diff options
| author | Lennart Weller <lhw@ring0.de> | 2016-09-05 08:27:21 +0000 |
|---|---|---|
| committer | Lennart Weller <lhw@ring0.de> | 2016-09-05 08:27:21 +0000 |
| commit | 1746898cefcb17f58b5cf27b4dad3d28236f1152 (patch) | |
| tree | 9207f191cf39bbd077a1e1c73d6e82123e2fc710 | |
| parent | Imported Upstream version 1.2.0+dfsg (diff) | |
| download | netdata-1746898cefcb17f58b5cf27b4dad3d28236f1152.tar.xz netdata-1746898cefcb17f58b5cf27b4dad3d28236f1152.zip | |
Imported Upstream version 1.3.0+dfsgupstream/1.3.0+dfsg
256 files changed, 62474 insertions, 33636 deletions
diff --git a/.gitignore b/.gitignore index 02801b778..c90442ce4 100644 --- a/.gitignore +++ b/.gitignore @@ -58,9 +58,11 @@ web/gadget.xml web/index_new.html web/version.txt +system/netdata-lsb system/netdata-openrc system/netdata-init-d system/netdata.logrotate +python.d/python-modules-installer.sh netdata-installer.log netdata-uninstaller.sh @@ -78,3 +80,5 @@ cmake_install.cmake contrib/debian/changelog profile/benchmark-dictionary profile/benchmark-registry + +*.pyc @@ -1,3 +1,124 @@ +netdata (1.3.0) - 2016-08-28 + + At a glance: + + - netdata has health monitoring / alarms! + - netdata has badges that can be embeded anywhere! + - netdata plugins are now written in python! + - new plugins: redis, memcached, nginx_log, ipfs, apache_cache + + IMPORTANT: + Since netdata now uses python plugins, new packages are + required to be installed on a system to allow it work. + For more information, [lease check the installation page: + + https://github.com/firehol/netdata/wiki/Installation + + In detail: + + * netdata has alarms! + + Based on the POLL we made on github + (https://github.com/firehol/netdata/issues/436), + health monitoring was the winner. So here it is! + + netdata now has a poweful health monitoring system embedded. + Please check the wiki page: + + https://github.com/firehol/netdata/wiki/health-monitoring + + * netdata has badges! + + netdata can generate badges with live information from the + collected metrics. + Please check the wiki page: + + https://github.com/firehol/netdata/wiki/Generating-Badges + + * netdata plugins are now written in python! + + Thanks to the great work of Paweł Krupa (@paulfantom), most BASH + plugins have been ported to python. + + The new python.d.plugin supports both python2 and python3 and + data collection from multiple sources for all modules. + + The following pre-existing modules have been ported to python: + + - apache + - cpufreq + - example + - exim + - hddtemp + - mysql + - nginx + - phpfm + - postfix + - sensors + - squid + - tomcat + + The following new modules have been added: + + - apache_cache + - dovecot + - ipfs + - memcached + - nginx_log + - redis + + * other data collectors: + + - Thanks to @simonnagl netdata now reports disk space usage. + + * dashboards now transfer a certain settings from server to server + when changing servers via the my-netdata menu. + + The settings transferred are the dashboard theme, the online + help status and current pan and zoom timeframe of the dashboard. + + * API improvements: + + - reduction functions now support 'min', 'sum' and 'incremental-sum'. + + - netdata now offers a multi-threaded and a single threaded + web server (single threaded is better for IoT). + + * apps.plugin improvements: + + - can now run with command line argument 'without-files' + to prevent it from enumating all the open files/sockets/pipes + of all running processes. + + - apps.plugin now scales the collected values to match the + the total system usage. + + - apps.plugin can now report guest CPU usage per process. + + - repeating errors are now logged once per process. + + * netdata now runs with IDLE process priority (lower than nice 19) + + * netdata now instructs the kernel to kill it first when it starves + for memory. + + * netdata listens for signals: + + - SIGHUP to netdata instructs it to re-open its log files + (new logrotate files added too). + + - SIGUSR1 to netdata saves the database + + - SIGUSR2 to netdata reloads health / alarms configuration + + * netdata can now bind to multiple IPs and ports. + + * netdata now has new systemd service file (it starts as user + netdata and does not fork). + + * Dozens of other improvements and bugfixes + + netdata (1.2.0) - 2016-05-16 At a glance: diff --git a/Dockerfile b/Dockerfile new file mode 100644 index 000000000..955befb54 --- /dev/null +++ b/Dockerfile @@ -0,0 +1,15 @@ +# author : titpetric +# original: https://github.com/titpetric/netdata + +FROM debian:jessie + +ADD docker-build.sh /docker-build.sh + +RUN chmod +x /docker-build.sh && sync && sleep 1 && /docker-build.sh + +WORKDIR / + +ENV NETDATA_PORT 19999 +EXPOSE $NETDATA_PORT + +CMD /usr/sbin/netdata -D -s /host -p ${NETDATA_PORT} diff --git a/LICENSE.md b/LICENSE.md index 380e86eee..9b6415d10 100644 --- a/LICENSE.md +++ b/LICENSE.md @@ -138,4 +138,12 @@ connectivity is not available. Copyright 2015, Joseph Huckaby [MIT License](https://github.com/jhuckaby/pixl-xml) - +- [sensors](https://github.com/paroj/sensors.py) + + Copyright 2014, Pavel Rojtberg + [LGPL 2.1 License](http://opensource.org/licenses/LGPL-2.1) + +- [PyYAML](https://bitbucket.org/blackjack/pysensors) + + Copyright 2006, Kirill Simonov + [MIT License](http://pyyaml.org) diff --git a/Makefile.am b/Makefile.am index dc109cbd1..3ed5e62bf 100644 --- a/Makefile.am +++ b/Makefile.am @@ -23,7 +23,6 @@ EXTRA_DIST = \ LICENSE.md \ COPYING \ autogen.sh \ - netdata-9999.ebuild \ tests/stress.sh \ $(NULL) @@ -32,14 +31,22 @@ SUBDIRS = \ conf.d \ node.d \ plugins.d \ + python.d \ src \ system \ web \ contrib \ $(NULL) -dist_noinst_DATA = netdata.spec +dist_noinst_DATA= \ + configs.signatures \ + Dockerfile \ + netdata.spec \ + $(NULL) # until integrated within build # should be proper init.d/openrc/systemd usable -dist_noinst_SCRIPTS = netdata-installer.sh +dist_noinst_SCRIPTS= \ + docker-build.sh \ + netdata-installer.sh \ + $(NULL) diff --git a/Makefile.in b/Makefile.in index 9cfa9bead..c4dd9b3ff 100644 --- a/Makefile.in +++ b/Makefile.in @@ -318,6 +318,8 @@ pluginsdir = @pluginsdir@ prefix = @prefix@ program_transform_name = @program_transform_name@ psdir = @psdir@ +pythondir = @pythondir@ +registrydir = @registrydir@ sbindir = @sbindir@ sharedstatedir = @sharedstatedir@ srcdir = @srcdir@ @@ -353,7 +355,6 @@ EXTRA_DIST = \ LICENSE.md \ COPYING \ autogen.sh \ - netdata-9999.ebuild \ tests/stress.sh \ $(NULL) @@ -362,17 +363,27 @@ SUBDIRS = \ conf.d \ node.d \ plugins.d \ + python.d \ src \ system \ web \ contrib \ $(NULL) -dist_noinst_DATA = netdata.spec +dist_noinst_DATA = \ + configs.signatures \ + Dockerfile \ + netdata.spec \ + $(NULL) + # until integrated within build # should be proper init.d/openrc/systemd usable -dist_noinst_SCRIPTS = netdata-installer.sh +dist_noinst_SCRIPTS = \ + docker-build.sh \ + netdata-installer.sh \ + $(NULL) + all: config.h $(MAKE) $(AM_MAKEFLAGS) all-recursive @@ -1,15 +1,40 @@ [](https://travis-ci.org/firehol/netdata) +<a href="https://scan.coverity.com/projects/firehol-netdata"><img alt="Coverity Scan Build Status" src="https://scan.coverity.com/projects/9140/badge.svg"/></a> +[](https://registry.my-netdata.io/#netdata_registry) +[](https://registry.my-netdata.io/#netdata_registry) +[](https://registry.my-netdata.io/#netdata_registry) + +[](https://registry.my-netdata.io/#netdata_registry) +[](https://registry.my-netdata.io/#netdata_registry) +[](https://registry.my-netdata.io/#netdata_registry) + # netdata + +> Aug 21st, 2016: Netdata got **[health monitoring](https://github.com/firehol/netdata/wiki/health-monitoring)** - alarms + +--- + +> May 16th, 2016 +> +> [netdata v1.2.0 released!](https://github.com/firehol/netdata/releases) +> +> - 30% faster! +> - **[netdata registry](https://github.com/firehol/netdata/wiki/mynetdata-menu-item)**, the first step towards scaling out performance monitoring! +> - real-time Linux Containers monitoring! +> - dozens of additional new features, optimizations, bug-fixes + +--- + +May 1st, 2016 + ##### 320.000+ views, 92.000+ visitors, 28.500+ downloads, 11.000+ github stars, 700+ forks, 1 month! And it still runs with 600+ git downloads... per day! **[Check what our users say about netdata](https://github.com/firehol/netdata/issues/148)**. -Thank you! - --- **Real-time performance monitoring, done right!** @@ -53,7 +78,7 @@ This is what it currently monitors (most with zero configuration): - **RAM, swap and kernel memory usage** (including KSM and kernel memory deduper) -- **Disks** (per disk: I/O, operations, backlog, utilization, etc) +- **Disks** (per disk: I/O, operations, backlog, utilization, space, etc)  @@ -87,17 +112,25 @@ This is what it currently monitors (most with zero configuration): - **Users and User Groups resource usage**, by summarizing the process tree per user and group (CPU, memory, disk reads, disk writes, swap, threads, pipes, sockets, etc) -- **Apache web server** mod-status (v2.2, v2.4) +- **Apache web server** mod-status (v2.2, v2.4) and cache log statistics (multiple servers) -- **Nginx web server** stub-status +- **Nginx web server** stub-status (multiple servers) - **mySQL databases** (multiple servers, each showing: bandwidth, queries/s, handlers, locks, issues, tmp operations, connections, binlog metrics, threads, innodb metrics, etc) +- **Redis databases** (multiple servers, each showing: operations, hit rate, memory, keys, clients, slaves) + +- **memcached databases** (multiple servers, each showing: bandwidth, connections, items, etc) + - **ISC Bind name server** (multiple servers, each showing: clients, requests, queries, updates, failures and several per view metrics) - **Postfix email server** message queue (entries, size) -- **Squid proxy server** (clients bandwidth and requests, servers bandwidth and requests) +- **exim email server** message queue (emails queued) + +- **IPFS** (Bandwidth, Peers) + +- **Squid proxy server** (multiple servers, each showing: clients bandwidth and requests, servers bandwidth and requests) - **Hardware sensors** (temperature, voltage, fans, power, humidity, etc) @@ -107,6 +140,8 @@ This is what it currently monitors (most with zero configuration): - **PHP-FPM** (multiple instances, each reporting connections, requests, performance) +- **hddtemp** (disk temperatures) + - **SNMP devices** can be monitored too (although you will need to configure these) And you can extend it, by writing plugins that collect data from any source, using any computer language. @@ -140,4 +175,3 @@ It should run on **any Linux** system. It has been tested on: ## Documentation Check the **[netdata wiki](https://github.com/firehol/netdata/wiki)**. - diff --git a/build/subst.inc b/build/subst.inc index 99cac7f4f..9682cf882 100644 --- a/build/subst.inc +++ b/build/subst.inc @@ -3,6 +3,7 @@ -e 's#[@]localstatedir_POST@#$(localstatedir)#g' \ -e 's#[@]sbindir_POST@#$(sbindir)#g' \ -e 's#[@]sysconfdir_POST@#$(sysconfdir)#g' \ + -e 's#[@]pythondir_POST@#$(pythondir)#g' \ $< > $@.tmp; then \ mv "$@.tmp" "$@"; \ else \ diff --git a/charts.d/Makefile.am b/charts.d/Makefile.am index ad11e972a..e131d508f 100644 --- a/charts.d/Makefile.am +++ b/charts.d/Makefile.am @@ -4,13 +4,13 @@ MAINTAINERCLEANFILES= $(srcdir)/Makefile.in dist_charts_SCRIPTS = \ - airsearches.chart.sh \ ap.chart.sh \ apache.chart.sh \ cpu_apps.chart.sh \ cpufreq.chart.sh \ - crsproxy.chart.sh \ example.chart.sh \ + exim.chart.sh \ + hddtemp.chart.sh \ load_average.chart.sh \ mem_apps.chart.sh \ mysql.chart.sh \ diff --git a/charts.d/Makefile.in b/charts.d/Makefile.in index 3aff5a94c..5dfb3cc4b 100644 --- a/charts.d/Makefile.in +++ b/charts.d/Makefile.in @@ -263,6 +263,8 @@ pluginsdir = @pluginsdir@ prefix = @prefix@ program_transform_name = @program_transform_name@ psdir = @psdir@ +pythondir = @pythondir@ +registrydir = @registrydir@ sbindir = @sbindir@ sharedstatedir = @sharedstatedir@ srcdir = @srcdir@ @@ -279,13 +281,13 @@ webdir = @webdir@ # MAINTAINERCLEANFILES = $(srcdir)/Makefile.in dist_charts_SCRIPTS = \ - airsearches.chart.sh \ ap.chart.sh \ apache.chart.sh \ cpu_apps.chart.sh \ cpufreq.chart.sh \ - crsproxy.chart.sh \ example.chart.sh \ + exim.chart.sh \ + hddtemp.chart.sh \ load_average.chart.sh \ mem_apps.chart.sh \ mysql.chart.sh \ diff --git a/charts.d/README.md b/charts.d/README.md index fd66c0d6a..37c9d22ec 100644 --- a/charts.d/README.md +++ b/charts.d/README.md @@ -181,6 +181,36 @@ sensors_source_update=1 # how frequently to collect sensor data # the default is to collect it at every iteration of charts.d sensors_update_every= + +# array of sensors which are excluded +# the default is to include all +sensors_excluded=() +``` + +--- + +# hddtemp + +The plugin will collect temperatures from disks + +It will create one chart with all active disks + +1. **temperature in Celsius** + +### configuration + +hddtemp needs to be running in daemonized mode + +```sh +# host with daemonized hddtemp +hddtemp_host="localhost" + +# port on which hddtemp is showing data +hddtemp_port="7634" + +# array of included disks +# the default is to include all +hddtemp_disks=() ``` --- diff --git a/charts.d/airsearches.chart.sh b/charts.d/airsearches.chart.sh deleted file mode 100755 index 449b14255..000000000 --- a/charts.d/airsearches.chart.sh +++ /dev/null @@ -1,91 +0,0 @@ -#!/bin/sh - -airsearches_url= -airsearches_cmds= -airsearches_update_every=15 - -airsearches_get() { - wget 2>/dev/null -O - "$airsearches_url" |\ - sed -e "s|<br />|\n|g" -e "s|: |=|g" -e "s| \+|_|g" -e "s/^/airsearches_/g" |\ - tr "[A-Z]\.\!@#\$%^&*()_+\-" "[a-z]_____________" |\ - egrep "^airsearches_[a-z0-9_]+=[0-9]+$" -} - -airsearches_check() { - # make sure we have all the commands we need - require_cmd wget || return 1 - - # make sure we are configured - if [ -z "$airsearches_url" ] - then - echo >&2 "$PROGRAM_NAME: airsearches: not configured. Please set airsearches_url='url' in $confd/airsearches.conf" - return 1 - fi - - # check once if the url works - wget 2>/dev/null -O /dev/null "$airsearches_url" - if [ ! $? -eq 0 ] - then - echo >&2 "$PROGRAM_NAME: airsearches: cannot fetch the url: $airsearches_url. Please set airsearches_url='url' in $confd/airsearches.conf" - return 1 - fi - - # if the admin did not give any commands - # find the available ones - if [ -z "$airsearches_cmds" ] - then - airsearches_cmds="$(airsearches_get | cut -d '=' -f 1 | sed "s/^airsearches_//g" | sort -u)" - echo - fi - - # did we find any commands? - if [ -z "$airsearches_cmds" ] - then - echo >&2 "$PROGRAM_NAME: airsearches: cannot find command list automatically. Please set airsearches_cmds='...' in $confd/airsearches.conf" - return 1 - fi - - # ok we can do it - return 0 -} - -airsearches_create() { - [ -z "$airsearches_cmds" ] && return 1 - - # create the charts - local x= - echo "CHART airsearches.affiliates '' 'Air Searches per affiliate' 'requests / min' airsearches '' stacked 20000 $airsearches_update_every" - for x in $airsearches_cmds - do - echo "DIMENSION $x '' incremental 60 1" - done - - return 0 -} - -airsearches_update() { - # the first argument to this function is the microseconds since last update - # pass this parameter to the BEGIN statement (see bellow). - - # do all the work to collect / calculate the values - # for each dimension - # remember: KEEP IT SIMPLE AND SHORT - - # get the values from airsearches - eval "$(airsearches_get)" - - # write the result of the work. - local x= - - echo "BEGIN airsearches.affiliates $1" - for x in $airsearches_cmds - do - eval "v=\$airsearches_$x" - echo "SET $x = $v" - done - echo "END" - - airsearches_dt=0 - - return 0 -} diff --git a/charts.d/ap.chart.sh b/charts.d/ap.chart.sh index aed51c1b6..7b4f690bb 100755 --- a/charts.d/ap.chart.sh +++ b/charts.d/ap.chart.sh @@ -1,4 +1,4 @@ -#!/bin/bash +# no need for shebang - this file is loaded from charts.d.plugin # _update_every is a special variable - it holds the number of seconds # between the calls of the _update() function @@ -11,6 +11,8 @@ export PATH="${PATH}:/sbin:/usr/sbin:/usr/local/sbin" # _check is called once, to find out if this chart should be enabled or not ap_check() { + require_cmd iw || return 1 + local ev=$(iw dev | awk ' BEGIN { i = ""; diff --git a/charts.d/apache.chart.sh b/charts.d/apache.chart.sh index dbf14a432..2d68d43b2 100755 --- a/charts.d/apache.chart.sh +++ b/charts.d/apache.chart.sh @@ -1,7 +1,8 @@ -#!/bin/bash +# no need for shebang - this file is loaded from charts.d.plugin # the URL to download apache status info apache_url="http://127.0.0.1:80/server-status?auto" +apache_curl_opts= # _update_every is a special variable - it holds the number of seconds # between the calls of the _update() function @@ -66,19 +67,14 @@ apache_detect() { # we will not check of the Conns* # keys, since these are apache 2.4 specific - if [ -z "${apache_key_accesses}" \ - -o -z "${apache_key_kbytes}" \ - -o -z "${apache_key_reqpersec}" \ - -o -z "${apache_key_bytespersec}" \ - -o -z "${apache_key_bytesperreq}" \ - -o -z "${apache_key_busyworkers}" \ - -o -z "${apache_key_idleworkers}" \ - -o -z "${apache_key_scoreboard}" \ - ] - then - echo >&2 "apache: Invalid response or missing keys from apache server: ${*}" - return 1 - fi + [ -z "${apache_key_accesses}" ] && echo >&2 "apache: missing 'Total Accesses' from apache server: ${*}" && return 1 + [ -z "${apache_key_kbytes}" ] && echo >&2 "apache: missing 'Total kBytes' from apache server: ${*}" && return 1 + [ -z "${apache_key_reqpersec}" ] && echo >&2 "apache: missing 'ReqPerSec' from apache server: ${*}" && return 1 + [ -z "${apache_key_bytespersec}" ] && echo >&2 "apache: missing 'BytesPerSec' from apache server: ${*}" && return 1 + [ -z "${apache_key_bytesperreq}" ] && echo >&2 "apache: missing 'BytesPerReq' from apache server: ${*}" && return 1 + [ -z "${apache_key_busyworkers}" ] && echo >&2 "apache: missing 'BusyWorkers' from apache server: ${*}" && return 1 + [ -z "${apache_key_idleworkers}" ] && echo >&2 "apache: missing 'IdleWorkers' from apache server: ${*}" && return 1 + [ -z "${apache_key_scoreboard}" ] && echo >&2 "apache: missing 'Scoreboard' from apache server: ${*}" && return 1 if [ ! -z "${apache_key_connstotal}" \ -a ! -z "${apache_key_connsasyncwriting}" \ @@ -87,6 +83,8 @@ apache_detect() { ] then apache_has_conns=1 + else + apache_has_conns=0 fi return 0 @@ -94,7 +92,7 @@ apache_detect() { apache_get() { local oIFS="${IFS}" ret - IFS=$':\n' apache_response=($(curl -Ss "${apache_url}")) + IFS=$':\n' apache_response=($(curl -Ss ${apache_curl_opts} "${apache_url}")) ret=$? IFS="${oIFS}" @@ -167,27 +165,27 @@ apache_check() { # _create is called once, to create the charts apache_create() { cat <<EOF -CHART apache.bytesperreq '' "apache Lifetime Avg. Response Size" "bytes/request" statistics apache.bytesperreq area $((apache_priority + 8)) $apache_update_every +CHART apache_local.bytesperreq '' "apache Lifetime Avg. Response Size" "bytes/request" statistics apache.bytesperreq area $((apache_priority + 8)) $apache_update_every DIMENSION size '' absolute 1 ${apache_decimal_detail} -CHART apache.workers '' "apache Workers" "workers" workers apache.workers stacked $((apache_priority + 5)) $apache_update_every +CHART apache_local.workers '' "apache Workers" "workers" workers apache.workers stacked $((apache_priority + 5)) $apache_update_every DIMENSION idle '' absolute 1 1 DIMENSION busy '' absolute 1 1 -CHART apache.reqpersec '' "apache Lifetime Avg. Requests/s" "requests/s" statistics apache.reqpersec line $((apache_priority + 6)) $apache_update_every +CHART apache_local.reqpersec '' "apache Lifetime Avg. Requests/s" "requests/s" statistics apache.reqpersec line $((apache_priority + 6)) $apache_update_every DIMENSION requests '' absolute 1 ${apache_decimal_detail} -CHART apache.bytespersec '' "apache Lifetime Avg. Bandwidth/s" "kilobits/s" statistics apache.bytespersec area $((apache_priority + 7)) $apache_update_every +CHART apache_local.bytespersec '' "apache Lifetime Avg. Bandwidth/s" "kilobits/s" statistics apache.bytespersec area $((apache_priority + 7)) $apache_update_every DIMENSION sent '' absolute 8 $((apache_decimal_detail * 1000)) -CHART apache.requests '' "apache Requests" "requests/s" requests apache.requests line $((apache_priority + 1)) $apache_update_every +CHART apache_local.requests '' "apache Requests" "requests/s" requests apache.requests line $((apache_priority + 1)) $apache_update_every DIMENSION requests '' incremental 1 1 -CHART apache.net '' "apache Bandwidth" "kilobits/s" bandwidth apache.net area $((apache_priority + 3)) $apache_update_every +CHART apache_local.net '' "apache Bandwidth" "kilobits/s" bandwidth apache.net area $((apache_priority + 3)) $apache_update_every DIMENSION sent '' incremental 8 1 EOF if [ ${apache_has_conns} -eq 1 ] then cat <<EOF2 -CHART apache.connections '' "apache Connections" "connections" connections apache.connections line $((apache_priority + 2)) $apache_update_every +CHART apache_local.connections '' "apache Connections" "connections" connections apache.connections line $((apache_priority + 2)) $apache_update_every DIMENSION connections '' absolute 1 1 -CHART apache.conns_async '' "apache Async Connections" "connections" connections apache.conns_async stacked $((apache_priority + 4)) $apache_update_every +CHART apache_local.conns_async '' "apache Async Connections" "connections" connections apache.conns_async stacked $((apache_priority + 4)) $apache_update_every DIMENSION keepalive '' absolute 1 1 DIMENSION closing '' absolute 1 1 DIMENSION writing '' absolute 1 1 @@ -211,22 +209,22 @@ apache_update() { # write the result of the work. cat <<VALUESEOF -BEGIN apache.requests $1 +BEGIN apache_local.requests $1 SET requests = $((apache_accesses)) END -BEGIN apache.net $1 +BEGIN apache_local.net $1 SET sent = $((apache_kbytes)) END -BEGIN apache.reqpersec $1 +BEGIN apache_local.reqpersec $1 SET requests = $((apache_reqpersec)) END -BEGIN apache.bytespersec $1 +BEGIN apache_local.bytespersec $1 SET sent = $((apache_bytespersec)) END -BEGIN apache.bytesperreq $1 +BEGIN apache_local.bytesperreq $1 SET size = $((apache_bytesperreq)) END -BEGIN apache.workers $1 +BEGIN apache_local.workers $1 SET idle = $((apache_idleworkers)) SET busy = $((apache_busyworkers)) END @@ -235,12 +233,12 @@ VALUESEOF if [ ${apache_has_conns} -eq 1 ] then cat <<VALUESEOF2 -BEGIN apache.connections $1 +BEGIN apache_local.connections $1 SET connections = $((apache_connstotal)) END -BEGIN apache.conns_async $1 +BEGIN apache_local.conns_async $1 SET keepalive = $((apache_connsasynckeepalive)) -SET closing = $((apache_connsasyncwriting)) +SET closing = $((apache_connsasyncclosing)) SET writing = $((apache_connsasyncwriting)) END VALUESEOF2 diff --git a/charts.d/cpu_apps.chart.sh b/charts.d/cpu_apps.chart.sh index 5a25163e1..6b2513dcf 100755 --- a/charts.d/cpu_apps.chart.sh +++ b/charts.d/cpu_apps.chart.sh @@ -1,4 +1,4 @@ -#!/bin/sh +# no need for shebang - this file is loaded from charts.d.plugin # THIS PLUGIN IS OBSOLETE # USE apps.plugin INSTEAD diff --git a/charts.d/cpufreq.chart.sh b/charts.d/cpufreq.chart.sh index 008ffe1d7..06f692fa6 100755 --- a/charts.d/cpufreq.chart.sh +++ b/charts.d/cpufreq.chart.sh @@ -1,4 +1,4 @@ -#!/bin/sh +# no need for shebang - this file is loaded from charts.d.plugin # if this chart is called X.chart.sh, then all functions and global variables # must start with X_ @@ -54,7 +54,7 @@ cpufreq_create() { echo >&2 "charts.d: cpufreq: on file='$file', dir='$dir', cpu='$cpu', id='$id'" echo "DIMENSION $id '$id' absolute 1 1000" - echo >>$TMP_DIR/cpufreq.sh "printf \"SET $id = \"; cat $file " + echo >>$TMP_DIR/cpufreq.sh "echo \"SET $id = \"\$(< $file )" done echo >>$TMP_DIR/cpufreq.sh "echo END" diff --git a/charts.d/crsproxy.chart.sh b/charts.d/crsproxy.chart.sh deleted file mode 100755 index 9ad8b3382..000000000 --- a/charts.d/crsproxy.chart.sh +++ /dev/null @@ -1,151 +0,0 @@ -#!/bin/sh - -crsproxy_url= -crsproxy_cmds= -crsproxy_update_every=15 - -crsproxy_get() { - wget 2>/dev/null -O - "$crsproxy_url" |\ - sed \ - -e "s/ \+/ /g" \ - -e "s/\./_/g" \ - -e "s/ =/=/g" \ - -e "s/= /=/g" \ - -e "s/^/crsproxy_/g" |\ - egrep "^crsproxy_[a-zA-Z][a-zA-Z0-9_]*=[0-9]+$" -} - -crsproxy_check() { - # make sure we have all the commands we need - require_cmd wget || return 1 - - if [ -z "$crsproxy_url" ] - then - echo >&2 "$PROGRAM_NAME: crsproxy: not configured. Please set crsproxy_url='url' in $confd/crsproxy.conf" - return 1 - fi - - # check once if the url works - wget 2>/dev/null -O /dev/null "$crsproxy_url" - if [ ! $? -eq 0 ] - then - echo >&2 "$PROGRAM_NAME: crsproxy: cannot fetch the url: $crsproxy_url. Please set crsproxy_url='url' in $confd/crsproxy.conf" - return 1 - fi - - # if the user did not request specific commands - # find the commands available - if [ -z "$crsproxy_cmds" ] - then - crsproxy_cmds="$(crsproxy_get | cut -d '=' -f 1 | sed "s/^crsproxy_cmd_//g" | sort -u)" - fi - - # if no commands are available - if [ -z "$crsproxy_cmds" ] - then - echo >&2 "$PROGRAM_NAME: crsproxy: cannot find command list automatically. Please set crsproxy_cmds='...' in $confd/crsproxy.conf" - return 1 - fi - return 0 -} - -crsproxy_create() { - # create the charts - cat <<EOF -CHART crsproxy.connected '' "CRS Proxy Connected Clients" "clients" crsproxy '' line 20000 $crsproxy_update_every -DIMENSION web '' absolute 1 1 -DIMENSION native '' absolute 1 1 -DIMENSION virtual '' absolute 1 1 -CHART crsproxy.requests '' "CRS Proxy Requests Rate" "requests / min" crsproxy '' area 20001 $crsproxy_update_every -DIMENSION web '' incremental 60 1 -DIMENSION native '' incremental -60 1 -CHART crsproxy.clients '' "CRS Proxy Clients Rate" "clients / min" crsproxy '' area 20010 $crsproxy_update_every -DIMENSION web '' incremental 60 1 -DIMENSION native '' incremental -60 1 -DIMENSION virtual '' incremental 60 1 -CHART crsproxy.replies '' "CRS Replies Rate" "replies / min" crsproxy '' area 20020 $crsproxy_update_every -DIMENSION ok '' incremental 60 1 -DIMENSION failed '' incremental -60 1 -CHART crsproxy.bconnections '' "Back-End Connections Rate" "connections / min" crsproxy '' area 20030 $crsproxy_update_every -DIMENSION ok '' incremental 60 1 -DIMENSION failed '' incremental -60 1 -EOF - - local x= - echo "CHART crsproxy.commands '' 'CRS Commands Requests' 'requests / min' crsproxy '' stacked 20100 $crsproxy_update_every" - for x in $crsproxy_cmds - do - echo "DIMENSION $x '' incremental 60 $crsproxy_update_every" - done - - echo "CHART crsproxy.commands_failed '' 'CRS Failed Commands' 'replies / min' crsproxy '' stacked 20110 $crsproxy_update_every" - for x in $crsproxy_cmds - do - echo "DIMENSION $x '' incremental 60 $crsproxy_update_every" - done - - return 0 -} - - -crsproxy_update() { - # the first argument to this function is the microseconds since last update - # pass this parameter to the BEGIN statement (see bellow). - - # do all the work to collect / calculate the values - # for each dimension - # remember: KEEP IT SIMPLE AND SHORT - - # get the values from crsproxy - eval "$(crsproxy_get)" - - - # write the result of the work. - cat <<VALUESEOF -BEGIN crsproxy.connected $1 -SET web = $((crsproxy_web_clients_opened - crsproxy_web_clients_closed)) -SET native = $((crsproxy_crs_clients_opened - crsproxy_crs_clients_closed)) -SET virtual = $((crsproxy_virtual_clients_opened - crsproxy_virtual_clients_closed)) -END -BEGIN crsproxy.requests $1 -SET web = $crsproxy_web_requests -SET native = $crsproxy_native_requests -END -BEGIN crsproxy.clients $1 -SET web = $crsproxy_web_clients_opened -SET native = $crsproxy_crs_clients_opened -SET virtual = $crsproxy_virtual_clients_opened -END -BEGIN crsproxy.replies $1 -SET ok = $crsproxy_replies_success -SET failed = $crsproxy_replies_error -END -BEGIN crsproxy.bconnections $1 -SET ok = $crsproxy_connections_nonblocking_established -SET failed = $crsproxy_connections_nonblocking_failed -END -VALUESEOF - - local native_requests="_native_requests" - local web_requests="_web_requests" - local replies_error="_replies_error" - local x= - - echo "BEGIN crsproxy.commands $1" - for x in $crsproxy_cmds - do - eval "v=\$(( crsproxy_cmd_$x$native_requests + crsproxy_cmd_$x$web_requests ))" - echo "SET $x = $v" - done - echo "END" - - echo "BEGIN crsproxy.commands_failed $1" - for x in $crsproxy_cmds - do - eval "v=\$crsproxy_cmd_$x$replies_error" - echo "SET $x = $v" - done - echo "END" - - return 0 -} diff --git a/charts.d/example.chart.sh b/charts.d/example.chart.sh index ad2050462..1f020dee2 100755 --- a/charts.d/example.chart.sh +++ b/charts.d/example.chart.sh @@ -1,4 +1,4 @@ -#!/bin/bash +# no need for shebang - this file is loaded from charts.d.plugin # if this chart is called X.chart.sh, then all functions and global variables # must start with X_ @@ -7,14 +7,71 @@ # between the calls of the _update() function example_update_every= +# the priority is used to sort the charts on the dashboard +# 1 = the first chart example_priority=150000 +# to enable this chart, you have to set this to 12345 +# (just a demonstration for something that needs to be checked) +example_magic_number= + +# global variables to store our collected data +# remember: they need to start with the module name example_ +example_value1= +example_value2= +example_value3= +example_value4= +example_last=0 +example_count=0 + +example_get() { + # do all the work to collect / calculate the values + # for each dimension + # + # Remember: + # 1. KEEP IT SIMPLE AND SHORT + # 2. AVOID FORKS (avoid piping commands) + # 3. AVOID CALLING TOO MANY EXTERNAL PROGRAMS + # 4. USE LOCAL VARIABLES (global variables may overlap with other modules) + + example_value1=$RANDOM + example_value2=$RANDOM + example_value3=$RANDOM + example_value4=$((8192 + (RANDOM * 16383 / 32767) )) + + if [ $example_count -gt 0 ] + then + example_count=$((example_count - 1)) + + [ $example_last -gt 16383 ] && example_value4=$((example_last + (RANDOM * ( (32767 - example_last) / 2) / 32767))) + [ $example_last -le 16383 ] && example_value4=$((example_last - (RANDOM * (example_last / 2) / 32767))) + else + example_count=$((1 + (RANDOM * 5 / 32767) )) + + [ $example_last -gt 16383 -a $example_value4 -gt 16383 ] && example_value4=$((value4 - 16383)) + [ $example_last -le 16383 -a $example_value4 -lt 16383 ] && example_value4=$((value4 + 16383)) + fi + example_last=$example_value4 + + # this should return: + # - 0 to send the data to netdata + # - 1 to report a failure to collect the data + + return 0 +} + # _check is called once, to find out if this chart should be enabled or not example_check() { # this should return: # - 0 to enable the chart # - 1 to disable the chart + # check something + [ "${example_magic_number}" != "12345" ] && echo >&2 "example: you have to set example_magic_number=$example_magic_number in example.conf to start example chart." && return 1 + + # check that we can collect data + example_get || return 1 + return 0 } @@ -34,46 +91,21 @@ EOF } # _update is called continiously, to collect the values -example_last=0 -example_count=0 example_update() { - local value1 value2 value3 value4 mode - # the first argument to this function is the microseconds since last update # pass this parameter to the BEGIN statement (see bellow). - # do all the work to collect / calculate the values - # for each dimension - # remember: KEEP IT SIMPLE AND SHORT - - value1=$RANDOM - value2=$RANDOM - value3=$RANDOM - value4=$((8192 + (RANDOM * 16383 / 32767) )) - - if [ $example_count -gt 0 ] - then - example_count=$((example_count - 1)) - - [ $example_last -gt 16383 ] && value4=$((example_last + (RANDOM * ( (32767 - example_last) / 2) / 32767))) - [ $example_last -le 16383 ] && value4=$((example_last - (RANDOM * (example_last / 2) / 32767))) - else - example_count=$((1 + (RANDOM * 5 / 32767) )) - - [ $example_last -gt 16383 -a $value4 -gt 16383 ] && value4=$((value4 - 16383)) - [ $example_last -le 16383 -a $value4 -lt 16383 ] && value4=$((value4 + 16383)) - fi - example_last=$value4 + example_get || return 1 # write the result of the work. cat <<VALUESEOF BEGIN example.random $1 -SET random1 = $value1 -SET random2 = $value2 -SET random3 = $value3 +SET random1 = $example_value1 +SET random2 = $example_value2 +SET random3 = $example_value3 END BEGIN example.random2 $1 -SET random = $value4 +SET random = $example_value4 END VALUESEOF # echo >&2 "example_count = $example_count value = $value4" diff --git a/charts.d/exim.chart.sh b/charts.d/exim.chart.sh new file mode 100644 index 000000000..c60ae9460 --- /dev/null +++ b/charts.d/exim.chart.sh @@ -0,0 +1,52 @@ +# no need for shebang - this file is loaded from charts.d.plugin + +exim_command= + +# how frequently to collect queue size +exim_update_every=5 + +exim_priority=60000 + +exim_check() { + if [ -z "$exim_command" -o ! -x "$exim_command" ] + then + local d= + for d in /sbin /usr/sbin /usr/local/sbin + do + if [ -x "$d/exim" ] + then + exim_command="$d/exim" + break + fi + done + fi + + if [ -z "$exim_command" -o ! -x "$exim_command" ] + then + echo >&2 "$PROGRAM_NAME: exim: cannot find exim executable. Please set 'exim_command=/path/to/exim' in $confd/exim.conf" + return 1 + fi + + if [ `$exim_command -bpc 2>&1 | grep -c denied` -ne 0 ] + then + echo >&2 "$PROGRAM_NAME: exim: permission denied. Please set 'queue_list_requires_admin = false' in your exim options file" + return 1 + fi + + return 0 +} + +exim_create() { +cat <<EOF +CHART exim_local.qemails '' "Exim Queue Emails" "emails" queue exim.queued.emails line $((exim_priority + 1)) $exim_update_every +DIMENSION emails '' absolute 1 1 +EOF +return 0 +} + +exim_update() { +echo "BEGIN exim_local.qemails $1" +echo "SET emails = " `$exim_command -bpc` +echo "END" +return 0 +} diff --git a/charts.d/hddtemp.chart.sh b/charts.d/hddtemp.chart.sh new file mode 100755 index 000000000..41c3e2478 --- /dev/null +++ b/charts.d/hddtemp.chart.sh @@ -0,0 +1,66 @@ +# no need for shebang - this file is loaded from charts.d.plugin + +# if this chart is called X.chart.sh, then all functions and global variables +# must start with X_ +hddtemp_host="localhost" +hddtemp_port="7634" +declare -A hddtemp_disks=() + +# _update_every is a special variable - it holds the number of seconds +# between the calls of the _update() function +hddtemp_update_every=3 +hddtemp_priority=90000 + +# _check is called once, to find out if this chart should be enabled or not +hddtemp_check() { + nc $hddtemp_host $hddtemp_port &>/dev/null && return 0 || return 1 +} + +# _create is called once, to create the charts +hddtemp_create() { + if [ ${#hddtemp_disks[@]} -eq 0 ]; then + local all + all=$(nc $hddtemp_host $hddtemp_port ) + unset hddtemp_disks + hddtemp_disks=( `grep -Po '/dev/[^|]+' <<< "$all" | cut -c 6-` ) + fi +# local disk_names +# disk_names=(`sed -e 's/||/\n/g;s/^|//' <<< "$all" | cut -d '|' -f2 | tr ' ' '_'`) + + echo "CHART hddtemp.temperature 'disks_temp' 'temperature' 'Celsius' 'Disks temperature' 'hddtemp.temp' line $((hddtemp_priority)) $hddtemp_update_every" + for i in `seq 0 $((${#hddtemp_disks[@]}-1))`; do +# echo "DIMENSION ${hddtemp_disks[i]} ${disk_names[i]} absolute 1 1" + echo "DIMENSION ${hddtemp_disks[$i]} '' absolute 1 1" + done + return 0 +} + +# _update is called continiously, to collect the values +hddtemp_last=0 +hddtemp_count=0 +hddtemp_update() { +# local all=( `nc $hddtemp_host $hddtemp_port | sed -e 's/||/\n/g;s/^|//' | cut -d '|' -f3` ) +# local all=( `nc $hddtemp_host $hddtemp_port | awk 'BEGIN { FS="|" };{i=4; while (i <= NF) {print $i+0;i+=5;};}'` ) + OLD_IFS=$IFS + set -f + IFS="|" all=( $(nc $hddtemp_host $hddtemp_port 2>/dev/null) ) + set +f + IFS=$OLD_IFS + + # check if there is some data + if [ -z "${all[3]}" ]; then + return 1 + fi + + # write the result of the work. + echo "BEGIN hddtemp.temperature $1" + end=${#hddtemp_disks[@]} + for ((i=0; i<end; i++)); do + # temperature - this will turn SLP to zero + t=$(( ${all[ $((i * 5 + 3)) ]} )) + echo "SET ${hddtemp_disks[$i]} = $t" + done + echo "END" + + return 0 +} diff --git a/charts.d/load_average.chart.sh b/charts.d/load_average.chart.sh index 4d86a8f4c..e6790d807 100755 --- a/charts.d/load_average.chart.sh +++ b/charts.d/load_average.chart.sh @@ -1,4 +1,4 @@ -#!/bin/sh +# no need for shebang - this file is loaded from charts.d.plugin load_average_update_every=5 load_priority=100 diff --git a/charts.d/mem_apps.chart.sh b/charts.d/mem_apps.chart.sh index f537ada48..ab95b361c 100755 --- a/charts.d/mem_apps.chart.sh +++ b/charts.d/mem_apps.chart.sh @@ -1,4 +1,4 @@ -#!/bin/sh +# no need for shebang - this file is loaded from charts.d.plugin mem_apps_apps= diff --git a/charts.d/mysql.chart.sh b/charts.d/mysql.chart.sh index 56dce42d7..120fec66e 100755 --- a/charts.d/mysql.chart.sh +++ b/charts.d/mysql.chart.sh @@ -1,4 +1,4 @@ -#!/bin/bash +# no need for shebang - this file is loaded from charts.d.plugin # http://dev.mysql.com/doc/refman/5.0/en/server-status-variables.html # @@ -7,139 +7,31 @@ # This statement does not require any privilege. # It requires only the ability to connect to the server. -mysql_update_every=5 +mysql_update_every=2 mysql_priority=60000 -declare -A mysql_cmds=() mysql_opts=() mysql_ids=() +declare -A mysql_cmds=() mysql_opts=() mysql_ids=() mysql_data=() -mysql_exec() { - local ret - - "${@}" -s -e "show global status;" - ret=$? +mysql_get() { + local arr + local oIFS="${IFS}" + mysql_data=() + IFS=$'\t'$'\n' + #arr=($("${@}" -e "SHOW GLOBAL STATUS WHERE value REGEXP '^[0-9]';" | egrep "^(Bytes|Slow_|Que|Handl|Table|Selec|Sort_|Creat|Conne|Abort|Binlo|Threa|Innod|Qcach|Key_|Open)" )) + #arr=($("${@}" -N -e "SHOW GLOBAL STATUS;" | egrep "^(Bytes|Slow_|Que|Handl|Table|Selec|Sort_|Creat|Conne|Abort|Binlo|Threa|Innod|Qcach|Key_|Open)[^ ]+\s[0-9]" )) + arr=($("${@}" -N -e "SHOW GLOBAL STATUS;" | egrep "^(Bytes|Slow_|Que|Handl|Table|Selec|Sort_|Creat|Conne|Abort|Binlo|Threa|Innod|Qcach|Key_|Open)[^[:space:]]+[[:space:]]+[0-9]+" )) + IFS="${oIFS}" + + [ "${#arr[@]}" -lt 3 ] && return 1 + local end=${#arr[@]} + for ((i=2;i<end;i+=2)); do + mysql_data["${arr[$i]}"]=${arr[$i+1]} + done - [ $ret -ne 0 ] && echo "plugin_command_failure $ret" - return $ret -} + [ -z "${mysql_data[Connections]}" ] && return 1 -mysql_get() { - unset \ - mysql_Bytes_received \ - mysql_Bytes_sent \ - mysql_Queries \ - mysql_Questions \ - mysql_Slow_queries \ - mysql_Handler_commit \ - mysql_Handler_delete \ - mysql_Handler_prepare \ - mysql_Handler_read_first \ - mysql_Handler_read_key \ - mysql_Handler_read_next \ - mysql_Handler_read_prev \ - mysql_Handler_read_rnd \ - mysql_Handler_read_rnd_next \ - mysql_Handler_rollback \ - mysql_Handler_savepoint \ - mysql_Handler_savepoint_rollback \ - mysql_Handler_update \ - mysql_Handler_write \ - mysql_Table_locks_immediate \ - mysql_Table_locks_waited \ - mysql_Select_full_join \ - mysql_Select_full_range_join \ - mysql_Select_range \ - mysql_Select_range_check \ - mysql_Select_scan \ - mysql_Sort_merge_passes \ - mysql_Sort_range \ - mysql_Sort_scan \ - mysql_Created_tmp_disk_tables \ - mysql_Created_tmp_files \ - mysql_Created_tmp_tables \ - mysql_Connection_errors_accept \ - mysql_Connection_errors_internal \ - mysql_Connection_errors_max_connections \ - mysql_Connection_errors_peer_addr \ - mysql_Connection_errors_select \ - mysql_Connection_errors_tcpwrap \ - mysql_Connections \ - mysql_Aborted_connects \ - mysql_Binlog_cache_disk_use \ - mysql_Binlog_cache_use \ - mysql_Binlog_stmt_cache_disk_use \ - mysql_Binlog_stmt_cache_use \ - mysql_Threads_connected \ - mysql_Threads_created \ - mysql_Threads_cached \ - mysql_Threads_running \ - mysql_Innodb_data_read \ - mysql_Innodb_data_written \ - mysql_Innodb_data_reads \ - mysql_Innodb_data_writes \ - mysql_Innodb_data_fsyncs \ - mysql_Innodb_data_pending_reads \ - mysql_Innodb_data_pending_writes \ - mysql_Innodb_data_pending_fsyncs \ - mysql_Innodb_log_waits \ - mysql_Innodb_log_write_requests \ - mysql_Innodb_log_writes \ - mysql_Innodb_os_log_fsyncs \ - mysql_Innodb_os_log_pending_fsyncs \ - mysql_Innodb_os_log_pending_writes \ - mysql_Innodb_os_log_written \ - mysql_Innodb_row_lock_current_waits \ - mysql_Innodb_rows_inserted \ - mysql_Innodb_rows_read \ - mysql_Innodb_rows_updated \ - mysql_Innodb_rows_deleted \ - mysql_Innodb_buffer_pool_pages_data \ - mysql_Innodb_buffer_pool_pages_dirty \ - mysql_Innodb_buffer_pool_pages_flushed \ - mysql_Innodb_buffer_pool_pages_free \ - mysql_Innodb_buffer_pool_pages_misc \ - mysql_Innodb_buffer_pool_pages_total \ - mysql_Innodb_buffer_pool_bytes_data \ - mysql_Innodb_buffer_pool_bytes_dirty \ - mysql_Innodb_buffer_pool_read_ahead_rnd \ - mysql_Innodb_buffer_pool_read_ahead \ - mysql_Innodb_buffer_pool_read_ahead_evicted \ - mysql_Innodb_buffer_pool_read_requests \ - mysql_Innodb_buffer_pool_reads \ - mysql_Innodb_buffer_pool_wait_free \ - mysql_Innodb_buffer_pool_write_requests \ - mysql_Qcache_free_blocks \ - mysql_Qcache_free_memory \ - mysql_Qcache_hits \ - mysql_Qcache_inserts \ - mysql_Qcache_lowmem_prunes \ - mysql_Qcache_not_cached \ - mysql_Qcache_queries_in_cache \ - mysql_Qcache_total_blocks \ - mysql_Key_blocks_not_flushed \ - mysql_Key_blocks_unused \ - mysql_Key_blocks_used \ - mysql_Key_read_requests \ - mysql_Key_reads \ - mysql_Key_write_requests \ - mysql_Key_writes \ - mysql_Open_files \ - mysql_Opened_files - - - mysql_plugin_command_failure=0 - - eval "$(mysql_exec "${@}" |\ - sed \ - -e "s/[[:space:]]\+/ /g" \ - -e "s/\./_/g" \ - -e "s/^\([a-zA-Z0-9_]\+\)[[:space:]]\+\([0-9]\+\)$/mysql_\1=\2/g" |\ - egrep "^mysql_[a-zA-Z0-9_]+=[[:digit:]]+$")" - - [ $mysql_plugin_command_failure -eq 1 ] && return 1 - [ -z "$mysql_Connections" ] && return 1 - - mysql_Thread_cache_misses=0 - [ $(( mysql_Connections + 1 - 1 )) -gt 0 ] && mysql_Thread_cache_misses=$(( mysql_Threads_created * 10000 / mysql_Connections )) + mysql_data[Thread_cache_misses]=0 + [ $(( mysql_data[Connections] + 1 - 1 )) -gt 0 ] && mysql_data[Thread_cache_misses]=$(( mysql_data[Threads_created] * 10000 / mysql_data[Connections] )) return 0 } @@ -200,7 +92,7 @@ mysql_check() { if [ ${#mysql_opts[@]} -eq 0 ] then - if [ ${unconfigured} -eq 1 && ${tryroot} -eq 0 ] + if [ ${unconfigured} -eq 1 -a ${tryroot} -eq 0 ] then mysql_check tryroot "${@}" return $? @@ -380,7 +272,7 @@ CHART mysql_$x.files_rate '' "mysql Opened Files Rate" "files/s" files mysql.fil DIMENSION Opened_files files incremental 1 1 EOF - if [ ! -z "$mysql_Binlog_stmt_cache_disk_use" ] + if [ ! -z "${mysql_data[Binlog_stmt_cache_disk_use]}" ] then cat <<EOF CHART mysql_$x.binlog_stmt_cache '' "mysql Binlog Statement Cache" "statements/s" binlog mysql.binlog_stmt_cache line $((mysql_priority + 50)) $mysql_update_every @@ -389,7 +281,7 @@ DIMENSION Binlog_stmt_cache_use all incremental 1 1 EOF fi - if [ ! -z "$mysql_Connection_errors_accept" ] + if [ ! -z "${mysql_data[Connection_errors_accept]}" ] then cat <<EOF CHART mysql_$x.connection_errors '' "mysql Connection Errors" "connections/s" connections mysql.connection_errors line $((mysql_priority + 51)) $mysql_update_every @@ -415,23 +307,12 @@ mysql_update() { # for each dimension # remember: KEEP IT SIMPLE AND SHORT - # 1. get the counters page from mysql - # 2. sed to remove spaces; replace . with _; remove spaces around =; prepend each line with: local mysql_ - # 3. egrep lines starting with: - # local mysql_client_http_ then one or more of these a-z 0-9 _ then = and one of more of 0-9 - # local mysql_server_all_ then one or more of these a-z 0-9 _ then = and one of more of 0-9 - # 4. then execute this as a script with the eval - # - # be very carefull with eval: - # prepare the script and always grep at the end the lines that are usefull, so that - # even if something goes wrong, no other code can be executed - local m x for m in "${!mysql_ids[@]}" do x="${mysql_ids[$m]}" - mysql_get "${mysql_cmds[$m]}" ${mysql_opts[$m]} + if [ $? -ne 0 ] then unset mysql_ids[$m] @@ -444,186 +325,186 @@ mysql_update() { # write the result of the work. cat <<VALUESEOF BEGIN mysql_$x.net $1 -SET Bytes_received = $mysql_Bytes_received -SET Bytes_sent = $mysql_Bytes_sent +SET Bytes_received = ${mysql_data[Bytes_received]} +SET Bytes_sent = ${mysql_data[Bytes_sent]} END BEGIN mysql_$x.queries $1 -SET Queries = $mysql_Queries -SET Questions = $mysql_Questions -SET Slow_queries = $mysql_Slow_queries +SET Queries = ${mysql_data[Queries]} +SET Questions = ${mysql_data[Questions]} +SET Slow_queries = ${mysql_data[Slow_queries]} END BEGIN mysql_$x.handlers $1 -SET Handler_commit = $mysql_Handler_commit -SET Handler_delete = $mysql_Handler_delete -SET Handler_prepare = $mysql_Handler_prepare -SET Handler_read_first = $mysql_Handler_read_first -SET Handler_read_key = $mysql_Handler_read_key -SET Handler_read_next = $mysql_Handler_read_next -SET Handler_read_prev = $mysql_Handler_read_prev -SET Handler_read_rnd = $mysql_Handler_read_rnd -SET Handler_read_rnd_next = $mysql_Handler_read_rnd_next -SET Handler_rollback = $mysql_Handler_rollback -SET Handler_savepoint = $mysql_Handler_savepoint -SET Handler_savepoint_rollback = $mysql_Handler_savepoint_rollback -SET Handler_update = $mysql_Handler_update -SET Handler_write = $mysql_Handler_write +SET Handler_commit = ${mysql_data[Handler_commit]} +SET Handler_delete = ${mysql_data[Handler_delete]} +SET Handler_prepare = ${mysql_data[Handler_prepare]} +SET Handler_read_first = ${mysql_data[Handler_read_first]} +SET Handler_read_key = ${mysql_data[Handler_read_key]} +SET Handler_read_next = ${mysql_data[Handler_read_next]} +SET Handler_read_prev = ${mysql_data[Handler_read_prev]} +SET Handler_read_rnd = ${mysql_data[Handler_read_rnd]} +SET Handler_read_rnd_next = ${mysql_data[Handler_read_rnd_next]} +SET Handler_rollback = ${mysql_data[Handler_rollback]} +SET Handler_savepoint = ${mysql_data[Handler_savepoint]} +SET Handler_savepoint_rollback = ${mysql_data[Handler_savepoint_rollback]} +SET Handler_update = ${mysql_data[Handler_update]} +SET Handler_write = ${mysql_data[Handler_write]} END BEGIN mysql_$x.table_locks $1 -SET Table_locks_immediate = $mysql_Table_locks_immediate -SET Table_locks_waited = $mysql_Table_locks_waited +SET Table_locks_immediate = ${mysql_data[Table_locks_immediate]} +SET Table_locks_waited = ${mysql_data[Table_locks_waited]} END BEGIN mysql_$x.join_issues $1 -SET Select_full_join = $mysql_Select_full_join -SET Select_full_range_join = $mysql_Select_full_range_join -SET Select_range = $mysql_Select_range -SET Select_range_check = $mysql_Select_range_check -SET Select_scan = $mysql_Select_scan +SET Select_full_join = ${mysql_data[Select_full_join]} +SET Select_full_range_join = ${mysql_data[Select_full_range_join]} +SET Select_range = ${mysql_data[Select_range]} +SET Select_range_check = ${mysql_data[Select_range_check]} +SET Select_scan = ${mysql_data[Select_scan]} END BEGIN mysql_$x.sort_issues $1 -SET Sort_merge_passes = $mysql_Sort_merge_passes -SET Sort_range = $mysql_Sort_range -SET Sort_scan = $mysql_Sort_scan +SET Sort_merge_passes = ${mysql_data[Sort_merge_passes]} +SET Sort_range = ${mysql_data[Sort_range]} +SET Sort_scan = ${mysql_data[Sort_scan]} END BEGIN mysql_$x.tmp $1 -SET Created_tmp_disk_tables = $mysql_Created_tmp_disk_tables -SET Created_tmp_files = $mysql_Created_tmp_files -SET Created_tmp_tables = $mysql_Created_tmp_tables +SET Created_tmp_disk_tables = ${mysql_data[Created_tmp_disk_tables]} +SET Created_tmp_files = ${mysql_data[Created_tmp_files]} +SET Created_tmp_tables = ${mysql_data[Created_tmp_tables]} END BEGIN mysql_$x.connections $1 -SET Connections = $mysql_Connections -SET Aborted_connects = $mysql_Aborted_connects +SET Connections = ${mysql_data[Connections]} +SET Aborted_connects = ${mysql_data[Aborted_connects]} END BEGIN mysql_$x.binlog_cache $1 -SET Binlog_cache_disk_use = $mysql_Binlog_cache_disk_use -SET Binlog_cache_use = $mysql_Binlog_cache_use +SET Binlog_cache_disk_use = ${mysql_data[Binlog_cache_disk_use]} +SET Binlog_cache_use = ${mysql_data[Binlog_cache_use]} END BEGIN mysql_$x.threads $1 -SET Threads_connected = $mysql_Threads_connected -SET Threads_created = $mysql_Threads_created -SET Threads_cached = $mysql_Threads_cached -SET Threads_running = $mysql_Threads_running +SET Threads_connected = ${mysql_data[Threads_connected]} +SET Threads_created = ${mysql_data[Threads_created]} +SET Threads_cached = ${mysql_data[Threads_cached]} +SET Threads_running = ${mysql_data[Threads_running]} END BEGIN mysql_$x.thread_cache_misses $1 -SET misses = $mysql_Thread_cache_misses +SET misses = ${mysql_data[Thread_cache_misses]} END BEGIN mysql_$x.innodb_io $1 -SET Innodb_data_read = $mysql_Innodb_data_read -SET Innodb_data_written = $mysql_Innodb_data_written +SET Innodb_data_read = ${mysql_data[Innodb_data_read]} +SET Innodb_data_written = ${mysql_data[Innodb_data_written]} END BEGIN mysql_$x.innodb_io_ops $1 -SET Innodb_data_reads = $mysql_Innodb_data_reads -SET Innodb_data_writes = $mysql_Innodb_data_writes -SET Innodb_data_fsyncs = $mysql_Innodb_data_fsyncs +SET Innodb_data_reads = ${mysql_data[Innodb_data_reads]} +SET Innodb_data_writes = ${mysql_data[Innodb_data_writes]} +SET Innodb_data_fsyncs = ${mysql_data[Innodb_data_fsyncs]} END BEGIN mysql_$x.innodb_io_pending_ops $1 -SET Innodb_data_pending_reads = $mysql_Innodb_data_pending_reads -SET Innodb_data_pending_writes = $mysql_Innodb_data_pending_writes -SET Innodb_data_pending_fsyncs = $mysql_Innodb_data_pending_fsyncs +SET Innodb_data_pending_reads = ${mysql_data[Innodb_data_pending_reads]} +SET Innodb_data_pending_writes = ${mysql_data[Innodb_data_pending_writes]} +SET Innodb_data_pending_fsyncs = ${mysql_data[Innodb_data_pending_fsyncs]} END BEGIN mysql_$x.innodb_log $1 -SET Innodb_log_waits = $mysql_Innodb_log_waits -SET Innodb_log_write_requests = $mysql_Innodb_log_write_requests -SET Innodb_log_writes = $mysql_Innodb_log_writes +SET Innodb_log_waits = ${mysql_data[Innodb_log_waits]} +SET Innodb_log_write_requests = ${mysql_data[Innodb_log_write_requests]} +SET Innodb_log_writes = ${mysql_data[Innodb_log_writes]} END BEGIN mysql_$x.innodb_os_log $1 -SET Innodb_os_log_fsyncs = $mysql_Innodb_os_log_fsyncs -SET Innodb_os_log_pending_fsyncs = $mysql_Innodb_os_log_pending_fsyncs -SET Innodb_os_log_pending_writes = $mysql_Innodb_os_log_pending_writes +SET Innodb_os_log_fsyncs = ${mysql_data[Innodb_os_log_fsyncs]} +SET Innodb_os_log_pending_fsyncs = ${mysql_data[Innodb_os_log_pending_fsyncs]} +SET Innodb_os_log_pending_writes = ${mysql_data[Innodb_os_log_pending_writes]} END BEGIN mysql_$x.innodb_os_log_io $1 -SET Innodb_os_log_written = $mysql_Innodb_os_log_written +SET Innodb_os_log_written = ${mysql_data[Innodb_os_log_written]} END BEGIN mysql_$x.innodb_cur_row_lock $1 -SET Innodb_row_lock_current_waits = $mysql_Innodb_row_lock_current_waits +SET Innodb_row_lock_current_waits = ${mysql_data[Innodb_row_lock_current_waits]} END BEGIN mysql_$x.innodb_rows $1 -SET Innodb_rows_inserted = $mysql_Innodb_rows_inserted -SET Innodb_rows_read = $mysql_Innodb_rows_read -SET Innodb_rows_updated = $mysql_Innodb_rows_updated -SET Innodb_rows_deleted = $mysql_Innodb_rows_deleted +SET Innodb_rows_inserted = ${mysql_data[Innodb_rows_inserted]} +SET Innodb_rows_read = ${mysql_data[Innodb_rows_read]} +SET Innodb_rows_updated = ${mysql_data[Innodb_rows_updated]} +SET Innodb_rows_deleted = ${mysql_data[Innodb_rows_deleted]} END BEGIN mysql_$x.innodb_buffer_pool_pages $1 -SET Innodb_buffer_pool_pages_data = $mysql_Innodb_buffer_pool_pages_data -SET Innodb_buffer_pool_pages_dirty = $mysql_Innodb_buffer_pool_pages_dirty -SET Innodb_buffer_pool_pages_free = $mysql_Innodb_buffer_pool_pages_free -SET Innodb_buffer_pool_pages_flushed = $mysql_Innodb_buffer_pool_pages_flushed -SET Innodb_buffer_pool_pages_misc = $mysql_Innodb_buffer_pool_pages_misc -SET Innodb_buffer_pool_pages_total = $mysql_Innodb_buffer_pool_pages_total +SET Innodb_buffer_pool_pages_data = ${mysql_data[Innodb_buffer_pool_pages_data]} +SET Innodb_buffer_pool_pages_dirty = ${mysql_data[Innodb_buffer_pool_pages_dirty]} +SET Innodb_buffer_pool_pages_free = ${mysql_data[Innodb_buffer_pool_pages_free]} +SET Innodb_buffer_pool_pages_flushed = ${mysql_data[Innodb_buffer_pool_pages_flushed]} +SET Innodb_buffer_pool_pages_misc = ${mysql_data[Innodb_buffer_pool_pages_misc]} +SET Innodb_buffer_pool_pages_total = ${mysql_data[Innodb_buffer_pool_pages_total]} END BEGIN mysql_$x.innodb_buffer_pool_bytes $1 -SET Innodb_buffer_pool_bytes_data = $mysql_Innodb_buffer_pool_bytes_data -SET Innodb_buffer_pool_bytes_dirty = $mysql_Innodb_buffer_pool_bytes_dirty +SET Innodb_buffer_pool_bytes_data = ${mysql_data[Innodb_buffer_pool_bytes_data]} +SET Innodb_buffer_pool_bytes_dirty = ${mysql_data[Innodb_buffer_pool_bytes_dirty]} END BEGIN mysql_$x.innodb_buffer_pool_read_ahead $1 -SET Innodb_buffer_pool_read_ahead = $mysql_Innodb_buffer_pool_read_ahead -SET Innodb_buffer_pool_read_ahead_evicted = $mysql_Innodb_buffer_pool_read_ahead_evicted -SET Innodb_buffer_pool_read_ahead_rnd = $mysql_Innodb_buffer_pool_read_ahead_rnd +SET Innodb_buffer_pool_read_ahead = ${mysql_data[Innodb_buffer_pool_read_ahead]} +SET Innodb_buffer_pool_read_ahead_evicted = ${mysql_data[Innodb_buffer_pool_read_ahead_evicted]} +SET Innodb_buffer_pool_read_ahead_rnd = ${mysql_data[Innodb_buffer_pool_read_ahead_rnd]} END BEGIN mysql_$x.innodb_buffer_pool_reqs $1 -SET Innodb_buffer_pool_read_requests = $mysql_Innodb_buffer_pool_read_requests -SET Innodb_buffer_pool_write_requests = $mysql_Innodb_buffer_pool_write_requests +SET Innodb_buffer_pool_read_requests = ${mysql_data[Innodb_buffer_pool_read_requests]} +SET Innodb_buffer_pool_write_requests = ${mysql_data[Innodb_buffer_pool_write_requests]} END BEGIN mysql_$x.innodb_buffer_pool_ops $1 -SET Innodb_buffer_pool_reads = $mysql_Innodb_buffer_pool_reads -SET Innodb_buffer_pool_wait_free = $mysql_Innodb_buffer_pool_wait_free +SET Innodb_buffer_pool_reads = ${mysql_data[Innodb_buffer_pool_reads]} +SET Innodb_buffer_pool_wait_free = ${mysql_data[Innodb_buffer_pool_wait_free]} END BEGIN mysql_$x.qcache_ops $1 -SET Qcache_hits hits = $mysql_Qcache_hits -SET Qcache_lowmem_prunes = $mysql_Qcache_lowmem_prunes -SET Qcache_inserts inserts = $mysql_Qcache_inserts inserts -SET Qcache_not_cached = $mysql_Qcache_not_cached +SET Qcache_hits hits = ${mysql_data[Qcache_hits]} +SET Qcache_lowmem_prunes = ${mysql_data[Qcache_lowmem_prunes]} +SET Qcache_inserts = ${mysql_data[Qcache_inserts]} +SET Qcache_not_cached = ${mysql_data[Qcache_not_cached]} END BEGIN mysql_$x.qcache $1 -SET Qcache_queries_in_cache = $mysql_Qcache_queries_in_cache +SET Qcache_queries_in_cache = ${mysql_data[Qcache_queries_in_cache]} END BEGIN mysql_$x.qcache_freemem $1 -SET Qcache_free_memory = $mysql_Qcache_free_memory +SET Qcache_free_memory = ${mysql_data[Qcache_free_memory]} END BEGIN mysql_$x.qcache_memblocks $1 -SET Qcache_free_blocks = $mysql_Qcache_free_blocks -SET Qcache_total_blocks = $mysql_Qcache_total_blocks +SET Qcache_free_blocks = ${mysql_data[Qcache_free_blocks]} +SET Qcache_total_blocks = ${mysql_data[Qcache_total_blocks]} END BEGIN mysql_$x.key_blocks $1 -SET Key_blocks_unused = $mysql_Key_blocks_unused -SET Key_blocks_used = $mysql_Key_blocks_used -SET Key_blocks_not_flushed = $mysql_Key_blocks_not_flushed +SET Key_blocks_unused = ${mysql_data[Key_blocks_unused]} +SET Key_blocks_used = ${mysql_data[Key_blocks_used]} +SET Key_blocks_not_flushed = ${mysql_data[Key_blocks_not_flushed]} END BEGIN mysql_$x.key_requests $1 -SET Key_read_requests = $mysql_Key_read_requests -SET Key_write_requests = $mysql_Key_write_requests +SET Key_read_requests = ${mysql_data[Key_read_requests]} +SET Key_write_requests = ${mysql_data[Key_write_requests]} END BEGIN mysql_$x.key_disk_ops $1 -SET Key_reads = $mysql_Key_reads -SET Key_writes = $mysql_Key_writes +SET Key_reads = ${mysql_data[Key_reads]} +SET Key_writes = ${mysql_data[Key_writes]} END BEGIN mysql_$x.files $1 -SET Open_files = $mysql_Open_files +SET Open_files = ${mysql_data[Open_files]} END BEGIN mysql_$x.files_rate $1 -SET Opened_files = $mysql_Opened_files +SET Opened_files = ${mysql_data[Opened_files]} END VALUESEOF - if [ ! -z "$mysql_Binlog_stmt_cache_disk_use" ] + if [ ! -z "${mysql_data[Binlog_stmt_cache_disk_use]}" ] then cat <<VALUESEOF BEGIN mysql_$x.binlog_stmt_cache $1 -SET Binlog_stmt_cache_disk_use = $mysql_Binlog_stmt_cache_disk_use -SET Binlog_stmt_cache_use = $mysql_Binlog_stmt_cache_use +SET Binlog_stmt_cache_disk_use = ${mysql_data[Binlog_stmt_cache_disk_use]} +SET Binlog_stmt_cache_use = ${mysql_data[Binlog_stmt_cache_use]} END VALUESEOF fi - if [ ! -z "$mysql_Connection_errors_accept" ] + if [ ! -z "${mysql_data[Connection_errors_accept]}" ] then cat <<VALUESEOF BEGIN mysql_$x.connection_errors $1 -SET Connection_errors_accept = $mysql_Connection_errors_accept -SET Connection_errors_internal = $mysql_Connection_errors_internal -SET Connection_errors_max_connections = $mysql_Connection_errors_max_connections -SET Connection_errors_peer_addr = $mysql_Connection_errors_peer_addr -SET Connection_errors_select = $mysql_Connection_errors_select -SET Connection_errors_tcpwrap = $mysql_Connection_errors_tcpwrap +SET Connection_errors_accept = ${mysql_data[Connection_errors_accept]} +SET Connection_errors_internal = ${mysql_data[Connection_errors_internal]} +SET Connection_errors_max_connections = ${mysql_data[Connection_errors_max_connections]} +SET Connection_errors_peer_addr = ${mysql_data[Connection_errors_peer_addr]} +SET Connection_errors_select = ${mysql_data[Connection_errors_select]} +SET Connection_errors_tcpwrap = ${mysql_data[Connection_errors_tcpwrap]} END VALUESEOF fi diff --git a/charts.d/nginx.chart.sh b/charts.d/nginx.chart.sh index 450aa94b3..a2a9b320f 100755 --- a/charts.d/nginx.chart.sh +++ b/charts.d/nginx.chart.sh @@ -1,9 +1,10 @@ -#!/bin/bash +# no need for shebang - this file is loaded from charts.d.plugin # if this chart is called X.chart.sh, then all functions and global variables # must start with X_ nginx_url="http://127.0.0.1:80/stub_status" +nginx_curl_opts="" # _update_every is a special variable - it holds the number of seconds # between the calls of the _update() function @@ -19,7 +20,7 @@ nginx_reading=0 nginx_writing=0 nginx_waiting=0 nginx_get() { - nginx_response=($(curl -Ss "${nginx_url}")) + nginx_response=($(curl -Ss ${nginx_curl_opts} "${nginx_url}")) [ $? -ne 0 -o "${#nginx_response[@]}" -eq 0 ] && return 1 if [ "${nginx_response[0]}" != "Active" \ @@ -81,18 +82,18 @@ nginx_check() { # _create is called once, to create the charts nginx_create() { cat <<EOF -CHART nginx.connections '' "nginx Active Connections" "connections" nginx nginx.connections line $((nginx_priority + 1)) $nginx_update_every +CHART nginx_local.connections '' "nginx Active Connections" "connections" nginx nginx.connections line $((nginx_priority + 1)) $nginx_update_every DIMENSION active '' absolute 1 1 -CHART nginx.requests '' "nginx Requests" "requests/s" nginx nginx.requests line $((nginx_priority + 2)) $nginx_update_every +CHART nginx_local.requests '' "nginx Requests" "requests/s" nginx nginx.requests line $((nginx_priority + 2)) $nginx_update_every DIMENSION requests '' incremental 1 1 -CHART nginx.connections_status '' "nginx Active Connections by Status" "connections" nginx nginx.connections.status line $((nginx_priority + 3)) $nginx_update_every +CHART nginx_local.connections_status '' "nginx Active Connections by Status" "connections" nginx nginx.connections.status line $((nginx_priority + 3)) $nginx_update_every DIMENSION reading '' absolute 1 1 DIMENSION writing '' absolute 1 1 DIMENSION waiting idle absolute 1 1 -CHART nginx.connect_rate '' "nginx Connections Rate" "connections/s" nginx nginx.connections.rate line $((nginx_priority + 4)) $nginx_update_every +CHART nginx_local.connect_rate '' "nginx Connections Rate" "connections/s" nginx nginx.connections.rate line $((nginx_priority + 4)) $nginx_update_every DIMENSION accepts accepted incremental 1 1 DIMENSION handled '' incremental 1 1 EOF @@ -113,18 +114,18 @@ nginx_update() { # write the result of the work. cat <<VALUESEOF -BEGIN nginx.connections $1 +BEGIN nginx_local.connections $1 SET active = $((nginx_active_connections)) END -BEGIN nginx.requests $1 +BEGIN nginx_local.requests $1 SET requests = $((nginx_requests)) END -BEGIN nginx.connections_status $1 +BEGIN nginx_local.connections_status $1 SET reading = $((nginx_reading)) SET writing = $((nginx_writing)) SET waiting = $((nginx_waiting)) END -BEGIN nginx.connect_rate $1 +BEGIN nginx_local.connect_rate $1 SET accepts = $((nginx_accepts)) SET handled = $((nginx_handled)) END diff --git a/charts.d/nut.chart.sh b/charts.d/nut.chart.sh index a47208451..3c8e1c9d0 100755 --- a/charts.d/nut.chart.sh +++ b/charts.d/nut.chart.sh @@ -1,4 +1,4 @@ -#!/bin/bash +# no need for shebang - this file is loaded from charts.d.plugin # a space separated list of UPS names # if empty, the list returned by 'upsc -l' will be used diff --git a/charts.d/opensips.chart.sh b/charts.d/opensips.chart.sh index c7066ec05..ce42ccdd7 100755 --- a/charts.d/opensips.chart.sh +++ b/charts.d/opensips.chart.sh @@ -1,4 +1,4 @@ -#!/bin/sh +# no need for shebang - this file is loaded from charts.d.plugin opensips_opts="fifo get_statistics all" opensips_cmd= diff --git a/charts.d/phpfpm.chart.sh b/charts.d/phpfpm.chart.sh index c0532fab1..976ce91b1 100755 --- a/charts.d/phpfpm.chart.sh +++ b/charts.d/phpfpm.chart.sh @@ -1,4 +1,4 @@ -#!/bin/bash +# no need for shebang - this file is loaded from charts.d.plugin # if this chart is called X.chart.sh, then all functions and global variables # must start with X_ @@ -8,6 +8,7 @@ # you can see, https://easyengine.io/tutorials/php/fpm-status-page/ declare -A phpfpm_urls=() +declare -A phpfpm_curl_opts=() # _update_every is a special variable - it holds the number of seconds # between the calls of the _update() function @@ -29,8 +30,9 @@ phpfpm_max_active_processes=0 phpfpm_max_children_reached=0 phpfpm_slow_requests=0 phpfpm_get() { - url=$1 - phpfpm_response=($(curl -Ss "${url}")) + local opts="${1}" url="${2}" + + phpfpm_response=($(curl -Ss ${opts} "${url}")) [ $? -ne 0 -o "${#phpfpm_response[@]}" -eq 0 ] && return 1 if [[ "${phpfpm_response[0]}" != "pool:" \ @@ -42,7 +44,6 @@ phpfpm_get() { || "${phpfpm_response[26]}" != "idle" \ || "${phpfpm_response[29]}" != "active" \ || "${phpfpm_response[32]}" != "total" \ - || "${phpfpm_response[43]}" != "slow" \ ]] then echo >&2 "phpfpm: invalid response from phpfpm status server: ${phpfpm_response[*]}" @@ -61,7 +62,12 @@ phpfpm_get() { phpfpm_total_processes="${phpfpm_response[34]}" phpfpm_max_active_processes="${phpfpm_response[38]}" phpfpm_max_children_reached="${phpfpm_response[42]}" - phpfpm_slow_requests="${phpfpm_response[45]}" + if [ "${phpfpm_response[43]}" == "slow" ] + then + phpfpm_slow_requests="${phpfpm_response[45]}" + else + phpfpm_slow_requests="-1" + fi if [[ -z "${phpfpm_pool}" \ || -z "${phpfpm_start_time}" \ @@ -75,7 +81,6 @@ phpfpm_get() { || -z "${phpfpm_total_processes}" \ || -z "${phpfpm_max_active_processes}" \ || -z "${phpfpm_max_children_reached}" \ - || -z "${phpfpm_slow_requests}" \ ]] then echo >&2 "phpfpm: empty values got from phpfpm status server: ${phpfpm_response[*]}" @@ -94,7 +99,7 @@ phpfpm_check() { local m for m in "${!phpfpm_urls[@]}" do - phpfpm_get "${phpfpm_urls[$m]}" + phpfpm_get "${phpfpm_curl_opts[$m]}" "${phpfpm_urls[$m]}" if [ $? -ne 0 ]; then echo >&2 "phpfpm: cannot find status on URL '${phpfpm_url[$m]}'. Please set phpfpm_urls[$m]='http://localhost/status' in $confd/phpfpm.conf" unset phpfpm_urls[$m] @@ -130,8 +135,11 @@ DIMENSION requests '' incremental 1 1 CHART phpfpm_$m.performance '' "PHP-FPM Performance" "status" phpfpm phpfpm.performance line $((phpfpm_priority + 3)) $phpfpm_update_every DIMENSION reached 'max children reached' absolute 1 1 -DIMENSION slow 'slow requests' absolute 1 1 EOF + if [ $((phpfpm_slow_requests)) -ne -1 ] + then + echo "DIMENSION slow 'slow requests' absolute 1 1" + fi done return 0 @@ -149,7 +157,7 @@ phpfpm_update() { local m for m in "${!phpfpm_urls[@]}" do - phpfpm_get "${phpfpm_urls[$m]}" + phpfpm_get "${phpfpm_curl_opts[$m]}" "${phpfpm_urls[$m]}" if [ $? -ne 0 ]; then continue fi @@ -166,10 +174,17 @@ SET requests = $((phpfpm_accepted_conn)) END BEGIN phpfpm_$m.performance $1 SET reached = $((phpfpm_max_children_reached)) -SET slow = $((phpfpm_slow_requests)) -END EOF + if [ $((phpfpm_slow_requests)) -ne -1 ] + then + echo "SET slow = $((phpfpm_slow_requests))" + fi + echo "END" done return 0 } + +phpfpm_check +phpfpm_create +phpfpm_update diff --git a/charts.d/postfix.chart.sh b/charts.d/postfix.chart.sh index f4f710275..7f07a1868 100755 --- a/charts.d/postfix.chart.sh +++ b/charts.d/postfix.chart.sh @@ -1,4 +1,4 @@ -#!/bin/sh +# no need for shebang - this file is loaded from charts.d.plugin # the postqueue command # if empty, it will use the one found in the system path @@ -43,9 +43,9 @@ postfix_check() { postfix_create() { cat <<EOF -CHART postfix.qemails '' "Postfix Queue Emails" "emails" queue postfix.queued.emails line $((postfix_priority + 1)) $postfix_update_every +CHART postfix_local.qemails '' "Postfix Queue Emails" "emails" queue postfix.queued.emails line $((postfix_priority + 1)) $postfix_update_every DIMENSION emails '' absolute 1 1 -CHART postfix.qsize '' "Postfix Queue Emails Size" "emails size in KB" queue postfix.queued.size area $((postfix_priority + 2)) $postfix_update_every +CHART postfix_local.qsize '' "Postfix Queue Emails Size" "emails size in KB" queue postfix.queued.size area $((postfix_priority + 2)) $postfix_update_every DIMENSION size '' absolute 1 1 EOF @@ -80,10 +80,10 @@ postfix_update() { # write the result of the work. cat <<VALUESEOF -BEGIN postfix.qemails $1 +BEGIN postfix_local.qemails $1 SET emails = $postfix_q_emails END -BEGIN postfix.qsize $1 +BEGIN postfix_local.qsize $1 SET size = $postfix_q_size END VALUESEOF diff --git a/charts.d/sensors.chart.sh b/charts.d/sensors.chart.sh index 19e938586..9652f896a 100755 --- a/charts.d/sensors.chart.sh +++ b/charts.d/sensors.chart.sh @@ -1,4 +1,4 @@ -#!/bin/sh +# no need for shebang - this file is loaded from charts.d.plugin # sensors docs # https://www.kernel.org/doc/Documentation/hwmon/sysfs-interface @@ -23,6 +23,8 @@ sensors_update_every= sensors_priority=90000 +declare -A sensors_excluded=() + sensors_find_all_files() { find $1 -maxdepth $sensors_sys_depth -name \*_input -o -name temp 2>/dev/null } @@ -47,15 +49,20 @@ sensors_check() { sensors_check_files() { # we only need sensors that report a non-zero value + # also remove not needed sensors - local f= v= + local f= v= excluded= for f in $* do [ ! -f "$f" ] && continue + for ex in ${sensors_excluded[@]}; do + [[ $f =~ .*$ex$ ]] && excluded='1' && break + done - v="$( cat $f )" + [ "$excluded" != "1" ] && v="$( cat $f )" || v=0 v=$(( v + 1 - 1 )) [ $v -ne 0 ] && echo "$f" && continue + excluded= echo >&2 "$PROGRAM_NAME: sensors: $f gives zero values" done @@ -206,7 +213,7 @@ sensors_create() { fi echo "DIMENSION $fid '$labelname' $algorithm $multiplier $divisor" - echo >>$TMP_DIR/sensors.sh "printf \"SET $fid = \"; cat $file " + echo >>$TMP_DIR/sensors.sh "echo \"SET $fid = \"\$(< $file )" done echo >>$TMP_DIR/sensors.sh "echo END" diff --git a/charts.d/squid.chart.sh b/charts.d/squid.chart.sh index f6154b256..3e72ba6df 100755 --- a/charts.d/squid.chart.sh +++ b/charts.d/squid.chart.sh @@ -1,10 +1,10 @@ -#!/bin/sh +# no need for shebang - this file is loaded from charts.d.plugin squid_host= squid_port= squid_url= squid_timeout=2 -squid_update_every=5 +squid_update_every=2 squid_priority=60000 squid_get_stats_internal() { @@ -63,21 +63,21 @@ squid_check() { squid_create() { # create the charts cat <<EOF -CHART squid.clients_net '' "Squid Client Bandwidth" "kilobits / sec" clients squid.clients.net area $((squid_priority + 1)) $squid_update_every +CHART squid_local.clients_net '' "Squid Client Bandwidth" "kilobits / sec" clients squid.clients.net area $((squid_priority + 1)) $squid_update_every DIMENSION client_http_kbytes_in in incremental 8 1 DIMENSION client_http_kbytes_out out incremental -8 1 DIMENSION client_http_hit_kbytes_out hits incremental -8 1 -CHART squid.clients_requests '' "Squid Client Requests" "requests / sec" clients squid.clients.requests line $((squid_priority + 3)) $squid_update_every +CHART squid_local.clients_requests '' "Squid Client Requests" "requests / sec" clients squid.clients.requests line $((squid_priority + 3)) $squid_update_every DIMENSION client_http_requests requests incremental 1 1 DIMENSION client_http_hits hits incremental 1 1 DIMENSION client_http_errors errors incremental -1 1 -CHART squid.servers_net '' "Squid Server Bandwidth" "kilobits / sec" servers squid.servers.net area $((squid_priority + 2)) $squid_update_every +CHART squid_local.servers_net '' "Squid Server Bandwidth" "kilobits / sec" servers squid.servers.net area $((squid_priority + 2)) $squid_update_every DIMENSION server_all_kbytes_in in incremental 8 1 DIMENSION server_all_kbytes_out out incremental -8 1 -CHART squid.servers_requests '' "Squid Server Requests" "requests / sec" servers squid.servers.requests line $((squid_priority + 4)) $squid_update_every +CHART squid_local.servers_requests '' "Squid Server Requests" "requests / sec" servers squid.servers.requests line $((squid_priority + 4)) $squid_update_every DIMENSION server_all_requests requests incremental 1 1 DIMENSION server_all_errors errors incremental -1 1 EOF @@ -111,24 +111,24 @@ squid_update() { # write the result of the work. cat <<VALUESEOF -BEGIN squid.clients_net $1 +BEGIN squid_local.clients_net $1 SET client_http_kbytes_in = $squid_client_http_kbytes_in SET client_http_kbytes_out = $squid_client_http_kbytes_out SET client_http_hit_kbytes_out = $squid_client_http_hit_kbytes_out END -BEGIN squid.clients_requests $1 +BEGIN squid_local.clients_requests $1 SET client_http_requests = $squid_client_http_requests SET client_http_hits = $squid_client_http_hits SET client_http_errors = $squid_client_http_errors END -BEGIN squid.servers_net $1 +BEGIN squid_local.servers_net $1 SET server_all_kbytes_in = $squid_server_all_kbytes_in SET server_all_kbytes_out = $squid_server_all_kbytes_out END -BEGIN squid.servers_requests $1 +BEGIN squid_local.servers_requests $1 SET server_all_requests = $squid_server_all_requests SET server_all_errors = $squid_server_all_errors END diff --git a/charts.d/tomcat.chart.sh b/charts.d/tomcat.chart.sh index 4e10a9183..cc6baea1a 100755 --- a/charts.d/tomcat.chart.sh +++ b/charts.d/tomcat.chart.sh @@ -1,4 +1,4 @@ -#!/bin/bash +# no need for shebang - this file is loaded from charts.d.plugin # Description: Tomcat netdata charts.d plugin # Author: Jorge Romero @@ -6,10 +6,11 @@ # the URL to download tomcat status info # usually http://localhost:8080/manager/status?XML=true tomcat_url="" +tomcat_curl_opts="" # set tomcat username/password here -tomcatUser="" -tomcatPassword="" +tomcat_user="" +tomcat_password="" # _update_every is a special variable - it holds the number of seconds # between the calls of the _update() function @@ -36,13 +37,23 @@ tomcat_check() { echo >&2 "tomcat url is unset or set to the empty string" return 1 fi - if [ -z "${tomcatUser}" ]; then - echo >&2 "tomcat user is unset or set to the empty string" - return 1 + if [ -z "${tomcat_user}" ]; then + # check backwards compatibility + if [ -z "${tomcatUser}" ]; then + echo >&2 "tomcat user is unset or set to the empty string" + return 1 + else + tomcat_user="${tomcatUser}" + fi fi - if [ -z "${tomcatPassword}" ]; then - echo >&2 "tomcat password is unset or set to the empty string" - return 1 + if [ -z "${tomcat_password}" ]; then + # check backwards compatibility + if [ -z "${tomcatPassword}" ]; then + echo >&2 "tomcat password is unset or set to the empty string" + return 1 + else + tomcat_password="${tomcatPassword}" + fi fi # check if we can get to tomcat's status page @@ -62,13 +73,14 @@ tomcat_check() { } tomcat_get() { - # Collect tomcat values - mapfile -t lines < <(curl -u "$tomcatUser":"$tomcatPassword" -Ss "$tomcat_url" |\ + # collect tomcat values + tomcat_port="$(IFS=/ read -ra a <<< "$tomcat_url"; hostport=${a[2]}; echo "${hostport#*:}")" + mapfile -t lines < <(curl -u "$tomcat_user":"$tomcat_password" -Ss ${tomcat_curl_opts} "$tomcat_url" |\ xmlstarlet sel \ -t -m "/status/jvm/memory" -v @free \ - -n -m "/status/connector[@name='\"http-bio-8080\"']/threadInfo" -v @currentThreadCount \ + -n -m "/status/connector[@name='\"http-bio-$tomcat_port\"']/threadInfo" -v @currentThreadCount \ -n -v @currentThreadsBusy \ - -n -m "/status/connector[@name='\"http-bio-8080\"']/requestInfo" -v @requestCount \ + -n -m "/status/connector[@name='\"http-bio-$tomcat_port\"']/requestInfo" -v @requestCount \ -n -v @bytesSent -n -) tomcat_jvm_freememory="${lines[0]}" diff --git a/conf.d/Makefile.am b/conf.d/Makefile.am index 381b546e3..02fe86b01 100644 --- a/conf.d/Makefile.am +++ b/conf.d/Makefile.am @@ -6,4 +6,52 @@ MAINTAINERCLEANFILES= $(srcdir)/Makefile.in dist_config_DATA = \ apps_groups.conf \ charts.d.conf \ + python.d.conf \ + $(NULL) + +chartsconfigdir=$(configdir)/charts.d +dist_chartsconfig_DATA = \ + $(NULL) + +nodeconfigdir=$(configdir)/node.d +dist_nodeconfig_DATA = \ + $(NULL) + +pythonconfigdir=$(configdir)/python.d +dist_pythonconfig_DATA = \ + python.d/apache.conf \ + python.d/apache_cache.conf \ + python.d/cpufreq.conf \ + python.d/dovecot.conf \ + python.d/example.conf \ + python.d/exim.conf \ + python.d/hddtemp.conf \ + python.d/ipfs.conf \ + python.d/memcached.conf \ + python.d/mysql.conf \ + python.d/nginx.conf \ + python.d/nginx_log.conf \ + python.d/phpfpm.conf \ + python.d/postfix.conf \ + python.d/redis.conf \ + python.d/sensors.conf \ + python.d/squid.conf \ + python.d/tomcat.conf \ + $(NULL) + +healthconfigdir=$(configdir)/health.d +dist_healthconfig_DATA = \ + health.d/apache.conf \ + health.d/cpu.conf \ + health.d/disks.conf \ + health.d/entropy.conf \ + health.d/memcached.conf \ + health.d/named.conf \ + health.d/net.conf \ + health.d/nginx.conf \ + health.d/qos.conf \ + health.d/ram.conf \ + health.d/redis.conf \ + health.d/swap.conf \ + health.d/squid.conf \ $(NULL) diff --git a/conf.d/Makefile.in b/conf.d/Makefile.in index 1938bd940..9356f60e2 100644 --- a/conf.d/Makefile.in +++ b/conf.d/Makefile.in @@ -80,7 +80,9 @@ build_triplet = @build@ host_triplet = @host@ subdir = conf.d DIST_COMMON = $(srcdir)/Makefile.in $(srcdir)/Makefile.am \ - $(dist_config_DATA) + $(dist_chartsconfig_DATA) $(dist_config_DATA) \ + $(dist_healthconfig_DATA) $(dist_nodeconfig_DATA) \ + $(dist_pythonconfig_DATA) ACLOCAL_M4 = $(top_srcdir)/aclocal.m4 am__aclocal_m4_deps = $(top_srcdir)/m4/ax_pthread.m4 \ $(top_srcdir)/configure.ac @@ -136,8 +138,12 @@ am__uninstall_files_from_dir = { \ || { echo " ( cd '$$dir' && rm -f" $$files ")"; \ $(am__cd) "$$dir" && rm -f $$files; }; \ } -am__installdirs = "$(DESTDIR)$(configdir)" -DATA = $(dist_config_DATA) +am__installdirs = "$(DESTDIR)$(chartsconfigdir)" \ + "$(DESTDIR)$(configdir)" "$(DESTDIR)$(healthconfigdir)" \ + "$(DESTDIR)$(nodeconfigdir)" "$(DESTDIR)$(pythonconfigdir)" +DATA = $(dist_chartsconfig_DATA) $(dist_config_DATA) \ + $(dist_healthconfig_DATA) $(dist_nodeconfig_DATA) \ + $(dist_pythonconfig_DATA) am__tagged_files = $(HEADERS) $(SOURCES) $(TAGS_FILES) $(LISP) DISTFILES = $(DIST_COMMON) $(DIST_SOURCES) $(TEXINFOS) $(EXTRA_DIST) ACLOCAL = @ACLOCAL@ @@ -261,6 +267,8 @@ pluginsdir = @pluginsdir@ prefix = @prefix@ program_transform_name = @program_transform_name@ psdir = @psdir@ +pythondir = @pythondir@ +registrydir = @registrydir@ sbindir = @sbindir@ sharedstatedir = @sharedstatedir@ srcdir = @srcdir@ @@ -279,6 +287,54 @@ MAINTAINERCLEANFILES = $(srcdir)/Makefile.in dist_config_DATA = \ apps_groups.conf \ charts.d.conf \ + python.d.conf \ + $(NULL) + +chartsconfigdir = $(configdir)/charts.d +dist_chartsconfig_DATA = \ + $(NULL) + +nodeconfigdir = $(configdir)/node.d +dist_nodeconfig_DATA = \ + $(NULL) + +pythonconfigdir = $(configdir)/python.d +dist_pythonconfig_DATA = \ + python.d/apache.conf \ + python.d/apache_cache.conf \ + python.d/cpufreq.conf \ + python.d/dovecot.conf \ + python.d/example.conf \ + python.d/exim.conf \ + python.d/hddtemp.conf \ + python.d/ipfs.conf \ + python.d/memcached.conf \ + python.d/mysql.conf \ + python.d/nginx.conf \ + python.d/nginx_log.conf \ + python.d/phpfpm.conf \ + python.d/postfix.conf \ + python.d/redis.conf \ + python.d/sensors.conf \ + python.d/squid.conf \ + python.d/tomcat.conf \ + $(NULL) + +healthconfigdir = $(configdir)/health.d +dist_healthconfig_DATA = \ + health.d/apache.conf \ + health.d/cpu.conf \ + health.d/disks.conf \ + health.d/entropy.conf \ + health.d/memcached.conf \ + health.d/named.conf \ + health.d/net.conf \ + health.d/nginx.conf \ + health.d/qos.conf \ + health.d/ram.conf \ + health.d/redis.conf \ + health.d/swap.conf \ + health.d/squid.conf \ $(NULL) all: all-am @@ -314,6 +370,27 @@ $(top_srcdir)/configure: @MAINTAINER_MODE_TRUE@ $(am__configure_deps) $(ACLOCAL_M4): @MAINTAINER_MODE_TRUE@ $(am__aclocal_m4_deps) cd $(top_builddir) && $(MAKE) $(AM_MAKEFLAGS) am--refresh $(am__aclocal_m4_deps): +install-dist_chartsconfigDATA: $(dist_chartsconfig_DATA) + @$(NORMAL_INSTALL) + @list='$(dist_chartsconfig_DATA)'; test -n "$(chartsconfigdir)" || list=; \ + if test -n "$$list"; then \ + echo " $(MKDIR_P) '$(DESTDIR)$(chartsconfigdir)'"; \ + $(MKDIR_P) "$(DESTDIR)$(chartsconfigdir)" || exit 1; \ + fi; \ + for p in $$list; do \ + if test -f "$$p"; then d=; else d="$(srcdir)/"; fi; \ + echo "$$d$$p"; \ + done | $(am__base_list) | \ + while read files; do \ + echo " $(INSTALL_DATA) $$files '$(DESTDIR)$(chartsconfigdir)'"; \ + $(INSTALL_DATA) $$files "$(DESTDIR)$(chartsconfigdir)" || exit $$?; \ + done + +uninstall-dist_chartsconfigDATA: + @$(NORMAL_UNINSTALL) + @list='$(dist_chartsconfig_DATA)'; test -n "$(chartsconfigdir)" || list=; \ + files=`for p in $$list; do echo $$p; done | sed -e 's|^.*/||'`; \ + dir='$(DESTDIR)$(chartsconfigdir)'; $(am__uninstall_files_from_dir) install-dist_configDATA: $(dist_config_DATA) @$(NORMAL_INSTALL) @list='$(dist_config_DATA)'; test -n "$(configdir)" || list=; \ @@ -335,6 +412,69 @@ uninstall-dist_configDATA: @list='$(dist_config_DATA)'; test -n "$(configdir)" || list=; \ files=`for p in $$list; do echo $$p; done | sed -e 's|^.*/||'`; \ dir='$(DESTDIR)$(configdir)'; $(am__uninstall_files_from_dir) +install-dist_healthconfigDATA: $(dist_healthconfig_DATA) + @$(NORMAL_INSTALL) + @list='$(dist_healthconfig_DATA)'; test -n "$(healthconfigdir)" || list=; \ + if test -n "$$list"; then \ + echo " $(MKDIR_P) '$(DESTDIR)$(healthconfigdir)'"; \ + $(MKDIR_P) "$(DESTDIR)$(healthconfigdir)" || exit 1; \ + fi; \ + for p in $$list; do \ + if test -f "$$p"; then d=; else d="$(srcdir)/"; fi; \ + echo "$$d$$p"; \ + done | $(am__base_list) | \ + while read files; do \ + echo " $(INSTALL_DATA) $$files '$(DESTDIR)$(healthconfigdir)'"; \ + $(INSTALL_DATA) $$files "$(DESTDIR)$(healthconfigdir)" || exit $$?; \ + done + +uninstall-dist_healthconfigDATA: + @$(NORMAL_UNINSTALL) + @list='$(dist_healthconfig_DATA)'; test -n "$(healthconfigdir)" || list=; \ + files=`for p in $$list; do echo $$p; done | sed -e 's|^.*/||'`; \ + dir='$(DESTDIR)$(healthconfigdir)'; $(am__uninstall_files_from_dir) +install-dist_nodeconfigDATA: $(dist_nodeconfig_DATA) + @$(NORMAL_INSTALL) + @list='$(dist_nodeconfig_DATA)'; test -n "$(nodeconfigdir)" || list=; \ + if test -n "$$list"; then \ + echo " $(MKDIR_P) '$(DESTDIR)$(nodeconfigdir)'"; \ + $(MKDIR_P) "$(DESTDIR)$(nodeconfigdir)" || exit 1; \ + fi; \ + for p in $$list; do \ + if test -f "$$p"; then d=; else d="$(srcdir)/"; fi; \ + echo "$$d$$p"; \ + done | $(am__base_list) | \ + while read files; do \ + echo " $(INSTALL_DATA) $$files '$(DESTDIR)$(nodeconfigdir)'"; \ + $(INSTALL_DATA) $$files "$(DESTDIR)$(nodeconfigdir)" || exit $$?; \ + done + +uninstall-dist_nodeconfigDATA: + @$(NORMAL_UNINSTALL) + @list='$(dist_nodeconfig_DATA)'; test -n "$(nodeconfigdir)" || list=; \ + files=`for p in $$list; do echo $$p; done | sed -e 's|^.*/||'`; \ + dir='$(DESTDIR)$(nodeconfigdir)'; $(am__uninstall_files_from_dir) +install-dist_pythonconfigDATA: $(dist_pythonconfig_DATA) + @$(NORMAL_INSTALL) + @list='$(dist_pythonconfig_DATA)'; test -n "$(pythonconfigdir)" || list=; \ + if test -n "$$list"; then \ + echo " $(MKDIR_P) '$(DESTDIR)$(pythonconfigdir)'"; \ + $(MKDIR_P) "$(DESTDIR)$(pythonconfigdir)" || exit 1; \ + fi; \ + for p in $$list; do \ + if test -f "$$p"; then d=; else d="$(srcdir)/"; fi; \ + echo "$$d$$p"; \ + done | $(am__base_list) | \ + while read files; do \ + echo " $(INSTALL_DATA) $$files '$(DESTDIR)$(pythonconfigdir)'"; \ + $(INSTALL_DATA) $$files "$(DESTDIR)$(pythonconfigdir)" || exit $$?; \ + done + +uninstall-dist_pythonconfigDATA: + @$(NORMAL_UNINSTALL) + @list='$(dist_pythonconfig_DATA)'; test -n "$(pythonconfigdir)" || list=; \ + files=`for p in $$list; do echo $$p; done | sed -e 's|^.*/||'`; \ + dir='$(DESTDIR)$(pythonconfigdir)'; $(am__uninstall_files_from_dir) tags TAGS: ctags CTAGS: @@ -376,7 +516,7 @@ check-am: all-am check: check-am all-am: Makefile $(DATA) installdirs: - for dir in "$(DESTDIR)$(configdir)"; do \ + for dir in "$(DESTDIR)$(chartsconfigdir)" "$(DESTDIR)$(configdir)" "$(DESTDIR)$(healthconfigdir)" "$(DESTDIR)$(nodeconfigdir)" "$(DESTDIR)$(pythonconfigdir)"; do \ test -z "$$dir" || $(MKDIR_P) "$$dir"; \ done install: install-am @@ -430,7 +570,9 @@ info: info-am info-am: -install-data-am: install-dist_configDATA +install-data-am: install-dist_chartsconfigDATA install-dist_configDATA \ + install-dist_healthconfigDATA install-dist_nodeconfigDATA \ + install-dist_pythonconfigDATA install-dvi: install-dvi-am @@ -474,21 +616,27 @@ ps: ps-am ps-am: -uninstall-am: uninstall-dist_configDATA +uninstall-am: uninstall-dist_chartsconfigDATA \ + uninstall-dist_configDATA uninstall-dist_healthconfigDATA \ + uninstall-dist_nodeconfigDATA uninstall-dist_pythonconfigDATA .MAKE: install-am install-strip .PHONY: all all-am check check-am clean clean-generic cscopelist-am \ ctags-am distclean distclean-generic distdir dvi dvi-am html \ html-am info info-am install install-am install-data \ - install-data-am install-dist_configDATA install-dvi \ - install-dvi-am install-exec install-exec-am install-html \ - install-html-am install-info install-info-am install-man \ - install-pdf install-pdf-am install-ps install-ps-am \ - install-strip installcheck installcheck-am installdirs \ - maintainer-clean maintainer-clean-generic mostlyclean \ - mostlyclean-generic pdf pdf-am ps ps-am tags-am uninstall \ - uninstall-am uninstall-dist_configDATA + install-data-am install-dist_chartsconfigDATA \ + install-dist_configDATA install-dist_healthconfigDATA \ + install-dist_nodeconfigDATA install-dist_pythonconfigDATA \ + install-dvi install-dvi-am install-exec install-exec-am \ + install-html install-html-am install-info install-info-am \ + install-man install-pdf install-pdf-am install-ps \ + install-ps-am install-strip installcheck installcheck-am \ + installdirs maintainer-clean maintainer-clean-generic \ + mostlyclean mostlyclean-generic pdf pdf-am ps ps-am tags-am \ + uninstall uninstall-am uninstall-dist_chartsconfigDATA \ + uninstall-dist_configDATA uninstall-dist_healthconfigDATA \ + uninstall-dist_nodeconfigDATA uninstall-dist_pythonconfigDATA # Tell versions [3.59,3.63) of GNU make to not export all variables. diff --git a/conf.d/apps_groups.conf b/conf.d/apps_groups.conf index 887563c44..0a6f55cd7 100644 --- a/conf.d/apps_groups.conf +++ b/conf.d/apps_groups.conf @@ -28,7 +28,7 @@ # *name* substring mode: will search for 'name' in the whole command line (/proc/PID/cmdline) # # If you enter even just one *name* (substring), apps.plugin will process -# /proc/PID/cmdline for all processes, on every iteration. +# /proc/PID/cmdline for all processes, just once (when they are first seen). # # To add process names with single quotes, enclose them in double quotes # example: "process with this ' single quote" @@ -44,53 +44,176 @@ # You can add any number of groups you like. Only the ones found running will # affect the charts generated. However, producing charts with hundreds of # dimensions may slow down your web browser. +# +# The order of the entries in this list is important: the first that matches +# a process is used, so put important ones at the top. Processes not matched +# by any row, will inherit it from their parents or children. +# +# The order also controls the order of the dimensions on the generated charts +# (although applications started after apps.plugin is started, will be appended +# to the existing list of dimensions the netdata daemon maintains). + +# ----------------------------------------------------------------------------- +# NETDATA processes accounting + +# netdata main process +netdata: netdata + +# netdata known plugins +# plugins not defined here will be accumulated in netdata, above +apps.plugin: apps.plugin +charts.d.plugin: *charts.d.plugin* +node.d.plugin: *node.d.plugin* +python.d.plugin: *python.d.plugin* +tc-qos-helper: *tc-qos-helper.sh* -compile: cc1 cc1plus as gcc* ld make automake autoconf git -rsync: rsync -media: mplayer vlc xine mediatomb omxplayer* kodi* xbmc* mediacenter eventlircd -squid: squid* c-icap -apache: apache* httpd -mysql: mysql* -asterisk: asterisk -opensips: opensips* stund -radius: radius* +# ----------------------------------------------------------------------------- +# authentication/authorization related servers + +auth: radius* openldap* ldap* fail2ban: fail2ban* -mail: dovecot imapd pop3d -postfix: master -nginx: nginx + +# ----------------------------------------------------------------------------- +# web/ftp servers + +httpd: apache* httpd nginx* lighttpd +proxy: squid* c-icap squidGuard varnish* +php: php* +ftpd: proftpd in.tftpd vsftpd +uwsgi: uwsgi +unicorn: *unicorn* + +# ----------------------------------------------------------------------------- +# database servers + +sql: mysqld* mariad* postgres* +nosql: mongod redis* + +# ----------------------------------------------------------------------------- +# email servers + +email: dovecot imapd pop3d amavis* master zmstat* zmmailboxdmgr qmgr oqmgr + +# ----------------------------------------------------------------------------- +# networking and VPN servers + +ppp: ppp* +vpn: openvpn pptp* cjdroute +wifi: hostapd wpa_supplicant + +# ----------------------------------------------------------------------------- +# high availability and balancers + +camo: *camo* +balancer: ipvs_* haproxy +ha: corosync hs_logd ha_logd stonithd + +# ----------------------------------------------------------------------------- +# telephony + +pbx: asterisk safe_asterisk *vicidial* +sip: opensips* stund +murmur: murmurd +vines: *vines* + +# ----------------------------------------------------------------------------- +# monitoring + +logs: ulogd* syslog* rsyslog* logrotate +nms: snmpd vnstatd smokeping zabbix* monit munin* mon openhpid watchdog tailon nrpe splunk: splunkd -mongo: mongod -lighttpd: lighttpd -ftpd: proftpd in.tftpd + +# ----------------------------------------------------------------------------- +# file systems and file servers + samba: smbd nmbd winbindd nfs: rpcbind rpc.* nfs* +zfs: spl_* z_* txg_* zil_* arc_* l2arc* +btrfs: btrfs* + +# ----------------------------------------------------------------------------- +# containers & virtual machines + +containers: lxc* docker* +VMs: vbox* VBox* qemu* + +# ----------------------------------------------------------------------------- +# ssh servers and clients + ssh: ssh* scp -X: X lightdm xdm pulseaudio gkrellm -xfce: xfwm4 xfdesktop xfce* Thunar xfsettingsd -gnome: gnome-* gdm gconfd-2 -named: named rncd -clam: clam* *clam -cups: cups* -ntp: ntp* -deluge: deluge* -vbox: vbox* VBox* -log: ulogd syslog* rsyslog* logrotate -nms: snmpd vnstatd smokeping zabbix* monit munin* mon openhpid -ppp: ppp* pptp* -inetd: inetd xinetd -openvpn: openvpn -cjdns: cjdroute -cron: cron atd -ha: corosync hs_logd ha_logd stonithd -ipvs: ipvs_* + +# ----------------------------------------------------------------------------- +# print servers and clients + +print: cups* lpd lpq + +# ----------------------------------------------------------------------------- +# time servers and clients + +time: ntp* + +# ----------------------------------------------------------------------------- +# dhcp servers and clients + +dhcp: *dhcp* + +# ----------------------------------------------------------------------------- +# name servers and clients + +named: named rncd dig + +# ----------------------------------------------------------------------------- +# installation / compilation / debugging + +build: cc1 cc1plus as gcc* ld make automake autoconf autoreconf git valgrind* + +# ----------------------------------------------------------------------------- +# antivirus + +antivirus: clam* *clam + +# ----------------------------------------------------------------------------- +# torrent clients + +torrents: *deluge* transmission* *SickBeard* + +# ----------------------------------------------------------------------------- +# backup servers and clients + +backup: rsync bacula* + +# ----------------------------------------------------------------------------- +# cron + +cron: cron atd anacron + +# ----------------------------------------------------------------------------- +# UPS + +ups: upsmon upsd */nut/* + +# ----------------------------------------------------------------------------- +# Kernel / System + +system: systemd* udisks* udevd* *udevd connmand ipv6_addrconf dbus-* inetd xinetd mdadm kernel: kthreadd kauditd lockd khelper kdevtmpfs khungtaskd rpciod fsnotify_mark kthrotld iscsi_eh deferwq -netdata: netdata -crsproxy: crsproxy -wifi: hostapd wpa_supplicant -system: systemd* udisks* udevd connmand ipv6_addrconf dbus-* ksmd: ksmd -lxc: lxc* -zfs-spl: spl_* -zfs-posix: z_* -zfs-txg: txg_* zil_* -zfs-arc: arc_* l2arc* + +# ----------------------------------------------------------------------------- +# media players, servers, clients + +media: mplayer vlc xine mediatomb omxplayer* kodi* xbmc* mediacenter eventlircd mpd minidlnad mt-daapd avahi* + +# ----------------------------------------------------------------------------- +# X + +X: X lightdm xdm pulseaudio gkrellm xfwm4 xfdesktop xfce* Thunar xfsettingsd xfconfd gnome-* gdm gconfd-2 *gvfsd gvfsd* kdm slim + +# ----------------------------------------------------------------------------- +# other application servers + +crsproxy: crsproxy +sidekiq: *sidekiq* +java: java +chat: irssi +ipfs: ipfs diff --git a/conf.d/charts.d.conf b/conf.d/charts.d.conf index daec33251..acb2a6fae 100644 --- a/conf.d/charts.d.conf +++ b/conf.d/charts.d.conf @@ -29,14 +29,35 @@ # ----------------------------------------------------------------------------- # the default enable/disable for all charts.d collectors -#enable_all_charts="yes" - -# per charts.d collector enable/disable -#nut=yes -#squid=yes -#postfix=yes -#sensors=yes -#cpufreq=yes -#mysql=yes -#example=yes -#load_average=yes +# the default is "yes" +# enable_all_charts="yes" + +# BY DEFAULT ENABLED MODULES +# ap=yes +# nut=yes +# opensips=yes + +# ----------------------------------------------------------------------------- +# THESE NEED TO BE SET TO "force" TO BE ENABLED + +# Nothing useful. +# Just an example charts.d plugin you can use as a template. +# example=force + +# OLD MODULES THAT ARE NOW SERVED BY python.d.plugin +# apache=force +# cpufreq=force +# exim=force +# hddtemp=force +# mysql=force +# nginx=force +# phpfpm=force +# postfix=force +# sensors=force +# squid=force +# tomcat=force + +# OLD MODULES THAT ARE NOW SERVED BY NETDATA DAEMON +# cpu_apps=force +# mem_apps=force +# load_average=force diff --git a/conf.d/health.d/apache.conf b/conf.d/health.d/apache.conf new file mode 100644 index 000000000..1fddbc99f --- /dev/null +++ b/conf.d/health.d/apache.conf @@ -0,0 +1,13 @@ + +# make sure apache is running + +template: apache_last_collected_secs + on: apache.requests + calc: $now - $last_collected_t + every: 10s + warn: $this > ( 5 * $update_every) + crit: $this > (10 * $update_every) + units: seconds ago + info: number of seconds since the last successful data collection + + diff --git a/conf.d/health.d/cpu.conf b/conf.d/health.d/cpu.conf new file mode 100644 index 000000000..9332e508a --- /dev/null +++ b/conf.d/health.d/cpu.conf @@ -0,0 +1,24 @@ + +template: 5min_cpu_pcent + on: system.cpu + lookup: average -5m unaligned of user,system,nice,softirq,irq,guest,guest_nice + every: 1m + warn: $this > 90 + units: % + info: average cpu utilization for the last 5 minutes + +template: 5min_iowait_cpu_pcent + on: system.cpu + lookup: average -5m unaligned of iowait + every: 1m + warn: $this > 10 + units: % + info: average wait I/O for the last 5 minutes + +template: 20min_steal_cpu_pcent + on: system.cpu + lookup: average -20m unaligned of steal + every: 5m + warn: $this > 10 + units: % + info: average stolen CPU time for the last 20 minutes diff --git a/conf.d/health.d/disks.conf b/conf.d/health.d/disks.conf new file mode 100644 index 000000000..c38f1a0a0 --- /dev/null +++ b/conf.d/health.d/disks.conf @@ -0,0 +1,85 @@ +# ----------------------------------------------------------------------------- +# low disk space + +# checking the latest collected values +# raise an alarm if the disk is low on +# available disk space + +template: disk_full_percent + on: disk.space + calc: $used * 100 / ($avail + $used) + every: 1m + warn: $this > 80 + crit: $this > 95 + units: % + info: current disk space usage + + +# ----------------------------------------------------------------------------- +# disk fill rate + +# calculate the rate the disk fills +# use as base, the available space change +# during the last 30 minutes + +# this is just a calculation - it has no alarm +# we will use it in the next template to find +# the hours remaining + +template: disk_fill_rate + on: disk.space + lookup: max -1s at -30m unaligned of avail + calc: ($this - $avail) / ($now - $after) + every: 15s + units: MB/s + info: average rate the disk fills up (positive), or frees up (negative) space, for the last 30 minutes + + +# calculate the hours remaining +# if the disk continues to fill +# in this rate + +template: disk_full_after_hours + on: disk.space + calc: $avail / $disk_fill_rate / 3600 + every: 10s + warn: $this > 0 and $this < 48 + crit: $this > 0 and $this < 24 + units: hours + info: estimated time the disk will run out of space, if the system continues to add data with the rate of the last 30 minutes + + +# ----------------------------------------------------------------------------- +# disk congestion + +# raise an alarm if the disk is congested +# by calculating the average disk utilization +# for the last 10 minutes + +template: 10min_disk_utilization + on: disk.util + lookup: average -10m unaligned + every: 1m + green: 90 + red: 98 + warn: $this > $green + crit: $this > $red + units: % + info: the percentage of time the disk was busy, during the last 10 minutes + + +# raise an alarm if the disk backlog +# is above 1000ms (1s) per second +# for 10 minutes +# (i.e. the disk cannot catch up) + +template: 10min_disk_backlog + on: disk.backlog + lookup: average -10m unaligned + every: 1m + green: 1000 + red: 2000 + warn: $this > $green + crit: $this > $red + units: ms + info: average of the kernel estimated disk backlog, for the last 10 minutes diff --git a/conf.d/health.d/entropy.conf b/conf.d/health.d/entropy.conf new file mode 100644 index 000000000..6f8b6e851 --- /dev/null +++ b/conf.d/health.d/entropy.conf @@ -0,0 +1,13 @@ + +# check if entropy is too low +# the alarm is checked every 1 minute +# and examines the last 30 minutes of data + + alarm: min_30min_entropy + on: system.entropy + lookup: min -30m unaligned + every: 1m + warn: $this < 200 + crit: $this < 100 + units: entries + info: minimum entries in the random numbers pool (entropy), for the last 30 minutes diff --git a/conf.d/health.d/memcached.conf b/conf.d/health.d/memcached.conf new file mode 100644 index 000000000..05ff14711 --- /dev/null +++ b/conf.d/health.d/memcached.conf @@ -0,0 +1,46 @@ + +# make sure memcached is running + +template: memcached_last_collected_secs + on: memcached.cache + calc: $now - $last_collected_t + every: 10s + warn: $this > ( 5 * $update_every) + crit: $this > (10 * $update_every) + units: seconds ago + info: number of seconds since the last successful data collection + + +# detect if memcached cache is full + +template: cache_full_pcent + on: memcached.cache + calc: $used * 100 / ($used + $available) + every: 10s + warn: $this > 80 + crit: $this > 90 + units: % + info: current cache memory usage + + +# find the rate memcached cache is filling + +template: cache_fill_rate + on: memcached.cache + lookup: max -1s at -30m unaligned of available + calc: ($this - $available) / ($now - $after) + every: 15s + units: KB/s + info: average rate the cache fills up (positive), or frees up (negative) space, for the last 30 minutes + + +# find the hours remaining until memcached cache is full + +template: cache_full_after_hours + on: memcached.cache + calc: $available / $cache_fill_rate / 3600 + every: 10s + warn: $this > 0 and $this < 48 + crit: $this > 0 and $this < 24 + units: hours + info: estimated time the cache will run out of space, if the system continues to add data with the rate of the last 30 minutes diff --git a/conf.d/health.d/named.conf b/conf.d/health.d/named.conf new file mode 100644 index 000000000..e46d1d330 --- /dev/null +++ b/conf.d/health.d/named.conf @@ -0,0 +1,12 @@ + +# make sure named is running + +template: named_last_collected_secs + on: named.global_queries + calc: $now - $last_collected_t + every: 10s + warn: $this > ( 5 * $update_every) + crit: $this > (10 * $update_every) + units: seconds ago + info: number of seconds since the last successful data collection + diff --git a/conf.d/health.d/net.conf b/conf.d/health.d/net.conf new file mode 100644 index 000000000..f65bc4fcb --- /dev/null +++ b/conf.d/health.d/net.conf @@ -0,0 +1,27 @@ + +# check if an interface is dropping packets +# the alarm is checked every 10 seconds +# and examines the last 30 minutes of data + +template: 30min_packet_drops + on: net.drops + lookup: sum -30m unaligned absolute + every: 1m + crit: $this > 0 + units: packets + info: dropped packets in the last 30 minutes + + +# check if an interface is having FIFO +# buffer errors +# the alarm is checked every 10 seconds +# and examines the last 30 minutes of data + +template: 30min_fifo_errors + on: net.fifo + lookup: sum -30m unaligned absolute + every: 1m + crit: $this > 0 + units: errors + info: network interface fifo errors in the last 30 minutes + diff --git a/conf.d/health.d/nginx.conf b/conf.d/health.d/nginx.conf new file mode 100644 index 000000000..da13008e3 --- /dev/null +++ b/conf.d/health.d/nginx.conf @@ -0,0 +1,12 @@ + +# make sure nginx is running + +template: nginx_last_collected_secs + on: nginx.requests + calc: $now - $last_collected_t + every: 10s + warn: $this > ( 5 * $update_every) + crit: $this > (10 * $update_every) + units: seconds ago + info: number of seconds since the last successful data collection + diff --git a/conf.d/health.d/qos.conf b/conf.d/health.d/qos.conf new file mode 100644 index 000000000..ac3bf8ff4 --- /dev/null +++ b/conf.d/health.d/qos.conf @@ -0,0 +1,12 @@ + +# check if a QoS class is dropping packets +# the alarm is checked every 10 seconds +# and examines the last minute of data + +#template: 10min_qos_packet_drops +# on: tc.qos_dropped +# lookup: sum -10m unaligned absolute +# every: 30s +# warn: $this > 0 +# units: packets +# info: dropped packets in the last 30 minutes diff --git a/conf.d/health.d/ram.conf b/conf.d/health.d/ram.conf new file mode 100644 index 000000000..1d3681128 --- /dev/null +++ b/conf.d/health.d/ram.conf @@ -0,0 +1,9 @@ + + alarm: used_ram_pcent + on: system.ram + calc: $used * 100 / ($used + $cached + $free) + every: 10s + warn: $this > 80 + crit: $this > 90 + units: % + info: system RAM usage diff --git a/conf.d/health.d/redis.conf b/conf.d/health.d/redis.conf new file mode 100644 index 000000000..3750176c5 --- /dev/null +++ b/conf.d/health.d/redis.conf @@ -0,0 +1,12 @@ + +# make sure redis is running + +template: redis_last_collected_secs + on: redis.operations + calc: $now - $last_collected_t + every: 10s + warn: $this > ( 5 * $update_every) + crit: $this > (10 * $update_every) + units: seconds ago + info: number of seconds since the last successful data collection + diff --git a/conf.d/health.d/squid.conf b/conf.d/health.d/squid.conf new file mode 100644 index 000000000..cc5ce1c3a --- /dev/null +++ b/conf.d/health.d/squid.conf @@ -0,0 +1,12 @@ + +# make sure squid is running + +template: squid_last_collected_secs + on: squid.clients_requests + calc: $now - $last_collected_t + every: 10s + warn: $this > ( 5 * $update_every) + crit: $this > (10 * $update_every) + units: seconds ago + info: number of seconds since the last successful data collection + diff --git a/conf.d/health.d/swap.conf b/conf.d/health.d/swap.conf new file mode 100644 index 000000000..552dd310a --- /dev/null +++ b/conf.d/health.d/swap.conf @@ -0,0 +1,20 @@ + + alarm: 30min_ram_swapped_out + on: system.swapio + lookup: sum -30m unaligned absolute of out + # we have to convert KB to MB by dividing $this (i.e. the result of the lookup) with 1024 + calc: $this / 1024 * 100 / ( $system.ram.used + $system.ram.cached + $system.ram.free ) + every: 1m + warn: $this > 1 + crit: $this > 10 + units: % of RAM + info: the sum of all memory swapped out during the last 30 minutes, as a percentage of the available RAM + + alarm: pcent_of_ram_in_swap + on: system.swap + calc: $used * 100 / ( $system.ram.used + $system.ram.cached + $system.ram.free ) + every: 10s + warn: $this > 10 + crit: $this > 50 + units: % of RAM + info: the currently used swap space, as a percentage of the available RAM diff --git a/conf.d/python.d.conf b/conf.d/python.d.conf new file mode 100644 index 000000000..940bd9183 --- /dev/null +++ b/conf.d/python.d.conf @@ -0,0 +1,42 @@ +# netdata python.d.plugin configuration +# +# This file is in YaML format. +# Generally the format is: +# +# name: value +# + +# Enable / disable the whole python.d.plugin (all its modules) +enabled: yes + +# Prevent log flood +# Define how many log messages can be written to log file in one log_interval +logs_per_interval: 200 + +# Define how long is one logging interval (in seconds) +log_interval: 3600 + +# ---------------------------------------------------------------------- +# Enable / Disable python.d.plugin modules +# +# The default for all modules is enabled (yes). +# Setting any of these to no will disable it. + +# apache: yes +# apache_cache: yes +# cpufreq: yes +# dovecot: yes +example: no +# exim: yes +# hddtemp: yes +# ipfs: yes +# memcached: yes +# mysql: yes +# nginx: yes +# nginx_log: yes +# phpfpm: yes +# postfix: yes +# redis: yes +# sensors: yes +# squid: yes +# tomcat: yes diff --git a/conf.d/python.d/apache.conf b/conf.d/python.d/apache.conf new file mode 100644 index 000000000..5b151ef70 --- /dev/null +++ b/conf.d/python.d/apache.conf @@ -0,0 +1,80 @@ +# netdata python.d.plugin configuration for apache +# +# This file is in YaML format. Generally the format is: +# +# name: value +# +# There are 2 sections: +# - global variables +# - one or more JOBS +# +# JOBS allow you to collect values from multiple sources. +# Each source will have its own set of charts. +# +# JOB parameters have to be indented (using spaces only, example below). + +# ---------------------------------------------------------------------- +# Global Variables +# These variables set the defaults for all JOBs, however each JOB +# may define its own, overriding the defaults. + +# update_every sets the default data collection frequency. +# If unset, the python.d.plugin default is used. +# update_every: 1 + +# priority controls the order of charts at the netdata dashboard. +# Lower numbers move the charts towards the top of the page. +# If unset, the default for python.d.plugin is used. +# priority: 60000 + +# retries sets the number of retries to be made in case of failures. +# If unset, the default for python.d.plugin is used. +# Attempts to restore the service are made once every update_every +# and only if the module has collected values in the past. +# retries: 5 + +# ---------------------------------------------------------------------- +# JOBS (data collection sources) +# +# The default JOBS share the same *name*. JOBS with the same name +# are mutually exclusive. Only one of them will be allowed running at +# any time. This allows autodetection to try several alternatives and +# pick the one that works. +# +# Any number of jobs is supported. +# +# All python.d.plugin JOBS (for all its modules) support a set of +# predefined parameters. These are: +# +# job_name: +# name: myname # the JOB's name as it will appear at the +# # dashboard (by default is the job_name) +# # JOBs sharing a name are mutually exclusive +# update_every: 1 # the JOB's data collection frequency +# priority: 60000 # the JOB's order on the dashboard +# retries: 5 # the JOB's number of restoration attempts +# +# Additionally to the above, apache also supports the following: +# +# url: 'URL' # the URL to fetch apache's mod_status stats +# +# if the URL is password protected, the following are supported: +# +# user: 'username' +# pass: 'password' + +# ---------------------------------------------------------------------- +# AUTO-DETECTION JOBS +# only one of them will run (they have the same name) + +localhost: + name : 'local' + url : 'http://localhost/server-status?auto' + +localipv4: + name : 'local' + url : 'http://127.0.0.1/server-status?auto' + +localipv6: + name : 'local' + url : 'http://::1/server-status?auto' diff --git a/conf.d/python.d/apache_cache.conf b/conf.d/python.d/apache_cache.conf new file mode 100644 index 000000000..98eecd0e8 --- /dev/null +++ b/conf.d/python.d/apache_cache.conf @@ -0,0 +1,76 @@ +# netdata python.d.plugin configuration for apache cache +# +# This file is in YaML format. Generally the format is: +# +# name: value +# +# There are 2 sections: +# - global variables +# - one or more JOBS +# +# JOBS allow you to collect values from multiple sources. +# Each source will have its own set of charts. +# +# JOB parameters have to be indented (using spaces only, example below). + +# ---------------------------------------------------------------------- +# Global Variables +# These variables set the defaults for all JOBs, however each JOB +# may define its own, overriding the defaults. + +# update_every sets the default data collection frequency. +# If unset, the python.d.plugin default is used. +# update_every: 1 + +# priority controls the order of charts at the netdata dashboard. +# Lower numbers move the charts towards the top of the page. +# If unset, the default for python.d.plugin is used. +# priority: 60000 + +# retries sets the number of retries to be made in case of failures. +# If unset, the default for python.d.plugin is used. +# Attempts to restore the service are made once every update_every +# and only if the module has collected values in the past. +# retries: 5 + +# ---------------------------------------------------------------------- +# JOBS (data collection sources) +# +# The default JOBS share the same *name*. JOBS with the same name +# are mutually exclusive. Only one of them will be allowed running at +# any time. This allows autodetection to try several alternatives and +# pick the one that works. +# +# Any number of jobs is supported. +# +# All python.d.plugin JOBS (for all its modules) support a set of +# predefined parameters. These are: +# +# job_name: +# name: myname # the JOB's name as it will appear at the +# # dashboard (by default is the job_name) +# # JOBs sharing a name are mutually exclusive +# update_every: 1 # the JOB's data collection frequency +# priority: 60000 # the JOB's order on the dashboard +# retries: 5 # the JOB's number of restoration attempts +# +# Additionally to the above, apache_cache also supports the following: +# +# path: 'PATH' # the path to apache's cache.log +# + +# ---------------------------------------------------------------------- +# AUTO-DETECTION JOBS +# only one of them will run (they have the same name) + +apache: + name: 'local' + path: '/var/log/apache/cache.log' + +apache2: + name: 'local' + path: '/var/log/apache2/cache.log' + +httpd: + name: 'local' + path: '/var/log/httpd/cache.log' diff --git a/conf.d/python.d/cpufreq.conf b/conf.d/python.d/cpufreq.conf new file mode 100644 index 000000000..10c96917f --- /dev/null +++ b/conf.d/python.d/cpufreq.conf @@ -0,0 +1,37 @@ +# netdata python.d.plugin configuration for cpufreq +# +# This file is in YaML format. Generally the format is: +# +# name: value +# +# There are 2 sections: +# - global variables +# - one or more JOBS +# +# JOBS allow you to collect values from multiple sources. +# Each source will have its own set of charts. +# +# JOB parameters have to be indented (using spaces only, example below). + +# ---------------------------------------------------------------------- +# Global Variables +# These variables set the defaults for all JOBs, however each JOB +# may define its own, overriding the defaults. + +# update_every sets the default data collection frequency. +# If unset, the python.d.plugin default is used. +# update_every: 1 + +# priority controls the order of charts at the netdata dashboard. +# Lower numbers move the charts towards the top of the page. +# If unset, the default for python.d.plugin is used. +# priority: 60000 + +# retries sets the number of retries to be made in case of failures. +# If unset, the default for python.d.plugin is used. +# Attempts to restore the service are made once every update_every +# and only if the module has collected values in the past. +# retries: 5 + +# The directory to search for the file scaling_cur_freq +sys_dir: "/sys/devices" diff --git a/conf.d/python.d/dovecot.conf b/conf.d/python.d/dovecot.conf new file mode 100644 index 000000000..917c5272e --- /dev/null +++ b/conf.d/python.d/dovecot.conf @@ -0,0 +1,89 @@ +# netdata python.d.plugin configuration for dovecot +# +# This file is in YaML format. Generally the format is: +# +# name: value +# +# There are 2 sections: +# - global variables +# - one or more JOBS +# +# JOBS allow you to collect values from multiple sources. +# Each source will have its own set of charts. +# +# JOB parameters have to be indented (using spaces only, example below). + +# ---------------------------------------------------------------------- +# Global Variables +# These variables set the defaults for all JOBs, however each JOB +# may define its own, overriding the defaults. + +# update_every sets the default data collection frequency. +# If unset, the python.d.plugin default is used. +# update_every: 1 + +# priority controls the order of charts at the netdata dashboard. +# Lower numbers move the charts towards the top of the page. +# If unset, the default for python.d.plugin is used. +# priority: 60000 + +# retries sets the number of retries to be made in case of failures. +# If unset, the default for python.d.plugin is used. +# Attempts to restore the service are made once every update_every +# and only if the module has collected values in the past. +# retries: 5 + +# ---------------------------------------------------------------------- +# JOBS (data collection sources) +# +# The default JOBS share the same *name*. JOBS with the same name +# are mutually exclusive. Only one of them will be allowed running at +# any time. This allows autodetection to try several alternatives and +# pick the one that works. +# +# Any number of jobs is supported. +# +# All python.d.plugin JOBS (for all its modules) support a set of +# predefined parameters. These are: +# +# job_name: +# name: myname # the JOB's name as it will appear at the +# # dashboard (by default is the job_name) +# # JOBs sharing a name are mutually exclusive +# update_every: 1 # the JOB's data collection frequency +# priority: 60000 # the JOB's order on the dashboard +# retries: 5 # the JOB's number of restoration attempts +# +# Additionally to the above, dovecot also supports the following: +# +# socket: 'path/to/dovecot/stats' +# +# or +# host: 'IP or HOSTNAME' # the host to connect to +# port: PORT # the port to connect to +# +# + +# ---------------------------------------------------------------------- +# AUTO-DETECTION JOBS +# only one of them will run (they have the same name) + +localhost: + name : 'local' + host : 'localhost' + port : 24242 + +localipv4: + name : 'local' + host : '127.0.0.1' + port : 24242 + +localipv6: + name : 'local' + host : '::1' + port : 24242 + +localsocket: + name : 'local' + socket : '/var/run/dovecot/stats' + diff --git a/conf.d/python.d/example.conf b/conf.d/python.d/example.conf new file mode 100644 index 000000000..31f9a49a0 --- /dev/null +++ b/conf.d/python.d/example.conf @@ -0,0 +1,63 @@ +# netdata python.d.plugin configuration for example +# +# This file is in YaML format. Generally the format is: +# +# name: value +# +# There are 2 sections: +# - global variables +# - one or more JOBS +# +# JOBS allow you to collect values from multiple sources. +# Each source will have its own set of charts. +# +# JOB parameters have to be indented (using spaces only, example below). + +# ---------------------------------------------------------------------- +# Global Variables +# These variables set the defaults for all JOBs, however each JOB +# may define its own, overriding the defaults. + +# update_every sets the default data collection frequency. +# If unset, the python.d.plugin default is used. +# update_every: 1 + +# priority controls the order of charts at the netdata dashboard. +# Lower numbers move the charts towards the top of the page. +# If unset, the default for python.d.plugin is used. +# priority: 60000 + +# retries sets the number of retries to be made in case of failures. +# If unset, the default for python.d.plugin is used. +# Attempts to restore the service are made once every update_every +# and only if the module has collected values in the past. +# retries: 5 + +# ---------------------------------------------------------------------- +# JOBS (data collection sources) +# +# The default JOBS share the same *name*. JOBS with the same name +# are mutually exclusive. Only one of them will be allowed running at +# any time. This allows autodetection to try several alternatives and +# pick the one that works. +# +# Any number of jobs is supported. +# +# All python.d.plugin JOBS (for all its modules) support a set of +# predefined parameters. These are: +# +# job_name: +# name: myname # the JOB's name as it will appear at the +# # dashboard (by default is the job_name) +# # JOBs sharing a name are mutually exclusive +# update_every: 1 # the JOB's data collection frequency +# priority: 60000 # the JOB's order on the dashboard +# retries: 5 # the JOB's number of restoration attempts +# +# Additionally to the above, example also supports the following: +# +# - none +# +# ---------------------------------------------------------------------- +# AUTO-DETECTION JOBS +# only one of them will run (they have the same name) diff --git a/conf.d/python.d/exim.conf b/conf.d/python.d/exim.conf new file mode 100644 index 000000000..6aca13c34 --- /dev/null +++ b/conf.d/python.d/exim.conf @@ -0,0 +1,86 @@ +# netdata python.d.plugin configuration for postfix +# +# This file is in YaML format. Generally the format is: +# +# name: value +# +# There are 2 sections: +# - global variables +# - one or more JOBS +# +# JOBS allow you to collect values from multiple sources. +# Each source will have its own set of charts. +# +# JOB parameters have to be indented (using spaces only, example below). + +# ---------------------------------------------------------------------- +# Global Variables +# These variables set the defaults for all JOBs, however each JOB +# may define its own, overriding the defaults. + +# update_every sets the default data collection frequency. +# If unset, the python.d.plugin default is used. +# exim is slow, so once every 10 seconds +update_every: 10 + +# priority controls the order of charts at the netdata dashboard. +# Lower numbers move the charts towards the top of the page. +# If unset, the default for python.d.plugin is used. +# priority: 60000 + +# retries sets the number of retries to be made in case of failures. +# If unset, the default for python.d.plugin is used. +# Attempts to restore the service are made once every update_every +# and only if the module has collected values in the past. +# retries: 5 + +# ---------------------------------------------------------------------- +# JOBS (data collection sources) +# +# The default JOBS share the same *name*. JOBS with the same name +# are mutually exclusive. Only one of them will be allowed running at +# any time. This allows autodetection to try several alternatives and +# pick the one that works. +# +# Any number of jobs is supported. +# +# All python.d.plugin JOBS (for all its modules) support a set of +# predefined parameters. These are: +# +# job_name: +# name: myname # the JOB's name as it will appear at the +# # dashboard (by default is the job_name) +# # JOBs sharing a name are mutually exclusive +# update_every: 1 # the JOB's data collection frequency +# priority: 60000 # the JOB's order on the dashboard +# retries: 5 # the JOB's number of restoration attempts +# +# Additionally to the above, postfix also supports the following: +# +# command: 'exim -bpc' # the command to run +# + +# ---------------------------------------------------------------------- +# REQUIRED exim CONFIGURATION +# +# netdata will query exim as user netdata. +# By default exim will refuse to respond. +# +# To allow querying exim as non-admin user, please set the following +# to your exim configuration: +# +# queue_list_requires_admin = false +# +# Your exim configuration should be in +# +# /etc/exim/exim4.conf +# or +# /etc/exim4/conf.d/main/000_local_options +# +# Please consult your distribution information to find the exact file. + +# ---------------------------------------------------------------------- +# AUTO-DETECTION JOBS + +local: + command: 'exim -bpc' diff --git a/conf.d/python.d/hddtemp.conf b/conf.d/python.d/hddtemp.conf new file mode 100644 index 000000000..0c78449b4 --- /dev/null +++ b/conf.d/python.d/hddtemp.conf @@ -0,0 +1,90 @@ +# netdata python.d.plugin configuration for hddtemp +# +# This file is in YaML format. Generally the format is: +# +# name: value +# +# There are 2 sections: +# - global variables +# - one or more JOBS +# +# JOBS allow you to collect values from multiple sources. +# Each source will have its own set of charts. +# +# JOB parameters have to be indented (using spaces only, example below). + +# ---------------------------------------------------------------------- +# Global Variables +# These variables set the defaults for all JOBs, however each JOB +# may define its own, overriding the defaults. + +# update_every sets the default data collection frequency. +# If unset, the python.d.plugin default is used. +# update_every: 1 + +# priority controls the order of charts at the netdata dashboard. +# Lower numbers move the charts towards the top of the page. +# If unset, the default for python.d.plugin is used. +# priority: 60000 + +# retries sets the number of retries to be made in case of failures. +# If unset, the default for python.d.plugin is used. +# Attempts to restore the service are made once every update_every +# and only if the module has collected values in the past. +# retries: 5 + +# ---------------------------------------------------------------------- +# JOBS (data collection sources) +# +# The default JOBS share the same *name*. JOBS with the same name +# are mutually exclusive. Only one of them will be allowed running at +# any time. This allows autodetection to try several alternatives and +# pick the one that works. +# +# Any number of jobs is supported. +# +# All python.d.plugin JOBS (for all its modules) support a set of +# predefined parameters. These are: +# +# job_name: +# name: myname # the JOB's name as it will appear at the +# # dashboard (by default is the job_name) +# # JOBs sharing a name are mutually exclusive +# update_every: 1 # the JOB's data collection frequency +# priority: 60000 # the JOB's order on the dashboard +# retries: 5 # the JOB's number of restoration attempts +# +# Additionally to the above, hddtemp also supports the following: +# +# host: 'IP or HOSTNAME' # the host to connect to +# port: PORT # the port to connect to +# + +# By default this module will try to autodetect disks +# (autodetection works only for disk which names start with "sd"). +# However this can be overridden by setting variable `disks` to +# array of desired disks. Example for two disks: +# +# devices: +# - sda +# - sdb +# + +# ---------------------------------------------------------------------- +# AUTO-DETECTION JOBS +# only one of them will run (they have the same name) + +localhost: + name: 'local' + host: 'localhost' + port: 7634 + +localipv4: + name: 'local' + host: '127.0.0.1' + port: 7634 + +localipv6: + name: 'local' + host: '::1' + port: 7634 diff --git a/conf.d/python.d/ipfs.conf b/conf.d/python.d/ipfs.conf new file mode 100644 index 000000000..e039026cc --- /dev/null +++ b/conf.d/python.d/ipfs.conf @@ -0,0 +1,67 @@ +# netdata python.d.plugin configuration for ipfs +# +# This file is in YaML format. Generally the format is: +# +# name: value +# +# There are 2 sections: +# - global variables +# - one or more JOBS +# +# JOBS allow you to collect values from multiple sources. +# Each source will have its own set of charts. +# +# JOB parameters have to be indented (using spaces only, example below). + +# ---------------------------------------------------------------------- +# Global Variables +# These variables set the defaults for all JOBs, however each JOB +# may define its own, overriding the defaults. + +# update_every sets the default data collection frequency. +# If unset, the python.d.plugin default is used. +# update_every: 1 + +# priority controls the order of charts at the netdata dashboard. +# Lower numbers move the charts towards the top of the page. +# If unset, the default for python.d.plugin is used. +# priority: 60000 + +# retries sets the number of retries to be made in case of failures. +# If unset, the default for python.d.plugin is used. +# Attempts to restore the service are made once every update_every +# and only if the module has collected values in the past. +# retries: 5 + +# ---------------------------------------------------------------------- +# JOBS (data collection sources) +# +# The default JOBS share the same *name*. JOBS with the same name +# are mutually exclusive. Only one of them will be allowed running at +# any time. This allows autodetection to try several alternatives and +# pick the one that works. +# +# Any number of jobs is supported. +# +# All python.d.plugin JOBS (for all its modules) support a set of +# predefined parameters. These are: +# +# job_name: +# name: myname # the JOB's name as it will appear at the +# # dashboard (by default is the job_name) +# # JOBs sharing a name are mutually exclusive +# update_every: 1 # the JOB's data collection frequency +# priority: 60000 # the JOB's order on the dashboard +# retries: 5 # the JOB's number of restoration attempts +# +# Additionally to the above, ipfs also supports the following: +# +# url: 'URL' # URL to the IPFS API +# +# ---------------------------------------------------------------------- +# AUTO-DETECTION JOBS +# only one of them will run (they have the same name) + +localhost: + name : 'local' + url : 'http://localhost:5001' diff --git a/conf.d/python.d/memcached.conf b/conf.d/python.d/memcached.conf new file mode 100644 index 000000000..f1723dc81 --- /dev/null +++ b/conf.d/python.d/memcached.conf @@ -0,0 +1,85 @@ +# netdata python.d.plugin configuration for memcached +# +# This file is in YaML format. Generally the format is: +# +# name: value +# +# There are 2 sections: +# - global variables +# - one or more JOBS +# +# JOBS allow you to collect values from multiple sources. +# Each source will have its own set of charts. +# +# JOB parameters have to be indented (using spaces only, example below). + +# ---------------------------------------------------------------------- +# Global Variables +# These variables set the defaults for all JOBs, however each JOB +# may define its own, overriding the defaults. + +# update_every sets the default data collection frequency. +# If unset, the python.d.plugin default is used. +# update_every: 1 + +# priority controls the order of charts at the netdata dashboard. +# Lower numbers move the charts towards the top of the page. +# If unset, the default for python.d.plugin is used. +# priority: 60000 + +# retries sets the number of retries to be made in case of failures. +# If unset, the default for python.d.plugin is used. +# Attempts to restore the service are made once every update_every +# and only if the module has collected values in the past. +# retries: 5 + +# ---------------------------------------------------------------------- +# JOBS (data collection sources) +# +# The default JOBS share the same *name*. JOBS with the same name +# are mutually exclusive. Only one of them will be allowed running at +# any time. This allows autodetection to try several alternatives and +# pick the one that works. +# +# Any number of jobs is supported. +# +# All python.d.plugin JOBS (for all its modules) support a set of +# predefined parameters. These are: +# +# job_name: +# name: myname # the JOB's name as it will appear at the +# # dashboard (by default is the job_name) +# # JOBs sharing a name are mutually exclusive +# update_every: 1 # the JOB's data collection frequency +# priority: 60000 # the JOB's order on the dashboard +# retries: 5 # the JOB's number of restoration attempts +# +# Additionally to the above, memcached also supports the following: +# +# socket: 'path/to/memcached.sock' +# +# or +# host: 'IP or HOSTNAME' # the host to connect to +# port: PORT # the port to connect to +# +# + +# ---------------------------------------------------------------------- +# AUTO-DETECTION JOBS +# only one of them will run (they have the same name) + +localhost: + name : 'local' + host : 'localhost' + port : 11211 + +localipv4: + name : 'local' + host : '127.0.0.1' + port : 11211 + +localipv6: + name : 'local' + host : '::1' + port : 11211 + diff --git a/conf.d/python.d/mysql.conf b/conf.d/python.d/mysql.conf new file mode 100644 index 000000000..d247b89a0 --- /dev/null +++ b/conf.d/python.d/mysql.conf @@ -0,0 +1,175 @@ +# netdata python.d.plugin configuration for mysql +# +# This file is in YaML format. Generally the format is: +# +# name: value +# +# There are 2 sections: +# - global variables +# - one or more JOBS +# +# JOBS allow you to collect values from multiple sources. +# Each source will have its own set of charts. +# +# JOB parameters have to be indented (using spaces only, example below). + +# ---------------------------------------------------------------------- +# Global Variables +# These variables set the defaults for all JOBs, however each JOB +# may define its own, overriding the defaults. + +# update_every sets the default data collection frequency. +# If unset, the python.d.plugin default is used. +# update_every: 1 + +# priority controls the order of charts at the netdata dashboard. +# Lower numbers move the charts towards the top of the page. +# If unset, the default for python.d.plugin is used. +# priority: 60000 + +# retries sets the number of retries to be made in case of failures. +# If unset, the default for python.d.plugin is used. +# Attempts to restore the service are made once every update_every +# and only if the module has collected values in the past. +# retries: 5 + +# ---------------------------------------------------------------------- +# JOBS (data collection sources) +# +# The default JOBS share the same *name*. JOBS with the same name +# are mutually exclusive. Only one of them will be allowed running at +# any time. This allows autodetection to try several alternatives and +# pick the one that works. +# +# Any number of jobs is supported. +# +# All python.d.plugin JOBS (for all its modules) support a set of +# predefined parameters. These are: +# +# job_name: +# name: myname # the JOB's name as it will appear at the +# # dashboard (by default is the job_name) +# # JOBs sharing a name are mutually exclusive +# update_every: 1 # the JOB's data collection frequency +# priority: 60000 # the JOB's order on the dashboard +# retries: 5 # the JOB's number of restoration attempts +# +# Additionally to the above, mysql also supports the following: +# +# socket: 'path/to/mysql.sock' +# +# or +# host: 'IP or HOSTNAME' # the host to connect to +# port: PORT # the port to connect to +# +# in all cases, the following can also be set: +# +# user: 'username' # the mysql username to use +# pass: 'password' # the mysql password to use +# + +# ---------------------------------------------------------------------- +# AUTO-DETECTION JOBS +# only one of them will run (they have the same name) + +mycnf1: + name : 'local' + 'my.cnf' : '/etc/my.cnf' + +mycnf2: + name : 'local' + 'my.cnf' : '/etc/mysql/my.cnf' + +socket1: + name : 'local' + # user : '' + # pass : '' + socket : '/var/run/mysqld/mysqld.sock' + +socket2: + name : 'local' + # user : '' + # pass : '' + socket : '/var/lib/mysql/mysql.sock' + +socket3: + name : 'local' + # user : '' + # pass : '' + socket : '/tmp/mysql.sock' + +tcp: + name : 'local' + # user : '' + # pass : '' + host : 'localhost' + port : '3306' + +tcpipv4: + name : 'local' + # user : '' + # pass : '' + host : '127.0.0.1' + port : '3306' + +tcpipv6: + name : 'local' + # user : '' + # pass : '' + host : '::1' + port : '3306' + + +# Now we try the same as above with user: root +# A few systems configure mysql to accept passwordless +# root access. + +mycnf1_root: + name : 'local' + user : 'root' + 'my.cnf' : '/etc/my.cnf' + +mycnf2_root: + name : 'local' + user : 'root' + 'my.cnf' : '/etc/mysql/my.cnf' + +socket1_root: + name : 'local' + user : 'root' + # pass : '' + socket : '/var/run/mysqld/mysqld.sock' + +socket2_root: + name : 'local' + user : 'root' + # pass : '' + socket : '/var/lib/mysql/mysql.sock' + +socket3_root: + name : 'local' + user : 'root' + # pass : '' + socket : '/tmp/mysql.sock' + +tcp_root: + name : 'local' + user : 'root' + # pass : '' + host : 'localhost' + port : '3306' + +tcpipv4_root: + name : 'local' + user : 'root' + # pass : '' + host : '127.0.0.1' + port : '3306' + +tcpipv6_root: + name : 'local' + user : 'root' + # pass : '' + host : '::1' + port : '3306' + diff --git a/conf.d/python.d/nginx.conf b/conf.d/python.d/nginx.conf new file mode 100644 index 000000000..1a27d67c5 --- /dev/null +++ b/conf.d/python.d/nginx.conf @@ -0,0 +1,82 @@ +# netdata python.d.plugin configuration for nginx +# +# This file is in YaML format. Generally the format is: +# +# name: value +# +# There are 2 sections: +# - global variables +# - one or more JOBS +# +# JOBS allow you to collect values from multiple sources. +# Each source will have its own set of charts. +# +# JOB parameters have to be indented (using spaces only, example below). + +# ---------------------------------------------------------------------- +# Global Variables +# These variables set the defaults for all JOBs, however each JOB +# may define its own, overriding the defaults. + +# update_every sets the default data collection frequency. +# If unset, the python.d.plugin default is used. +# update_every: 1 + +# priority controls the order of charts at the netdata dashboard. +# Lower numbers move the charts towards the top of the page. +# If unset, the default for python.d.plugin is used. +# priority: 60000 + +# retries sets the number of retries to be made in case of failures. +# If unset, the default for python.d.plugin is used. +# Attempts to restore the service are made once every update_every +# and only if the module has collected values in the past. +# retries: 5 + +# ---------------------------------------------------------------------- +# JOBS (data collection sources) +# +# The default JOBS share the same *name*. JOBS with the same name +# are mutually exclusive. Only one of them will be allowed running at +# any time. This allows autodetection to try several alternatives and +# pick the one that works. +# +# Any number of jobs is supported. +# +# All python.d.plugin JOBS (for all its modules) support a set of +# predefined parameters. These are: +# +# job_name: +# name: myname # the JOB's name as it will appear at the +# # dashboard (by default is the job_name) +# # JOBs sharing a name are mutually exclusive +# update_every: 1 # the JOB's data collection frequency +# priority: 60000 # the JOB's order on the dashboard +# retries: 5 # the JOB's number of restoration attempts +# +# Additionally to the above, nginx also supports the following: +# +# url: 'URL' # the URL to fetch nginx's status stats +# +# if the URL is password protected, the following are supported: +# +# user: 'username' +# pass: 'password' +# + +# ---------------------------------------------------------------------- +# AUTO-DETECTION JOBS +# only one of them will run (they have the same name) + +localhost: + name : 'local' + url : 'http://localhost/stub_status' + +localipv4: + name : 'local' + url : 'http://127.0.0.1/stub_status' + +localipv6: + name : 'local' + url : 'http://::1/stub_status' + diff --git a/conf.d/python.d/nginx_log.conf b/conf.d/python.d/nginx_log.conf new file mode 100644 index 000000000..6a53c5204 --- /dev/null +++ b/conf.d/python.d/nginx_log.conf @@ -0,0 +1,72 @@ +# netdata python.d.plugin configuration for nginx log +# +# This file is in YaML format. Generally the format is: +# +# name: value +# +# There are 2 sections: +# - global variables +# - one or more JOBS +# +# JOBS allow you to collect values from multiple sources. +# Each source will have its own set of charts. +# +# JOB parameters have to be indented (using spaces only, example below). + +# ---------------------------------------------------------------------- +# Global Variables +# These variables set the defaults for all JOBs, however each JOB +# may define its own, overriding the defaults. + +# update_every sets the default data collection frequency. +# If unset, the python.d.plugin default is used. +# update_every: 1 + +# priority controls the order of charts at the netdata dashboard. +# Lower numbers move the charts towards the top of the page. +# If unset, the default for python.d.plugin is used. +# priority: 60000 + +# retries sets the number of retries to be made in case of failures. +# If unset, the default for python.d.plugin is used. +# Attempts to restore the service are made once every update_every +# and only if the module has collected values in the past. +# retries: 5 + +# ---------------------------------------------------------------------- +# JOBS (data collection sources) +# +# The default JOBS share the same *name*. JOBS with the same name +# are mutually exclusive. Only one of them will be allowed running at +# any time. This allows autodetection to try several alternatives and +# pick the one that works. +# +# Any number of jobs is supported. +# +# All python.d.plugin JOBS (for all its modules) support a set of +# predefined parameters. These are: +# +# job_name: +# name: myname # the JOB's name as it will appear at the +# # dashboard (by default is the job_name) +# # JOBs sharing a name are mutually exclusive +# update_every: 1 # the JOB's data collection frequency +# priority: 60000 # the JOB's order on the dashboard +# retries: 5 # the JOB's number of restoration attempts +# +# Additionally to the above, nginx_log also supports the following: +# +# path: 'PATH' # the path to nginx's access.log +# + +# ---------------------------------------------------------------------- +# AUTO-DETECTION JOBS +# only one of them will run (they have the same name) + +nginx_log: + name: 'local' + path: '/var/log/nginx/access.log' + +nginx_log2: + name: 'local' + path: '/var/log/nginx/nginx-access.log' diff --git a/conf.d/python.d/phpfpm.conf b/conf.d/python.d/phpfpm.conf new file mode 100644 index 000000000..06d2367ae --- /dev/null +++ b/conf.d/python.d/phpfpm.conf @@ -0,0 +1,82 @@ +# netdata python.d.plugin configuration for PHP-FPM +# +# This file is in YaML format. Generally the format is: +# +# name: value +# +# There are 2 sections: +# - global variables +# - one or more JOBS +# +# JOBS allow you to collect values from multiple sources. +# Each source will have its own set of charts. +# +# JOB parameters have to be indented (using spaces only, example below). + +# ---------------------------------------------------------------------- +# Global Variables +# These variables set the defaults for all JOBs, however each JOB +# may define its own, overriding the defaults. + +# update_every sets the default data collection frequency. +# If unset, the python.d.plugin default is used. +# update_every: 1 + +# priority controls the order of charts at the netdata dashboard. +# Lower numbers move the charts towards the top of the page. +# If unset, the default for python.d.plugin is used. +# priority: 60000 + +# retries sets the number of retries to be made in case of failures. +# If unset, the default for python.d.plugin is used. +# Attempts to restore the service are made once every update_every +# and only if the module has collected values in the past. +# retries: 5 + +# ---------------------------------------------------------------------- +# JOBS (data collection sources) +# +# The default JOBS share the same *name*. JOBS with the same name +# are mutually exclusive. Only one of them will be allowed running at +# any time. This allows autodetection to try several alternatives and +# pick the one that works. +# +# Any number of jobs is supported. +# +# All python.d.plugin JOBS (for all its modules) support a set of +# predefined parameters. These are: +# +# job_name: +# name: myname # the JOB's name as it will appear at the +# # dashboard (by default is the job_name) +# # JOBs sharing a name are mutually exclusive +# update_every: 1 # the JOB's data collection frequency +# priority: 60000 # the JOB's order on the dashboard +# retries: 5 # the JOB's number of restoration attempts +# +# Additionally to the above, PHP-FPM also supports the following: +# +# url: 'URL' # the URL to fetch nginx's status stats +# +# if the URL is password protected, the following are supported: +# +# user: 'username' +# pass: 'password' +# + +# ---------------------------------------------------------------------- +# AUTO-DETECTION JOBS +# only one of them will run (they have the same name) + +localhost: + name : 'local' + url : "http://localhost/status" + +localipv4: + name : 'local' + url : "http://127.0.0.1/status" + +localipv6: + name : 'local' + url : "http://::1/status" + diff --git a/conf.d/python.d/postfix.conf b/conf.d/python.d/postfix.conf new file mode 100644 index 000000000..ca9d8fada --- /dev/null +++ b/conf.d/python.d/postfix.conf @@ -0,0 +1,67 @@ +# netdata python.d.plugin configuration for postfix +# +# This file is in YaML format. Generally the format is: +# +# name: value +# +# There are 2 sections: +# - global variables +# - one or more JOBS +# +# JOBS allow you to collect values from multiple sources. +# Each source will have its own set of charts. +# +# JOB parameters have to be indented (using spaces only, example below). + +# ---------------------------------------------------------------------- +# Global Variables +# These variables set the defaults for all JOBs, however each JOB +# may define its own, overriding the defaults. + +# update_every sets the default data collection frequency. +# If unset, the python.d.plugin default is used. +# postfix is slow, so once every 10 seconds +update_every: 10 + +# priority controls the order of charts at the netdata dashboard. +# Lower numbers move the charts towards the top of the page. +# If unset, the default for python.d.plugin is used. +# priority: 60000 + +# retries sets the number of retries to be made in case of failures. +# If unset, the default for python.d.plugin is used. +# Attempts to restore the service are made once every update_every +# and only if the module has collected values in the past. +# retries: 5 + +# ---------------------------------------------------------------------- +# JOBS (data collection sources) +# +# The default JOBS share the same *name*. JOBS with the same name +# are mutually exclusive. Only one of them will be allowed running at +# any time. This allows autodetection to try several alternatives and +# pick the one that works. +# +# Any number of jobs is supported. +# +# All python.d.plugin JOBS (for all its modules) support a set of +# predefined parameters. These are: +# +# job_name: +# name: myname # the JOB's name as it will appear at the +# # dashboard (by default is the job_name) +# # JOBs sharing a name are mutually exclusive +# update_every: 1 # the JOB's data collection frequency +# priority: 60000 # the JOB's order on the dashboard +# retries: 5 # the JOB's number of restoration attempts +# +# Additionally to the above, postfix also supports the following: +# +# command: 'postqueue -p' # the command to run +# + +# ---------------------------------------------------------------------- +# AUTO-DETECTION JOBS + +local: + command: 'postqueue -p' diff --git a/conf.d/python.d/redis.conf b/conf.d/python.d/redis.conf new file mode 100644 index 000000000..9935bff77 --- /dev/null +++ b/conf.d/python.d/redis.conf @@ -0,0 +1,97 @@ +# netdata python.d.plugin configuration for redis +# +# This file is in YaML format. Generally the format is: +# +# name: value +# +# There are 2 sections: +# - global variables +# - one or more JOBS +# +# JOBS allow you to collect values from multiple sources. +# Each source will have its own set of charts. +# +# JOB parameters have to be indented (using spaces only, example below). + +# ---------------------------------------------------------------------- +# Global Variables +# These variables set the defaults for all JOBs, however each JOB +# may define its own, overriding the defaults. + +# update_every sets the default data collection frequency. +# If unset, the python.d.plugin default is used. +# update_every: 1 + +# priority controls the order of charts at the netdata dashboard. +# Lower numbers move the charts towards the top of the page. +# If unset, the default for python.d.plugin is used. +# priority: 60000 + +# retries sets the number of retries to be made in case of failures. +# If unset, the default for python.d.plugin is used. +# Attempts to restore the service are made once every update_every +# and only if the module has collected values in the past. +# retries: 5 + +# ---------------------------------------------------------------------- +# JOBS (data collection sources) +# +# The default JOBS share the same *name*. JOBS with the same name +# are mutually exclusive. Only one of them will be allowed running at +# any time. This allows autodetection to try several alternatives and +# pick the one that works. +# +# Any number of jobs is supported. +# +# All python.d.plugin JOBS (for all its modules) support a set of +# predefined parameters. These are: +# +# job_name: +# name: myname # the JOB's name as it will appear at the +# # dashboard (by default is the job_name) +# # JOBs sharing a name are mutually exclusive +# update_every: 1 # the JOB's data collection frequency +# priority: 60000 # the JOB's order on the dashboard +# retries: 5 # the JOB's number of restoration attempts +# +# Additionally to the above, redis also supports the following: +# +# socket: 'path/to/mysql.sock' +# +# or +# host: 'IP or HOSTNAME' # the host to connect to +# port: PORT # the port to connect to +# +# + +# ---------------------------------------------------------------------- +# AUTO-DETECTION JOBS +# only one of them will run (they have the same name) + +socket1: + name : 'local' + socket : '/tmp/redis.sock' + +socket2: + name : 'local' + socket : '/var/run/redis/redis.sock' + +socket3: + name : 'local' + socket : '/var/lib/redis/redis.sock' + +localhost: + name : 'local' + host : 'localhost' + port : 6379 + +localipv4: + name : 'local' + host : '127.0.0.1' + port : 6379 + +localipv6: + name : 'local' + host : '::1' + port : 6379 + diff --git a/conf.d/python.d/sensors.conf b/conf.d/python.d/sensors.conf new file mode 100644 index 000000000..7d895c348 --- /dev/null +++ b/conf.d/python.d/sensors.conf @@ -0,0 +1,54 @@ +# netdata python.d.plugin configuration for sensors +# +# This file is in YaML format. Generally the format is: +# +# name: value +# + +# ---------------------------------------------------------------------- +# Global Variables +# These variables set the defaults for all JOBs, however each JOB +# may define its own, overriding the defaults. + +# update_every sets the default data collection frequency. +# If unset, the python.d.plugin default is used. +# update_every: 1 + +# priority controls the order of charts at the netdata dashboard. +# Lower numbers move the charts towards the top of the page. +# If unset, the default for python.d.plugin is used. +# priority: 60000 + +# retries sets the number of retries to be made in case of failures. +# If unset, the default for python.d.plugin is used. +# Attempts to restore the service are made once every update_every +# and only if the module has collected values in the past. +# retries: 5 + +# ---------------------------------------------------------------------- +# Limit the number of sensors types. +# Comment the ones you want to disable. +# Also, re-arranging this list controls the order of the charts at the +# netdata dashboard. + +types: + - temperature + - fan + - voltage + - current + - power + - energy + - humidity + +# ---------------------------------------------------------------------- +# Limit the number of sensors chips. +# Uncomment the first line (chips:) and add chip names below it. +# The chip names that start with like that will be matched. +# You can find the chip names using the sensors command. + +#chips: +# - i8k +# - coretemp +# +# chip names can be found using the sensors shell command +# the prefix is matched (anything that starts like that) diff --git a/conf.d/python.d/squid.conf b/conf.d/python.d/squid.conf new file mode 100644 index 000000000..27800bde7 --- /dev/null +++ b/conf.d/python.d/squid.conf @@ -0,0 +1,162 @@ +# netdata python.d.plugin configuration for squid +# +# This file is in YaML format. Generally the format is: +# +# name: value +# +# There are 2 sections: +# - global variables +# - one or more JOBS +# +# JOBS allow you to collect values from multiple sources. +# Each source will have its own set of charts. +# +# JOB parameters have to be indented (using spaces only, example below). + +# ---------------------------------------------------------------------- +# Global Variables +# These variables set the defaults for all JOBs, however each JOB +# may define its own, overriding the defaults. + +# update_every sets the default data collection frequency. +# If unset, the python.d.plugin default is used. +# update_every: 1 + +# priority controls the order of charts at the netdata dashboard. +# Lower numbers move the charts towards the top of the page. +# If unset, the default for python.d.plugin is used. +# priority: 60000 + +# retries sets the number of retries to be made in case of failures. +# If unset, the default for python.d.plugin is used. +# Attempts to restore the service are made once every update_every +# and only if the module has collected values in the past. +# retries: 5 + +# ---------------------------------------------------------------------- +# JOBS (data collection sources) +# +# The default JOBS share the same *name*. JOBS with the same name +# are mutually exclusive. Only one of them will be allowed running at +# any time. This allows autodetection to try several alternatives and +# pick the one that works. +# +# Any number of jobs is supported. +# +# All python.d.plugin JOBS (for all its modules) support a set of +# predefined parameters. These are: +# +# job_name: +# name: myname # the JOB's name as it will appear at the +# # dashboard (by default is the job_name) +# # JOBs sharing a name are mutually exclusive +# update_every: 1 # the JOB's data collection frequency +# priority: 60000 # the JOB's order on the dashboard +# retries: 5 # the JOB's number of restoration attempts +# +# Additionally to the above, squid also supports the following: +# +# host : 'IP or HOSTNAME' # the host to connect to +# port : PORT # the port to connect to +# request: 'URL' # the URL to request from squid +# + +# ---------------------------------------------------------------------- +# SQUID CONFIGURATION +# +# See: +# http://wiki.squid-cache.org/Features/CacheManager +# +# In short, add to your squid configuration these: +# +# http_access allow localhost manager +# http_access deny manager +# +# To remotely monitor a squid: +# +# acl managerAdmin src 192.0.2.1 +# http_access allow localhost manager +# http_access allow managerAdmin manager +# http_access deny manager +# + +# ---------------------------------------------------------------------- +# AUTO-DETECTION JOBS +# only one of them will run (they have the same name) + +tcp3128old: + name : 'local' + host : 'localhost' + port : 3128 + request : 'cache_object://localhost:3128/counters' + +tcp8080old: + name : 'local' + host : 'localhost' + port : 8080 + request : 'cache_object://localhost:3128/counters' + +tcp3128new: + name : 'local' + host : 'localhost' + port : 3128 + request : '/squid-internal-mgr/counters' + +tcp8080new: + name : 'local' + host : 'localhost' + port : 8080 + request : '/squid-internal-mgr/counters' + +# IPv4 + +tcp3128oldipv4: + name : 'local' + host : '127.0.0.1' + port : 3128 + request : 'cache_object://127.0.0.1:3128/counters' + +tcp8080oldipv4: + name : 'local' + host : '127.0.0.1' + port : 8080 + request : 'cache_object://127.0.0.1:3128/counters' + +tcp3128newipv4: + name : 'local' + host : '127.0.0.1' + port : 3128 + request : '/squid-internal-mgr/counters' + +tcp8080newipv4: + name : 'local' + host : '127.0.0.1' + port : 8080 + request : '/squid-internal-mgr/counters' + +# IPv6 + +tcp3128oldipv6: + name : 'local' + host : '::1' + port : 3128 + request : 'cache_object://[::1]:3128/counters' + +tcp8080oldipv6: + name : 'local' + host : '::1' + port : 8080 + request : 'cache_object://[::1]:3128/counters' + +tcp3128newipv6: + name : 'local' + host : '::1' + port : 3128 + request : '/squid-internal-mgr/counters' + +tcp8080newipv6: + name : 'local' + host : '::1' + port : 8080 + request : '/squid-internal-mgr/counters' + diff --git a/conf.d/python.d/tomcat.conf b/conf.d/python.d/tomcat.conf new file mode 100644 index 000000000..aef9631b9 --- /dev/null +++ b/conf.d/python.d/tomcat.conf @@ -0,0 +1,81 @@ +# netdata python.d.plugin configuration for tomcat +# +# This file is in YaML format. Generally the format is: +# +# name: value +# +# There are 2 sections: +# - global variables +# - one or more JOBS +# +# JOBS allow you to collect values from multiple sources. +# Each source will have its own set of charts. +# +# JOB parameters have to be indented (using spaces only, example below). + +# ---------------------------------------------------------------------- +# Global Variables +# These variables set the defaults for all JOBs, however each JOB +# may define its own, overriding the defaults. + +# update_every sets the default data collection frequency. +# If unset, the python.d.plugin default is used. +# update_every: 1 + +# priority controls the order of charts at the netdata dashboard. +# Lower numbers move the charts towards the top of the page. +# If unset, the default for python.d.plugin is used. +# priority: 60000 + +# retries sets the number of retries to be made in case of failures. +# If unset, the default for python.d.plugin is used. +# Attempts to restore the service are made once every update_every +# and only if the module has collected values in the past. +# retries: 5 + +# ---------------------------------------------------------------------- +# JOBS (data collection sources) +# +# The default JOBS share the same *name*. JOBS with the same name +# are mutually exclusive. Only one of them will be allowed running at +# any time. This allows autodetection to try several alternatives and +# pick the one that works. +# +# Any number of jobs is supported. +# +# All python.d.plugin JOBS (for all its modules) support a set of +# predefined parameters. These are: +# +# job_name: +# name: myname # the JOB's name as it will appear at the +# # dashboard (by default is the job_name) +# # JOBs sharing a name are mutually exclusive +# update_every: 1 # the JOB's data collection frequency +# priority: 60000 # the JOB's order on the dashboard +# retries: 5 # the JOB's number of restoration attempts +# +# Additionally to the above, tomcat also supports the following: +# +# url: 'URL' # the URL to fetch nginx's status stats +# +# if the URL is password protected, the following are supported: +# +# user: 'username' +# pass: 'password' +# + +# ---------------------------------------------------------------------- +# AUTO-DETECTION JOBS +# only one of them will run (they have the same name) + +localhost: + name : 'local' + url : 'http://localhost:8080/manager/status?XML=true' + +localipv4: + name : 'local' + url : 'http://127.0.0.1:8080/manager/status?XML=true' + +localipv6: + name : 'local' + url : 'http://[::1]:8080/manager/status?XML=true' diff --git a/config.h.in b/config.h.in index ce8cd7450..065ff0334 100644 --- a/config.h.in +++ b/config.h.in @@ -1,5 +1,8 @@ /* config.h.in. Generated from configure.ac by autoheader. */ +/* Define to 1 if you have the `accept4' function. */ +#undef HAVE_ACCEPT4 + /* Define to 1 if you have the <inttypes.h> header file. */ #undef HAVE_INTTYPES_H diff --git a/configs.signatures b/configs.signatures new file mode 100644 index 000000000..6a0194bf5 --- /dev/null +++ b/configs.signatures @@ -0,0 +1,152 @@ +declare -A configs_signatures=( + ['0056936ce99788ed9ae1c611c87aa6d8']='apps_groups.conf' + ['18ee1c6197a4381b1c1631ef6129824f']='apps_groups.conf' + ['2f4a85fedecce1bf425fa1039f6b021e']='apps_groups.conf' + ['3af522d65b50a5e447607ffb28c81ff5']='apps_groups.conf' + ['3b1bfa40a4ff6a200bb2fc00bc51a664']='apps_groups.conf' + ['4a448831776de8acf2e0bdc4cc994cb4']='apps_groups.conf' + ['5bf51bb24fb41db9b1e448bd060d3f8c']='apps_groups.conf' + ['636d032928ea0f4741eab264fb49c099']='apps_groups.conf' + ['647361e99b5f4e0d73470c569bb9461c']='apps_groups.conf' + ['6a47af861ad3dd112124c37fbf09672b']='apps_groups.conf' + ['79a37756869d9b4629285922572d6b9b']='apps_groups.conf' + ['99c1617448abbdc493976ab9bda5ce02']='apps_groups.conf' + ['9c0185ceff15415bc59b2ce2c1f04367']='apps_groups.conf' + ['a0ee8f351f213c0e8af9eb7a4a09cb95']='apps_groups.conf' + ['a7cceeafb1e6ef1ead503ab65f687902']='apps_groups.conf' + ['a837986be634fd7648bcdf939019424a']='apps_groups.conf' + ['a9cd91675467c5426f5b51c47602c889']='apps_groups.conf' + ['acaa6731a272f6d251afb357e99b518f']='apps_groups.conf' + ['bb51112d01ff20053196a57632df8962']='apps_groups.conf' + ['d9258e671d0d0b6498af1ce16ef030d2']='apps_groups.conf' + ['ebd0612ccc5807524ebb2b647e3e56c9']='apps_groups.conf' + ['f2f1b8656f5011e965ac45b818cf668d']='apps_groups.conf' + ['fdea185e0e52b459b48852aa37f20e0f']='apps_groups.conf' + ['4ccb06fff1ce06dc5bc80e0a9f568f6e']='charts.d.conf' + ['4e995acb0d6fd77403a2a9dca984b55b']='charts.d.conf' + ['535e5113b07b0fc6f3abd59546c276f6']='charts.d.conf' + ['7cf6402b51e5070f2be3ad6fe059ff89']='charts.d.conf' + ['a02d14124b19c635c1426cee2e98bac5']='charts.d.conf' + ['ca026d7c779f0a7cb7787713c5be5c47']='charts.d.conf' + ['3848172053221b95279ba9bf789cd4e0']='health.d/apache.conf' + ['842b1ad5b89bfa5f421d9c5b72e001a4']='health.d/apache.conf' + ['084ee72d64760f2641b0720e79c922f3']='health.d/cpu.conf' + ['623771eecb3c277fc728b5304793f93b']='health.d/cpu.conf' + ['7596ae54d46ce199ac599429ef753caf']='health.d/cpu.conf' + ['fdd11640ba626cc2064c2fe3ea3eee4c']='health.d/cpu.conf' + ['02fa10fa85ab88e9723998de48d1aca0']='health.d/disks.conf' + ['22ceb822983134a7ca67343241f30341']='health.d/disks.conf' + ['2385e5d35b440619621c4af62492d91b']='health.d/disks.conf' + ['3bc2776623889744a98178bad6fb3b79']='health.d/disks.conf' + ['573398335c0c71c075fa57f702bce287']='health.d/disks.conf' + ['5da15d6e17a15213a720749045e5d419']='health.d/disks.conf' + ['8989b5e2f4ef9cd278ef58be0fae4074']='health.d/disks.conf' + ['8dc0bd0a70b5117454bd5f5b98f91c2c']='health.d/disks.conf' + ['a8167dafeac0b66696a1d9b08e815cda']='health.d/disks.conf' + ['afdae4646c755ff2d117527fbf761c8e']='health.d/disks.conf' + ['cb60badf376d246ad8ec9d3f524db430']='health.d/disks.conf' + ['2d1d7498c72f4245cf32902c2b7e71e0']='health.d/entropy.conf' + ['450667c552ab7a7d8d4a2c214fdacca5']='health.d/entropy.conf' + ['a8feb36776005bf419c90278787a1be8']='health.d/entropy.conf' + ['d8dc489e32f7114c6298fce94e86a8ef']='health.d/entropy.conf' + ['e449e5582279742496550df14b6fca95']='health.d/entropy.conf' + ['45a77ac36ba9f1898144b902de17204b']='health.d/memcached.conf' + ['b81b8f331161b0d48e03f6fbf6b6d062']='health.d/memcached.conf' + ['c1a7e634b5b8aad523a0d115a93379cd']='health.d/memcached.conf' + ['f8c30f22df92765e2c0fab3c8174e2fc']='health.d/memcached.conf' + ['373c1276dc9e65884ff2b26e1f08afe7']='health.d/named.conf' + ['846ce94bfeeb90c0dc6a89e8d25f1a68']='health.d/named.conf' + ['ddda2bb1c88be03b637d3285406f7910']='health.d/named.conf' + ['2827de41cf34a91b7a8e4d8724f59668']='health.d/net.conf' + ['2ad55a5d1e885cf142849a78d4b00401']='health.d/net.conf' + ['43ebb7f224c3b232d8ad044d7e9508b6']='health.d/net.conf' + ['d11711b3647bc2bdd0292dd7deebbeb1']='health.d/net.conf' + ['de02f899a61f21b86adb646940f0bcae']='health.d/net.conf' + ['64c48f9726ab987baec9c617a9fef7a6']='health.d/nginx.conf' + ['7a985528cc9176564640001aa73e3492']='health.d/nginx.conf' + ['36fdd55665cf10b0db164c2a0cca5e57']='health.d/qos.conf' + ['80f109ff293ac94222bf3959432751bd']='health.d/qos.conf' + ['20be73f473e59bc7de1fe61d53466aba']='health.d/ram.conf' + ['b7d769ce86a7aebba01315da5c0799e6']='health.d/ram.conf' + ['325617412a628e3bc776e3fbb777a2a6']='health.d/redis.conf' + ['4f6a5b47a13f5912cc89e9286701dd08']='health.d/redis.conf' + ['23ae815aefa221b1929f96752a1f7556']='health.d/squid.conf' + ['a4a8660728c6afcb528cc6b378897d6b']='health.d/squid.conf' + ['ef9916ea144878a9f37cbb6b1b29da10']='health.d/squid.conf' + ['a44899a5795bed2863c1d11aa3e85586']='health.d/swap.conf' + ['c9b792755de59d842ba95f8c315d94c8']='health.d/swap.conf' + ['da29d2ab1ab7b8fda189960c840e5144']='health.d/swap.conf' + ['13141998a5d71308d9c119834c27bfd3']='python.d.conf' + ['38d1bf04fe9901481dd6febcc0404a86']='python.d.conf' + ['4b775fb31342f1478b3773d041a72911']='python.d.conf' + ['4fdf72784296326e0b46cb526a5d77a1']='python.d.conf' + ['7a21ccc76be2968ce5d0b52ec1166788']='python.d.conf' + ['99a3de85d1e7826ed64a5f8576712e5d']='python.d.conf' + ['9e0553ebdc21b64295873fc104cfa79d']='python.d.conf' + ['a2944a309f8ce1a3195451856478d6ae']='python.d.conf' + ['af44cc53aa2bc5cc8935667119567522']='python.d.conf' + ['b27f10a38a95edbbec20f44a4728b7c4']='python.d.conf' + ['b32164929eda7449a9677044e11151bf']='python.d.conf' + ['b8969be5b3ceb4a99477937119bd4323']='python.d.conf' + ['bba2f3886587f137ea08a6e63dd3d376']='python.d.conf' + ['d55be5bb5e108da1e7645da007c53cd4']='python.d.conf' + ['f82924563e41d99cdae5431f0af69155']='python.d.conf' + ['5278ebbae19c60db600f0a119cb3664e']='python.d/apache.conf' + ['5829812db29598db5857c9f433e96fef']='python.d/apache.conf' + ['6bf0de6e3b251b765b10a71d8c5c319d']='python.d/apache.conf' + ['6b917300747e7e8314844237e2462261']='python.d/apache_cache.conf' + ['e0e96cc47ed61d6492416be5236cd4d3']='python.d/apache_cache.conf' + ['7830066c46a7e5f9682b8d3f4566b4e5']='python.d/cpufreq.conf' + ['b5b5a8d6d991fb1cef8d80afa23ba114']='python.d/cpufreq.conf' + ['dc0d2b96378f290eec3fcf98b89ad824']='python.d/cpufreq.conf' + ['e40947d22f7ed5359f12fc89e3512963']='python.d/dovecot.conf' + ['a8bb4e1d0525f59692778ad8f675a77a']='python.d/example.conf' + ['ae5ac0a3521e50aa6f6eda2a330b4075']='python.d/example.conf' + ['e5f32f54d6d6728f21f9ac26f37d6573']='python.d/example.conf' + ['15e32114994b92be7853b88091e7c6fb']='python.d/exim.conf' + ['54614490a14e1a4b7b3d9fecb6b4cfa5']='python.d/exim.conf' + ['73125ae64d5c6e9361944cd9bd14844e']='python.d/exim.conf' + ['a94af1c808aafdf00537d85ff2197ec8']='python.d/exim.conf' + ['2a0794fd43eadf30a51805bc9ba2c64d']='python.d/hddtemp.conf' + ['3fc45cc18e884c22482524dff6d27833']='python.d/hddtemp.conf' + ['421d5dc6c2fce22d0816b6e6363bea57']='python.d/hddtemp.conf' + ['47180421d580baeaedf8c0ef3d647fb5']='python.d/hddtemp.conf' + ['731a1fcfe9b2da1b9d685056a59541b8']='python.d/hddtemp.conf' + ['d74dc63fbe631dab9a2ff1b0f5d71719']='python.d/hddtemp.conf' + ['0e59bc11d0a869ea0247c04c08c8d72e']='python.d/ipfs.conf' + ['70105b1744a8e13f49083d7f1981aea2']='python.d/ipfs.conf' + ['97f337eb96213f3ede05e522e3743a6c']='python.d/memcached.conf' + ['1ea8e8ef1fa8a3a0fcdfba236f4cb195']='python.d/mysql.conf' + ['5379cdc26d7725e2b0d688d785816cef']='python.d/mysql.conf' + ['7deb236ec68a512b9bdd18e6a51d76f7']='python.d/mysql.conf' + ['88f77865f75c9fb61c97d700bd4561ee']='python.d/mysql.conf' + ['b0f0a0ac415e4b1a82187b80d211e83b']='python.d/mysql.conf' + ['df381f3a7ca9fb2b4b43ae7cb7a4c492']='python.d/mysql.conf' + ['061c45b0e34170d357e47883166ecf40']='python.d/nginx.conf' + ['21924a6ab8008d16ffac340f226ebad9']='python.d/nginx.conf' + ['c61948101e0e6846679682794ee48c5b']='python.d/nginx.conf' + ['6cba40e32a7e98a98c31a209913839cc']='python.d/nginx_log.conf' + ['3ca696189911fb38a0319ddd71e9a395']='python.d/phpfpm.conf' + ['8c1d41e2c88aeca78bc319ed74c8748c']='python.d/phpfpm.conf' + ['b8b87574fd496a66ede884c5336493bd']='python.d/phpfpm.conf' + ['c88fb430f35b7d8f08775d84debffbd2']='python.d/phpfpm.conf' + ['d7e0bd12d4a60a761dcab3531a841711']='python.d/phpfpm.conf' + ['142a5b693d34b0308bb0b8aec71fad79']='python.d/postfix.conf' + ['ca249db7a0637d55abb938d969f9b486']='python.d/postfix.conf' + ['39571e9fad9b759200c5d5b2ee13feb4']='python.d/redis.conf' + ['777f4da70f461ef675bde07fb3644312']='python.d/redis.conf' + ['b915126262d08aa9da81de539a58a3fb']='python.d/redis.conf' + ['837480f77ba1a85677a36747fbc2cd2e']='python.d/sensors.conf' + ['cfecf298bdafaa7e0a3a263548e82132']='python.d/sensors.conf' + ['48195c5c8c0476a49b714b4c76bdb570']='python.d/squid.conf' + ['64070d856ab1b47a18ec871e49bbc13b']='python.d/squid.conf' + ['78bb08809dffcb62e9bc493840f9c039']='python.d/squid.conf' + ['78e0065738394f5bf15023f41d66ed4b']='python.d/squid.conf' + ['7d8bd884ec26cb35d16c4fc05f969799']='python.d/squid.conf' + ['91cf3b3d42cac969b8b3fd4f531ecfb3']='python.d/squid.conf' + ['ade389c1b6efe0cff47c33e662731f0a']='python.d/squid.conf' + ['b846ca1f99fa6a65303b58186f47d7a4']='python.d/squid.conf' + ['e3e5bc57335c489f01b8559f5c70e112']='python.d/squid.conf' + ['0388b873d0d7e47c19005b7241db77d8']='python.d/tomcat.conf' + ['f7a99e94231beda85c6254912d8d31c1']='python.d/tomcat.conf' +) @@ -1,6 +1,6 @@ #! /bin/sh # Guess values for system-dependent variables and create Makefiles. -# Generated by GNU Autoconf 2.69 for netdata 1.2.0. +# Generated by GNU Autoconf 2.69 for netdata 1.3.0. # # # Copyright (C) 1992-1996, 1998-2012 Free Software Foundation, Inc. @@ -577,8 +577,8 @@ MAKEFLAGS= # Identity of this package. PACKAGE_NAME='netdata' PACKAGE_TARNAME='netdata' -PACKAGE_VERSION='1.2.0' -PACKAGE_STRING='netdata 1.2.0' +PACKAGE_VERSION='1.3.0' +PACKAGE_STRING='netdata 1.3.0' PACKAGE_BUGREPORT='' PACKAGE_URL='' @@ -619,6 +619,7 @@ ac_includes_default="\ # include <unistd.h> #endif" +ac_func_list= ac_subst_vars='am__EXEEXT_FALSE am__EXEEXT_TRUE LTLIBOBJS @@ -635,9 +636,11 @@ webdir pluginsdir logdir configdir +pythondir nodedir chartsdir cachedir +registrydir varlibdir ZLIB_LIBS ZLIB_CFLAGS @@ -1329,7 +1332,7 @@ if test "$ac_init_help" = "long"; then # Omit some internal or obsolete options to make the list less imposing. # This message is too long to be a string in the A/UX 3.1 sh. cat <<_ACEOF -\`configure' configures netdata 1.2.0 to adapt to many kinds of systems. +\`configure' configures netdata 1.3.0 to adapt to many kinds of systems. Usage: $0 [OPTION]... [VAR=VALUE]... @@ -1399,7 +1402,7 @@ fi if test -n "$ac_init_help"; then case $ac_init_help in - short | recursive ) echo "Configuration of netdata 1.2.0:";; + short | recursive ) echo "Configuration of netdata 1.3.0:";; esac cat <<\_ACEOF @@ -1520,7 +1523,7 @@ fi test -n "$ac_init_help" && exit $ac_status if $ac_init_version; then cat <<\_ACEOF -netdata configure 1.2.0 +netdata configure 1.3.0 generated by GNU Autoconf 2.69 Copyright (C) 2012 Free Software Foundation, Inc. @@ -1815,6 +1818,73 @@ fi } # ac_fn_c_try_link +# ac_fn_c_check_func LINENO FUNC VAR +# ---------------------------------- +# Tests whether FUNC exists, setting the cache variable VAR accordingly +ac_fn_c_check_func () +{ + as_lineno=${as_lineno-"$1"} as_lineno_stack=as_lineno_stack=$as_lineno_stack + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $2" >&5 +$as_echo_n "checking for $2... " >&6; } +if eval \${$3+:} false; then : + $as_echo_n "(cached) " >&6 +else + cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ +/* Define $2 to an innocuous variant, in case <limits.h> declares $2. + For example, HP-UX 11i <limits.h> declares gettimeofday. */ +#define $2 innocuous_$2 + +/* System header to define __stub macros and hopefully few prototypes, + which can conflict with char $2 (); below. + Prefer <limits.h> to <assert.h> if __STDC__ is defined, since + <limits.h> exists even on freestanding compilers. */ + +#ifdef __STDC__ +# include <limits.h> +#else +# include <assert.h> +#endif + +#undef $2 + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char $2 (); +/* The GNU C library defines this for functions which it implements + to always fail with ENOSYS. Some functions are actually named + something starting with __ and the normal name is an alias. */ +#if defined __stub_$2 || defined __stub___$2 +choke me +#endif + +int +main () +{ +return $2 (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + eval "$3=yes" +else + eval "$3=no" +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +fi +eval ac_res=\$$3 + { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_res" >&5 +$as_echo "$ac_res" >&6; } + eval $as_lineno_stack; ${as_lineno_stack:+:} unset as_lineno + +} # ac_fn_c_check_func + # ac_fn_c_find_uintX_t LINENO BITS VAR # ------------------------------------ # Finds an unsigned integer type with width BITS, setting cache variable VAR @@ -1872,7 +1942,7 @@ cat >config.log <<_ACEOF This file contains any messages produced by compilers while running configure, to aid debugging if configure makes a mistake. -It was created by netdata $as_me 1.2.0, which was +It was created by netdata $as_me 1.3.0, which was generated by GNU Autoconf 2.69. Invocation command line was $ $0 $@ @@ -2152,6 +2222,7 @@ $as_echo "$as_me: creating cache $cache_file" >&6;} >$cache_file fi +as_fn_append ac_func_list " accept4" # Check that the precious variables saved in the cache have kept the same # value. ac_cache_corrupted=false @@ -2250,7 +2321,7 @@ $as_echo "$as_me: ***************** MAINTAINER MODE *****************" >&6;} PACKAGE_BUILT_DATE=$(date '+%d %b %Y') fi -PACKAGE_RPM_VERSION="1.2.0" +PACKAGE_RPM_VERSION="1.3.0" @@ -2773,7 +2844,7 @@ fi # Define the identity of the package. PACKAGE='netdata' - VERSION='1.2.0' + VERSION='1.3.0' cat >>confdefs.h <<_ACEOF @@ -4556,6 +4627,24 @@ $as_echo "$ac_cv_safe_to_define___extensions__" >&6; } + + + for ac_func in $ac_func_list +do : + as_ac_var=`$as_echo "ac_cv_func_$ac_func" | $as_tr_sh` +ac_fn_c_check_func "$LINENO" "$ac_func" "$as_ac_var" +if eval test \"x\$"$as_ac_var"\" = x"yes"; then : + cat >>confdefs.h <<_ACEOF +#define `$as_echo "HAVE_$ac_func" | $as_tr_cpp` 1 +_ACEOF + +fi +done + + + + + # Check whether --enable-plugin-nfacct was given. if test "${enable_plugin_nfacct+set}" = set; then : enableval=$enable_plugin_nfacct; @@ -5618,12 +5707,16 @@ _ACEOF varlibdir="\$(localstatedir)/lib/netdata" +registrydir="\$(localstatedir)/lib/netdata/registry" + cachedir="\$(localstatedir)/cache/netdata" chartsdir="\$(libexecdir)/netdata/charts.d" nodedir="\$(libexecdir)/netdata/node.d" +pythondir="\$(libexecdir)/netdata/python.d" + configdir="\$(sysconfdir)/netdata" logdir="\$(localstatedir)/log/netdata" @@ -5641,7 +5734,7 @@ pluginsdir="\$(libexecdir)/netdata/plugins.d" -ac_config_files="$ac_config_files Makefile charts.d/Makefile conf.d/Makefile netdata.spec node.d/Makefile plugins.d/Makefile src/Makefile system/Makefile web/Makefile contrib/Makefile" +ac_config_files="$ac_config_files Makefile charts.d/Makefile conf.d/Makefile netdata.spec python.d/Makefile node.d/Makefile plugins.d/Makefile src/Makefile system/Makefile web/Makefile contrib/Makefile" cat >confcache <<\_ACEOF # This file is a shell script that caches the results of configure @@ -6177,7 +6270,7 @@ cat >>$CONFIG_STATUS <<\_ACEOF || ac_write_fail=1 # report actual input values of CONFIG_FILES etc. instead of their # values after options handling. ac_log=" -This file was extended by netdata $as_me 1.2.0, which was +This file was extended by netdata $as_me 1.3.0, which was generated by GNU Autoconf 2.69. Invocation command line was CONFIG_FILES = $CONFIG_FILES @@ -6243,7 +6336,7 @@ _ACEOF cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1 ac_cs_config="`$as_echo "$ac_configure_args" | sed 's/^ //; s/[\\""\`\$]/\\\\&/g'`" ac_cs_version="\\ -netdata config.status 1.2.0 +netdata config.status 1.3.0 configured by $0, generated by GNU Autoconf 2.69, with options \\"\$ac_cs_config\\" @@ -6378,6 +6471,7 @@ do "charts.d/Makefile") CONFIG_FILES="$CONFIG_FILES charts.d/Makefile" ;; "conf.d/Makefile") CONFIG_FILES="$CONFIG_FILES conf.d/Makefile" ;; "netdata.spec") CONFIG_FILES="$CONFIG_FILES netdata.spec" ;; + "python.d/Makefile") CONFIG_FILES="$CONFIG_FILES python.d/Makefile" ;; "node.d/Makefile") CONFIG_FILES="$CONFIG_FILES node.d/Makefile" ;; "plugins.d/Makefile") CONFIG_FILES="$CONFIG_FILES plugins.d/Makefile" ;; "src/Makefile") CONFIG_FILES="$CONFIG_FILES src/Makefile" ;; diff --git a/configure.ac b/configure.ac index 3d85f6d1c..fd84f133a 100644 --- a/configure.ac +++ b/configure.ac @@ -4,7 +4,7 @@ AC_PREREQ(2.60) define([VERSION_MAJOR], [1]) -define([VERSION_MINOR], [2]) +define([VERSION_MINOR], [3]) define([VERSION_FIX], [0]) define([VERSION_NUMBER], VERSION_MAJOR[.]VERSION_MINOR[.]VERSION_FIX) define([VERSION_SUFFIX], []) @@ -34,6 +34,7 @@ AC_PROG_CC AC_PROG_INSTALL PKG_PROG_PKG_CONFIG AC_USE_SYSTEM_EXTENSIONS +AC_CHECK_FUNCS_ONCE(accept4) AC_ARG_ENABLE( [plugin-nfacct], @@ -149,9 +150,11 @@ fi AC_DEFINE_UNQUOTED([NETDATA_USER], ["${with_user}"], [use this user to drop privileged]) AC_SUBST([varlibdir], ["\$(localstatedir)/lib/netdata"]) +AC_SUBST([registrydir], ["\$(localstatedir)/lib/netdata/registry"]) AC_SUBST([cachedir], ["\$(localstatedir)/cache/netdata"]) AC_SUBST([chartsdir], ["\$(libexecdir)/netdata/charts.d"]) AC_SUBST([nodedir], ["\$(libexecdir)/netdata/node.d"]) +AC_SUBST([pythondir], ["\$(libexecdir)/netdata/python.d"]) AC_SUBST([configdir], ["\$(sysconfdir)/netdata"]) AC_SUBST([logdir], ["\$(localstatedir)/log/netdata"]) AC_SUBST([pluginsdir], ["\$(libexecdir)/netdata/plugins.d"]) @@ -171,6 +174,7 @@ AC_CONFIG_FILES([ charts.d/Makefile conf.d/Makefile netdata.spec + python.d/Makefile node.d/Makefile plugins.d/Makefile src/Makefile diff --git a/contrib/Makefile.am b/contrib/Makefile.am index 19e5df77f..b58b40e26 100644 --- a/contrib/Makefile.am +++ b/contrib/Makefile.am @@ -5,6 +5,7 @@ dist_noinst_DATA = \ debian/copyright \ debian/netdata.conf \ debian/source/format \ + debian/control.wheezy \ debian/compat \ debian/netdata.install \ debian/netdata.lintian-overrides \ diff --git a/contrib/Makefile.in b/contrib/Makefile.in index f2d5ffa25..7b14a18de 100644 --- a/contrib/Makefile.in +++ b/contrib/Makefile.in @@ -235,6 +235,8 @@ pluginsdir = @pluginsdir@ prefix = @prefix@ program_transform_name = @program_transform_name@ psdir = @psdir@ +pythondir = @pythondir@ +registrydir = @registrydir@ sbindir = @sbindir@ sharedstatedir = @sharedstatedir@ srcdir = @srcdir@ @@ -251,6 +253,7 @@ dist_noinst_DATA = \ debian/copyright \ debian/netdata.conf \ debian/source/format \ + debian/control.wheezy \ debian/compat \ debian/netdata.install \ debian/netdata.lintian-overrides \ diff --git a/contrib/README.md b/contrib/README.md index 4578989a4..d3643d753 100644 --- a/contrib/README.md +++ b/contrib/README.md @@ -35,8 +35,16 @@ updates first. * edit `contrib/debian/rules` and adjust the `dh` rule near the top to remove systemd (see comments in that file). -* edit `contrib/debian/control`: remove `dh-systemd` from the - Build-Depends list, and add `pkg-config` to it. +* rename `contrib/debian/control.wheezy` to `contrib/debian/control`. + +* uncomment `EXTRA_OPTS="-P /var/run/netdata.pid"` in + `contrib/debian/netdata.default` + +* edit `contrib/debian/netdata.init` and change `PIDFILE` to + `/var/run/netdata.pid` + +* uncomment `postrotate` in `system/netdata.logrotate.in` and change + `try-restart` to `restart` Then proceed as the main instructions above. @@ -46,4 +54,3 @@ The recommended way to upgrade netdata packages built from this source is to remove the current package from your system, then install the new package. Upgrading on wheezy is known to not work cleanly; Jessie may behave as expected. - diff --git a/contrib/debian/changelog b/contrib/debian/changelog index ed818d4df..f1dddc4ca 100644 --- a/contrib/debian/changelog +++ b/contrib/debian/changelog @@ -1,3 +1,3 @@ -netdata (1.2.0) UNRELEASED; urgency=medium +netdata (1.3.0) UNRELEASED; urgency=medium * Latest release - -- Netdata Team <> Mon, 16 May 2016 22:12:32 +0200 + -- Netdata Team <> Sat, 27 Aug 2016 23:38:15 +0200 diff --git a/contrib/debian/control b/contrib/debian/control index c0312c6c6..24c5f4c4b 100644 --- a/contrib/debian/control +++ b/contrib/debian/control @@ -3,7 +3,8 @@ Build-Depends: debhelper (>= 9), dh-autoreconf, dh-systemd (>= 1.5), dpkg-dev (>= 1.13.19), - zlib1g-dev + zlib1g-dev, + uuid-dev Section: net Priority: optional Maintainer: Costa Tsaousis <costa@tsaousis.gr> diff --git a/contrib/debian/control.wheezy b/contrib/debian/control.wheezy new file mode 100644 index 000000000..4103908a2 --- /dev/null +++ b/contrib/debian/control.wheezy @@ -0,0 +1,25 @@ +Source: netdata +Build-Depends: debhelper (>= 9), + dh-autoreconf, + pkg-config, + dpkg-dev (>= 1.13.19), + zlib1g-dev, + uuid-dev +Section: net +Priority: optional +Maintainer: Costa Tsaousis <costa@tsaousis.gr> +Standards-Version: 3.9.6 +Homepage: https://github.com/firehol/netdata/wiki + +Package: netdata +Architecture: any +Depends: adduser, + libcap2-bin (>= 1:2.0), + lsb-base (>= 3.1-23.2), + ${misc:Depends}, + ${shlibs:Depends} +Description: real-time charts for system monitoring + Netdata is a daemon that collects data in realtime (per second) + and presents a web site to view and analyze them. The presentation + is also real-time and full of interactive charts that precisely + render all collected values. diff --git a/contrib/debian/netdata.default b/contrib/debian/netdata.default index eb1a69037..9e7f8ae6e 100644 --- a/contrib/debian/netdata.default +++ b/contrib/debian/netdata.default @@ -1,4 +1,5 @@ # Extra arguments to pass to netdata # #EXTRA_OPTS="" - +#uncomment following line if you are building a wheezy-package +#EXTRA_OPTS="-P /var/run/netdata.pid" diff --git a/docker-build.sh b/docker-build.sh new file mode 100644 index 000000000..462276c32 --- /dev/null +++ b/docker-build.sh @@ -0,0 +1,44 @@ +#!/bin/bash + +# author : titpetric +# original: https://github.com/titpetric/netdata + +set -e +DEBIAN_FRONTEND=noninteractive + +# some mirrors have issues, i skipped httpredir in favor of an eu mirror + +echo "deb http://ftp.nl.debian.org/debian/ jessie main" > /etc/apt/sources.list +echo "deb http://security.debian.org/debian-security jessie/updates main" >> /etc/apt/sources.list + +# install dependencies for build + +apt-get -qq update +apt-get -y install zlib1g-dev uuid-dev libmnl-dev gcc make curl git autoconf autogen automake pkg-config netcat-openbsd jq +apt-get -y install autoconf-archive lm-sensors nodejs python python-mysqldb python-yaml + +# fetch netdata + +git clone https://github.com/firehol/netdata.git /netdata.git --depth=1 +cd /netdata.git + +# use the provided installer + +./netdata-installer.sh --dont-wait --dont-start-it + +# remove build dependencies + +cd / +rm -rf /netdata.git + +dpkg -P zlib1g-dev uuid-dev libmnl-dev gcc make git autoconf autogen automake pkg-config +apt-get -y autoremove +apt-get clean +rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* + + +# symlink access log and error log to stdout/stderr + +ln -sf /dev/stdout /var/log/netdata/access.log +ln -sf /dev/stdout /var/log/netdata/debug.log +ln -sf /dev/stderr /var/log/netdata/error.log diff --git a/netdata-9999.ebuild b/netdata-9999.ebuild deleted file mode 100644 index 7bcb90801..000000000 --- a/netdata-9999.ebuild +++ /dev/null @@ -1,116 +0,0 @@ -# Copyright 1999-2016 Gentoo Foundation -# Distributed under the terms of the GNU General Public License v2 -# $Id$ - -EAPI=6 - -inherit linux-info systemd user - -if [[ ${PV} == "9999" ]] ; then - EGIT_REPO_URI="git://github.com/firehol/${PN}.git" - inherit git-r3 autotools - SRC_URI="" - KEYWORDS="" -else - SRC_URI="http://firehol.org/download/${PN}/releases/v${PV}/${P}.tar.xz" - KEYWORDS="~amd64 ~x86" -fi - - -DESCRIPTION="Linux real time system monitoring, done right!" -HOMEPAGE="https://github.com/firehol/netdata http://netdata.firehol.org/" - -LICENSE="GPL-3+ MIT BSD" -SLOT="0" -IUSE="+compression nfacct nodejs" - -# most unconditional dependencies are for plugins.d/charts.d.plugin: -RDEPEND=" - >=app-shells/bash-4:0 - net-misc/curl - net-misc/wget - virtual/awk - compression? ( sys-libs/zlib ) - nfacct? ( - net-firewall/nfacct - net-libs/libmnl - ) - nodejs? ( - net-libs/nodejs - )" - -DEPEND="${RDEPEND} - virtual/pkgconfig" - -# check for Kernel-Samepage-Merging (CONFIG_KSM) -CONFIG_CHECK=" - ~KSM -" - -: ${NETDATA_USER:=netdata} -: ${NETDATA_GROUP:=netdata} - -pkg_setup() { - linux-info_pkg_setup - - enewgroup ${PN} - enewuser ${PN} -1 -1 / ${PN} -} - -src_prepare() { - default - [[ ${PV} == "9999" ]] && eautoreconf -} - -src_configure() { - econf \ - --localstatedir=/var \ - --with-user=${NETDATA_USER} \ - $(use_enable nfacct plugin-nfacct) \ - $(use_with compression zlib) -} - -src_install() { - default - - fowners ${NETDATA_USER}:${NETDATA_GROUP} /var/log/netdata - fowners ${NETDATA_USER}:${NETDATA_GROUP} /var/cache/netdata - - chown -Rc ${NETDATA_USER}:${NETDATA_GROUP} "${ED}"/usr/share/${PN} || die - - cat >> "${T}"/${PN}-sysctl <<- EOF - kernel.mm.ksm.run = 1 - kernel.mm.ksm.sleep_millisecs = 1000 - EOF - - dodoc "${T}"/${PN}-sysctl - - newinitd system/netdata-openrc ${PN} - systemd_dounit system/netdata.service -} - -pkg_postinst() { - if [[ -e "/sys/kernel/mm/ksm/run" ]]; then - elog "INFORMATION:" - echo "" - elog "I see you have kernel memory de-duper (called Kernel Same-page Merging," - elog "or KSM) available, but it is not currently enabled." - echo "" - elog "To enable it run:" - echo "" - elog "echo 1 >/sys/kernel/mm/ksm/run" - elog "echo 1000 >/sys/kernel/mm/ksm/sleep_millisecs" - echo "" - elog "If you enable it, you will save 20-60% of netdata memory." - else - elog "INFORMATION:" - echo "" - elog "I see you do not have kernel memory de-duper (called Kernel Same-page" - elog "Merging, or KSM) available." - echo "" - elog "To enable it, you need a kernel built with CONFIG_KSM=y" - echo "" - elog "If you can have it, you will save 20-60% of netdata memory." - fi - -} diff --git a/netdata-installer.sh b/netdata-installer.sh index 42ab0e156..d6e78681b 100755 --- a/netdata-installer.sh +++ b/netdata-installer.sh @@ -1,8 +1,10 @@ -#!/bin/bash +#!/usr/bin/env bash # reload the user profile [ -f /etc/profile ] && . /etc/profile +export PATH="${PATH}:/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin" + # fix PKG_CHECK_MODULES error if [ -d /usr/share/aclocal ] then @@ -23,6 +25,23 @@ printf "CFLAGS=\"%s\" " "${CFLAGS}" >>netdata-installer.log printf "%q " "$0" "${@}" >>netdata-installer.log printf "\n" >>netdata-installer.log +service="$(which service 2>/dev/null || command -v service 2>/dev/null)" +systemctl="$(which systemctl 2>/dev/null || command -v systemctl 2>/dev/null)" +service() { + local cmd="${1}" action="${2}" + + if [ ! -z "${service}" ] + then + run "${service}" "${cmd}" "${action}" + return $? + elif [ ! -z "${systemctl}" ] + then + run "${systemctl}" "${action}" "${cmd}" + return $? + fi + return 1 +} + ME="$0" DONOTSTART=0 DONOTWAIT=0 @@ -30,378 +49,510 @@ NETDATA_PREFIX= LIBS_ARE_HERE=0 usage() { - cat <<-USAGE + cat <<USAGE - ${ME} <installer options> +${ME} <installer options> - Valid <installer options> are: +Valid <installer options> are: - --install /PATH/TO/INSTALL + --install /PATH/TO/INSTALL - If your give: --install /opt - netdata will be installed in /opt/netdata + If your give: --install /opt + netdata will be installed in /opt/netdata - --dont-start-it + --dont-start-it - Do not (re)start netdata. - Just install it. + Do not (re)start netdata. + Just install it. - --dont-wait + --dont-wait - Do not wait for the user to press ENTER. - Start immediately building it. + Do not wait for the user to press ENTER. + Start immediately building it. - --zlib-is-really-here - --libs-are-really-here + --zlib-is-really-here + --libs-are-really-here - If you get errors about missing zlib, - or libuuid but you know it is available, - you have a broken pkg-config. - Use this option to allow it continue - without checking pkg-config. + If you get errors about missing zlib, + or libuuid but you know it is available, + you have a broken pkg-config. + Use this option to allow it continue + without checking pkg-config. - Netdata will by default be compiled with gcc optimization -O3 - If you need to pass different CFLAGS, use something like this: +Netdata will by default be compiled with gcc optimization -O3 +If you need to pass different CFLAGS, use something like this: - CFLAGS="<gcc options>" ${ME} <installer options> + CFLAGS="<gcc options>" ${ME} <installer options> - For the installer to complete successfully, you will need - these packages installed: +For the installer to complete successfully, you will need +these packages installed: - gcc make autoconf automake pkg-config zlib1g-dev (or zlib-devel) - uuid-dev (or libuuid-devel) + gcc make autoconf automake pkg-config zlib1g-dev (or zlib-devel) + uuid-dev (or libuuid-devel) - For the plugins, you will at least need: +For the plugins, you will at least need: - curl nodejs + curl nodejs USAGE } +md5sum="$(which md5sum 2>/dev/null || command -v md5sum 2>/dev/null)" +get_git_config_signatures() { + local x s file md5 + + [ ! -d "conf.d" ] && echo >&2 "Wrong directory." && return 1 + [ -z "${md5sum}" -o ! -x "${md5sum}" ] && echo >&2 "No md5sum command." && return 1 + + echo >configs.signatures.tmp + + for x in $(find conf.d -name \*.conf) + do + x="${x/conf.d\//}" + echo "${x}" + for c in $(git log --follow "conf.d/${x}" | grep ^commit | cut -d ' ' -f 2) + do + git checkout ${c} "conf.d/${x}" || continue + s="$(cat "conf.d/${x}" | md5sum | cut -d ' ' -f 1)" + echo >>configs.signatures.tmp "${x}:${s}" + echo " ${s}" + done + git checkout HEAD "conf.d/${x}" || break + done + + cat configs.signatures.tmp |\ + grep -v "^$" |\ + sort -u |\ + { + echo "declare -A configs_signatures=(" + IFS=":" + while read file md5 + do + echo " ['${md5}']='${file}'" + done + echo ")" + } >configs.signatures + + rm configs.signatures.tmp + + return 0 +} + + while [ ! -z "${1}" ] do - if [ "$1" = "--install" ] - then - NETDATA_PREFIX="${2}/netdata" - shift 2 - elif [ "$1" = "--zlib-is-really-here" -o "$1" = "--libs-are-really-here" ] - then - LIBS_ARE_HERE=1 - shift 1 - elif [ "$1" = "--dont-start-it" ] - then - DONOTSTART=1 - shift 1 - elif [ "$1" = "--dont-wait" ] - then - DONOTWAIT=1 - shift 1 - elif [ "$1" = "--help" -o "$1" = "-h" ] - then - usage - exit 1 - else - echo >&2 - echo >&2 "ERROR:" - echo >&2 "I cannot understand option '$1'." - usage - exit 1 - fi + if [ "$1" = "--install" ] + then + NETDATA_PREFIX="${2}/netdata" + shift 2 + elif [ "$1" = "--zlib-is-really-here" -o "$1" = "--libs-are-really-here" ] + then + LIBS_ARE_HERE=1 + shift 1 + elif [ "$1" = "--dont-start-it" ] + then + DONOTSTART=1 + shift 1 + elif [ "$1" = "--dont-wait" ] + then + DONOTWAIT=1 + shift 1 + elif [ "$1" = "--help" -o "$1" = "-h" ] + then + usage + exit 1 + elif [ "$1" = "get_git_config_signatures" ] + then + get_git_config_signatures && exit 0 + exit 1 + else + echo >&2 + echo >&2 "ERROR:" + echo >&2 "I cannot understand option '$1'." + usage + exit 1 + fi done -cat <<-BANNER +cat <<BANNER - Welcome to netdata! - Nice to see you are giving it a try! +Welcome to netdata! +The real-time performance monitoring system. - You are about to build and install netdata to your system. +You are about to build and install netdata to your system. - It will be installed at these locations: +It will be installed at these locations: - - the daemon at ${NETDATA_PREFIX}/usr/sbin/netdata - - config files at ${NETDATA_PREFIX}/etc/netdata - - web files at ${NETDATA_PREFIX}/usr/share/netdata - - plugins at ${NETDATA_PREFIX}/usr/libexec/netdata - - cache files at ${NETDATA_PREFIX}/var/cache/netdata - - db files at ${NETDATA_PREFIX}/var/lib/netdata - - log files at ${NETDATA_PREFIX}/var/log/netdata - - pid file at ${NETDATA_PREFIX}/var/run + - the daemon at ${NETDATA_PREFIX}/usr/sbin/netdata + - config files at ${NETDATA_PREFIX}/etc/netdata + - web files at ${NETDATA_PREFIX}/usr/share/netdata + - plugins at ${NETDATA_PREFIX}/usr/libexec/netdata + - cache files at ${NETDATA_PREFIX}/var/cache/netdata + - db files at ${NETDATA_PREFIX}/var/lib/netdata + - log files at ${NETDATA_PREFIX}/var/log/netdata + - pid file at ${NETDATA_PREFIX}/var/run - This installer allows you to change the installation path. - Press Control-C and run the same command with --help for help. +This installer allows you to change the installation path. +Press Control-C and run the same command with --help for help. BANNER if [ "${UID}" -ne 0 ] - then - if [ -z "${NETDATA_PREFIX}" ] - then - cat <<-NONROOTNOPREFIX + then + if [ -z "${NETDATA_PREFIX}" ] + then + cat <<NONROOTNOPREFIX - Sorry! This will fail! +Sorry! This will fail! - You are attempting to install netdata as non-root, but you plan to install it - in system paths. +You are attempting to install netdata as non-root, but you plan to install it +in system paths. - Please set an installation prefix, like this: +Please set an installation prefix, like this: - $0 ${@} --install /tmp + $0 ${@} --install /tmp - or, run the installer as root: +or, run the installer as root: - sudo $0 ${@} + sudo $0 ${@} - We suggest to install it as root, or certain data collectors will not be able - to work. Netdata drops root privileges when running. So, if you plan to keep - it, install it as root to get the full functionality. +We suggest to install it as root, or certain data collectors will not be able +to work. Netdata drops root privileges when running. So, if you plan to keep +it, install it as root to get the full functionality. NONROOTNOPREFIX - exit 1 + exit 1 - else - cat <<-NONROOT + else + cat <<NONROOT - IMPORTANT: - You are about to install netdata as a non-root user. - Netdata will work, but a few data collection modules that - require root access will fail. +IMPORTANT: +You are about to install netdata as a non-root user. +Netdata will work, but a few data collection modules that +require root access will fail. - If you installing permanently on your system, run the - installer like this: +If you installing permanently on your system, run the +installer like this: - sudo $0 ${@} + sudo $0 ${@} NONROOT - fi + fi fi have_autotools= if [ "$(type autoreconf 2> /dev/null)" ] then - autoconf_maj_min() { - local maj min IFS=.- - - maj=$1 - min=$2 - - set -- $(autoreconf -V | sed -ne '1s/.* \([^ ]*\)$/\1/p') - eval $maj=\$1 $min=\$2 - } - autoconf_maj_min AMAJ AMIN - - if [ "$AMAJ" -gt 2 ] - then - have_autotools=Y - elif [ "$AMAJ" -eq 2 -a "$AMIN" -ge 60 ] - then - have_autotools=Y - else - echo "Found autotools $AMAJ.$AMIN" - fi + autoconf_maj_min() { + local maj min IFS=.- + + maj=$1 + min=$2 + + set -- $(autoreconf -V | sed -ne '1s/.* \([^ ]*\)$/\1/p') + eval $maj=\$1 $min=\$2 + } + autoconf_maj_min AMAJ AMIN + + if [ "$AMAJ" -gt 2 ] + then + have_autotools=Y + elif [ "$AMAJ" -eq 2 -a "$AMIN" -ge 60 ] + then + have_autotools=Y + else + echo "Found autotools $AMAJ.$AMIN" + fi else - echo "No autotools found" + echo "No autotools found" fi if [ ! "$have_autotools" ] then - if [ -f configure ] - then - echo "Will skip autoreconf step" - else - cat <<-"EOF" + if [ -f configure ] + then + echo "Will skip autoreconf step" + else + cat <<"EOF" - ------------------------------------------------------------------------------- - autotools 2.60 or later is required +------------------------------------------------------------------------------- +autotools 2.60 or later is required - Sorry, you do not seem to have autotools 2.60 or later, which is - required to build from the git sources of netdata. +Sorry, you do not seem to have autotools 2.60 or later, which is +required to build from the git sources of netdata. - You can either install a suitable version of autotools and automake - or download a netdata package which does not have these dependencies. +You can either install a suitable version of autotools and automake +or download a netdata package which does not have these dependencies. - Source packages where autotools have already been run are available - here: - https://firehol.org/download/netdata/ +Source packages where autotools have already been run are available +here: + https://firehol.org/download/netdata/ - The unsigned/master folder tracks the head of the git tree and released - packages are also available. +The unsigned/master folder tracks the head of the git tree and released +packages are also available. EOF - exit 1 - fi + exit 1 + fi fi if [ ${DONOTWAIT} -eq 0 ] - then - if [ ! -z "${NETDATA_PREFIX}" ] - then - read -p "Press ENTER to build and install netdata to '${NETDATA_PREFIX}' > " - else - read -p "Press ENTER to build and install netdata to your system > " - fi + then + if [ ! -z "${NETDATA_PREFIX}" ] + then + read -p "Press ENTER to build and install netdata to '${NETDATA_PREFIX}' > " + else + read -p "Press ENTER to build and install netdata to your system > " + fi fi build_error() { - cat <<-EOF + cat <<EOF - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - Sorry! NetData failed to build... +Sorry! NetData failed to build... - You many need to check these: +You many need to check these: - 1. The package uuid-dev (or libuuid-devel) has to be installed. +1. The package uuid-dev (or libuuid-devel) has to be installed. - If your system cannot find libuuid, although it is installed - run me with the option: --libs-are-really-here + If your system cannot find libuuid, although it is installed + run me with the option: --libs-are-really-here - 2. The package zlib1g-dev (or zlib-devel) has to be installed. +2. The package zlib1g-dev (or zlib-devel) has to be installed. - If your system cannot find zlib, although it is installed - run me with the option: --libs-are-really-here + If your system cannot find zlib, although it is installed + run me with the option: --libs-are-really-here - 3. You need basic build tools installed, like: +3. You need basic build tools installed, like: - gcc make autoconf automake pkg-config + gcc make autoconf automake pkg-config - Autoconf version 2.60 or higher is required. + Autoconf version 2.60 or higher is required. - If you still cannot get it to build, ask for help at github: +If you still cannot get it to build, ask for help at github: - https://github.com/firehol/netdata/issues + https://github.com/firehol/netdata/issues EOF - trap - EXIT - exit 1 + trap - EXIT + exit 1 } run() { - printf >>netdata-installer.log "# " - printf >>netdata-installer.log "%q " "${@}" - printf >>netdata-installer.log " ... " - - printf >&2 "\n" - printf >&2 ":-----------------------------------------------------------------------------\n" - printf >&2 "Running command:\n" - printf >&2 "\n" - printf >&2 "%q " "${@}" - printf >&2 "\n" - - "${@}" - - local ret=$? - if [ ${ret} -ne 0 ] - then - printf >>netdata-installer.log "FAILED!\n" - else - printf >>netdata-installer.log "OK\n" - fi - - return ${ret} + printf >>netdata-installer.log "# " + printf >>netdata-installer.log "%q " "${@}" + printf >>netdata-installer.log " ... " + + printf >&2 "\n" + printf >&2 ":-----------------------------------------------------------------------------\n" + printf >&2 "Running command:\n" + printf >&2 "\n" + printf >&2 "%q " "${@}" + printf >&2 "\n" + + "${@}" + + local ret=$? + if [ ${ret} -ne 0 ] + then + printf >>netdata-installer.log "FAILED!\n" + else + printf >>netdata-installer.log "OK\n" + fi + + return ${ret} } if [ ${LIBS_ARE_HERE} -eq 1 ] - then - shift - echo >&2 "ok, assuming libs are really installed." - export ZLIB_CFLAGS=" " - export ZLIB_LIBS="-lz" - export UUID_CFLAGS=" " - export UUID_LIBS="-luuid" + then + shift + echo >&2 "ok, assuming libs are really installed." + export ZLIB_CFLAGS=" " + export ZLIB_LIBS="-lz" + export UUID_CFLAGS=" " + export UUID_LIBS="-luuid" fi trap build_error EXIT if [ "$have_autotools" ] then - run ./autogen.sh || exit 1 + run ./autogen.sh || exit 1 fi run ./configure \ - --prefix="${NETDATA_PREFIX}/usr" \ - --sysconfdir="${NETDATA_PREFIX}/etc" \ - --localstatedir="${NETDATA_PREFIX}/var" \ - --with-zlib --with-math --with-user=netdata \ - CFLAGS="${CFLAGS}" || exit 1 + --prefix="${NETDATA_PREFIX}/usr" \ + --sysconfdir="${NETDATA_PREFIX}/etc" \ + --localstatedir="${NETDATA_PREFIX}/var" \ + --with-zlib --with-math --with-user=netdata \ + CFLAGS="${CFLAGS}" || exit 1 # remove the build_error hook trap - EXIT if [ -f src/netdata ] - then - echo >&2 "Cleaning a possibly old compilation ..." - run make clean + then + echo >&2 "Cleaning a possibly old compilation ..." + run make clean fi echo >&2 "Compiling netdata ..." run make || exit 1 +if [ "${BASH_VERSINFO[0]}" -ge "4" ] +then + declare -A configs_signatures=() + if [ -f "configs.signatures" ] + then + source "configs.signatures" || echo >&2 "ERROR: Failed to load configs.signatures !" + fi +fi + +# migrate existing configuration files +# for node.d and charts.d +if [ -d "${NETDATA_PREFIX}/etc/netdata" ] + then + # the configuration directory exists + + if [ ! -d "${NETDATA_PREFIX}/etc/netdata/charts.d" ] + then + run mkdir "${NETDATA_PREFIX}/etc/netdata/charts.d" + fi + + # move the charts.d config files + for x in apache ap cpu_apps cpufreq example exim hddtemp load_average mem_apps mysql nginx nut opensips phpfpm postfix sensors squid tomcat + do + for y in "" ".old" ".orig" + do + if [ -f "${NETDATA_PREFIX}/etc/netdata/${x}.conf${y}" -a ! -f "${NETDATA_PREFIX}/etc/netdata/charts.d/${x}.conf${y}" ] + then + run mv -f "${NETDATA_PREFIX}/etc/netdata/${x}.conf${y}" "${NETDATA_PREFIX}/etc/netdata/charts.d/${x}.conf${y}" + fi + done + done + + if [ ! -d "${NETDATA_PREFIX}/etc/netdata/node.d" ] + then + run mkdir "${NETDATA_PREFIX}/etc/netdata/node.d" + fi + + # move the node.d config files + for x in named sma_webbox snmp + do + for y in "" ".old" ".orig" + do + if [ -f "${NETDATA_PREFIX}/etc/netdata/${x}.conf${y}" -a ! -f "${NETDATA_PREFIX}/etc/netdata/node.d/${x}.conf${y}" ] + then + run mv -f "${NETDATA_PREFIX}/etc/netdata/${x}.conf${y}" "${NETDATA_PREFIX}/etc/netdata/node.d/${x}.conf${y}" + fi + done + done +fi + # backup user configurations installer_backup_suffix="${PID}.${RANDOM}" -for x in apps_groups.conf charts.d.conf +for x in $(find "${NETDATA_PREFIX}/etc/netdata/" -name '*.conf' -type f) do - if [ -f "${NETDATA_PREFIX}/etc/netdata/${x}" ] - then - cp -p "${NETDATA_PREFIX}/etc/netdata/${x}" "${NETDATA_PREFIX}/etc/netdata/${x}.installer_backup.${installer_backup_suffix}" - - elif [ -f "${NETDATA_PREFIX}/etc/netdata/${x}.installer_backup.${installer_backup_suffix}" ] - then - rm -f "${NETDATA_PREFIX}/etc/netdata/${x}.installer_backup.${installer_backup_suffix}" - fi + if [ -f "${x}" ] + then + # make a backup of the configuration file + cp -p "${x}" "${x}.old" + + if [ -z "${md5sum}" -o ! -x "${md5sum}" ] + then + # we don't have md5sum - keep it + cp -p "${x}" "${x}.installer_backup.${installer_backup_suffix}" + else + # find it relative filename + f="${x/*\/etc\/netdata\//}" + + # find its checksum + md5="$(cat "${x}" | ${md5sum} | cut -d ' ' -f 1)" + + # copy the original + if [ -f "conf.d/${f}" ] + then + cp "conf.d/${f}" "${x}.orig" + fi + + if [ "${BASH_VERSINFO[0]}" -ge "4" ] + then + if [ "${configs_signatures[${md5}]}" = "${f}" ] + then + # it is a stock version - don't keep it + echo >&2 "File '${x}' is stock version." + else + # edited by user - keep it + echo >&2 "File '${x}' has been edited by user." + cp -p "${x}" "${x}.installer_backup.${installer_backup_suffix}" + fi + else + echo >&2 "File '${x}' cannot be check for custom edits." + cp -p "${x}" "${x}.installer_backup.${installer_backup_suffix}" + fi + fi + + elif [ -f "${x}.installer_backup.${installer_backup_suffix}" ] + then + rm -f "${x}.installer_backup.${installer_backup_suffix}" + fi done echo >&2 "Installing netdata ..." run make install || exit 1 # restore user configurations -for x in apps_groups.conf charts.d.conf +for x in $(find "${NETDATA_PREFIX}/etc/netdata/" -name '*.conf' -type f) do - if [ -f "${NETDATA_PREFIX}/etc/netdata/${x}.installer_backup.${installer_backup_suffix}" ] - then - cp -p "${NETDATA_PREFIX}/etc/netdata/${x}.installer_backup.${installer_backup_suffix}" "${NETDATA_PREFIX}/etc/netdata/${x}" - fi + if [ -f "${x}.installer_backup.${installer_backup_suffix}" ] + then + cp -p "${x}.installer_backup.${installer_backup_suffix}" "${x}" + rm -f "${x}.installer_backup.${installer_backup_suffix}" + fi done NETDATA_ADDED_TO_DOCKER=0 if [ ${UID} -eq 0 ] - then - getent group netdata > /dev/null - if [ $? -ne 0 ] - then - echo >&2 "Adding netdata user group ..." - run groupadd -r netdata - fi - - getent passwd netdata > /dev/null - if [ $? -ne 0 ] - then - echo >&2 "Adding netdata user account ..." - run useradd -r -g netdata -c netdata -s /sbin/nologin -d / netdata - fi - - getent group docker > /dev/null - if [ $? -eq 0 ] - then - # find the users in the docker group - docker=$(getent group docker | cut -d ':' -f 4) - if [[ ",${docker}," =~ ,netdata, ]] - then - # netdata is already there - : - else - # netdata is not in docker group - echo >&2 "Adding netdata user to the docker group (needed to get container names) ..." - run usermod -a -G docker netdata - fi - # let the uninstall script know - NETDATA_ADDED_TO_DOCKER=1 - fi - - if [ -d /etc/logrotate.d -a ! -f /etc/logrotate.d/netdata ] - then - echo >&2 "Adding netdata logrotate configuration ..." - run cp system/netdata.logrotate /etc/logrotate.d/netdata - fi + then + getent group netdata > /dev/null + if [ $? -ne 0 ] + then + echo >&2 "Adding netdata user group ..." + run groupadd -r netdata + fi + + getent passwd netdata > /dev/null + if [ $? -ne 0 ] + then + echo >&2 "Adding netdata user account ..." + run useradd -r -g netdata -c netdata -s $(which nologin || echo '/bin/false') -d / netdata + fi + + getent group docker > /dev/null + if [ $? -eq 0 ] + then + # find the users in the docker group + docker=$(getent group docker | cut -d ':' -f 4) + if [[ ",${docker}," =~ ,netdata, ]] + then + # netdata is already there + : + else + # netdata is not in docker group + echo >&2 "Adding netdata user to the docker group (needed to get container names) ..." + run usermod -a -G docker netdata + fi + # let the uninstall script know + NETDATA_ADDED_TO_DOCKER=1 + fi + + if [ -d /etc/logrotate.d -a ! -f /etc/logrotate.d/netdata ] + then + echo >&2 "Adding netdata logrotate configuration ..." + run cp system/netdata.logrotate /etc/logrotate.d/netdata + fi fi @@ -413,15 +564,15 @@ fi # function to extract values from the config file config_option() { - local key="${1}" value="${2}" line= + local key="${1}" value="${2}" line= - if [ -s "${NETDATA_PREFIX}/etc/netdata/netdata.conf" ] - then - line="$( grep "^[[:space:]]*${key}[[:space:]]*=[[:space:]]*" "${NETDATA_PREFIX}/etc/netdata/netdata.conf" | head -n 1 )" - [ ! -z "${line}" ] && value="$( echo "${line}" | cut -d '=' -f 2 | sed -e "s/^[[:space:]]\+//g" -e "s/[[:space:]]\+$//g" )" - fi + if [ -s "${NETDATA_PREFIX}/etc/netdata/netdata.conf" ] + then + line="$( grep "^[[:space:]]*${key}[[:space:]]*=[[:space:]]*" "${NETDATA_PREFIX}/etc/netdata/netdata.conf" | head -n 1 )" + [ ! -z "${line}" ] && value="$( echo "${line}" | cut -d '=' -f 2 | sed -e "s/^[[:space:]]\+//g" -e "s/[[:space:]]\+$//g" )" + fi - echo "${value}" + echo "${value}" } # user @@ -438,7 +589,16 @@ NETDATA_DEBUG="$( config_option "debug flags" ${defdebug} )" # port defport=19999 -NETDATA_PORT="$( config_option "port" ${defport} )" +NETDATA_PORT="$( config_option "default port" ${defport} )" +NETDATA_PORT2="$( config_option "port" ${defport} )" + +if [ "${NETDATA_PORT}" != "${NETDATA_PORT2}" ] +then + if [ "${NETDATA_PORT2}" != "${defport}" ] + then + NETDATA_PORT="${NETDATA_PORT2}" + fi +fi # directories NETDATA_LIB_DIR="$( config_option "lib directory" "${NETDATA_PREFIX}/var/lib/netdata" )" @@ -446,7 +606,6 @@ NETDATA_CACHE_DIR="$( config_option "cache directory" "${NETDATA_PREFIX}/var/cac NETDATA_WEB_DIR="$( config_option "web files directory" "${NETDATA_PREFIX}/usr/share/netdata/web" )" NETDATA_LOG_DIR="$( config_option "log directory" "${NETDATA_PREFIX}/var/log/netdata" )" NETDATA_CONF_DIR="$( config_option "config directory" "${NETDATA_PREFIX}/etc/netdata" )" -NETDATA_BIND="$( config_option "bind socket to IP" "*" )" NETDATA_RUN_DIR="${NETDATA_PREFIX}/var/run" @@ -455,240 +614,304 @@ NETDATA_RUN_DIR="${NETDATA_PREFIX}/var/run" # this is needed if NETDATA_PREFIX is not empty if [ ! -d "${NETDATA_RUN_DIR}" ] - then - mkdir -p "${NETDATA_RUN_DIR}" || exit 1 + then + mkdir -p "${NETDATA_RUN_DIR}" || exit 1 fi echo >&2 echo >&2 "Fixing directories (user: ${NETDATA_USER})..." -for x in "${NETDATA_WEB_DIR}" "${NETDATA_CONF_DIR}" "${NETDATA_CACHE_DIR}" "${NETDATA_LOG_DIR}" "${NETDATA_LIB_DIR}" +for x in "${NETDATA_WEB_DIR}" "${NETDATA_CONF_DIR}" "${NETDATA_CACHE_DIR}" "${NETDATA_LOG_DIR}" "${NETDATA_LIB_DIR}" "${NETDATA_CONF_DIR}/python.d" "${NETDATA_CONF_DIR}/charts.d" "${NETDATA_CONF_DIR}/node.d" do - if [ ! -d "${x}" ] - then - echo >&2 "Creating directory '${x}'" - run mkdir -p "${x}" || exit 1 - fi - - if [ ${UID} -eq 0 ] - then - if [ "${x}" = "${NETDATA_WEB_DIR}" ] - then - run chown -R "${NETDATA_WEB_USER}:${NETDATA_WEB_GROUP}" "${x}" || echo >&2 "WARNING: Cannot change the ownership of the files in directory ${x} to ${NETDATA_WEB_USER}:${NETDATA_WEB_GROUP}..." - else - run chown -R "${NETDATA_USER}:${NETDATA_USER}" "${x}" || echo >&2 "WARNING: Cannot change the ownership of the files in directory ${x} to ${NETDATA_USER}..." - fi - fi - - run chmod 0755 "${x}" || echo >&2 "WARNING: Cannot change the permissions of the directory ${x} to 0755..." + if [ ! -d "${x}" ] + then + echo >&2 "Creating directory '${x}'" + run mkdir -p "${x}" || exit 1 + fi + + if [ ${UID} -eq 0 ] + then + if [ "${x}" = "${NETDATA_WEB_DIR}" ] + then + run chown -R "${NETDATA_WEB_USER}:${NETDATA_WEB_GROUP}" "${x}" || echo >&2 "WARNING: Cannot change the ownership of the files in directory ${x} to ${NETDATA_WEB_USER}:${NETDATA_WEB_GROUP}..." + else + run chown -R "${NETDATA_USER}:${NETDATA_USER}" "${x}" || echo >&2 "WARNING: Cannot change the ownership of the files in directory ${x} to ${NETDATA_USER}..." + fi + fi + + run chmod 0755 "${x}" || echo >&2 "WARNING: Cannot change the permissions of the directory ${x} to 0755..." done if [ ${UID} -eq 0 ] - then - run chown root "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/apps.plugin" - run chmod 0755 "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/apps.plugin" - run setcap cap_dac_read_search,cap_sys_ptrace+ep "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/apps.plugin" - if [ $? -ne 0 ] - then - # fix apps.plugin to be setuid to root - run chown root "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/apps.plugin" - run chmod 4755 "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/apps.plugin" - fi + then + run chown root "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/apps.plugin" + run chmod 0755 "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/apps.plugin" + run setcap cap_dac_read_search,cap_sys_ptrace+ep "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/apps.plugin" + if [ $? -ne 0 ] + then + # fix apps.plugin to be setuid to root + run chown root "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/apps.plugin" + run chmod 4755 "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/apps.plugin" + fi fi # ----------------------------------------------------------------------------- # check if we can re-start netdata if [ ${DONOTSTART} -eq 1 ] - then - if [ ! -s "${NETDATA_PREFIX}/etc/netdata/netdata.conf" ] - then - echo >&2 "Generating empty config file in: ${NETDATA_PREFIX}/etc/netdata/netdata.conf" - echo "# Get config from http://127.0.0.1:${NETDATA_PORT}/netdata.conf" >"${NETDATA_PREFIX}/etc/netdata/netdata.conf" - - if [ "${UID}" -eq 0 ] - then - chown "${NETDATA_USER}" "${NETDATA_PREFIX}/etc/netdata/netdata.conf" - fi - chmod 0664 "${NETDATA_PREFIX}/etc/netdata/netdata.conf" - fi - echo >&2 "OK. It is now installed and ready." - exit 0 + then + if [ ! -s "${NETDATA_PREFIX}/etc/netdata/netdata.conf" ] + then + echo >&2 "Generating empty config file in: ${NETDATA_PREFIX}/etc/netdata/netdata.conf" + echo "# Get config from http://127.0.0.1:${NETDATA_PORT}/netdata.conf" >"${NETDATA_PREFIX}/etc/netdata/netdata.conf" + + if [ "${UID}" -eq 0 ] + then + chown "${NETDATA_USER}" "${NETDATA_PREFIX}/etc/netdata/netdata.conf" + fi + chmod 0664 "${NETDATA_PREFIX}/etc/netdata/netdata.conf" + fi + echo >&2 "OK. It is now installed and ready." + exit 0 fi # ----------------------------------------------------------------------------- # stop a running netdata isnetdata() { - [ -z "$1" -o ! -f "/proc/$1/stat" ] && return 1 - [ "$(cat "/proc/$1/stat" | cut -d '(' -f 2 | cut -d ')' -f 1)" = "netdata" ] && return 0 - return 1 + [ -z "$1" -o ! -f "/proc/$1/stat" ] && return 1 + [ "$(cat "/proc/$1/stat" | cut -d '(' -f 2 | cut -d ')' -f 1)" = "netdata" ] && return 0 + return 1 } +stop_netdata_on_pid() { + local pid="$1" ret=0 count=0 -echo >&2 -echo >&2 "-------------------------------------------------------------------------------" -echo >&2 -printf >&2 "Stopping a (possibly) running netdata..." -ret=0 -count=0 -while [ $ret -eq 0 ] -do - if [ $count -gt 30 ] - then - echo >&2 "Cannot stop the running netdata." - exit 1 - fi - - count=$((count + 1)) - - pid=$(cat "${NETDATA_RUN_DIR}/netdata.pid" 2>/dev/null) - # backwards compatibility - [ -z "${pid}" ] && pid=$(cat /var/run/netdata.pid 2>/dev/null) - [ -z "${pid}" ] && pid=$(cat /var/run/netdata/netdata.pid 2>/dev/null) - - isnetdata $pid || pid= - if [ ! -z "${pid}" ] - then - run kill $pid 2>/dev/null - ret=$? - else - run killall netdata 2>/dev/null - ret=$? - fi - - test $ret -eq 0 && printf >&2 "." && sleep 2 -done -echo >&2 -echo >&2 + isnetdata $pid || return 0 + + printf >&2 "Stopping netdata on pid $pid ..." + while [ ! -z "$pid" -a $ret -eq 0 ] + do + if [ $count -gt 45 ] + then + echo >&2 "Cannot stop the running netdata on pid $pid." + return 1 + fi + + count=$(( count + 1 )) + + run kill $pid 2>/dev/null + ret=$? + + test $ret -eq 0 && printf >&2 "." && sleep 2 + done + echo >&2 + if [ $ret -eq 0 ] + then + echo >&2 "SORRY! CANNOT STOP netdata ON PID $pid !" + return 1 + fi + + echo >&2 "netdata on pid $pid stopped." + return 0 +} + +stop_all_netdata() { + local p + + echo >&2 "Stopping a (possibly) running netdata..." + + for p in $(cat "${NETDATA_RUN_DIR}/netdata.pid" 2>/dev/null) \ + $(cat /var/run/netdata.pid 2>/dev/null) \ + $(cat /var/run/netdata/netdata.pid 2>/dev/null) \ + $(pidof netdata 2>/dev/null) + do + stop_netdata_on_pid $p + done +} # ----------------------------------------------------------------------------- -# run netdata +# check netdata for systemd -echo >&2 "Starting netdata..." -run ${NETDATA_PREFIX}/usr/sbin/netdata -pidfile ${NETDATA_RUN_DIR}/netdata.pid "${@}" +issystemd() { + # if the directory /etc/systemd/system does not exit, it is not systemd + [ ! -d /etc/systemd/system ] && return 1 -if [ $? -ne 0 ] - then - echo >&2 - echo >&2 "SORRY! FAILED TO START NETDATA!" - exit 1 -else - echo >&2 "OK. NetData Started!" + # if pid 1 is systemd, it is systemd + [ "$(basename $(readlink /proc/1/exe) 2>/dev/null)" = "systemd" ] && return 0 + + # if systemd is running, it is systemd + pidof systemd >/dev/null 2>&1 && return 0 + + # else, it is not systemd + return 1 +} + +started=0 +if [ "${UID}" -eq 0 ] + then + + if issystemd + then + # systemd is running on this system + + if [ ! -f /etc/systemd/system/netdata.service ] + then + echo >&2 "Installing systemd service..." + run cp system/netdata.service /etc/systemd/system/netdata.service && \ + run systemctl daemon-reload && \ + run systemctl enable netdata + else + service netdata stop + fi + + stop_all_netdata + service netdata restart && started=1 + fi + + if [ ${started} -eq 0 ] + then + # check if we can use the system service + service netdata stop + stop_all_netdata + service netdata restart && started=1 + if [ ${started} -eq 0 ] + then + service netdata start && started=1 + fi + fi fi -echo >&2 +if [ ${started} -eq 0 ] +then + # still not started... + + stop_all_netdata + + echo >&2 "Starting netdata..." + run ${NETDATA_PREFIX}/usr/sbin/netdata -P ${NETDATA_RUN_DIR}/netdata.pid "${@}" + if [ $? -ne 0 ] + then + echo >&2 + echo >&2 "SORRY! FAILED TO START NETDATA!" + exit 1 + else + echo >&2 "OK. NetData Started!" + fi + + echo >&2 +fi # ----------------------------------------------------------------------------- # save a config file, if it is not already there if [ ! -s "${NETDATA_PREFIX}/etc/netdata/netdata.conf" ] - then - echo >&2 - echo >&2 "-------------------------------------------------------------------------------" - echo >&2 - echo >&2 "Downloading default configuration from netdata..." - sleep 5 - - # remove a possibly obsolete download - [ -f "${NETDATA_PREFIX}/etc/netdata/netdata.conf.new" ] && rm "${NETDATA_PREFIX}/etc/netdata/netdata.conf.new" - - # disable a proxy to get data from the local netdata - export http_proxy= - export https_proxy= - - # try wget - wget 2>/dev/null -O "${NETDATA_PREFIX}/etc/netdata/netdata.conf.new" "http://localhost:${NETDATA_PORT}/netdata.conf" - ret=$? - - # try curl - if [ $ret -ne 0 -o ! -s "${NETDATA_PREFIX}/etc/netdata/netdata.conf.new" ] - then - curl -s -o "${NETDATA_PREFIX}/etc/netdata/netdata.conf.new" "http://localhost:${NETDATA_PORT}/netdata.conf" - ret=$? - fi - - if [ $ret -eq 0 -a -s "${NETDATA_PREFIX}/etc/netdata/netdata.conf.new" ] - then - mv "${NETDATA_PREFIX}/etc/netdata/netdata.conf.new" "${NETDATA_PREFIX}/etc/netdata/netdata.conf" - echo >&2 "New configuration saved for you to edit at ${NETDATA_PREFIX}/etc/netdata/netdata.conf" - - if [ "${UID}" -eq 0 ] - then - chown "${NETDATA_USER}" "${NETDATA_PREFIX}/etc/netdata/netdata.conf" - fi - chmod 0664 "${NETDATA_PREFIX}/etc/netdata/netdata.conf" - else - echo >&2 "Cannnot download configuration from netdata daemon using url 'http://localhost:${NETDATA_PORT}/netdata.conf'" - [ -f "${NETDATA_PREFIX}/etc/netdata/netdata.conf.new" ] && rm "${NETDATA_PREFIX}/etc/netdata/netdata.conf.new" - fi + then + echo >&2 + echo >&2 "-------------------------------------------------------------------------------" + echo >&2 + echo >&2 "Downloading default configuration from netdata..." + sleep 5 + + # remove a possibly obsolete download + [ -f "${NETDATA_PREFIX}/etc/netdata/netdata.conf.new" ] && rm "${NETDATA_PREFIX}/etc/netdata/netdata.conf.new" + + # disable a proxy to get data from the local netdata + export http_proxy= + export https_proxy= + + # try wget + wget 2>/dev/null -O "${NETDATA_PREFIX}/etc/netdata/netdata.conf.new" "http://localhost:${NETDATA_PORT}/netdata.conf" + ret=$? + + # try curl + if [ $ret -ne 0 -o ! -s "${NETDATA_PREFIX}/etc/netdata/netdata.conf.new" ] + then + curl -s -o "${NETDATA_PREFIX}/etc/netdata/netdata.conf.new" "http://localhost:${NETDATA_PORT}/netdata.conf" + ret=$? + fi + + if [ $ret -eq 0 -a -s "${NETDATA_PREFIX}/etc/netdata/netdata.conf.new" ] + then + mv "${NETDATA_PREFIX}/etc/netdata/netdata.conf.new" "${NETDATA_PREFIX}/etc/netdata/netdata.conf" + echo >&2 "New configuration saved for you to edit at ${NETDATA_PREFIX}/etc/netdata/netdata.conf" + + if [ "${UID}" -eq 0 ] + then + chown "${NETDATA_USER}" "${NETDATA_PREFIX}/etc/netdata/netdata.conf" + fi + chmod 0664 "${NETDATA_PREFIX}/etc/netdata/netdata.conf" + else + echo >&2 "Cannnot download configuration from netdata daemon using url 'http://localhost:${NETDATA_PORT}/netdata.conf'" + [ -f "${NETDATA_PREFIX}/etc/netdata/netdata.conf.new" ] && rm "${NETDATA_PREFIX}/etc/netdata/netdata.conf.new" + fi fi # ----------------------------------------------------------------------------- # Check for KSM ksm_is_available_but_disabled() { - cat <<-KSM1 + cat <<KSM1 - ------------------------------------------------------------------------------- - Memory de-duplication instructions +------------------------------------------------------------------------------- +Memory de-duplication instructions - I see you have kernel memory de-duper (called Kernel Same-page Merging, - or KSM) available, but it is not currently enabled. +You have kernel memory de-duper (called Kernel Same-page Merging, +or KSM) available, but it is not currently enabled. - To enable it run: +To enable it run: - echo 1 >/sys/kernel/mm/ksm/run - echo 1000 >/sys/kernel/mm/ksm/sleep_millisecs +echo 1 >/sys/kernel/mm/ksm/run +echo 1000 >/sys/kernel/mm/ksm/sleep_millisecs - If you enable it, you will save 40-60% of netdata memory. +If you enable it, you will save 40-60% of netdata memory. KSM1 } ksm_is_not_available() { - cat <<-KSM2 + cat <<KSM2 - ------------------------------------------------------------------------------- - Memory de-duplication not present in your kernel +------------------------------------------------------------------------------- +Memory de-duplication not present in your kernel - It seems you do not have kernel memory de-duper (called Kernel Same-page - Merging, or KSM) available. +It seems you do not have kernel memory de-duper (called Kernel Same-page +Merging, or KSM) available. - To enable it, you need a kernel built with CONFIG_KSM=y +To enable it, you need a kernel built with CONFIG_KSM=y - If you can have it, you will save 40-60% of netdata memory. +If you can have it, you will save 40-60% of netdata memory. KSM2 } if [ -f "/sys/kernel/mm/ksm/run" ] - then - if [ $(cat "/sys/kernel/mm/ksm/run") != "1" ] - then - ksm_is_available_but_disabled - fi + then + if [ $(cat "/sys/kernel/mm/ksm/run") != "1" ] + then + ksm_is_available_but_disabled + fi else - ksm_is_not_available + ksm_is_not_available fi # ----------------------------------------------------------------------------- # Check for version.txt if [ ! -s web/version.txt ] - then - cat <<-VERMSG + then + cat <<VERMSG - ------------------------------------------------------------------------------- - Version update check warning +------------------------------------------------------------------------------- +Version update check warning - The way you downloaded netdata, we cannot find its version. This means the - Update check on the dashboard, will not work. +The way you downloaded netdata, we cannot find its version. This means the +Update check on the dashboard, will not work. - If you want to have version update check, please re-install it - following the procedure in: +If you want to have version update check, please re-install it +following the procedure in: - https://github.com/firehol/netdata/wiki/Installation +https://github.com/firehol/netdata/wiki/Installation VERMSG fi @@ -697,30 +920,30 @@ fi # apps.plugin warning if [ "${UID}" -ne 0 ] - then - cat <<-SETUID_WARNING + then + cat <<SETUID_WARNING - ------------------------------------------------------------------------------- - apps.plugin needs privileges +------------------------------------------------------------------------------- +apps.plugin needs privileges - Since you have installed netdata as a normal user, to have apps.plugin collect - all the needed data, you have to give it the access rights it needs, by running - either of the following sets of commands: +Since you have installed netdata as a normal user, to have apps.plugin collect +all the needed data, you have to give it the access rights it needs, by running +either of the following sets of commands: - To run apps.plugin with escalated capabilities: +To run apps.plugin with escalated capabilities: - sudo chown root:${NETDATA_USER} "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/apps.plugin" - sudo chmod 0750 "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/apps.plugin" - sudo setcap cap_dac_read_search,cap_sys_ptrace+ep "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/apps.plugin" + sudo chown root:${NETDATA_USER} "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/apps.plugin" + sudo chmod 0750 "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/apps.plugin" + sudo setcap cap_dac_read_search,cap_sys_ptrace+ep "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/apps.plugin" - or, to run apps.plugin as root: +or, to run apps.plugin as root: - sudo chown root "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/apps.plugin" - sudo chmod 4755 "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/apps.plugin" + sudo chown root "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/apps.plugin" + sudo chmod 4755 "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/apps.plugin" - apps.plugin is performing a hard-coded function of data collection for all - running processes. It cannot be instructed from the netdata daemon to perform - any task, so it is pretty safe to do this. +apps.plugin is performing a hard-coded function of data collection for all +running processes. It cannot be instructed from the netdata daemon to perform +any task, so it is pretty safe to do this. SETUID_WARNING fi @@ -728,121 +951,122 @@ fi # ----------------------------------------------------------------------------- # Keep un-install info -cat >netdata-uninstaller.sh <<-UNINSTALL - #!/bin/bash - - # this script will uninstall netdata - - if [ "\$1" != "--force" ] - then - echo >&2 "This script will REMOVE netdata from your system." - echo >&2 "Run it again with --force to do it." - exit 1 - fi - - echo >&2 "Stopping a possibly running netdata..." - killall netdata - sleep 2 - - deletedir() { - if [ ! -z "\$1" -a -d "\$1" ] - then - echo - echo "Deleting directory '\$1' ..." - rm -I -R "\$1" - fi - } - - if [ ! -z "${NETDATA_PREFIX}" -a -d "${NETDATA_PREFIX}" ] - then - # installation prefix was given - - deletedir "${NETDATA_PREFIX}" - - else - # installation prefix was NOT given - - if [ -f "${NETDATA_PREFIX}/usr/sbin/netdata" ] - then - echo "Deleting ${NETDATA_PREFIX}/usr/sbin/netdata ..." - rm -i "${NETDATA_PREFIX}/usr/sbin/netdata" - fi - - deletedir "${NETDATA_PREFIX}/etc/netdata" - deletedir "${NETDATA_PREFIX}/usr/share/netdata" - deletedir "${NETDATA_PREFIX}/usr/libexec/netdata" - deletedir "${NETDATA_PREFIX}/var/lib/netdata" - deletedir "${NETDATA_PREFIX}/var/cache/netdata" - deletedir "${NETDATA_PREFIX}/var/log/netdata" - fi - - if [ -f /etc/logrotate.d/netdata ] - then - echo "Deleting /etc/logrotate.d/netdata ..." - rm -i /etc/logrotate.d/netdata - fi - - getent passwd netdata > /dev/null - if [ $? -eq 0 ] - then - echo - echo "You may also want to remove the user netdata" - echo "by running:" - echo " userdel netdata" - fi - - getent group netdata > /dev/null - if [ $? -eq 0 ] - then - echo - echo "You may also want to remove the group netdata" - echo "by running:" - echo " groupdel netdata" - fi - - getent group docker > /dev/null - if [ $? -eq 0 -a "${NETDATA_ADDED_TO_DOCKER}" = "1" ] - then - echo - echo "You may also want to remove the netdata user from the docker group" - echo "by running:" - echo " gpasswd -d netdata docker" - fi +cat >netdata-uninstaller.sh <<UNINSTALL +#!/usr/bin/env bash -UNINSTALL -chmod 750 netdata-uninstaller.sh +# this script will uninstall netdata -# ----------------------------------------------------------------------------- +if [ "\$1" != "--force" ] + then + echo >&2 "This script will REMOVE netdata from your system." + echo >&2 "Run it again with --force to do it." + exit 1 +fi + +echo >&2 "Stopping a possibly running netdata..." +for p in \$(pidof netdata); do kill \$p; done +sleep 2 + +deletedir() { + if [ ! -z "\$1" -a -d "\$1" ] + then + echo + echo "Deleting directory '\$1' ..." + rm -I -R "\$1" + fi +} + +if [ ! -z "${NETDATA_PREFIX}" -a -d "${NETDATA_PREFIX}" ] + then + # installation prefix was given + + deletedir "${NETDATA_PREFIX}" -if [ "${NETDATA_BIND}" = "*" ] - then - access="localhost" else - access="${NETDATA_BIND}" + # installation prefix was NOT given + + if [ -f "${NETDATA_PREFIX}/usr/sbin/netdata" ] + then + echo "Deleting ${NETDATA_PREFIX}/usr/sbin/netdata ..." + rm -i "${NETDATA_PREFIX}/usr/sbin/netdata" + fi + + deletedir "${NETDATA_PREFIX}/etc/netdata" + deletedir "${NETDATA_PREFIX}/usr/share/netdata" + deletedir "${NETDATA_PREFIX}/usr/libexec/netdata" + deletedir "${NETDATA_PREFIX}/var/lib/netdata" + deletedir "${NETDATA_PREFIX}/var/cache/netdata" + deletedir "${NETDATA_PREFIX}/var/log/netdata" fi -cat <<-END +if [ -f /etc/logrotate.d/netdata ] + then + echo "Deleting /etc/logrotate.d/netdata ..." + rm -i /etc/logrotate.d/netdata +fi + +if [ -f /etc/systemd/system/netdata.service ] + then + echo "Deleting /etc/systemd/system/netdata.service ..." + rm -i /etc/systemd/system/netdata.service +fi + +getent passwd netdata > /dev/null +if [ $? -eq 0 ] + then + echo + echo "You may also want to remove the user netdata" + echo "by running:" + echo " userdel netdata" +fi + +getent group netdata > /dev/null +if [ $? -eq 0 ] + then + echo + echo "You may also want to remove the group netdata" + echo "by running:" + echo " groupdel netdata" +fi + +getent group docker > /dev/null +if [ $? -eq 0 -a "${NETDATA_ADDED_TO_DOCKER}" = "1" ] + then + echo + echo "You may also want to remove the netdata user from the docker group" + echo "by running:" + echo " gpasswd -d netdata docker" +fi + +UNINSTALL +chmod 750 netdata-uninstaller.sh + +# ----------------------------------------------------------------------------- + +cat <<END - ------------------------------------------------------------------------------- +------------------------------------------------------------------------------- - OK. NetData is installed and it is running (listening to ${NETDATA_BIND}:${NETDATA_PORT}). +OK. NetData is installed and it is running. - ------------------------------------------------------------------------------- +------------------------------------------------------------------------------- +By default netdata listens on all IPs on port ${NETDATA_PORT}, +so you can access it with: - Hit http://${access}:${NETDATA_PORT}/ from your browser. +http://this.machine.ip:${NETDATA_PORT}/ - To stop netdata, just kill it, with: +To stop netdata, just kill it, with: - killall netdata + killall netdata - To start it, just run it: +To start it, just run it: - ${NETDATA_PREFIX}/usr/sbin/netdata + ${NETDATA_PREFIX}/usr/sbin/netdata - Enjoy! +Enjoy! END echo >&2 "Uninstall script generated: ./netdata-uninstaller.sh" diff --git a/netdata.spec b/netdata.spec index 88a7c7641..5db7717a6 100644 --- a/netdata.spec +++ b/netdata.spec @@ -1,7 +1,14 @@ +%define contentdir %{_datadir}/netdata + +# This is temporary and should eventually be resolved. This bypasses +# the default rhel __os_install_post which throws a python compile +# error. +%define __os_install_post %{nil} + # # Conditional build: -%bcond_without systemd # systemd -%bcond_without nfacct # build with nfacct plugin +%bcond_without systemd # systemd +%bcond_with nfacct # build with nfacct plugin %if 0%{?fedora} || 0%{?rhel} >= 7 %else @@ -10,25 +17,33 @@ Summary: Real-time performance monitoring, done right Name: netdata -Version: 1.2.0 -Release: %{?release_suffix}%{?dist} +Version: 1.3.0 +Release: 1%{?dist} License: GPL v3+ Group: Applications/System -Source0: http://firehol.org/download/netdata/releases/v1.2.0/%{name}-1.2.0.tar.xz +Source0: http://firehol.org/download/netdata/releases/v1.3.0/%{name}-1.3.0.tar.xz URL: http://netdata.firehol.org/ BuildRequires: pkgconfig BuildRequires: xz BuildRequires: zlib-devel + +# Packages can be found in the EPEL repo %if %{with nfacct} BuildRequires: libmnl-devel BuildRequires: libnetfilter_acct-devel %endif + +Requires(pre): /usr/sbin/groupadd +Requires(pre): /usr/sbin/useradd + %if %{with systemd} -BuildRequires: systemd -Requires(post): systemd -Requires(preun): systemd -Requires(postun): systemd +Requires(preun): systemd-units +Requires(postun): systemd-units +Requires(post): systemd-units +%else +Requires(post): chkconfig %endif + BuildRoot: %{tmpdir}/%{name}-%{version}-root-%(id -u -n) %description @@ -42,11 +57,10 @@ so that you can get insights of what is happening now and what just happened, on your systems and applications. %prep -%setup -q -n %{name}-1.2.0 +%setup -q -n %{name}-1.3.0 %build %configure \ - --docdir=%{_docdir}/%{name}-%{version} \ --with-zlib \ --with-math \ %{?with_nfacct:--enable-plugin-nfacct} \ @@ -55,23 +69,32 @@ happened, on your systems and applications. %install rm -rf $RPM_BUILD_ROOT -%{__make} %{?_smp_mflags} install \ - DESTDIR=$RPM_BUILD_ROOT +%{__make} %{?_smp_mflags} DESTDIR=$RPM_BUILD_ROOT install -install -m 644 -p system/netdata.conf $RPM_BUILD_ROOT%{_sysconfdir}/%{name}/*.conf +find $RPM_BUILD_ROOT -name .keep -print0 | xargs --null --no-run-if-empty rm -find $RPM_BUILD_ROOT -name .keep | xargs rm +install -m 644 -p system/netdata.conf $RPM_BUILD_ROOT%{_sysconfdir}/%{name} + +mkdir -p $RPM_BUILD_ROOT%{_sysconfdir}/logrotate.d +install -m 644 -p system/netdata.logrotate $RPM_BUILD_ROOT%{_sysconfdir}/logrotate.d/%{name} %if %{with systemd} install -d $RPM_BUILD_ROOT%{_unitdir} install -m 644 -p system/netdata.service $RPM_BUILD_ROOT%{_unitdir}/netdata.service +%else +# install SYSV init stuff +mkdir -p $RPM_BUILD_ROOT/etc/rc.d/init.d +install -m755 system/netdata-init-d \ + $RPM_BUILD_ROOT/etc/rc.d/init.d/netdata %endif +%if %{with systemd} %pre -getent group netdata > /dev/null || groupadd -r netdata -getent passwd netdata > /dev/null || useradd -r -g netdata -c netdata -s /sbin/nologin -d / netdata +# Add the "netdata" user +/usr/sbin/groupadd -r netdata 2> /dev/null || : +/usr/sbin/useradd -c "netdata" -g netdata \ + -s /sbin/nologin -r -d %{contentdir} netdata 2> /dev/null || : -%if %{with systemd} %post %systemd_post netdata.service @@ -80,26 +103,84 @@ getent passwd netdata > /dev/null || useradd -r -g netdata -c netdata -s /sbin/n %postun %systemd_postun_with_restart netdata.service +%else +%pre +# Add the "netdata" user +getent group netdata >/dev/null || groupadd -r netdata +getent passwd netdata >/dev/null || \ + useradd -r -g netdata -s /sbin/nologin \ + -d %{contentdir} -c "netdata" netdata +exit 0 + +%post +# Register the netdata service +/sbin/chkconfig --add netdata +# Only gets run on initial install (not upgrades or uninstalls) +if [ $1 = 1 ]; then + # Start the netdata service + /sbin/service netdata start +fi +exit 0 + +%preun +# Only gets run on uninstall (not upgrades) +if [ $1 = 0 ]; then + /sbin/service netdata stop > /dev/null 2>&1 + /sbin/chkconfig --del netdata +fi +exit 0 + +%postun +# Only gets run on upgrade (not uninstalls) +if [ $1 != 0 ]; then + /sbin/service netdata condrestart 2>&1 > /dev/null +fi +exit 0 %endif %clean rm -rf $RPM_BUILD_ROOT %files -%attr(-,netdata,netdata) %dir %{_localstatedir}/cache/%{name} -%attr(-,netdata,netdata) %dir %{_localstatedir}/log/%{name} -%config(noreplace) %verify(not md5 mtime size) %{_sysconfdir}/%{name}/*.conf +%defattr(-,root,root) + %dir %{_sysconfdir}/%{name} -%{?with_systemd:%{_unitdir}/netdata.service} +%config(noreplace) %{_sysconfdir}/%{name}/*.conf +%config(noreplace) %{_sysconfdir}/%{name}/health.d/*.conf +%config(noreplace) %{_sysconfdir}/%{name}/python.d/*.conf +%config(noreplace) %{_sysconfdir}/logrotate.d/%{name} + %{_libexecdir}/%{name} %{_sbindir}/%{name} + +%attr(0700,netdata,netdata) %dir %{_localstatedir}/cache/%{name} +%attr(0700,netdata,netdata) %dir %{_localstatedir}/log/%{name} +%attr(0700,netdata,netdata) %dir %{_localstatedir}/lib/%{name} + %dir %{_datadir}/%{name} -# override defattr for web files -%defattr(644,root,netdata,755) +%if %{with systemd} +%{_unitdir}/netdata.service +%else +%{_sysconfdir}/rc.d/init.d/netdata +%endif + +# Enforce 0644 for files and 0755 for directories +# for the netdata web directory +%defattr(0644,root,netdata,0755) %{_datadir}/%{name}/web %changelog +* Sun Aug 28 2016 Costa Tsaousis <costa@tsaousis.gr> - 1.3.0-1 +- netdata now has health monitoring +- netdata now generates badges +- netdata now has python plugins +- Several more improvements, new features and bugfixes. +* Tue Jul 26 2016 Jason Barnett <J@sonBarnett.com> - 1.2.0-2 +- Added support for EL6 +- Corrected several Requires statements +- Changed default to build without nfacct +- Removed --docdir from configure * Mon May 16 2016 Costa Tsaousis <costa@tsaousis.gr> - 1.2.0-1 - netdata is now 30% faster. - netdata now has a registry (my-netdata menu on the dashboard). diff --git a/netdata.spec.in b/netdata.spec.in index 339c5c63d..04956b6b4 100644 --- a/netdata.spec.in +++ b/netdata.spec.in @@ -1,7 +1,14 @@ +%define contentdir %{_datadir}/netdata + +# This is temporary and should eventually be resolved. This bypasses +# the default rhel __os_install_post which throws a python compile +# error. +%define __os_install_post %{nil} + # # Conditional build: -%bcond_without systemd # systemd -%bcond_without nfacct # build with nfacct plugin +%bcond_without systemd # systemd +%bcond_with nfacct # build with nfacct plugin %if 0%{?fedora} || 0%{?rhel} >= 7 %else @@ -11,7 +18,7 @@ Summary: Real-time performance monitoring, done right Name: @PACKAGE_NAME@ Version: @PACKAGE_RPM_VERSION@ -Release: @PACKAGE_RPM_RELEASE@%{?release_suffix}%{?dist} +Release: 1%{?dist} License: GPL v3+ Group: Applications/System Source0: http://firehol.org/download/netdata/releases/v@PACKAGE_VERSION@/%{name}-@PACKAGE_VERSION@.tar.xz @@ -19,16 +26,24 @@ URL: http://netdata.firehol.org/ BuildRequires: pkgconfig BuildRequires: xz BuildRequires: zlib-devel + +# Packages can be found in the EPEL repo %if %{with nfacct} BuildRequires: libmnl-devel BuildRequires: libnetfilter_acct-devel %endif + +Requires(pre): /usr/sbin/groupadd +Requires(pre): /usr/sbin/useradd + %if %{with systemd} -BuildRequires: systemd -Requires(post): systemd -Requires(preun): systemd -Requires(postun): systemd +Requires(preun): systemd-units +Requires(postun): systemd-units +Requires(post): systemd-units +%else +Requires(post): chkconfig %endif + BuildRoot: %{tmpdir}/%{name}-%{version}-root-%(id -u -n) %description @@ -46,7 +61,6 @@ happened, on your systems and applications. %build %configure \ - --docdir=%{_docdir}/%{name}-%{version} \ --with-zlib \ --with-math \ %{?with_nfacct:--enable-plugin-nfacct} \ @@ -55,23 +69,32 @@ happened, on your systems and applications. %install rm -rf $RPM_BUILD_ROOT -%{__make} %{?_smp_mflags} install \ - DESTDIR=$RPM_BUILD_ROOT +%{__make} %{?_smp_mflags} DESTDIR=$RPM_BUILD_ROOT install -install -m 644 -p system/netdata.conf $RPM_BUILD_ROOT%{_sysconfdir}/%{name}/*.conf +find $RPM_BUILD_ROOT -name .keep -print0 | xargs --null --no-run-if-empty rm -find $RPM_BUILD_ROOT -name .keep | xargs rm +install -m 644 -p system/netdata.conf $RPM_BUILD_ROOT%{_sysconfdir}/%{name} + +mkdir -p $RPM_BUILD_ROOT%{_sysconfdir}/logrotate.d +install -m 644 -p system/netdata.logrotate $RPM_BUILD_ROOT%{_sysconfdir}/logrotate.d/%{name} %if %{with systemd} install -d $RPM_BUILD_ROOT%{_unitdir} install -m 644 -p system/netdata.service $RPM_BUILD_ROOT%{_unitdir}/netdata.service +%else +# install SYSV init stuff +mkdir -p $RPM_BUILD_ROOT/etc/rc.d/init.d +install -m755 system/netdata-init-d \ + $RPM_BUILD_ROOT/etc/rc.d/init.d/netdata %endif +%if %{with systemd} %pre -getent group netdata > /dev/null || groupadd -r netdata -getent passwd netdata > /dev/null || useradd -r -g netdata -c netdata -s /sbin/nologin -d / netdata +# Add the "netdata" user +/usr/sbin/groupadd -r netdata 2> /dev/null || : +/usr/sbin/useradd -c "netdata" -g netdata \ + -s /sbin/nologin -r -d %{contentdir} netdata 2> /dev/null || : -%if %{with systemd} %post %systemd_post netdata.service @@ -80,26 +103,84 @@ getent passwd netdata > /dev/null || useradd -r -g netdata -c netdata -s /sbin/n %postun %systemd_postun_with_restart netdata.service +%else +%pre +# Add the "netdata" user +getent group netdata >/dev/null || groupadd -r netdata +getent passwd netdata >/dev/null || \ + useradd -r -g netdata -s /sbin/nologin \ + -d %{contentdir} -c "netdata" netdata +exit 0 + +%post +# Register the netdata service +/sbin/chkconfig --add netdata +# Only gets run on initial install (not upgrades or uninstalls) +if [ $1 = 1 ]; then + # Start the netdata service + /sbin/service netdata start +fi +exit 0 + +%preun +# Only gets run on uninstall (not upgrades) +if [ $1 = 0 ]; then + /sbin/service netdata stop > /dev/null 2>&1 + /sbin/chkconfig --del netdata +fi +exit 0 + +%postun +# Only gets run on upgrade (not uninstalls) +if [ $1 != 0 ]; then + /sbin/service netdata condrestart 2>&1 > /dev/null +fi +exit 0 %endif %clean rm -rf $RPM_BUILD_ROOT %files -%attr(-,netdata,netdata) %dir %{_localstatedir}/cache/%{name} -%attr(-,netdata,netdata) %dir %{_localstatedir}/log/%{name} -%config(noreplace) %verify(not md5 mtime size) %{_sysconfdir}/%{name}/*.conf +%defattr(-,root,root) + %dir %{_sysconfdir}/%{name} -%{?with_systemd:%{_unitdir}/netdata.service} +%config(noreplace) %{_sysconfdir}/%{name}/*.conf +%config(noreplace) %{_sysconfdir}/%{name}/health.d/*.conf +%config(noreplace) %{_sysconfdir}/%{name}/python.d/*.conf +%config(noreplace) %{_sysconfdir}/logrotate.d/%{name} + %{_libexecdir}/%{name} %{_sbindir}/%{name} + +%attr(0700,netdata,netdata) %dir %{_localstatedir}/cache/%{name} +%attr(0700,netdata,netdata) %dir %{_localstatedir}/log/%{name} +%attr(0700,netdata,netdata) %dir %{_localstatedir}/lib/%{name} + %dir %{_datadir}/%{name} -# override defattr for web files -%defattr(644,root,netdata,755) +%if %{with systemd} +%{_unitdir}/netdata.service +%else +%{_sysconfdir}/rc.d/init.d/netdata +%endif + +# Enforce 0644 for files and 0755 for directories +# for the netdata web directory +%defattr(0644,root,netdata,0755) %{_datadir}/%{name}/web %changelog +* Sun Aug 28 2016 Costa Tsaousis <costa@tsaousis.gr> - 1.3.0-1 +- netdata now has health monitoring +- netdata now generates badges +- netdata now has python plugins +- Several more improvements, new features and bugfixes. +* Tue Jul 26 2016 Jason Barnett <J@sonBarnett.com> - 1.2.0-2 +- Added support for EL6 +- Corrected several Requires statements +- Changed default to build without nfacct +- Removed --docdir from configure * Mon May 16 2016 Costa Tsaousis <costa@tsaousis.gr> - 1.2.0-1 - netdata is now 30% faster. - netdata now has a registry (my-netdata menu on the dashboard). diff --git a/node.d/Makefile.in b/node.d/Makefile.in index cb073117c..12a870f59 100644 --- a/node.d/Makefile.in +++ b/node.d/Makefile.in @@ -264,6 +264,8 @@ pluginsdir = @pluginsdir@ prefix = @prefix@ program_transform_name = @program_transform_name@ psdir = @psdir@ +pythondir = @pythondir@ +registrydir = @registrydir@ sbindir = @sbindir@ sharedstatedir = @sharedstatedir@ srcdir = @srcdir@ diff --git a/node.d/named.node.js b/node.d/named.node.js index c2b15eae7..19b8ce29b 100644 --- a/node.d/named.node.js +++ b/node.d/named.node.js @@ -11,20 +11,20 @@ /* { - "enable_autodetect": true, - "update_every": 5, - "servers": [ - { - "name": "bind1", - "url": "http://127.0.0.1:8888/json/v1/server", - "update_every": 1 - }, - { - "name": "bind2", - "url": "http://10.0.0.1:8888/xml/v3/server", - "update_every": 2 - } - ] + "enable_autodetect": true, + "update_every": 5, + "servers": [ + { + "name": "bind1", + "url": "http://127.0.0.1:8888/json/v1/server", + "update_every": 1 + }, + { + "name": "bind2", + "url": "http://10.0.0.1:8888/xml/v3/server", + "update_every": 2 + } + ] } */ @@ -44,543 +44,543 @@ var netdata = require('netdata'); if(netdata.options.DEBUG === true) netdata.debug('loaded ' + __filename + ' plugin'); var named = { - name: __filename, - enable_autodetect: true, - update_every: 1, - base_priority: 60000, - charts: {}, - - chartFromMembersCreate: function(service, obj, id, title_suffix, units, family, context, type, priority, algorithm, multiplier, divisor) { - var chart = { - id: id, // the unique id of the chart - name: '', // the unique name of the chart - title: service.name + ' ' + title_suffix, // the title of the chart - units: units, // the units of the chart dimensions - family: family, // the family of the chart - context: context, // the context of the chart - type: type, // the type of the chart - priority: priority, // the priority relative to others in the same family - update_every: service.update_every, // the expected update frequency of the chart - dimensions: {} - } - - var found = 0; - for(var x in obj) { - if(typeof(obj[x]) !== 'undefined' && obj[x] !== 0) { - found++; - chart.dimensions[x] = { - id: x, // the unique id of the dimension - name: x, // the name of the dimension - algorithm: algorithm, // the id of the netdata algorithm - multiplier: multiplier, // the multiplier - divisor: divisor, // the divisor - hidden: false // is hidden (boolean) - } - } - } - - if(found === false) - return null; - - chart = service.chart(id, chart); - this.charts[id] = chart; - return chart; - }, - - chartFromMembers: function(service, obj, id_suffix, title_suffix, units, family, context, type, priority, algorithm, multiplier, divisor) { - var id = 'named_' + service.name + '.' + id_suffix; - var chart = this.charts[id]; - - if(typeof chart === 'undefined') { - chart = this.chartFromMembersCreate(service, obj, id, title_suffix, units, family, context, type, priority, algorithm, multiplier, divisor); - if(chart === null) return false; - } - else { - // check if we need to re-generate the chart - for(var x in obj) { - if(typeof(chart.dimensions[x]) === 'undefined') { - chart = this.chartFromMembersCreate(service, obj, id, title_suffix, units, family, context, type, priority, algorithm, multiplier, divisor); - if(chart === null) return false; - break; - } - } - } - - var found = 0; - service.begin(chart); - for(var x in obj) { - if(typeof(chart.dimensions[x]) !== 'undefined') { - found++; - service.set(x, obj[x]); - } - } - service.end(); - - if(found > 0) return true; - return false; - }, - - // an index to map values to different charts - lookups: { - nsstats: {}, - resolver_stats: {}, - numfetch: {} - }, - - // transform the XML response of bind - // to the JSON response of bind - xml2js: function(service, data_xml) { - var d = XML.parse(data_xml); - if(d === null) return null; - - var data = {}; - var len = d.server.counters.length; - while(len--) { - var a = d.server.counters[len]; - if(typeof a.counter === 'undefined') continue; - if(a.type === 'opcode') a.type = 'opcodes'; - else if(a.type === 'qtype') a.type = 'qtypes'; - else if(a.type === 'nsstat') a.type = 'nsstats'; - var aa = data[a.type] = {}; - var alen = 0 - var alen2 = a.counter.length; - while(alen < alen2) { - aa[a.counter[alen].name] = parseInt(a.counter[alen]._Data); - alen++; - } - } - - data.views = {}; - var vlen = d.views.view.length; - while(vlen--) { - var vname = d.views.view[vlen].name; - data.views[vname] = { resolver: {} }; - var len = d.views.view[vlen].counters.length; - while(len--) { - var a = d.views.view[vlen].counters[len]; - if(typeof a.counter === 'undefined') continue; - if(a.type === 'resstats') a.type = 'stats'; - else if(a.type === 'resqtype') a.type = 'qtypes'; - else if(a.type === 'adbstat') a.type = 'adb'; - var aa = data.views[vname].resolver[a.type] = {}; - var alen = 0; - var alen2 = a.counter.length; - while(alen < alen2) { - aa[a.counter[alen].name] = parseInt(a.counter[alen]._Data); - alen++; - } - } - } - - return data; - }, - - processResponse: function(service, data) { - if(data !== null) { - var r; - - // parse XML or JSON - // pepending on the URL given - if(service.request.path.match(/^\/xml/) !== null) - r = named.xml2js(service, data); - else - r = JSON.parse(data); - - if(typeof r === 'undefined' || r === null) { - netdata.serviceError(service, "Cannot parse these data: " + data); - return; - } - - if(service.added !== true) - service.commit(); - - if(typeof r.nsstats !== 'undefined') { - // we split the nsstats object to several others - var global_requests = {}, global_requests_enable = false; - var global_failures = {}, global_failures_enable = false; - var global_failures_detail = {}, global_failures_detail_enable = false; - var global_updates = {}, global_updates_enable = false; - var protocol_queries = {}, protocol_queries_enable = false; - var global_queries = {}, global_queries_enable = false; - var global_queries_success = {}, global_queries_success_enable = false; - var default_enable = false; - var RecursClients = 0; - - // RecursClients is an absolute value - if(typeof r.nsstats['RecursClients'] !== 'undefined') { - RecursClients = r.nsstats['RecursClients']; - delete r.nsstats['RecursClients']; - } - - for( var x in r.nsstats ) { - // we maintain an index of the values found - // mapping them to objects splitted - - var look = named.lookups.nsstats[x]; - if(typeof look === 'undefined') { - // a new value, not found in the index - // index it: - if(x === 'Requestv4') { - named.lookups.nsstats[x] = { - name: 'IPv4', - type: 'global_requests' - }; - } - else if(x === 'Requestv6') { - named.lookups.nsstats[x] = { - name: 'IPv6', - type: 'global_requests' - }; - } - else if(x === 'QryFailure') { - named.lookups.nsstats[x] = { - name: 'failures', - type: 'global_failures' - }; - } - else if(x === 'QryUDP') { - named.lookups.nsstats[x] = { - name: 'UDP', - type: 'protocol_queries' - }; - } - else if(x === 'QryTCP') { - named.lookups.nsstats[x] = { - name: 'TCP', - type: 'protocol_queries' - }; - } - else if(x === 'QrySuccess') { - named.lookups.nsstats[x] = { - name: 'queries', - type: 'global_queries_success' - }; - } - else if(x.match(/QryRej$/) !== null) { - named.lookups.nsstats[x] = { - name: x, - type: 'global_failures_detail' - }; - } - else if(x.match(/^Qry/) !== null) { - named.lookups.nsstats[x] = { - name: x, - type: 'global_queries' - }; - } - else if(x.match(/^Update/) !== null) { - named.lookups.nsstats[x] = { - name: x, - type: 'global_updates' - }; - } - else { - // values not mapped, will remain - // in the default map - named.lookups.nsstats[x] = { - name: x, - type: 'default' - }; - } - - look = named.lookups.nsstats[x]; - // netdata.error('lookup nsstats value: ' + x + ' >>> ' + named.lookups.nsstats[x].type); - } - - switch(look.type) { - case 'global_requests': global_requests[look.name] = r.nsstats[x]; delete r.nsstats[x]; global_requests_enable = true; break; - case 'global_queries': global_queries[look.name] = r.nsstats[x]; delete r.nsstats[x]; global_queries_enable = true; break; - case 'global_queries_success': global_queries_success[look.name] = r.nsstats[x]; delete r.nsstats[x]; global_queries_success_enable = true; break; - case 'global_updates': global_updates[look.name] = r.nsstats[x]; delete r.nsstats[x]; global_updates_enable = true; break; - case 'protocol_queries': protocol_queries[look.name] = r.nsstats[x]; delete r.nsstats[x]; protocol_queries_enable = true; break; - case 'global_failures': global_failures[look.name] = r.nsstats[x]; delete r.nsstats[x]; global_failures_enable = true; break; - case 'global_failures_detail': global_failures_detail[look.name] = r.nsstats[x]; delete r.nsstats[x]; global_failures_detail_enable = true; break; - default: default_enable = true; break; - } - } - - if(global_requests_enable == true) - service.module.chartFromMembers(service, global_requests, 'received_requests', 'Bind, Global Received Requests by IP version', 'requests/s', 'requests', 'named.requests', netdata.chartTypes.stacked, named.base_priority + 1, netdata.chartAlgorithms.incremental, 1, 1); - - if(global_queries_success_enable == true) - service.module.chartFromMembers(service, global_queries_success, 'global_queries_success', 'Bind, Global Successful Queries', 'queries/s', 'queries', 'named.queries.succcess', netdata.chartTypes.line, named.base_priority + 2, netdata.chartAlgorithms.incremental, 1, 1); - - if(protocol_queries_enable == true) - service.module.chartFromMembers(service, protocol_queries, 'protocols_queries', 'Bind, Global Queries by IP Protocol', 'queries/s', 'queries', 'named.protocol.queries', netdata.chartTypes.stacked, named.base_priority + 3, netdata.chartAlgorithms.incremental, 1, 1); - - if(global_queries_enable == true) - service.module.chartFromMembers(service, global_queries, 'global_queries', 'Bind, Global Queries Analysis', 'queries/s', 'queries', 'named.global.queries', netdata.chartTypes.stacked, named.base_priority + 4, netdata.chartAlgorithms.incremental, 1, 1); - - if(global_updates_enable == true) - service.module.chartFromMembers(service, global_updates, 'received_updates', 'Bind, Global Received Updates', 'updates/s', 'updates', 'named.global.updates', netdata.chartTypes.stacked, named.base_priority + 5, netdata.chartAlgorithms.incremental, 1, 1); - - if(global_failures_enable == true) - service.module.chartFromMembers(service, global_failures, 'query_failures', 'Bind, Global Query Failures', 'failures/s', 'failures', 'named.global.failures', netdata.chartTypes.line, named.base_priority + 6, netdata.chartAlgorithms.incremental, 1, 1); - - if(global_failures_detail_enable == true) - service.module.chartFromMembers(service, global_failures_detail, 'query_failures_detail', 'Bind, Global Query Failures Analysis', 'failures/s', 'failures', 'named.global.failures.detail', netdata.chartTypes.stacked, named.base_priority + 7, netdata.chartAlgorithms.incremental, 1, 1); - - if(default_enable === true) - service.module.chartFromMembers(service, r.nsstats, 'nsstats', 'Bind, Other Global Server Statistics', 'operations/s', 'other', 'named.nsstats', netdata.chartTypes.line, named.base_priority + 8, netdata.chartAlgorithms.incremental, 1, 1); - - // RecursClients chart - { - var id = 'named_' + service.name + '.recursive_clients'; - var chart = named.charts[id]; - - if(typeof chart === 'undefined') { - chart = { - id: id, // the unique id of the chart - name: '', // the unique name of the chart - title: service.name + ' Bind, Current Recursive Clients', // the title of the chart - units: 'clients', // the units of the chart dimensions - family: 'clients', // the family of the chart - context: 'named.recursive.clients', // the context of the chart - type: netdata.chartTypes.line, // the type of the chart - priority: named.base_priority + 1, // the priority relative to others in the same family - update_every: service.update_every, // the expected update frequency of the chart - dimensions: { - 'clients': { - id: 'clients', // the unique id of the dimension - name: '', // the name of the dimension - algorithm: netdata.chartAlgorithms.absolute,// the id of the netdata algorithm - multiplier: 1, // the multiplier - divisor: 1, // the divisor - hidden: false // is hidden (boolean) - } - } - }; - - chart = service.chart(id, chart); - named.charts[id] = chart; - } - - service.begin(chart); - service.set('clients', RecursClients); - service.end(); - } - } - - if(typeof r.opcodes !== 'undefined') - service.module.chartFromMembers(service, r.opcodes, 'in_opcodes', 'Bind, Global Incoming Requests by OpCode', 'requests/s', 'requests', 'named.in.opcodes', netdata.chartTypes.stacked, named.base_priority + 9, netdata.chartAlgorithms.incremental, 1, 1); - - if(typeof r.qtypes !== 'undefined') - service.module.chartFromMembers(service, r.qtypes, 'in_qtypes', 'Bind, Global Incoming Requests by Query Type', 'requests/s', 'requests', 'named.in.qtypes', netdata.chartTypes.stacked, named.base_priority + 10, netdata.chartAlgorithms.incremental, 1, 1); - - if(typeof r.sockstats !== 'undefined') - service.module.chartFromMembers(service, r.sockstats, 'in_sockstats', 'Bind, Global Socket Statistics', 'operations/s', 'sockets', 'named.in.sockstats', netdata.chartTypes.line, named.base_priority + 11, netdata.chartAlgorithms.incremental, 1, 1); - - if(typeof r.views !== 'undefined') { - for( var x in r.views ) { - var resolver = r.views[x].resolver; - - if(typeof resolver !== 'undefined') { - if(typeof resolver.stats !== 'undefined') { - var NumFetch = 0; - var key = service.name + '.' + x; - var default_enable = false; - var rtt = {}, rtt_enable = false; - - // NumFetch is an absolute value - if(typeof resolver.stats['NumFetch'] !== 'undefined') { - named.lookups.numfetch[key] = true; - NumFetch = resolver.stats['NumFetch']; - delete resolver.stats['NumFetch']; - } - if(typeof resolver.stats['BucketSize'] !== 'undefined') { - delete resolver.stats['BucketSize']; - } - - // split the QryRTT* from the main chart - for( var y in resolver.stats ) { - // we maintain an index of the values found - // mapping them to objects splitted - - var look = named.lookups.resolver_stats[y]; - if(typeof look === 'undefined') { - if(y.match(/^QryRTT/) !== null) { - named.lookups.resolver_stats[y] = { - name: y, - type: 'rtt' - }; - } - else { - named.lookups.resolver_stats[y] = { - name: y, - type: 'default' - }; - } - - look = named.lookups.resolver_stats[y]; - // netdata.error('lookup resolver stats value: ' + y + ' >>> ' + look.type); - } - - switch(look.type) { - case 'rtt': rtt[look.name] = resolver.stats[y]; delete resolver.stats[y]; rtt_enable = true; break; - default: default_enable = true; break; - } - } - - if(rtt_enable) - service.module.chartFromMembers(service, rtt, 'view_resolver_rtt_' + x, 'Bind, ' + x + ' View, Resolver Round Trip Timings', 'queries/s', 'view_' + x, 'named.resolver.rtt', netdata.chartTypes.stacked, named.base_priority + 12, netdata.chartAlgorithms.incremental, 1, 1); - - if(default_enable) - service.module.chartFromMembers(service, resolver.stats, 'view_resolver_stats_' + x, 'Bind, ' + x + ' View, Resolver Statistics', 'operations/s', 'view_' + x, 'named.resolver.stats', netdata.chartTypes.line, named.base_priority + 13, netdata.chartAlgorithms.incremental, 1, 1); - - // NumFetch chart - if(typeof named.lookups.numfetch[key] !== 'undefined') { - var id = 'named_' + service.name + '.view_resolver_numfetch_' + x; - var chart = named.charts[id]; - - if(typeof chart === 'undefined') { - chart = { - id: id, // the unique id of the chart - name: '', // the unique name of the chart - title: service.name + ' Bind, ' + x + ' View, Resolver Active Queries', // the title of the chart - units: 'queries', // the units of the chart dimensions - family: 'view_' + x, // the family of the chart - context: 'named.resolver.active.queries', // the context of the chart - type: netdata.chartTypes.line, // the type of the chart - priority: named.base_priority + 1001, // the priority relative to others in the same family - update_every: service.update_every, // the expected update frequency of the chart - dimensions: { - 'queries': { - id: 'queries', // the unique id of the dimension - name: '', // the name of the dimension - algorithm: netdata.chartAlgorithms.absolute,// the id of the netdata algorithm - multiplier: 1, // the multiplier - divisor: 1, // the divisor - hidden: false // is hidden (boolean) - } - } - }; - - chart = service.chart(id, chart); - named.charts[id] = chart; - } - - service.begin(chart); - service.set('queries', NumFetch); - service.end(); - } - } - } - - if(typeof resolver.qtypes !== 'undefined') - service.module.chartFromMembers(service, resolver.qtypes, 'view_resolver_qtypes_' + x, 'Bind, ' + x + ' View, Requests by Query Type', 'requests/s', 'view_' + x, 'named.resolver.qtypes', netdata.chartTypes.stacked, named.base_priority + 14, netdata.chartAlgorithms.incremental, 1, 1); - - //if(typeof resolver.cache !== 'undefined') - // service.module.chartFromMembers(service, resolver.cache, 'view_resolver_cache_' + x, 'Bind, ' + x + ' View, Cache Entries', 'entries', 'view_' + x, 'named.resolver.cache', netdata.chartTypes.stacked, named.base_priority + 15, netdata.chartAlgorithms.absolute, 1, 1); - - if(typeof resolver.cachestats['CacheHits'] !== 'undefined' && resolver.cachestats['CacheHits'] > 0) { - var id = 'named_' + service.name + '.view_resolver_cachehits_' + x; - var chart = named.charts[id]; - - if(typeof chart === 'undefined') { - chart = { - id: id, // the unique id of the chart - name: '', // the unique name of the chart - title: service.name + ' Bind, ' + x + ' View, Resolver Cache Hits', // the title of the chart - units: 'operations/s', // the units of the chart dimensions - family: 'view_' + x, // the family of the chart - context: 'named.resolver.cache.hits', // the context of the chart - type: netdata.chartTypes.area, // the type of the chart - priority: named.base_priority + 1100, // the priority relative to others in the same family - update_every: service.update_every, // the expected update frequency of the chart - dimensions: { - 'CacheHits': { - id: 'CacheHits', // the unique id of the dimension - name: 'hits', // the name of the dimension - algorithm: netdata.chartAlgorithms.incremental,// the id of the netdata algorithm - multiplier: 1, // the multiplier - divisor: 1, // the divisor - hidden: false // is hidden (boolean) - }, - 'CacheMisses': { - id: 'CacheMisses', // the unique id of the dimension - name: 'misses', // the name of the dimension - algorithm: netdata.chartAlgorithms.incremental,// the id of the netdata algorithm - multiplier: -1, // the multiplier - divisor: 1, // the divisor - hidden: false // is hidden (boolean) - } - } - }; - - chart = service.chart(id, chart); - named.charts[id] = chart; - } - - service.begin(chart); - service.set('CacheHits', resolver.cachestats['CacheHits']); - service.set('CacheMisses', resolver.cachestats['CacheMisses']); - service.end(); - } - - // this is wrong, it contains many types of info: - // 1. CacheHits, CacheMisses - incremental (added above) - // 2. QueryHits, QueryMisses - incremental - // 3. DeleteLRU, DeleteTTL - incremental - // 4. CacheNodes, CacheBuckets - absolute - // 5. TreeMemTotal, TreeMemInUse - absolute - // 6. HeapMemMax, HeapMemTotal, HeapMemInUse - absolute - //if(typeof resolver.cachestats !== 'undefined') - // service.module.chartFromMembers(service, resolver.cachestats, 'view_resolver_cachestats_' + x, 'Bind, ' + x + ' View, Cache Statistics', 'requests/s', 'view_' + x, 'named.resolver.cache.stats', netdata.chartTypes.line, named.base_priority + 1001, netdata.chartAlgorithms.incremental, 1, 1); - - //if(typeof resolver.adb !== 'undefined') - // service.module.chartFromMembers(service, resolver.adb, 'view_resolver_adb_' + x, 'Bind, ' + x + ' View, ADB Statistics', 'entries', 'view_' + x, 'named.resolver.adb', netdata.chartTypes.line, named.base_priority + 1002, netdata.chartAlgorithms.absolute, 1, 1); - } - } - } - }, - - // module.serviceExecute() - // this function is called only from this module - // its purpose is to prepare the request and call - // netdata.serviceExecute() - serviceExecute: function(name, a_url, update_every) { - if(netdata.options.DEBUG === true) netdata.debug(this.name + ': ' + name + ': url: ' + a_url + ', update_every: ' + update_every); - var service = netdata.service({ - name: name, - request: netdata.requestFromURL(a_url), - update_every: update_every, - module: this - }); - - service.execute(this.processResponse); - }, - - configure: function(config) { - var added = 0; - - if(this.enable_autodetect === true) { - this.serviceExecute('local', 'http://localhost:8888/json/v1/server', this.update_every); - added++; - } - - if(typeof(config.servers) !== 'undefined') { - var len = config.servers.length; - while(len--) { - if(typeof config.servers[len].update_every === 'undefined') - config.servers[len].update_every = this.update_every; - - this.serviceExecute(config.servers[len].name, config.servers[len].url, config.servers[len].update_every); - added++; - } - } - - return added; - }, - - // module.update() - // this is called repeatidly to collect data, by calling - // netdata.serviceExecute() - update: function(service, callback) { - service.execute(function(serv, data) { - service.module.processResponse(serv, data); - callback(); - }); - }, + name: __filename, + enable_autodetect: true, + update_every: 1, + base_priority: 60000, + charts: {}, + + chartFromMembersCreate: function(service, obj, id, title_suffix, units, family, context, type, priority, algorithm, multiplier, divisor) { + var chart = { + id: id, // the unique id of the chart + name: '', // the unique name of the chart + title: service.name + ' ' + title_suffix, // the title of the chart + units: units, // the units of the chart dimensions + family: family, // the family of the chart + context: context, // the context of the chart + type: type, // the type of the chart + priority: priority, // the priority relative to others in the same family + update_every: service.update_every, // the expected update frequency of the chart + dimensions: {} + } + + var found = 0; + for(var x in obj) { + if(typeof(obj[x]) !== 'undefined' && obj[x] !== 0) { + found++; + chart.dimensions[x] = { + id: x, // the unique id of the dimension + name: x, // the name of the dimension + algorithm: algorithm, // the id of the netdata algorithm + multiplier: multiplier, // the multiplier + divisor: divisor, // the divisor + hidden: false // is hidden (boolean) + } + } + } + + if(found === false) + return null; + + chart = service.chart(id, chart); + this.charts[id] = chart; + return chart; + }, + + chartFromMembers: function(service, obj, id_suffix, title_suffix, units, family, context, type, priority, algorithm, multiplier, divisor) { + var id = 'named_' + service.name + '.' + id_suffix; + var chart = this.charts[id]; + + if(typeof chart === 'undefined') { + chart = this.chartFromMembersCreate(service, obj, id, title_suffix, units, family, context, type, priority, algorithm, multiplier, divisor); + if(chart === null) return false; + } + else { + // check if we need to re-generate the chart + for(var x in obj) { + if(typeof(chart.dimensions[x]) === 'undefined') { + chart = this.chartFromMembersCreate(service, obj, id, title_suffix, units, family, context, type, priority, algorithm, multiplier, divisor); + if(chart === null) return false; + break; + } + } + } + + var found = 0; + service.begin(chart); + for(var x in obj) { + if(typeof(chart.dimensions[x]) !== 'undefined') { + found++; + service.set(x, obj[x]); + } + } + service.end(); + + if(found > 0) return true; + return false; + }, + + // an index to map values to different charts + lookups: { + nsstats: {}, + resolver_stats: {}, + numfetch: {} + }, + + // transform the XML response of bind + // to the JSON response of bind + xml2js: function(service, data_xml) { + var d = XML.parse(data_xml); + if(d === null) return null; + + var data = {}; + var len = d.server.counters.length; + while(len--) { + var a = d.server.counters[len]; + if(typeof a.counter === 'undefined') continue; + if(a.type === 'opcode') a.type = 'opcodes'; + else if(a.type === 'qtype') a.type = 'qtypes'; + else if(a.type === 'nsstat') a.type = 'nsstats'; + var aa = data[a.type] = {}; + var alen = 0 + var alen2 = a.counter.length; + while(alen < alen2) { + aa[a.counter[alen].name] = parseInt(a.counter[alen]._Data); + alen++; + } + } + + data.views = {}; + var vlen = d.views.view.length; + while(vlen--) { + var vname = d.views.view[vlen].name; + data.views[vname] = { resolver: {} }; + var len = d.views.view[vlen].counters.length; + while(len--) { + var a = d.views.view[vlen].counters[len]; + if(typeof a.counter === 'undefined') continue; + if(a.type === 'resstats') a.type = 'stats'; + else if(a.type === 'resqtype') a.type = 'qtypes'; + else if(a.type === 'adbstat') a.type = 'adb'; + var aa = data.views[vname].resolver[a.type] = {}; + var alen = 0; + var alen2 = a.counter.length; + while(alen < alen2) { + aa[a.counter[alen].name] = parseInt(a.counter[alen]._Data); + alen++; + } + } + } + + return data; + }, + + processResponse: function(service, data) { + if(data !== null) { + var r; + + // parse XML or JSON + // pepending on the URL given + if(service.request.path.match(/^\/xml/) !== null) + r = named.xml2js(service, data); + else + r = JSON.parse(data); + + if(typeof r === 'undefined' || r === null) { + netdata.serviceError(service, "Cannot parse these data: " + data); + return; + } + + if(service.added !== true) + service.commit(); + + if(typeof r.nsstats !== 'undefined') { + // we split the nsstats object to several others + var global_requests = {}, global_requests_enable = false; + var global_failures = {}, global_failures_enable = false; + var global_failures_detail = {}, global_failures_detail_enable = false; + var global_updates = {}, global_updates_enable = false; + var protocol_queries = {}, protocol_queries_enable = false; + var global_queries = {}, global_queries_enable = false; + var global_queries_success = {}, global_queries_success_enable = false; + var default_enable = false; + var RecursClients = 0; + + // RecursClients is an absolute value + if(typeof r.nsstats['RecursClients'] !== 'undefined') { + RecursClients = r.nsstats['RecursClients']; + delete r.nsstats['RecursClients']; + } + + for( var x in r.nsstats ) { + // we maintain an index of the values found + // mapping them to objects splitted + + var look = named.lookups.nsstats[x]; + if(typeof look === 'undefined') { + // a new value, not found in the index + // index it: + if(x === 'Requestv4') { + named.lookups.nsstats[x] = { + name: 'IPv4', + type: 'global_requests' + }; + } + else if(x === 'Requestv6') { + named.lookups.nsstats[x] = { + name: 'IPv6', + type: 'global_requests' + }; + } + else if(x === 'QryFailure') { + named.lookups.nsstats[x] = { + name: 'failures', + type: 'global_failures' + }; + } + else if(x === 'QryUDP') { + named.lookups.nsstats[x] = { + name: 'UDP', + type: 'protocol_queries' + }; + } + else if(x === 'QryTCP') { + named.lookups.nsstats[x] = { + name: 'TCP', + type: 'protocol_queries' + }; + } + else if(x === 'QrySuccess') { + named.lookups.nsstats[x] = { + name: 'queries', + type: 'global_queries_success' + }; + } + else if(x.match(/QryRej$/) !== null) { + named.lookups.nsstats[x] = { + name: x, + type: 'global_failures_detail' + }; + } + else if(x.match(/^Qry/) !== null) { + named.lookups.nsstats[x] = { + name: x, + type: 'global_queries' + }; + } + else if(x.match(/^Update/) !== null) { + named.lookups.nsstats[x] = { + name: x, + type: 'global_updates' + }; + } + else { + // values not mapped, will remain + // in the default map + named.lookups.nsstats[x] = { + name: x, + type: 'default' + }; + } + + look = named.lookups.nsstats[x]; + // netdata.error('lookup nsstats value: ' + x + ' >>> ' + named.lookups.nsstats[x].type); + } + + switch(look.type) { + case 'global_requests': global_requests[look.name] = r.nsstats[x]; delete r.nsstats[x]; global_requests_enable = true; break; + case 'global_queries': global_queries[look.name] = r.nsstats[x]; delete r.nsstats[x]; global_queries_enable = true; break; + case 'global_queries_success': global_queries_success[look.name] = r.nsstats[x]; delete r.nsstats[x]; global_queries_success_enable = true; break; + case 'global_updates': global_updates[look.name] = r.nsstats[x]; delete r.nsstats[x]; global_updates_enable = true; break; + case 'protocol_queries': protocol_queries[look.name] = r.nsstats[x]; delete r.nsstats[x]; protocol_queries_enable = true; break; + case 'global_failures': global_failures[look.name] = r.nsstats[x]; delete r.nsstats[x]; global_failures_enable = true; break; + case 'global_failures_detail': global_failures_detail[look.name] = r.nsstats[x]; delete r.nsstats[x]; global_failures_detail_enable = true; break; + default: default_enable = true; break; + } + } + + if(global_requests_enable == true) + service.module.chartFromMembers(service, global_requests, 'received_requests', 'Bind, Global Received Requests by IP version', 'requests/s', 'requests', 'named.requests', netdata.chartTypes.stacked, named.base_priority + 1, netdata.chartAlgorithms.incremental, 1, 1); + + if(global_queries_success_enable == true) + service.module.chartFromMembers(service, global_queries_success, 'global_queries_success', 'Bind, Global Successful Queries', 'queries/s', 'queries', 'named.queries_succcess', netdata.chartTypes.line, named.base_priority + 2, netdata.chartAlgorithms.incremental, 1, 1); + + if(protocol_queries_enable == true) + service.module.chartFromMembers(service, protocol_queries, 'protocols_queries', 'Bind, Global Queries by IP Protocol', 'queries/s', 'queries', 'named.protocol_queries', netdata.chartTypes.stacked, named.base_priority + 3, netdata.chartAlgorithms.incremental, 1, 1); + + if(global_queries_enable == true) + service.module.chartFromMembers(service, global_queries, 'global_queries', 'Bind, Global Queries Analysis', 'queries/s', 'queries', 'named.global_queries', netdata.chartTypes.stacked, named.base_priority + 4, netdata.chartAlgorithms.incremental, 1, 1); + + if(global_updates_enable == true) + service.module.chartFromMembers(service, global_updates, 'received_updates', 'Bind, Global Received Updates', 'updates/s', 'updates', 'named.global_updates', netdata.chartTypes.stacked, named.base_priority + 5, netdata.chartAlgorithms.incremental, 1, 1); + + if(global_failures_enable == true) + service.module.chartFromMembers(service, global_failures, 'query_failures', 'Bind, Global Query Failures', 'failures/s', 'failures', 'named.global_failures', netdata.chartTypes.line, named.base_priority + 6, netdata.chartAlgorithms.incremental, 1, 1); + + if(global_failures_detail_enable == true) + service.module.chartFromMembers(service, global_failures_detail, 'query_failures_detail', 'Bind, Global Query Failures Analysis', 'failures/s', 'failures', 'named.global_failures_detail', netdata.chartTypes.stacked, named.base_priority + 7, netdata.chartAlgorithms.incremental, 1, 1); + + if(default_enable === true) + service.module.chartFromMembers(service, r.nsstats, 'nsstats', 'Bind, Other Global Server Statistics', 'operations/s', 'other', 'named.nsstats', netdata.chartTypes.line, named.base_priority + 8, netdata.chartAlgorithms.incremental, 1, 1); + + // RecursClients chart + { + var id = 'named_' + service.name + '.recursive_clients'; + var chart = named.charts[id]; + + if(typeof chart === 'undefined') { + chart = { + id: id, // the unique id of the chart + name: '', // the unique name of the chart + title: service.name + ' Bind, Current Recursive Clients', // the title of the chart + units: 'clients', // the units of the chart dimensions + family: 'clients', // the family of the chart + context: 'named.recursive_clients', // the context of the chart + type: netdata.chartTypes.line, // the type of the chart + priority: named.base_priority + 1, // the priority relative to others in the same family + update_every: service.update_every, // the expected update frequency of the chart + dimensions: { + 'clients': { + id: 'clients', // the unique id of the dimension + name: '', // the name of the dimension + algorithm: netdata.chartAlgorithms.absolute,// the id of the netdata algorithm + multiplier: 1, // the multiplier + divisor: 1, // the divisor + hidden: false // is hidden (boolean) + } + } + }; + + chart = service.chart(id, chart); + named.charts[id] = chart; + } + + service.begin(chart); + service.set('clients', RecursClients); + service.end(); + } + } + + if(typeof r.opcodes !== 'undefined') + service.module.chartFromMembers(service, r.opcodes, 'in_opcodes', 'Bind, Global Incoming Requests by OpCode', 'requests/s', 'requests', 'named.in_opcodes', netdata.chartTypes.stacked, named.base_priority + 9, netdata.chartAlgorithms.incremental, 1, 1); + + if(typeof r.qtypes !== 'undefined') + service.module.chartFromMembers(service, r.qtypes, 'in_qtypes', 'Bind, Global Incoming Requests by Query Type', 'requests/s', 'requests', 'named.in_qtypes', netdata.chartTypes.stacked, named.base_priority + 10, netdata.chartAlgorithms.incremental, 1, 1); + + if(typeof r.sockstats !== 'undefined') + service.module.chartFromMembers(service, r.sockstats, 'in_sockstats', 'Bind, Global Socket Statistics', 'operations/s', 'sockets', 'named.in_sockstats', netdata.chartTypes.line, named.base_priority + 11, netdata.chartAlgorithms.incremental, 1, 1); + + if(typeof r.views !== 'undefined') { + for( var x in r.views ) { + var resolver = r.views[x].resolver; + + if(typeof resolver !== 'undefined') { + if(typeof resolver.stats !== 'undefined') { + var NumFetch = 0; + var key = service.name + '.' + x; + var default_enable = false; + var rtt = {}, rtt_enable = false; + + // NumFetch is an absolute value + if(typeof resolver.stats['NumFetch'] !== 'undefined') { + named.lookups.numfetch[key] = true; + NumFetch = resolver.stats['NumFetch']; + delete resolver.stats['NumFetch']; + } + if(typeof resolver.stats['BucketSize'] !== 'undefined') { + delete resolver.stats['BucketSize']; + } + + // split the QryRTT* from the main chart + for( var y in resolver.stats ) { + // we maintain an index of the values found + // mapping them to objects splitted + + var look = named.lookups.resolver_stats[y]; + if(typeof look === 'undefined') { + if(y.match(/^QryRTT/) !== null) { + named.lookups.resolver_stats[y] = { + name: y, + type: 'rtt' + }; + } + else { + named.lookups.resolver_stats[y] = { + name: y, + type: 'default' + }; + } + + look = named.lookups.resolver_stats[y]; + // netdata.error('lookup resolver stats value: ' + y + ' >>> ' + look.type); + } + + switch(look.type) { + case 'rtt': rtt[look.name] = resolver.stats[y]; delete resolver.stats[y]; rtt_enable = true; break; + default: default_enable = true; break; + } + } + + if(rtt_enable) + service.module.chartFromMembers(service, rtt, 'view_resolver_rtt_' + x, 'Bind, ' + x + ' View, Resolver Round Trip Timings', 'queries/s', 'view_' + x, 'named.resolver_rtt', netdata.chartTypes.stacked, named.base_priority + 12, netdata.chartAlgorithms.incremental, 1, 1); + + if(default_enable) + service.module.chartFromMembers(service, resolver.stats, 'view_resolver_stats_' + x, 'Bind, ' + x + ' View, Resolver Statistics', 'operations/s', 'view_' + x, 'named.resolver_stats', netdata.chartTypes.line, named.base_priority + 13, netdata.chartAlgorithms.incremental, 1, 1); + + // NumFetch chart + if(typeof named.lookups.numfetch[key] !== 'undefined') { + var id = 'named_' + service.name + '.view_resolver_numfetch_' + x; + var chart = named.charts[id]; + + if(typeof chart === 'undefined') { + chart = { + id: id, // the unique id of the chart + name: '', // the unique name of the chart + title: service.name + ' Bind, ' + x + ' View, Resolver Active Queries', // the title of the chart + units: 'queries', // the units of the chart dimensions + family: 'view_' + x, // the family of the chart + context: 'named.resolver_active_queries', // the context of the chart + type: netdata.chartTypes.line, // the type of the chart + priority: named.base_priority + 1001, // the priority relative to others in the same family + update_every: service.update_every, // the expected update frequency of the chart + dimensions: { + 'queries': { + id: 'queries', // the unique id of the dimension + name: '', // the name of the dimension + algorithm: netdata.chartAlgorithms.absolute,// the id of the netdata algorithm + multiplier: 1, // the multiplier + divisor: 1, // the divisor + hidden: false // is hidden (boolean) + } + } + }; + + chart = service.chart(id, chart); + named.charts[id] = chart; + } + + service.begin(chart); + service.set('queries', NumFetch); + service.end(); + } + } + } + + if(typeof resolver.qtypes !== 'undefined') + service.module.chartFromMembers(service, resolver.qtypes, 'view_resolver_qtypes_' + x, 'Bind, ' + x + ' View, Requests by Query Type', 'requests/s', 'view_' + x, 'named.resolver_qtypes', netdata.chartTypes.stacked, named.base_priority + 14, netdata.chartAlgorithms.incremental, 1, 1); + + //if(typeof resolver.cache !== 'undefined') + // service.module.chartFromMembers(service, resolver.cache, 'view_resolver_cache_' + x, 'Bind, ' + x + ' View, Cache Entries', 'entries', 'view_' + x, 'named.resolver_cache', netdata.chartTypes.stacked, named.base_priority + 15, netdata.chartAlgorithms.absolute, 1, 1); + + if(typeof resolver.cachestats['CacheHits'] !== 'undefined' && resolver.cachestats['CacheHits'] > 0) { + var id = 'named_' + service.name + '.view_resolver_cachehits_' + x; + var chart = named.charts[id]; + + if(typeof chart === 'undefined') { + chart = { + id: id, // the unique id of the chart + name: '', // the unique name of the chart + title: service.name + ' Bind, ' + x + ' View, Resolver Cache Hits', // the title of the chart + units: 'operations/s', // the units of the chart dimensions + family: 'view_' + x, // the family of the chart + context: 'named.resolver_cache_hits', // the context of the chart + type: netdata.chartTypes.area, // the type of the chart + priority: named.base_priority + 1100, // the priority relative to others in the same family + update_every: service.update_every, // the expected update frequency of the chart + dimensions: { + 'CacheHits': { + id: 'CacheHits', // the unique id of the dimension + name: 'hits', // the name of the dimension + algorithm: netdata.chartAlgorithms.incremental,// the id of the netdata algorithm + multiplier: 1, // the multiplier + divisor: 1, // the divisor + hidden: false // is hidden (boolean) + }, + 'CacheMisses': { + id: 'CacheMisses', // the unique id of the dimension + name: 'misses', // the name of the dimension + algorithm: netdata.chartAlgorithms.incremental,// the id of the netdata algorithm + multiplier: -1, // the multiplier + divisor: 1, // the divisor + hidden: false // is hidden (boolean) + } + } + }; + + chart = service.chart(id, chart); + named.charts[id] = chart; + } + + service.begin(chart); + service.set('CacheHits', resolver.cachestats['CacheHits']); + service.set('CacheMisses', resolver.cachestats['CacheMisses']); + service.end(); + } + + // this is wrong, it contains many types of info: + // 1. CacheHits, CacheMisses - incremental (added above) + // 2. QueryHits, QueryMisses - incremental + // 3. DeleteLRU, DeleteTTL - incremental + // 4. CacheNodes, CacheBuckets - absolute + // 5. TreeMemTotal, TreeMemInUse - absolute + // 6. HeapMemMax, HeapMemTotal, HeapMemInUse - absolute + //if(typeof resolver.cachestats !== 'undefined') + // service.module.chartFromMembers(service, resolver.cachestats, 'view_resolver_cachestats_' + x, 'Bind, ' + x + ' View, Cache Statistics', 'requests/s', 'view_' + x, 'named.resolver_cache_stats', netdata.chartTypes.line, named.base_priority + 1001, netdata.chartAlgorithms.incremental, 1, 1); + + //if(typeof resolver.adb !== 'undefined') + // service.module.chartFromMembers(service, resolver.adb, 'view_resolver_adb_' + x, 'Bind, ' + x + ' View, ADB Statistics', 'entries', 'view_' + x, 'named.resolver_adb', netdata.chartTypes.line, named.base_priority + 1002, netdata.chartAlgorithms.absolute, 1, 1); + } + } + } + }, + + // module.serviceExecute() + // this function is called only from this module + // its purpose is to prepare the request and call + // netdata.serviceExecute() + serviceExecute: function(name, a_url, update_every) { + if(netdata.options.DEBUG === true) netdata.debug(this.name + ': ' + name + ': url: ' + a_url + ', update_every: ' + update_every); + var service = netdata.service({ + name: name, + request: netdata.requestFromURL(a_url), + update_every: update_every, + module: this + }); + + service.execute(this.processResponse); + }, + + configure: function(config) { + var added = 0; + + if(this.enable_autodetect === true) { + this.serviceExecute('local', 'http://localhost:8888/json/v1/server', this.update_every); + added++; + } + + if(typeof(config.servers) !== 'undefined') { + var len = config.servers.length; + while(len--) { + if(typeof config.servers[len].update_every === 'undefined') + config.servers[len].update_every = this.update_every; + + this.serviceExecute(config.servers[len].name, config.servers[len].url, config.servers[len].update_every); + added++; + } + } + + return added; + }, + + // module.update() + // this is called repeatidly to collect data, by calling + // netdata.serviceExecute() + update: function(service, callback) { + service.execute(function(serv, data) { + service.module.processResponse(serv, data); + callback(); + }); + }, }; module.exports = named; diff --git a/node.d/sma_webbox.node.js b/node.d/sma_webbox.node.js index 5ed1c55a7..a6ce12052 100644 --- a/node.d/sma_webbox.node.js +++ b/node.d/sma_webbox.node.js @@ -6,20 +6,20 @@ // example configuration in /etc/netdata/sma_webbox.conf /* { - "enable_autodetect": false, - "update_every": 5, - "servers": [ - { - "name": "plant1", - "hostname": "10.0.1.1", - "update_every": 10 - }, - { - "name": "plant2", - "hostname": "10.0.2.1", - "update_every": 15 - } - ] + "enable_autodetect": false, + "update_every": 5, + "servers": [ + { + "name": "plant1", + "hostname": "10.0.1.1", + "update_every": 10 + }, + { + "name": "plant2", + "hostname": "10.0.2.1", + "update_every": 15 + } + ] } */ @@ -30,208 +30,208 @@ var netdata = require('netdata'); if(netdata.options.DEBUG === true) netdata.debug('loaded ' + __filename + ' plugin'); var webbox = { - name: __filename, - enable_autodetect: true, - update_every: 1, - base_priority: 60000, - charts: {}, - - processResponse: function(service, data) { - if(data !== null) { - var r = JSON.parse(data); - - var d = { - 'GriPwr': { - unit: null, - value: null - }, - 'GriEgyTdy': { - unit: null, - value: null - }, - 'GriEgyTot': { - unit: null, - value: null - } - }; - - // parse the webbox response - // and put it in our d object - var found = 0; - var len = r.result.overview.length; - while(len--) { - var e = r.result.overview[len]; - if(typeof(d[e.meta]) !== 'undefined') { - found++; - d[e.meta].value = e.value; - d[e.meta].unit = e.unit; - } - } - - // add the service - if(found > 0 && service.added !== true) - service.commit(); - - // Grid Current Power Chart - if(d['GriPwr'].value !== null) { - var id = 'sma_webbox_' + service.name + '.current'; - var chart = webbox.charts[id]; - - if(typeof chart === 'undefined') { - chart = { - id: id, // the unique id of the chart - name: '', // the unique name of the chart - title: service.name + ' Current Grid Power', // the title of the chart - units: d['GriPwr'].unit, // the units of the chart dimensions - family: 'now', // the family of the chart - context: 'sma_webbox.grid.power', // the context of the chart - type: netdata.chartTypes.area, // the type of the chart - priority: webbox.base_priority + 1, // the priority relative to others in the same family - update_every: service.update_every, // the expected update frequency of the chart - dimensions: { - 'GriPwr': { - id: 'GriPwr', // the unique id of the dimension - name: 'power', // the name of the dimension - algorithm: netdata.chartAlgorithms.absolute,// the id of the netdata algorithm - multiplier: 1, // the multiplier - divisor: 1, // the divisor - hidden: false // is hidden (boolean) - } - } - }; - - chart = service.chart(id, chart); - webbox.charts[id] = chart; - } - - service.begin(chart); - service.set('GriPwr', Math.round(d['GriPwr'].value)); - service.end(); - } - - if(d['GriEgyTdy'].value !== null) { - var id = 'sma_webbox_' + service.name + '.today'; - var chart = webbox.charts[id]; - - if(typeof chart === 'undefined') { - chart = { - id: id, // the unique id of the chart - name: '', // the unique name of the chart - title: service.name + ' Today Grid Power', // the title of the chart - units: d['GriEgyTdy'].unit, // the units of the chart dimensions - family: 'today', // the family of the chart - context: 'sma_webbox.grid.power.today', // the context of the chart - type: netdata.chartTypes.area, // the type of the chart - priority: webbox.base_priority + 2, // the priority relative to others in the same family - update_every: service.update_every, // the expected update frequency of the chart - dimensions: { - 'GriEgyTdy': { - id: 'GriEgyTdy', // the unique id of the dimension - name: 'power', // the name of the dimension - algorithm: netdata.chartAlgorithms.absolute,// the id of the netdata algorithm - multiplier: 1, // the multiplier - divisor: 1000, // the divisor - hidden: false // is hidden (boolean) - } - } - }; - - chart = service.chart(id, chart); - webbox.charts[id] = chart; - } - - service.begin(chart); - service.set('GriEgyTdy', Math.round(d['GriEgyTdy'].value * 1000)); - service.end(); - } - - if(d['GriEgyTot'].value !== null) { - var id = 'sma_webbox_' + service.name + '.total'; - var chart = webbox.charts[id]; - - if(typeof chart === 'undefined') { - chart = { - id: id, // the unique id of the chart - name: '', // the unique name of the chart - title: service.name + ' Total Grid Power', // the title of the chart - units: d['GriEgyTot'].unit, // the units of the chart dimensions - family: 'total', // the family of the chart - context: 'sma_webbox.grid.power.total', // the context of the chart - type: netdata.chartTypes.area, // the type of the chart - priority: webbox.base_priority + 3, // the priority relative to others in the same family - update_every: service.update_every, // the expected update frequency of the chart - dimensions: { - 'GriEgyTot': { - id: 'GriEgyTot', // the unique id of the dimension - name: 'power', // the name of the dimension - algorithm: netdata.chartAlgorithms.absolute,// the id of the netdata algorithm - multiplier: 1, // the multiplier - divisor: 1000, // the divisor - hidden: false // is hidden (boolean) - } - } - }; - - chart = service.chart(id, chart); - webbox.charts[id] = chart; - } - - service.begin(chart); - service.set('GriEgyTot', Math.round(d['GriEgyTot'].value * 1000)); - service.end(); - } - } - }, - - // module.serviceExecute() - // this function is called only from this module - // its purpose is to prepare the request and call - // netdata.serviceExecute() - serviceExecute: function(name, hostname, update_every) { - if(netdata.options.DEBUG === true) netdata.debug(this.name + ': ' + name + ': hostname: ' + hostname + ', update_every: ' + update_every); - - var service = netdata.service({ - name: name, - request: netdata.requestFromURL('http://' + hostname + '/rpc'), - update_every: update_every, - module: this - }); - service.postData = 'RPC={"proc":"GetPlantOverview","format":"JSON","version":"1.0","id":"1"}'; - service.request.method = 'POST'; - service.request.headers['Content-Length'] = service.postData.length; - - service.execute(this.processResponse); - }, - - configure: function(config) { - var added = 0; - - if(typeof(config.servers) !== 'undefined') { - var len = config.servers.length; - while(len--) { - if(typeof config.servers[len].update_every === 'undefined') - config.servers[len].update_every = this.update_every; - - if(config.servers[len].update_every < 5) - config.servers[len].update_every = 5; - - this.serviceExecute(config.servers[len].name, config.servers[len].hostname, config.servers[len].update_every); - added++; - } - } - - return added; - }, - - // module.update() - // this is called repeatidly to collect data, by calling - // netdata.serviceExecute() - update: function(service, callback) { - service.execute(function(serv, data) { - service.module.processResponse(serv, data); - callback(); - }); - }, + name: __filename, + enable_autodetect: true, + update_every: 1, + base_priority: 60000, + charts: {}, + + processResponse: function(service, data) { + if(data !== null) { + var r = JSON.parse(data); + + var d = { + 'GriPwr': { + unit: null, + value: null + }, + 'GriEgyTdy': { + unit: null, + value: null + }, + 'GriEgyTot': { + unit: null, + value: null + } + }; + + // parse the webbox response + // and put it in our d object + var found = 0; + var len = r.result.overview.length; + while(len--) { + var e = r.result.overview[len]; + if(typeof(d[e.meta]) !== 'undefined') { + found++; + d[e.meta].value = e.value; + d[e.meta].unit = e.unit; + } + } + + // add the service + if(found > 0 && service.added !== true) + service.commit(); + + // Grid Current Power Chart + if(d['GriPwr'].value !== null) { + var id = 'sma_webbox_' + service.name + '.current'; + var chart = webbox.charts[id]; + + if(typeof chart === 'undefined') { + chart = { + id: id, // the unique id of the chart + name: '', // the unique name of the chart + title: service.name + ' Current Grid Power', // the title of the chart + units: d['GriPwr'].unit, // the units of the chart dimensions + family: 'now', // the family of the chart + context: 'sma_webbox.grid.power', // the context of the chart + type: netdata.chartTypes.area, // the type of the chart + priority: webbox.base_priority + 1, // the priority relative to others in the same family + update_every: service.update_every, // the expected update frequency of the chart + dimensions: { + 'GriPwr': { + id: 'GriPwr', // the unique id of the dimension + name: 'power', // the name of the dimension + algorithm: netdata.chartAlgorithms.absolute,// the id of the netdata algorithm + multiplier: 1, // the multiplier + divisor: 1, // the divisor + hidden: false // is hidden (boolean) + } + } + }; + + chart = service.chart(id, chart); + webbox.charts[id] = chart; + } + + service.begin(chart); + service.set('GriPwr', Math.round(d['GriPwr'].value)); + service.end(); + } + + if(d['GriEgyTdy'].value !== null) { + var id = 'sma_webbox_' + service.name + '.today'; + var chart = webbox.charts[id]; + + if(typeof chart === 'undefined') { + chart = { + id: id, // the unique id of the chart + name: '', // the unique name of the chart + title: service.name + ' Today Grid Power', // the title of the chart + units: d['GriEgyTdy'].unit, // the units of the chart dimensions + family: 'today', // the family of the chart + context: 'sma_webbox.grid.power.today', // the context of the chart + type: netdata.chartTypes.area, // the type of the chart + priority: webbox.base_priority + 2, // the priority relative to others in the same family + update_every: service.update_every, // the expected update frequency of the chart + dimensions: { + 'GriEgyTdy': { + id: 'GriEgyTdy', // the unique id of the dimension + name: 'power', // the name of the dimension + algorithm: netdata.chartAlgorithms.absolute,// the id of the netdata algorithm + multiplier: 1, // the multiplier + divisor: 1000, // the divisor + hidden: false // is hidden (boolean) + } + } + }; + + chart = service.chart(id, chart); + webbox.charts[id] = chart; + } + + service.begin(chart); + service.set('GriEgyTdy', Math.round(d['GriEgyTdy'].value * 1000)); + service.end(); + } + + if(d['GriEgyTot'].value !== null) { + var id = 'sma_webbox_' + service.name + '.total'; + var chart = webbox.charts[id]; + + if(typeof chart === 'undefined') { + chart = { + id: id, // the unique id of the chart + name: '', // the unique name of the chart + title: service.name + ' Total Grid Power', // the title of the chart + units: d['GriEgyTot'].unit, // the units of the chart dimensions + family: 'total', // the family of the chart + context: 'sma_webbox.grid.power.total', // the context of the chart + type: netdata.chartTypes.area, // the type of the chart + priority: webbox.base_priority + 3, // the priority relative to others in the same family + update_every: service.update_every, // the expected update frequency of the chart + dimensions: { + 'GriEgyTot': { + id: 'GriEgyTot', // the unique id of the dimension + name: 'power', // the name of the dimension + algorithm: netdata.chartAlgorithms.absolute,// the id of the netdata algorithm + multiplier: 1, // the multiplier + divisor: 1000, // the divisor + hidden: false // is hidden (boolean) + } + } + }; + + chart = service.chart(id, chart); + webbox.charts[id] = chart; + } + + service.begin(chart); + service.set('GriEgyTot', Math.round(d['GriEgyTot'].value * 1000)); + service.end(); + } + } + }, + + // module.serviceExecute() + // this function is called only from this module + // its purpose is to prepare the request and call + // netdata.serviceExecute() + serviceExecute: function(name, hostname, update_every) { + if(netdata.options.DEBUG === true) netdata.debug(this.name + ': ' + name + ': hostname: ' + hostname + ', update_every: ' + update_every); + + var service = netdata.service({ + name: name, + request: netdata.requestFromURL('http://' + hostname + '/rpc'), + update_every: update_every, + module: this + }); + service.postData = 'RPC={"proc":"GetPlantOverview","format":"JSON","version":"1.0","id":"1"}'; + service.request.method = 'POST'; + service.request.headers['Content-Length'] = service.postData.length; + + service.execute(this.processResponse); + }, + + configure: function(config) { + var added = 0; + + if(typeof(config.servers) !== 'undefined') { + var len = config.servers.length; + while(len--) { + if(typeof config.servers[len].update_every === 'undefined') + config.servers[len].update_every = this.update_every; + + if(config.servers[len].update_every < 5) + config.servers[len].update_every = 5; + + this.serviceExecute(config.servers[len].name, config.servers[len].hostname, config.servers[len].update_every); + added++; + } + } + + return added; + }, + + // module.update() + // this is called repeatidly to collect data, by calling + // netdata.serviceExecute() + update: function(service, callback) { + service.execute(function(serv, data) { + service.module.processResponse(serv, data); + callback(); + }); + }, }; module.exports = webbox; diff --git a/node.d/snmp.node.js b/node.d/snmp.node.js index 21615f623..ddc898527 100644 --- a/node.d/snmp.node.js +++ b/node.d/snmp.node.js @@ -5,60 +5,60 @@ // example configuration in /etc/netdata/snmp.conf /* { - "enable_autodetect": false, - "update_every": 5, - "max_request_size": 50, - "servers": [ - { - "hostname": "10.11.12.8", - "community": "public", - "update_every": 10, - "max_request_size": 50, - "options": { "timeout": 10000 }, - "charts": { - "snmp_switch.bandwidth_port1": { - "title": "Switch Bandwidth for port 1", - "units": "kilobits/s", - "type": "area", - "priority": 1, - "dimensions": { - "in": { - "oid": ".1.3.6.1.2.1.2.2.1.10.1", - "algorithm": "incremental", - "multiplier": 8, - "divisor": 1024 - }, - "out": { - "oid": ".1.3.6.1.2.1.2.2.1.16.1", - "algorithm": "incremental", - "multiplier": -8, - "divisor": 1024 - } - } - }, - "snmp_switch.bandwidth_port2": { - "title": "Switch Bandwidth for port 2", - "units": "kilobits/s", - "type": "area", - "priority": 1, - "dimensions": { - "in": { - "oid": ".1.3.6.1.2.1.2.2.1.10.2", - "algorithm": "incremental", - "multiplier": 8, - "divisor": 1024 - }, - "out": { - "oid": ".1.3.6.1.2.1.2.2.1.16.2", - "algorithm": "incremental", - "multiplier": -8, - "divisor": 1024 - } - } - } - } - } - ] + "enable_autodetect": false, + "update_every": 5, + "max_request_size": 50, + "servers": [ + { + "hostname": "10.11.12.8", + "community": "public", + "update_every": 10, + "max_request_size": 50, + "options": { "timeout": 10000 }, + "charts": { + "snmp_switch.bandwidth_port1": { + "title": "Switch Bandwidth for port 1", + "units": "kilobits/s", + "type": "area", + "priority": 1, + "dimensions": { + "in": { + "oid": ".1.3.6.1.2.1.2.2.1.10.1", + "algorithm": "incremental", + "multiplier": 8, + "divisor": 1024 + }, + "out": { + "oid": ".1.3.6.1.2.1.2.2.1.16.1", + "algorithm": "incremental", + "multiplier": -8, + "divisor": 1024 + } + } + }, + "snmp_switch.bandwidth_port2": { + "title": "Switch Bandwidth for port 2", + "units": "kilobits/s", + "type": "area", + "priority": 1, + "dimensions": { + "in": { + "oid": ".1.3.6.1.2.1.2.2.1.10.2", + "algorithm": "incremental", + "multiplier": 8, + "divisor": 1024 + }, + "out": { + "oid": ".1.3.6.1.2.1.2.2.1.16.2", + "algorithm": "incremental", + "multiplier": -8, + "divisor": 1024 + } + } + } + } + } + ] } */ @@ -67,41 +67,41 @@ // so that 24 charts will be created. /* { - "enable_autodetect": false, - "update_every": 10, - "max_request_size": 50, - "servers": [ - { - "hostname": "10.11.12.8", - "community": "public", - "update_every": 10, - "max_request_size": 50, - "options": { "timeout": 20000 }, - "charts": { - "snmp_switch.bandwidth_port": { - "title": "Switch Bandwidth for port ", - "units": "kilobits/s", - "type": "area", - "priority": 1, - "multiply_range": [ 1, 24 ], - "dimensions": { - "in": { - "oid": ".1.3.6.1.2.1.2.2.1.10.", - "algorithm": "incremental", - "multiplier": 8, - "divisor": 1024 - }, - "out": { - "oid": ".1.3.6.1.2.1.2.2.1.16.", - "algorithm": "incremental", - "multiplier": -8, - "divisor": 1024 - } - } - } - } - } - ] + "enable_autodetect": false, + "update_every": 10, + "max_request_size": 50, + "servers": [ + { + "hostname": "10.11.12.8", + "community": "public", + "update_every": 10, + "max_request_size": 50, + "options": { "timeout": 20000 }, + "charts": { + "snmp_switch.bandwidth_port": { + "title": "Switch Bandwidth for port ", + "units": "kilobits/s", + "type": "area", + "priority": 1, + "multiply_range": [ 1, 24 ], + "dimensions": { + "in": { + "oid": ".1.3.6.1.2.1.2.2.1.10.", + "algorithm": "incremental", + "multiplier": 8, + "divisor": 1024 + }, + "out": { + "oid": ".1.3.6.1.2.1.2.2.1.16.", + "algorithm": "incremental", + "multiplier": -8, + "divisor": 1024 + } + } + } + } + } + ] } */ @@ -112,325 +112,325 @@ var netdata = require('netdata'); if(netdata.options.DEBUG === true) netdata.debug('loaded ' + __filename + ' plugin'); netdata.processors.snmp = { - name: 'snmp', - - fixoid: function(oid) { - if(typeof oid !== 'string') - return oid; - - if(oid.charAt(0) === '.') - return oid.substring(1, oid.length); - - return oid; - }, - - prepare: function(service) { - if(typeof service.snmp_oids === 'undefined' || service.snmp_oids === null || service.snmp_oids.length === 0) { - // this is the first time we see this service - - if(netdata.options.DEBUG === true) - netdata.debug(service.module.name + ': ' + service.name + ': preparing ' + this.name + ' OIDs'); - - // build an index of all OIDs - service.snmp_oids_index = {}; - for(var c in service.request.charts) { - // for each chart - - if(netdata.options.DEBUG === true) - netdata.debug(service.module.name + ': ' + service.name + ': indexing ' + this.name + ' chart: ' + c); - - if(typeof service.request.charts[c].titleoid !== 'undefined') { - service.snmp_oids_index[this.fixoid(service.request.charts[c].titleoid)] = { - type: 'title', - link: service.request.charts[c] - }; - } - - for(var d in service.request.charts[c].dimensions) { - // for each dimension in the chart - - var oid = this.fixoid(service.request.charts[c].dimensions[d].oid); - var oidname = this.fixoid(service.request.charts[c].dimensions[d].oidname); - - if(netdata.options.DEBUG === true) - netdata.debug(service.module.name + ': ' + service.name + ': indexing ' + this.name + ' chart: ' + c + ', dimension: ' + d + ', OID: ' + oid + ", OID name: " + oidname); - - // link it to the point we need to set the value to - service.snmp_oids_index[oid] = { - type: 'value', - link: service.request.charts[c].dimensions[d] - }; - - if(typeof oidname !== 'undefined') - service.snmp_oids_index[oidname] = { - type: 'name', - link: service.request.charts[c].dimensions[d] - }; - - // and set the value to null - service.request.charts[c].dimensions[d].value = null; - } - } - - if(netdata.options.DEBUG === true) - netdata.debug(service.module.name + ': ' + service.name + ': indexed ' + this.name + ' OIDs: ' + netdata.stringify(service.snmp_oids_index)); - - // now create the array of OIDs needed by net-snmp - service.snmp_oids = new Array(); - for(var o in service.snmp_oids_index) - service.snmp_oids.push(o); - - if(netdata.options.DEBUG === true) - netdata.debug(service.module.name + ': ' + service.name + ': final list of ' + this.name + ' OIDs: ' + netdata.stringify(service.snmp_oids)); - - service.snmp_oids_cleaned = 0; - } - else if(service.snmp_oids_cleaned === 0) { - service.snmp_oids_cleaned = 1; - - // the second time, keep only values - service.snmp_oids = new Array(); - for(var o in service.snmp_oids_index) - if(service.snmp_oids_index[o].type === 'value') - service.snmp_oids.push(o); - } - }, - - getdata: function(service, index, ok, failed, callback) { - var that = this; - - if(index >= service.snmp_oids.length) { - callback((ok > 0)?{ ok: ok, failed: failed }:null); - return; - } - - var slice; - if(service.snmp_oids.length <= service.request.max_request_size) { - slice = service.snmp_oids; - index = service.snmp_oids.length; - } - else if(service.snmp_oids.length - index <= service.request.max_request_size) { - slice = service.snmp_oids.slice(index, service.snmp_oids.length); - index = service.snmp_oids.length; - } - else { - slice = service.snmp_oids.slice(index, index + service.request.max_request_size); - index += service.request.max_request_size; - } - - if(netdata.options.DEBUG === true) - netdata.debug(service.module.name + ': ' + service.name + ': making ' + slice.length + ' entries request, max is: ' + service.request.max_request_size); - - service.snmp_session.get(slice, function(error, varbinds) { - if(error) { - service.error('Received error = ' + netdata.stringify(error) + ' varbinds = ' + netdata.stringify(varbinds)); - - // make all values null - var len = slice.length; - while(len--) - service.snmp_oids_index[slice[len]].value = null; - } - else { - if(netdata.options.DEBUG === true) - netdata.debug(service.module.name + ': ' + service.name + ': got valid ' + service.module.name + ' response: ' + netdata.stringify(varbinds)); - - for(var i = 0; i < varbinds.length; i++) { - var value = null; - - if(net_snmp.isVarbindError(varbinds[i])) { - if(netdata.options.DEBUG === true) - netdata.debug(service.module.name + ': ' + service.name + ': failed ' + service.module.name + ' get for OIDs ' + varbinds[i].oid); - - service.error('OID ' + varbinds[i].oid + ' gave error: ' + snmp.varbindError(varbinds[i])); - value = null; - failed++; - } - else { - if(netdata.options.DEBUG === true) - netdata.debug(service.module.name + ': ' + service.name + ': found ' + service.module.name + ' value of OIDs ' + varbinds[i].oid + " = " + varbinds[i].value); - - value = varbinds[i].value; - ok++; - } - - if(value !== null) { - switch(service.snmp_oids_index[varbinds[i].oid].type) { - case 'title': service.snmp_oids_index[varbinds[i].oid].link.title += ' ' + value; break; - case 'name' : service.snmp_oids_index[varbinds[i].oid].link.name = value; break; - case 'value': service.snmp_oids_index[varbinds[i].oid].link.value = value; break; - } - } - } - - if(netdata.options.DEBUG === true) - netdata.debug(service.module.name + ': ' + service.name + ': finished ' + service.module.name + ' with ' + ok + ' successful and ' + failed + ' failed values'); - } - that.getdata(service, index, ok, failed, callback); - }); - }, - - process: function(service, callback) { - this.prepare(service); - - if(service.snmp_oids.length === 0) { - // no OIDs found for this service - - if(netdata.options.DEBUG === true) - service.error('no OIDs to process.'); - - callback(null); - return; - } - - if(typeof service.snmp_session === 'undefined' || service.snmp_session === null) { - // no SNMP session has been created for this service - // the SNMP session is just the initialization of NET-SNMP - - if(netdata.options.DEBUG === true) - netdata.debug(service.module.name + ': ' + service.name + ': opening ' + this.name + ' session on ' + service.request.hostname + ' community ' + service.request.community + ' options ' + netdata.stringify(service.request.options)); - - // create the SNMP session - service.snmp_session = net_snmp.createSession (service.request.hostname, service.request.community, service.request.options); - - if(netdata.options.DEBUG === true) - netdata.debug(service.module.name + ': ' + service.name + ': got ' + this.name + ' session: ' + netdata.stringify(service.snmp_session)); - - // if we later need traps, this is how to do it: - //service.snmp_session.trap(net_snmp.TrapType.LinkDown, function(error) { - // if(error) console.error('trap error: ' + netdata.stringify(error)); - //}); - } - - // do it, get the SNMP values for the sessions we need - this.getdata(service, 0, 0, 0, callback); - } + name: 'snmp', + + fixoid: function(oid) { + if(typeof oid !== 'string') + return oid; + + if(oid.charAt(0) === '.') + return oid.substring(1, oid.length); + + return oid; + }, + + prepare: function(service) { + if(typeof service.snmp_oids === 'undefined' || service.snmp_oids === null || service.snmp_oids.length === 0) { + // this is the first time we see this service + + if(netdata.options.DEBUG === true) + netdata.debug(service.module.name + ': ' + service.name + ': preparing ' + this.name + ' OIDs'); + + // build an index of all OIDs + service.snmp_oids_index = {}; + for(var c in service.request.charts) { + // for each chart + + if(netdata.options.DEBUG === true) + netdata.debug(service.module.name + ': ' + service.name + ': indexing ' + this.name + ' chart: ' + c); + + if(typeof service.request.charts[c].titleoid !== 'undefined') { + service.snmp_oids_index[this.fixoid(service.request.charts[c].titleoid)] = { + type: 'title', + link: service.request.charts[c] + }; + } + + for(var d in service.request.charts[c].dimensions) { + // for each dimension in the chart + + var oid = this.fixoid(service.request.charts[c].dimensions[d].oid); + var oidname = this.fixoid(service.request.charts[c].dimensions[d].oidname); + + if(netdata.options.DEBUG === true) + netdata.debug(service.module.name + ': ' + service.name + ': indexing ' + this.name + ' chart: ' + c + ', dimension: ' + d + ', OID: ' + oid + ", OID name: " + oidname); + + // link it to the point we need to set the value to + service.snmp_oids_index[oid] = { + type: 'value', + link: service.request.charts[c].dimensions[d] + }; + + if(typeof oidname !== 'undefined') + service.snmp_oids_index[oidname] = { + type: 'name', + link: service.request.charts[c].dimensions[d] + }; + + // and set the value to null + service.request.charts[c].dimensions[d].value = null; + } + } + + if(netdata.options.DEBUG === true) + netdata.debug(service.module.name + ': ' + service.name + ': indexed ' + this.name + ' OIDs: ' + netdata.stringify(service.snmp_oids_index)); + + // now create the array of OIDs needed by net-snmp + service.snmp_oids = new Array(); + for(var o in service.snmp_oids_index) + service.snmp_oids.push(o); + + if(netdata.options.DEBUG === true) + netdata.debug(service.module.name + ': ' + service.name + ': final list of ' + this.name + ' OIDs: ' + netdata.stringify(service.snmp_oids)); + + service.snmp_oids_cleaned = 0; + } + else if(service.snmp_oids_cleaned === 0) { + service.snmp_oids_cleaned = 1; + + // the second time, keep only values + service.snmp_oids = new Array(); + for(var o in service.snmp_oids_index) + if(service.snmp_oids_index[o].type === 'value') + service.snmp_oids.push(o); + } + }, + + getdata: function(service, index, ok, failed, callback) { + var that = this; + + if(index >= service.snmp_oids.length) { + callback((ok > 0)?{ ok: ok, failed: failed }:null); + return; + } + + var slice; + if(service.snmp_oids.length <= service.request.max_request_size) { + slice = service.snmp_oids; + index = service.snmp_oids.length; + } + else if(service.snmp_oids.length - index <= service.request.max_request_size) { + slice = service.snmp_oids.slice(index, service.snmp_oids.length); + index = service.snmp_oids.length; + } + else { + slice = service.snmp_oids.slice(index, index + service.request.max_request_size); + index += service.request.max_request_size; + } + + if(netdata.options.DEBUG === true) + netdata.debug(service.module.name + ': ' + service.name + ': making ' + slice.length + ' entries request, max is: ' + service.request.max_request_size); + + service.snmp_session.get(slice, function(error, varbinds) { + if(error) { + service.error('Received error = ' + netdata.stringify(error) + ' varbinds = ' + netdata.stringify(varbinds)); + + // make all values null + var len = slice.length; + while(len--) + service.snmp_oids_index[slice[len]].value = null; + } + else { + if(netdata.options.DEBUG === true) + netdata.debug(service.module.name + ': ' + service.name + ': got valid ' + service.module.name + ' response: ' + netdata.stringify(varbinds)); + + for(var i = 0; i < varbinds.length; i++) { + var value = null; + + if(net_snmp.isVarbindError(varbinds[i])) { + if(netdata.options.DEBUG === true) + netdata.debug(service.module.name + ': ' + service.name + ': failed ' + service.module.name + ' get for OIDs ' + varbinds[i].oid); + + service.error('OID ' + varbinds[i].oid + ' gave error: ' + snmp.varbindError(varbinds[i])); + value = null; + failed++; + } + else { + if(netdata.options.DEBUG === true) + netdata.debug(service.module.name + ': ' + service.name + ': found ' + service.module.name + ' value of OIDs ' + varbinds[i].oid + " = " + varbinds[i].value); + + value = varbinds[i].value; + ok++; + } + + if(value !== null) { + switch(service.snmp_oids_index[varbinds[i].oid].type) { + case 'title': service.snmp_oids_index[varbinds[i].oid].link.title += ' ' + value; break; + case 'name' : service.snmp_oids_index[varbinds[i].oid].link.name = value; break; + case 'value': service.snmp_oids_index[varbinds[i].oid].link.value = value; break; + } + } + } + + if(netdata.options.DEBUG === true) + netdata.debug(service.module.name + ': ' + service.name + ': finished ' + service.module.name + ' with ' + ok + ' successful and ' + failed + ' failed values'); + } + that.getdata(service, index, ok, failed, callback); + }); + }, + + process: function(service, callback) { + this.prepare(service); + + if(service.snmp_oids.length === 0) { + // no OIDs found for this service + + if(netdata.options.DEBUG === true) + service.error('no OIDs to process.'); + + callback(null); + return; + } + + if(typeof service.snmp_session === 'undefined' || service.snmp_session === null) { + // no SNMP session has been created for this service + // the SNMP session is just the initialization of NET-SNMP + + if(netdata.options.DEBUG === true) + netdata.debug(service.module.name + ': ' + service.name + ': opening ' + this.name + ' session on ' + service.request.hostname + ' community ' + service.request.community + ' options ' + netdata.stringify(service.request.options)); + + // create the SNMP session + service.snmp_session = net_snmp.createSession (service.request.hostname, service.request.community, service.request.options); + + if(netdata.options.DEBUG === true) + netdata.debug(service.module.name + ': ' + service.name + ': got ' + this.name + ' session: ' + netdata.stringify(service.snmp_session)); + + // if we later need traps, this is how to do it: + //service.snmp_session.trap(net_snmp.TrapType.LinkDown, function(error) { + // if(error) console.error('trap error: ' + netdata.stringify(error)); + //}); + } + + // do it, get the SNMP values for the sessions we need + this.getdata(service, 0, 0, 0, callback); + } }; var snmp = { - name: __filename, - enable_autodetect: true, - update_every: 1, - base_priority: 50000, - - charts: {}, - - processResponse: function(service, data) { - if(data !== null) { - if(service.added !== true) - service.commit(); - - for(var c in service.request.charts) { - var chart = snmp.charts[c]; - - if(typeof chart === 'undefined') { - chart = service.chart(c, service.request.charts[c]); - snmp.charts[c] = chart; - } - - service.begin(chart); - - for( var d in service.request.charts[c].dimensions ) - if(service.request.charts[c].dimensions[d].value !== null) - service.set(d, service.request.charts[c].dimensions[d].value); - - service.end(); - } - } - }, - - // module.serviceExecute() - // this function is called only from this module - // its purpose is to prepare the request and call - // netdata.serviceExecute() - serviceExecute: function(conf) { - if(netdata.options.DEBUG === true) - netdata.debug(this.name + ': snmp hostname: ' + conf.hostname + ', update_every: ' + conf.update_every); - - var service = netdata.service({ - name: conf.hostname, - request: conf, - update_every: conf.update_every, - module: this, - processor: netdata.processors.snmp - }); - - // multiply the charts, if required - for(var c in service.request.charts) { - if(netdata.options.DEBUG === true) - netdata.debug(this.name + ': snmp hostname: ' + conf.hostname + ', examining chart: ' + c); - - if(typeof service.request.charts[c].update_every === 'undefined') - service.request.charts[c].update_every = service.update_every; - - if(typeof service.request.charts[c].multiply_range !== 'undefined') { - var from = service.request.charts[c].multiply_range[0]; - var to = service.request.charts[c].multiply_range[1]; - var prio = service.request.charts[c].priority || 1; - - if(prio < snmp.base_priority) prio += snmp.base_priority; - - while(from <= to) { - var id = c + from.toString(); - var chart = extend(true, {}, service.request.charts[c]); - chart.title += from.toString(); - - if(typeof chart.titleoid !== 'undefined') - chart.titleoid += from.toString(); - - chart.priority = prio++; - for(var d in chart.dimensions) { - chart.dimensions[d].oid += from.toString(); - - if(typeof chart.dimensions[d].oidname !== 'undefined') - chart.dimensions[d].oidname += from.toString(); - } - service.request.charts[id] = chart; - from++; - } - - delete service.request.charts[c]; - } - else { - if(service.request.charts[c].priority < snmp.base_priority) - service.request.charts[c].priority += snmp.base_priority; - } - } - - service.execute(this.processResponse); - }, - - configure: function(config) { - var added = 0; - - if(typeof config.max_request_size === 'undefined') - config.max_request_size = 50; - - if(typeof(config.servers) !== 'undefined') { - var len = config.servers.length; - while(len--) { - if(typeof config.servers[len].update_every === 'undefined') - config.servers[len].update_every = this.update_every; - - if(typeof config.servers[len].max_request_size === 'undefined') - config.servers[len].max_request_size = config.max_request_size; - - this.serviceExecute(config.servers[len]); - added++; - } - } - - return added; - }, - - // module.update() - // this is called repeatidly to collect data, by calling - // service.execute() - update: function(service, callback) { - service.execute(function(serv, data) { - service.module.processResponse(serv, data); - callback(); - }); - }, + name: __filename, + enable_autodetect: true, + update_every: 1, + base_priority: 50000, + + charts: {}, + + processResponse: function(service, data) { + if(data !== null) { + if(service.added !== true) + service.commit(); + + for(var c in service.request.charts) { + var chart = snmp.charts[c]; + + if(typeof chart === 'undefined') { + chart = service.chart(c, service.request.charts[c]); + snmp.charts[c] = chart; + } + + service.begin(chart); + + for( var d in service.request.charts[c].dimensions ) + if(service.request.charts[c].dimensions[d].value !== null) + service.set(d, service.request.charts[c].dimensions[d].value); + + service.end(); + } + } + }, + + // module.serviceExecute() + // this function is called only from this module + // its purpose is to prepare the request and call + // netdata.serviceExecute() + serviceExecute: function(conf) { + if(netdata.options.DEBUG === true) + netdata.debug(this.name + ': snmp hostname: ' + conf.hostname + ', update_every: ' + conf.update_every); + + var service = netdata.service({ + name: conf.hostname, + request: conf, + update_every: conf.update_every, + module: this, + processor: netdata.processors.snmp + }); + + // multiply the charts, if required + for(var c in service.request.charts) { + if(netdata.options.DEBUG === true) + netdata.debug(this.name + ': snmp hostname: ' + conf.hostname + ', examining chart: ' + c); + + if(typeof service.request.charts[c].update_every === 'undefined') + service.request.charts[c].update_every = service.update_every; + + if(typeof service.request.charts[c].multiply_range !== 'undefined') { + var from = service.request.charts[c].multiply_range[0]; + var to = service.request.charts[c].multiply_range[1]; + var prio = service.request.charts[c].priority || 1; + + if(prio < snmp.base_priority) prio += snmp.base_priority; + + while(from <= to) { + var id = c + from.toString(); + var chart = extend(true, {}, service.request.charts[c]); + chart.title += from.toString(); + + if(typeof chart.titleoid !== 'undefined') + chart.titleoid += from.toString(); + + chart.priority = prio++; + for(var d in chart.dimensions) { + chart.dimensions[d].oid += from.toString(); + + if(typeof chart.dimensions[d].oidname !== 'undefined') + chart.dimensions[d].oidname += from.toString(); + } + service.request.charts[id] = chart; + from++; + } + + delete service.request.charts[c]; + } + else { + if(service.request.charts[c].priority < snmp.base_priority) + service.request.charts[c].priority += snmp.base_priority; + } + } + + service.execute(this.processResponse); + }, + + configure: function(config) { + var added = 0; + + if(typeof config.max_request_size === 'undefined') + config.max_request_size = 50; + + if(typeof(config.servers) !== 'undefined') { + var len = config.servers.length; + while(len--) { + if(typeof config.servers[len].update_every === 'undefined') + config.servers[len].update_every = this.update_every; + + if(typeof config.servers[len].max_request_size === 'undefined') + config.servers[len].max_request_size = config.max_request_size; + + this.serviceExecute(config.servers[len]); + added++; + } + } + + return added; + }, + + // module.update() + // this is called repeatidly to collect data, by calling + // service.execute() + update: function(service, callback) { + service.execute(function(serv, data) { + service.module.processResponse(serv, data); + callback(); + }); + }, }; module.exports = snmp; diff --git a/plugins.d/Makefile.am b/plugins.d/Makefile.am index a717cbed1..b8a28610a 100644 --- a/plugins.d/Makefile.am +++ b/plugins.d/Makefile.am @@ -8,10 +8,12 @@ dist_plugins_DATA = \ $(NULL) dist_plugins_SCRIPTS = \ + alarm-email.sh \ cgroup-name.sh \ charts.d.dryrun-helper.sh \ charts.d.plugin \ node.d.plugin \ + python.d.plugin \ tc-qos-helper.sh \ loopsleepms.sh.inc \ $(NULL) diff --git a/plugins.d/Makefile.in b/plugins.d/Makefile.in index c74997688..06211d51c 100644 --- a/plugins.d/Makefile.in +++ b/plugins.d/Makefile.in @@ -263,6 +263,8 @@ pluginsdir = @pluginsdir@ prefix = @prefix@ program_transform_name = @program_transform_name@ psdir = @psdir@ +pythondir = @pythondir@ +registrydir = @registrydir@ sbindir = @sbindir@ sharedstatedir = @sharedstatedir@ srcdir = @srcdir@ @@ -283,10 +285,12 @@ dist_plugins_DATA = \ $(NULL) dist_plugins_SCRIPTS = \ + alarm-email.sh \ cgroup-name.sh \ charts.d.dryrun-helper.sh \ charts.d.plugin \ node.d.plugin \ + python.d.plugin \ tc-qos-helper.sh \ loopsleepms.sh.inc \ $(NULL) diff --git a/plugins.d/alarm-email.sh b/plugins.d/alarm-email.sh new file mode 100755 index 000000000..78c79ccdb --- /dev/null +++ b/plugins.d/alarm-email.sh @@ -0,0 +1,264 @@ +#!/usr/bin/env bash + +me="${0}" + +sendmail="$(which sendmail 2>/dev/null || command -v sendmail 2>/dev/null)" +if [ -z "${sendmail}" ] +then + echo >&2 "I cannot send emails - there is no sendmail command available." +fi + +sendmail_from_pipe() { + "${sendmail}" -t + + if [ $? -eq 0 ] + then + echo >&2 "${me}: Sent notification email for ${status} on '${chart}.${name}'" + return 0 + else + echo >&2 "${me}: FAILED to send notification email for ${status} on '${chart}.${name}'" + return 1 + fi +} + +name="${1}" # the name of the alarm, as given in netdata health.d entries +chart="${2}" # the name of the chart (type.id) +family="${3}" # the family of the chart +status="${4}" # the current status : UNITIALIZED, UNDEFINED, CLEAR, WARNING, CRITICAL +old_status="${5}" # the previous status: UNITIALIZED, UNDEFINED, CLEAR, WARNING, CRITICAL +value="${6}" # the current value +old_value="${7}" # the previous value +src="${8}" # the line number and file the alarm has been configured +duration="${9}" # the duration in seconds the previous state took +non_clear_duration="${10}" # the total duration in seconds this is non-clear +units="${11}" # the units of the value +info="${12}" # a short description of the alarm + +[ ! -z "${info}" ] && info=" <small><br/>${info}</small>" + +# get the system hostname +hostname="${NETDATA_HOSTNAME}" +[ -z "${hostname}" ] && hostname="${NETDATA_REGISTRY_HOSTNAME}" +[ -z "${hostname}" ] && hostname="$(hostname)" + +goto_url="${NETDATA_REGISTRY_URL}/goto-host-from-alarm.html?machine_guid=${NETDATA_REGISTRY_UNIQUE_ID}&chart=${chart}&family=${family}" + +# get the current date +date="$(date)" + +duration4human() { + local s="${1}" d=0 h=0 m=0 ds="day" hs="hour" ms="minute" ss="second" + d=$(( s / 86400 )) + s=$(( s - (d * 86400) )) + h=$(( s / 3600 )) + s=$(( s - (h * 3600) )) + m=$(( s / 60 )) + s=$(( s - (m * 60) )) + + if [ ${d} -gt 0 ] + then + [ ${m} -ge 30 ] && h=$(( h + 1 )) + [ ${d} -gt 1 ] && ds="days" + [ ${h} -gt 1 ] && hs="hours" + if [ ${h} -gt 0 ] + then + echo "${d} ${ds} and ${h} ${hs}" + else + echo "${d} ${ds}" + fi + elif [ ${h} -gt 0 ] + then + [ ${s} -ge 30 ] && m=$(( m + 1 )) + [ ${h} -gt 1 ] && hs="hours" + [ ${m} -gt 1 ] && ms="minutes" + if [ ${m} -gt 0 ] + then + echo "${h} ${hs} and ${m} ${ms}" + else + echo "${h} ${hs}" + fi + elif [ ${m} -gt 0 ] + then + [ ${m} -gt 1 ] && ms="minutes" + [ ${s} -gt 1 ] && ss="seconds" + if [ ${s} -gt 0 ] + then + echo "${m} ${ms} and ${s} ${ss}" + else + echo "${m} ${ms}" + fi + else + [ ${s} -gt 1 ] && ss="seconds" + echo "${s} ${ss}" + fi +} + +severity="${status}" +raised_for="<br/>(was ${old_status,,} for $(duration4human ${duration}))" +status_message="status unknown" +color="grey" +alarm="${name} = ${value} ${units}" + +# prepare the title based on status +case "${status}" in + CRITICAL) + status_message="is critical" + color="#ca414b" + ;; + + WARNING) + status_message="needs attention" + color="#caca4b" + ;; + + CLEAR) + status_message="recovered" + color="#77ca6d" + + # don't show the value when the status is CLEAR + # for certain alarms, this value might not have any meaning + alarm="${name}" + ;; +esac + +if [ "${status}" != "WARNING" -a "${status}" != "CRITICAL" -a "${status}" != "CLEAR" ] +then + # don't do anything if this is not WARNING, CRITICAL or CLEAR + echo >&2 "${me}: not sending notification email for ${status} on '${chart}.${name}'" + exit 0 +elif [ "${old_status}" != "WARNING" -a "${old_status}" != "CRITICAL" -a "${status}" = "CLEAR" ] +then + # don't do anything if this is CLEAR, but it was not WARNING or CRITICAL + echo >&2 "${me}: not sending notification email for ${status} on '${chart}.${name}' (last status was ${old_status})" + exit 0 +elif [ "${status}" = "CLEAR" ] +then + severity="Recovered from ${old_status}" + if [ $non_clear_duration -gt $duration ] + then + raised_for="<br/>(had issues for $(duration4human ${non_clear_duration}))" + fi + +elif [ "${old_status}" = "WARNING" -a "${status}" = "CRITICAL" ] +then + severity="Escalated to ${status}" + if [ $non_clear_duration -gt $duration ] + then + raised_for="<br/>(has issues for $(duration4human ${non_clear_duration}))" + fi + +elif [ "${old_status}" = "CRITICAL" -a "${status}" = "WARNING" ] +then + severity="Demoted to ${status}" + if [ $non_clear_duration -gt $duration ] + then + raised_for="<br/>(has issues for $(duration4human ${non_clear_duration}))" + fi + +else + raised_for= +fi + +# send the email +cat <<EOF | sendmail_from_pipe +To: root +Subject: ${hostname} ${status_message} - ${chart}.${name} +Content-Type: text/html + +<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> +<html xmlns="http://www.w3.org/1999/xhtml" style="font-family: 'Helvetica Neue', 'Helvetica', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0; padding: 0;"> +<body style="font-family:'Helvetica Neue','Helvetica',Helvetica,Arial,sans-serif;font-size:14px;width:100%!important;min-height:100%;line-height:1.6;background:#f6f6f6;margin:0;padding:0"> +<table> + <tbody> + <tr> + <td style="vertical-align:top;" valign="top"></td> + <td width="700" style="vertical-align:top;display:block!important;max-width:700px!important;clear:both!important;margin:0 auto;padding:0" valign="top"> + <div style="max-width:700px;display:block;margin:0 auto;padding:20px"> + <table width="100%" cellpadding="0" cellspacing="0" + style="background:#fff;border:1px solid #e9e9e9"> + <tbody> + <tr> + <td bgcolor="#eee" + style="padding: 5px 20px 5px 20px;background-color:#eee;"> + <div style="font-size:20px;color:#777;font-weight: bold;">netdata notification</div> + </td> + </tr> + <tr> + <td bgcolor="${color}" + style="font-size:16px;vertical-align:top;font-weight:400;text-align:center;margin:0;padding:10px;color:#ffffff;background:${color}!important;border:1px solid ${color};border-top-color:${color}" align="center" valign="top"> + <h1 style="font-weight:400;margin:0">${hostname} ${status_message}</h1> + </td> + </tr> + <tr> + <td style="vertical-align:top" valign="top"> + <div style="margin:0;padding:20px;max-width:700px"> + <table width="100%" cellpadding="0" cellspacing="0" style="max-width:700px"> + <tbody> + <tr> + <td style="font-size:18px;vertical-align:top;margin:0;padding:0 0 20px" + align="left" valign="top"> + <span>${chart}</span> + <span style="display:block;color:#666666;font-size:12px;font-weight:300;line-height:1;text-transform:uppercase">Chart</span> + </td> + </tr> + <tr style="margin:0;padding:0"> + <td style="font-size:18px;vertical-align:top;margin:0;padding:0 0 20px" + align="left" valign="top"> + <span><b>${alarm}</b>${info}</span> + <span style="display:block;color:#666666;font-size:12px;font-weight:300;line-height:1;text-transform:uppercase">Alarm</span> + </td> + </tr> + <tr> + <td style="font-size:18px;vertical-align:top;margin:0;padding:0 0 20px" + align="left" valign="top"> + <span>${family}</span> + <span style="display:block;color:#666666;font-size:12px;font-weight:300;line-height:1;text-transform:uppercase">Family</span> + </td> + </tr> + <tr style="margin:0;padding:0"> + <td style="font-size:18px;vertical-align:top;margin:0;padding:0 0 20px" + align="left" valign="top"> + <span>${severity}</span> + <span style="display:block;color:#666666;font-size:12px;font-weight:300;line-height:1;text-transform:uppercase">Severity</span> + </td> + </tr> + <tr style="margin:0;padding:0"> + <td style="font-size:18px;vertical-align:top;margin:0;padding:0 0 20px" + align="left" valign="top"><span>${date}</span> + <span>${raised_for}</span> <span + style="display:block;color:#666666;font-size:12px;font-weight:300;line-height:1;text-transform:uppercase">Time</span> + </td> + </tr> + <!-- + <tr style="margin:0;padding:0"> + <td style="font-size:18px;vertical-align:top;margin:0;padding:0 0 20px"> + <a href="${goto_url}" style="font-size:14px;color:#ffffff;text-decoration:none;line-height:1.5;font-weight:bold;text-align:center;display:inline-block;text-transform:capitalize;background:#35568d;border-width:1px;border-style:solid;border-color:#2b4c86;margin:0;padding:10px 15px" target="_blank">View Netdata</a> + </td> + </tr> + --> + <tr style="text-align:center;margin:0;padding:0"> + <td style="font-size:11px;vertical-align:top;margin:0;padding:10px 0 0 0;color:#666666" + align="center" valign="bottom">The source of this alarm is line <code>${src}</code> + </td> + </tr> + <tr style="text-align:center;margin:0;padding:0"> + <td style="font-size:12px;vertical-align:top;margin:0;padding:20px 0 0 0;color:#666666;border-top:1px solid #f0f0f0" + align="center" valign="bottom">Sent by + <a href="https://mynetdata.io/" target="_blank">netdata</a>, the real-time performance monitoring. + </td> + </tr> + </tbody> + </table> + </div> + </td> + </tr> + </tbody> + </table> + </div> + </td> + </tr> + </tbody> +</table> +</body> +</html> +EOF diff --git a/plugins.d/cgroup-name.sh b/plugins.d/cgroup-name.sh index 8ce64b3d7..8bfc984c2 100755 --- a/plugins.d/cgroup-name.sh +++ b/plugins.d/cgroup-name.sh @@ -1,4 +1,4 @@ -#!/bin/bash +#!/usr/bin/env bash export PATH="${PATH}:/sbin:/usr/sbin:/usr/local/sbin" export LC_ALL=C @@ -9,66 +9,69 @@ CGROUP="${1}" NAME= if [ -z "${CGROUP}" ] - then - echo >&2 "${0}: called without a cgroup name. Nothing to do." - exit 1 + then + echo >&2 "${0}: called without a cgroup name. Nothing to do." + exit 1 fi if [ -f "${CONFIG}" ] - then - NAME="$(cat "${CONFIG}" | grep "^${CGROUP} " | sed "s/[[:space:]]\+/ /g" | cut -d ' ' -f 2)" - if [ -z "${NAME}" ] - then - echo >&2 "${0}: cannot find cgroup '${CGROUP}' in '${CONFIG}'." - fi + then + NAME="$(grep "^${CGROUP} " "${CONFIG}" | sed "s/[[:space:]]\+/ /g" | cut -d ' ' -f 2)" + if [ -z "${NAME}" ] + then + echo >&2 "${0}: cannot find cgroup '${CGROUP}' in '${CONFIG}'." + fi #else -# echo >&2 "${0}: configuration file '${CONFIG}' is not available." +# echo >&2 "${0}: configuration file '${CONFIG}' is not available." fi function get_name_classic { - DOCKERID=$1 - echo >&2 "Running command: docker ps --filter=id=\"${DOCKERID}\" --format=\"{{.Names}}\"" - NAME="$( docker ps --filter=id="${DOCKERID}" --format="{{.Names}}" )" + local DOCKERID="$1" + echo >&2 "Running command: docker ps --filter=id=\"${DOCKERID}\" --format=\"{{.Names}}\"" + NAME="$( docker ps --filter=id="${DOCKERID}" --format="{{.Names}}" )" + return 0 } function get_name_api { - DOCKERID=$1 - if [ ! -S "/var/run/docker.sock" ] - then - echo >&2 "Can't find /var/run/docker.sock" - return - fi - echo >&2 "Running API command: /containers/${DOCKERID}/json" - JSON=$(echo -e "GET /containers/${DOCKERID}/json HTTP/1.0\r\n" | nc -U /var/run/docker.sock | egrep '^{.*') - NAME=$(echo $JSON | jq -r .Name,.Config.Hostname | grep -v null | head -n1 | sed 's|^/||') + local DOCKERID="$1" + if [ ! -S "/var/run/docker.sock" ] + then + echo >&2 "Can't find /var/run/docker.sock" + return 1 + fi + echo >&2 "Running API command: /containers/${DOCKERID}/json" + JSON=$(echo -e "GET /containers/${DOCKERID}/json HTTP/1.0\r\n" | nc -U /var/run/docker.sock | egrep '^{.*') + NAME=$(echo $JSON | jq -r .Name,.Config.Hostname | grep -v null | head -n1 | sed 's|^/||') + return 0 } if [ -z "${NAME}" ] - then - if [[ "${CGROUP}" =~ ^.*docker[-/\.][a-fA-F0-9]+[-\.]?.*$ ]] - then - DOCKERID="$( echo "${CGROUP}" | sed "s|^.*docker[-/]\([a-fA-F0-9]\+\)[-\.]\?.*$|\1|" )" + then + if [[ "${CGROUP}" =~ ^.*docker[-_/\.][a-fA-F0-9]+[-_\.]?.*$ ]] + then + DOCKERID="$( echo "${CGROUP}" | sed "s|^.*docker[-_/]\([a-fA-F0-9]\+\)[-_\.]\?.*$|\1|" )" + # echo "DOCKERID=${DOCKERID}" - if [ ! -z "${DOCKERID}" -a \( ${#DOCKERID} -eq 64 -o ${#DOCKERID} -eq 12 \) ] - then - if hash docker 2>/dev/null - then - get_name_classic $DOCKERID - else - get_name_api $DOCKERID - fi - if [ -z "${NAME}" ] - then - echo >&2 "Cannot find the name of docker container '${DOCKERID}'" - NAME="${DOCKERID:0:12}" - else - echo >&2 "Docker container '${DOCKERID}' is named '${NAME}'" - fi - fi - fi + if [ ! -z "${DOCKERID}" -a \( ${#DOCKERID} -eq 64 -o ${#DOCKERID} -eq 12 \) ] + then + if hash docker 2>/dev/null + then + get_name_classic $DOCKERID + else + get_name_api $DOCKERID || get_name_classic $DOCKERID + fi + if [ -z "${NAME}" ] + then + echo >&2 "Cannot find the name of docker container '${DOCKERID}'" + NAME="${DOCKERID:0:12}" + else + echo >&2 "Docker container '${DOCKERID}' is named '${NAME}'" + fi + fi + fi - [ -z "${NAME}" ] && NAME="${CGROUP}" - [ ${#NAME} -gt 50 ] && NAME="${NAME:0:50}" + [ -z "${NAME}" ] && NAME="${CGROUP}" + [ ${#NAME} -gt 50 ] && NAME="${NAME:0:50}" fi echo >&2 "${0}: cgroup '${CGROUP}' is called '${NAME}'" diff --git a/plugins.d/charts.d.dryrun-helper.sh b/plugins.d/charts.d.dryrun-helper.sh index 646452892..8142f9882 100755 --- a/plugins.d/charts.d.dryrun-helper.sh +++ b/plugins.d/charts.d.dryrun-helper.sh @@ -1,4 +1,4 @@ -#!/bin/sh +#!/usr/bin/env bash # will stop the script for any error set -e @@ -14,7 +14,7 @@ tmp1="`mktemp`" tmp2="`mktemp`" myset() { - set | grep -v "^_=" | grep -v "^PIPESTATUS=" | grep -v "^BASH_LINENO=" + set | grep -v "^_=" | grep -v "^PIPESTATUS=" | grep -v "^BASH_LINENO=" } # save 2 'set' @@ -25,9 +25,9 @@ myset >"$tmp2" diff "$tmp1" "$tmp2" >/dev/null 2>&1 if [ $? -ne 0 ] then - # they differ, we cannot do the check - echo >&2 "$me: cannot check with diff." - can_diff=0 + # they differ, we cannot do the check + echo >&2 "$me: cannot check with diff." + can_diff=0 fi # do it again, now including the script @@ -36,21 +36,21 @@ myset >"$tmp1" # include the plugin and its config if [ -f "$conf" ] then - . "$conf" - if [ $? -ne 0 ] - then - echo >&2 "$me: cannot load config file $conf" - rm "$tmp1" "$tmp2" - exit 1 - fi + . "$conf" + if [ $? -ne 0 ] + then + echo >&2 "$me: cannot load config file $conf" + rm "$tmp1" "$tmp2" + exit 1 + fi fi . "$chart" if [ $? -ne 0 ] then - echo >&2 "$me: cannot load chart file $chart" - rm "$tmp1" "$tmp2" - exit 1 + echo >&2 "$me: cannot load chart file $chart" + rm "$tmp1" "$tmp2" + exit 1 fi # remove all variables starting with the plugin name @@ -58,15 +58,15 @@ myset | grep -v "^$name" >"$tmp2" if [ $can_diff -eq 1 ] then - # check if they are different - # make sure they don't differ - diff "$tmp1" "$tmp2" >&2 - if [ $? -ne 0 ] - then - # they differ - rm "$tmp1" "$tmp2" - exit 1 - fi + # check if they are different + # make sure they don't differ + diff "$tmp1" "$tmp2" >&2 + if [ $? -ne 0 ] + then + # they differ + rm "$tmp1" "$tmp2" + exit 1 + fi fi rm "$tmp1" "$tmp2" diff --git a/plugins.d/charts.d.plugin b/plugins.d/charts.d.plugin index 2824fa3c6..9aaadc168 100755 --- a/plugins.d/charts.d.plugin +++ b/plugins.d/charts.d.plugin @@ -1,4 +1,4 @@ -#!/bin/bash +#!/usr/bin/env bash PROGRAM_FILE="$0" PROGRAM_NAME="$(basename $0)" @@ -14,24 +14,24 @@ echo >&2 "$PROGRAM_NAME: started from '$PROGRAM_FILE' with options: $*" if [ $(( ${BASH_VERSINFO[0]} )) -lt 4 ] then - echo >&2 - echo >&2 "$PROGRAM_NAME: ERROR" - echo >&2 "BASH version 4 or later is required." - echo >&2 "You are running version: ${BASH_VERSION}" - echo >&2 "Please upgrade." - echo >&2 - exit 1 + echo >&2 + echo >&2 "$PROGRAM_NAME: ERROR" + echo >&2 "BASH version 4 or later is required." + echo >&2 "You are running version: ${BASH_VERSION}" + echo >&2 "Please upgrade." + echo >&2 + exit 1 fi # check a few commands require_cmd() { - which "$1" >/dev/null - if [ $? -ne 0 ] - then - echo >&2 "$PROGRAM_NAME: ERROR: Command '$1' is not found in the system path." - return 1 - fi - return 0 + which "$1" >/dev/null + if [ $? -ne 0 ] + then + echo >&2 "$PROGRAM_NAME: ERROR: Command '$1' is not found in the system path." + return 1 + fi + return 0 } require_cmd date || exit 1 @@ -61,7 +61,7 @@ chartsd="$pluginsd/../charts.d" myconfig="$confd/$PROGRAM_NAME.conf" minimum_update_frequency="${NETDATA_UPDATE_EVERY-1}" -update_every=${minimum_update_frequency} # this will be overwritten by the command line +update_every=${minimum_update_frequency} # this will be overwritten by the command line # work around for non BASH shells charts_create="_create" @@ -106,50 +106,50 @@ check=0 chart_only= while [ ! -z "$1" ] do - if [ "$1" = "check" ] - then - check=1 - shift - continue - fi - - if [ "$1" = "debug" -o "$1" = "all" ] - then - debug=1 - shift - continue - fi - - if [ -f "$chartsd/$1.chart.sh" ] - then - debug=1 - chart_only="$( echo $1.chart.sh | sed "s/\.chart\.sh$//g" )" - shift - continue - fi - - if [ -f "$chartsd/$1" ] - then - debug=1 - chart_only="$( echo $1 | sed "s/\.chart\.sh$//g" )" - shift - continue - fi - - # number check - n="$1" - x=$(( n )) - if [ "$x" = "$n" ] - then - shift - update_every=$x - [ $update_every -lt $minimum_update_frequency ] && update_every=$minimum_update_frequency - continue - fi - - echo >&2 "Cannot understand parameter $1. Aborting." - echo "DISABLE" - exit 1 + if [ "$1" = "check" ] + then + check=1 + shift + continue + fi + + if [ "$1" = "debug" -o "$1" = "all" ] + then + debug=1 + shift + continue + fi + + if [ -f "$chartsd/$1.chart.sh" ] + then + debug=1 + chart_only="$( echo $1.chart.sh | sed "s/\.chart\.sh$//g" )" + shift + continue + fi + + if [ -f "$chartsd/$1" ] + then + debug=1 + chart_only="$( echo $1 | sed "s/\.chart\.sh$//g" )" + shift + continue + fi + + # number check + n="$1" + x=$(( n )) + if [ "$x" = "$n" ] + then + shift + update_every=$x + [ $update_every -lt $minimum_update_frequency ] && update_every=$minimum_update_frequency + continue + fi + + echo >&2 "Cannot understand parameter $1. Aborting." + echo "DISABLE" + exit 1 done @@ -157,26 +157,26 @@ done # load my configuration if [ -f "$myconfig" ] - then - . "$myconfig" - if [ $? -ne 0 ] - then - echo >&2 "$PROGRAM_NAME: cannot load $myconfig" - echo "DISABLE" - exit 1 - fi - time_divisor=$((time_divisor)) - [ $time_divisor -lt 10 ] && time_divisor=10 - [ $time_divisor -gt 100 ] && time_divisor=100 + then + . "$myconfig" + if [ $? -ne 0 ] + then + echo >&2 "$PROGRAM_NAME: cannot load $myconfig" + echo "DISABLE" + exit 1 + fi + time_divisor=$((time_divisor)) + [ $time_divisor -lt 10 ] && time_divisor=10 + [ $time_divisor -gt 100 ] && time_divisor=100 else - echo >&2 "$PROGRAM_NAME: configuration file '$myconfig' not found. Using defaults." + echo >&2 "$PROGRAM_NAME: configuration file '$myconfig' not found. Using defaults." fi if [ "$pause_method" = "suspend" ] then - # enable bash job control - # this is required for suspend to work - set -m + # enable bash job control + # this is required for suspend to work + set -m fi # we check for the timeout command, after we load our @@ -185,22 +185,22 @@ fi # can emulate the timeout command we need: # > timeout SECONDS command ... if [ $check_for_timeout -eq 1 ] - then - require_cmd timeout || exit 1 + then + require_cmd timeout || exit 1 fi # ----------------------------------------------------------------------------- # internal checks # netdata passes the requested update frequency as the first argument -update_every=$(( update_every + 1 - 1)) # makes sure it is a number +update_every=$(( update_every + 1 - 1)) # makes sure it is a number test $update_every -eq 0 && update_every=1 # if it is zero, make it 1 # check the charts.d directory if [ ! -d "$chartsd" ] - then - echo >&2 "$PROGRAM_NAME: cannot find charts directory '$chartsd'" - echo "DISABLE" + then + echo >&2 "$PROGRAM_NAME: cannot find charts directory '$chartsd'" + echo "DISABLE" fi @@ -210,13 +210,13 @@ fi # default sleep function LOOPSLEEPMS_HIGHRES=0 loopsleepms() { - [ "$1" = "tellwork" ] && shift - sleep $1 + [ "$1" = "tellwork" ] && shift + sleep $1 } now_ms= current_time_ms() { - now_ms="$(date +'%s')000" + now_ms="$(date +'%s')000" } # if found and included, this file overwrites loopsleepms() @@ -229,10 +229,10 @@ current_time_ms() { # library functions fixid() { - echo "$*" |\ - tr -c "[A-Z][a-z][0-9]" "_" |\ - sed -e "s|^_\+||g" -e "s|_\+$||g" -e "s|_\+|_|g" |\ - tr "[A-Z]" "[a-z]" + echo "$*" |\ + tr -c "[A-Z][a-z][0-9]" "_" |\ + sed -e "s|^_\+||g" -e "s|_\+$||g" -e "s|_\+|_|g" |\ + tr "[A-Z]" "[a-z]" } # convert any floating point number @@ -241,57 +241,57 @@ fixid() { # so that no fork is necessary # the multiplier must be a power of 10 float2int() { - local f m="$2" a b l v=($1) - f=${v[0]} - - # echo >&2 "value='${1}' f='${f}', m='${m}'" - - # the length of the multiplier - 1 - l=$(( ${#m} - 1 )) - - # check if the number is in scientific notation - if [[ ${f} =~ ^[[:space:]]*(-)?[0-9.]+(e|E)(\+|-)[0-9]+ ]] - then - # convert it to decimal - # unfortunately, this fork cannot be avoided - # if you know of a way to avoid it, please let me know - f=$(printf "%0.${l}f" ${f}) - fi - - # split the floating point number - # in integer (a) and decimal (b) - a=${f/.*/} - b=${f/*./} - - # if the integer part is missing - # set it to zero - [ -z "${a}" ] && a="0" - - # strip leading zeros from the integer part - # base 10 convertion - a=$((10#$a)) - - # check the length of the decimal part - # against the length of the multiplier - if [ ${#b} -gt ${l} ] - then - # too many digits - take the most significant - b=${b:0:${l}} - - elif [ ${#b} -lt ${l} ] - then - # too few digits - pad with zero on the right - local z="00000000000000000000000" r=$((l - ${#b})) - b="${b}${z:0:${r}}" - fi - - # strip leading zeros from the decimal part - # base 10 convertion - b=$((10#$b)) - - # store the result - FLOAT2INT_RESULT=$(( (a * m) + b )) - #echo >&2 "FLOAT2INT_RESULT='${FLOAT2INT_RESULT}'" + local f m="$2" a b l v=($1) + f=${v[0]} + + # echo >&2 "value='${1}' f='${f}', m='${m}'" + + # the length of the multiplier - 1 + l=$(( ${#m} - 1 )) + + # check if the number is in scientific notation + if [[ ${f} =~ ^[[:space:]]*(-)?[0-9.]+(e|E)(\+|-)[0-9]+ ]] + then + # convert it to decimal + # unfortunately, this fork cannot be avoided + # if you know of a way to avoid it, please let me know + f=$(printf "%0.${l}f" ${f}) + fi + + # split the floating point number + # in integer (a) and decimal (b) + a=${f/.*/} + b=${f/*./} + + # if the integer part is missing + # set it to zero + [ -z "${a}" ] && a="0" + + # strip leading zeros from the integer part + # base 10 convertion + a=$((10#$a)) + + # check the length of the decimal part + # against the length of the multiplier + if [ ${#b} -gt ${l} ] + then + # too many digits - take the most significant + b=${b:0:${l}} + + elif [ ${#b} -lt ${l} ] + then + # too few digits - pad with zero on the right + local z="00000000000000000000000" r=$((l - ${#b})) + b="${b}${z:0:${r}}" + fi + + # strip leading zeros from the decimal part + # base 10 convertion + b=$((10#$b)) + + # store the result + FLOAT2INT_RESULT=$(( (a * m) + b )) + #echo >&2 "FLOAT2INT_RESULT='${FLOAT2INT_RESULT}'" } @@ -299,84 +299,105 @@ float2int() { # charts check functions all_charts() { - cd "$chartsd" - [ $? -ne 0 ] && echo >&2 "$PROGRAM_NAME: Cannot cd to $chartsd" && return 1 + cd "$chartsd" + [ $? -ne 0 ] && echo >&2 "$PROGRAM_NAME: Cannot cd to $chartsd" && return 1 - ls *.chart.sh | sed "s/\.chart\.sh$//g" + ls *.chart.sh | sed "s/\.chart\.sh$//g" } +declare -A charts_enable_keyword=( + ['apache']="force" + ['cpu_apps']="force" + ['cpufreq']="force" + ['example']="force" + ['exim']="force" + ['hddtemp']="force" + ['load_average']="force" + ['mem_apps']="force" + ['mysql']="force" + ['nginx']="force" + ['phpfpm']="force" + ['postfix']="force" + ['sensors']="force" + ['squid']="force" + ['tomcat']="force" + ) + all_enabled_charts() { - local charts= enabled= - - # find all enabled charts - - for chart in $( all_charts ) - do - eval "enabled=\$$chart" - if [ -z "${enabled}" ] - then - enabled="${enable_all_charts}" - fi - - if [ ! "${enabled}" = "yes" ] - then - echo >&2 "$PROGRAM_NAME: '$chart' is NOT enabled. Add a line with $chart=yes in $myconfig to enable it (or remove the line that disables it)." - else - [ $debug -eq 1 ] && echo >&2 "$PROGRAM_NAME: '$chart' is enabled." - local charts="$charts $chart" - fi - done - - local charts2= - for chart in $charts - do - # check the enabled charts - local check="$( cat "$chartsd/$chart.chart.sh" | sed "s/^ \+//g" | grep "^$chart$charts_check()" )" - if [ -z "$check" ] - then - echo >&2 "$PROGRAM_NAME: chart '$chart' does not seem to have a $chart$charts_check() function. Disabling it." - continue - fi - - local create="$( cat "$chartsd/$chart.chart.sh" | sed "s/^ \+//g" | grep "^$chart$charts_create()" )" - if [ -z "$create" ] - then - echo >&2 "$PROGRAM_NAME: chart '$chart' does not seem to have a $chart$charts_create() function. Disabling it." - continue - fi - - local update="$( cat "$chartsd/$chart.chart.sh" | sed "s/^ \+//g" | grep "^$chart$charts_update()" )" - if [ -z "$update" ] - then - echo >&2 "$PROGRAM_NAME: chart '$chart' does not seem to have a $chart$charts_update() function. Disabling it." - continue - fi - - # check its config - #if [ -f "$confd/$chart.conf" ] - #then - # if [ ! -z "$( cat "$confd/$chart.conf" | sed "s/^ \+//g" | grep -v "^$" | grep -v "^#" | grep -v "^$chart$charts_undescore" )" ] - # then - # echo >&2 "$PROGRAM_NAME: chart's $chart config $confd/$chart.conf should only have lines starting with $chart$charts_undescore . Disabling it." - # continue - # fi - #fi - - #if [ $dryrunner -eq 1 ] - # then - # "$pluginsd/charts.d.dryrun-helper.sh" "$chart" "$chartsd/$chart.chart.sh" "$confd/$chart.conf" >/dev/null - # if [ $? -ne 0 ] - # then - # echo >&2 "$PROGRAM_NAME: chart's $chart did not pass the dry run check. This means it uses global variables not starting with $chart. Disabling it." - # continue - # fi - #fi - - local charts2="$charts2 $chart" - done - - echo $charts2 - [ $debug -eq 1 ] && echo >&2 "$PROGRAM_NAME: enabled charts: $charts2" + local charts= enabled= required= + + # find all enabled charts + + for chart in $( all_charts ) + do + eval "enabled=\$$chart" + if [ -z "${enabled}" ] + then + enabled="${enable_all_charts}" + fi + + required="${charts_enable_keyword[${chart}]}" + [ -z "${required}" ] && required="yes" + + if [ ! "${enabled}" = "${required}" ] + then + echo >&2 "$PROGRAM_NAME: '$chart' is NOT enabled. Add a line with $chart=$required in $myconfig to enable it (or remove the line that disables it)." + else + [ $debug -eq 1 ] && echo >&2 "$PROGRAM_NAME: '$chart' is enabled." + local charts="$charts $chart" + fi + done + + local charts2= + for chart in $charts + do + # check the enabled charts + local check="$( cat "$chartsd/$chart.chart.sh" | sed "s/^ \+//g" | grep "^$chart$charts_check()" )" + if [ -z "$check" ] + then + echo >&2 "$PROGRAM_NAME: chart '$chart' does not seem to have a $chart$charts_check() function. Disabling it." + continue + fi + + local create="$( cat "$chartsd/$chart.chart.sh" | sed "s/^ \+//g" | grep "^$chart$charts_create()" )" + if [ -z "$create" ] + then + echo >&2 "$PROGRAM_NAME: chart '$chart' does not seem to have a $chart$charts_create() function. Disabling it." + continue + fi + + local update="$( cat "$chartsd/$chart.chart.sh" | sed "s/^ \+//g" | grep "^$chart$charts_update()" )" + if [ -z "$update" ] + then + echo >&2 "$PROGRAM_NAME: chart '$chart' does not seem to have a $chart$charts_update() function. Disabling it." + continue + fi + + # check its config + #if [ -f "$confd/$chart.conf" ] + #then + # if [ ! -z "$( cat "$confd/$chart.conf" | sed "s/^ \+//g" | grep -v "^$" | grep -v "^#" | grep -v "^$chart$charts_undescore" )" ] + # then + # echo >&2 "$PROGRAM_NAME: chart's $chart config $confd/$chart.conf should only have lines starting with $chart$charts_undescore . Disabling it." + # continue + # fi + #fi + + #if [ $dryrunner -eq 1 ] + # then + # "$pluginsd/charts.d.dryrun-helper.sh" "$chart" "$chartsd/$chart.chart.sh" "$confd/$chart.conf" >/dev/null + # if [ $? -ne 0 ] + # then + # echo >&2 "$PROGRAM_NAME: chart's $chart did not pass the dry run check. This means it uses global variables not starting with $chart. Disabling it." + # continue + # fi + #fi + + local charts2="$charts2 $chart" + done + + echo $charts2 + [ $debug -eq 1 ] && echo >&2 "$PROGRAM_NAME: enabled charts: $charts2" } @@ -387,32 +408,36 @@ suffix_update_every="_update_every" active_charts= for chart in $( all_enabled_charts ) do - [ $debug -eq 1 ] && echo >&2 "$PROGRAM_NAME: loading chart: '$chartsd/$chart.chart.sh'" - . "$chartsd/$chart.chart.sh" - - if [ -f "$confd/$chart.conf" ] - then - [ $debug -eq 1 ] && echo >&2 "$PROGRAM_NAME: loading chart options: '$confd/$chart.conf'" - . "$confd/$chart.conf" - else - echo >&2 "$PROGRAM_NAME: $chart: configuration file '$confd/$chart.conf' not found. Using defaults." - fi - - eval "dt=\$$chart$suffix_update_every" - dt=$(( dt + 1 - 1 )) # make sure it is a number - if [ $dt -lt $update_every ] - then - eval "$chart$suffix_update_every=$update_every" - fi - - $chart$charts_check - if [ $? -eq 0 ] - then - [ $debug -eq 1 ] && echo >&2 "$PROGRAM_NAME: '$chart' activated" - active_charts="$active_charts $chart" - else - echo >&2 "$PROGRAM_NAME: chart '$chart' check() function reports failure." - fi + [ $debug -eq 1 ] && echo >&2 "$PROGRAM_NAME: loading chart: '$chartsd/$chart.chart.sh'" + . "$chartsd/$chart.chart.sh" + + if [ -f "$confd/charts.d/$chart.conf" ] + then + [ $debug -eq 1 ] && echo >&2 "$PROGRAM_NAME: loading chart options: '$confd/charts.d/$chart.conf'" + . "$confd/charts.d/$chart.conf" + elif [ -f "$confd/$chart.conf" ] + then + [ $debug -eq 1 ] && echo >&2 "$PROGRAM_NAME: loading chart options: '$confd/$chart.conf'" + . "$confd/$chart.conf" + else + echo >&2 "$PROGRAM_NAME: $chart: configuration file '$confd/charts.d/$chart.conf' not found. Using defaults." + fi + + eval "dt=\$$chart$suffix_update_every" + dt=$(( dt + 1 - 1 )) # make sure it is a number + if [ $dt -lt $update_every ] + then + eval "$chart$suffix_update_every=$update_every" + fi + + $chart$charts_check + if [ $? -eq 0 ] + then + [ $debug -eq 1 ] && echo >&2 "$PROGRAM_NAME: '$chart' activated" + active_charts="$active_charts $chart" + else + echo >&2 "$PROGRAM_NAME: chart '$chart' check() function reports failure." + fi done [ $debug -eq 1 ] && echo >&2 "$PROGRAM_NAME: activated charts: $active_charts" @@ -427,26 +452,26 @@ test $debug -eq 1 && debug_time=tellwork # if we only need a specific chart, remove all the others if [ ! -z "${chart_only}" ] then - [ $debug -eq 1 ] && echo >&2 "$PROGRAM_NAME: requested to run only for: '${chart_only}'" - check_charts= - for chart in $active_charts - do - if [ "$chart" = "$chart_only" ] - then - check_charts="$chart" - break - fi - done - active_charts="$check_charts" + [ $debug -eq 1 ] && echo >&2 "$PROGRAM_NAME: requested to run only for: '${chart_only}'" + check_charts= + for chart in $active_charts + do + if [ "$chart" = "$chart_only" ] + then + check_charts="$chart" + break + fi + done + active_charts="$check_charts" fi [ $debug -eq 1 ] && echo >&2 "$PROGRAM_NAME: activated charts: $active_charts" # stop if we just need a pre-check if [ $check -eq 1 ] then - echo >&2 "CHECK RESULT" - echo >&2 "Will run the charts: $active_charts" - exit 0 + echo >&2 "CHECK RESULT" + echo >&2 "Will run the charts: $active_charts" + exit 0 fi # ----------------------------------------------------------------------------- @@ -454,13 +479,13 @@ fi TMP_DIR= chartsd_cleanup() { - cd /tmp - if [ ! -z "$TMP_DIR" -a -d "$TMP_DIR" ] - then - [ $debug -eq 1 ] && echo >&2 "$PROGRAM_NAME: cleaning up temporary directory $TMP_DIR ..." - rm -rf "$TMP_DIR" - fi - exit 0 + cd /tmp + if [ ! -z "$TMP_DIR" -a -d "$TMP_DIR" ] + then + [ $debug -eq 1 ] && echo >&2 "$PROGRAM_NAME: cleaning up temporary directory $TMP_DIR ..." + rm -rf "$TMP_DIR" + fi + exit 0 } trap chartsd_cleanup EXIT trap chartsd_cleanup SIGHUP @@ -468,9 +493,9 @@ trap chartsd_cleanup INT if [ $UID = "0" ] then - TMP_DIR="$( mktemp -d /var/run/netdata-${PROGRAM_NAME}-XXXXXXXXXX )" + TMP_DIR="$( mktemp -d /var/run/netdata-${PROGRAM_NAME}-XXXXXXXXXX )" else - TMP_DIR="$( mktemp -d /tmp/.netdata-${PROGRAM_NAME}-XXXXXXXXXX )" + TMP_DIR="$( mktemp -d /tmp/.netdata-${PROGRAM_NAME}-XXXXXXXXXX )" fi cd "$TMP_DIR" || exit 1 @@ -481,15 +506,15 @@ cd "$TMP_DIR" || exit 1 run_charts= for chart in $active_charts do - [ $debug -eq 1 ] && echo >&2 "$PROGRAM_NAME: Calling '$chart$charts_create()'..." - $chart$charts_create - if [ $? -eq 0 ] - then - run_charts="$run_charts $chart" - [ $debug -eq 1 ] && echo >&2 "$PROGRAM_NAME: '$chart' has initialized." - else - echo >&2 "$PROGRAM_NAME: chart '$chart' function '$chart$charts_create()' reports failure." - fi + [ $debug -eq 1 ] && echo >&2 "$PROGRAM_NAME: Calling '$chart$charts_create()'..." + $chart$charts_create + if [ $? -eq 0 ] + then + run_charts="$run_charts $chart" + [ $debug -eq 1 ] && echo >&2 "$PROGRAM_NAME: '$chart' has initialized." + else + echo >&2 "$PROGRAM_NAME: chart '$chart' function '$chart$charts_create()' reports failure." + fi done [ $debug -eq 1 ] && echo >&2 "$PROGRAM_NAME: run_charts='$run_charts'" @@ -498,109 +523,131 @@ done # update dimensions if [ -z "$run_charts" ] - then - echo >&2 "$PROGRAM_NAME: No charts to collect data from." - echo "DISABLE" - exit 1 + then + echo >&2 "$PROGRAM_NAME: No charts to collect data from." + echo "DISABLE" + exit 1 fi -declare -A charts_last_update=() charts_update_every=() charts_next_update=() charts_run_counter=() +declare -A charts_last_update=() charts_update_every=() charts_next_update=() charts_run_counter=() charts_serial_failures=() global_update() { - local exit_at \ - c=0 dt ret last_ms exec_start_ms exec_end_ms \ - chart now_charts=() next_charts=($run_charts) - - # return the current time in ms in $now_ms - current_time_ms - - exit_at=$(( now_ms + (restart_timeout * 1000) )) - - for chart in $run_charts - do - eval "charts_update_every[$chart]=\$$chart$suffix_update_every" - test -z "${charts_update_every[$chart]}" && charts_update_every[$charts]=$update_every - charts_last_update[$chart]=$((now_ms - (now_ms % (charts_update_every[$chart] * 1000) ) )) - charts_next_update[$chart]=$(( charts_last_update[$chart] + (charts_update_every[$chart] * 1000) )) - charts_run_counter[$chart]=0 - - echo "CHART netdata.plugin_chartsd_$chart '' 'Execution time for $chart plugin' 'milliseconds / run' charts.d netdata.plugin_charts area 145000 ${charts_update_every[$chart]}" - echo "DIMENSION run_time 'run time' absolute 1 1" - done - - # the main loop - while [ 1 ] - do - c=$((c + 1)) - now_charts=("${next_charts[@]}") - next_charts=() - - # return the current time in ms in $now_ms - current_time_ms - - for chart in "${now_charts[@]}" - do - # echo >&2 "DEBUG: chart: $chart last: ${charts_last_update[$chart]}, next: ${charts_next_update[$chart]}, now: ${now_ms}" - if [ ${now_ms} -ge ${charts_next_update[$chart]} ] - then - last_ms=${charts_last_update[$chart]} - dt=$(( (now_ms - last_ms) )) - # echo >&2 "DEBUG: chart: $chart last: ${charts_last_update[$chart]}, next: ${charts_next_update[$chart]}, now: ${now_ms}, dt: ${dt}" - - charts_last_update[$chart]=${now_ms} - - while [ ${charts_next_update[$chart]} -lt ${now_ms} ] - do - charts_next_update[$chart]=$(( charts_next_update[$chart] + (charts_update_every[$chart] * 1000) )) - done - - # the first call should not give a duration - # so that netdata calibrates to current time - dt=$(( dt * 1000 )) - charts_run_counter[$chart]=$(( charts_run_counter[$chart] + 1 )) - if [ ${charts_run_counter[$chart]} -eq 1 ] - then - dt= - fi - - exec_start_ms=$now_ms - $chart$charts_update $dt - ret=$? - - # return the current time in ms in $now_ms - current_time_ms; exec_end_ms=$now_ms - - echo "BEGIN netdata.plugin_chartsd_$chart $dt" - if [ $ret -eq 0 ] - then - echo "SET run_time = $(( exec_end_ms - exec_start_ms ))" - next_charts+=($chart) - else - echo "SET run_time = $(( (exec_end_ms - exec_start_ms) * -1 ))" - echo >&2 "$PROGRAM_NAME: chart '$chart' update() function reports failure. Disabling it." - fi - echo "END" - else - next_charts+=($chart) - fi - done - - if [ "$pause_method" = "suspend" ] - then - echo "STOPPING_WAKE_ME_UP_PLEASE" - suspend || ( echo >&2 "$PROGRAM_NAME: suspend returned error $?, falling back to sleep."; loopsleepms $debug_time $update_every $time_divisor) - else - # wait the time you are required to - #loopsleepms $debug_time $update_every $time_divisor - if [ ${LOOPSLEEPMS_HIGHRES} -eq 1 ] - then - sleep 0.2 - else - sleep 1 - fi - fi - - test ${now_ms} -ge ${exit_at} && exit 0 - done + local exit_at \ + c=0 dt ret last_ms exec_start_ms exec_end_ms \ + chart now_charts=() next_charts=($run_charts) \ + next_ms x seconds millis + + # return the current time in ms in $now_ms + current_time_ms + + exit_at=$(( now_ms + (restart_timeout * 1000) )) + + for chart in $run_charts + do + eval "charts_update_every[$chart]=\$$chart$suffix_update_every" + test -z "${charts_update_every[$chart]}" && charts_update_every[$charts]=$update_every + charts_last_update[$chart]=$((now_ms - (now_ms % (charts_update_every[$chart] * 1000) ) )) + charts_next_update[$chart]=$(( charts_last_update[$chart] + (charts_update_every[$chart] * 1000) )) + charts_run_counter[$chart]=0 + charts_serial_failures[$chart]=0 + + echo "CHART netdata.plugin_chartsd_$chart '' 'Execution time for $chart plugin' 'milliseconds / run' charts.d netdata.plugin_charts area 145000 ${charts_update_every[$chart]}" + echo "DIMENSION run_time 'run time' absolute 1 1" + done + + # the main loop + while [ "${#next_charts[@]}" -gt 0 ] + do + c=$((c + 1)) + now_charts=("${next_charts[@]}") + next_charts=() + + # return the current time in ms in $now_ms + current_time_ms + + for chart in "${now_charts[@]}" + do + # echo >&2 "DEBUG: chart: $chart last: ${charts_last_update[$chart]}, next: ${charts_next_update[$chart]}, now: ${now_ms}" + if [ ${now_ms} -ge ${charts_next_update[$chart]} ] + then + last_ms=${charts_last_update[$chart]} + dt=$(( (now_ms - last_ms) )) + # echo >&2 "DEBUG: chart: $chart last: ${charts_last_update[$chart]}, next: ${charts_next_update[$chart]}, now: ${now_ms}, dt: ${dt}" + + charts_last_update[$chart]=${now_ms} + + while [ ${charts_next_update[$chart]} -lt ${now_ms} ] + do + charts_next_update[$chart]=$(( charts_next_update[$chart] + (charts_update_every[$chart] * 1000) )) + done + + # the first call should not give a duration + # so that netdata calibrates to current time + dt=$(( dt * 1000 )) + charts_run_counter[$chart]=$(( charts_run_counter[$chart] + 1 )) + if [ ${charts_run_counter[$chart]} -eq 1 ] + then + dt= + fi + + exec_start_ms=$now_ms + $chart$charts_update $dt + ret=$? + + # return the current time in ms in $now_ms + current_time_ms; exec_end_ms=$now_ms + + echo "BEGIN netdata.plugin_chartsd_$chart $dt" + echo "SET run_time = $(( exec_end_ms - exec_start_ms ))" + echo "END" + + if [ $ret -eq 0 ] + then + charts_serial_failures[$chart]=0 + next_charts+=($chart) + else + charts_serial_failures[$chart]=$(( charts_serial_failures[$chart] + 1 )) + + if [ ${charts_serial_failures[$chart]} -gt 10 ] + then + echo >&2 "$PROGRAM_NAME: chart '$chart' update() function reported failure ${charts_serial_failures[$chart]} times. Disabling it." + else + echo >&2 "$PROGRAM_NAME: chart '$chart' update() function reports failure. Will keep trying for a while." + next_charts+=($chart) + fi + fi + else + next_charts+=($chart) + fi + done + + if [ "$pause_method" = "suspend" ] + then + echo "STOPPING_WAKE_ME_UP_PLEASE" + suspend || ( echo >&2 "$PROGRAM_NAME: suspend returned error $?, falling back to sleep."; loopsleepms $debug_time $update_every $time_divisor) + else + # wait the time you are required to + next_ms=$((now_ms + (update_every * 1000 * 100) )) + for x in "${charts_next_update[@]}"; do [ ${x} -lt ${next_ms} ] && next_ms=${x}; done + next_ms=$((next_ms - now_ms)) + + if [ ${LOOPSLEEPMS_HIGHRES} -eq 1 -a ${next_ms} -gt 0 ] + then + seconds=$(( next_ms / 1000 )) + millis=$(( next_ms % 1000 )) + [ ${millis} -lt 10 ] && millis="0${millis}" + [ ${millis} -lt 100 ] && millis="0${millis}" + [ $debug -eq 1 ] && echo >&2 "$PROGRAM_NAME: sleeping for ${seconds}.${millis} seconds." + sleep ${seconds}.${millis} + else + sleep $update_every + fi + fi + + test ${now_ms} -ge ${exit_at} && exit 0 + done + + echo >&2 "$PROGRAM_NAME: Nothing left to do. Disabling charts.d.plugin." + echo "DISABLE" } global_update diff --git a/plugins.d/loopsleepms.sh.inc b/plugins.d/loopsleepms.sh.inc index 2e22de3d8..02ab694d2 100644 --- a/plugins.d/loopsleepms.sh.inc +++ b/plugins.d/loopsleepms.sh.inc @@ -1,4 +1,4 @@ -#!/bin/bash +# no need for shebang - this file is included from other scripts # this function is used to sleep a fraction of a second # it calculates the difference between every time is called @@ -7,9 +7,9 @@ LOOPSLEEP_DATE="$(which date)" if [ -z "$LOOPSLEEP_DATE" ] - then - echo >&2 "$0: ERROR: Cannot find the command 'date' in the system path." - exit 1 + then + echo >&2 "$0: ERROR: Cannot find the command 'date' in the system path." + exit 1 fi LOOPSLEEPMS_LASTRUN=0 @@ -21,70 +21,70 @@ test "$($LOOPSLEEP_DATE +%N)" = "%N" && LOOPSLEEPMS_HIGHRES=0 now_ms= current_time_ms() { - # if high resolution is not supported - # just sleep the time requested, in seconds - if [ $LOOPSLEEPMS_HIGHRES -eq 0 ] - then - now_ms="$($LOOPSLEEP_DATE +'%s')000" - else - now_ms="$(( $( $LOOPSLEEP_DATE +'%s * 1000 + %-N / 1000000' ) ))" - fi + # if high resolution is not supported + # just sleep the time requested, in seconds + if [ $LOOPSLEEPMS_HIGHRES -eq 0 ] + then + now_ms="$($LOOPSLEEP_DATE +'%s')000" + else + now_ms="$(( $( $LOOPSLEEP_DATE +'%s * 1000 + %-N / 1000000' ) ))" + fi } loopsleepms() { - local tellwork=0 t="$1" div s m now mstosleep + local tellwork=0 t="$1" div s m now mstosleep - if [ "$t" = "tellwork" ] - then - tellwork=1 - shift - t="$1" - fi - div="${2-100}" + if [ "$t" = "tellwork" ] + then + tellwork=1 + shift + t="$1" + fi + div="${2-100}" - # $t = the time in seconds to wait + # $t = the time in seconds to wait - # if high resolution is not supported - # just sleep the time requested, in seconds - if [ $LOOPSLEEPMS_HIGHRES -eq 0 ] - then - sleep $t - return - fi + # if high resolution is not supported + # just sleep the time requested, in seconds + if [ $LOOPSLEEPMS_HIGHRES -eq 0 ] + then + sleep $t + return + fi - # get the current time, in ms - # milliseconds since epoch (1-1-1970) - now="$(( $( $LOOPSLEEP_DATE +'%s * 1000 + %-N / 1000000' ) ))" + # get the current time, in ms + # milliseconds since epoch (1-1-1970) + now="$(( $( $LOOPSLEEP_DATE +'%s * 1000 + %-N / 1000000' ) ))" - # calculate required sleep in ms - t=$((t * 1000 * div / 100)) + # calculate required sleep in ms + t=$((t * 1000 * div / 100)) - # this is our first run - # just wait the requested time - test $LOOPSLEEPMS_LASTRUN -eq 0 && LOOPSLEEPMS_LASTRUN=$now + # this is our first run + # just wait the requested time + test $LOOPSLEEPMS_LASTRUN -eq 0 && LOOPSLEEPMS_LASTRUN=$now - # calculate ms since last run - LOOPSLEEPMS_LASTWORK=$((now - LOOPSLEEPMS_LASTRUN - LOOPSLEEPMS_LASTSLEEP)) - # echo "# last loop's work took $LOOPSLEEPMS_LASTWORK ms" + # calculate ms since last run + LOOPSLEEPMS_LASTWORK=$((now - LOOPSLEEPMS_LASTRUN - LOOPSLEEPMS_LASTSLEEP)) + # echo "# last loop's work took $LOOPSLEEPMS_LASTWORK ms" - # calculate ms to sleep - mstosleep=$(( t - LOOPSLEEPMS_LASTWORK )) - # echo "# mstosleep is $mstosleep ms" + # calculate ms to sleep + mstosleep=$(( t - LOOPSLEEPMS_LASTWORK )) + # echo "# mstosleep is $mstosleep ms" - # if we are too slow, sleep some time - test $mstosleep -lt 200 && mstosleep=200 + # if we are too slow, sleep some time + test $mstosleep -lt 200 && mstosleep=200 - s=$((mstosleep / 1000)) - m=$((mstosleep - (s * 1000))) + s=$((mstosleep / 1000)) + m=$((mstosleep - (s * 1000))) - test $tellwork -eq 1 && echo >&2 " >>> PERFORMANCE >>> WORK TOOK $LOOPSLEEPMS_LASTWORK ms ( $((LOOPSLEEPMS_LASTWORK * 100 / 1000)).$((LOOPSLEEPMS_LASTWORK % 10))% cpu ) >>> SLEEPING $mstosleep ms" + test $tellwork -eq 1 && echo >&2 " >>> PERFORMANCE >>> WORK TOOK $LOOPSLEEPMS_LASTWORK ms ( $((LOOPSLEEPMS_LASTWORK * 100 / 1000)).$((LOOPSLEEPMS_LASTWORK % 10))% cpu ) >>> SLEEPING $mstosleep ms" - # echo "# sleeping $s.$m" - # echo - sleep $s.$m + # echo "# sleeping $s.$m" + # echo + sleep $s.$m - # keep the values we need - # for our next run - LOOPSLEEPMS_LASTRUN=$now - LOOPSLEEPMS_LASTSLEEP=$mstosleep + # keep the values we need + # for our next run + LOOPSLEEPMS_LASTRUN=$now + LOOPSLEEPMS_LASTSLEEP=$mstosleep } diff --git a/plugins.d/node.d.plugin b/plugins.d/node.d.plugin index a1fa754fa..21b04384e 100755 --- a/plugins.d/node.d.plugin +++ b/plugins.d/node.d.plugin @@ -1,4 +1,4 @@ -#!/bin/sh +#!/usr/bin/env bash ':' //; exec "$(command -v nodejs || command -v node || command -v js || echo "ERROR node.js IS NOT AVAILABLE IN THIS SYSTEM")" "$0" "$@" // shebang hack from: @@ -40,55 +40,60 @@ var netdata = require('netdata'); // configuration function pluginConfig(filename) { - var f = path.basename(filename); + var f = path.basename(filename); - var m = f.match('.plugin' + '$'); - if(m === null) m = f.match('.node.js' + '$'); - if(m !== null) - return netdata.options.paths.config + '/' + f.substring(0, m.index) + '.conf'; + // node.d.plugin configuration + var m = f.match('.plugin' + '$'); + if(m !== null) + return netdata.options.paths.config + '/' + f.substring(0, m.index) + '.conf'; - return netdata.options.paths.config + '/' + f + '.conf'; + // node.d modules configuration + m = f.match('.node.js' + '$'); + if(m !== null) + return netdata.options.paths.config + '/node.d/' + f.substring(0, m.index) + '.conf'; + + return netdata.options.paths.config + '/node.d/' + f + '.conf'; } // internal defaults extend(true, netdata.options, { - filename: path.basename(__filename), + filename: path.basename(__filename), - update_every: NETDATA_UPDATE_EVERY, + update_every: NETDATA_UPDATE_EVERY, - exit_after_ms: 3600 * 4 * 1000, + exit_after_ms: 3600 * 4 * 1000, - paths: { - plugins: NETDATA_PLUGINS_DIR, - config: NETDATA_CONFIG_DIR, - modules: [], - }, + paths: { + plugins: NETDATA_PLUGINS_DIR, + config: NETDATA_CONFIG_DIR, + modules: [], + }, - modules_enable_autodetect: true, - modules_enable_all: true, - modules: {}, + modules_enable_autodetect: true, + modules_enable_all: true, + modules: {}, }); netdata.options.config_filename = pluginConfig(__filename); // load configuration file try { - netdata.options_loaded = JSON.parse(fs.readFileSync(netdata.options.config_filename, 'utf8')); - extend(true, netdata.options, netdata.options_loaded); - console.error('merged netdata object:'); - console.error(netdata); + netdata.options_loaded = JSON.parse(fs.readFileSync(netdata.options.config_filename, 'utf8')); + extend(true, netdata.options, netdata.options_loaded); + console.error('merged netdata object:'); + console.error(netdata); } catch(e) { - netdata.error('Cannot read configuration file ' + netdata.options.config_filename + ': ' + e.message + ', using internal defaults.'); - netdata.options_loaded = undefined; - dumpError(e); + netdata.error('Cannot read configuration file ' + netdata.options.config_filename + ': ' + e.message + ', using internal defaults.'); + netdata.options_loaded = undefined; + dumpError(e); } // apply module paths to node.js process function applyModulePaths() { - var len = netdata.options.paths.modules.length; - while(len--) - process.mainModule.paths.unshift(netdata.options.paths.modules[len]); + var len = netdata.options.paths.modules.length; + while(len--) + process.mainModule.paths.unshift(netdata.options.paths.modules[len]); } applyModulePaths(); @@ -97,165 +102,165 @@ applyModulePaths(); // tracing function dumpError(err) { - if (typeof err === 'object') { - if (err.stack) { - netdata.debug(err.stack); - } - } + if (typeof err === 'object') { + if (err.stack) { + netdata.debug(err.stack); + } + } } // -------------------------------------------------------------------------------------------------------------------- // get command line arguments { - var found_myself = false; - var found_number = false; - var found_modules = false; - process.argv.forEach(function (val, index, array) { - netdata.debug('PARAM: ' + val); - - if(!found_myself) { - if(val === __filename) - found_myself = true; - } - else { - switch(val) { - case 'debug': - netdata.options.DEBUG = true; - netdata.debug('DEBUG enabled'); - break; - - default: - if(found_number === true) { - if(found_modules === false) { - for(var i in netdata.options.modules) - netdata.options.modules[i].enabled = false; - } - - if(typeof netdata.options.modules[val] === 'undefined') - netdata.options.modules[val] = {}; - - netdata.options.modules[val].enabled = true; - netdata.options.modules_enable_all = false; - netdata.debug('enabled module ' + val); - } - else { - try { - var x = parseInt(val); - if(x > 0) { - netdata.options.update_every = x; - if(netdata.options.update_every < NETDATA_UPDATE_EVERY) { - netdata.options.update_every = NETDATA_UPDATE_EVERY; - netdata.debug('Update frequency ' + x + 's is too low'); - } - - found_number = true; - netdata.debug('Update frequency set to ' + netdata.options.update_every + ' seconds'); - } - else netdata.error('Ignoring parameter: ' + val); - } - catch(e) { - netdata.error('Cannot get value of parameter: ' + val); - dumpError(e); - } - } - break; - } - } - }); + var found_myself = false; + var found_number = false; + var found_modules = false; + process.argv.forEach(function (val, index, array) { + netdata.debug('PARAM: ' + val); + + if(!found_myself) { + if(val === __filename) + found_myself = true; + } + else { + switch(val) { + case 'debug': + netdata.options.DEBUG = true; + netdata.debug('DEBUG enabled'); + break; + + default: + if(found_number === true) { + if(found_modules === false) { + for(var i in netdata.options.modules) + netdata.options.modules[i].enabled = false; + } + + if(typeof netdata.options.modules[val] === 'undefined') + netdata.options.modules[val] = {}; + + netdata.options.modules[val].enabled = true; + netdata.options.modules_enable_all = false; + netdata.debug('enabled module ' + val); + } + else { + try { + var x = parseInt(val); + if(x > 0) { + netdata.options.update_every = x; + if(netdata.options.update_every < NETDATA_UPDATE_EVERY) { + netdata.options.update_every = NETDATA_UPDATE_EVERY; + netdata.debug('Update frequency ' + x + 's is too low'); + } + + found_number = true; + netdata.debug('Update frequency set to ' + netdata.options.update_every + ' seconds'); + } + else netdata.error('Ignoring parameter: ' + val); + } + catch(e) { + netdata.error('Cannot get value of parameter: ' + val); + dumpError(e); + } + } + break; + } + } + }); } if(netdata.options.update_every < 1) { - netdata.debug('Adjusting update frequency to 1 second'); - netdata.options.update_every = 1; + netdata.debug('Adjusting update frequency to 1 second'); + netdata.options.update_every = 1; } // -------------------------------------------------------------------------------------------------------------------- // find modules function findModules() { - var found = 0; - - var files = fs.readdirSync(NODE_D_DIR); - var len = files.length; - while(len--) { - var m = files[len].match('.node.js' + '$'); - if(m !== null) { - var n = files[len].substring(0, m.index); - - if(typeof(netdata.options.modules[n]) === 'undefined') - netdata.options.modules[n] = { name: n, enabled: netdata.options.modules_enable_all }; - - if(netdata.options.modules[n].enabled === true) { - netdata.options.modules[n].name = n; - netdata.options.modules[n].filename = NODE_D_DIR + '/' + files[len]; - netdata.options.modules[n].loaded = false; - - if(typeof(netdata.options.modules[n].config_filename) !== 'string') - netdata.options.modules[n].config_filename = pluginConfig(files[len]); - - // load the module - try { - netdata.debug('loading module ' + netdata.options.modules[n].filename); - netdata.options.modules[n].module = require(netdata.options.modules[n].filename); - netdata.options.modules[n].module.name = n; - netdata.debug('loaded module ' + netdata.options.modules[n].name + ' from ' + netdata.options.modules[n].filename); - } - catch(e) { - netdata.options.modules[n].enabled = false; - netdata.error('Cannot load module: ' + netdata.options.modules[n].filename + ' exception: ' + e); - dumpError(e); - continue; - } - - // load its configuration - var c = { - enable_autodetect: netdata.options.modules_enable_autodetect, - update_every: netdata.options.update_every - }; - try { - netdata.debug('loading module\'s ' + netdata.options.modules[n].name + ' config ' + netdata.options.modules[n].config_filename); - var c2 = JSON.parse(fs.readFileSync(netdata.options.modules[n].config_filename, 'utf8')); - extend(true, c, c2); - netdata.debug('loaded module\'s ' + netdata.options.modules[n].name + ' config ' + netdata.options.modules[n].config_filename); - } - catch(e) { - netdata.error('Cannot load module\'s ' + netdata.options.modules[n].name + ' config from ' + netdata.options.modules[n].config_filename + ' exception: ' + e + ', using internal defaults.'); - dumpError(e); - } - - // call module auto-detection / configuration - try { - netdata.modules_configuring++; - netdata.debug('Configuring module ' + netdata.options.modules[n].name); - var serv = netdata.configure(netdata.options.modules[n].module, c, function() { - netdata.debug('Configured module ' + netdata.options.modules[n].name); - netdata.modules_configuring--; - }); - - netdata.debug('Configuring module ' + netdata.options.modules[n].name + ' reports ' + serv + ' eligible services.'); - } - catch(e) { - netdata.modules_configuring--; - netdata.options.modules[n].enabled = false; - netdata.error('Failed module auto-detection: ' + netdata.options.modules[n].name + ' exception: ' + e + ', disabling module.'); - dumpError(e); - continue; - } - - netdata.options.modules[n].loaded = true; - found++; - } - } - } - - // netdata.debug(netdata.options.modules); - return found; + var found = 0; + + var files = fs.readdirSync(NODE_D_DIR); + var len = files.length; + while(len--) { + var m = files[len].match('.node.js' + '$'); + if(m !== null) { + var n = files[len].substring(0, m.index); + + if(typeof(netdata.options.modules[n]) === 'undefined') + netdata.options.modules[n] = { name: n, enabled: netdata.options.modules_enable_all }; + + if(netdata.options.modules[n].enabled === true) { + netdata.options.modules[n].name = n; + netdata.options.modules[n].filename = NODE_D_DIR + '/' + files[len]; + netdata.options.modules[n].loaded = false; + + if(typeof(netdata.options.modules[n].config_filename) !== 'string') + netdata.options.modules[n].config_filename = pluginConfig(files[len]); + + // load the module + try { + netdata.debug('loading module ' + netdata.options.modules[n].filename); + netdata.options.modules[n].module = require(netdata.options.modules[n].filename); + netdata.options.modules[n].module.name = n; + netdata.debug('loaded module ' + netdata.options.modules[n].name + ' from ' + netdata.options.modules[n].filename); + } + catch(e) { + netdata.options.modules[n].enabled = false; + netdata.error('Cannot load module: ' + netdata.options.modules[n].filename + ' exception: ' + e); + dumpError(e); + continue; + } + + // load its configuration + var c = { + enable_autodetect: netdata.options.modules_enable_autodetect, + update_every: netdata.options.update_every + }; + try { + netdata.debug('loading module\'s ' + netdata.options.modules[n].name + ' config ' + netdata.options.modules[n].config_filename); + var c2 = JSON.parse(fs.readFileSync(netdata.options.modules[n].config_filename, 'utf8')); + extend(true, c, c2); + netdata.debug('loaded module\'s ' + netdata.options.modules[n].name + ' config ' + netdata.options.modules[n].config_filename); + } + catch(e) { + netdata.error('Cannot load module\'s ' + netdata.options.modules[n].name + ' config from ' + netdata.options.modules[n].config_filename + ' exception: ' + e + ', using internal defaults.'); + dumpError(e); + } + + // call module auto-detection / configuration + try { + netdata.modules_configuring++; + netdata.debug('Configuring module ' + netdata.options.modules[n].name); + var serv = netdata.configure(netdata.options.modules[n].module, c, function() { + netdata.debug('Configured module ' + netdata.options.modules[n].name); + netdata.modules_configuring--; + }); + + netdata.debug('Configuring module ' + netdata.options.modules[n].name + ' reports ' + serv + ' eligible services.'); + } + catch(e) { + netdata.modules_configuring--; + netdata.options.modules[n].enabled = false; + netdata.error('Failed module auto-detection: ' + netdata.options.modules[n].name + ' exception: ' + e + ', disabling module.'); + dumpError(e); + continue; + } + + netdata.options.modules[n].loaded = true; + found++; + } + } + } + + // netdata.debug(netdata.options.modules); + return found; } if(findModules() === 0) { - netdata.error('Cannot load any .node.js module from: ' + NODE_D_DIR); - netdata.disableNodePlugin(); - process.exit(1); + netdata.error('Cannot load any .node.js module from: ' + NODE_D_DIR); + netdata.disableNodePlugin(); + process.exit(1); } @@ -263,14 +268,14 @@ if(findModules() === 0) { // start function start_when_configuring_ends() { - if(netdata.modules_configuring > 0) { - netdata.debug('Waiting modules configuration, still running ' + netdata.modules_configuring); - setTimeout(start_when_configuring_ends, 500); - return; - } - - netdata.modules_configuring = 0; - netdata.start(); + if(netdata.modules_configuring > 0) { + netdata.debug('Waiting modules configuration, still running ' + netdata.modules_configuring); + setTimeout(start_when_configuring_ends, 500); + return; + } + + netdata.modules_configuring = 0; + netdata.start(); } start_when_configuring_ends(); diff --git a/plugins.d/python.d.plugin b/plugins.d/python.d.plugin new file mode 100755 index 000000000..5e81fb263 --- /dev/null +++ b/plugins.d/python.d.plugin @@ -0,0 +1,533 @@ +#!/usr/bin/env bash +'''':; exec "$(command -v python || command -v python3 || command -v python2 || echo "ERROR python IS NOT AVAILABLE IN THIS SYSTEM")" "$0" "$@" # ''' +# -*- coding: utf-8 -*- + +# Description: netdata python modules supervisor +# Author: Pawel Krupa (paulfantom) + +import os +import sys +import time +import threading + +# ----------------------------------------------------------------------------- +# globals & environment setup +# https://github.com/firehol/netdata/wiki/External-Plugins#environment-variables +MODULE_EXTENSION = ".chart.py" +BASE_CONFIG = {'update_every': os.getenv('NETDATA_UPDATE_EVERY', 1), + 'priority': 90000, + 'retries': 10} + +MODULES_DIR = os.path.abspath(os.getenv('NETDATA_PLUGINS_DIR', + os.path.dirname(__file__)) + "/../python.d") + "/" +CONFIG_DIR = os.getenv('NETDATA_CONFIG_DIR', "/etc/netdata/") +# directories should end with '/' +if CONFIG_DIR[-1] != "/": + CONFIG_DIR += "/" +sys.path.append(MODULES_DIR + "python_modules") + +PROGRAM = os.path.basename(__file__).replace(".plugin", "") +DEBUG_FLAG = False +OVERRIDE_UPDATE_EVERY = False + +# ----------------------------------------------------------------------------- +# custom, third party and version specific python modules management +import msg + +try: + assert sys.version_info >= (3, 1) + import importlib.machinery + PY_VERSION = 3 + # change this hack below if we want PY_VERSION to be used in modules + # import builtins + # builtins.PY_VERSION = 3 + msg.info('Using python v3') +except (AssertionError, ImportError): + try: + import imp + + # change this hack below if we want PY_VERSION to be used in modules + # import __builtin__ + # __builtin__.PY_VERSION = 2 + PY_VERSION = 2 + msg.info('Using python v2') + except ImportError: + msg.fatal('Cannot start. No importlib.machinery on python3 or lack of imp on python2') +# try: +# import yaml +# except ImportError: +# msg.fatal('Cannot find yaml library') +try: + if PY_VERSION == 3: + import pyyaml3 as yaml + else: + import pyyaml2 as yaml +except ImportError: + msg.fatal('Cannot find yaml library') + + +class PythonCharts(object): + """ + Main class used to control every python module. + """ + + def __init__(self, + modules=None, + modules_path='../python.d/', + modules_configs='../conf.d/', + modules_disabled=None): + """ + :param modules: list + :param modules_path: str + :param modules_configs: str + :param modules_disabled: list + """ + + if modules is None: + modules = [] + if modules_disabled is None: + modules_disabled = [] + + self.first_run = True + # set configuration directory + self.configs = modules_configs + + # load modules + loaded_modules = self._load_modules(modules_path, modules, modules_disabled) + + # load configuration files + configured_modules = self._load_configs(loaded_modules) + + # good economy and prosperity: + self.jobs = self._create_jobs(configured_modules) # type: list + + # enable timetable override like `python.d.plugin mysql debug 1` + if DEBUG_FLAG and OVERRIDE_UPDATE_EVERY: + for job in self.jobs: + job.create_timetable(BASE_CONFIG['update_every']) + + @staticmethod + def _import_module(path, name=None): + """ + Try to import module using only its path. + :param path: str + :param name: str + :return: object + """ + + if name is None: + name = path.split('/')[-1] + if name[-len(MODULE_EXTENSION):] != MODULE_EXTENSION: + return None + name = name[:-len(MODULE_EXTENSION)] + try: + if PY_VERSION == 3: + return importlib.machinery.SourceFileLoader(name, path).load_module() + else: + return imp.load_source(name, path) + except Exception as e: + msg.error("Problem loading", name, str(e)) + return None + + def _load_modules(self, path, modules, disabled): + """ + Load modules from 'modules' list or dynamically every file from 'path' (only .chart.py files) + :param path: str + :param modules: list + :param disabled: list + :return: list + """ + + # check if plugin directory exists + if not os.path.isdir(path): + msg.fatal("cannot find charts directory ", path) + + # load modules + loaded = [] + if len(modules) > 0: + for m in modules: + if m in disabled: + continue + mod = self._import_module(path + m + MODULE_EXTENSION) + if mod is not None: + loaded.append(mod) + else: # exit if plugin is not found + msg.fatal('no modules found.') + else: + # scan directory specified in path and load all modules from there + names = os.listdir(path) + for mod in names: + if mod.replace(MODULE_EXTENSION, "") in disabled: + msg.error(mod + ": disabled module ", mod.replace(MODULE_EXTENSION, "")) + continue + m = self._import_module(path + mod) + if m is not None: + msg.debug(mod + ": loading module '" + path + mod + "'") + loaded.append(m) + return loaded + + def _load_configs(self, modules): + """ + Append configuration in list named `config` to every module. + For multi-job modules `config` list is created in _parse_config, + otherwise it is created here based on BASE_CONFIG prototype with None as identifier. + :param modules: list + :return: list + """ + for mod in modules: + configfile = self.configs + mod.__name__ + ".conf" + if os.path.isfile(configfile): + msg.debug(mod.__name__ + ": loading module configuration: '" + configfile + "'") + try: + if not hasattr(mod, 'config'): + mod.config = {} + setattr(mod, + 'config', + self._parse_config(mod, read_config(configfile))) + except Exception as e: + msg.error(mod.__name__ + ": cannot parse configuration file '" + configfile + "':", str(e)) + else: + msg.error(mod.__name__ + ": configuration file '" + configfile + "' not found. Using defaults.") + # set config if not found + if not hasattr(mod, 'config'): + msg.debug(mod.__name__ + ": setting configuration for only one job") + mod.config = {None: {}} + for var in BASE_CONFIG: + try: + mod.config[None][var] = getattr(mod, var) + except AttributeError: + mod.config[None][var] = BASE_CONFIG[var] + return modules + + @staticmethod + def _parse_config(module, config): + """ + Parse configuration file or extract configuration from module file. + Example of returned dictionary: + config = {'name': { + 'update_every': 2, + 'retries': 3, + 'priority': 30000 + 'other_val': 123}} + :param module: object + :param config: dict + :return: dict + """ + if config is None: + config = {} + # get default values + defaults = {} + msg.debug(module.__name__ + ": reading configuration") + for key in BASE_CONFIG: + try: + # get defaults from module config + defaults[key] = int(config.pop(key)) + except (KeyError, ValueError): + try: + # get defaults from module source code + defaults[key] = getattr(module, key) + except (KeyError, ValueError, AttributeError): + # if above failed, get defaults from global dict + defaults[key] = BASE_CONFIG[key] + + # check if there are dict in config dict + many_jobs = False + for name in config: + if type(config[name]) is dict: + many_jobs = True + break + + # assign variables needed by supervisor to every job configuration + if many_jobs: + for name in config: + for key in defaults: + if key not in config[name]: + config[name][key] = defaults[key] + # if only one job is needed, values doesn't have to be in dict (in YAML) + else: + config = {None: config.copy()} + config[None].update(defaults) + + # return dictionary of jobs where every job has BASE_CONFIG variables + return config + + @staticmethod + def _create_jobs(modules): + """ + Create jobs based on module.config dictionary and module.Service class definition. + :param modules: list + :return: list + """ + jobs = [] + for module in modules: + for name in module.config: + # register a new job + conf = module.config[name] + try: + job = module.Service(configuration=conf, name=name) + except Exception as e: + msg.error(module.__name__ + + ("/" + str(name) if name is not None else "") + + ": cannot start job: '" + + str(e)) + return None + else: + # set chart_name (needed to plot run time graphs) + job.chart_name = module.__name__ + if name is not None: + job.chart_name += "_" + name + jobs.append(job) + msg.debug(module.__name__ + ("/" + str(name) if name is not None else "") + ": job added") + + return [j for j in jobs if j is not None] + + def _stop(self, job, reason=None): + """ + Stop specified job and remove it from self.jobs list + Also notifies user about job failure if DEBUG_FLAG is set + :param job: object + :param reason: str + """ + prefix = job.__module__ + if job.name is not None and len(job.name) != 0: + prefix += "/" + job.name + try: + self.jobs.remove(job) + msg.info("Disabled", prefix) + except Exception as e: + msg.debug("This shouldn't happen. NO " + prefix + " IN LIST:" + str(self.jobs) + " ERROR: " + str(e)) + + # TODO remove section below and remove `reason`. + prefix += ": " + if reason is None: + return + elif reason[:3] == "no ": + msg.error(prefix + + "does not seem to have " + + reason[3:] + + "() function. Disabling it.") + elif reason[:7] == "failed ": + msg.error(prefix + + reason[7:] + + "() function reports failure.") + elif reason[:13] == "configuration": + msg.error(prefix + + "configuration file '" + + self.configs + + job.__module__ + + ".conf' not found. Using defaults.") + elif reason[:11] == "misbehaving": + msg.error(prefix + "is " + reason) + + def check(self): + """ + Tries to execute check() on every job. + This cannot fail thus it is catching every exception + If job.check() fails job is stopped + """ + i = 0 + overridden = [] + msg.debug("all job objects", str(self.jobs)) + while i < len(self.jobs): + job = self.jobs[i] + try: + if not job.check(): + msg.error(job.chart_name, "check function failed.") + self._stop(job) + else: + msg.debug(job.chart_name, "check succeeded") + i += 1 + try: + if job.override_name is not None: + new_name = job.__module__ + '_' + job.override_name + if new_name in overridden: + msg.error(job.override_name + " already exists. Stopping '" + job.name + "'") + self._stop(job) + i -= 1 + else: + job.name = job.override_name + msg.debug(job.chart_name + " changing chart name to: '" + new_name + "'") + job.chart_name = new_name + overridden.append(job.chart_name) + except Exception: + pass + except AttributeError as e: + self._stop(job) + msg.error(job.chart_name, "cannot find check() function or it thrown unhandled exception.") + msg.debug(str(e)) + except (UnboundLocalError, Exception) as e: + msg.error(job.chart_name, str(e)) + self._stop(job) + msg.debug("overridden job names:", str(overridden)) + msg.debug("all remaining job objects:", str(self.jobs)) + + def create(self): + """ + Tries to execute create() on every job. + This cannot fail thus it is catching every exception. + If job.create() fails job is stopped. + This is also creating job run time chart. + """ + i = 0 + while i < len(self.jobs): + job = self.jobs[i] + try: + if not job.create(): + msg.error(job.chart_name, "create function failed.") + self._stop(job) + else: + chart = job.chart_name + sys.stdout.write( + "CHART netdata.plugin_pythond_" + + chart + + " '' 'Execution time for " + + chart + + " plugin' 'milliseconds / run' python.d netdata.plugin_python area 145000 " + + str(job.timetable['freq']) + + '\n') + sys.stdout.write("DIMENSION run_time 'run time' absolute 1 1\n\n") + msg.debug("created charts for", job.chart_name) + # sys.stdout.flush() + i += 1 + except AttributeError: + msg.error(job.chart_name, "cannot find create() function or it thrown unhandled exception.") + self._stop(job) + except (UnboundLocalError, Exception) as e: + msg.error(job.chart_name, str(e)) + self._stop(job) + + def update(self): + """ + Creates and supervises every job thread. + This will stay forever and ever and ever forever and ever it'll be the one... + """ + for job in self.jobs: + job.start() + + while True: + if threading.active_count() <= 1: + msg.fatal("no more jobs") + time.sleep(1) + + +def read_config(path): + """ + Read YAML configuration from specified file + :param path: str + :return: dict + """ + try: + with open(path, 'r') as stream: + config = yaml.load(stream) + except (OSError, IOError): + msg.error(str(path), "is not a valid configuration file") + return None + except yaml.YAMLError as e: + msg.error(str(path), "is malformed:", e) + return None + return config + + +def parse_cmdline(directory, *commands): + """ + Parse parameters from command line. + :param directory: str + :param commands: list of str + :return: dict + """ + global DEBUG_FLAG + global OVERRIDE_UPDATE_EVERY + global BASE_CONFIG + + changed_update = False + mods = [] + for cmd in commands[1:]: + if cmd == "check": + pass + elif cmd == "debug" or cmd == "all": + DEBUG_FLAG = True + # redirect stderr to stdout? + elif os.path.isfile(directory + cmd + ".chart.py") or os.path.isfile(directory + cmd): + #DEBUG_FLAG = True + mods.append(cmd.replace(".chart.py", "")) + else: + try: + BASE_CONFIG['update_every'] = int(cmd) + changed_update = True + except ValueError: + pass + if changed_update and DEBUG_FLAG: + OVERRIDE_UPDATE_EVERY = True + msg.debug(PROGRAM, "overriding update interval to", str(BASE_CONFIG['update_every'])) + + msg.debug("started from", commands[0], "with options:", *commands[1:]) + + return mods + + +# if __name__ == '__main__': +def run(): + """ + Main program. + """ + global DEBUG_FLAG, BASE_CONFIG + + # read configuration file + disabled = [] + configfile = CONFIG_DIR + "python.d.conf" + msg.PROGRAM = PROGRAM + msg.info("reading configuration file:", configfile) + log_counter = 200 + log_interval = 3600 + + conf = read_config(configfile) + if conf is not None: + try: + # exit the whole plugin when 'enabled: no' is set in 'python.d.conf' + if conf['enabled'] is False: + msg.fatal('disabled in configuration file.\n') + except (KeyError, TypeError): + pass + try: + for param in BASE_CONFIG: + BASE_CONFIG[param] = conf[param] + except (KeyError, TypeError): + pass # use default update_every from NETDATA_UPDATE_EVERY + try: + DEBUG_FLAG = conf['debug'] + except (KeyError, TypeError): + pass + try: + log_counter = conf['logs_per_interval'] + except (KeyError, TypeError): + pass + try: + log_interval = conf['log_interval'] + except (KeyError, TypeError): + pass + for k, v in conf.items(): + if k in ("update_every", "debug", "enabled"): + continue + if v is False: + disabled.append(k) + + # parse passed command line arguments + modules = parse_cmdline(MODULES_DIR, *sys.argv) + msg.DEBUG_FLAG = DEBUG_FLAG + msg.LOG_COUNTER = log_counter + msg.LOG_INTERVAL = log_interval + msg.info("MODULES_DIR='" + MODULES_DIR + + "', CONFIG_DIR='" + CONFIG_DIR + + "', UPDATE_EVERY=" + str(BASE_CONFIG['update_every']) + + ", ONLY_MODULES=" + str(modules)) + + # run plugins + charts = PythonCharts(modules, MODULES_DIR, CONFIG_DIR + "python.d/", disabled) + charts.check() + charts.create() + charts.update() + msg.fatal("finished") + + +if __name__ == '__main__': + run() diff --git a/plugins.d/tc-qos-helper.sh b/plugins.d/tc-qos-helper.sh index 7b4739815..bff5217d2 100755 --- a/plugins.d/tc-qos-helper.sh +++ b/plugins.d/tc-qos-helper.sh @@ -1,107 +1,126 @@ -#!/bin/bash +#!/usr/bin/env bash export PATH="${PATH}:/sbin:/usr/sbin:/usr/local/sbin" +PROGRAM_FILE="$0" +PROGRAM_NAME="$(basename $0)" +PROGRAM_NAME="${PROGRAM_NAME/.plugin}" + +plugins_dir="${NETDATA_PLUGINS_DIR}" +[ -z "$plugins_dir" ] && plugins_dir="$( dirname $PROGRAM_FILE )" + +config_dir=${NETDATA_CONFIG_DIR-/etc/netdata} +tc="$(which tc 2>/dev/null)" +fireqos_run_dir="/var/run/fireqos" +qos_get_class_names_every=120 +qos_exit_every=3600 + +# check if we have a valid number for interval +t=${1} +update_every=$((t)) +[ $((update_every)) -lt 1 ] && update_every=${NETDATA_UPDATE_EVERY} +[ $((update_every)) -lt 1 ] && update_every=1 + +# allow the user to override our defaults +if [ -f "${config_dir}/tc-qos-helper.conf" ] + then + source "${config_dir}/tc-qos-helper.conf" +fi + # default time function now_ms= current_time_ms() { - now_ms="$(date +'%s')000" + now_ms="$(date +'%s')000" } # default sleep function LOOPSLEEPMS_LASTWORK=0 loopsleepms() { - [ "$1" = "tellwork" ] && shift - sleep $1 + [ "$1" = "tellwork" ] && shift + sleep $1 } # if found and included, this file overwrites loopsleepms() # with a high resolution timer function for precise looping. -. "$NETDATA_PLUGINS_DIR/loopsleepms.sh.inc" +. "${plugins_dir}/loopsleepms.sh.inc" -# check if we have a valid number for interval -t=$1 -sleep_time=$((t)) -[ $((sleep_time)) -lt 1 ] && $NETDATA_UPDATE_EVERY -[ $((sleep_time)) -lt 1 ] && sleep_time=1 - -tc_cmd="$(which tc)" -if [ -z "$tc_cmd" ] - then - echo >&2 "tc: Cannot find a 'tc' command in this system." - exit 1 +if [ -z "${tc}" -o ! -x "${tc}" ] + then + echo >&2 "${PROGRAM_NAME}: Cannot find command 'tc' in this system." + exit 1 fi devices= fix_names= setclassname() { - echo "SETCLASSNAME $3 $2" + echo "SETCLASSNAME $3 $2" } show_tc() { - local x="$1" - - echo "BEGIN $x" - $tc_cmd -s class show dev $x - - # check FireQOS names for classes - if [ ! -z "$fix_names" -a -f /var/run/fireqos/ifaces/$x ] - then - name="$(cat /var/run/fireqos/ifaces/$x)" - echo "SETDEVICENAME $name" - - interface_classes= - interface_classes_monitor= - . /var/run/fireqos/$name.conf - for n in $interface_classes_monitor - do - setclassname $(echo $n | tr '|' ' ') - done - echo "SETDEVICEGROUP $interface_dev" - fi - echo "END $x" + local x="${1}" interface_dev interface_classes interface_classes_monitor + + echo "BEGIN ${x}" + ${tc} -s class show dev ${x} + + # check FireQOS names for classes + if [ ! -z "${fix_names}" -a -f "${fireqos_run_dir}/ifaces/${x}" ] + then + name="$(<"${fireqos_run_dir}/ifaces/${x}")" + echo "SETDEVICENAME ${name}" + + interface_dev= + interface_classes= + interface_classes_monitor= + source "${fireqos_run_dir}/${name}.conf" + for n in ${interface_classes_monitor} + do + setclassname ${n//|/ } + done + [ ! -z "${interface_dev}" ] && echo "SETDEVICEGROUP ${interface_dev}" + fi + echo "END ${x}" } all_devices() { - cat /proc/net/dev | grep ":" | cut -d ':' -f 1 | while read dev - do - l=$($tc_cmd class show dev $dev | wc -l) - [ $l -ne 0 ] && echo $dev - done + cat /proc/net/dev | grep ":" | cut -d ':' -f 1 | while read dev + do + l=$(${tc} class show dev ${dev} | wc -l) + [ $l -ne 0 ] && echo ${dev} + done } # update devices and class names # once every 2 minutes -names_every=$((120 / sleep_time)) +names_every=$((qos_get_class_names_every / update_every)) # exit this script every hour # it will be restarted automatically -exit_after=$((3600 / sleep_time)) +exit_after=$((qos_exit_every / update_every)) c=0 gc=0 while [ 1 ] do - fix_names= - c=$((c + 1)) - gc=$((gc + 1)) + fix_names= + c=$((c + 1)) + gc=$((gc + 1)) - if [ $c -le 1 -o $c -ge $names_every ] - then - c=1 - fix_names="YES" - devices="$( all_devices )" - fi + if [ ${c} -le 1 -o ${c} -ge ${names_every} ] + then + c=1 + fix_names="YES" + devices="$( all_devices )" + fi - for d in $devices - do - show_tc $d - done + for d in ${devices} + do + show_tc ${d} + done - echo "WORKTIME $LOOPSLEEPMS_LASTWORK" + echo "WORKTIME ${LOOPSLEEPMS_LASTWORK}" - loopsleepms $sleep_time + loopsleepms ${update_every} - [ $gc -gt $exit_after ] && exit 0 + [ ${gc} -gt ${exit_after} ] && exit 0 done diff --git a/python.d/Makefile.am b/python.d/Makefile.am new file mode 100644 index 000000000..8bccba378 --- /dev/null +++ b/python.d/Makefile.am @@ -0,0 +1,84 @@ +MAINTAINERCLEANFILES= $(srcdir)/Makefile.in +CLEANFILES = \ + python-modules-installer.sh \ + $(NULL) + +include $(top_srcdir)/build/subst.inc + +SUFFIXES = .in + +dist_python_SCRIPTS = \ + apache.chart.py \ + apache_cache.chart.py \ + cpufreq.chart.py \ + dovecot.chart.py \ + example.chart.py \ + exim.chart.py \ + hddtemp.chart.py \ + ipfs.chart.py \ + memcached.chart.py \ + mysql.chart.py \ + nginx.chart.py \ + nginx_log.chart.py \ + phpfpm.chart.py \ + postfix.chart.py \ + redis.chart.py \ + sensors.chart.py \ + squid.chart.py \ + tomcat.chart.py \ + python-modules-installer.sh \ + $(NULL) + +dist_python_DATA = \ + README.md \ + $(NULL) + +pythonmodulesdir=$(pythondir)/python_modules +dist_pythonmodules_DATA = \ + python_modules/__init__.py \ + python_modules/base.py \ + python_modules/msg.py \ + python_modules/lm_sensors.py \ + $(NULL) + +pythonyaml2dir=$(pythonmodulesdir)/pyyaml2 +dist_pythonyaml2_DATA = \ + python_modules/pyyaml2/__init__.py \ + python_modules/pyyaml2/composer.py \ + python_modules/pyyaml2/constructor.py \ + python_modules/pyyaml2/cyaml.py \ + python_modules/pyyaml2/dumper.py \ + python_modules/pyyaml2/emitter.py \ + python_modules/pyyaml2/error.py \ + python_modules/pyyaml2/events.py \ + python_modules/pyyaml2/loader.py \ + python_modules/pyyaml2/nodes.py \ + python_modules/pyyaml2/parser.py \ + python_modules/pyyaml2/reader.py \ + python_modules/pyyaml2/representer.py \ + python_modules/pyyaml2/resolver.py \ + python_modules/pyyaml2/scanner.py \ + python_modules/pyyaml2/serializer.py \ + python_modules/pyyaml2/tokens.py \ + $(NULL) + +pythonyaml3dir=$(pythonmodulesdir)/pyyaml3 +dist_pythonyaml3_DATA = \ + python_modules/pyyaml3/__init__.py \ + python_modules/pyyaml3/composer.py \ + python_modules/pyyaml3/constructor.py \ + python_modules/pyyaml3/cyaml.py \ + python_modules/pyyaml3/dumper.py \ + python_modules/pyyaml3/emitter.py \ + python_modules/pyyaml3/error.py \ + python_modules/pyyaml3/events.py \ + python_modules/pyyaml3/loader.py \ + python_modules/pyyaml3/nodes.py \ + python_modules/pyyaml3/parser.py \ + python_modules/pyyaml3/reader.py \ + python_modules/pyyaml3/representer.py \ + python_modules/pyyaml3/resolver.py \ + python_modules/pyyaml3/scanner.py \ + python_modules/pyyaml3/serializer.py \ + python_modules/pyyaml3/tokens.py \ + $(NULL) diff --git a/python.d/Makefile.in b/python.d/Makefile.in new file mode 100644 index 000000000..0e1925c56 --- /dev/null +++ b/python.d/Makefile.in @@ -0,0 +1,698 @@ +# Makefile.in generated by automake 1.14.1 from Makefile.am. +# @configure_input@ + +# Copyright (C) 1994-2013 Free Software Foundation, Inc. + +# This Makefile.in is free software; the Free Software Foundation +# gives unlimited permission to copy and/or distribute it, +# with or without modifications, as long as this notice is preserved. + +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY, to the extent permitted by law; without +# even the implied warranty of MERCHANTABILITY or FITNESS FOR A +# PARTICULAR PURPOSE. + +@SET_MAKE@ + + +VPATH = @srcdir@ +am__is_gnu_make = test -n '$(MAKEFILE_LIST)' && test -n '$(MAKELEVEL)' +am__make_running_with_option = \ + case $${target_option-} in \ + ?) ;; \ + *) echo "am__make_running_with_option: internal error: invalid" \ + "target option '$${target_option-}' specified" >&2; \ + exit 1;; \ + esac; \ + has_opt=no; \ + sane_makeflags=$$MAKEFLAGS; \ + if $(am__is_gnu_make); then \ + sane_makeflags=$$MFLAGS; \ + else \ + case $$MAKEFLAGS in \ + *\\[\ \ ]*) \ + bs=\\; \ + sane_makeflags=`printf '%s\n' "$$MAKEFLAGS" \ + | sed "s/$$bs$$bs[$$bs $$bs ]*//g"`;; \ + esac; \ + fi; \ + skip_next=no; \ + strip_trailopt () \ + { \ + flg=`printf '%s\n' "$$flg" | sed "s/$$1.*$$//"`; \ + }; \ + for flg in $$sane_makeflags; do \ + test $$skip_next = yes && { skip_next=no; continue; }; \ + case $$flg in \ + *=*|--*) continue;; \ + -*I) strip_trailopt 'I'; skip_next=yes;; \ + -*I?*) strip_trailopt 'I';; \ + -*O) strip_trailopt 'O'; skip_next=yes;; \ + -*O?*) strip_trailopt 'O';; \ + -*l) strip_trailopt 'l'; skip_next=yes;; \ + -*l?*) strip_trailopt 'l';; \ + -[dEDm]) skip_next=yes;; \ + -[JT]) skip_next=yes;; \ + esac; \ + case $$flg in \ + *$$target_option*) has_opt=yes; break;; \ + esac; \ + done; \ + test $$has_opt = yes +am__make_dryrun = (target_option=n; $(am__make_running_with_option)) +am__make_keepgoing = (target_option=k; $(am__make_running_with_option)) +pkgdatadir = $(datadir)/@PACKAGE@ +pkgincludedir = $(includedir)/@PACKAGE@ +pkglibdir = $(libdir)/@PACKAGE@ +pkglibexecdir = $(libexecdir)/@PACKAGE@ +am__cd = CDPATH="$${ZSH_VERSION+.}$(PATH_SEPARATOR)" && cd +install_sh_DATA = $(install_sh) -c -m 644 +install_sh_PROGRAM = $(install_sh) -c +install_sh_SCRIPT = $(install_sh) -c +INSTALL_HEADER = $(INSTALL_DATA) +transform = $(program_transform_name) +NORMAL_INSTALL = : +PRE_INSTALL = : +POST_INSTALL = : +NORMAL_UNINSTALL = : +PRE_UNINSTALL = : +POST_UNINSTALL = : +build_triplet = @build@ +host_triplet = @host@ +DIST_COMMON = $(top_srcdir)/build/subst.inc $(srcdir)/Makefile.in \ + $(srcdir)/Makefile.am $(dist_python_SCRIPTS) \ + $(dist_python_DATA) $(dist_pythonmodules_DATA) \ + $(dist_pythonyaml2_DATA) $(dist_pythonyaml3_DATA) +subdir = python.d +ACLOCAL_M4 = $(top_srcdir)/aclocal.m4 +am__aclocal_m4_deps = $(top_srcdir)/m4/ax_pthread.m4 \ + $(top_srcdir)/configure.ac +am__configure_deps = $(am__aclocal_m4_deps) $(CONFIGURE_DEPENDENCIES) \ + $(ACLOCAL_M4) +mkinstalldirs = $(install_sh) -d +CONFIG_HEADER = $(top_builddir)/config.h +CONFIG_CLEAN_FILES = +CONFIG_CLEAN_VPATH_FILES = +am__vpath_adj_setup = srcdirstrip=`echo "$(srcdir)" | sed 's|.|.|g'`; +am__vpath_adj = case $$p in \ + $(srcdir)/*) f=`echo "$$p" | sed "s|^$$srcdirstrip/||"`;; \ + *) f=$$p;; \ + esac; +am__strip_dir = f=`echo $$p | sed -e 's|^.*/||'`; +am__install_max = 40 +am__nobase_strip_setup = \ + srcdirstrip=`echo "$(srcdir)" | sed 's/[].[^$$\\*|]/\\\\&/g'` +am__nobase_strip = \ + for p in $$list; do echo "$$p"; done | sed -e "s|$$srcdirstrip/||" +am__nobase_list = $(am__nobase_strip_setup); \ + for p in $$list; do echo "$$p $$p"; done | \ + sed "s| $$srcdirstrip/| |;"' / .*\//!s/ .*/ ./; s,\( .*\)/[^/]*$$,\1,' | \ + $(AWK) 'BEGIN { files["."] = "" } { files[$$2] = files[$$2] " " $$1; \ + if (++n[$$2] == $(am__install_max)) \ + { print $$2, files[$$2]; n[$$2] = 0; files[$$2] = "" } } \ + END { for (dir in files) print dir, files[dir] }' +am__base_list = \ + sed '$$!N;$$!N;$$!N;$$!N;$$!N;$$!N;$$!N;s/\n/ /g' | \ + sed '$$!N;$$!N;$$!N;$$!N;s/\n/ /g' +am__uninstall_files_from_dir = { \ + test -z "$$files" \ + || { test ! -d "$$dir" && test ! -f "$$dir" && test ! -r "$$dir"; } \ + || { echo " ( cd '$$dir' && rm -f" $$files ")"; \ + $(am__cd) "$$dir" && rm -f $$files; }; \ + } +am__installdirs = "$(DESTDIR)$(pythondir)" "$(DESTDIR)$(pythondir)" \ + "$(DESTDIR)$(pythonmodulesdir)" "$(DESTDIR)$(pythonyaml2dir)" \ + "$(DESTDIR)$(pythonyaml3dir)" +SCRIPTS = $(dist_python_SCRIPTS) +AM_V_P = $(am__v_P_@AM_V@) +am__v_P_ = $(am__v_P_@AM_DEFAULT_V@) +am__v_P_0 = false +am__v_P_1 = : +AM_V_GEN = $(am__v_GEN_@AM_V@) +am__v_GEN_ = $(am__v_GEN_@AM_DEFAULT_V@) +am__v_GEN_0 = @echo " GEN " $@; +am__v_GEN_1 = +AM_V_at = $(am__v_at_@AM_V@) +am__v_at_ = $(am__v_at_@AM_DEFAULT_V@) +am__v_at_0 = @ +am__v_at_1 = +SOURCES = +DIST_SOURCES = +am__can_run_installinfo = \ + case $$AM_UPDATE_INFO_DIR in \ + n|no|NO) false;; \ + *) (install-info --version) >/dev/null 2>&1;; \ + esac +DATA = $(dist_python_DATA) $(dist_pythonmodules_DATA) \ + $(dist_pythonyaml2_DATA) $(dist_pythonyaml3_DATA) +am__tagged_files = $(HEADERS) $(SOURCES) $(TAGS_FILES) $(LISP) +DISTFILES = $(DIST_COMMON) $(DIST_SOURCES) $(TEXINFOS) $(EXTRA_DIST) +ACLOCAL = @ACLOCAL@ +AMTAR = @AMTAR@ +AM_DEFAULT_VERBOSITY = @AM_DEFAULT_VERBOSITY@ +AUTOCONF = @AUTOCONF@ +AUTOHEADER = @AUTOHEADER@ +AUTOMAKE = @AUTOMAKE@ +AWK = @AWK@ +CC = @CC@ +CCDEPMODE = @CCDEPMODE@ +CFLAGS = @CFLAGS@ +CPP = @CPP@ +CPPFLAGS = @CPPFLAGS@ +CYGPATH_W = @CYGPATH_W@ +DEFS = @DEFS@ +DEPDIR = @DEPDIR@ +ECHO_C = @ECHO_C@ +ECHO_N = @ECHO_N@ +ECHO_T = @ECHO_T@ +EGREP = @EGREP@ +EXEEXT = @EXEEXT@ +GREP = @GREP@ +INSTALL = @INSTALL@ +INSTALL_DATA = @INSTALL_DATA@ +INSTALL_PROGRAM = @INSTALL_PROGRAM@ +INSTALL_SCRIPT = @INSTALL_SCRIPT@ +INSTALL_STRIP_PROGRAM = @INSTALL_STRIP_PROGRAM@ +LDFLAGS = @LDFLAGS@ +LIBMNL_CFLAGS = @LIBMNL_CFLAGS@ +LIBMNL_LIBS = @LIBMNL_LIBS@ +LIBOBJS = @LIBOBJS@ +LIBS = @LIBS@ +LTLIBOBJS = @LTLIBOBJS@ +MAINT = @MAINT@ +MAKEINFO = @MAKEINFO@ +MATH_CFLAGS = @MATH_CFLAGS@ +MATH_LIBS = @MATH_LIBS@ +MKDIR_P = @MKDIR_P@ +NFACCT_CFLAGS = @NFACCT_CFLAGS@ +NFACCT_LIBS = @NFACCT_LIBS@ +OBJEXT = @OBJEXT@ +OPTIONAL_MATH_CLFAGS = @OPTIONAL_MATH_CLFAGS@ +OPTIONAL_MATH_LIBS = @OPTIONAL_MATH_LIBS@ +OPTIONAL_NFACCT_CLFAGS = @OPTIONAL_NFACCT_CLFAGS@ +OPTIONAL_NFACCT_LIBS = @OPTIONAL_NFACCT_LIBS@ +OPTIONAL_UUID_CLFAGS = @OPTIONAL_UUID_CLFAGS@ +OPTIONAL_UUID_LIBS = @OPTIONAL_UUID_LIBS@ +OPTIONAL_ZLIB_CLFAGS = @OPTIONAL_ZLIB_CLFAGS@ +OPTIONAL_ZLIB_LIBS = @OPTIONAL_ZLIB_LIBS@ +PACKAGE = @PACKAGE@ +PACKAGE_BUGREPORT = @PACKAGE_BUGREPORT@ +PACKAGE_NAME = @PACKAGE_NAME@ +PACKAGE_RPM_RELEASE = @PACKAGE_RPM_RELEASE@ +PACKAGE_RPM_VERSION = @PACKAGE_RPM_VERSION@ +PACKAGE_STRING = @PACKAGE_STRING@ +PACKAGE_TARNAME = @PACKAGE_TARNAME@ +PACKAGE_URL = @PACKAGE_URL@ +PACKAGE_VERSION = @PACKAGE_VERSION@ +PATH_SEPARATOR = @PATH_SEPARATOR@ +PKG_CONFIG = @PKG_CONFIG@ +PKG_CONFIG_LIBDIR = @PKG_CONFIG_LIBDIR@ +PKG_CONFIG_PATH = @PKG_CONFIG_PATH@ +PTHREAD_CC = @PTHREAD_CC@ +PTHREAD_CFLAGS = @PTHREAD_CFLAGS@ +PTHREAD_LIBS = @PTHREAD_LIBS@ +SET_MAKE = @SET_MAKE@ +SHELL = @SHELL@ +STRIP = @STRIP@ +UUID_CFLAGS = @UUID_CFLAGS@ +UUID_LIBS = @UUID_LIBS@ +VERSION = @VERSION@ +ZLIB_CFLAGS = @ZLIB_CFLAGS@ +ZLIB_LIBS = @ZLIB_LIBS@ +abs_builddir = @abs_builddir@ +abs_srcdir = @abs_srcdir@ +abs_top_builddir = @abs_top_builddir@ +abs_top_srcdir = @abs_top_srcdir@ +ac_ct_CC = @ac_ct_CC@ +am__include = @am__include@ +am__leading_dot = @am__leading_dot@ +am__quote = @am__quote@ +am__tar = @am__tar@ +am__untar = @am__untar@ +ax_pthread_config = @ax_pthread_config@ +bindir = @bindir@ +build = @build@ +build_alias = @build_alias@ +build_cpu = @build_cpu@ +build_os = @build_os@ +build_vendor = @build_vendor@ +builddir = @builddir@ +cachedir = @cachedir@ +chartsdir = @chartsdir@ +configdir = @configdir@ +datadir = @datadir@ +datarootdir = @datarootdir@ +docdir = @docdir@ +dvidir = @dvidir@ +exec_prefix = @exec_prefix@ +host = @host@ +host_alias = @host_alias@ +host_cpu = @host_cpu@ +host_os = @host_os@ +host_vendor = @host_vendor@ +htmldir = @htmldir@ +includedir = @includedir@ +infodir = @infodir@ +install_sh = @install_sh@ +libdir = @libdir@ +libexecdir = @libexecdir@ +localedir = @localedir@ +localstatedir = @localstatedir@ +logdir = @logdir@ +mandir = @mandir@ +mkdir_p = @mkdir_p@ +nodedir = @nodedir@ +oldincludedir = @oldincludedir@ +pdfdir = @pdfdir@ +pluginsdir = @pluginsdir@ +prefix = @prefix@ +program_transform_name = @program_transform_name@ +psdir = @psdir@ +pythondir = @pythondir@ +registrydir = @registrydir@ +sbindir = @sbindir@ +sharedstatedir = @sharedstatedir@ +srcdir = @srcdir@ +sysconfdir = @sysconfdir@ +target_alias = @target_alias@ +top_build_prefix = @top_build_prefix@ +top_builddir = @top_builddir@ +top_srcdir = @top_srcdir@ +varlibdir = @varlibdir@ +webdir = @webdir@ +MAINTAINERCLEANFILES = $(srcdir)/Makefile.in +CLEANFILES = \ + python-modules-installer.sh \ + $(NULL) + +SUFFIXES = .in +dist_python_SCRIPTS = \ + apache.chart.py \ + apache_cache.chart.py \ + cpufreq.chart.py \ + dovecot.chart.py \ + example.chart.py \ + exim.chart.py \ + hddtemp.chart.py \ + ipfs.chart.py \ + memcached.chart.py \ + mysql.chart.py \ + nginx.chart.py \ + nginx_log.chart.py \ + phpfpm.chart.py \ + postfix.chart.py \ + redis.chart.py \ + sensors.chart.py \ + squid.chart.py \ + tomcat.chart.py \ + python-modules-installer.sh \ + $(NULL) + +dist_python_DATA = \ + README.md \ + $(NULL) + +pythonmodulesdir = $(pythondir)/python_modules +dist_pythonmodules_DATA = \ + python_modules/__init__.py \ + python_modules/base.py \ + python_modules/msg.py \ + python_modules/lm_sensors.py \ + $(NULL) + +pythonyaml2dir = $(pythonmodulesdir)/pyyaml2 +dist_pythonyaml2_DATA = \ + python_modules/pyyaml2/__init__.py \ + python_modules/pyyaml2/composer.py \ + python_modules/pyyaml2/constructor.py \ + python_modules/pyyaml2/cyaml.py \ + python_modules/pyyaml2/dumper.py \ + python_modules/pyyaml2/emitter.py \ + python_modules/pyyaml2/error.py \ + python_modules/pyyaml2/events.py \ + python_modules/pyyaml2/loader.py \ + python_modules/pyyaml2/nodes.py \ + python_modules/pyyaml2/parser.py \ + python_modules/pyyaml2/reader.py \ + python_modules/pyyaml2/representer.py \ + python_modules/pyyaml2/resolver.py \ + python_modules/pyyaml2/scanner.py \ + python_modules/pyyaml2/serializer.py \ + python_modules/pyyaml2/tokens.py \ + $(NULL) + +pythonyaml3dir = $(pythonmodulesdir)/pyyaml3 +dist_pythonyaml3_DATA = \ + python_modules/pyyaml3/__init__.py \ + python_modules/pyyaml3/composer.py \ + python_modules/pyyaml3/constructor.py \ + python_modules/pyyaml3/cyaml.py \ + python_modules/pyyaml3/dumper.py \ + python_modules/pyyaml3/emitter.py \ + python_modules/pyyaml3/error.py \ + python_modules/pyyaml3/events.py \ + python_modules/pyyaml3/loader.py \ + python_modules/pyyaml3/nodes.py \ + python_modules/pyyaml3/parser.py \ + python_modules/pyyaml3/reader.py \ + python_modules/pyyaml3/representer.py \ + python_modules/pyyaml3/resolver.py \ + python_modules/pyyaml3/scanner.py \ + python_modules/pyyaml3/serializer.py \ + python_modules/pyyaml3/tokens.py \ + $(NULL) + +all: all-am + +.SUFFIXES: +.SUFFIXES: .in +$(srcdir)/Makefile.in: @MAINTAINER_MODE_TRUE@ $(srcdir)/Makefile.am $(top_srcdir)/build/subst.inc $(am__configure_deps) + @for dep in $?; do \ + case '$(am__configure_deps)' in \ + *$$dep*) \ + ( cd $(top_builddir) && $(MAKE) $(AM_MAKEFLAGS) am--refresh ) \ + && { if test -f $@; then exit 0; else break; fi; }; \ + exit 1;; \ + esac; \ + done; \ + echo ' cd $(top_srcdir) && $(AUTOMAKE) --gnu python.d/Makefile'; \ + $(am__cd) $(top_srcdir) && \ + $(AUTOMAKE) --gnu python.d/Makefile +.PRECIOUS: Makefile +Makefile: $(srcdir)/Makefile.in $(top_builddir)/config.status + @case '$?' in \ + *config.status*) \ + cd $(top_builddir) && $(MAKE) $(AM_MAKEFLAGS) am--refresh;; \ + *) \ + echo ' cd $(top_builddir) && $(SHELL) ./config.status $(subdir)/$@ $(am__depfiles_maybe)'; \ + cd $(top_builddir) && $(SHELL) ./config.status $(subdir)/$@ $(am__depfiles_maybe);; \ + esac; +$(top_srcdir)/build/subst.inc: + +$(top_builddir)/config.status: $(top_srcdir)/configure $(CONFIG_STATUS_DEPENDENCIES) + cd $(top_builddir) && $(MAKE) $(AM_MAKEFLAGS) am--refresh + +$(top_srcdir)/configure: @MAINTAINER_MODE_TRUE@ $(am__configure_deps) + cd $(top_builddir) && $(MAKE) $(AM_MAKEFLAGS) am--refresh +$(ACLOCAL_M4): @MAINTAINER_MODE_TRUE@ $(am__aclocal_m4_deps) + cd $(top_builddir) && $(MAKE) $(AM_MAKEFLAGS) am--refresh +$(am__aclocal_m4_deps): +install-dist_pythonSCRIPTS: $(dist_python_SCRIPTS) + @$(NORMAL_INSTALL) + @list='$(dist_python_SCRIPTS)'; test -n "$(pythondir)" || list=; \ + if test -n "$$list"; then \ + echo " $(MKDIR_P) '$(DESTDIR)$(pythondir)'"; \ + $(MKDIR_P) "$(DESTDIR)$(pythondir)" || exit 1; \ + fi; \ + for p in $$list; do \ + if test -f "$$p"; then d=; else d="$(srcdir)/"; fi; \ + if test -f "$$d$$p"; then echo "$$d$$p"; echo "$$p"; else :; fi; \ + done | \ + sed -e 'p;s,.*/,,;n' \ + -e 'h;s|.*|.|' \ + -e 'p;x;s,.*/,,;$(transform)' | sed 'N;N;N;s,\n, ,g' | \ + $(AWK) 'BEGIN { files["."] = ""; dirs["."] = 1; } \ + { d=$$3; if (dirs[d] != 1) { print "d", d; dirs[d] = 1 } \ + if ($$2 == $$4) { files[d] = files[d] " " $$1; \ + if (++n[d] == $(am__install_max)) { \ + print "f", d, files[d]; n[d] = 0; files[d] = "" } } \ + else { print "f", d "/" $$4, $$1 } } \ + END { for (d in files) print "f", d, files[d] }' | \ + while read type dir files; do \ + if test "$$dir" = .; then dir=; else dir=/$$dir; fi; \ + test -z "$$files" || { \ + echo " $(INSTALL_SCRIPT) $$files '$(DESTDIR)$(pythondir)$$dir'"; \ + $(INSTALL_SCRIPT) $$files "$(DESTDIR)$(pythondir)$$dir" || exit $$?; \ + } \ + ; done + +uninstall-dist_pythonSCRIPTS: + @$(NORMAL_UNINSTALL) + @list='$(dist_python_SCRIPTS)'; test -n "$(pythondir)" || exit 0; \ + files=`for p in $$list; do echo "$$p"; done | \ + sed -e 's,.*/,,;$(transform)'`; \ + dir='$(DESTDIR)$(pythondir)'; $(am__uninstall_files_from_dir) +install-dist_pythonDATA: $(dist_python_DATA) + @$(NORMAL_INSTALL) + @list='$(dist_python_DATA)'; test -n "$(pythondir)" || list=; \ + if test -n "$$list"; then \ + echo " $(MKDIR_P) '$(DESTDIR)$(pythondir)'"; \ + $(MKDIR_P) "$(DESTDIR)$(pythondir)" || exit 1; \ + fi; \ + for p in $$list; do \ + if test -f "$$p"; then d=; else d="$(srcdir)/"; fi; \ + echo "$$d$$p"; \ + done | $(am__base_list) | \ + while read files; do \ + echo " $(INSTALL_DATA) $$files '$(DESTDIR)$(pythondir)'"; \ + $(INSTALL_DATA) $$files "$(DESTDIR)$(pythondir)" || exit $$?; \ + done + +uninstall-dist_pythonDATA: + @$(NORMAL_UNINSTALL) + @list='$(dist_python_DATA)'; test -n "$(pythondir)" || list=; \ + files=`for p in $$list; do echo $$p; done | sed -e 's|^.*/||'`; \ + dir='$(DESTDIR)$(pythondir)'; $(am__uninstall_files_from_dir) +install-dist_pythonmodulesDATA: $(dist_pythonmodules_DATA) + @$(NORMAL_INSTALL) + @list='$(dist_pythonmodules_DATA)'; test -n "$(pythonmodulesdir)" || list=; \ + if test -n "$$list"; then \ + echo " $(MKDIR_P) '$(DESTDIR)$(pythonmodulesdir)'"; \ + $(MKDIR_P) "$(DESTDIR)$(pythonmodulesdir)" || exit 1; \ + fi; \ + for p in $$list; do \ + if test -f "$$p"; then d=; else d="$(srcdir)/"; fi; \ + echo "$$d$$p"; \ + done | $(am__base_list) | \ + while read files; do \ + echo " $(INSTALL_DATA) $$files '$(DESTDIR)$(pythonmodulesdir)'"; \ + $(INSTALL_DATA) $$files "$(DESTDIR)$(pythonmodulesdir)" || exit $$?; \ + done + +uninstall-dist_pythonmodulesDATA: + @$(NORMAL_UNINSTALL) + @list='$(dist_pythonmodules_DATA)'; test -n "$(pythonmodulesdir)" || list=; \ + files=`for p in $$list; do echo $$p; done | sed -e 's|^.*/||'`; \ + dir='$(DESTDIR)$(pythonmodulesdir)'; $(am__uninstall_files_from_dir) +install-dist_pythonyaml2DATA: $(dist_pythonyaml2_DATA) + @$(NORMAL_INSTALL) + @list='$(dist_pythonyaml2_DATA)'; test -n "$(pythonyaml2dir)" || list=; \ + if test -n "$$list"; then \ + echo " $(MKDIR_P) '$(DESTDIR)$(pythonyaml2dir)'"; \ + $(MKDIR_P) "$(DESTDIR)$(pythonyaml2dir)" || exit 1; \ + fi; \ + for p in $$list; do \ + if test -f "$$p"; then d=; else d="$(srcdir)/"; fi; \ + echo "$$d$$p"; \ + done | $(am__base_list) | \ + while read files; do \ + echo " $(INSTALL_DATA) $$files '$(DESTDIR)$(pythonyaml2dir)'"; \ + $(INSTALL_DATA) $$files "$(DESTDIR)$(pythonyaml2dir)" || exit $$?; \ + done + +uninstall-dist_pythonyaml2DATA: + @$(NORMAL_UNINSTALL) + @list='$(dist_pythonyaml2_DATA)'; test -n "$(pythonyaml2dir)" || list=; \ + files=`for p in $$list; do echo $$p; done | sed -e 's|^.*/||'`; \ + dir='$(DESTDIR)$(pythonyaml2dir)'; $(am__uninstall_files_from_dir) +install-dist_pythonyaml3DATA: $(dist_pythonyaml3_DATA) + @$(NORMAL_INSTALL) + @list='$(dist_pythonyaml3_DATA)'; test -n "$(pythonyaml3dir)" || list=; \ + if test -n "$$list"; then \ + echo " $(MKDIR_P) '$(DESTDIR)$(pythonyaml3dir)'"; \ + $(MKDIR_P) "$(DESTDIR)$(pythonyaml3dir)" || exit 1; \ + fi; \ + for p in $$list; do \ + if test -f "$$p"; then d=; else d="$(srcdir)/"; fi; \ + echo "$$d$$p"; \ + done | $(am__base_list) | \ + while read files; do \ + echo " $(INSTALL_DATA) $$files '$(DESTDIR)$(pythonyaml3dir)'"; \ + $(INSTALL_DATA) $$files "$(DESTDIR)$(pythonyaml3dir)" || exit $$?; \ + done + +uninstall-dist_pythonyaml3DATA: + @$(NORMAL_UNINSTALL) + @list='$(dist_pythonyaml3_DATA)'; test -n "$(pythonyaml3dir)" || list=; \ + files=`for p in $$list; do echo $$p; done | sed -e 's|^.*/||'`; \ + dir='$(DESTDIR)$(pythonyaml3dir)'; $(am__uninstall_files_from_dir) +tags TAGS: + +ctags CTAGS: + +cscope cscopelist: + + +distdir: $(DISTFILES) + @srcdirstrip=`echo "$(srcdir)" | sed 's/[].[^$$\\*]/\\\\&/g'`; \ + topsrcdirstrip=`echo "$(top_srcdir)" | sed 's/[].[^$$\\*]/\\\\&/g'`; \ + list='$(DISTFILES)'; \ + dist_files=`for file in $$list; do echo $$file; done | \ + sed -e "s|^$$srcdirstrip/||;t" \ + -e "s|^$$topsrcdirstrip/|$(top_builddir)/|;t"`; \ + case $$dist_files in \ + */*) $(MKDIR_P) `echo "$$dist_files" | \ + sed '/\//!d;s|^|$(distdir)/|;s,/[^/]*$$,,' | \ + sort -u` ;; \ + esac; \ + for file in $$dist_files; do \ + if test -f $$file || test -d $$file; then d=.; else d=$(srcdir); fi; \ + if test -d $$d/$$file; then \ + dir=`echo "/$$file" | sed -e 's,/[^/]*$$,,'`; \ + if test -d "$(distdir)/$$file"; then \ + find "$(distdir)/$$file" -type d ! -perm -700 -exec chmod u+rwx {} \;; \ + fi; \ + if test -d $(srcdir)/$$file && test $$d != $(srcdir); then \ + cp -fpR $(srcdir)/$$file "$(distdir)$$dir" || exit 1; \ + find "$(distdir)/$$file" -type d ! -perm -700 -exec chmod u+rwx {} \;; \ + fi; \ + cp -fpR $$d/$$file "$(distdir)$$dir" || exit 1; \ + else \ + test -f "$(distdir)/$$file" \ + || cp -p $$d/$$file "$(distdir)/$$file" \ + || exit 1; \ + fi; \ + done +check-am: all-am +check: check-am +all-am: Makefile $(SCRIPTS) $(DATA) +installdirs: + for dir in "$(DESTDIR)$(pythondir)" "$(DESTDIR)$(pythondir)" "$(DESTDIR)$(pythonmodulesdir)" "$(DESTDIR)$(pythonyaml2dir)" "$(DESTDIR)$(pythonyaml3dir)"; do \ + test -z "$$dir" || $(MKDIR_P) "$$dir"; \ + done +install: install-am +install-exec: install-exec-am +install-data: install-data-am +uninstall: uninstall-am + +install-am: all-am + @$(MAKE) $(AM_MAKEFLAGS) install-exec-am install-data-am + +installcheck: installcheck-am +install-strip: + if test -z '$(STRIP)'; then \ + $(MAKE) $(AM_MAKEFLAGS) INSTALL_PROGRAM="$(INSTALL_STRIP_PROGRAM)" \ + install_sh_PROGRAM="$(INSTALL_STRIP_PROGRAM)" INSTALL_STRIP_FLAG=-s \ + install; \ + else \ + $(MAKE) $(AM_MAKEFLAGS) INSTALL_PROGRAM="$(INSTALL_STRIP_PROGRAM)" \ + install_sh_PROGRAM="$(INSTALL_STRIP_PROGRAM)" INSTALL_STRIP_FLAG=-s \ + "INSTALL_PROGRAM_ENV=STRIPPROG='$(STRIP)'" install; \ + fi +mostlyclean-generic: + +clean-generic: + -test -z "$(CLEANFILES)" || rm -f $(CLEANFILES) + +distclean-generic: + -test -z "$(CONFIG_CLEAN_FILES)" || rm -f $(CONFIG_CLEAN_FILES) + -test . = "$(srcdir)" || test -z "$(CONFIG_CLEAN_VPATH_FILES)" || rm -f $(CONFIG_CLEAN_VPATH_FILES) + +maintainer-clean-generic: + @echo "This command is intended for maintainers to use" + @echo "it deletes files that may require special tools to rebuild." + -test -z "$(MAINTAINERCLEANFILES)" || rm -f $(MAINTAINERCLEANFILES) +clean: clean-am + +clean-am: clean-generic mostlyclean-am + +distclean: distclean-am + -rm -f Makefile +distclean-am: clean-am distclean-generic + +dvi: dvi-am + +dvi-am: + +html: html-am + +html-am: + +info: info-am + +info-am: + +install-data-am: install-dist_pythonDATA install-dist_pythonSCRIPTS \ + install-dist_pythonmodulesDATA install-dist_pythonyaml2DATA \ + install-dist_pythonyaml3DATA + +install-dvi: install-dvi-am + +install-dvi-am: + +install-exec-am: + +install-html: install-html-am + +install-html-am: + +install-info: install-info-am + +install-info-am: + +install-man: + +install-pdf: install-pdf-am + +install-pdf-am: + +install-ps: install-ps-am + +install-ps-am: + +installcheck-am: + +maintainer-clean: maintainer-clean-am + -rm -f Makefile +maintainer-clean-am: distclean-am maintainer-clean-generic + +mostlyclean: mostlyclean-am + +mostlyclean-am: mostlyclean-generic + +pdf: pdf-am + +pdf-am: + +ps: ps-am + +ps-am: + +uninstall-am: uninstall-dist_pythonDATA uninstall-dist_pythonSCRIPTS \ + uninstall-dist_pythonmodulesDATA \ + uninstall-dist_pythonyaml2DATA uninstall-dist_pythonyaml3DATA + +.MAKE: install-am install-strip + +.PHONY: all all-am check check-am clean clean-generic cscopelist-am \ + ctags-am distclean distclean-generic distdir dvi dvi-am html \ + html-am info info-am install install-am install-data \ + install-data-am install-dist_pythonDATA \ + install-dist_pythonSCRIPTS install-dist_pythonmodulesDATA \ + install-dist_pythonyaml2DATA install-dist_pythonyaml3DATA \ + install-dvi install-dvi-am install-exec install-exec-am \ + install-html install-html-am install-info install-info-am \ + install-man install-pdf install-pdf-am install-ps \ + install-ps-am install-strip installcheck installcheck-am \ + installdirs maintainer-clean maintainer-clean-generic \ + mostlyclean mostlyclean-generic pdf pdf-am ps ps-am tags-am \ + uninstall uninstall-am uninstall-dist_pythonDATA \ + uninstall-dist_pythonSCRIPTS uninstall-dist_pythonmodulesDATA \ + uninstall-dist_pythonyaml2DATA uninstall-dist_pythonyaml3DATA + +.in: + if sed \ + -e 's#[@]localstatedir_POST@#$(localstatedir)#g' \ + -e 's#[@]sbindir_POST@#$(sbindir)#g' \ + -e 's#[@]sysconfdir_POST@#$(sysconfdir)#g' \ + -e 's#[@]pythondir_POST@#$(pythondir)#g' \ + $< > $@.tmp; then \ + mv "$@.tmp" "$@"; \ + else \ + rm -f "$@.tmp"; \ + false; \ + fi + +# Tell versions [3.59,3.63) of GNU make to not export all variables. +# Otherwise a system limit (for SysV at least) may be exceeded. +.NOEXPORT: diff --git a/python.d/README.md b/python.d/README.md new file mode 100644 index 000000000..4cc4d8d2f --- /dev/null +++ b/python.d/README.md @@ -0,0 +1,698 @@ +# Disclaimer + +Every module should be compatible with python2 and python3. +All third party libraries should be installed system-wide or in `python_modules` directory. +Module configurations are written in YAML and **pyYAML is required**. + +Every configuration file must have one of two formats: + +- Configuration for only one job: + +```yaml +update_every : 2 # update frequency +retries : 1 # how many failures in update() is tolerated +priority : 20000 # where it is shown on dashboard + +other_var1 : bla # variables passed to module +other_var2 : alb +``` + +- Configuration for many jobs (ex. mysql): + +```yaml +# module defaults: +update_every : 2 +retries : 1 +priority : 20000 + +local: # job name + update_every : 5 # job update frequency + other_var1 : some_val # module specific variable + +other_job: + priority : 5 # job position on dashboard + retries : 20 # job retries + other_var2 : val # module specific variable +``` + +`update_every`, `retries`, and `priority` are always optional. + +--- + +The following python.d modules are supported: + +# apache + +This module will monitor one or more apache servers depending on configuration. + +**Requirements:** + * apache with enabled `mod_status` + +It produces following charts: + +1. **Requests** in requests/s + * requests + +2. **Connections** + * connections + +3. **Async Connections** + * keepalive + * closing + * writing + +4. **Bandwidth** in kilobytes/s + * sent + +5. **Workers** + * idle + * busy + +6. **Lifetime Avg. Requests/s** in requests/s + * requests_sec + +7. **Lifetime Avg. Bandwidth/s** in kilobytes/s + * size_sec + +8. **Lifetime Avg. Response Size** in bytes/request + * size_req + +### configuration + +Needs only `url` to server's `server-status?auto` + +Here is an example for 2 servers: + +```yaml +update_every : 10 +priority : 90100 + +local: + url : 'http://localhost/server-status?auto' + retries : 20 + +remote: + url : 'http://www.apache.org/server-status?auto' + update_every : 5 + retries : 4 +``` + +Without configuration, module attempts to connect to `http://localhost/server-status?auto` + +--- + +# apache_cache + +Module monitors apache mod_cache log and produces only one chart: + +**cached responses** in percent cached + * hit + * miss + * other + +### configuration + +Sample: + +```yaml +update_every : 10 +priority : 120000 +retries : 5 +log_path : '/var/log/apache2/cache.log' +``` + +If no configuration is given, module will attempt to read log file at `/var/log/apache2/cache.log` + +--- + +# cpufreq + +Module shows current cpu frequency by looking at appropriate files in /sys/devices + +**Requirement:** +Processor which presents data scaling frequency data + +It produces one chart with multiple lines (one line per core). + +### configuration + +Sample: + +```yaml +sys_dir: "/sys/devices" +``` + +If no configuration is given, module will search for `scaling_cur_freq` files in `/sys/devices` directory. +Directory is also prefixed with `NETDATA_HOST_PREFIX` if specified. + +--- + +# dovecot + +This module provides statistics information from dovecot server. +Statistics are taken from dovecot socket by executing `EXPORT global` command. +More information about dovecot stats can be found on [project wiki page.](http://wiki2.dovecot.org/Statistics) + +**Requirement:** +Dovecot unix socket with R/W permissions for user netdata or dovecot with configured TCP/IP socket. + +Module gives information with following charts: + +1. **logins and sessions** + * logins + * active sessions + +2. **commands** - number of IMAP commands + * commands + +3. **Faults** + * minor + * major + +4. **Context Switches** + * volountary + * involountary + +5. **disk** in bytes/s + * read + * write + +6. **bytes** in bytes/s + * read + * write + +7. **number of syscalls** in syscalls/s + * read + * write + +8. **lookups** - number of lookups per second + * path + * attr + +9. **hits** - number of cache hits + * hits + +10. **attempts** - authorization attemts + * success + * failure + +11. **cache** - cached authorization hits + * hit + * miss + +### configuration + +Sample: + +```yaml +localtcpip: + name : 'local' + host : '127.0.0.1' + port : 24242 + +localsocket: + name : 'local' + socket : '/var/run/dovecot/stats' +``` + +If no configuration is given, module will attempt to connect to dovecot using unix socket localized in `/var/run/dovecot/stats` + +--- + +# exim + +Simple module executing `exim -bpc` to grab exim queue. +This command can take a lot of time to finish its execution thus it is not recommended to run it every second. + +It produces only one chart: + +1. **Exim Queue Emails** + * emails + +Configuration is not needed. + +--- + +# hddtemp + +Module monitors disk temperatures from one or more hddtemp daemons + +**Requirement:** +Running `hddtemp` in daemonized mode with access on tcp port + +It produces one chart **Temperature** with dynamic number of dimensions (one per disk) + +### configuration + +Sample: + +```yaml +update_every: 3 +host: "127.0.0.1" +port: 7634 +``` + +If no configuration is given, module will attempt to connect to hddtemp daemon on `127.0.0.1:7634` address + +--- + +# IPFS + +Module monitors [IPFS](https://ipfs.io) basic information. + +1. **Bandwidth** in kbits/s + * in + * out + +2. **Peers** + * peers + +### configuration + +Only url to IPFS server is needed. + +Sample: + +```yaml +localhost: + name : 'local' + url : 'http://localhost:5001' +``` + +--- + +# memcached + +Memcached monitoring module. Data grabbed from [stats interface](https://github.com/memcached/memcached/wiki/Commands#stats). + +1. **Network** in kilobytes/s + * read + * written + +2. **Connections** per second + * current + * rejected + * total + +3. **Items** in cluster + * current + * total + +4. **Evicted and Reclaimed** items + * evicted + * reclaimed + +5. **GET** requests/s + * hits + * misses + +6. **GET rate** rate in requests/s + * rate + +7. **SET rate** rate in requests/s + * rate + +8. **DELETE** requests/s + * hits + * misses + +9. **CAS** requests/s + * hits + * misses + * bad value + +10. **Increment** requests/s + * hits + * misses + +11. **Decrement** requests/s + * hits + * misses + +12. **Touch** requests/s + * hits + * misses + +13. **Touch rate** rate in requests/s + * rate + +### configuration + +Sample: + +```yaml +localtcpip: + name : 'local' + host : '127.0.0.1' + port : 24242 +``` + +If no configuration is given, module will attempt to connect to memcached instance on `127.0.0.1:11211` address. + +--- + +# mysql + +Module monitors one or more mysql servers + +**Requirements:** + * python library [MySQLdb](https://github.com/PyMySQL/mysqlclient-python) (faster) or [PyMySQL](https://github.com/PyMySQL/PyMySQL) (slower) + +It will produce following charts (if data is available): + +1. **Bandwidth** in kbps + * in + * out + +2. **Queries** in queries/sec + * queries + * questions + * slow queries + +3. **Operations** in operations/sec + * opened tables + * flush + * commit + * delete + * prepare + * read first + * read key + * read next + * read prev + * read random + * read random next + * rollback + * save point + * update + * write + +4. **Table Locks** in locks/sec + * immediate + * waited + +5. **Select Issues** in issues/sec + * full join + * full range join + * range + * range check + * scan + +6. **Sort Issues** in issues/sec + * merge passes + * range + * scan + +### configuration + +You can provide, per server, the following: + +1. username which have access to database (deafults to 'root') +2. password (defaults to none) +3. mysql my.cnf configuration file +4. mysql socket (optional) +5. mysql host (ip or hostname) +6. mysql port (defaults to 3306) + +Here is an example for 3 servers: + +```yaml +update_every : 10 +priority : 90100 +retries : 5 + +local: + 'my.cnf' : '/etc/mysql/my.cnf' + priority : 90000 + +local_2: + user : 'root' + pass : 'blablablabla' + socket : '/var/run/mysqld/mysqld.sock' + update_every : 1 + +remote: + user : 'admin' + pass : 'bla' + host : 'example.org' + port : 9000 + retries : 20 +``` + +If no configuration is given, module will attempt to connect to mysql server via unix socket at `/var/run/mysqld/mysqld.sock` without password and with username `root` + +--- + +# nginx + +This module will monitor one or more nginx servers depending on configuration. + +**Requirements:** + * nginx with configured `stub_status` + +It produces following charts: + +1. **Active Connections** + * active + +2. **Requests** in requests/s + * requests + +3. **Active Connections by Status** + * reading + * writing + * waiting + +4. **Connections Rate** in connections/s + * accepts + * handled + +### configuration + +Needs only `url` to server's `stub_status` + +Here is an example for local server: + +```yaml +update_every : 10 +priority : 90100 + +local: + url : 'http://localhost/stub_status' + retries : 10 +``` + +Without configuration, module attempts to connect to `http://localhost/stub_status` + +--- + +# nginx_log + +Module monitors nginx access log and produces only one chart: + +1. **nginx status codes** in requests/s + * 2xx + * 3xx + * 4xx + * 5xx + +### configuration + +Sample for two vhosts: + +```yaml +site_A: + path: '/var/log/nginx/access-A.log' + +site_B: + name: 'local' + path: '/var/log/nginx/access-B.log' +``` + +When no configuration file is found, module tries to parse `/var/log/nginx/access.log` file. + +--- + +# phpfpm + +This module will monitor one or more php-fpm instances depending on configuration. + +**Requirements:** + * php-fpm with enabled `status` page + * access to `status` page via web server + +It produces following charts: + +1. **Active Connections** + * active + * maxActive + * idle + +2. **Requests** in requests/s + * requests + +3. **Performance** + * reached + * slow + +### configuration + +Needs only `url` to server's `status` + +Here is an example for local instance: + +```yaml +update_every : 3 +priority : 90100 + +local: + url : 'http://localhost/status' + retries : 10 +``` + +Without configuration, module attempts to connect to `http://localhost/status` + +--- + +# postfix + +Simple module executing `postfix -p` to grab postfix queue. + +It produces only two charts: + +1. **Postfix Queue Emails** + * emails + +2. **Postfix Queue Emails Size** in KB + * size + +Configuration is not needed. + +--- + +# redis + +Get INFO data from redis instance. + +Following charts are drawn: + +1. **Operations** per second + * operations + +2. **Hit rate** in percent + * rate + +3. **Memory utilization** in kilobytes + * total + * lua + +4. **Database keys** + * lines are creates dynamically based on how many databases are there + +5. **Clients** + * connected + * blocked + +6. **Slaves** + * connected + +### configuration + +```yaml +socket: + name : 'local' + socket : '/var/lib/redis/redis.sock' + +localhost: + name : 'local' + host : 'localhost' + port : 6379 +``` + +When no configuration file is found, module tries to connect to TCP/IP socket: `localhost:6379`. + +--- + +# sensors + +System sensors information. + +Charts are created dynamically. + +### configuration + +For detailed configuration information please read [`sensors.conf`](https://github.com/firehol/netdata/blob/master/conf.d/python.d/sensors.conf) file. + +--- + +# squid + +This module will monitor one or more squid instances depending on configuration. + +It produces following charts: + +1. **Client Bandwidth** in kilobits/s + * in + * out + * hits + +2. **Client Requests** in requests/s + * requests + * hits + * errors + +3. **Server Bandwidth** in kilobits/s + * in + * out + +4. **Server Requests** in requests/s + * requests + * errors + +### configuration + +```yaml +priority : 50000 + +local: + request : 'cache_object://localhost:3128/counters' + host : 'localhost' + port : 3128 +``` + +Without any configuration module will try to autodetect where squid presents its `counters` data + +--- + +# tomcat + +Present tomcat containers memory utilization. + +Charts: + +1. **Requests** per second + * accesses + +2. **Volume** in KB/s + * volume + +3. **Threads** + * current + * busy + +4. **JVM Free Memory** in MB + * jvm + +### configuration + +```yaml +localhost: + name : 'local' + url : 'http://127.0.0.1:8080/manager/status?XML=true' + user : 'tomcat_username' + pass : 'secret_tomcat_password' +``` + +Without configuration, module attempts to connect to `http://localhost:8080/manager/status?XML=true`, without any credentials. +So it will probably fail. + +--- diff --git a/python.d/apache.chart.py b/python.d/apache.chart.py new file mode 100644 index 000000000..2e4d16dd3 --- /dev/null +++ b/python.d/apache.chart.py @@ -0,0 +1,108 @@ +# -*- coding: utf-8 -*- +# Description: apache netdata python.d module +# Author: Pawel Krupa (paulfantom) + +from base import UrlService + +# default module values (can be overridden per job in `config`) +# update_every = 2 +priority = 60000 +retries = 60 + +# default job configuration (overridden by python.d.plugin) +# config = {'local': { +# 'update_every': update_every, +# 'retries': retries, +# 'priority': priority, +# 'url': 'http://www.apache.org/server-status?auto' +# }} + +# charts order (can be overridden if you want less charts, or different order) +ORDER = ['requests', 'connections', 'conns_async', 'net', 'workers', 'reqpersec', 'bytespersec', 'bytesperreq'] + +CHARTS = { + 'bytesperreq': { + 'options': [None, 'apache Lifetime Avg. Response Size', 'bytes/request', 'statistics', 'apache.bytesperreq', 'area'], + 'lines': [ + ["size_req"] + ]}, + 'workers': { + 'options': [None, 'apache Workers', 'workers', 'workers', 'apache.workers', 'stacked'], + 'lines': [ + ["idle"], + ["busy"] + ]}, + 'reqpersec': { + 'options': [None, 'apache Lifetime Avg. Requests/s', 'requests/s', 'statistics', 'apache.reqpersec', 'area'], + 'lines': [ + ["requests_sec"] + ]}, + 'bytespersec': { + 'options': [None, 'apache Lifetime Avg. Bandwidth/s', 'kilobytes/s', 'statistics', 'apache.bytesperreq', 'area'], + 'lines': [ + ["size_sec", None, 'absolute', 1, 1000] + ]}, + 'requests': { + 'options': [None, 'apache Requests', 'requests/s', 'requests', 'apache.requests', 'line'], + 'lines': [ + ["requests", None, 'incremental'] + ]}, + 'net': { + 'options': [None, 'apache Bandwidth', 'kilobytes/s', 'bandwidth', 'apache.net', 'area'], + 'lines': [ + ["sent", None, 'incremental'] + ]}, + 'connections': { + 'options': [None, 'apache Connections', 'connections', 'connections', 'apache.connections', 'line'], + 'lines': [ + ["connections"] + ]}, + 'conns_async': { + 'options': [None, 'apache Async Connections', 'connections', 'connections', 'apache.conns_async', 'stacked'], + 'lines': [ + ["keepalive"], + ["closing"], + ["writing"] + ]} +} + + +class Service(UrlService): + def __init__(self, configuration=None, name=None): + UrlService.__init__(self, configuration=configuration, name=name) + if len(self.url) == 0: + self.url = "http://localhost/server-status?auto" + self.order = ORDER + self.definitions = CHARTS + self.assignment = {"BytesPerReq": 'size_req', + "IdleWorkers": 'idle', + "BusyWorkers": 'busy', + "ReqPerSec": 'requests_sec', + "BytesPerSec": 'size_sec', + "Total Accesses": 'requests', + "Total kBytes": 'sent', + "ConnsTotal": 'connections', + "ConnsAsyncKeepAlive": 'keepalive', + "ConnsAsyncClosing": 'closing', + "ConnsAsyncWriting": 'writing'} + + def _get_data(self): + """ + Format data received from http request + :return: dict + """ + try: + raw = self._get_raw_data().split("\n") + except AttributeError: + return None + data = {} + for row in raw: + tmp = row.split(":") + if str(tmp[0]) in self.assignment: + try: + data[self.assignment[tmp[0]]] = int(float(tmp[1])) + except (IndexError, ValueError): + pass + if len(data) == 0: + return None + return data diff --git a/python.d/apache_cache.chart.py b/python.d/apache_cache.chart.py new file mode 100644 index 000000000..3681a8511 --- /dev/null +++ b/python.d/apache_cache.chart.py @@ -0,0 +1,60 @@ +# -*- coding: utf-8 -*- +# Description: apache cache netdata python.d module +# Author: Pawel Krupa (paulfantom) + +from base import LogService + +priority = 60000 +retries = 60 +# update_every = 3 + +ORDER = ['cache'] +CHARTS = { + 'cache': { + 'options': [None, 'apache cached responses', 'percent cached', 'cached', 'apache_cache.cache', 'stacked'], + 'lines': [ + ["hit", 'cache', "percentage-of-absolute-row"], + ["miss", None, "percentage-of-absolute-row"], + ["other", None, "percentage-of-absolute-row"] + ]} +} + + +class Service(LogService): + def __init__(self, configuration=None, name=None): + LogService.__init__(self, configuration=configuration, name=name) + if len(self.log_path) == 0: + self.log_path = "/var/log/apache2/cache.log" + self.order = ORDER + self.definitions = CHARTS + + def _get_data(self): + """ + Parse new log lines + :return: dict + """ + try: + raw = self._get_raw_data() + if raw is None: + return None + elif not raw: + return {'hit': 0, + 'miss': 0, + 'other': 0} + except (ValueError, AttributeError): + return None + + hit = 0 + miss = 0 + other = 0 + for line in raw: + if "cache hit" in line: + hit += 1 + elif "cache miss" in line: + miss += 1 + else: + other += 1 + + return {'hit': hit, + 'miss': miss, + 'other': other} diff --git a/python.d/cpufreq.chart.py b/python.d/cpufreq.chart.py new file mode 100644 index 000000000..a9de5cedd --- /dev/null +++ b/python.d/cpufreq.chart.py @@ -0,0 +1,84 @@ +# -*- coding: utf-8 -*- +# Description: cpufreq netdata python.d module +# Author: Pawel Krupa (paulfantom) + +import os +from base import SimpleService + +# default module values (can be overridden per job in `config`) +# update_every = 2 + +ORDER = ['cpufreq'] + +CHARTS = { + 'cpufreq': { + 'options': [None, 'CPU Clock', 'MHz', 'cpufreq', None, 'line'], + 'lines': [ + # lines are created dynamically in `check()` method + ]} +} + + +class Service(SimpleService): + def __init__(self, configuration=None, name=None): + prefix = os.getenv('NETDATA_HOST_PREFIX', "") + if prefix.endswith('/'): + prefix = prefix[:-1] + self.sys_dir = prefix + "/sys/devices" + self.filename = "scaling_cur_freq" + SimpleService.__init__(self, configuration=configuration, name=name) + self.order = ORDER + self.definitions = CHARTS + self._orig_name = "" + self.assignment = {} + self.paths = [] + + def _get_data(self): + raw = {} + for path in self.paths: + with open(path, 'r') as f: + raw[path] = f.read() + data = {} + for path in self.paths: + data[self.assignment[path]] = raw[path] + return data + + def check(self): + try: + self.sys_dir = str(self.configuration['sys_dir']) + except (KeyError, TypeError): + self.error("No path specified. Using: '" + self.sys_dir + "'") + + self._orig_name = self.chart_name + + for dirpath, _, filenames in os.walk(self.sys_dir): + if self.filename in filenames: + self.paths.append(dirpath + "/" + self.filename) + + if len(self.paths) == 0: + self.error("cannot find", self.filename) + return False + + self.paths.sort() + i = 0 + for path in self.paths: + self.assignment[path] = "cpu" + str(i) + i += 1 + + for name in self.assignment: + dim = self.assignment[name] + self.definitions[ORDER[0]]['lines'].append([dim, dim, 'absolute', 1, 1000]) + + return True + + def create(self): + self.chart_name = "cpu" + status = SimpleService.create(self) + self.chart_name = self._orig_name + return status + + def update(self, interval): + self.chart_name = "cpu" + status = SimpleService.update(self, interval=interval) + self.chart_name = self._orig_name + return status diff --git a/python.d/dovecot.chart.py b/python.d/dovecot.chart.py new file mode 100644 index 000000000..586478cda --- /dev/null +++ b/python.d/dovecot.chart.py @@ -0,0 +1,127 @@ +# -*- coding: utf-8 -*- +# Description: dovecot netdata python.d module +# Author: Pawel Krupa (paulfantom) + +from base import SocketService + +# default module values (can be overridden per job in `config`) +# update_every = 2 +priority = 60000 +retries = 60 + +# charts order (can be overridden if you want less charts, or different order) +ORDER = ['sessions', 'commands', + 'faults', + 'context_switches', + 'disk', 'bytes', 'syscalls', + 'lookup', 'cache', + 'auth', 'auth_cache'] + +CHARTS = { + 'sessions': { + 'options': [None, "logins and sessions", 'number', 'IMAP', 'dovecot.sessions', 'line'], + 'lines': [ + ['num_logins', 'logins', 'absolute'], + ['num_connected_sessions', 'active sessions', 'absolute'] + ]}, + 'commands': { + 'options': [None, "commands", "commands", 'IMAP', 'dovecot.commands', 'line'], + 'lines': [ + ['num_cmds', 'commands', 'absolute'] + ]}, + 'faults': { + 'options': [None, "faults", "faults", 'Faults', 'dovecot.faults', 'line'], + 'lines': [ + ['min_faults', 'minor', 'absolute'], + ['maj_faults', 'major', 'absolute'] + ]}, + 'context_switches': { + 'options': [None, "context switches", '', 'Context Switches', 'dovecot.context_switches', 'line'], + 'lines': [ + ['vol_cs', 'volountary', 'absolute'], + ['invol_cs', 'involountary', 'absolute'] + ]}, + 'disk': { + 'options': [None, "disk", 'bytes/s', 'Reads and Writes', 'dovecot.disk', 'line'], + 'lines': [ + ['disk_input', 'read', 'incremental'], + ['disk_output', 'write', 'incremental'] + ]}, + 'bytes': { + 'options': [None, "bytes", 'bytes/s', 'Reads and Writes', 'dovecot.bytes', 'line'], + 'lines': [ + ['read_bytes', 'read', 'incremental'], + ['write_bytes', 'write', 'incremental'] + ]}, + 'syscalls': { + 'options': [None, "number of syscalls", 'syscalls/s', 'Reads and Writes', 'dovecot.syscalls', 'line'], + 'lines': [ + ['read_count', 'read', 'incremental'], + ['write_count', 'write', 'incremental'] + ]}, + 'lookup': { + 'options': [None, "lookups", 'number/s', 'Mail', 'dovecot.lookup', 'line'], + 'lines': [ + ['mail_lookup_path', 'path', 'incremental'], + ['mail_lookup_attr', 'attr', 'incremental'] + ]}, + 'cache': { + 'options': [None, "hits", 'hits/s', 'Mail', 'dovecot.cache', 'line'], + 'lines': [ + ['mail_cache_hits', 'hits', 'incremental'] + ]}, + 'auth': { + 'options': [None, "attempts", 'attempts', 'Authentication', 'dovecot.auth', 'stacked'], + 'lines': [ + ['auth_successes', 'success', 'absolute'], + ['auth_failures', 'failure', 'absolute'] + ]}, + 'auth_cache': { + 'options': [None, "cache", 'number', 'Authentication', 'dovecot.auth_cache', 'stacked'], + 'lines': [ + ['auth_cache_hits', 'hit', 'absolute'], + ['auth_cache_misses', 'miss', 'absolute'] + ]} +} + + +class Service(SocketService): + def __init__(self, configuration=None, name=None): + SocketService.__init__(self, configuration=configuration, name=name) + self.request = "EXPORT\tglobal\r\n" + self.host = None # localhost + self.port = None # 24242 + # self._keep_alive = True + self.unix_socket = "/var/run/dovecot/stats" + self.order = ORDER + self.definitions = CHARTS + + def _get_data(self): + """ + Format data received from socket + :return: dict + """ + try: + raw = self._get_raw_data() + except (ValueError, AttributeError): + return None + + data = raw.split('\n')[:2] + desc = data[0].split('\t') + vals = data[1].split('\t') + # ret = dict(zip(desc, vals)) + ret = {} + for i in range(len(desc)): + try: + #d = str(desc[i]) + #if d in ('user_cpu', 'sys_cpu', 'clock_time'): + # val = float(vals[i]) + #else: + # val = int(vals[i]) + #ret[d] = val + ret[str(desc[i])] = int(vals[i]) + except ValueError: + pass + if len(ret) == 0: + return None + return ret diff --git a/python.d/example.chart.py b/python.d/example.chart.py new file mode 100644 index 000000000..adf97a921 --- /dev/null +++ b/python.d/example.chart.py @@ -0,0 +1,36 @@ +# -*- coding: utf-8 -*- +# Description: example netdata python.d module +# Author: Pawel Krupa (paulfantom) + +import os +import random +from base import SimpleService + +NAME = os.path.basename(__file__).replace(".chart.py", "") + +# default module values +# update_every = 4 +priority = 90000 +retries = 60 + + +class Service(SimpleService): + def __init__(self, configuration=None, name=None): + super(self.__class__,self).__init__(configuration=configuration, name=name) + + def check(self): + return True + + def create(self): + self.chart("example.python_random", '', 'A random number', 'random number', + 'random', 'random', 'line', self.priority, self.update_every) + self.dimension('random1') + self.commit() + return True + + def update(self, interval): + self.begin("example.python_random", interval) + self.set("random1", random.randint(0, 100)) + self.end() + self.commit() + return True diff --git a/python.d/exim.chart.py b/python.d/exim.chart.py new file mode 100644 index 000000000..1858cbc70 --- /dev/null +++ b/python.d/exim.chart.py @@ -0,0 +1,39 @@ +# -*- coding: utf-8 -*- +# Description: exim netdata python.d module +# Author: Pawel Krupa (paulfantom) + +from base import ExecutableService + +# default module values (can be overridden per job in `config`) +# update_every = 2 +priority = 60000 +retries = 60 + +# charts order (can be overridden if you want less charts, or different order) +ORDER = ['qemails'] + +CHARTS = { + 'qemails': { + 'options': [None, "Exim Queue Emails", "emails", 'queue', 'exim.qemails', 'line'], + 'lines': [ + ['emails', None, 'absolute'] + ]} +} + + +class Service(ExecutableService): + def __init__(self, configuration=None, name=None): + ExecutableService.__init__(self, configuration=configuration, name=name) + self.command = "exim -bpc" + self.order = ORDER + self.definitions = CHARTS + + def _get_data(self): + """ + Format data received from shell command + :return: dict + """ + try: + return {'emails': int(self._get_raw_data()[0])} + except (ValueError, AttributeError): + return None diff --git a/python.d/hddtemp.chart.py b/python.d/hddtemp.chart.py new file mode 100644 index 000000000..4271001b7 --- /dev/null +++ b/python.d/hddtemp.chart.py @@ -0,0 +1,114 @@ +# -*- coding: utf-8 -*- +# Description: hddtemp netdata python.d module +# Author: Pawel Krupa (paulfantom) + +import os +from base import SocketService + +# default module values (can be overridden per job in `config`) +#update_every = 2 +priority = 60000 +retries = 60 + +# default job configuration (overridden by python.d.plugin) +# config = {'local': { +# 'update_every': update_every, +# 'retries': retries, +# 'priority': priority, +# 'host': 'localhost', +# 'port': 7634 +# }} + +ORDER = ['temperatures'] + +CHARTS = { + 'temperatures': { + 'options': ['disks_temp', 'temperature', 'Celsius', 'Disks temperature', 'hddtemp.temperatures', 'line'], + 'lines': [ + # lines are created dynamically in `check()` method + ]} +} + + +class Service(SocketService): + def __init__(self, configuration=None, name=None): + SocketService.__init__(self, configuration=configuration, name=name) + self._keep_alive = False + self.request = "" + self.host = "127.0.0.1" + self.port = 7634 + self.order = ORDER + self.definitions = CHARTS + self.disks = [] + + def _get_disks(self): + try: + disks = self.configuration['devices'] + print(disks) + except (KeyError, TypeError) as e: + self.info("Autodetecting disks") + return ["/dev/" + f for f in os.listdir("/dev") if len(f) == 3 and f.startswith("sd")] + + ret = [] + for disk in disks: + if not disk.startswith('/dev/'): + disk = "/dev/" + disk + if os.path.exists(disk): + ret.append(disk) + if len(ret) == 0: + self.error("Provided disks cannot be found in /dev directory.") + return ret + + def _check_raw_data(self, data): + if not data.endswith('|'): + return False + + if all(disk in data for disk in self.disks): + return True + + return False + + def _get_data(self): + """ + Get data from TCP/IP socket + :return: dict + """ + try: + raw = self._get_raw_data().split("|")[:-1] + except AttributeError: + self.error("no data received") + return None + data = {} + for i in range(len(raw) // 5): + if not raw[i*5+1] in self.disks: + continue + try: + val = int(raw[i*5+3]) + except ValueError: + val = 0 + data[raw[i*5+1].replace("/dev/", "")] = val + + if len(data) == 0: + self.error("received data doesn't have needed records") + return None + else: + return data + + def check(self): + """ + Parse configuration, check if hddtemp is available, and dynamically create chart lines data + :return: boolean + """ + self._parse_config() + self.disks = self._get_disks() + + data = self._get_data() + if data is None: + return False + + for name in data: + self.definitions[ORDER[0]]['lines'].append([name]) + + return True + + diff --git a/python.d/ipfs.chart.py b/python.d/ipfs.chart.py new file mode 100644 index 000000000..b0b2a9659 --- /dev/null +++ b/python.d/ipfs.chart.py @@ -0,0 +1,107 @@ +# -*- coding: utf-8 -*- +# Description: IPFS netdata python.d module +# Authors: Pawel Krupa (paulfantom), davidak + +from base import UrlService +import json + +# default module values (can be overridden per job in `config`) +# update_every = 2 +priority = 60000 +retries = 60 + +# default job configuration (overridden by python.d.plugin) +# config = {'local': { +# 'update_every': update_every, +# 'retries': retries, +# 'priority': priority, +# 'url': 'http://localhost:5001' +# }} + +# charts order (can be overridden if you want less charts, or different order) +ORDER = ['bandwidth', 'peers'] + +CHARTS = { + 'bandwidth': { + 'options': [None, 'IPFS Bandwidth', 'kbits/s', 'Bandwidth', 'ipfs.bandwidth', 'line'], + 'lines': [ + ["in", None, "absolute", 8, 1000], + ["out", None, "absolute", -8, 1000] + ]}, + 'peers': { + 'options': [None, 'IPFS Peers', 'peers', 'Peers', 'ipfs.peers', 'line'], + 'lines': [ + ["peers", None, 'absolute'] + ]} +} + + +class Service(UrlService): + def __init__(self, configuration=None, name=None): + UrlService.__init__(self, configuration=configuration, name=name) + try: + self.baseurl = str(self.configuration['url']) + except (KeyError, TypeError): + self.baseurl = "http://localhost:5001" + self.order = ORDER + self.definitions = CHARTS + + def _get_bandwidth(self): + """ + Format data received from http request + :return: int, int + """ + self.url = self.baseurl + "/api/v0/stats/bw" + try: + raw = self._get_raw_data() + except AttributeError: + return None + + try: + parsed = json.loads(raw) + bw_in = int(parsed['RateIn']) + bw_out = int(parsed['RateOut']) + except: + return None + + return bw_in, bw_out + + def _get_peers(self): + """ + Format data received from http request + :return: int + """ + self.url = self.baseurl + "/api/v0/swarm/peers" + try: + raw = self._get_raw_data() + except AttributeError: + return None + + try: + parsed = json.loads(raw) + peers = len(parsed['Strings']) + except: + return None + + return peers + + def _get_data(self): + """ + Get data from API + :return: dict + """ + try: + peers = self._get_peers() + bandwidth_in, bandwidth_out = self._get_bandwidth() + except: + return None + data = {} + if peers is not None: + data['peers'] = peers + if bandwidth_in is not None and bandwidth_out is not None: + data['in'] = bandwidth_in + data['out'] = bandwidth_out + + if len(data) == 0: + return None + return data diff --git a/python.d/memcached.chart.py b/python.d/memcached.chart.py new file mode 100644 index 000000000..5a1400c99 --- /dev/null +++ b/python.d/memcached.chart.py @@ -0,0 +1,180 @@ +# -*- coding: utf-8 -*- +# Description: memcached netdata python.d module +# Author: Pawel Krupa (paulfantom) + +from base import SocketService + +# default module values (can be overridden per job in `config`) +#update_every = 2 +priority = 60000 +retries = 60 + +# default job configuration (overridden by python.d.plugin) +# config = {'local': { +# 'update_every': update_every, +# 'retries': retries, +# 'priority': priority, +# 'host': 'localhost', +# 'port': 11211, +# 'unix_socket': None +# }} + +ORDER = ['cache', 'net', 'connections', 'items', 'evicted_reclaimed', + 'get', 'get_rate', 'set_rate', 'delete', 'cas', 'increment', 'decrement', 'touch', 'touch_rate'] + +CHARTS = { + 'cache': { + 'options': [None, 'Cache Size', 'megabytes', 'Cache', 'memcached.cache', 'stacked'], + 'lines': [ + ['used', 'used', 'absolute', 1, 1048576], + ['avail', 'available', 'absolute', 1, 1048576] + ]}, + 'net': { + 'options': [None, 'Network', 'kilobytes/s', 'Network', 'memcached.net', 'line'], + 'lines': [ + ['bytes_read', 'read', 'incremental', 1, 1024], + ['bytes_written', 'written', 'incremental', 1, 1024] + ]}, + 'connections': { + 'options': [None, 'Connections', 'connections/s', 'Cluster', 'memcached.connections', 'line'], + 'lines': [ + ['curr_connections', 'current', 'incremental'], + ['rejected_connections', 'rejected', 'incremental'], + ['total_connections', 'total', 'incremental'] + ]}, + 'items': { + 'options': [None, 'Items', 'items', 'Cluster', 'memcached.items', 'line'], + 'lines': [ + ['curr_items', 'current', 'absolute'], + ['total_items', 'total', 'absolute'] + ]}, + 'evicted_reclaimed': { + 'options': [None, 'Items', 'items', 'Evicted and Reclaimed', 'memcached.evicted_reclaimed', 'line'], + 'lines': [ + ['evictions', 'evicted', 'absolute'], + ['reclaimed', 'reclaimed', 'absolute'] + ]}, + 'get': { + 'options': [None, 'Requests', 'requests', 'GET', 'memcached.get', 'stacked'], + 'lines': [ + ['get_hits', 'hits', 'percent-of-absolute-row'], + ['get_misses', 'misses', 'percent-of-absolute-row'] + ]}, + 'get_rate': { + 'options': [None, 'Rate', 'requests/s', 'GET', 'memcached.get_rate', 'line'], + 'lines': [ + ['cmd_get', 'rate', 'incremental'] + ]}, + 'set_rate': { + 'options': [None, 'Rate', 'requests/s', 'SET', 'memcached.set_rate', 'line'], + 'lines': [ + ['cmd_set', 'rate', 'incremental'] + ]}, + 'delete': { + 'options': [None, 'Requests', 'requests', 'DELETE', 'memcached.delete', 'stacked'], + 'lines': [ + ['delete_hits', 'hits', 'percent-of-absolute-row'], + ['delete_misses', 'misses', 'percent-of-absolute-row'], + ]}, + 'cas': { + 'options': [None, 'Requests', 'requests', 'CAS', 'memcached.cas', 'stacked'], + 'lines': [ + ['cas_hits', 'hits', 'percent-of-absolute-row'], + ['cas_misses', 'misses', 'percent-of-absolute-row'], + ['cas_badval', 'bad value', 'percent-of-absolute-row'] + ]}, + 'increment': { + 'options': [None, 'Requests', 'requests', 'Increment', 'memcached.increment', 'stacked'], + 'lines': [ + ['incr_hits', 'hits', 'percent-of-absolute-row'], + ['incr_misses', 'misses', 'percent-of-absolute-row'] + ]}, + 'decrement': { + 'options': [None, 'Requests', 'requests', 'Decrement', 'memcached.decrement', 'stacked'], + 'lines': [ + ['decr_hits', 'hits', 'percent-of-absolute-row'], + ['decr_misses', 'misses', 'percent-of-absolute-row'] + ]}, + 'touch': { + 'options': [None, 'Requests', 'requests', 'Touch', 'memcached.touch', 'stacked'], + 'lines': [ + ['touch_hits', 'hits', 'percent-of-absolute-row'], + ['touch_misses', 'misses', 'percent-of-absolute-row'] + ]}, + 'touch_rate': { + 'options': [None, 'Rate', 'requests/s', 'Touch', 'memcached.touch_rate', 'line'], + 'lines': [ + ['cmd_touch', 'rate', 'incremental'] + ]} +} + + +class Service(SocketService): + def __init__(self, configuration=None, name=None): + SocketService.__init__(self, configuration=configuration, name=name) + self.request = "stats\r\n" + self.host = "localhost" + self.port = 11211 + self._keep_alive = True + self.unix_socket = None + self.order = ORDER + self.definitions = CHARTS + + def _get_data(self): + """ + Get data from socket + :return: dict + """ + try: + raw = self._get_raw_data().split("\n") + except AttributeError: + self.error("no data received") + return None + if raw[0].startswith('ERROR'): + self.error("Memcached returned ERROR") + return None + data = {} + for line in raw: + if line.startswith('STAT'): + try: + t = line[5:].split(' ') + data[t[0]] = int(t[1]) + except (IndexError, ValueError): + pass + try: + data['hit_rate'] = int((data['keyspace_hits'] / float(data['keyspace_hits'] + data['keyspace_misses'])) * 100) + except: + data['hit_rate'] = 0 + + try: + data['avail'] = int(data['limit_maxbytes']) - int(data['bytes']) + data['used'] = data['bytes'] + except: + pass + + if len(data) == 0: + self.error("received data doesn't have needed records") + return None + else: + return data + + def _check_raw_data(self, data): + if data.endswith('END\r\n'): + return True + else: + return False + + def check(self): + """ + Parse configuration, check if memcached is available + :return: boolean + """ + self._parse_config() + if self.name == "": + self.name = "local" + self.chart_name += "_" + self.name + data = self._get_data() + if data is None: + return False + + return True diff --git a/python.d/mysql.chart.py b/python.d/mysql.chart.py new file mode 100644 index 000000000..7c3931acf --- /dev/null +++ b/python.d/mysql.chart.py @@ -0,0 +1,406 @@ +# -*- coding: utf-8 -*- +# Description: MySQL netdata python.d module +# Author: Pawel Krupa (paulfantom) + +from base import SimpleService +import msg + +# import 3rd party library to handle MySQL communication +try: + import MySQLdb + + # https://github.com/PyMySQL/mysqlclient-python + msg.info("using MySQLdb") +except ImportError: + try: + import pymysql as MySQLdb + + # https://github.com/PyMySQL/PyMySQL + msg.info("using pymysql") + except ImportError: + msg.error("MySQLdb or PyMySQL module is needed to use mysql.chart.py plugin") + raise ImportError + +# default module values (can be overridden per job in `config`) +# update_every = 3 +priority = 90000 +retries = 60 + +# default configuration (overridden by python.d.plugin) +# config = { +# 'local': { +# 'user': 'root', +# 'pass': '', +# 'socket': '/var/run/mysqld/mysqld.sock', +# 'update_every': update_every, +# 'retries': retries, +# 'priority': priority +# } +#} + +# query executed on MySQL server +QUERY = "SHOW GLOBAL STATUS;" + +ORDER = ['net', + 'queries', + 'handlers', + 'table_locks', + 'join_issues', 'sort_issues', + 'tmp', + 'connections', 'connection_errors', + 'binlog_cache', 'binlog_stmt_cache', + 'threads', 'thread_cache_misses', + 'innodb_io', 'innodb_io_ops', 'innodb_io_pending_ops', 'innodb_log', 'innodb_os_log', 'innodb_os_log_io', + 'innodb_cur_row_lock', 'innodb_rows', 'innodb_buffer_pool_pages', 'innodb_buffer_pool_bytes', + 'innodb_buffer_pool_read_ahead', 'innodb_buffer_pool_reqs', 'innodb_buffer_pool_ops', + 'qcache_ops', 'qcache', 'qcache_freemem', 'qcache_memblocks', + 'key_blocks', 'key_requests', 'key_disk_ops', + 'files', 'files_rate'] + +CHARTS = { + 'net': { + 'options': [None, 'mysql Bandwidth', 'kilobits/s', 'bandwidth', 'mysql.net', 'area'], + 'lines': [ + ["Bytes_received", "in", "incremental", 8, 1024], + ["Bytes_sent", "out", "incremental", -8, 1024] + ]}, + 'queries': { + 'options': [None, 'mysql Queries', 'queries/s', 'queries', 'mysql.queries', 'line'], + 'lines': [ + ["Queries", "queries", "incremental"], + ["Questions", "questions", "incremental"], + ["Slow_queries", "slow_queries", "incremental"] + ]}, + 'handlers': { + 'options': [None, 'mysql Handlers', 'handlers/s', 'handlers', 'mysql.handlers', 'line'], + 'lines': [ + ["Handler_commit", "commit", "incremental"], + ["Handler_delete", "delete", "incremental"], + ["Handler_prepare", "prepare", "incremental"], + ["Handler_read_first", "read_first", "incremental"], + ["Handler_read_key", "read_key", "incremental"], + ["Handler_read_next", "read_next", "incremental"], + ["Handler_read_prev", "read_prev", "incremental"], + ["Handler_read_rnd", "read_rnd", "incremental"], + ["Handler_read_rnd_next", "read_rnd_next", "incremental"], + ["Handler_rollback", "rollback", "incremental"], + ["Handler_savepoint", "savepoint", "incremental"], + ["Handler_savepoint_rollback", "savepoint_rollback", "incremental"], + ["Handler_update", "update", "incremental"], + ["Handler_write", "write", "incremental"] + ]}, + 'table_locks': { + 'options': [None, 'mysql Tables Locks', 'locks/s', 'locks', 'mysql.table_locks', 'line'], + 'lines': [ + ["Table_locks_immediate", "immediate", "incremental"], + ["Table_locks_waited", "waited", "incremental", -1, 1] + ]}, + 'join_issues': { + 'options': [None, 'mysql Select Join Issues', 'joins/s', 'issues', 'mysql.join_issues', 'line'], + 'lines': [ + ["Select_full_join", "full_join", "incremental"], + ["Select_full_range_join", "full_range_join", "incremental"], + ["Select_range", "range", "incremental"], + ["Select_range_check", "range_check", "incremental"], + ["Select_scan", "scan", "incremental"] + ]}, + 'sort_issues': { + 'options': [None, 'mysql Sort Issues', 'issues/s', 'issues', 'mysql.sort_issues', 'line'], + 'lines': [ + ["Sort_merge_passes", "merge_passes", "incremental"], + ["Sort_range", "range", "incremental"], + ["Sort_scan", "scan", "incremental"] + ]}, + 'tmp': { + 'options': [None, 'mysql Tmp Operations', 'counter', 'temporaries', 'mysql.tmp', 'line'], + 'lines': [ + ["Created_tmp_disk_tables", "disk_tables", "incremental"], + ["Created_tmp_files", "files", "incremental"], + ["Created_tmp_tables", "tables", "incremental"] + ]}, + 'connections': { + 'options': [None, 'mysql Connections', 'connections/s', 'connections', 'mysql.connections', 'line'], + 'lines': [ + ["Connections", "all", "incremental"], + ["Aborted_connects", "aborted", "incremental"] + ]}, + 'binlog_cache': { + 'options': [None, 'mysql Binlog Cache', 'transactions/s', 'binlog', 'mysql.binlog_cache', 'line'], + 'lines': [ + ["Binlog_cache_disk_use", "disk", "incremental"], + ["Binlog_cache_use", "all", "incremental"] + ]}, + 'threads': { + 'options': [None, 'mysql Threads', 'threads', 'threads', 'mysql.threads', 'line'], + 'lines': [ + ["Threads_connected", "connected", "absolute"], + ["Threads_created", "created", "incremental"], + ["Threads_cached", "cached", "absolute", -1, 1], + ["Threads_running", "running", "absolute"], + ]}, + 'thread_cache_misses': { + 'options': [None, 'mysql Threads Cache Misses', 'misses', 'threads', 'mysql.thread_cache_misses', 'area'], + 'lines': [ + ["Thread_cache_misses", "misses", "absolute", 1, 100] + ]}, + 'innodb_io': { + 'options': [None, 'mysql InnoDB I/O Bandwidth', 'kilobytes/s', 'innodb', 'mysql.innodb_io', 'area'], + 'lines': [ + ["Innodb_data_read", "read", "incremental", 1, 1024], + ["Innodb_data_written", "write", "incremental", -1, 1024] + ]}, + 'innodb_io_ops': { + 'options': [None, 'mysql InnoDB I/O Operations', 'operations/s', 'innodb', 'mysql.innodb_io_ops', 'line'], + 'lines': [ + ["Innodb_data_reads", "reads", "incremental"], + ["Innodb_data_writes", "writes", "incremental", -1, 1], + ["Innodb_data_fsyncs", "fsyncs", "incremental"] + ]}, + 'innodb_io_pending_ops': { + 'options': [None, 'mysql InnoDB Pending I/O Operations', 'operations', 'innodb', 'mysql.innodb_io_pending_ops', 'line'], + 'lines': [ + ["Innodb_data_pending_reads", "reads", "absolute"], + ["Innodb_data_pending_writes", "writes", "absolute", -1, 1], + ["Innodb_data_pending_fsyncs", "fsyncs", "absolute"] + ]}, + 'innodb_log': { + 'options': [None, 'mysql InnoDB Log Operations', 'operations/s', 'innodb', 'mysql.innodb_log', 'line'], + 'lines': [ + ["Innodb_log_waits", "waits", "incremental"], + ["Innodb_log_write_requests", "write_requests", "incremental", -1, 1], + ["Innodb_log_writes", "writes", "incremental", -1, 1], + ]}, + 'innodb_os_log': { + 'options': [None, 'mysql InnoDB OS Log Operations', 'operations', 'innodb', 'mysql.innodb_os_log', 'line'], + 'lines': [ + ["Innodb_os_log_fsyncs", "fsyncs", "incremental"], + ["Innodb_os_log_pending_fsyncs", "pending_fsyncs", "absolute"], + ["Innodb_os_log_pending_writes", "pending_writes", "absolute", -1, 1], + ]}, + 'innodb_os_log_io': { + 'options': [None, 'mysql InnoDB OS Log Bandwidth', 'kilobytes/s', 'innodb', 'mysql.innodb_os_log_io', 'area'], + 'lines': [ + ["Innodb_os_log_written", "write", "incremental", -1, 1024], + ]}, + 'innodb_cur_row_lock': { + 'options': [None, 'mysql InnoDB Current Row Locks', 'operations', 'innodb', 'mysql.innodb_cur_row_lock', 'area'], + 'lines': [ + ["Innodb_row_lock_current_waits", "current_waits", "absolute"] + ]}, + 'innodb_rows': { + 'options': [None, 'mysql InnoDB Row Operations', 'operations/s', 'innodb', 'mysql.innodb_rows', 'area'], + 'lines': [ + ["Innodb_rows_inserted", "read", "incremental"], + ["Innodb_rows_read", "deleted", "incremental", -1, 1], + ["Innodb_rows_updated", "inserted", "incremental", 1, 1], + ["Innodb_rows_deleted", "updated", "incremental", -1, 1], + ]}, + 'innodb_buffer_pool_pages': { + 'options': [None, 'mysql InnoDB Buffer Pool Pages', 'pages', 'innodb', 'mysql.innodb_buffer_pool_pages', 'line'], + 'lines': [ + ["Innodb_buffer_pool_pages_data", "data", "absolute"], + ["Innodb_buffer_pool_pages_dirty", "dirty", "absolute", -1, 1], + ["Innodb_buffer_pool_pages_free", "free", "absolute"], + ["Innodb_buffer_pool_pages_flushed", "flushed", "incremental", -1, 1], + ["Innodb_buffer_pool_pages_misc", "misc", "absolute", -1, 1], + ["Innodb_buffer_pool_pages_total", "total", "absolute"] + ]}, + 'innodb_buffer_pool_bytes': { + 'options': [None, 'mysql InnoDB Buffer Pool Bytes', 'MB', 'innodb', 'mysql.innodb_buffer_pool_bytes', 'area'], + 'lines': [ + ["Innodb_buffer_pool_bytes_data", "data", "absolute", 1, 1024 * 1024], + ["Innodb_buffer_pool_bytes_dirty", "dirty", "absolute", -1, 1024 * 1024] + ]}, + 'innodb_buffer_pool_read_ahead': { + 'options': [None, 'mysql InnoDB Buffer Pool Read Ahead', 'operations/s', 'innodb', 'mysql.innodb_buffer_pool_read_ahead', 'area'], + 'lines': [ + ["Innodb_buffer_pool_read_ahead", "all", "incremental"], + ["Innodb_buffer_pool_read_ahead_evicted", "evicted", "incremental", -1, 1], + ["Innodb_buffer_pool_read_ahead_rnd", "random", "incremental"] + ]}, + 'innodb_buffer_pool_reqs': { + 'options': [None, 'mysql InnoDB Buffer Pool Requests', 'requests/s', 'innodb', 'mysql.innodb_buffer_pool_reqs', 'area'], + 'lines': [ + ["Innodb_buffer_pool_read_requests", "reads", "incremental"], + ["Innodb_buffer_pool_write_requests", "writes", "incremental", -1, 1] + ]}, + 'innodb_buffer_pool_ops': { + 'options': [None, 'mysql InnoDB Buffer Pool Operations', 'operations/s', 'innodb', 'mysql.innodb_buffer_pool_ops', 'area'], + 'lines': [ + ["Innodb_buffer_pool_reads", "disk reads", "incremental"], + ["Innodb_buffer_pool_wait_free", "wait free", "incremental", -1, 1] + ]}, + 'qcache_ops': { + 'options': [None, 'mysql QCache Operations', 'queries/s', 'qcache', 'mysql.qcache_ops', 'line'], + 'lines': [ + ["Qcache_hits", "hits", "incremental"], + ["Qcache_lowmem_prunes", "lowmem prunes", "incremental", -1, 1], + ["Qcache_inserts", "inserts", "incremental"], + ["Qcache_not_cached", "not cached", "incremental", -1, 1] + ]}, + 'qcache': { + 'options': [None, 'mysql QCache Queries in Cache', 'queries', 'qcache', 'mysql.qcache', 'line'], + 'lines': [ + ["Qcache_queries_in_cache", "queries", "absolute"] + ]}, + 'qcache_freemem': { + 'options': [None, 'mysql QCache Free Memory', 'MB', 'qcache', 'mysql.qcache_freemem', 'area'], + 'lines': [ + ["Qcache_free_memory", "free", "absolute", 1, 1024 * 1024] + ]}, + 'qcache_memblocks': { + 'options': [None, 'mysql QCache Memory Blocks', 'blocks', 'qcache', 'mysql.qcache_memblocks', 'line'], + 'lines': [ + ["Qcache_free_blocks", "free", "absolute"], + ["Qcache_total_blocks", "total", "absolute"] + ]}, + 'key_blocks': { + 'options': [None, 'mysql MyISAM Key Cache Blocks', 'blocks', 'myisam', 'mysql.key_blocks', 'line'], + 'lines': [ + ["Key_blocks_unused", "unused", "absolute"], + ["Key_blocks_used", "used", "absolute", -1, 1], + ["Key_blocks_not_flushed", "not flushed", "absolute"] + ]}, + 'key_requests': { + 'options': [None, 'mysql MyISAM Key Cache Requests', 'requests/s', 'myisam', 'mysql.key_requests', 'area'], + 'lines': [ + ["Key_read_requests", "reads", "incremental"], + ["Key_write_requests", "writes", "incremental", -1, 1] + ]}, + 'key_disk_ops': { + 'options': [None, 'mysql MyISAM Key Cache Disk Operations', 'operations/s', 'myisam', 'mysql.key_disk_ops', 'area'], + 'lines': [ + ["Key_reads", "reads", "incremental"], + ["Key_writes", "writes", "incremental", -1, 1] + ]}, + 'files': { + 'options': [None, 'mysql Open Files', 'files', 'files', 'mysql.files', 'line'], + 'lines': [ + ["Open_files", "files", "absolute"] + ]}, + 'files_rate': { + 'options': [None, 'mysql Opened Files Rate', 'files/s', 'files', 'mysql.files_rate', 'line'], + 'lines': [ + ["Opened_files", "files", "incremental"] + ]}, + 'binlog_stmt_cache': { + 'options': [None, 'mysql Binlog Statement Cache', 'statements/s', 'binlog', 'mysql.binlog_stmt_cache', 'line'], + 'lines': [ + ["Binlog_stmt_cache_disk_use", "disk", "incremental"], + ["Binlog_stmt_cache_use", "all", "incremental"] + ]}, + 'connection_errors': { + 'options': [None, 'mysql Connection Errors', 'connections/s', 'connections', 'mysql.connection_errors', 'line'], + 'lines': [ + ["Connection_errors_accept", "accept", "incremental"], + ["Connection_errors_internal", "internal", "incremental"], + ["Connection_errors_max_connections", "max", "incremental"], + ["Connection_errors_peer_address", "peer_addr", "incremental"], + ["Connection_errors_select", "select", "incremental"], + ["Connection_errors_tcpwrap", "tcpwrap", "incremental"] + ]} + +} + + +class Service(SimpleService): + def __init__(self, configuration=None, name=None): + SimpleService.__init__(self, configuration=configuration, name=name) + self._parse_config(configuration) + self.order = ORDER + self.definitions = CHARTS + self.connection = None + + def _parse_config(self, configuration): + """ + Parse configuration to collect data from MySQL server + :param configuration: dict + :return: dict + """ + if self.name is None: + self.name = 'local' + if 'user' not in configuration: + self.configuration['user'] = 'root' + if 'pass' not in configuration: + self.configuration['pass'] = '' + if 'my.cnf' in configuration: + self.configuration['socket'] = '' + self.configuration['host'] = '' + self.configuration['port'] = 0 + elif 'socket' in configuration: + self.configuration['my.cnf'] = '' + self.configuration['host'] = '' + self.configuration['port'] = 0 + elif 'host' in configuration: + self.configuration['my.cnf'] = '' + self.configuration['socket'] = '' + if 'port' in configuration: + self.configuration['port'] = int(configuration['port']) + else: + self.configuration['port'] = 3306 + + def _connect(self): + """ + Try to connect to MySQL server + """ + try: + self.connection = MySQLdb.connect(user=self.configuration['user'], + passwd=self.configuration['pass'], + read_default_file=self.configuration['my.cnf'], + unix_socket=self.configuration['socket'], + host=self.configuration['host'], + port=self.configuration['port'], + connect_timeout=self.update_every) + except MySQLdb.OperationalError as e: + self.error("Cannot establish connection to MySQL.") + self.debug(str(e)) + raise RuntimeError + except Exception as e: + self.error("problem connecting to server:", e) + raise RuntimeError + + def _get_data(self): + """ + Get raw data from MySQL server + :return: dict + """ + if self.connection is None: + try: + self._connect() + except RuntimeError: + return None + try: + cursor = self.connection.cursor() + cursor.execute(QUERY) + raw_data = cursor.fetchall() + except MySQLdb.OperationalError as e: + self.debug("Reconnecting due to", str(e)) + self._connect() + cursor = self.connection.cursor() + cursor.execute(QUERY) + raw_data = cursor.fetchall() + except Exception as e: + self.error("cannot execute query.", e) + self.connection.close() + self.connection = None + return None + + data = dict(raw_data) + try: + data["Thread_cache_misses"] = int(data["Threads_created"] * 10000 / float(data["Connections"])) + except: + data["Thread_cache_misses"] = 0 + + return data + + def check(self): + """ + Check if service is able to connect to server + :return: boolean + """ + try: + self.connection = self._connect() + return True + except RuntimeError: + self.connection = None + return False diff --git a/python.d/nginx.chart.py b/python.d/nginx.chart.py new file mode 100644 index 000000000..ff8bfede8 --- /dev/null +++ b/python.d/nginx.chart.py @@ -0,0 +1,73 @@ +# -*- coding: utf-8 -*- +# Description: nginx netdata python.d module +# Author: Pawel Krupa (paulfantom) + +from base import UrlService + +# default module values (can be overridden per job in `config`) +# update_every = 2 +priority = 60000 +retries = 60 + +# default job configuration (overridden by python.d.plugin) +# config = {'local': { +# 'update_every': update_every, +# 'retries': retries, +# 'priority': priority, +# 'url': 'http://localhost/stub_status' +# }} + +# charts order (can be overridden if you want less charts, or different order) +ORDER = ['connections', 'requests', 'connection_status', 'connect_rate'] + +CHARTS = { + 'connections': { + 'options': [None, 'nginx Active Connections', 'connections', 'nginx', 'nginx.connections', 'line'], + 'lines': [ + ["active"] + ]}, + 'requests': { + 'options': [None, 'nginx Requests', 'requests/s', 'nginx', 'nginx.requests', 'line'], + 'lines': [ + ["requests", None, 'incremental'] + ]}, + 'connection_status': { + 'options': [None, 'nginx Active Connections by Status', 'connections', 'nginx', 'nginx.connection_status', 'line'], + 'lines': [ + ["reading"], + ["writing"], + ["waiting", "idle"] + ]}, + 'connect_rate': { + 'options': [None, 'nginx Connections Rate', 'connections/s', 'nginx', 'nginx.connect_rate', 'line'], + 'lines': [ + ["accepts", "accepted", "incremental"], + ["handled", None, "incremental"] + ]} +} + + +class Service(UrlService): + def __init__(self, configuration=None, name=None): + UrlService.__init__(self, configuration=configuration, name=name) + if len(self.url) == 0: + self.url = "http://localhost/stub_status" + self.order = ORDER + self.definitions = CHARTS + + def _get_data(self): + """ + Format data received from http request + :return: dict + """ + try: + raw = self._get_raw_data().split(" ") + return {'active': int(raw[2]), + 'requests': int(raw[9]), + 'reading': int(raw[11]), + 'writing': int(raw[13]), + 'waiting': int(raw[15]), + 'accepts': int(raw[7]), + 'handled': int(raw[8])} + except (ValueError, AttributeError): + return None diff --git a/python.d/nginx_log.chart.py b/python.d/nginx_log.chart.py new file mode 100644 index 000000000..95fb123d2 --- /dev/null +++ b/python.d/nginx_log.chart.py @@ -0,0 +1,69 @@ +# -*- coding: utf-8 -*- +# Description: nginx log netdata python.d module +# Author: Pawel Krupa (paulfantom) + +from base import LogService +import re + +priority = 60000 +retries = 60 +# update_every = 3 + +ORDER = ['codes'] +CHARTS = { + 'codes': { + 'options': [None, 'nginx status codes', 'requests/s', 'requests', 'nginx_log.codes', 'stacked'], + 'lines': [ + ["2xx", None, "incremental"], + ["3xx", None, "incremental"], + ["4xx", None, "incremental"], + ["5xx", None, "incremental"] + ]} +} + + +class Service(LogService): + def __init__(self, configuration=None, name=None): + LogService.__init__(self, configuration=configuration, name=name) + if len(self.log_path) == 0: + self.log_path = "/var/log/nginx/access.log" + self.order = ORDER + self.definitions = CHARTS + pattern = r'" ([0-9]{3}) ?' + #pattern = r'(?:" )([0-9][0-9][0-9]) ?' + self.regex = re.compile(pattern) + + def _get_data(self): + """ + Parse new log lines + :return: dict + """ + data = {'2xx': 0, + '3xx': 0, + '4xx': 0, + '5xx': 0} + try: + raw = self._get_raw_data() + if raw is None: + return None + elif not raw: + return data + except (ValueError, AttributeError): + return None + + regex = self.regex + for line in raw: + code = regex.search(line) + beginning = code.group(1)[0] + + if beginning == '2': + data["2xx"] += 1 + elif beginning == '3': + data["3xx"] += 1 + elif beginning == '4': + data["4xx"] += 1 + elif beginning == '5': + data["5xx"] += 1 + + return data + diff --git a/python.d/phpfpm.chart.py b/python.d/phpfpm.chart.py new file mode 100755 index 000000000..d1791d42e --- /dev/null +++ b/python.d/phpfpm.chart.py @@ -0,0 +1,78 @@ +# -*- coding: utf-8 -*- +# Description: PHP-FPM netdata python.d module +# Author: Pawel Krupa (paulfantom) + +from base import UrlService + +# default module values (can be overridden per job in `config`) +# update_every = 2 +priority = 60000 +retries = 60 + +# default job configuration (overridden by python.d.plugin) +# config = {'local': { +# 'update_every': update_every, +# 'retries': retries, +# 'priority': priority, +# 'url': 'http://localhost/status' +# }} + +# charts order (can be overridden if you want less charts, or different order) +ORDER = ['connections', 'requests', 'performance'] + +CHARTS = { + 'connections': { + 'options': [None, 'PHP-FPM Active Connections', 'connections', 'phpfpm', 'phpfpm.connections', 'line'], + 'lines': [ + ["active"], + ["maxActive", 'max active'], + ["idle"] + ]}, + 'requests': { + 'options': [None, 'PHP-FPM Requests', 'requests/s', 'phpfpm', 'phpfpm.requests', 'line'], + 'lines': [ + ["requests", None, "incremental"] + ]}, + 'performance': { + 'options': [None, 'PHP-FPM Performance', 'status', 'phpfpm', 'phpfpm.performance', 'line'], + 'lines': [ + ["reached", 'max children reached'], + ["slow", 'slow requests'] + ]} +} + + +class Service(UrlService): + def __init__(self, configuration=None, name=None): + UrlService.__init__(self, configuration=configuration, name=name) + if len(self.url) == 0: + self.url = "http://localhost/status" + self.order = ORDER + self.definitions = CHARTS + self.assignment = {"active processes": 'active', + "max active processes": 'maxActive', + "idle processes": 'idle', + "accepted conn": 'requests', + "max children reached": 'reached', + "slow requests": 'slow'} + + def _get_data(self): + """ + Format data received from http request + :return: dict + """ + try: + raw = self._get_raw_data().split('\n') + except AttributeError: + return None + data = {} + for row in raw: + tmp = row.split(":") + if str(tmp[0]) in self.assignment: + try: + data[self.assignment[tmp[0]]] = int(tmp[1]) + except (IndexError, ValueError): + pass + if len(data) == 0: + return None + return data diff --git a/python.d/postfix.chart.py b/python.d/postfix.chart.py new file mode 100644 index 000000000..ee4142aaf --- /dev/null +++ b/python.d/postfix.chart.py @@ -0,0 +1,50 @@ +# -*- coding: utf-8 -*- +# Description: postfix netdata python.d module +# Author: Pawel Krupa (paulfantom) + +from base import ExecutableService + +# default module values (can be overridden per job in `config`) +# update_every = 2 +priority = 60000 +retries = 60 + +# charts order (can be overridden if you want less charts, or different order) +ORDER = ['qemails', 'qsize'] + +CHARTS = { + 'qemails': { + 'options': [None, "Postfix Queue Emails", "emails", 'queue', 'postfix.qemails', 'line'], + 'lines': [ + ['emails', None, 'absolute'] + ]}, + 'qsize': { + 'options': [None, "Postfix Queue Emails Size", "emails size in KB", 'queue', 'postfix.qsize', 'area'], + 'lines': [ + ["size", None, 'absolute'] + ]} +} + + +class Service(ExecutableService): + def __init__(self, configuration=None, name=None): + ExecutableService.__init__(self, configuration=configuration, name=name) + self.command = "postqueue -p" + self.order = ORDER + self.definitions = CHARTS + + def _get_data(self): + """ + Format data received from shell command + :return: dict + """ + try: + raw = self._get_raw_data()[-1].split(' ') + if raw[0] == 'Mail' and raw[1] == 'queue': + return {'emails': 0, + 'size': 0} + + return {'emails': raw[4], + 'size': raw[1]} + except (ValueError, AttributeError): + return None diff --git a/python.d/python-modules-installer.sh b/python.d/python-modules-installer.sh new file mode 100644 index 000000000..cda3c6662 --- /dev/null +++ b/python.d/python-modules-installer.sh @@ -0,0 +1,158 @@ +#!/usr/bin/env bash + +umask 022 + +dir="/usr/local/libexec/netdata/python.d" +target="${dir}/python_modules" +pv="$(python -V 2>&1)" + +# parse parameters +while [ ! -z "${1}" ] +do + case "${1}" in + -p|--python) + pv="Python ${2}" + shift 2 + ;; + + -d|--dir) + dir="${2}" + target="${dir}/python_modules" + echo >&2 "Will install python modules in: '${target}'" + shift 2 + ;; + + -s|--system) + target= + echo >&2 "Will install python modules system-wide" + shift + ;; + + -h|--help) + echo "${0} [--dir netdata-python.d-path] [--system]" + echo "Please make sure you have installed packages: python-pip (or python3-pip) python-dev libyaml-dev libmysqlclient-dev" + exit 0 + ;; + + *) + echo >&2 "Cannot understand parameter: ${1}" + exit 1 + ;; + esac +done + + +if [ ! -z "${target}" -a ! -d "${target}" ] +then + echo >&2 "Cannot find directory: '${target}'" + exit 1 +fi + +if [[ "${pv}" =~ ^Python\ 2.* ]] +then + pv=2 + pip="$(which pip2 2>/dev/null)" +elif [[ "${pv}" =~ ^Python\ 3.* ]] +then + pv=3 + pip="$(which pip3 2>/dev/null)" +else + echo >&2 "Cannot detect python version. Is python installed?" + exit 1 +fi + +[ -z "${pip}" ] && pip="$(which pip 2>/dev/null)" +if [ -z "${pip}" ] +then + echo >&2 "pip command is required to install python v${pv} modules." + [ "${pv}" = "2" ] && echo >&2 "Please install python-pip." + [ "${pv}" = "3" ] && echo >&2 "Please install python3-pip." + exit 1 +fi + +echo >&2 "Working for python version ${pv} (pip command: '${pip}')" +echo >&2 "Installing netdata python modules in: '${target}'" + +run() { + printf "Running command:\n# " + printf "%q " "${@}" + printf "\n" + "${@}" +} + +# try to install all the python modules given as parameters +# until the first that succeeds +failed="" +installed="" +errors=0 +pip_install() { + local ret x msg="${1}" + shift + + echo >&2 + echo >&2 + echo >&2 "Installing one of: ${*}" + + for x in "${@}" + do + echo >&2 + echo >&2 "attempting to install: ${x}" + if [ ! -z "${target}" ] + then + run "${pip}" install --target "${target}" "${x}" + ret=$? + else + run "${pip}" install "${x}" + ret=$? + fi + [ ${ret} -eq 0 ] && break + echo >&2 "failed to install: ${x}. ${msg}" + done + + if [ ${ret} -ne 0 ] + then + echo >&2 + echo >&2 + echo >&2 "FAILED: could not install any of: ${*}. ${msg}" + echo >&2 + echo >&2 + errors=$(( errors + 1 )) + failed="${failed}|${*}" + else + echo >&2 + echo >&2 + echo >&2 "SUCCESS: we have: ${x}" + echo >&2 + echo >&2 + installed="${installed} ${x}" + fi + return ${ret} +} + +if [ "${pv}" = "2" ] +then + pip_install "is libyaml-dev and python-dev installed?" pyyaml + pip_install "is libmysqlclient-dev and python-dev installed?" mysqlclient mysql-python pymysql +else + pip_install "is libyaml-dev and python-dev installed?" pyyaml + pip_install "is libmysqlclient-dev and python-dev installed?" mysql-python mysqlclient pymysql +fi + +echo >&2 +echo >&2 +if [ ${errors} -ne 0 ] +then + echo >&2 "Failed to install ${errors} modules: ${failed}" + if [ ! -z "${target}" ] + then + echo >&2 + echo >&2 "If you are getting errors during cleanup from pip, there is a known bug" + echo >&2 "in certain versions of pip that prevents installing packages local to an" + echo >&2 "application. To install them system-wide please run:" + echo >&2 "$0 --system" + fi + exit 1 +else + echo >&2 "All done. We have: ${installed}" + exit 0 +fi diff --git a/python.d/python_modules/__init__.py b/python.d/python_modules/__init__.py new file mode 100755 index 000000000..8d1c8b69c --- /dev/null +++ b/python.d/python_modules/__init__.py @@ -0,0 +1 @@ + diff --git a/python.d/python_modules/base.py b/python.d/python_modules/base.py new file mode 100644 index 000000000..c2bbed2a5 --- /dev/null +++ b/python.d/python_modules/base.py @@ -0,0 +1,798 @@ +# -*- coding: utf-8 -*- +# Description: netdata python modules framework +# Author: Pawel Krupa (paulfantom) + +# Remember: +# ALL CODE NEEDS TO BE COMPATIBLE WITH Python > 2.7 and Python > 3.1 +# Follow PEP8 as much as it is possible +# "check" and "create" CANNOT be blocking. +# "update" CAN be blocking +# "update" function needs to be fast, so follow: +# https://wiki.python.org/moin/PythonSpeed/PerformanceTips +# basically: +# - use local variables wherever it is possible +# - avoid dots in expressions that are executed many times +# - use "join()" instead of "+" +# - use "import" only at the beginning +# +# using ".encode()" in one thread can block other threads as well (only in python2) + +import time +# import sys +import os +import socket +import select +try: + import urllib.request as urllib2 +except ImportError: + import urllib2 + +from subprocess import Popen, PIPE + +import threading +import msg + + +# class BaseService(threading.Thread): +class SimpleService(threading.Thread): + """ + Prototype of Service class. + Implemented basic functionality to run jobs by `python.d.plugin` + """ + def __init__(self, configuration=None, name=None): + """ + This needs to be initialized in child classes + :param configuration: dict + :param name: str + """ + threading.Thread.__init__(self) + self._data_stream = "" + self.daemon = True + self.retries = 0 + self.retries_left = 0 + self.priority = 140000 + self.update_every = 1 + self.name = name + self.override_name = None + self.chart_name = "" + self._dimensions = [] + self._charts = [] + self.__chart_set = False + self.__first_run = True + self.order = [] + self.definitions = {} + if configuration is None: + self.error("BaseService: no configuration parameters supplied. Cannot create Service.") + raise RuntimeError + else: + self._extract_base_config(configuration) + self.timetable = {} + self.create_timetable() + + # --- BASIC SERVICE CONFIGURATION --- + + def _extract_base_config(self, config): + """ + Get basic parameters to run service + Minimum config: + config = {'update_every':1, + 'priority':100000, + 'retries':0} + :param config: dict + """ + pop = config.pop + try: + self.override_name = pop('name') + except KeyError: + pass + self.update_every = int(pop('update_every')) + self.priority = int(pop('priority')) + self.retries = int(pop('retries')) + self.retries_left = self.retries + self.configuration = config + + def create_timetable(self, freq=None): + """ + Create service timetable. + `freq` is optional + Example: + timetable = {'last': 1466370091.3767564, + 'next': 1466370092, + 'freq': 1} + :param freq: int + """ + if freq is None: + freq = self.update_every + now = time.time() + self.timetable = {'last': now, + 'next': now - (now % freq) + freq, + 'freq': freq} + + # --- THREAD CONFIGURATION --- + + def _run_once(self): + """ + Executes self.update(interval) and draws run time chart. + Return value presents exit status of update() + :return: boolean + """ + t_start = time.time() + timetable = self.timetable + chart_name = self.chart_name + # check if it is time to execute job update() function + if timetable['next'] > t_start: + self.debug(chart_name, "will be run in", str(int((timetable['next'] - t_start) * 1000)), "ms") + return True + + since_last = int((t_start - timetable['last']) * 1000000) + self.debug(chart_name, + "ready to run, after", str(int((t_start - timetable['last']) * 1000)), + "ms (update_every:", str(timetable['freq'] * 1000), + "ms, latency:", str(int((t_start - timetable['next']) * 1000)), "ms") + if self.__first_run: + since_last = 0 + if not self.update(since_last): + self.error("update function failed.") + return False + t_end = time.time() + self.timetable['next'] = t_end - (t_end % timetable['freq']) + timetable['freq'] + # draw performance graph + run_time = str(int((t_end - t_start) * 1000)) + # noinspection SqlNoDataSourceInspection + print("BEGIN netdata.plugin_pythond_%s %s\nSET run_time = %s\nEND\n" % + (self.chart_name, str(since_last), run_time)) + # sys.stdout.write("BEGIN netdata.plugin_pythond_%s %s\nSET run_time = %s\nEND\n" % + # (self.chart_name, str(since_last), run_time)) + + self.debug(chart_name, "updated in", str(run_time), "ms") + self.timetable['last'] = t_start + self.__first_run = False + return True + + def run(self): + """ + Runs job in thread. Handles retries. + Exits when job failed or timed out. + :return: None + """ + self.timetable['last'] = time.time() + while True: # run forever, unless something is wrong + try: + status = self._run_once() + except Exception as e: + self.error("Something wrong: ", str(e)) + return + if status: # handle retries if update failed + time.sleep(self.timetable['next'] - time.time()) + self.retries_left = self.retries + else: + self.retries_left -= 1 + if self.retries_left <= 0: + self.error("no more retries. Exiting") + return + else: + time.sleep(self.timetable['freq']) + + # --- CHART --- + + @staticmethod + def _format(*args): + """ + Escape and convert passed arguments. + :param args: anything + :return: list + """ + params = [] + append = params.append + for p in args: + if p is None: + append(p) + continue + if type(p) is not str: + p = str(p) + if ' ' in p: + p = "'" + p + "'" + append(p) + return params + + def _line(self, instruction, *params): + """ + Converts *params to string and joins them with one space between every one. + Result is appended to self._data_stream + :param params: str/int/float + """ + tmp = list(map((lambda x: "''" if x is None or len(x) == 0 else x), params)) + self._data_stream += "%s %s\n" % (instruction, str(" ".join(tmp))) + + def chart(self, type_id, name="", title="", units="", family="", + category="", chart_type="line", priority="", update_every=""): + """ + Defines a new chart. + :param type_id: str + :param name: str + :param title: str + :param units: str + :param family: str + :param category: str + :param chart_type: str + :param priority: int/str + :param update_every: int/str + """ + self._charts.append(type_id) + + p = self._format(type_id, name, title, units, family, category, chart_type, priority, update_every) + self._line("CHART", *p) + + def dimension(self, id, name=None, algorithm="absolute", multiplier=1, divisor=1, hidden=False): + """ + Defines a new dimension for the chart + :param id: str + :param name: str + :param algorithm: str + :param multiplier: int/str + :param divisor: int/str + :param hidden: boolean + :return: + """ + try: + int(multiplier) + except TypeError: + self.error("malformed dimension: multiplier is not a number:", multiplier) + multiplier = 1 + try: + int(divisor) + except TypeError: + self.error("malformed dimension: divisor is not a number:", divisor) + divisor = 1 + if name is None: + name = id + if algorithm not in ("absolute", "incremental", "percentage-of-absolute-row", "percentage-of-incremental-row"): + algorithm = "absolute" + + self._dimensions.append(str(id)) + if hidden: + p = self._format(id, name, algorithm, multiplier, divisor, "hidden") + else: + p = self._format(id, name, algorithm, multiplier, divisor) + + self._line("DIMENSION", *p) + + def begin(self, type_id, microseconds=0): + """ + Begin data set + :param type_id: str + :param microseconds: int + :return: boolean + """ + if type_id not in self._charts: + self.error("wrong chart type_id:", type_id) + return False + try: + int(microseconds) + except TypeError: + self.error("malformed begin statement: microseconds are not a number:", microseconds) + microseconds = "" + + self._line("BEGIN", type_id, str(microseconds)) + return True + + def set(self, id, value): + """ + Set value to dimension + :param id: str + :param value: int/float + :return: boolean + """ + if id not in self._dimensions: + self.error("wrong dimension id:", id, "Available dimensions are:", *self._dimensions) + return False + try: + value = str(int(value)) + except TypeError: + self.error("cannot set non-numeric value:", str(value)) + return False + self._line("SET", id, "=", str(value)) + self.__chart_set = True + return True + + def end(self): + if self.__chart_set: + self._line("END") + self.__chart_set = False + else: + pos = self._data_stream.rfind("BEGIN") + self._data_stream = self._data_stream[:pos] + + def commit(self): + """ + Upload new data to netdata. + """ + print(self._data_stream) + self._data_stream = "" + + # --- ERROR HANDLING --- + + def error(self, *params): + """ + Show error message on stderr + """ + msg.error(self.chart_name, *params) + + def debug(self, *params): + """ + Show debug message on stderr + """ + msg.debug(self.chart_name, *params) + + def info(self, *params): + """ + Show information message on stderr + """ + msg.info(self.chart_name, *params) + + # --- MAIN METHODS --- + + def _get_data(self): + """ + Get some data + :return: dict + """ + return {} + + def check(self): + """ + check() prototype + :return: boolean + """ + self.debug("Module", str(self.__module__), "doesn't implement check() function. Using default.") + if self._get_data() is None or len(self._get_data()) == 0: + return False + else: + return True + + def create(self): + """ + Create charts + :return: boolean + """ + data = self._get_data() + if data is None: + return False + + idx = 0 + for name in self.order: + options = self.definitions[name]['options'] + [self.priority + idx, self.update_every] + self.chart(self.chart_name + "." + name, *options) + # check if server has this datapoint + for line in self.definitions[name]['lines']: + if line[0] in data: + self.dimension(*line) + idx += 1 + + self.commit() + return True + + def update(self, interval): + """ + Update charts + :param interval: int + :return: boolean + """ + data = self._get_data() + if data is None: + self.debug("_get_data() returned no data") + return False + + updated = False + for chart in self.order: + if self.begin(self.chart_name + "." + chart, interval): + updated = True + for dim in self.definitions[chart]['lines']: + try: + self.set(dim[0], data[dim[0]]) + except KeyError: + pass + self.end() + + self.commit() + if not updated: + self.error("no charts to update") + + return updated + + +class UrlService(SimpleService): + # TODO add support for https connections + def __init__(self, configuration=None, name=None): + self.url = "" + self.user = None + self.password = None + self.proxies = {} + SimpleService.__init__(self, configuration=configuration, name=name) + + def __add_openers(self): + # TODO add error handling + self.opener = urllib2.build_opener() + + # Proxy handling + # TODO currently self.proxies isn't parsed from configuration file + # if len(self.proxies) > 0: + # for proxy in self.proxies: + # url = proxy['url'] + # # TODO test this: + # if "user" in proxy and "pass" in proxy: + # if url.lower().startswith('https://'): + # url = 'https://' + proxy['user'] + ':' + proxy['pass'] + '@' + url[8:] + # else: + # url = 'http://' + proxy['user'] + ':' + proxy['pass'] + '@' + url[7:] + # # FIXME move proxy auth to sth like this: + # # passman = urllib2.HTTPPasswordMgrWithDefaultRealm() + # # passman.add_password(None, url, proxy['user'], proxy['password']) + # # opener.add_handler(urllib2.HTTPBasicAuthHandler(passman)) + # + # if url.lower().startswith('https://'): + # opener.add_handler(urllib2.ProxyHandler({'https': url})) + # else: + # opener.add_handler(urllib2.ProxyHandler({'https': url})) + + # HTTP Basic Auth + if self.user is not None and self.password is not None: + passman = urllib2.HTTPPasswordMgrWithDefaultRealm() + passman.add_password(None, self.url, self.user, self.password) + self.opener.add_handler(urllib2.HTTPBasicAuthHandler(passman)) + self.debug("Enabling HTTP basic auth") + + #urllib2.install_opener(opener) + + def _get_raw_data(self): + """ + Get raw data from http request + :return: str + """ + raw = None + try: + f = self.opener.open(self.url, timeout=self.update_every * 2) + # f = urllib2.urlopen(self.url, timeout=self.update_every * 2) + except Exception as e: + self.error(str(e)) + return None + + try: + raw = f.read().decode('utf-8') + except Exception as e: + self.error(str(e)) + finally: + f.close() + return raw + + def check(self): + """ + Format configuration data and try to connect to server + :return: boolean + """ + if self.name is None or self.name == str(None): + self.name = 'local' + self.chart_name += "_" + self.name + else: + self.name = str(self.name) + try: + self.url = str(self.configuration['url']) + except (KeyError, TypeError): + pass + try: + self.user = str(self.configuration['user']) + except (KeyError, TypeError): + pass + try: + self.password = str(self.configuration['pass']) + except (KeyError, TypeError): + pass + + self.__add_openers() + + test = self._get_data() + if test is None or len(test) == 0: + return False + else: + return True + + +class SocketService(SimpleService): + def __init__(self, configuration=None, name=None): + self._sock = None + self._keep_alive = False + self.host = "localhost" + self.port = None + self.unix_socket = None + self.request = "" + self.__socket_config = None + SimpleService.__init__(self, configuration=configuration, name=name) + + def _connect(self): + """ + Recreate socket and connect to it since sockets cannot be reused after closing + Available configurations are IPv6, IPv4 or UNIX socket + :return: + """ + try: + if self.unix_socket is None: + if self.__socket_config is None: + # establish ipv6 or ipv4 connection. + for res in socket.getaddrinfo(self.host, self.port, socket.AF_UNSPEC, socket.SOCK_STREAM): + try: + # noinspection SpellCheckingInspection + af, socktype, proto, canonname, sa = res + self._sock = socket.socket(af, socktype, proto) + except socket.error as e: + self.debug("Cannot create socket:", str(e)) + self._sock = None + continue + try: + self._sock.connect(sa) + except socket.error as e: + self.debug("Cannot connect to socket:", str(e)) + self._disconnect() + continue + self.__socket_config = res + break + else: + # connect to socket with previously established configuration + try: + af, socktype, proto, canonname, sa = self.__socket_config + self._sock = socket.socket(af, socktype, proto) + self._sock.connect(sa) + except socket.error as e: + self.debug("Cannot create or connect to socket:", str(e)) + self._disconnect() + else: + # connect to unix socket + try: + self._sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM) + self._sock.connect(self.unix_socket) + except socket.error: + self._sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) + self._sock.connect(self.unix_socket) + + except Exception as e: + self.error(str(e), + "Cannot create socket with following configuration: host:", str(self.host), + "port:", str(self.port), + "socket:", str(self.unix_socket)) + self._sock = None + self._sock.setblocking(0) + + def _disconnect(self): + """ + Close socket connection + :return: + """ + try: + self._sock.shutdown(2) # 0 - read, 1 - write, 2 - all + self._sock.close() + except Exception: + pass + self._sock = None + + def _send(self): + """ + Send request. + :return: boolean + """ + # Send request if it is needed + if self.request != "".encode(): + try: + self._sock.send(self.request) + except Exception as e: + self._disconnect() + self.error(str(e), + "used configuration: host:", str(self.host), + "port:", str(self.port), + "socket:", str(self.unix_socket)) + return False + return True + + def _receive(self): + """ + Receive data from socket + :return: str + """ + data = "" + while True: + try: + ready_to_read, _, in_error = select.select([self._sock], [], [], 5) + except Exception as e: + self.debug("SELECT", str(e)) + self._disconnect() + break + if len(ready_to_read) > 0: + buf = self._sock.recv(4096) + if len(buf) == 0 or buf is None: # handle server disconnect + break + data += buf.decode() + if self._check_raw_data(data): + break + else: + self.error("Socket timed out.") + self._disconnect() + break + + return data + + def _get_raw_data(self): + """ + Get raw data with low-level "socket" module. + :return: str + """ + if self._sock is None: + self._connect() + + # Send request if it is needed + if not self._send(): + return None + + data = self._receive() + + if not self._keep_alive: + self._disconnect() + + return data + + def _check_raw_data(self, data): + """ + Check if all data has been gathered from socket + :param data: str + :return: boolean + """ + return True + + def _parse_config(self): + """ + Parse configuration data + :return: boolean + """ + if self.name is None or self.name == str(None): + self.name = "" + else: + self.name = str(self.name) + try: + self.unix_socket = str(self.configuration['socket']) + except (KeyError, TypeError): + self.debug("No unix socket specified. Trying TCP/IP socket.") + try: + self.host = str(self.configuration['host']) + except (KeyError, TypeError): + self.debug("No host specified. Using: '" + self.host + "'") + try: + self.port = int(self.configuration['port']) + except (KeyError, TypeError): + self.debug("No port specified. Using: '" + str(self.port) + "'") + try: + self.request = str(self.configuration['request']) + except (KeyError, TypeError): + self.debug("No request specified. Using: '" + str(self.request) + "'") + self.request = self.request.encode() + + def check(self): + return SimpleService.check(self) + + +class LogService(SimpleService): + def __init__(self, configuration=None, name=None): + self.log_path = "" + self._last_position = 0 + # self._log_reader = None + SimpleService.__init__(self, configuration=configuration, name=name) + self.retries = 100000 # basically always retry + + def _get_raw_data(self): + """ + Get log lines since last poll + :return: list + """ + lines = [] + try: + if os.path.getsize(self.log_path) < self._last_position: + self._last_position = 0 # read from beginning if file has shrunk + elif os.path.getsize(self.log_path) == self._last_position: + self.debug("Log file hasn't changed. No new data.") + return [] # return empty list if nothing has changed + with open(self.log_path, "r") as fp: + fp.seek(self._last_position) + for i, line in enumerate(fp): + lines.append(line) + self._last_position = fp.tell() + except Exception as e: + self.error(str(e)) + + if len(lines) != 0: + return lines + else: + self.error("No data collected.") + return None + + def check(self): + """ + Parse basic configuration and check if log file exists + :return: boolean + """ + if self.name is not None or self.name != str(None): + self.name = "" + else: + self.name = str(self.name) + try: + self.log_path = str(self.configuration['path']) + except (KeyError, TypeError): + self.error("No path to log specified. Using: '" + self.log_path + "'") + + if os.access(self.log_path, os.R_OK): + return True + else: + self.error("Cannot access file: '" + self.log_path + "'") + return False + + def create(self): + # set cursor at last byte of log file + self._last_position = os.path.getsize(self.log_path) + status = SimpleService.create(self) + # self._last_position = 0 + return status + + +class ExecutableService(SimpleService): + bad_substrings = ('&', '|', ';', '>', '<') + + def __init__(self, configuration=None, name=None): + self.command = "" + SimpleService.__init__(self, configuration=configuration, name=name) + + def _get_raw_data(self): + """ + Get raw data from executed command + :return: str + """ + try: + p = Popen(self.command, stdout=PIPE, stderr=PIPE) + except Exception as e: + self.error("Executing command", self.command, "resulted in error:", str(e)) + return None + data = [] + for line in p.stdout.readlines(): + data.append(str(line.decode())) + + if len(data) == 0: + self.error("No data collected.") + return None + + return data + + def check(self): + """ + Parse basic configuration, check if command is whitelisted and is returning values + :return: boolean + """ + if self.name is not None or self.name != str(None): + self.name = "" + else: + self.name = str(self.name) + try: + self.command = str(self.configuration['command']) + except (KeyError, TypeError): + self.error("No command specified. Using: '" + self.command + "'") + command = self.command.split(' ') + + for arg in command[1:]: + if any(st in arg for st in self.bad_substrings): + self.error("Bad command argument:" + " ".join(self.command[1:])) + return False + # test command and search for it in /usr/sbin or /sbin when failed + base = command[0].split('/')[-1] + if self._get_raw_data() is None: + for prefix in ['/sbin/', '/usr/sbin/']: + command[0] = prefix + base + if os.path.isfile(command[0]): + break + self.command = command + if self._get_data() is None or len(self._get_data()) == 0: + self.error("Command", self.command, "returned no data") + return False + return True diff --git a/python.d/python_modules/lm_sensors.py b/python.d/python_modules/lm_sensors.py new file mode 100644 index 000000000..1d868f0e2 --- /dev/null +++ b/python.d/python_modules/lm_sensors.py @@ -0,0 +1,257 @@ +""" +@package sensors.py +Python Bindings for libsensors3 + +use the documentation of libsensors for the low level API. +see example.py for high level API usage. + +@author: Pavel Rojtberg (http://www.rojtberg.net) +@see: https://github.com/paroj/sensors.py +@copyright: LGPLv2 (same as libsensors) <http://opensource.org/licenses/LGPL-2.1> +""" + +from ctypes import * +import ctypes.util + +_libc = cdll.LoadLibrary(ctypes.util.find_library("c")) +# see https://github.com/paroj/sensors.py/issues/1 +_libc.free.argtypes = [c_void_p] +_hdl = cdll.LoadLibrary(ctypes.util.find_library("sensors")) + +version = c_char_p.in_dll(_hdl, "libsensors_version").value.decode("ascii") + + +class bus_id(Structure): + _fields_ = [("type", c_short), + ("nr", c_short)] + + +class chip_name(Structure): + _fields_ = [("prefix", c_char_p), + ("bus", bus_id), + ("addr", c_int), + ("path", c_char_p)] + + +class feature(Structure): + _fields_ = [("name", c_char_p), + ("number", c_int), + ("type", c_int)] + + # sensors_feature_type + IN = 0x00 + FAN = 0x01 + TEMP = 0x02 + POWER = 0x03 + ENERGY = 0x04 + CURR = 0x05 + HUMIDITY = 0x06 + MAX_MAIN = 0x7 + VID = 0x10 + INTRUSION = 0x11 + MAX_OTHER = 0x12 + BEEP_ENABLE = 0x18 + + +class subfeature(Structure): + _fields_ = [("name", c_char_p), + ("number", c_int), + ("type", c_int), + ("mapping", c_int), + ("flags", c_uint)] + + +_hdl.sensors_get_detected_chips.restype = POINTER(chip_name) +_hdl.sensors_get_features.restype = POINTER(feature) +_hdl.sensors_get_all_subfeatures.restype = POINTER(subfeature) +_hdl.sensors_get_label.restype = c_void_p # return pointer instead of str so we can free it +_hdl.sensors_get_adapter_name.restype = c_char_p # docs do not say whether to free this or not +_hdl.sensors_strerror.restype = c_char_p + +### RAW API ### +MODE_R = 1 +MODE_W = 2 +COMPUTE_MAPPING = 4 + + +def init(cfg_file=None): + file = _libc.fopen(cfg_file.encode("utf-8"), "r") if cfg_file is not None else None + + if _hdl.sensors_init(file) != 0: + raise Exception("sensors_init failed") + + if file is not None: + _libc.fclose(file) + + +def cleanup(): + _hdl.sensors_cleanup() + + +def parse_chip_name(orig_name): + ret = chip_name() + err = _hdl.sensors_parse_chip_name(orig_name.encode("utf-8"), byref(ret)) + + if err < 0: + raise Exception(strerror(err)) + + return ret + + +def strerror(errnum): + return _hdl.sensors_strerror(errnum).decode("utf-8") + + +def free_chip_name(chip): + _hdl.sensors_free_chip_name(byref(chip)) + + +def get_detected_chips(match, nr): + """ + @return: (chip, next nr to query) + """ + _nr = c_int(nr) + + if match is not None: + match = byref(match) + + chip = _hdl.sensors_get_detected_chips(match, byref(_nr)) + chip = chip.contents if bool(chip) else None + return chip, _nr.value + + +def chip_snprintf_name(chip, buffer_size=200): + """ + @param buffer_size defaults to the size used in the sensors utility + """ + ret = create_string_buffer(buffer_size) + err = _hdl.sensors_snprintf_chip_name(ret, buffer_size, byref(chip)) + + if err < 0: + raise Exception(strerror(err)) + + return ret.value.decode("utf-8") + + +def do_chip_sets(chip): + """ + @attention this function was not tested + """ + err = _hdl.sensors_do_chip_sets(byref(chip)) + if err < 0: + raise Exception(strerror(err)) + + +def get_adapter_name(bus): + return _hdl.sensors_get_adapter_name(byref(bus)).decode("utf-8") + + +def get_features(chip, nr): + """ + @return: (feature, next nr to query) + """ + _nr = c_int(nr) + feature = _hdl.sensors_get_features(byref(chip), byref(_nr)) + feature = feature.contents if bool(feature) else None + return feature, _nr.value + + +def get_label(chip, feature): + ptr = _hdl.sensors_get_label(byref(chip), byref(feature)) + val = cast(ptr, c_char_p).value.decode("utf-8") + _libc.free(ptr) + return val + + +def get_all_subfeatures(chip, feature, nr): + """ + @return: (subfeature, next nr to query) + """ + _nr = c_int(nr) + subfeature = _hdl.sensors_get_all_subfeatures(byref(chip), byref(feature), byref(_nr)) + subfeature = subfeature.contents if bool(subfeature) else None + return subfeature, _nr.value + + +def get_value(chip, subfeature_nr): + val = c_double() + err = _hdl.sensors_get_value(byref(chip), subfeature_nr, byref(val)) + if err < 0: + raise Exception(strerror(err)) + return val.value + + +def set_value(chip, subfeature_nr, value): + """ + @attention this function was not tested + """ + val = c_double(value) + err = _hdl.sensors_set_value(byref(chip), subfeature_nr, byref(val)) + if err < 0: + raise Exception(strerror(err)) + + +### Convenience API ### +class ChipIterator: + def __init__(self, match=None): + self.match = parse_chip_name(match) if match is not None else None + self.nr = 0 + + def __iter__(self): + return self + + def __next__(self): + chip, self.nr = get_detected_chips(self.match, self.nr) + + if chip is None: + raise StopIteration + + return chip + + def __del__(self): + if self.match is not None: + free_chip_name(self.match) + + def next(self): # python2 compability + return self.__next__() + + +class FeatureIterator: + def __init__(self, chip): + self.chip = chip + self.nr = 0 + + def __iter__(self): + return self + + def __next__(self): + feature, self.nr = get_features(self.chip, self.nr) + + if feature is None: + raise StopIteration + + return feature + + def next(self): # python2 compability + return self.__next__() + + +class SubFeatureIterator: + def __init__(self, chip, feature): + self.chip = chip + self.feature = feature + self.nr = 0 + + def __iter__(self): + return self + + def __next__(self): + subfeature, self.nr = get_all_subfeatures(self.chip, self.feature, self.nr) + + if subfeature is None: + raise StopIteration + + return subfeature + + def next(self): # python2 compability + return self.__next__()
\ No newline at end of file diff --git a/python.d/python_modules/msg.py b/python.d/python_modules/msg.py new file mode 100644 index 000000000..ecefba0a4 --- /dev/null +++ b/python.d/python_modules/msg.py @@ -0,0 +1,74 @@ +# -*- coding: utf-8 -*- +# Description: logging for netdata python.d modules + +import sys +from time import time, strftime + +DEBUG_FLAG = False +PROGRAM = "" +LOG_COUNTER = 2 +LOG_INTERVAL = 5 +NEXT_CHECK = 0 + +WRITE = sys.stderr.write +FLUSH = sys.stderr.flush + + +def log_msg(msg_type, *args): + """ + Print message on stderr. + :param msg_type: str + """ + global LOG_COUNTER + if not DEBUG_FLAG: + LOG_COUNTER -= 1 + now = time() + if LOG_COUNTER >= 0: + timestamp = strftime('%y-%m-%d %X') + msg = "%s: %s %s: %s" % (timestamp, PROGRAM, str(msg_type), " ".join(args)) + WRITE(msg + "\n") + FLUSH() + + global NEXT_CHECK + if NEXT_CHECK <= now: + NEXT_CHECK = now - (now % LOG_INTERVAL) + LOG_INTERVAL + if LOG_COUNTER < 0: + timestamp = strftime('%y-%m-%d %X') + msg = "%s: Prevented %s log messages from displaying" % (timestamp, str(0 - LOG_COUNTER)) + WRITE(msg + "\n") + FLUSH() + + +def debug(*args): + """ + Print debug message on stderr. + """ + if not DEBUG_FLAG: + return + + log_msg("DEBUG", *args) + + +def error(*args): + """ + Print message on stderr. + """ + log_msg("ERROR", *args) + + +def info(*args): + """ + Print message on stderr. + """ + log_msg("INFO", *args) + + +def fatal(*args): + """ + Print message on stderr and exit. + """ + log_msg("FATAL", *args) + # using sys.stdout causes IOError: Broken Pipe + print('DISABLE') + # sys.stdout.write('DISABLE\n') + sys.exit(1)
\ No newline at end of file diff --git a/python.d/python_modules/pyyaml2/__init__.py b/python.d/python_modules/pyyaml2/__init__.py new file mode 100644 index 000000000..76e19e13f --- /dev/null +++ b/python.d/python_modules/pyyaml2/__init__.py @@ -0,0 +1,315 @@ + +from error import * + +from tokens import * +from events import * +from nodes import * + +from loader import * +from dumper import * + +__version__ = '3.11' + +try: + from cyaml import * + __with_libyaml__ = True +except ImportError: + __with_libyaml__ = False + +def scan(stream, Loader=Loader): + """ + Scan a YAML stream and produce scanning tokens. + """ + loader = Loader(stream) + try: + while loader.check_token(): + yield loader.get_token() + finally: + loader.dispose() + +def parse(stream, Loader=Loader): + """ + Parse a YAML stream and produce parsing events. + """ + loader = Loader(stream) + try: + while loader.check_event(): + yield loader.get_event() + finally: + loader.dispose() + +def compose(stream, Loader=Loader): + """ + Parse the first YAML document in a stream + and produce the corresponding representation tree. + """ + loader = Loader(stream) + try: + return loader.get_single_node() + finally: + loader.dispose() + +def compose_all(stream, Loader=Loader): + """ + Parse all YAML documents in a stream + and produce corresponding representation trees. + """ + loader = Loader(stream) + try: + while loader.check_node(): + yield loader.get_node() + finally: + loader.dispose() + +def load(stream, Loader=Loader): + """ + Parse the first YAML document in a stream + and produce the corresponding Python object. + """ + loader = Loader(stream) + try: + return loader.get_single_data() + finally: + loader.dispose() + +def load_all(stream, Loader=Loader): + """ + Parse all YAML documents in a stream + and produce corresponding Python objects. + """ + loader = Loader(stream) + try: + while loader.check_data(): + yield loader.get_data() + finally: + loader.dispose() + +def safe_load(stream): + """ + Parse the first YAML document in a stream + and produce the corresponding Python object. + Resolve only basic YAML tags. + """ + return load(stream, SafeLoader) + +def safe_load_all(stream): + """ + Parse all YAML documents in a stream + and produce corresponding Python objects. + Resolve only basic YAML tags. + """ + return load_all(stream, SafeLoader) + +def emit(events, stream=None, Dumper=Dumper, + canonical=None, indent=None, width=None, + allow_unicode=None, line_break=None): + """ + Emit YAML parsing events into a stream. + If stream is None, return the produced string instead. + """ + getvalue = None + if stream is None: + from StringIO import StringIO + stream = StringIO() + getvalue = stream.getvalue + dumper = Dumper(stream, canonical=canonical, indent=indent, width=width, + allow_unicode=allow_unicode, line_break=line_break) + try: + for event in events: + dumper.emit(event) + finally: + dumper.dispose() + if getvalue: + return getvalue() + +def serialize_all(nodes, stream=None, Dumper=Dumper, + canonical=None, indent=None, width=None, + allow_unicode=None, line_break=None, + encoding='utf-8', explicit_start=None, explicit_end=None, + version=None, tags=None): + """ + Serialize a sequence of representation trees into a YAML stream. + If stream is None, return the produced string instead. + """ + getvalue = None + if stream is None: + if encoding is None: + from StringIO import StringIO + else: + from cStringIO import StringIO + stream = StringIO() + getvalue = stream.getvalue + dumper = Dumper(stream, canonical=canonical, indent=indent, width=width, + allow_unicode=allow_unicode, line_break=line_break, + encoding=encoding, version=version, tags=tags, + explicit_start=explicit_start, explicit_end=explicit_end) + try: + dumper.open() + for node in nodes: + dumper.serialize(node) + dumper.close() + finally: + dumper.dispose() + if getvalue: + return getvalue() + +def serialize(node, stream=None, Dumper=Dumper, **kwds): + """ + Serialize a representation tree into a YAML stream. + If stream is None, return the produced string instead. + """ + return serialize_all([node], stream, Dumper=Dumper, **kwds) + +def dump_all(documents, stream=None, Dumper=Dumper, + default_style=None, default_flow_style=None, + canonical=None, indent=None, width=None, + allow_unicode=None, line_break=None, + encoding='utf-8', explicit_start=None, explicit_end=None, + version=None, tags=None): + """ + Serialize a sequence of Python objects into a YAML stream. + If stream is None, return the produced string instead. + """ + getvalue = None + if stream is None: + if encoding is None: + from StringIO import StringIO + else: + from cStringIO import StringIO + stream = StringIO() + getvalue = stream.getvalue + dumper = Dumper(stream, default_style=default_style, + default_flow_style=default_flow_style, + canonical=canonical, indent=indent, width=width, + allow_unicode=allow_unicode, line_break=line_break, + encoding=encoding, version=version, tags=tags, + explicit_start=explicit_start, explicit_end=explicit_end) + try: + dumper.open() + for data in documents: + dumper.represent(data) + dumper.close() + finally: + dumper.dispose() + if getvalue: + return getvalue() + +def dump(data, stream=None, Dumper=Dumper, **kwds): + """ + Serialize a Python object into a YAML stream. + If stream is None, return the produced string instead. + """ + return dump_all([data], stream, Dumper=Dumper, **kwds) + +def safe_dump_all(documents, stream=None, **kwds): + """ + Serialize a sequence of Python objects into a YAML stream. + Produce only basic YAML tags. + If stream is None, return the produced string instead. + """ + return dump_all(documents, stream, Dumper=SafeDumper, **kwds) + +def safe_dump(data, stream=None, **kwds): + """ + Serialize a Python object into a YAML stream. + Produce only basic YAML tags. + If stream is None, return the produced string instead. + """ + return dump_all([data], stream, Dumper=SafeDumper, **kwds) + +def add_implicit_resolver(tag, regexp, first=None, + Loader=Loader, Dumper=Dumper): + """ + Add an implicit scalar detector. + If an implicit scalar value matches the given regexp, + the corresponding tag is assigned to the scalar. + first is a sequence of possible initial characters or None. + """ + Loader.add_implicit_resolver(tag, regexp, first) + Dumper.add_implicit_resolver(tag, regexp, first) + +def add_path_resolver(tag, path, kind=None, Loader=Loader, Dumper=Dumper): + """ + Add a path based resolver for the given tag. + A path is a list of keys that forms a path + to a node in the representation tree. + Keys can be string values, integers, or None. + """ + Loader.add_path_resolver(tag, path, kind) + Dumper.add_path_resolver(tag, path, kind) + +def add_constructor(tag, constructor, Loader=Loader): + """ + Add a constructor for the given tag. + Constructor is a function that accepts a Loader instance + and a node object and produces the corresponding Python object. + """ + Loader.add_constructor(tag, constructor) + +def add_multi_constructor(tag_prefix, multi_constructor, Loader=Loader): + """ + Add a multi-constructor for the given tag prefix. + Multi-constructor is called for a node if its tag starts with tag_prefix. + Multi-constructor accepts a Loader instance, a tag suffix, + and a node object and produces the corresponding Python object. + """ + Loader.add_multi_constructor(tag_prefix, multi_constructor) + +def add_representer(data_type, representer, Dumper=Dumper): + """ + Add a representer for the given type. + Representer is a function accepting a Dumper instance + and an instance of the given data type + and producing the corresponding representation node. + """ + Dumper.add_representer(data_type, representer) + +def add_multi_representer(data_type, multi_representer, Dumper=Dumper): + """ + Add a representer for the given type. + Multi-representer is a function accepting a Dumper instance + and an instance of the given data type or subtype + and producing the corresponding representation node. + """ + Dumper.add_multi_representer(data_type, multi_representer) + +class YAMLObjectMetaclass(type): + """ + The metaclass for YAMLObject. + """ + def __init__(cls, name, bases, kwds): + super(YAMLObjectMetaclass, cls).__init__(name, bases, kwds) + if 'yaml_tag' in kwds and kwds['yaml_tag'] is not None: + cls.yaml_loader.add_constructor(cls.yaml_tag, cls.from_yaml) + cls.yaml_dumper.add_representer(cls, cls.to_yaml) + +class YAMLObject(object): + """ + An object that can dump itself to a YAML stream + and load itself from a YAML stream. + """ + + __metaclass__ = YAMLObjectMetaclass + __slots__ = () # no direct instantiation, so allow immutable subclasses + + yaml_loader = Loader + yaml_dumper = Dumper + + yaml_tag = None + yaml_flow_style = None + + def from_yaml(cls, loader, node): + """ + Convert a representation node to a Python object. + """ + return loader.construct_yaml_object(node, cls) + from_yaml = classmethod(from_yaml) + + def to_yaml(cls, dumper, data): + """ + Convert a Python object to a representation node. + """ + return dumper.represent_yaml_object(cls.yaml_tag, data, cls, + flow_style=cls.yaml_flow_style) + to_yaml = classmethod(to_yaml) + diff --git a/python.d/python_modules/pyyaml2/composer.py b/python.d/python_modules/pyyaml2/composer.py new file mode 100644 index 000000000..06e5ac782 --- /dev/null +++ b/python.d/python_modules/pyyaml2/composer.py @@ -0,0 +1,139 @@ + +__all__ = ['Composer', 'ComposerError'] + +from error import MarkedYAMLError +from events import * +from nodes import * + +class ComposerError(MarkedYAMLError): + pass + +class Composer(object): + + def __init__(self): + self.anchors = {} + + def check_node(self): + # Drop the STREAM-START event. + if self.check_event(StreamStartEvent): + self.get_event() + + # If there are more documents available? + return not self.check_event(StreamEndEvent) + + def get_node(self): + # Get the root node of the next document. + if not self.check_event(StreamEndEvent): + return self.compose_document() + + def get_single_node(self): + # Drop the STREAM-START event. + self.get_event() + + # Compose a document if the stream is not empty. + document = None + if not self.check_event(StreamEndEvent): + document = self.compose_document() + + # Ensure that the stream contains no more documents. + if not self.check_event(StreamEndEvent): + event = self.get_event() + raise ComposerError("expected a single document in the stream", + document.start_mark, "but found another document", + event.start_mark) + + # Drop the STREAM-END event. + self.get_event() + + return document + + def compose_document(self): + # Drop the DOCUMENT-START event. + self.get_event() + + # Compose the root node. + node = self.compose_node(None, None) + + # Drop the DOCUMENT-END event. + self.get_event() + + self.anchors = {} + return node + + def compose_node(self, parent, index): + if self.check_event(AliasEvent): + event = self.get_event() + anchor = event.anchor + if anchor not in self.anchors: + raise ComposerError(None, None, "found undefined alias %r" + % anchor.encode('utf-8'), event.start_mark) + return self.anchors[anchor] + event = self.peek_event() + anchor = event.anchor + if anchor is not None: + if anchor in self.anchors: + raise ComposerError("found duplicate anchor %r; first occurence" + % anchor.encode('utf-8'), self.anchors[anchor].start_mark, + "second occurence", event.start_mark) + self.descend_resolver(parent, index) + if self.check_event(ScalarEvent): + node = self.compose_scalar_node(anchor) + elif self.check_event(SequenceStartEvent): + node = self.compose_sequence_node(anchor) + elif self.check_event(MappingStartEvent): + node = self.compose_mapping_node(anchor) + self.ascend_resolver() + return node + + def compose_scalar_node(self, anchor): + event = self.get_event() + tag = event.tag + if tag is None or tag == u'!': + tag = self.resolve(ScalarNode, event.value, event.implicit) + node = ScalarNode(tag, event.value, + event.start_mark, event.end_mark, style=event.style) + if anchor is not None: + self.anchors[anchor] = node + return node + + def compose_sequence_node(self, anchor): + start_event = self.get_event() + tag = start_event.tag + if tag is None or tag == u'!': + tag = self.resolve(SequenceNode, None, start_event.implicit) + node = SequenceNode(tag, [], + start_event.start_mark, None, + flow_style=start_event.flow_style) + if anchor is not None: + self.anchors[anchor] = node + index = 0 + while not self.check_event(SequenceEndEvent): + node.value.append(self.compose_node(node, index)) + index += 1 + end_event = self.get_event() + node.end_mark = end_event.end_mark + return node + + def compose_mapping_node(self, anchor): + start_event = self.get_event() + tag = start_event.tag + if tag is None or tag == u'!': + tag = self.resolve(MappingNode, None, start_event.implicit) + node = MappingNode(tag, [], + start_event.start_mark, None, + flow_style=start_event.flow_style) + if anchor is not None: + self.anchors[anchor] = node + while not self.check_event(MappingEndEvent): + #key_event = self.peek_event() + item_key = self.compose_node(node, None) + #if item_key in node.value: + # raise ComposerError("while composing a mapping", start_event.start_mark, + # "found duplicate key", key_event.start_mark) + item_value = self.compose_node(node, item_key) + #node.value[item_key] = item_value + node.value.append((item_key, item_value)) + end_event = self.get_event() + node.end_mark = end_event.end_mark + return node + diff --git a/python.d/python_modules/pyyaml2/constructor.py b/python.d/python_modules/pyyaml2/constructor.py new file mode 100644 index 000000000..635faac3e --- /dev/null +++ b/python.d/python_modules/pyyaml2/constructor.py @@ -0,0 +1,675 @@ + +__all__ = ['BaseConstructor', 'SafeConstructor', 'Constructor', + 'ConstructorError'] + +from error import * +from nodes import * + +import datetime + +import binascii, re, sys, types + +class ConstructorError(MarkedYAMLError): + pass + +class BaseConstructor(object): + + yaml_constructors = {} + yaml_multi_constructors = {} + + def __init__(self): + self.constructed_objects = {} + self.recursive_objects = {} + self.state_generators = [] + self.deep_construct = False + + def check_data(self): + # If there are more documents available? + return self.check_node() + + def get_data(self): + # Construct and return the next document. + if self.check_node(): + return self.construct_document(self.get_node()) + + def get_single_data(self): + # Ensure that the stream contains a single document and construct it. + node = self.get_single_node() + if node is not None: + return self.construct_document(node) + return None + + def construct_document(self, node): + data = self.construct_object(node) + while self.state_generators: + state_generators = self.state_generators + self.state_generators = [] + for generator in state_generators: + for dummy in generator: + pass + self.constructed_objects = {} + self.recursive_objects = {} + self.deep_construct = False + return data + + def construct_object(self, node, deep=False): + if node in self.constructed_objects: + return self.constructed_objects[node] + if deep: + old_deep = self.deep_construct + self.deep_construct = True + if node in self.recursive_objects: + raise ConstructorError(None, None, + "found unconstructable recursive node", node.start_mark) + self.recursive_objects[node] = None + constructor = None + tag_suffix = None + if node.tag in self.yaml_constructors: + constructor = self.yaml_constructors[node.tag] + else: + for tag_prefix in self.yaml_multi_constructors: + if node.tag.startswith(tag_prefix): + tag_suffix = node.tag[len(tag_prefix):] + constructor = self.yaml_multi_constructors[tag_prefix] + break + else: + if None in self.yaml_multi_constructors: + tag_suffix = node.tag + constructor = self.yaml_multi_constructors[None] + elif None in self.yaml_constructors: + constructor = self.yaml_constructors[None] + elif isinstance(node, ScalarNode): + constructor = self.__class__.construct_scalar + elif isinstance(node, SequenceNode): + constructor = self.__class__.construct_sequence + elif isinstance(node, MappingNode): + constructor = self.__class__.construct_mapping + if tag_suffix is None: + data = constructor(self, node) + else: + data = constructor(self, tag_suffix, node) + if isinstance(data, types.GeneratorType): + generator = data + data = generator.next() + if self.deep_construct: + for dummy in generator: + pass + else: + self.state_generators.append(generator) + self.constructed_objects[node] = data + del self.recursive_objects[node] + if deep: + self.deep_construct = old_deep + return data + + def construct_scalar(self, node): + if not isinstance(node, ScalarNode): + raise ConstructorError(None, None, + "expected a scalar node, but found %s" % node.id, + node.start_mark) + return node.value + + def construct_sequence(self, node, deep=False): + if not isinstance(node, SequenceNode): + raise ConstructorError(None, None, + "expected a sequence node, but found %s" % node.id, + node.start_mark) + return [self.construct_object(child, deep=deep) + for child in node.value] + + def construct_mapping(self, node, deep=False): + if not isinstance(node, MappingNode): + raise ConstructorError(None, None, + "expected a mapping node, but found %s" % node.id, + node.start_mark) + mapping = {} + for key_node, value_node in node.value: + key = self.construct_object(key_node, deep=deep) + try: + hash(key) + except TypeError, exc: + raise ConstructorError("while constructing a mapping", node.start_mark, + "found unacceptable key (%s)" % exc, key_node.start_mark) + value = self.construct_object(value_node, deep=deep) + mapping[key] = value + return mapping + + def construct_pairs(self, node, deep=False): + if not isinstance(node, MappingNode): + raise ConstructorError(None, None, + "expected a mapping node, but found %s" % node.id, + node.start_mark) + pairs = [] + for key_node, value_node in node.value: + key = self.construct_object(key_node, deep=deep) + value = self.construct_object(value_node, deep=deep) + pairs.append((key, value)) + return pairs + + def add_constructor(cls, tag, constructor): + if not 'yaml_constructors' in cls.__dict__: + cls.yaml_constructors = cls.yaml_constructors.copy() + cls.yaml_constructors[tag] = constructor + add_constructor = classmethod(add_constructor) + + def add_multi_constructor(cls, tag_prefix, multi_constructor): + if not 'yaml_multi_constructors' in cls.__dict__: + cls.yaml_multi_constructors = cls.yaml_multi_constructors.copy() + cls.yaml_multi_constructors[tag_prefix] = multi_constructor + add_multi_constructor = classmethod(add_multi_constructor) + +class SafeConstructor(BaseConstructor): + + def construct_scalar(self, node): + if isinstance(node, MappingNode): + for key_node, value_node in node.value: + if key_node.tag == u'tag:yaml.org,2002:value': + return self.construct_scalar(value_node) + return BaseConstructor.construct_scalar(self, node) + + def flatten_mapping(self, node): + merge = [] + index = 0 + while index < len(node.value): + key_node, value_node = node.value[index] + if key_node.tag == u'tag:yaml.org,2002:merge': + del node.value[index] + if isinstance(value_node, MappingNode): + self.flatten_mapping(value_node) + merge.extend(value_node.value) + elif isinstance(value_node, SequenceNode): + submerge = [] + for subnode in value_node.value: + if not isinstance(subnode, MappingNode): + raise ConstructorError("while constructing a mapping", + node.start_mark, + "expected a mapping for merging, but found %s" + % subnode.id, subnode.start_mark) + self.flatten_mapping(subnode) + submerge.append(subnode.value) + submerge.reverse() + for value in submerge: + merge.extend(value) + else: + raise ConstructorError("while constructing a mapping", node.start_mark, + "expected a mapping or list of mappings for merging, but found %s" + % value_node.id, value_node.start_mark) + elif key_node.tag == u'tag:yaml.org,2002:value': + key_node.tag = u'tag:yaml.org,2002:str' + index += 1 + else: + index += 1 + if merge: + node.value = merge + node.value + + def construct_mapping(self, node, deep=False): + if isinstance(node, MappingNode): + self.flatten_mapping(node) + return BaseConstructor.construct_mapping(self, node, deep=deep) + + def construct_yaml_null(self, node): + self.construct_scalar(node) + return None + + bool_values = { + u'yes': True, + u'no': False, + u'true': True, + u'false': False, + u'on': True, + u'off': False, + } + + def construct_yaml_bool(self, node): + value = self.construct_scalar(node) + return self.bool_values[value.lower()] + + def construct_yaml_int(self, node): + value = str(self.construct_scalar(node)) + value = value.replace('_', '') + sign = +1 + if value[0] == '-': + sign = -1 + if value[0] in '+-': + value = value[1:] + if value == '0': + return 0 + elif value.startswith('0b'): + return sign*int(value[2:], 2) + elif value.startswith('0x'): + return sign*int(value[2:], 16) + elif value[0] == '0': + return sign*int(value, 8) + elif ':' in value: + digits = [int(part) for part in value.split(':')] + digits.reverse() + base = 1 + value = 0 + for digit in digits: + value += digit*base + base *= 60 + return sign*value + else: + return sign*int(value) + + inf_value = 1e300 + while inf_value != inf_value*inf_value: + inf_value *= inf_value + nan_value = -inf_value/inf_value # Trying to make a quiet NaN (like C99). + + def construct_yaml_float(self, node): + value = str(self.construct_scalar(node)) + value = value.replace('_', '').lower() + sign = +1 + if value[0] == '-': + sign = -1 + if value[0] in '+-': + value = value[1:] + if value == '.inf': + return sign*self.inf_value + elif value == '.nan': + return self.nan_value + elif ':' in value: + digits = [float(part) for part in value.split(':')] + digits.reverse() + base = 1 + value = 0.0 + for digit in digits: + value += digit*base + base *= 60 + return sign*value + else: + return sign*float(value) + + def construct_yaml_binary(self, node): + value = self.construct_scalar(node) + try: + return str(value).decode('base64') + except (binascii.Error, UnicodeEncodeError), exc: + raise ConstructorError(None, None, + "failed to decode base64 data: %s" % exc, node.start_mark) + + timestamp_regexp = re.compile( + ur'''^(?P<year>[0-9][0-9][0-9][0-9]) + -(?P<month>[0-9][0-9]?) + -(?P<day>[0-9][0-9]?) + (?:(?:[Tt]|[ \t]+) + (?P<hour>[0-9][0-9]?) + :(?P<minute>[0-9][0-9]) + :(?P<second>[0-9][0-9]) + (?:\.(?P<fraction>[0-9]*))? + (?:[ \t]*(?P<tz>Z|(?P<tz_sign>[-+])(?P<tz_hour>[0-9][0-9]?) + (?::(?P<tz_minute>[0-9][0-9]))?))?)?$''', re.X) + + def construct_yaml_timestamp(self, node): + value = self.construct_scalar(node) + match = self.timestamp_regexp.match(node.value) + values = match.groupdict() + year = int(values['year']) + month = int(values['month']) + day = int(values['day']) + if not values['hour']: + return datetime.date(year, month, day) + hour = int(values['hour']) + minute = int(values['minute']) + second = int(values['second']) + fraction = 0 + if values['fraction']: + fraction = values['fraction'][:6] + while len(fraction) < 6: + fraction += '0' + fraction = int(fraction) + delta = None + if values['tz_sign']: + tz_hour = int(values['tz_hour']) + tz_minute = int(values['tz_minute'] or 0) + delta = datetime.timedelta(hours=tz_hour, minutes=tz_minute) + if values['tz_sign'] == '-': + delta = -delta + data = datetime.datetime(year, month, day, hour, minute, second, fraction) + if delta: + data -= delta + return data + + def construct_yaml_omap(self, node): + # Note: we do not check for duplicate keys, because it's too + # CPU-expensive. + omap = [] + yield omap + if not isinstance(node, SequenceNode): + raise ConstructorError("while constructing an ordered map", node.start_mark, + "expected a sequence, but found %s" % node.id, node.start_mark) + for subnode in node.value: + if not isinstance(subnode, MappingNode): + raise ConstructorError("while constructing an ordered map", node.start_mark, + "expected a mapping of length 1, but found %s" % subnode.id, + subnode.start_mark) + if len(subnode.value) != 1: + raise ConstructorError("while constructing an ordered map", node.start_mark, + "expected a single mapping item, but found %d items" % len(subnode.value), + subnode.start_mark) + key_node, value_node = subnode.value[0] + key = self.construct_object(key_node) + value = self.construct_object(value_node) + omap.append((key, value)) + + def construct_yaml_pairs(self, node): + # Note: the same code as `construct_yaml_omap`. + pairs = [] + yield pairs + if not isinstance(node, SequenceNode): + raise ConstructorError("while constructing pairs", node.start_mark, + "expected a sequence, but found %s" % node.id, node.start_mark) + for subnode in node.value: + if not isinstance(subnode, MappingNode): + raise ConstructorError("while constructing pairs", node.start_mark, + "expected a mapping of length 1, but found %s" % subnode.id, + subnode.start_mark) + if len(subnode.value) != 1: + raise ConstructorError("while constructing pairs", node.start_mark, + "expected a single mapping item, but found %d items" % len(subnode.value), + subnode.start_mark) + key_node, value_node = subnode.value[0] + key = self.construct_object(key_node) + value = self.construct_object(value_node) + pairs.append((key, value)) + + def construct_yaml_set(self, node): + data = set() + yield data + value = self.construct_mapping(node) + data.update(value) + + def construct_yaml_str(self, node): + value = self.construct_scalar(node) + try: + return value.encode('ascii') + except UnicodeEncodeError: + return value + + def construct_yaml_seq(self, node): + data = [] + yield data + data.extend(self.construct_sequence(node)) + + def construct_yaml_map(self, node): + data = {} + yield data + value = self.construct_mapping(node) + data.update(value) + + def construct_yaml_object(self, node, cls): + data = cls.__new__(cls) + yield data + if hasattr(data, '__setstate__'): + state = self.construct_mapping(node, deep=True) + data.__setstate__(state) + else: + state = self.construct_mapping(node) + data.__dict__.update(state) + + def construct_undefined(self, node): + raise ConstructorError(None, None, + "could not determine a constructor for the tag %r" % node.tag.encode('utf-8'), + node.start_mark) + +SafeConstructor.add_constructor( + u'tag:yaml.org,2002:null', + SafeConstructor.construct_yaml_null) + +SafeConstructor.add_constructor( + u'tag:yaml.org,2002:bool', + SafeConstructor.construct_yaml_bool) + +SafeConstructor.add_constructor( + u'tag:yaml.org,2002:int', + SafeConstructor.construct_yaml_int) + +SafeConstructor.add_constructor( + u'tag:yaml.org,2002:float', + SafeConstructor.construct_yaml_float) + +SafeConstructor.add_constructor( + u'tag:yaml.org,2002:binary', + SafeConstructor.construct_yaml_binary) + +SafeConstructor.add_constructor( + u'tag:yaml.org,2002:timestamp', + SafeConstructor.construct_yaml_timestamp) + +SafeConstructor.add_constructor( + u'tag:yaml.org,2002:omap', + SafeConstructor.construct_yaml_omap) + +SafeConstructor.add_constructor( + u'tag:yaml.org,2002:pairs', + SafeConstructor.construct_yaml_pairs) + +SafeConstructor.add_constructor( + u'tag:yaml.org,2002:set', + SafeConstructor.construct_yaml_set) + +SafeConstructor.add_constructor( + u'tag:yaml.org,2002:str', + SafeConstructor.construct_yaml_str) + +SafeConstructor.add_constructor( + u'tag:yaml.org,2002:seq', + SafeConstructor.construct_yaml_seq) + +SafeConstructor.add_constructor( + u'tag:yaml.org,2002:map', + SafeConstructor.construct_yaml_map) + +SafeConstructor.add_constructor(None, + SafeConstructor.construct_undefined) + +class Constructor(SafeConstructor): + + def construct_python_str(self, node): + return self.construct_scalar(node).encode('utf-8') + + def construct_python_unicode(self, node): + return self.construct_scalar(node) + + def construct_python_long(self, node): + return long(self.construct_yaml_int(node)) + + def construct_python_complex(self, node): + return complex(self.construct_scalar(node)) + + def construct_python_tuple(self, node): + return tuple(self.construct_sequence(node)) + + def find_python_module(self, name, mark): + if not name: + raise ConstructorError("while constructing a Python module", mark, + "expected non-empty name appended to the tag", mark) + try: + __import__(name) + except ImportError, exc: + raise ConstructorError("while constructing a Python module", mark, + "cannot find module %r (%s)" % (name.encode('utf-8'), exc), mark) + return sys.modules[name] + + def find_python_name(self, name, mark): + if not name: + raise ConstructorError("while constructing a Python object", mark, + "expected non-empty name appended to the tag", mark) + if u'.' in name: + module_name, object_name = name.rsplit('.', 1) + else: + module_name = '__builtin__' + object_name = name + try: + __import__(module_name) + except ImportError, exc: + raise ConstructorError("while constructing a Python object", mark, + "cannot find module %r (%s)" % (module_name.encode('utf-8'), exc), mark) + module = sys.modules[module_name] + if not hasattr(module, object_name): + raise ConstructorError("while constructing a Python object", mark, + "cannot find %r in the module %r" % (object_name.encode('utf-8'), + module.__name__), mark) + return getattr(module, object_name) + + def construct_python_name(self, suffix, node): + value = self.construct_scalar(node) + if value: + raise ConstructorError("while constructing a Python name", node.start_mark, + "expected the empty value, but found %r" % value.encode('utf-8'), + node.start_mark) + return self.find_python_name(suffix, node.start_mark) + + def construct_python_module(self, suffix, node): + value = self.construct_scalar(node) + if value: + raise ConstructorError("while constructing a Python module", node.start_mark, + "expected the empty value, but found %r" % value.encode('utf-8'), + node.start_mark) + return self.find_python_module(suffix, node.start_mark) + + class classobj: pass + + def make_python_instance(self, suffix, node, + args=None, kwds=None, newobj=False): + if not args: + args = [] + if not kwds: + kwds = {} + cls = self.find_python_name(suffix, node.start_mark) + if newobj and isinstance(cls, type(self.classobj)) \ + and not args and not kwds: + instance = self.classobj() + instance.__class__ = cls + return instance + elif newobj and isinstance(cls, type): + return cls.__new__(cls, *args, **kwds) + else: + return cls(*args, **kwds) + + def set_python_instance_state(self, instance, state): + if hasattr(instance, '__setstate__'): + instance.__setstate__(state) + else: + slotstate = {} + if isinstance(state, tuple) and len(state) == 2: + state, slotstate = state + if hasattr(instance, '__dict__'): + instance.__dict__.update(state) + elif state: + slotstate.update(state) + for key, value in slotstate.items(): + setattr(object, key, value) + + def construct_python_object(self, suffix, node): + # Format: + # !!python/object:module.name { ... state ... } + instance = self.make_python_instance(suffix, node, newobj=True) + yield instance + deep = hasattr(instance, '__setstate__') + state = self.construct_mapping(node, deep=deep) + self.set_python_instance_state(instance, state) + + def construct_python_object_apply(self, suffix, node, newobj=False): + # Format: + # !!python/object/apply # (or !!python/object/new) + # args: [ ... arguments ... ] + # kwds: { ... keywords ... } + # state: ... state ... + # listitems: [ ... listitems ... ] + # dictitems: { ... dictitems ... } + # or short format: + # !!python/object/apply [ ... arguments ... ] + # The difference between !!python/object/apply and !!python/object/new + # is how an object is created, check make_python_instance for details. + if isinstance(node, SequenceNode): + args = self.construct_sequence(node, deep=True) + kwds = {} + state = {} + listitems = [] + dictitems = {} + else: + value = self.construct_mapping(node, deep=True) + args = value.get('args', []) + kwds = value.get('kwds', {}) + state = value.get('state', {}) + listitems = value.get('listitems', []) + dictitems = value.get('dictitems', {}) + instance = self.make_python_instance(suffix, node, args, kwds, newobj) + if state: + self.set_python_instance_state(instance, state) + if listitems: + instance.extend(listitems) + if dictitems: + for key in dictitems: + instance[key] = dictitems[key] + return instance + + def construct_python_object_new(self, suffix, node): + return self.construct_python_object_apply(suffix, node, newobj=True) + +Constructor.add_constructor( + u'tag:yaml.org,2002:python/none', + Constructor.construct_yaml_null) + +Constructor.add_constructor( + u'tag:yaml.org,2002:python/bool', + Constructor.construct_yaml_bool) + +Constructor.add_constructor( + u'tag:yaml.org,2002:python/str', + Constructor.construct_python_str) + +Constructor.add_constructor( + u'tag:yaml.org,2002:python/unicode', + Constructor.construct_python_unicode) + +Constructor.add_constructor( + u'tag:yaml.org,2002:python/int', + Constructor.construct_yaml_int) + +Constructor.add_constructor( + u'tag:yaml.org,2002:python/long', + Constructor.construct_python_long) + +Constructor.add_constructor( + u'tag:yaml.org,2002:python/float', + Constructor.construct_yaml_float) + +Constructor.add_constructor( + u'tag:yaml.org,2002:python/complex', + Constructor.construct_python_complex) + +Constructor.add_constructor( + u'tag:yaml.org,2002:python/list', + Constructor.construct_yaml_seq) + +Constructor.add_constructor( + u'tag:yaml.org,2002:python/tuple', + Constructor.construct_python_tuple) + +Constructor.add_constructor( + u'tag:yaml.org,2002:python/dict', + Constructor.construct_yaml_map) + +Constructor.add_multi_constructor( + u'tag:yaml.org,2002:python/name:', + Constructor.construct_python_name) + +Constructor.add_multi_constructor( + u'tag:yaml.org,2002:python/module:', + Constructor.construct_python_module) + +Constructor.add_multi_constructor( + u'tag:yaml.org,2002:python/object:', + Constructor.construct_python_object) + +Constructor.add_multi_constructor( + u'tag:yaml.org,2002:python/object/apply:', + Constructor.construct_python_object_apply) + +Constructor.add_multi_constructor( + u'tag:yaml.org,2002:python/object/new:', + Constructor.construct_python_object_new) + diff --git a/python.d/python_modules/pyyaml2/cyaml.py b/python.d/python_modules/pyyaml2/cyaml.py new file mode 100644 index 000000000..68dcd7519 --- /dev/null +++ b/python.d/python_modules/pyyaml2/cyaml.py @@ -0,0 +1,85 @@ + +__all__ = ['CBaseLoader', 'CSafeLoader', 'CLoader', + 'CBaseDumper', 'CSafeDumper', 'CDumper'] + +from _yaml import CParser, CEmitter + +from constructor import * + +from serializer import * +from representer import * + +from resolver import * + +class CBaseLoader(CParser, BaseConstructor, BaseResolver): + + def __init__(self, stream): + CParser.__init__(self, stream) + BaseConstructor.__init__(self) + BaseResolver.__init__(self) + +class CSafeLoader(CParser, SafeConstructor, Resolver): + + def __init__(self, stream): + CParser.__init__(self, stream) + SafeConstructor.__init__(self) + Resolver.__init__(self) + +class CLoader(CParser, Constructor, Resolver): + + def __init__(self, stream): + CParser.__init__(self, stream) + Constructor.__init__(self) + Resolver.__init__(self) + +class CBaseDumper(CEmitter, BaseRepresenter, BaseResolver): + + def __init__(self, stream, + default_style=None, default_flow_style=None, + canonical=None, indent=None, width=None, + allow_unicode=None, line_break=None, + encoding=None, explicit_start=None, explicit_end=None, + version=None, tags=None): + CEmitter.__init__(self, stream, canonical=canonical, + indent=indent, width=width, encoding=encoding, + allow_unicode=allow_unicode, line_break=line_break, + explicit_start=explicit_start, explicit_end=explicit_end, + version=version, tags=tags) + Representer.__init__(self, default_style=default_style, + default_flow_style=default_flow_style) + Resolver.__init__(self) + +class CSafeDumper(CEmitter, SafeRepresenter, Resolver): + + def __init__(self, stream, + default_style=None, default_flow_style=None, + canonical=None, indent=None, width=None, + allow_unicode=None, line_break=None, + encoding=None, explicit_start=None, explicit_end=None, + version=None, tags=None): + CEmitter.__init__(self, stream, canonical=canonical, + indent=indent, width=width, encoding=encoding, + allow_unicode=allow_unicode, line_break=line_break, + explicit_start=explicit_start, explicit_end=explicit_end, + version=version, tags=tags) + SafeRepresenter.__init__(self, default_style=default_style, + default_flow_style=default_flow_style) + Resolver.__init__(self) + +class CDumper(CEmitter, Serializer, Representer, Resolver): + + def __init__(self, stream, + default_style=None, default_flow_style=None, + canonical=None, indent=None, width=None, + allow_unicode=None, line_break=None, + encoding=None, explicit_start=None, explicit_end=None, + version=None, tags=None): + CEmitter.__init__(self, stream, canonical=canonical, + indent=indent, width=width, encoding=encoding, + allow_unicode=allow_unicode, line_break=line_break, + explicit_start=explicit_start, explicit_end=explicit_end, + version=version, tags=tags) + Representer.__init__(self, default_style=default_style, + default_flow_style=default_flow_style) + Resolver.__init__(self) + diff --git a/python.d/python_modules/pyyaml2/dumper.py b/python.d/python_modules/pyyaml2/dumper.py new file mode 100644 index 000000000..f811d2c91 --- /dev/null +++ b/python.d/python_modules/pyyaml2/dumper.py @@ -0,0 +1,62 @@ + +__all__ = ['BaseDumper', 'SafeDumper', 'Dumper'] + +from emitter import * +from serializer import * +from representer import * +from resolver import * + +class BaseDumper(Emitter, Serializer, BaseRepresenter, BaseResolver): + + def __init__(self, stream, + default_style=None, default_flow_style=None, + canonical=None, indent=None, width=None, + allow_unicode=None, line_break=None, + encoding=None, explicit_start=None, explicit_end=None, + version=None, tags=None): + Emitter.__init__(self, stream, canonical=canonical, + indent=indent, width=width, + allow_unicode=allow_unicode, line_break=line_break) + Serializer.__init__(self, encoding=encoding, + explicit_start=explicit_start, explicit_end=explicit_end, + version=version, tags=tags) + Representer.__init__(self, default_style=default_style, + default_flow_style=default_flow_style) + Resolver.__init__(self) + +class SafeDumper(Emitter, Serializer, SafeRepresenter, Resolver): + + def __init__(self, stream, + default_style=None, default_flow_style=None, + canonical=None, indent=None, width=None, + allow_unicode=None, line_break=None, + encoding=None, explicit_start=None, explicit_end=None, + version=None, tags=None): + Emitter.__init__(self, stream, canonical=canonical, + indent=indent, width=width, + allow_unicode=allow_unicode, line_break=line_break) + Serializer.__init__(self, encoding=encoding, + explicit_start=explicit_start, explicit_end=explicit_end, + version=version, tags=tags) + SafeRepresenter.__init__(self, default_style=default_style, + default_flow_style=default_flow_style) + Resolver.__init__(self) + +class Dumper(Emitter, Serializer, Representer, Resolver): + + def __init__(self, stream, + default_style=None, default_flow_style=None, + canonical=None, indent=None, width=None, + allow_unicode=None, line_break=None, + encoding=None, explicit_start=None, explicit_end=None, + version=None, tags=None): + Emitter.__init__(self, stream, canonical=canonical, + indent=indent, width=width, + allow_unicode=allow_unicode, line_break=line_break) + Serializer.__init__(self, encoding=encoding, + explicit_start=explicit_start, explicit_end=explicit_end, + version=version, tags=tags) + Representer.__init__(self, default_style=default_style, + default_flow_style=default_flow_style) + Resolver.__init__(self) + diff --git a/python.d/python_modules/pyyaml2/emitter.py b/python.d/python_modules/pyyaml2/emitter.py new file mode 100644 index 000000000..e5bcdcccb --- /dev/null +++ b/python.d/python_modules/pyyaml2/emitter.py @@ -0,0 +1,1140 @@ + +# Emitter expects events obeying the following grammar: +# stream ::= STREAM-START document* STREAM-END +# document ::= DOCUMENT-START node DOCUMENT-END +# node ::= SCALAR | sequence | mapping +# sequence ::= SEQUENCE-START node* SEQUENCE-END +# mapping ::= MAPPING-START (node node)* MAPPING-END + +__all__ = ['Emitter', 'EmitterError'] + +from error import YAMLError +from events import * + +class EmitterError(YAMLError): + pass + +class ScalarAnalysis(object): + def __init__(self, scalar, empty, multiline, + allow_flow_plain, allow_block_plain, + allow_single_quoted, allow_double_quoted, + allow_block): + self.scalar = scalar + self.empty = empty + self.multiline = multiline + self.allow_flow_plain = allow_flow_plain + self.allow_block_plain = allow_block_plain + self.allow_single_quoted = allow_single_quoted + self.allow_double_quoted = allow_double_quoted + self.allow_block = allow_block + +class Emitter(object): + + DEFAULT_TAG_PREFIXES = { + u'!' : u'!', + u'tag:yaml.org,2002:' : u'!!', + } + + def __init__(self, stream, canonical=None, indent=None, width=None, + allow_unicode=None, line_break=None): + + # The stream should have the methods `write` and possibly `flush`. + self.stream = stream + + # Encoding can be overriden by STREAM-START. + self.encoding = None + + # Emitter is a state machine with a stack of states to handle nested + # structures. + self.states = [] + self.state = self.expect_stream_start + + # Current event and the event queue. + self.events = [] + self.event = None + + # The current indentation level and the stack of previous indents. + self.indents = [] + self.indent = None + + # Flow level. + self.flow_level = 0 + + # Contexts. + self.root_context = False + self.sequence_context = False + self.mapping_context = False + self.simple_key_context = False + + # Characteristics of the last emitted character: + # - current position. + # - is it a whitespace? + # - is it an indention character + # (indentation space, '-', '?', or ':')? + self.line = 0 + self.column = 0 + self.whitespace = True + self.indention = True + + # Whether the document requires an explicit document indicator + self.open_ended = False + + # Formatting details. + self.canonical = canonical + self.allow_unicode = allow_unicode + self.best_indent = 2 + if indent and 1 < indent < 10: + self.best_indent = indent + self.best_width = 80 + if width and width > self.best_indent*2: + self.best_width = width + self.best_line_break = u'\n' + if line_break in [u'\r', u'\n', u'\r\n']: + self.best_line_break = line_break + + # Tag prefixes. + self.tag_prefixes = None + + # Prepared anchor and tag. + self.prepared_anchor = None + self.prepared_tag = None + + # Scalar analysis and style. + self.analysis = None + self.style = None + + def dispose(self): + # Reset the state attributes (to clear self-references) + self.states = [] + self.state = None + + def emit(self, event): + self.events.append(event) + while not self.need_more_events(): + self.event = self.events.pop(0) + self.state() + self.event = None + + # In some cases, we wait for a few next events before emitting. + + def need_more_events(self): + if not self.events: + return True + event = self.events[0] + if isinstance(event, DocumentStartEvent): + return self.need_events(1) + elif isinstance(event, SequenceStartEvent): + return self.need_events(2) + elif isinstance(event, MappingStartEvent): + return self.need_events(3) + else: + return False + + def need_events(self, count): + level = 0 + for event in self.events[1:]: + if isinstance(event, (DocumentStartEvent, CollectionStartEvent)): + level += 1 + elif isinstance(event, (DocumentEndEvent, CollectionEndEvent)): + level -= 1 + elif isinstance(event, StreamEndEvent): + level = -1 + if level < 0: + return False + return (len(self.events) < count+1) + + def increase_indent(self, flow=False, indentless=False): + self.indents.append(self.indent) + if self.indent is None: + if flow: + self.indent = self.best_indent + else: + self.indent = 0 + elif not indentless: + self.indent += self.best_indent + + # States. + + # Stream handlers. + + def expect_stream_start(self): + if isinstance(self.event, StreamStartEvent): + if self.event.encoding and not getattr(self.stream, 'encoding', None): + self.encoding = self.event.encoding + self.write_stream_start() + self.state = self.expect_first_document_start + else: + raise EmitterError("expected StreamStartEvent, but got %s" + % self.event) + + def expect_nothing(self): + raise EmitterError("expected nothing, but got %s" % self.event) + + # Document handlers. + + def expect_first_document_start(self): + return self.expect_document_start(first=True) + + def expect_document_start(self, first=False): + if isinstance(self.event, DocumentStartEvent): + if (self.event.version or self.event.tags) and self.open_ended: + self.write_indicator(u'...', True) + self.write_indent() + if self.event.version: + version_text = self.prepare_version(self.event.version) + self.write_version_directive(version_text) + self.tag_prefixes = self.DEFAULT_TAG_PREFIXES.copy() + if self.event.tags: + handles = self.event.tags.keys() + handles.sort() + for handle in handles: + prefix = self.event.tags[handle] + self.tag_prefixes[prefix] = handle + handle_text = self.prepare_tag_handle(handle) + prefix_text = self.prepare_tag_prefix(prefix) + self.write_tag_directive(handle_text, prefix_text) + implicit = (first and not self.event.explicit and not self.canonical + and not self.event.version and not self.event.tags + and not self.check_empty_document()) + if not implicit: + self.write_indent() + self.write_indicator(u'---', True) + if self.canonical: + self.write_indent() + self.state = self.expect_document_root + elif isinstance(self.event, StreamEndEvent): + if self.open_ended: + self.write_indicator(u'...', True) + self.write_indent() + self.write_stream_end() + self.state = self.expect_nothing + else: + raise EmitterError("expected DocumentStartEvent, but got %s" + % self.event) + + def expect_document_end(self): + if isinstance(self.event, DocumentEndEvent): + self.write_indent() + if self.event.explicit: + self.write_indicator(u'...', True) + self.write_indent() + self.flush_stream() + self.state = self.expect_document_start + else: + raise EmitterError("expected DocumentEndEvent, but got %s" + % self.event) + + def expect_document_root(self): + self.states.append(self.expect_document_end) + self.expect_node(root=True) + + # Node handlers. + + def expect_node(self, root=False, sequence=False, mapping=False, + simple_key=False): + self.root_context = root + self.sequence_context = sequence + self.mapping_context = mapping + self.simple_key_context = simple_key + if isinstance(self.event, AliasEvent): + self.expect_alias() + elif isinstance(self.event, (ScalarEvent, CollectionStartEvent)): + self.process_anchor(u'&') + self.process_tag() + if isinstance(self.event, ScalarEvent): + self.expect_scalar() + elif isinstance(self.event, SequenceStartEvent): + if self.flow_level or self.canonical or self.event.flow_style \ + or self.check_empty_sequence(): + self.expect_flow_sequence() + else: + self.expect_block_sequence() + elif isinstance(self.event, MappingStartEvent): + if self.flow_level or self.canonical or self.event.flow_style \ + or self.check_empty_mapping(): + self.expect_flow_mapping() + else: + self.expect_block_mapping() + else: + raise EmitterError("expected NodeEvent, but got %s" % self.event) + + def expect_alias(self): + if self.event.anchor is None: + raise EmitterError("anchor is not specified for alias") + self.process_anchor(u'*') + self.state = self.states.pop() + + def expect_scalar(self): + self.increase_indent(flow=True) + self.process_scalar() + self.indent = self.indents.pop() + self.state = self.states.pop() + + # Flow sequence handlers. + + def expect_flow_sequence(self): + self.write_indicator(u'[', True, whitespace=True) + self.flow_level += 1 + self.increase_indent(flow=True) + self.state = self.expect_first_flow_sequence_item + + def expect_first_flow_sequence_item(self): + if isinstance(self.event, SequenceEndEvent): + self.indent = self.indents.pop() + self.flow_level -= 1 + self.write_indicator(u']', False) + self.state = self.states.pop() + else: + if self.canonical or self.column > self.best_width: + self.write_indent() + self.states.append(self.expect_flow_sequence_item) + self.expect_node(sequence=True) + + def expect_flow_sequence_item(self): + if isinstance(self.event, SequenceEndEvent): + self.indent = self.indents.pop() + self.flow_level -= 1 + if self.canonical: + self.write_indicator(u',', False) + self.write_indent() + self.write_indicator(u']', False) + self.state = self.states.pop() + else: + self.write_indicator(u',', False) + if self.canonical or self.column > self.best_width: + self.write_indent() + self.states.append(self.expect_flow_sequence_item) + self.expect_node(sequence=True) + + # Flow mapping handlers. + + def expect_flow_mapping(self): + self.write_indicator(u'{', True, whitespace=True) + self.flow_level += 1 + self.increase_indent(flow=True) + self.state = self.expect_first_flow_mapping_key + + def expect_first_flow_mapping_key(self): + if isinstance(self.event, MappingEndEvent): + self.indent = self.indents.pop() + self.flow_level -= 1 + self.write_indicator(u'}', False) + self.state = self.states.pop() + else: + if self.canonical or self.column > self.best_width: + self.write_indent() + if not self.canonical and self.check_simple_key(): + self.states.append(self.expect_flow_mapping_simple_value) + self.expect_node(mapping=True, simple_key=True) + else: + self.write_indicator(u'?', True) + self.states.append(self.expect_flow_mapping_value) + self.expect_node(mapping=True) + + def expect_flow_mapping_key(self): + if isinstance(self.event, MappingEndEvent): + self.indent = self.indents.pop() + self.flow_level -= 1 + if self.canonical: + self.write_indicator(u',', False) + self.write_indent() + self.write_indicator(u'}', False) + self.state = self.states.pop() + else: + self.write_indicator(u',', False) + if self.canonical or self.column > self.best_width: + self.write_indent() + if not self.canonical and self.check_simple_key(): + self.states.append(self.expect_flow_mapping_simple_value) + self.expect_node(mapping=True, simple_key=True) + else: + self.write_indicator(u'?', True) + self.states.append(self.expect_flow_mapping_value) + self.expect_node(mapping=True) + + def expect_flow_mapping_simple_value(self): + self.write_indicator(u':', False) + self.states.append(self.expect_flow_mapping_key) + self.expect_node(mapping=True) + + def expect_flow_mapping_value(self): + if self.canonical or self.column > self.best_width: + self.write_indent() + self.write_indicator(u':', True) + self.states.append(self.expect_flow_mapping_key) + self.expect_node(mapping=True) + + # Block sequence handlers. + + def expect_block_sequence(self): + indentless = (self.mapping_context and not self.indention) + self.increase_indent(flow=False, indentless=indentless) + self.state = self.expect_first_block_sequence_item + + def expect_first_block_sequence_item(self): + return self.expect_block_sequence_item(first=True) + + def expect_block_sequence_item(self, first=False): + if not first and isinstance(self.event, SequenceEndEvent): + self.indent = self.indents.pop() + self.state = self.states.pop() + else: + self.write_indent() + self.write_indicator(u'-', True, indention=True) + self.states.append(self.expect_block_sequence_item) + self.expect_node(sequence=True) + + # Block mapping handlers. + + def expect_block_mapping(self): + self.increase_indent(flow=False) + self.state = self.expect_first_block_mapping_key + + def expect_first_block_mapping_key(self): + return self.expect_block_mapping_key(first=True) + + def expect_block_mapping_key(self, first=False): + if not first and isinstance(self.event, MappingEndEvent): + self.indent = self.indents.pop() + self.state = self.states.pop() + else: + self.write_indent() + if self.check_simple_key(): + self.states.append(self.expect_block_mapping_simple_value) + self.expect_node(mapping=True, simple_key=True) + else: + self.write_indicator(u'?', True, indention=True) + self.states.append(self.expect_block_mapping_value) + self.expect_node(mapping=True) + + def expect_block_mapping_simple_value(self): + self.write_indicator(u':', False) + self.states.append(self.expect_block_mapping_key) + self.expect_node(mapping=True) + + def expect_block_mapping_value(self): + self.write_indent() + self.write_indicator(u':', True, indention=True) + self.states.append(self.expect_block_mapping_key) + self.expect_node(mapping=True) + + # Checkers. + + def check_empty_sequence(self): + return (isinstance(self.event, SequenceStartEvent) and self.events + and isinstance(self.events[0], SequenceEndEvent)) + + def check_empty_mapping(self): + return (isinstance(self.event, MappingStartEvent) and self.events + and isinstance(self.events[0], MappingEndEvent)) + + def check_empty_document(self): + if not isinstance(self.event, DocumentStartEvent) or not self.events: + return False + event = self.events[0] + return (isinstance(event, ScalarEvent) and event.anchor is None + and event.tag is None and event.implicit and event.value == u'') + + def check_simple_key(self): + length = 0 + if isinstance(self.event, NodeEvent) and self.event.anchor is not None: + if self.prepared_anchor is None: + self.prepared_anchor = self.prepare_anchor(self.event.anchor) + length += len(self.prepared_anchor) + if isinstance(self.event, (ScalarEvent, CollectionStartEvent)) \ + and self.event.tag is not None: + if self.prepared_tag is None: + self.prepared_tag = self.prepare_tag(self.event.tag) + length += len(self.prepared_tag) + if isinstance(self.event, ScalarEvent): + if self.analysis is None: + self.analysis = self.analyze_scalar(self.event.value) + length += len(self.analysis.scalar) + return (length < 128 and (isinstance(self.event, AliasEvent) + or (isinstance(self.event, ScalarEvent) + and not self.analysis.empty and not self.analysis.multiline) + or self.check_empty_sequence() or self.check_empty_mapping())) + + # Anchor, Tag, and Scalar processors. + + def process_anchor(self, indicator): + if self.event.anchor is None: + self.prepared_anchor = None + return + if self.prepared_anchor is None: + self.prepared_anchor = self.prepare_anchor(self.event.anchor) + if self.prepared_anchor: + self.write_indicator(indicator+self.prepared_anchor, True) + self.prepared_anchor = None + + def process_tag(self): + tag = self.event.tag + if isinstance(self.event, ScalarEvent): + if self.style is None: + self.style = self.choose_scalar_style() + if ((not self.canonical or tag is None) and + ((self.style == '' and self.event.implicit[0]) + or (self.style != '' and self.event.implicit[1]))): + self.prepared_tag = None + return + if self.event.implicit[0] and tag is None: + tag = u'!' + self.prepared_tag = None + else: + if (not self.canonical or tag is None) and self.event.implicit: + self.prepared_tag = None + return + if tag is None: + raise EmitterError("tag is not specified") + if self.prepared_tag is None: + self.prepared_tag = self.prepare_tag(tag) + if self.prepared_tag: + self.write_indicator(self.prepared_tag, True) + self.prepared_tag = None + + def choose_scalar_style(self): + if self.analysis is None: + self.analysis = self.analyze_scalar(self.event.value) + if self.event.style == '"' or self.canonical: + return '"' + if not self.event.style and self.event.implicit[0]: + if (not (self.simple_key_context and + (self.analysis.empty or self.analysis.multiline)) + and (self.flow_level and self.analysis.allow_flow_plain + or (not self.flow_level and self.analysis.allow_block_plain))): + return '' + if self.event.style and self.event.style in '|>': + if (not self.flow_level and not self.simple_key_context + and self.analysis.allow_block): + return self.event.style + if not self.event.style or self.event.style == '\'': + if (self.analysis.allow_single_quoted and + not (self.simple_key_context and self.analysis.multiline)): + return '\'' + return '"' + + def process_scalar(self): + if self.analysis is None: + self.analysis = self.analyze_scalar(self.event.value) + if self.style is None: + self.style = self.choose_scalar_style() + split = (not self.simple_key_context) + #if self.analysis.multiline and split \ + # and (not self.style or self.style in '\'\"'): + # self.write_indent() + if self.style == '"': + self.write_double_quoted(self.analysis.scalar, split) + elif self.style == '\'': + self.write_single_quoted(self.analysis.scalar, split) + elif self.style == '>': + self.write_folded(self.analysis.scalar) + elif self.style == '|': + self.write_literal(self.analysis.scalar) + else: + self.write_plain(self.analysis.scalar, split) + self.analysis = None + self.style = None + + # Analyzers. + + def prepare_version(self, version): + major, minor = version + if major != 1: + raise EmitterError("unsupported YAML version: %d.%d" % (major, minor)) + return u'%d.%d' % (major, minor) + + def prepare_tag_handle(self, handle): + if not handle: + raise EmitterError("tag handle must not be empty") + if handle[0] != u'!' or handle[-1] != u'!': + raise EmitterError("tag handle must start and end with '!': %r" + % (handle.encode('utf-8'))) + for ch in handle[1:-1]: + if not (u'0' <= ch <= u'9' or u'A' <= ch <= u'Z' or u'a' <= ch <= u'z' \ + or ch in u'-_'): + raise EmitterError("invalid character %r in the tag handle: %r" + % (ch.encode('utf-8'), handle.encode('utf-8'))) + return handle + + def prepare_tag_prefix(self, prefix): + if not prefix: + raise EmitterError("tag prefix must not be empty") + chunks = [] + start = end = 0 + if prefix[0] == u'!': + end = 1 + while end < len(prefix): + ch = prefix[end] + if u'0' <= ch <= u'9' or u'A' <= ch <= u'Z' or u'a' <= ch <= u'z' \ + or ch in u'-;/?!:@&=+$,_.~*\'()[]': + end += 1 + else: + if start < end: + chunks.append(prefix[start:end]) + start = end = end+1 + data = ch.encode('utf-8') + for ch in data: + chunks.append(u'%%%02X' % ord(ch)) + if start < end: + chunks.append(prefix[start:end]) + return u''.join(chunks) + + def prepare_tag(self, tag): + if not tag: + raise EmitterError("tag must not be empty") + if tag == u'!': + return tag + handle = None + suffix = tag + prefixes = self.tag_prefixes.keys() + prefixes.sort() + for prefix in prefixes: + if tag.startswith(prefix) \ + and (prefix == u'!' or len(prefix) < len(tag)): + handle = self.tag_prefixes[prefix] + suffix = tag[len(prefix):] + chunks = [] + start = end = 0 + while end < len(suffix): + ch = suffix[end] + if u'0' <= ch <= u'9' or u'A' <= ch <= u'Z' or u'a' <= ch <= u'z' \ + or ch in u'-;/?:@&=+$,_.~*\'()[]' \ + or (ch == u'!' and handle != u'!'): + end += 1 + else: + if start < end: + chunks.append(suffix[start:end]) + start = end = end+1 + data = ch.encode('utf-8') + for ch in data: + chunks.append(u'%%%02X' % ord(ch)) + if start < end: + chunks.append(suffix[start:end]) + suffix_text = u''.join(chunks) + if handle: + return u'%s%s' % (handle, suffix_text) + else: + return u'!<%s>' % suffix_text + + def prepare_anchor(self, anchor): + if not anchor: + raise EmitterError("anchor must not be empty") + for ch in anchor: + if not (u'0' <= ch <= u'9' or u'A' <= ch <= u'Z' or u'a' <= ch <= u'z' \ + or ch in u'-_'): + raise EmitterError("invalid character %r in the anchor: %r" + % (ch.encode('utf-8'), anchor.encode('utf-8'))) + return anchor + + def analyze_scalar(self, scalar): + + # Empty scalar is a special case. + if not scalar: + return ScalarAnalysis(scalar=scalar, empty=True, multiline=False, + allow_flow_plain=False, allow_block_plain=True, + allow_single_quoted=True, allow_double_quoted=True, + allow_block=False) + + # Indicators and special characters. + block_indicators = False + flow_indicators = False + line_breaks = False + special_characters = False + + # Important whitespace combinations. + leading_space = False + leading_break = False + trailing_space = False + trailing_break = False + break_space = False + space_break = False + + # Check document indicators. + if scalar.startswith(u'---') or scalar.startswith(u'...'): + block_indicators = True + flow_indicators = True + + # First character or preceded by a whitespace. + preceeded_by_whitespace = True + + # Last character or followed by a whitespace. + followed_by_whitespace = (len(scalar) == 1 or + scalar[1] in u'\0 \t\r\n\x85\u2028\u2029') + + # The previous character is a space. + previous_space = False + + # The previous character is a break. + previous_break = False + + index = 0 + while index < len(scalar): + ch = scalar[index] + + # Check for indicators. + if index == 0: + # Leading indicators are special characters. + if ch in u'#,[]{}&*!|>\'\"%@`': + flow_indicators = True + block_indicators = True + if ch in u'?:': + flow_indicators = True + if followed_by_whitespace: + block_indicators = True + if ch == u'-' and followed_by_whitespace: + flow_indicators = True + block_indicators = True + else: + # Some indicators cannot appear within a scalar as well. + if ch in u',?[]{}': + flow_indicators = True + if ch == u':': + flow_indicators = True + if followed_by_whitespace: + block_indicators = True + if ch == u'#' and preceeded_by_whitespace: + flow_indicators = True + block_indicators = True + + # Check for line breaks, special, and unicode characters. + if ch in u'\n\x85\u2028\u2029': + line_breaks = True + if not (ch == u'\n' or u'\x20' <= ch <= u'\x7E'): + if (ch == u'\x85' or u'\xA0' <= ch <= u'\uD7FF' + or u'\uE000' <= ch <= u'\uFFFD') and ch != u'\uFEFF': + unicode_characters = True + if not self.allow_unicode: + special_characters = True + else: + special_characters = True + + # Detect important whitespace combinations. + if ch == u' ': + if index == 0: + leading_space = True + if index == len(scalar)-1: + trailing_space = True + if previous_break: + break_space = True + previous_space = True + previous_break = False + elif ch in u'\n\x85\u2028\u2029': + if index == 0: + leading_break = True + if index == len(scalar)-1: + trailing_break = True + if previous_space: + space_break = True + previous_space = False + previous_break = True + else: + previous_space = False + previous_break = False + + # Prepare for the next character. + index += 1 + preceeded_by_whitespace = (ch in u'\0 \t\r\n\x85\u2028\u2029') + followed_by_whitespace = (index+1 >= len(scalar) or + scalar[index+1] in u'\0 \t\r\n\x85\u2028\u2029') + + # Let's decide what styles are allowed. + allow_flow_plain = True + allow_block_plain = True + allow_single_quoted = True + allow_double_quoted = True + allow_block = True + + # Leading and trailing whitespaces are bad for plain scalars. + if (leading_space or leading_break + or trailing_space or trailing_break): + allow_flow_plain = allow_block_plain = False + + # We do not permit trailing spaces for block scalars. + if trailing_space: + allow_block = False + + # Spaces at the beginning of a new line are only acceptable for block + # scalars. + if break_space: + allow_flow_plain = allow_block_plain = allow_single_quoted = False + + # Spaces followed by breaks, as well as special character are only + # allowed for double quoted scalars. + if space_break or special_characters: + allow_flow_plain = allow_block_plain = \ + allow_single_quoted = allow_block = False + + # Although the plain scalar writer supports breaks, we never emit + # multiline plain scalars. + if line_breaks: + allow_flow_plain = allow_block_plain = False + + # Flow indicators are forbidden for flow plain scalars. + if flow_indicators: + allow_flow_plain = False + + # Block indicators are forbidden for block plain scalars. + if block_indicators: + allow_block_plain = False + + return ScalarAnalysis(scalar=scalar, + empty=False, multiline=line_breaks, + allow_flow_plain=allow_flow_plain, + allow_block_plain=allow_block_plain, + allow_single_quoted=allow_single_quoted, + allow_double_quoted=allow_double_quoted, + allow_block=allow_block) + + # Writers. + + def flush_stream(self): + if hasattr(self.stream, 'flush'): + self.stream.flush() + + def write_stream_start(self): + # Write BOM if needed. + if self.encoding and self.encoding.startswith('utf-16'): + self.stream.write(u'\uFEFF'.encode(self.encoding)) + + def write_stream_end(self): + self.flush_stream() + + def write_indicator(self, indicator, need_whitespace, + whitespace=False, indention=False): + if self.whitespace or not need_whitespace: + data = indicator + else: + data = u' '+indicator + self.whitespace = whitespace + self.indention = self.indention and indention + self.column += len(data) + self.open_ended = False + if self.encoding: + data = data.encode(self.encoding) + self.stream.write(data) + + def write_indent(self): + indent = self.indent or 0 + if not self.indention or self.column > indent \ + or (self.column == indent and not self.whitespace): + self.write_line_break() + if self.column < indent: + self.whitespace = True + data = u' '*(indent-self.column) + self.column = indent + if self.encoding: + data = data.encode(self.encoding) + self.stream.write(data) + + def write_line_break(self, data=None): + if data is None: + data = self.best_line_break + self.whitespace = True + self.indention = True + self.line += 1 + self.column = 0 + if self.encoding: + data = data.encode(self.encoding) + self.stream.write(data) + + def write_version_directive(self, version_text): + data = u'%%YAML %s' % version_text + if self.encoding: + data = data.encode(self.encoding) + self.stream.write(data) + self.write_line_break() + + def write_tag_directive(self, handle_text, prefix_text): + data = u'%%TAG %s %s' % (handle_text, prefix_text) + if self.encoding: + data = data.encode(self.encoding) + self.stream.write(data) + self.write_line_break() + + # Scalar streams. + + def write_single_quoted(self, text, split=True): + self.write_indicator(u'\'', True) + spaces = False + breaks = False + start = end = 0 + while end <= len(text): + ch = None + if end < len(text): + ch = text[end] + if spaces: + if ch is None or ch != u' ': + if start+1 == end and self.column > self.best_width and split \ + and start != 0 and end != len(text): + self.write_indent() + else: + data = text[start:end] + self.column += len(data) + if self.encoding: + data = data.encode(self.encoding) + self.stream.write(data) + start = end + elif breaks: + if ch is None or ch not in u'\n\x85\u2028\u2029': + if text[start] == u'\n': + self.write_line_break() + for br in text[start:end]: + if br == u'\n': + self.write_line_break() + else: + self.write_line_break(br) + self.write_indent() + start = end + else: + if ch is None or ch in u' \n\x85\u2028\u2029' or ch == u'\'': + if start < end: + data = text[start:end] + self.column += len(data) + if self.encoding: + data = data.encode(self.encoding) + self.stream.write(data) + start = end + if ch == u'\'': + data = u'\'\'' + self.column += 2 + if self.encoding: + data = data.encode(self.encoding) + self.stream.write(data) + start = end + 1 + if ch is not None: + spaces = (ch == u' ') + breaks = (ch in u'\n\x85\u2028\u2029') + end += 1 + self.write_indicator(u'\'', False) + + ESCAPE_REPLACEMENTS = { + u'\0': u'0', + u'\x07': u'a', + u'\x08': u'b', + u'\x09': u't', + u'\x0A': u'n', + u'\x0B': u'v', + u'\x0C': u'f', + u'\x0D': u'r', + u'\x1B': u'e', + u'\"': u'\"', + u'\\': u'\\', + u'\x85': u'N', + u'\xA0': u'_', + u'\u2028': u'L', + u'\u2029': u'P', + } + + def write_double_quoted(self, text, split=True): + self.write_indicator(u'"', True) + start = end = 0 + while end <= len(text): + ch = None + if end < len(text): + ch = text[end] + if ch is None or ch in u'"\\\x85\u2028\u2029\uFEFF' \ + or not (u'\x20' <= ch <= u'\x7E' + or (self.allow_unicode + and (u'\xA0' <= ch <= u'\uD7FF' + or u'\uE000' <= ch <= u'\uFFFD'))): + if start < end: + data = text[start:end] + self.column += len(data) + if self.encoding: + data = data.encode(self.encoding) + self.stream.write(data) + start = end + if ch is not None: + if ch in self.ESCAPE_REPLACEMENTS: + data = u'\\'+self.ESCAPE_REPLACEMENTS[ch] + elif ch <= u'\xFF': + data = u'\\x%02X' % ord(ch) + elif ch <= u'\uFFFF': + data = u'\\u%04X' % ord(ch) + else: + data = u'\\U%08X' % ord(ch) + self.column += len(data) + if self.encoding: + data = data.encode(self.encoding) + self.stream.write(data) + start = end+1 + if 0 < end < len(text)-1 and (ch == u' ' or start >= end) \ + and self.column+(end-start) > self.best_width and split: + data = text[start:end]+u'\\' + if start < end: + start = end + self.column += len(data) + if self.encoding: + data = data.encode(self.encoding) + self.stream.write(data) + self.write_indent() + self.whitespace = False + self.indention = False + if text[start] == u' ': + data = u'\\' + self.column += len(data) + if self.encoding: + data = data.encode(self.encoding) + self.stream.write(data) + end += 1 + self.write_indicator(u'"', False) + + def determine_block_hints(self, text): + hints = u'' + if text: + if text[0] in u' \n\x85\u2028\u2029': + hints += unicode(self.best_indent) + if text[-1] not in u'\n\x85\u2028\u2029': + hints += u'-' + elif len(text) == 1 or text[-2] in u'\n\x85\u2028\u2029': + hints += u'+' + return hints + + def write_folded(self, text): + hints = self.determine_block_hints(text) + self.write_indicator(u'>'+hints, True) + if hints[-1:] == u'+': + self.open_ended = True + self.write_line_break() + leading_space = True + spaces = False + breaks = True + start = end = 0 + while end <= len(text): + ch = None + if end < len(text): + ch = text[end] + if breaks: + if ch is None or ch not in u'\n\x85\u2028\u2029': + if not leading_space and ch is not None and ch != u' ' \ + and text[start] == u'\n': + self.write_line_break() + leading_space = (ch == u' ') + for br in text[start:end]: + if br == u'\n': + self.write_line_break() + else: + self.write_line_break(br) + if ch is not None: + self.write_indent() + start = end + elif spaces: + if ch != u' ': + if start+1 == end and self.column > self.best_width: + self.write_indent() + else: + data = text[start:end] + self.column += len(data) + if self.encoding: + data = data.encode(self.encoding) + self.stream.write(data) + start = end + else: + if ch is None or ch in u' \n\x85\u2028\u2029': + data = text[start:end] + self.column += len(data) + if self.encoding: + data = data.encode(self.encoding) + self.stream.write(data) + if ch is None: + self.write_line_break() + start = end + if ch is not None: + breaks = (ch in u'\n\x85\u2028\u2029') + spaces = (ch == u' ') + end += 1 + + def write_literal(self, text): + hints = self.determine_block_hints(text) + self.write_indicator(u'|'+hints, True) + if hints[-1:] == u'+': + self.open_ended = True + self.write_line_break() + breaks = True + start = end = 0 + while end <= len(text): + ch = None + if end < len(text): + ch = text[end] + if breaks: + if ch is None or ch not in u'\n\x85\u2028\u2029': + for br in text[start:end]: + if br == u'\n': + self.write_line_break() + else: + self.write_line_break(br) + if ch is not None: + self.write_indent() + start = end + else: + if ch is None or ch in u'\n\x85\u2028\u2029': + data = text[start:end] + if self.encoding: + data = data.encode(self.encoding) + self.stream.write(data) + if ch is None: + self.write_line_break() + start = end + if ch is not None: + breaks = (ch in u'\n\x85\u2028\u2029') + end += 1 + + def write_plain(self, text, split=True): + if self.root_context: + self.open_ended = True + if not text: + return + if not self.whitespace: + data = u' ' + self.column += len(data) + if self.encoding: + data = data.encode(self.encoding) + self.stream.write(data) + self.whitespace = False + self.indention = False + spaces = False + breaks = False + start = end = 0 + while end <= len(text): + ch = None + if end < len(text): + ch = text[end] + if spaces: + if ch != u' ': + if start+1 == end and self.column > self.best_width and split: + self.write_indent() + self.whitespace = False + self.indention = False + else: + data = text[start:end] + self.column += len(data) + if self.encoding: + data = data.encode(self.encoding) + self.stream.write(data) + start = end + elif breaks: + if ch not in u'\n\x85\u2028\u2029': + if text[start] == u'\n': + self.write_line_break() + for br in text[start:end]: + if br == u'\n': + self.write_line_break() + else: + self.write_line_break(br) + self.write_indent() + self.whitespace = False + self.indention = False + start = end + else: + if ch is None or ch in u' \n\x85\u2028\u2029': + data = text[start:end] + self.column += len(data) + if self.encoding: + data = data.encode(self.encoding) + self.stream.write(data) + start = end + if ch is not None: + spaces = (ch == u' ') + breaks = (ch in u'\n\x85\u2028\u2029') + end += 1 + diff --git a/python.d/python_modules/pyyaml2/error.py b/python.d/python_modules/pyyaml2/error.py new file mode 100644 index 000000000..577686db5 --- /dev/null +++ b/python.d/python_modules/pyyaml2/error.py @@ -0,0 +1,75 @@ + +__all__ = ['Mark', 'YAMLError', 'MarkedYAMLError'] + +class Mark(object): + + def __init__(self, name, index, line, column, buffer, pointer): + self.name = name + self.index = index + self.line = line + self.column = column + self.buffer = buffer + self.pointer = pointer + + def get_snippet(self, indent=4, max_length=75): + if self.buffer is None: + return None + head = '' + start = self.pointer + while start > 0 and self.buffer[start-1] not in u'\0\r\n\x85\u2028\u2029': + start -= 1 + if self.pointer-start > max_length/2-1: + head = ' ... ' + start += 5 + break + tail = '' + end = self.pointer + while end < len(self.buffer) and self.buffer[end] not in u'\0\r\n\x85\u2028\u2029': + end += 1 + if end-self.pointer > max_length/2-1: + tail = ' ... ' + end -= 5 + break + snippet = self.buffer[start:end].encode('utf-8') + return ' '*indent + head + snippet + tail + '\n' \ + + ' '*(indent+self.pointer-start+len(head)) + '^' + + def __str__(self): + snippet = self.get_snippet() + where = " in \"%s\", line %d, column %d" \ + % (self.name, self.line+1, self.column+1) + if snippet is not None: + where += ":\n"+snippet + return where + +class YAMLError(Exception): + pass + +class MarkedYAMLError(YAMLError): + + def __init__(self, context=None, context_mark=None, + problem=None, problem_mark=None, note=None): + self.context = context + self.context_mark = context_mark + self.problem = problem + self.problem_mark = problem_mark + self.note = note + + def __str__(self): + lines = [] + if self.context is not None: + lines.append(self.context) + if self.context_mark is not None \ + and (self.problem is None or self.problem_mark is None + or self.context_mark.name != self.problem_mark.name + or self.context_mark.line != self.problem_mark.line + or self.context_mark.column != self.problem_mark.column): + lines.append(str(self.context_mark)) + if self.problem is not None: + lines.append(self.problem) + if self.problem_mark is not None: + lines.append(str(self.problem_mark)) + if self.note is not None: + lines.append(self.note) + return '\n'.join(lines) + diff --git a/python.d/python_modules/pyyaml2/events.py b/python.d/python_modules/pyyaml2/events.py new file mode 100644 index 000000000..f79ad389c --- /dev/null +++ b/python.d/python_modules/pyyaml2/events.py @@ -0,0 +1,86 @@ + +# Abstract classes. + +class Event(object): + def __init__(self, start_mark=None, end_mark=None): + self.start_mark = start_mark + self.end_mark = end_mark + def __repr__(self): + attributes = [key for key in ['anchor', 'tag', 'implicit', 'value'] + if hasattr(self, key)] + arguments = ', '.join(['%s=%r' % (key, getattr(self, key)) + for key in attributes]) + return '%s(%s)' % (self.__class__.__name__, arguments) + +class NodeEvent(Event): + def __init__(self, anchor, start_mark=None, end_mark=None): + self.anchor = anchor + self.start_mark = start_mark + self.end_mark = end_mark + +class CollectionStartEvent(NodeEvent): + def __init__(self, anchor, tag, implicit, start_mark=None, end_mark=None, + flow_style=None): + self.anchor = anchor + self.tag = tag + self.implicit = implicit + self.start_mark = start_mark + self.end_mark = end_mark + self.flow_style = flow_style + +class CollectionEndEvent(Event): + pass + +# Implementations. + +class StreamStartEvent(Event): + def __init__(self, start_mark=None, end_mark=None, encoding=None): + self.start_mark = start_mark + self.end_mark = end_mark + self.encoding = encoding + +class StreamEndEvent(Event): + pass + +class DocumentStartEvent(Event): + def __init__(self, start_mark=None, end_mark=None, + explicit=None, version=None, tags=None): + self.start_mark = start_mark + self.end_mark = end_mark + self.explicit = explicit + self.version = version + self.tags = tags + +class DocumentEndEvent(Event): + def __init__(self, start_mark=None, end_mark=None, + explicit=None): + self.start_mark = start_mark + self.end_mark = end_mark + self.explicit = explicit + +class AliasEvent(NodeEvent): + pass + +class ScalarEvent(NodeEvent): + def __init__(self, anchor, tag, implicit, value, + start_mark=None, end_mark=None, style=None): + self.anchor = anchor + self.tag = tag + self.implicit = implicit + self.value = value + self.start_mark = start_mark + self.end_mark = end_mark + self.style = style + +class SequenceStartEvent(CollectionStartEvent): + pass + +class SequenceEndEvent(CollectionEndEvent): + pass + +class MappingStartEvent(CollectionStartEvent): + pass + +class MappingEndEvent(CollectionEndEvent): + pass + diff --git a/python.d/python_modules/pyyaml2/loader.py b/python.d/python_modules/pyyaml2/loader.py new file mode 100644 index 000000000..293ff467b --- /dev/null +++ b/python.d/python_modules/pyyaml2/loader.py @@ -0,0 +1,40 @@ + +__all__ = ['BaseLoader', 'SafeLoader', 'Loader'] + +from reader import * +from scanner import * +from parser import * +from composer import * +from constructor import * +from resolver import * + +class BaseLoader(Reader, Scanner, Parser, Composer, BaseConstructor, BaseResolver): + + def __init__(self, stream): + Reader.__init__(self, stream) + Scanner.__init__(self) + Parser.__init__(self) + Composer.__init__(self) + BaseConstructor.__init__(self) + BaseResolver.__init__(self) + +class SafeLoader(Reader, Scanner, Parser, Composer, SafeConstructor, Resolver): + + def __init__(self, stream): + Reader.__init__(self, stream) + Scanner.__init__(self) + Parser.__init__(self) + Composer.__init__(self) + SafeConstructor.__init__(self) + Resolver.__init__(self) + +class Loader(Reader, Scanner, Parser, Composer, Constructor, Resolver): + + def __init__(self, stream): + Reader.__init__(self, stream) + Scanner.__init__(self) + Parser.__init__(self) + Composer.__init__(self) + Constructor.__init__(self) + Resolver.__init__(self) + diff --git a/python.d/python_modules/pyyaml2/nodes.py b/python.d/python_modules/pyyaml2/nodes.py new file mode 100644 index 000000000..c4f070c41 --- /dev/null +++ b/python.d/python_modules/pyyaml2/nodes.py @@ -0,0 +1,49 @@ + +class Node(object): + def __init__(self, tag, value, start_mark, end_mark): + self.tag = tag + self.value = value + self.start_mark = start_mark + self.end_mark = end_mark + def __repr__(self): + value = self.value + #if isinstance(value, list): + # if len(value) == 0: + # value = '<empty>' + # elif len(value) == 1: + # value = '<1 item>' + # else: + # value = '<%d items>' % len(value) + #else: + # if len(value) > 75: + # value = repr(value[:70]+u' ... ') + # else: + # value = repr(value) + value = repr(value) + return '%s(tag=%r, value=%s)' % (self.__class__.__name__, self.tag, value) + +class ScalarNode(Node): + id = 'scalar' + def __init__(self, tag, value, + start_mark=None, end_mark=None, style=None): + self.tag = tag + self.value = value + self.start_mark = start_mark + self.end_mark = end_mark + self.style = style + +class CollectionNode(Node): + def __init__(self, tag, value, + start_mark=None, end_mark=None, flow_style=None): + self.tag = tag + self.value = value + self.start_mark = start_mark + self.end_mark = end_mark + self.flow_style = flow_style + +class SequenceNode(CollectionNode): + id = 'sequence' + +class MappingNode(CollectionNode): + id = 'mapping' + diff --git a/python.d/python_modules/pyyaml2/parser.py b/python.d/python_modules/pyyaml2/parser.py new file mode 100644 index 000000000..f9e3057f3 --- /dev/null +++ b/python.d/python_modules/pyyaml2/parser.py @@ -0,0 +1,589 @@ + +# The following YAML grammar is LL(1) and is parsed by a recursive descent +# parser. +# +# stream ::= STREAM-START implicit_document? explicit_document* STREAM-END +# implicit_document ::= block_node DOCUMENT-END* +# explicit_document ::= DIRECTIVE* DOCUMENT-START block_node? DOCUMENT-END* +# block_node_or_indentless_sequence ::= +# ALIAS +# | properties (block_content | indentless_block_sequence)? +# | block_content +# | indentless_block_sequence +# block_node ::= ALIAS +# | properties block_content? +# | block_content +# flow_node ::= ALIAS +# | properties flow_content? +# | flow_content +# properties ::= TAG ANCHOR? | ANCHOR TAG? +# block_content ::= block_collection | flow_collection | SCALAR +# flow_content ::= flow_collection | SCALAR +# block_collection ::= block_sequence | block_mapping +# flow_collection ::= flow_sequence | flow_mapping +# block_sequence ::= BLOCK-SEQUENCE-START (BLOCK-ENTRY block_node?)* BLOCK-END +# indentless_sequence ::= (BLOCK-ENTRY block_node?)+ +# block_mapping ::= BLOCK-MAPPING_START +# ((KEY block_node_or_indentless_sequence?)? +# (VALUE block_node_or_indentless_sequence?)?)* +# BLOCK-END +# flow_sequence ::= FLOW-SEQUENCE-START +# (flow_sequence_entry FLOW-ENTRY)* +# flow_sequence_entry? +# FLOW-SEQUENCE-END +# flow_sequence_entry ::= flow_node | KEY flow_node? (VALUE flow_node?)? +# flow_mapping ::= FLOW-MAPPING-START +# (flow_mapping_entry FLOW-ENTRY)* +# flow_mapping_entry? +# FLOW-MAPPING-END +# flow_mapping_entry ::= flow_node | KEY flow_node? (VALUE flow_node?)? +# +# FIRST sets: +# +# stream: { STREAM-START } +# explicit_document: { DIRECTIVE DOCUMENT-START } +# implicit_document: FIRST(block_node) +# block_node: { ALIAS TAG ANCHOR SCALAR BLOCK-SEQUENCE-START BLOCK-MAPPING-START FLOW-SEQUENCE-START FLOW-MAPPING-START } +# flow_node: { ALIAS ANCHOR TAG SCALAR FLOW-SEQUENCE-START FLOW-MAPPING-START } +# block_content: { BLOCK-SEQUENCE-START BLOCK-MAPPING-START FLOW-SEQUENCE-START FLOW-MAPPING-START SCALAR } +# flow_content: { FLOW-SEQUENCE-START FLOW-MAPPING-START SCALAR } +# block_collection: { BLOCK-SEQUENCE-START BLOCK-MAPPING-START } +# flow_collection: { FLOW-SEQUENCE-START FLOW-MAPPING-START } +# block_sequence: { BLOCK-SEQUENCE-START } +# block_mapping: { BLOCK-MAPPING-START } +# block_node_or_indentless_sequence: { ALIAS ANCHOR TAG SCALAR BLOCK-SEQUENCE-START BLOCK-MAPPING-START FLOW-SEQUENCE-START FLOW-MAPPING-START BLOCK-ENTRY } +# indentless_sequence: { ENTRY } +# flow_collection: { FLOW-SEQUENCE-START FLOW-MAPPING-START } +# flow_sequence: { FLOW-SEQUENCE-START } +# flow_mapping: { FLOW-MAPPING-START } +# flow_sequence_entry: { ALIAS ANCHOR TAG SCALAR FLOW-SEQUENCE-START FLOW-MAPPING-START KEY } +# flow_mapping_entry: { ALIAS ANCHOR TAG SCALAR FLOW-SEQUENCE-START FLOW-MAPPING-START KEY } + +__all__ = ['Parser', 'ParserError'] + +from error import MarkedYAMLError +from tokens import * +from events import * +from scanner import * + +class ParserError(MarkedYAMLError): + pass + +class Parser(object): + # Since writing a recursive-descendant parser is a straightforward task, we + # do not give many comments here. + + DEFAULT_TAGS = { + u'!': u'!', + u'!!': u'tag:yaml.org,2002:', + } + + def __init__(self): + self.current_event = None + self.yaml_version = None + self.tag_handles = {} + self.states = [] + self.marks = [] + self.state = self.parse_stream_start + + def dispose(self): + # Reset the state attributes (to clear self-references) + self.states = [] + self.state = None + + def check_event(self, *choices): + # Check the type of the next event. + if self.current_event is None: + if self.state: + self.current_event = self.state() + if self.current_event is not None: + if not choices: + return True + for choice in choices: + if isinstance(self.current_event, choice): + return True + return False + + def peek_event(self): + # Get the next event. + if self.current_event is None: + if self.state: + self.current_event = self.state() + return self.current_event + + def get_event(self): + # Get the next event and proceed further. + if self.current_event is None: + if self.state: + self.current_event = self.state() + value = self.current_event + self.current_event = None + return value + + # stream ::= STREAM-START implicit_document? explicit_document* STREAM-END + # implicit_document ::= block_node DOCUMENT-END* + # explicit_document ::= DIRECTIVE* DOCUMENT-START block_node? DOCUMENT-END* + + def parse_stream_start(self): + + # Parse the stream start. + token = self.get_token() + event = StreamStartEvent(token.start_mark, token.end_mark, + encoding=token.encoding) + + # Prepare the next state. + self.state = self.parse_implicit_document_start + + return event + + def parse_implicit_document_start(self): + + # Parse an implicit document. + if not self.check_token(DirectiveToken, DocumentStartToken, + StreamEndToken): + self.tag_handles = self.DEFAULT_TAGS + token = self.peek_token() + start_mark = end_mark = token.start_mark + event = DocumentStartEvent(start_mark, end_mark, + explicit=False) + + # Prepare the next state. + self.states.append(self.parse_document_end) + self.state = self.parse_block_node + + return event + + else: + return self.parse_document_start() + + def parse_document_start(self): + + # Parse any extra document end indicators. + while self.check_token(DocumentEndToken): + self.get_token() + + # Parse an explicit document. + if not self.check_token(StreamEndToken): + token = self.peek_token() + start_mark = token.start_mark + version, tags = self.process_directives() + if not self.check_token(DocumentStartToken): + raise ParserError(None, None, + "expected '<document start>', but found %r" + % self.peek_token().id, + self.peek_token().start_mark) + token = self.get_token() + end_mark = token.end_mark + event = DocumentStartEvent(start_mark, end_mark, + explicit=True, version=version, tags=tags) + self.states.append(self.parse_document_end) + self.state = self.parse_document_content + else: + # Parse the end of the stream. + token = self.get_token() + event = StreamEndEvent(token.start_mark, token.end_mark) + assert not self.states + assert not self.marks + self.state = None + return event + + def parse_document_end(self): + + # Parse the document end. + token = self.peek_token() + start_mark = end_mark = token.start_mark + explicit = False + if self.check_token(DocumentEndToken): + token = self.get_token() + end_mark = token.end_mark + explicit = True + event = DocumentEndEvent(start_mark, end_mark, + explicit=explicit) + + # Prepare the next state. + self.state = self.parse_document_start + + return event + + def parse_document_content(self): + if self.check_token(DirectiveToken, + DocumentStartToken, DocumentEndToken, StreamEndToken): + event = self.process_empty_scalar(self.peek_token().start_mark) + self.state = self.states.pop() + return event + else: + return self.parse_block_node() + + def process_directives(self): + self.yaml_version = None + self.tag_handles = {} + while self.check_token(DirectiveToken): + token = self.get_token() + if token.name == u'YAML': + if self.yaml_version is not None: + raise ParserError(None, None, + "found duplicate YAML directive", token.start_mark) + major, minor = token.value + if major != 1: + raise ParserError(None, None, + "found incompatible YAML document (version 1.* is required)", + token.start_mark) + self.yaml_version = token.value + elif token.name == u'TAG': + handle, prefix = token.value + if handle in self.tag_handles: + raise ParserError(None, None, + "duplicate tag handle %r" % handle.encode('utf-8'), + token.start_mark) + self.tag_handles[handle] = prefix + if self.tag_handles: + value = self.yaml_version, self.tag_handles.copy() + else: + value = self.yaml_version, None + for key in self.DEFAULT_TAGS: + if key not in self.tag_handles: + self.tag_handles[key] = self.DEFAULT_TAGS[key] + return value + + # block_node_or_indentless_sequence ::= ALIAS + # | properties (block_content | indentless_block_sequence)? + # | block_content + # | indentless_block_sequence + # block_node ::= ALIAS + # | properties block_content? + # | block_content + # flow_node ::= ALIAS + # | properties flow_content? + # | flow_content + # properties ::= TAG ANCHOR? | ANCHOR TAG? + # block_content ::= block_collection | flow_collection | SCALAR + # flow_content ::= flow_collection | SCALAR + # block_collection ::= block_sequence | block_mapping + # flow_collection ::= flow_sequence | flow_mapping + + def parse_block_node(self): + return self.parse_node(block=True) + + def parse_flow_node(self): + return self.parse_node() + + def parse_block_node_or_indentless_sequence(self): + return self.parse_node(block=True, indentless_sequence=True) + + def parse_node(self, block=False, indentless_sequence=False): + if self.check_token(AliasToken): + token = self.get_token() + event = AliasEvent(token.value, token.start_mark, token.end_mark) + self.state = self.states.pop() + else: + anchor = None + tag = None + start_mark = end_mark = tag_mark = None + if self.check_token(AnchorToken): + token = self.get_token() + start_mark = token.start_mark + end_mark = token.end_mark + anchor = token.value + if self.check_token(TagToken): + token = self.get_token() + tag_mark = token.start_mark + end_mark = token.end_mark + tag = token.value + elif self.check_token(TagToken): + token = self.get_token() + start_mark = tag_mark = token.start_mark + end_mark = token.end_mark + tag = token.value + if self.check_token(AnchorToken): + token = self.get_token() + end_mark = token.end_mark + anchor = token.value + if tag is not None: + handle, suffix = tag + if handle is not None: + if handle not in self.tag_handles: + raise ParserError("while parsing a node", start_mark, + "found undefined tag handle %r" % handle.encode('utf-8'), + tag_mark) + tag = self.tag_handles[handle]+suffix + else: + tag = suffix + #if tag == u'!': + # raise ParserError("while parsing a node", start_mark, + # "found non-specific tag '!'", tag_mark, + # "Please check 'http://pyyaml.org/wiki/YAMLNonSpecificTag' and share your opinion.") + if start_mark is None: + start_mark = end_mark = self.peek_token().start_mark + event = None + implicit = (tag is None or tag == u'!') + if indentless_sequence and self.check_token(BlockEntryToken): + end_mark = self.peek_token().end_mark + event = SequenceStartEvent(anchor, tag, implicit, + start_mark, end_mark) + self.state = self.parse_indentless_sequence_entry + else: + if self.check_token(ScalarToken): + token = self.get_token() + end_mark = token.end_mark + if (token.plain and tag is None) or tag == u'!': + implicit = (True, False) + elif tag is None: + implicit = (False, True) + else: + implicit = (False, False) + event = ScalarEvent(anchor, tag, implicit, token.value, + start_mark, end_mark, style=token.style) + self.state = self.states.pop() + elif self.check_token(FlowSequenceStartToken): + end_mark = self.peek_token().end_mark + event = SequenceStartEvent(anchor, tag, implicit, + start_mark, end_mark, flow_style=True) + self.state = self.parse_flow_sequence_first_entry + elif self.check_token(FlowMappingStartToken): + end_mark = self.peek_token().end_mark + event = MappingStartEvent(anchor, tag, implicit, + start_mark, end_mark, flow_style=True) + self.state = self.parse_flow_mapping_first_key + elif block and self.check_token(BlockSequenceStartToken): + end_mark = self.peek_token().start_mark + event = SequenceStartEvent(anchor, tag, implicit, + start_mark, end_mark, flow_style=False) + self.state = self.parse_block_sequence_first_entry + elif block and self.check_token(BlockMappingStartToken): + end_mark = self.peek_token().start_mark + event = MappingStartEvent(anchor, tag, implicit, + start_mark, end_mark, flow_style=False) + self.state = self.parse_block_mapping_first_key + elif anchor is not None or tag is not None: + # Empty scalars are allowed even if a tag or an anchor is + # specified. + event = ScalarEvent(anchor, tag, (implicit, False), u'', + start_mark, end_mark) + self.state = self.states.pop() + else: + if block: + node = 'block' + else: + node = 'flow' + token = self.peek_token() + raise ParserError("while parsing a %s node" % node, start_mark, + "expected the node content, but found %r" % token.id, + token.start_mark) + return event + + # block_sequence ::= BLOCK-SEQUENCE-START (BLOCK-ENTRY block_node?)* BLOCK-END + + def parse_block_sequence_first_entry(self): + token = self.get_token() + self.marks.append(token.start_mark) + return self.parse_block_sequence_entry() + + def parse_block_sequence_entry(self): + if self.check_token(BlockEntryToken): + token = self.get_token() + if not self.check_token(BlockEntryToken, BlockEndToken): + self.states.append(self.parse_block_sequence_entry) + return self.parse_block_node() + else: + self.state = self.parse_block_sequence_entry + return self.process_empty_scalar(token.end_mark) + if not self.check_token(BlockEndToken): + token = self.peek_token() + raise ParserError("while parsing a block collection", self.marks[-1], + "expected <block end>, but found %r" % token.id, token.start_mark) + token = self.get_token() + event = SequenceEndEvent(token.start_mark, token.end_mark) + self.state = self.states.pop() + self.marks.pop() + return event + + # indentless_sequence ::= (BLOCK-ENTRY block_node?)+ + + def parse_indentless_sequence_entry(self): + if self.check_token(BlockEntryToken): + token = self.get_token() + if not self.check_token(BlockEntryToken, + KeyToken, ValueToken, BlockEndToken): + self.states.append(self.parse_indentless_sequence_entry) + return self.parse_block_node() + else: + self.state = self.parse_indentless_sequence_entry + return self.process_empty_scalar(token.end_mark) + token = self.peek_token() + event = SequenceEndEvent(token.start_mark, token.start_mark) + self.state = self.states.pop() + return event + + # block_mapping ::= BLOCK-MAPPING_START + # ((KEY block_node_or_indentless_sequence?)? + # (VALUE block_node_or_indentless_sequence?)?)* + # BLOCK-END + + def parse_block_mapping_first_key(self): + token = self.get_token() + self.marks.append(token.start_mark) + return self.parse_block_mapping_key() + + def parse_block_mapping_key(self): + if self.check_token(KeyToken): + token = self.get_token() + if not self.check_token(KeyToken, ValueToken, BlockEndToken): + self.states.append(self.parse_block_mapping_value) + return self.parse_block_node_or_indentless_sequence() + else: + self.state = self.parse_block_mapping_value + return self.process_empty_scalar(token.end_mark) + if not self.check_token(BlockEndToken): + token = self.peek_token() + raise ParserError("while parsing a block mapping", self.marks[-1], + "expected <block end>, but found %r" % token.id, token.start_mark) + token = self.get_token() + event = MappingEndEvent(token.start_mark, token.end_mark) + self.state = self.states.pop() + self.marks.pop() + return event + + def parse_block_mapping_value(self): + if self.check_token(ValueToken): + token = self.get_token() + if not self.check_token(KeyToken, ValueToken, BlockEndToken): + self.states.append(self.parse_block_mapping_key) + return self.parse_block_node_or_indentless_sequence() + else: + self.state = self.parse_block_mapping_key + return self.process_empty_scalar(token.end_mark) + else: + self.state = self.parse_block_mapping_key + token = self.peek_token() + return self.process_empty_scalar(token.start_mark) + + # flow_sequence ::= FLOW-SEQUENCE-START + # (flow_sequence_entry FLOW-ENTRY)* + # flow_sequence_entry? + # FLOW-SEQUENCE-END + # flow_sequence_entry ::= flow_node | KEY flow_node? (VALUE flow_node?)? + # + # Note that while production rules for both flow_sequence_entry and + # flow_mapping_entry are equal, their interpretations are different. + # For `flow_sequence_entry`, the part `KEY flow_node? (VALUE flow_node?)?` + # generate an inline mapping (set syntax). + + def parse_flow_sequence_first_entry(self): + token = self.get_token() + self.marks.append(token.start_mark) + return self.parse_flow_sequence_entry(first=True) + + def parse_flow_sequence_entry(self, first=False): + if not self.check_token(FlowSequenceEndToken): + if not first: + if self.check_token(FlowEntryToken): + self.get_token() + else: + token = self.peek_token() + raise ParserError("while parsing a flow sequence", self.marks[-1], + "expected ',' or ']', but got %r" % token.id, token.start_mark) + + if self.check_token(KeyToken): + token = self.peek_token() + event = MappingStartEvent(None, None, True, + token.start_mark, token.end_mark, + flow_style=True) + self.state = self.parse_flow_sequence_entry_mapping_key + return event + elif not self.check_token(FlowSequenceEndToken): + self.states.append(self.parse_flow_sequence_entry) + return self.parse_flow_node() + token = self.get_token() + event = SequenceEndEvent(token.start_mark, token.end_mark) + self.state = self.states.pop() + self.marks.pop() + return event + + def parse_flow_sequence_entry_mapping_key(self): + token = self.get_token() + if not self.check_token(ValueToken, + FlowEntryToken, FlowSequenceEndToken): + self.states.append(self.parse_flow_sequence_entry_mapping_value) + return self.parse_flow_node() + else: + self.state = self.parse_flow_sequence_entry_mapping_value + return self.process_empty_scalar(token.end_mark) + + def parse_flow_sequence_entry_mapping_value(self): + if self.check_token(ValueToken): + token = self.get_token() + if not self.check_token(FlowEntryToken, FlowSequenceEndToken): + self.states.append(self.parse_flow_sequence_entry_mapping_end) + return self.parse_flow_node() + else: + self.state = self.parse_flow_sequence_entry_mapping_end + return self.process_empty_scalar(token.end_mark) + else: + self.state = self.parse_flow_sequence_entry_mapping_end + token = self.peek_token() + return self.process_empty_scalar(token.start_mark) + + def parse_flow_sequence_entry_mapping_end(self): + self.state = self.parse_flow_sequence_entry + token = self.peek_token() + return MappingEndEvent(token.start_mark, token.start_mark) + + # flow_mapping ::= FLOW-MAPPING-START + # (flow_mapping_entry FLOW-ENTRY)* + # flow_mapping_entry? + # FLOW-MAPPING-END + # flow_mapping_entry ::= flow_node | KEY flow_node? (VALUE flow_node?)? + + def parse_flow_mapping_first_key(self): + token = self.get_token() + self.marks.append(token.start_mark) + return self.parse_flow_mapping_key(first=True) + + def parse_flow_mapping_key(self, first=False): + if not self.check_token(FlowMappingEndToken): + if not first: + if self.check_token(FlowEntryToken): + self.get_token() + else: + token = self.peek_token() + raise ParserError("while parsing a flow mapping", self.marks[-1], + "expected ',' or '}', but got %r" % token.id, token.start_mark) + if self.check_token(KeyToken): + token = self.get_token() + if not self.check_token(ValueToken, + FlowEntryToken, FlowMappingEndToken): + self.states.append(self.parse_flow_mapping_value) + return self.parse_flow_node() + else: + self.state = self.parse_flow_mapping_value + return self.process_empty_scalar(token.end_mark) + elif not self.check_token(FlowMappingEndToken): + self.states.append(self.parse_flow_mapping_empty_value) + return self.parse_flow_node() + token = self.get_token() + event = MappingEndEvent(token.start_mark, token.end_mark) + self.state = self.states.pop() + self.marks.pop() + return event + + def parse_flow_mapping_value(self): + if self.check_token(ValueToken): + token = self.get_token() + if not self.check_token(FlowEntryToken, FlowMappingEndToken): + self.states.append(self.parse_flow_mapping_key) + return self.parse_flow_node() + else: + self.state = self.parse_flow_mapping_key + return self.process_empty_scalar(token.end_mark) + else: + self.state = self.parse_flow_mapping_key + token = self.peek_token() + return self.process_empty_scalar(token.start_mark) + + def parse_flow_mapping_empty_value(self): + self.state = self.parse_flow_mapping_key + return self.process_empty_scalar(self.peek_token().start_mark) + + def process_empty_scalar(self, mark): + return ScalarEvent(None, None, (True, False), u'', mark, mark) + diff --git a/python.d/python_modules/pyyaml2/reader.py b/python.d/python_modules/pyyaml2/reader.py new file mode 100644 index 000000000..3249e6b9f --- /dev/null +++ b/python.d/python_modules/pyyaml2/reader.py @@ -0,0 +1,190 @@ +# This module contains abstractions for the input stream. You don't have to +# looks further, there are no pretty code. +# +# We define two classes here. +# +# Mark(source, line, column) +# It's just a record and its only use is producing nice error messages. +# Parser does not use it for any other purposes. +# +# Reader(source, data) +# Reader determines the encoding of `data` and converts it to unicode. +# Reader provides the following methods and attributes: +# reader.peek(length=1) - return the next `length` characters +# reader.forward(length=1) - move the current position to `length` characters. +# reader.index - the number of the current character. +# reader.line, stream.column - the line and the column of the current character. + +__all__ = ['Reader', 'ReaderError'] + +from error import YAMLError, Mark + +import codecs, re + +class ReaderError(YAMLError): + + def __init__(self, name, position, character, encoding, reason): + self.name = name + self.character = character + self.position = position + self.encoding = encoding + self.reason = reason + + def __str__(self): + if isinstance(self.character, str): + return "'%s' codec can't decode byte #x%02x: %s\n" \ + " in \"%s\", position %d" \ + % (self.encoding, ord(self.character), self.reason, + self.name, self.position) + else: + return "unacceptable character #x%04x: %s\n" \ + " in \"%s\", position %d" \ + % (self.character, self.reason, + self.name, self.position) + +class Reader(object): + # Reader: + # - determines the data encoding and converts it to unicode, + # - checks if characters are in allowed range, + # - adds '\0' to the end. + + # Reader accepts + # - a `str` object, + # - a `unicode` object, + # - a file-like object with its `read` method returning `str`, + # - a file-like object with its `read` method returning `unicode`. + + # Yeah, it's ugly and slow. + + def __init__(self, stream): + self.name = None + self.stream = None + self.stream_pointer = 0 + self.eof = True + self.buffer = u'' + self.pointer = 0 + self.raw_buffer = None + self.raw_decode = None + self.encoding = None + self.index = 0 + self.line = 0 + self.column = 0 + if isinstance(stream, unicode): + self.name = "<unicode string>" + self.check_printable(stream) + self.buffer = stream+u'\0' + elif isinstance(stream, str): + self.name = "<string>" + self.raw_buffer = stream + self.determine_encoding() + else: + self.stream = stream + self.name = getattr(stream, 'name', "<file>") + self.eof = False + self.raw_buffer = '' + self.determine_encoding() + + def peek(self, index=0): + try: + return self.buffer[self.pointer+index] + except IndexError: + self.update(index+1) + return self.buffer[self.pointer+index] + + def prefix(self, length=1): + if self.pointer+length >= len(self.buffer): + self.update(length) + return self.buffer[self.pointer:self.pointer+length] + + def forward(self, length=1): + if self.pointer+length+1 >= len(self.buffer): + self.update(length+1) + while length: + ch = self.buffer[self.pointer] + self.pointer += 1 + self.index += 1 + if ch in u'\n\x85\u2028\u2029' \ + or (ch == u'\r' and self.buffer[self.pointer] != u'\n'): + self.line += 1 + self.column = 0 + elif ch != u'\uFEFF': + self.column += 1 + length -= 1 + + def get_mark(self): + if self.stream is None: + return Mark(self.name, self.index, self.line, self.column, + self.buffer, self.pointer) + else: + return Mark(self.name, self.index, self.line, self.column, + None, None) + + def determine_encoding(self): + while not self.eof and len(self.raw_buffer) < 2: + self.update_raw() + if not isinstance(self.raw_buffer, unicode): + if self.raw_buffer.startswith(codecs.BOM_UTF16_LE): + self.raw_decode = codecs.utf_16_le_decode + self.encoding = 'utf-16-le' + elif self.raw_buffer.startswith(codecs.BOM_UTF16_BE): + self.raw_decode = codecs.utf_16_be_decode + self.encoding = 'utf-16-be' + else: + self.raw_decode = codecs.utf_8_decode + self.encoding = 'utf-8' + self.update(1) + + NON_PRINTABLE = re.compile(u'[^\x09\x0A\x0D\x20-\x7E\x85\xA0-\uD7FF\uE000-\uFFFD]') + def check_printable(self, data): + match = self.NON_PRINTABLE.search(data) + if match: + character = match.group() + position = self.index+(len(self.buffer)-self.pointer)+match.start() + raise ReaderError(self.name, position, ord(character), + 'unicode', "special characters are not allowed") + + def update(self, length): + if self.raw_buffer is None: + return + self.buffer = self.buffer[self.pointer:] + self.pointer = 0 + while len(self.buffer) < length: + if not self.eof: + self.update_raw() + if self.raw_decode is not None: + try: + data, converted = self.raw_decode(self.raw_buffer, + 'strict', self.eof) + except UnicodeDecodeError, exc: + character = exc.object[exc.start] + if self.stream is not None: + position = self.stream_pointer-len(self.raw_buffer)+exc.start + else: + position = exc.start + raise ReaderError(self.name, position, character, + exc.encoding, exc.reason) + else: + data = self.raw_buffer + converted = len(data) + self.check_printable(data) + self.buffer += data + self.raw_buffer = self.raw_buffer[converted:] + if self.eof: + self.buffer += u'\0' + self.raw_buffer = None + break + + def update_raw(self, size=1024): + data = self.stream.read(size) + if data: + self.raw_buffer += data + self.stream_pointer += len(data) + else: + self.eof = True + +#try: +# import psyco +# psyco.bind(Reader) +#except ImportError: +# pass + diff --git a/python.d/python_modules/pyyaml2/representer.py b/python.d/python_modules/pyyaml2/representer.py new file mode 100644 index 000000000..5f4fc70db --- /dev/null +++ b/python.d/python_modules/pyyaml2/representer.py @@ -0,0 +1,484 @@ + +__all__ = ['BaseRepresenter', 'SafeRepresenter', 'Representer', + 'RepresenterError'] + +from error import * +from nodes import * + +import datetime + +import sys, copy_reg, types + +class RepresenterError(YAMLError): + pass + +class BaseRepresenter(object): + + yaml_representers = {} + yaml_multi_representers = {} + + def __init__(self, default_style=None, default_flow_style=None): + self.default_style = default_style + self.default_flow_style = default_flow_style + self.represented_objects = {} + self.object_keeper = [] + self.alias_key = None + + def represent(self, data): + node = self.represent_data(data) + self.serialize(node) + self.represented_objects = {} + self.object_keeper = [] + self.alias_key = None + + def get_classobj_bases(self, cls): + bases = [cls] + for base in cls.__bases__: + bases.extend(self.get_classobj_bases(base)) + return bases + + def represent_data(self, data): + if self.ignore_aliases(data): + self.alias_key = None + else: + self.alias_key = id(data) + if self.alias_key is not None: + if self.alias_key in self.represented_objects: + node = self.represented_objects[self.alias_key] + #if node is None: + # raise RepresenterError("recursive objects are not allowed: %r" % data) + return node + #self.represented_objects[alias_key] = None + self.object_keeper.append(data) + data_types = type(data).__mro__ + if type(data) is types.InstanceType: + data_types = self.get_classobj_bases(data.__class__)+list(data_types) + if data_types[0] in self.yaml_representers: + node = self.yaml_representers[data_types[0]](self, data) + else: + for data_type in data_types: + if data_type in self.yaml_multi_representers: + node = self.yaml_multi_representers[data_type](self, data) + break + else: + if None in self.yaml_multi_representers: + node = self.yaml_multi_representers[None](self, data) + elif None in self.yaml_representers: + node = self.yaml_representers[None](self, data) + else: + node = ScalarNode(None, unicode(data)) + #if alias_key is not None: + # self.represented_objects[alias_key] = node + return node + + def add_representer(cls, data_type, representer): + if not 'yaml_representers' in cls.__dict__: + cls.yaml_representers = cls.yaml_representers.copy() + cls.yaml_representers[data_type] = representer + add_representer = classmethod(add_representer) + + def add_multi_representer(cls, data_type, representer): + if not 'yaml_multi_representers' in cls.__dict__: + cls.yaml_multi_representers = cls.yaml_multi_representers.copy() + cls.yaml_multi_representers[data_type] = representer + add_multi_representer = classmethod(add_multi_representer) + + def represent_scalar(self, tag, value, style=None): + if style is None: + style = self.default_style + node = ScalarNode(tag, value, style=style) + if self.alias_key is not None: + self.represented_objects[self.alias_key] = node + return node + + def represent_sequence(self, tag, sequence, flow_style=None): + value = [] + node = SequenceNode(tag, value, flow_style=flow_style) + if self.alias_key is not None: + self.represented_objects[self.alias_key] = node + best_style = True + for item in sequence: + node_item = self.represent_data(item) + if not (isinstance(node_item, ScalarNode) and not node_item.style): + best_style = False + value.append(node_item) + if flow_style is None: + if self.default_flow_style is not None: + node.flow_style = self.default_flow_style + else: + node.flow_style = best_style + return node + + def represent_mapping(self, tag, mapping, flow_style=None): + value = [] + node = MappingNode(tag, value, flow_style=flow_style) + if self.alias_key is not None: + self.represented_objects[self.alias_key] = node + best_style = True + if hasattr(mapping, 'items'): + mapping = mapping.items() + mapping.sort() + for item_key, item_value in mapping: + node_key = self.represent_data(item_key) + node_value = self.represent_data(item_value) + if not (isinstance(node_key, ScalarNode) and not node_key.style): + best_style = False + if not (isinstance(node_value, ScalarNode) and not node_value.style): + best_style = False + value.append((node_key, node_value)) + if flow_style is None: + if self.default_flow_style is not None: + node.flow_style = self.default_flow_style + else: + node.flow_style = best_style + return node + + def ignore_aliases(self, data): + return False + +class SafeRepresenter(BaseRepresenter): + + def ignore_aliases(self, data): + if data in [None, ()]: + return True + if isinstance(data, (str, unicode, bool, int, float)): + return True + + def represent_none(self, data): + return self.represent_scalar(u'tag:yaml.org,2002:null', + u'null') + + def represent_str(self, data): + tag = None + style = None + try: + data = unicode(data, 'ascii') + tag = u'tag:yaml.org,2002:str' + except UnicodeDecodeError: + try: + data = unicode(data, 'utf-8') + tag = u'tag:yaml.org,2002:str' + except UnicodeDecodeError: + data = data.encode('base64') + tag = u'tag:yaml.org,2002:binary' + style = '|' + return self.represent_scalar(tag, data, style=style) + + def represent_unicode(self, data): + return self.represent_scalar(u'tag:yaml.org,2002:str', data) + + def represent_bool(self, data): + if data: + value = u'true' + else: + value = u'false' + return self.represent_scalar(u'tag:yaml.org,2002:bool', value) + + def represent_int(self, data): + return self.represent_scalar(u'tag:yaml.org,2002:int', unicode(data)) + + def represent_long(self, data): + return self.represent_scalar(u'tag:yaml.org,2002:int', unicode(data)) + + inf_value = 1e300 + while repr(inf_value) != repr(inf_value*inf_value): + inf_value *= inf_value + + def represent_float(self, data): + if data != data or (data == 0.0 and data == 1.0): + value = u'.nan' + elif data == self.inf_value: + value = u'.inf' + elif data == -self.inf_value: + value = u'-.inf' + else: + value = unicode(repr(data)).lower() + # Note that in some cases `repr(data)` represents a float number + # without the decimal parts. For instance: + # >>> repr(1e17) + # '1e17' + # Unfortunately, this is not a valid float representation according + # to the definition of the `!!float` tag. We fix this by adding + # '.0' before the 'e' symbol. + if u'.' not in value and u'e' in value: + value = value.replace(u'e', u'.0e', 1) + return self.represent_scalar(u'tag:yaml.org,2002:float', value) + + def represent_list(self, data): + #pairs = (len(data) > 0 and isinstance(data, list)) + #if pairs: + # for item in data: + # if not isinstance(item, tuple) or len(item) != 2: + # pairs = False + # break + #if not pairs: + return self.represent_sequence(u'tag:yaml.org,2002:seq', data) + #value = [] + #for item_key, item_value in data: + # value.append(self.represent_mapping(u'tag:yaml.org,2002:map', + # [(item_key, item_value)])) + #return SequenceNode(u'tag:yaml.org,2002:pairs', value) + + def represent_dict(self, data): + return self.represent_mapping(u'tag:yaml.org,2002:map', data) + + def represent_set(self, data): + value = {} + for key in data: + value[key] = None + return self.represent_mapping(u'tag:yaml.org,2002:set', value) + + def represent_date(self, data): + value = unicode(data.isoformat()) + return self.represent_scalar(u'tag:yaml.org,2002:timestamp', value) + + def represent_datetime(self, data): + value = unicode(data.isoformat(' ')) + return self.represent_scalar(u'tag:yaml.org,2002:timestamp', value) + + def represent_yaml_object(self, tag, data, cls, flow_style=None): + if hasattr(data, '__getstate__'): + state = data.__getstate__() + else: + state = data.__dict__.copy() + return self.represent_mapping(tag, state, flow_style=flow_style) + + def represent_undefined(self, data): + raise RepresenterError("cannot represent an object: %s" % data) + +SafeRepresenter.add_representer(type(None), + SafeRepresenter.represent_none) + +SafeRepresenter.add_representer(str, + SafeRepresenter.represent_str) + +SafeRepresenter.add_representer(unicode, + SafeRepresenter.represent_unicode) + +SafeRepresenter.add_representer(bool, + SafeRepresenter.represent_bool) + +SafeRepresenter.add_representer(int, + SafeRepresenter.represent_int) + +SafeRepresenter.add_representer(long, + SafeRepresenter.represent_long) + +SafeRepresenter.add_representer(float, + SafeRepresenter.represent_float) + +SafeRepresenter.add_representer(list, + SafeRepresenter.represent_list) + +SafeRepresenter.add_representer(tuple, + SafeRepresenter.represent_list) + +SafeRepresenter.add_representer(dict, + SafeRepresenter.represent_dict) + +SafeRepresenter.add_representer(set, + SafeRepresenter.represent_set) + +SafeRepresenter.add_representer(datetime.date, + SafeRepresenter.represent_date) + +SafeRepresenter.add_representer(datetime.datetime, + SafeRepresenter.represent_datetime) + +SafeRepresenter.add_representer(None, + SafeRepresenter.represent_undefined) + +class Representer(SafeRepresenter): + + def represent_str(self, data): + tag = None + style = None + try: + data = unicode(data, 'ascii') + tag = u'tag:yaml.org,2002:str' + except UnicodeDecodeError: + try: + data = unicode(data, 'utf-8') + tag = u'tag:yaml.org,2002:python/str' + except UnicodeDecodeError: + data = data.encode('base64') + tag = u'tag:yaml.org,2002:binary' + style = '|' + return self.represent_scalar(tag, data, style=style) + + def represent_unicode(self, data): + tag = None + try: + data.encode('ascii') + tag = u'tag:yaml.org,2002:python/unicode' + except UnicodeEncodeError: + tag = u'tag:yaml.org,2002:str' + return self.represent_scalar(tag, data) + + def represent_long(self, data): + tag = u'tag:yaml.org,2002:int' + if int(data) is not data: + tag = u'tag:yaml.org,2002:python/long' + return self.represent_scalar(tag, unicode(data)) + + def represent_complex(self, data): + if data.imag == 0.0: + data = u'%r' % data.real + elif data.real == 0.0: + data = u'%rj' % data.imag + elif data.imag > 0: + data = u'%r+%rj' % (data.real, data.imag) + else: + data = u'%r%rj' % (data.real, data.imag) + return self.represent_scalar(u'tag:yaml.org,2002:python/complex', data) + + def represent_tuple(self, data): + return self.represent_sequence(u'tag:yaml.org,2002:python/tuple', data) + + def represent_name(self, data): + name = u'%s.%s' % (data.__module__, data.__name__) + return self.represent_scalar(u'tag:yaml.org,2002:python/name:'+name, u'') + + def represent_module(self, data): + return self.represent_scalar( + u'tag:yaml.org,2002:python/module:'+data.__name__, u'') + + def represent_instance(self, data): + # For instances of classic classes, we use __getinitargs__ and + # __getstate__ to serialize the data. + + # If data.__getinitargs__ exists, the object must be reconstructed by + # calling cls(**args), where args is a tuple returned by + # __getinitargs__. Otherwise, the cls.__init__ method should never be + # called and the class instance is created by instantiating a trivial + # class and assigning to the instance's __class__ variable. + + # If data.__getstate__ exists, it returns the state of the object. + # Otherwise, the state of the object is data.__dict__. + + # We produce either a !!python/object or !!python/object/new node. + # If data.__getinitargs__ does not exist and state is a dictionary, we + # produce a !!python/object node . Otherwise we produce a + # !!python/object/new node. + + cls = data.__class__ + class_name = u'%s.%s' % (cls.__module__, cls.__name__) + args = None + state = None + if hasattr(data, '__getinitargs__'): + args = list(data.__getinitargs__()) + if hasattr(data, '__getstate__'): + state = data.__getstate__() + else: + state = data.__dict__ + if args is None and isinstance(state, dict): + return self.represent_mapping( + u'tag:yaml.org,2002:python/object:'+class_name, state) + if isinstance(state, dict) and not state: + return self.represent_sequence( + u'tag:yaml.org,2002:python/object/new:'+class_name, args) + value = {} + if args: + value['args'] = args + value['state'] = state + return self.represent_mapping( + u'tag:yaml.org,2002:python/object/new:'+class_name, value) + + def represent_object(self, data): + # We use __reduce__ API to save the data. data.__reduce__ returns + # a tuple of length 2-5: + # (function, args, state, listitems, dictitems) + + # For reconstructing, we calls function(*args), then set its state, + # listitems, and dictitems if they are not None. + + # A special case is when function.__name__ == '__newobj__'. In this + # case we create the object with args[0].__new__(*args). + + # Another special case is when __reduce__ returns a string - we don't + # support it. + + # We produce a !!python/object, !!python/object/new or + # !!python/object/apply node. + + cls = type(data) + if cls in copy_reg.dispatch_table: + reduce = copy_reg.dispatch_table[cls](data) + elif hasattr(data, '__reduce_ex__'): + reduce = data.__reduce_ex__(2) + elif hasattr(data, '__reduce__'): + reduce = data.__reduce__() + else: + raise RepresenterError("cannot represent object: %r" % data) + reduce = (list(reduce)+[None]*5)[:5] + function, args, state, listitems, dictitems = reduce + args = list(args) + if state is None: + state = {} + if listitems is not None: + listitems = list(listitems) + if dictitems is not None: + dictitems = dict(dictitems) + if function.__name__ == '__newobj__': + function = args[0] + args = args[1:] + tag = u'tag:yaml.org,2002:python/object/new:' + newobj = True + else: + tag = u'tag:yaml.org,2002:python/object/apply:' + newobj = False + function_name = u'%s.%s' % (function.__module__, function.__name__) + if not args and not listitems and not dictitems \ + and isinstance(state, dict) and newobj: + return self.represent_mapping( + u'tag:yaml.org,2002:python/object:'+function_name, state) + if not listitems and not dictitems \ + and isinstance(state, dict) and not state: + return self.represent_sequence(tag+function_name, args) + value = {} + if args: + value['args'] = args + if state or not isinstance(state, dict): + value['state'] = state + if listitems: + value['listitems'] = listitems + if dictitems: + value['dictitems'] = dictitems + return self.represent_mapping(tag+function_name, value) + +Representer.add_representer(str, + Representer.represent_str) + +Representer.add_representer(unicode, + Representer.represent_unicode) + +Representer.add_representer(long, + Representer.represent_long) + +Representer.add_representer(complex, + Representer.represent_complex) + +Representer.add_representer(tuple, + Representer.represent_tuple) + +Representer.add_representer(type, + Representer.represent_name) + +Representer.add_representer(types.ClassType, + Representer.represent_name) + +Representer.add_representer(types.FunctionType, + Representer.represent_name) + +Representer.add_representer(types.BuiltinFunctionType, + Representer.represent_name) + +Representer.add_representer(types.ModuleType, + Representer.represent_module) + +Representer.add_multi_representer(types.InstanceType, + Representer.represent_instance) + +Representer.add_multi_representer(object, + Representer.represent_object) + diff --git a/python.d/python_modules/pyyaml2/resolver.py b/python.d/python_modules/pyyaml2/resolver.py new file mode 100644 index 000000000..6b5ab8759 --- /dev/null +++ b/python.d/python_modules/pyyaml2/resolver.py @@ -0,0 +1,224 @@ + +__all__ = ['BaseResolver', 'Resolver'] + +from error import * +from nodes import * + +import re + +class ResolverError(YAMLError): + pass + +class BaseResolver(object): + + DEFAULT_SCALAR_TAG = u'tag:yaml.org,2002:str' + DEFAULT_SEQUENCE_TAG = u'tag:yaml.org,2002:seq' + DEFAULT_MAPPING_TAG = u'tag:yaml.org,2002:map' + + yaml_implicit_resolvers = {} + yaml_path_resolvers = {} + + def __init__(self): + self.resolver_exact_paths = [] + self.resolver_prefix_paths = [] + + def add_implicit_resolver(cls, tag, regexp, first): + if not 'yaml_implicit_resolvers' in cls.__dict__: + cls.yaml_implicit_resolvers = cls.yaml_implicit_resolvers.copy() + if first is None: + first = [None] + for ch in first: + cls.yaml_implicit_resolvers.setdefault(ch, []).append((tag, regexp)) + add_implicit_resolver = classmethod(add_implicit_resolver) + + def add_path_resolver(cls, tag, path, kind=None): + # Note: `add_path_resolver` is experimental. The API could be changed. + # `new_path` is a pattern that is matched against the path from the + # root to the node that is being considered. `node_path` elements are + # tuples `(node_check, index_check)`. `node_check` is a node class: + # `ScalarNode`, `SequenceNode`, `MappingNode` or `None`. `None` + # matches any kind of a node. `index_check` could be `None`, a boolean + # value, a string value, or a number. `None` and `False` match against + # any _value_ of sequence and mapping nodes. `True` matches against + # any _key_ of a mapping node. A string `index_check` matches against + # a mapping value that corresponds to a scalar key which content is + # equal to the `index_check` value. An integer `index_check` matches + # against a sequence value with the index equal to `index_check`. + if not 'yaml_path_resolvers' in cls.__dict__: + cls.yaml_path_resolvers = cls.yaml_path_resolvers.copy() + new_path = [] + for element in path: + if isinstance(element, (list, tuple)): + if len(element) == 2: + node_check, index_check = element + elif len(element) == 1: + node_check = element[0] + index_check = True + else: + raise ResolverError("Invalid path element: %s" % element) + else: + node_check = None + index_check = element + if node_check is str: + node_check = ScalarNode + elif node_check is list: + node_check = SequenceNode + elif node_check is dict: + node_check = MappingNode + elif node_check not in [ScalarNode, SequenceNode, MappingNode] \ + and not isinstance(node_check, basestring) \ + and node_check is not None: + raise ResolverError("Invalid node checker: %s" % node_check) + if not isinstance(index_check, (basestring, int)) \ + and index_check is not None: + raise ResolverError("Invalid index checker: %s" % index_check) + new_path.append((node_check, index_check)) + if kind is str: + kind = ScalarNode + elif kind is list: + kind = SequenceNode + elif kind is dict: + kind = MappingNode + elif kind not in [ScalarNode, SequenceNode, MappingNode] \ + and kind is not None: + raise ResolverError("Invalid node kind: %s" % kind) + cls.yaml_path_resolvers[tuple(new_path), kind] = tag + add_path_resolver = classmethod(add_path_resolver) + + def descend_resolver(self, current_node, current_index): + if not self.yaml_path_resolvers: + return + exact_paths = {} + prefix_paths = [] + if current_node: + depth = len(self.resolver_prefix_paths) + for path, kind in self.resolver_prefix_paths[-1]: + if self.check_resolver_prefix(depth, path, kind, + current_node, current_index): + if len(path) > depth: + prefix_paths.append((path, kind)) + else: + exact_paths[kind] = self.yaml_path_resolvers[path, kind] + else: + for path, kind in self.yaml_path_resolvers: + if not path: + exact_paths[kind] = self.yaml_path_resolvers[path, kind] + else: + prefix_paths.append((path, kind)) + self.resolver_exact_paths.append(exact_paths) + self.resolver_prefix_paths.append(prefix_paths) + + def ascend_resolver(self): + if not self.yaml_path_resolvers: + return + self.resolver_exact_paths.pop() + self.resolver_prefix_paths.pop() + + def check_resolver_prefix(self, depth, path, kind, + current_node, current_index): + node_check, index_check = path[depth-1] + if isinstance(node_check, basestring): + if current_node.tag != node_check: + return + elif node_check is not None: + if not isinstance(current_node, node_check): + return + if index_check is True and current_index is not None: + return + if (index_check is False or index_check is None) \ + and current_index is None: + return + if isinstance(index_check, basestring): + if not (isinstance(current_index, ScalarNode) + and index_check == current_index.value): + return + elif isinstance(index_check, int) and not isinstance(index_check, bool): + if index_check != current_index: + return + return True + + def resolve(self, kind, value, implicit): + if kind is ScalarNode and implicit[0]: + if value == u'': + resolvers = self.yaml_implicit_resolvers.get(u'', []) + else: + resolvers = self.yaml_implicit_resolvers.get(value[0], []) + resolvers += self.yaml_implicit_resolvers.get(None, []) + for tag, regexp in resolvers: + if regexp.match(value): + return tag + implicit = implicit[1] + if self.yaml_path_resolvers: + exact_paths = self.resolver_exact_paths[-1] + if kind in exact_paths: + return exact_paths[kind] + if None in exact_paths: + return exact_paths[None] + if kind is ScalarNode: + return self.DEFAULT_SCALAR_TAG + elif kind is SequenceNode: + return self.DEFAULT_SEQUENCE_TAG + elif kind is MappingNode: + return self.DEFAULT_MAPPING_TAG + +class Resolver(BaseResolver): + pass + +Resolver.add_implicit_resolver( + u'tag:yaml.org,2002:bool', + re.compile(ur'''^(?:yes|Yes|YES|no|No|NO + |true|True|TRUE|false|False|FALSE + |on|On|ON|off|Off|OFF)$''', re.X), + list(u'yYnNtTfFoO')) + +Resolver.add_implicit_resolver( + u'tag:yaml.org,2002:float', + re.compile(ur'''^(?:[-+]?(?:[0-9][0-9_]*)\.[0-9_]*(?:[eE][-+][0-9]+)? + |\.[0-9_]+(?:[eE][-+][0-9]+)? + |[-+]?[0-9][0-9_]*(?::[0-5]?[0-9])+\.[0-9_]* + |[-+]?\.(?:inf|Inf|INF) + |\.(?:nan|NaN|NAN))$''', re.X), + list(u'-+0123456789.')) + +Resolver.add_implicit_resolver( + u'tag:yaml.org,2002:int', + re.compile(ur'''^(?:[-+]?0b[0-1_]+ + |[-+]?0[0-7_]+ + |[-+]?(?:0|[1-9][0-9_]*) + |[-+]?0x[0-9a-fA-F_]+ + |[-+]?[1-9][0-9_]*(?::[0-5]?[0-9])+)$''', re.X), + list(u'-+0123456789')) + +Resolver.add_implicit_resolver( + u'tag:yaml.org,2002:merge', + re.compile(ur'^(?:<<)$'), + [u'<']) + +Resolver.add_implicit_resolver( + u'tag:yaml.org,2002:null', + re.compile(ur'''^(?: ~ + |null|Null|NULL + | )$''', re.X), + [u'~', u'n', u'N', u'']) + +Resolver.add_implicit_resolver( + u'tag:yaml.org,2002:timestamp', + re.compile(ur'''^(?:[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9] + |[0-9][0-9][0-9][0-9] -[0-9][0-9]? -[0-9][0-9]? + (?:[Tt]|[ \t]+)[0-9][0-9]? + :[0-9][0-9] :[0-9][0-9] (?:\.[0-9]*)? + (?:[ \t]*(?:Z|[-+][0-9][0-9]?(?::[0-9][0-9])?))?)$''', re.X), + list(u'0123456789')) + +Resolver.add_implicit_resolver( + u'tag:yaml.org,2002:value', + re.compile(ur'^(?:=)$'), + [u'=']) + +# The following resolver is only for documentation purposes. It cannot work +# because plain scalars cannot start with '!', '&', or '*'. +Resolver.add_implicit_resolver( + u'tag:yaml.org,2002:yaml', + re.compile(ur'^(?:!|&|\*)$'), + list(u'!&*')) + diff --git a/python.d/python_modules/pyyaml2/scanner.py b/python.d/python_modules/pyyaml2/scanner.py new file mode 100644 index 000000000..5228fad65 --- /dev/null +++ b/python.d/python_modules/pyyaml2/scanner.py @@ -0,0 +1,1457 @@ + +# Scanner produces tokens of the following types: +# STREAM-START +# STREAM-END +# DIRECTIVE(name, value) +# DOCUMENT-START +# DOCUMENT-END +# BLOCK-SEQUENCE-START +# BLOCK-MAPPING-START +# BLOCK-END +# FLOW-SEQUENCE-START +# FLOW-MAPPING-START +# FLOW-SEQUENCE-END +# FLOW-MAPPING-END +# BLOCK-ENTRY +# FLOW-ENTRY +# KEY +# VALUE +# ALIAS(value) +# ANCHOR(value) +# TAG(value) +# SCALAR(value, plain, style) +# +# Read comments in the Scanner code for more details. +# + +__all__ = ['Scanner', 'ScannerError'] + +from error import MarkedYAMLError +from tokens import * + +class ScannerError(MarkedYAMLError): + pass + +class SimpleKey(object): + # See below simple keys treatment. + + def __init__(self, token_number, required, index, line, column, mark): + self.token_number = token_number + self.required = required + self.index = index + self.line = line + self.column = column + self.mark = mark + +class Scanner(object): + + def __init__(self): + """Initialize the scanner.""" + # It is assumed that Scanner and Reader will have a common descendant. + # Reader do the dirty work of checking for BOM and converting the + # input data to Unicode. It also adds NUL to the end. + # + # Reader supports the following methods + # self.peek(i=0) # peek the next i-th character + # self.prefix(l=1) # peek the next l characters + # self.forward(l=1) # read the next l characters and move the pointer. + + # Had we reached the end of the stream? + self.done = False + + # The number of unclosed '{' and '['. `flow_level == 0` means block + # context. + self.flow_level = 0 + + # List of processed tokens that are not yet emitted. + self.tokens = [] + + # Add the STREAM-START token. + self.fetch_stream_start() + + # Number of tokens that were emitted through the `get_token` method. + self.tokens_taken = 0 + + # The current indentation level. + self.indent = -1 + + # Past indentation levels. + self.indents = [] + + # Variables related to simple keys treatment. + + # A simple key is a key that is not denoted by the '?' indicator. + # Example of simple keys: + # --- + # block simple key: value + # ? not a simple key: + # : { flow simple key: value } + # We emit the KEY token before all keys, so when we find a potential + # simple key, we try to locate the corresponding ':' indicator. + # Simple keys should be limited to a single line and 1024 characters. + + # Can a simple key start at the current position? A simple key may + # start: + # - at the beginning of the line, not counting indentation spaces + # (in block context), + # - after '{', '[', ',' (in the flow context), + # - after '?', ':', '-' (in the block context). + # In the block context, this flag also signifies if a block collection + # may start at the current position. + self.allow_simple_key = True + + # Keep track of possible simple keys. This is a dictionary. The key + # is `flow_level`; there can be no more that one possible simple key + # for each level. The value is a SimpleKey record: + # (token_number, required, index, line, column, mark) + # A simple key may start with ALIAS, ANCHOR, TAG, SCALAR(flow), + # '[', or '{' tokens. + self.possible_simple_keys = {} + + # Public methods. + + def check_token(self, *choices): + # Check if the next token is one of the given types. + while self.need_more_tokens(): + self.fetch_more_tokens() + if self.tokens: + if not choices: + return True + for choice in choices: + if isinstance(self.tokens[0], choice): + return True + return False + + def peek_token(self): + # Return the next token, but do not delete if from the queue. + while self.need_more_tokens(): + self.fetch_more_tokens() + if self.tokens: + return self.tokens[0] + + def get_token(self): + # Return the next token. + while self.need_more_tokens(): + self.fetch_more_tokens() + if self.tokens: + self.tokens_taken += 1 + return self.tokens.pop(0) + + # Private methods. + + def need_more_tokens(self): + if self.done: + return False + if not self.tokens: + return True + # The current token may be a potential simple key, so we + # need to look further. + self.stale_possible_simple_keys() + if self.next_possible_simple_key() == self.tokens_taken: + return True + + def fetch_more_tokens(self): + + # Eat whitespaces and comments until we reach the next token. + self.scan_to_next_token() + + # Remove obsolete possible simple keys. + self.stale_possible_simple_keys() + + # Compare the current indentation and column. It may add some tokens + # and decrease the current indentation level. + self.unwind_indent(self.column) + + # Peek the next character. + ch = self.peek() + + # Is it the end of stream? + if ch == u'\0': + return self.fetch_stream_end() + + # Is it a directive? + if ch == u'%' and self.check_directive(): + return self.fetch_directive() + + # Is it the document start? + if ch == u'-' and self.check_document_start(): + return self.fetch_document_start() + + # Is it the document end? + if ch == u'.' and self.check_document_end(): + return self.fetch_document_end() + + # TODO: support for BOM within a stream. + #if ch == u'\uFEFF': + # return self.fetch_bom() <-- issue BOMToken + + # Note: the order of the following checks is NOT significant. + + # Is it the flow sequence start indicator? + if ch == u'[': + return self.fetch_flow_sequence_start() + + # Is it the flow mapping start indicator? + if ch == u'{': + return self.fetch_flow_mapping_start() + + # Is it the flow sequence end indicator? + if ch == u']': + return self.fetch_flow_sequence_end() + + # Is it the flow mapping end indicator? + if ch == u'}': + return self.fetch_flow_mapping_end() + + # Is it the flow entry indicator? + if ch == u',': + return self.fetch_flow_entry() + + # Is it the block entry indicator? + if ch == u'-' and self.check_block_entry(): + return self.fetch_block_entry() + + # Is it the key indicator? + if ch == u'?' and self.check_key(): + return self.fetch_key() + + # Is it the value indicator? + if ch == u':' and self.check_value(): + return self.fetch_value() + + # Is it an alias? + if ch == u'*': + return self.fetch_alias() + + # Is it an anchor? + if ch == u'&': + return self.fetch_anchor() + + # Is it a tag? + if ch == u'!': + return self.fetch_tag() + + # Is it a literal scalar? + if ch == u'|' and not self.flow_level: + return self.fetch_literal() + + # Is it a folded scalar? + if ch == u'>' and not self.flow_level: + return self.fetch_folded() + + # Is it a single quoted scalar? + if ch == u'\'': + return self.fetch_single() + + # Is it a double quoted scalar? + if ch == u'\"': + return self.fetch_double() + + # It must be a plain scalar then. + if self.check_plain(): + return self.fetch_plain() + + # No? It's an error. Let's produce a nice error message. + raise ScannerError("while scanning for the next token", None, + "found character %r that cannot start any token" + % ch.encode('utf-8'), self.get_mark()) + + # Simple keys treatment. + + def next_possible_simple_key(self): + # Return the number of the nearest possible simple key. Actually we + # don't need to loop through the whole dictionary. We may replace it + # with the following code: + # if not self.possible_simple_keys: + # return None + # return self.possible_simple_keys[ + # min(self.possible_simple_keys.keys())].token_number + min_token_number = None + for level in self.possible_simple_keys: + key = self.possible_simple_keys[level] + if min_token_number is None or key.token_number < min_token_number: + min_token_number = key.token_number + return min_token_number + + def stale_possible_simple_keys(self): + # Remove entries that are no longer possible simple keys. According to + # the YAML specification, simple keys + # - should be limited to a single line, + # - should be no longer than 1024 characters. + # Disabling this procedure will allow simple keys of any length and + # height (may cause problems if indentation is broken though). + for level in self.possible_simple_keys.keys(): + key = self.possible_simple_keys[level] + if key.line != self.line \ + or self.index-key.index > 1024: + if key.required: + raise ScannerError("while scanning a simple key", key.mark, + "could not found expected ':'", self.get_mark()) + del self.possible_simple_keys[level] + + def save_possible_simple_key(self): + # The next token may start a simple key. We check if it's possible + # and save its position. This function is called for + # ALIAS, ANCHOR, TAG, SCALAR(flow), '[', and '{'. + + # Check if a simple key is required at the current position. + required = not self.flow_level and self.indent == self.column + + # A simple key is required only if it is the first token in the current + # line. Therefore it is always allowed. + assert self.allow_simple_key or not required + + # The next token might be a simple key. Let's save it's number and + # position. + if self.allow_simple_key: + self.remove_possible_simple_key() + token_number = self.tokens_taken+len(self.tokens) + key = SimpleKey(token_number, required, + self.index, self.line, self.column, self.get_mark()) + self.possible_simple_keys[self.flow_level] = key + + def remove_possible_simple_key(self): + # Remove the saved possible key position at the current flow level. + if self.flow_level in self.possible_simple_keys: + key = self.possible_simple_keys[self.flow_level] + + if key.required: + raise ScannerError("while scanning a simple key", key.mark, + "could not found expected ':'", self.get_mark()) + + del self.possible_simple_keys[self.flow_level] + + # Indentation functions. + + def unwind_indent(self, column): + + ## In flow context, tokens should respect indentation. + ## Actually the condition should be `self.indent >= column` according to + ## the spec. But this condition will prohibit intuitively correct + ## constructions such as + ## key : { + ## } + #if self.flow_level and self.indent > column: + # raise ScannerError(None, None, + # "invalid intendation or unclosed '[' or '{'", + # self.get_mark()) + + # In the flow context, indentation is ignored. We make the scanner less + # restrictive then specification requires. + if self.flow_level: + return + + # In block context, we may need to issue the BLOCK-END tokens. + while self.indent > column: + mark = self.get_mark() + self.indent = self.indents.pop() + self.tokens.append(BlockEndToken(mark, mark)) + + def add_indent(self, column): + # Check if we need to increase indentation. + if self.indent < column: + self.indents.append(self.indent) + self.indent = column + return True + return False + + # Fetchers. + + def fetch_stream_start(self): + # We always add STREAM-START as the first token and STREAM-END as the + # last token. + + # Read the token. + mark = self.get_mark() + + # Add STREAM-START. + self.tokens.append(StreamStartToken(mark, mark, + encoding=self.encoding)) + + + def fetch_stream_end(self): + + # Set the current intendation to -1. + self.unwind_indent(-1) + + # Reset simple keys. + self.remove_possible_simple_key() + self.allow_simple_key = False + self.possible_simple_keys = {} + + # Read the token. + mark = self.get_mark() + + # Add STREAM-END. + self.tokens.append(StreamEndToken(mark, mark)) + + # The steam is finished. + self.done = True + + def fetch_directive(self): + + # Set the current intendation to -1. + self.unwind_indent(-1) + + # Reset simple keys. + self.remove_possible_simple_key() + self.allow_simple_key = False + + # Scan and add DIRECTIVE. + self.tokens.append(self.scan_directive()) + + def fetch_document_start(self): + self.fetch_document_indicator(DocumentStartToken) + + def fetch_document_end(self): + self.fetch_document_indicator(DocumentEndToken) + + def fetch_document_indicator(self, TokenClass): + + # Set the current intendation to -1. + self.unwind_indent(-1) + + # Reset simple keys. Note that there could not be a block collection + # after '---'. + self.remove_possible_simple_key() + self.allow_simple_key = False + + # Add DOCUMENT-START or DOCUMENT-END. + start_mark = self.get_mark() + self.forward(3) + end_mark = self.get_mark() + self.tokens.append(TokenClass(start_mark, end_mark)) + + def fetch_flow_sequence_start(self): + self.fetch_flow_collection_start(FlowSequenceStartToken) + + def fetch_flow_mapping_start(self): + self.fetch_flow_collection_start(FlowMappingStartToken) + + def fetch_flow_collection_start(self, TokenClass): + + # '[' and '{' may start a simple key. + self.save_possible_simple_key() + + # Increase the flow level. + self.flow_level += 1 + + # Simple keys are allowed after '[' and '{'. + self.allow_simple_key = True + + # Add FLOW-SEQUENCE-START or FLOW-MAPPING-START. + start_mark = self.get_mark() + self.forward() + end_mark = self.get_mark() + self.tokens.append(TokenClass(start_mark, end_mark)) + + def fetch_flow_sequence_end(self): + self.fetch_flow_collection_end(FlowSequenceEndToken) + + def fetch_flow_mapping_end(self): + self.fetch_flow_collection_end(FlowMappingEndToken) + + def fetch_flow_collection_end(self, TokenClass): + + # Reset possible simple key on the current level. + self.remove_possible_simple_key() + + # Decrease the flow level. + self.flow_level -= 1 + + # No simple keys after ']' or '}'. + self.allow_simple_key = False + + # Add FLOW-SEQUENCE-END or FLOW-MAPPING-END. + start_mark = self.get_mark() + self.forward() + end_mark = self.get_mark() + self.tokens.append(TokenClass(start_mark, end_mark)) + + def fetch_flow_entry(self): + + # Simple keys are allowed after ','. + self.allow_simple_key = True + + # Reset possible simple key on the current level. + self.remove_possible_simple_key() + + # Add FLOW-ENTRY. + start_mark = self.get_mark() + self.forward() + end_mark = self.get_mark() + self.tokens.append(FlowEntryToken(start_mark, end_mark)) + + def fetch_block_entry(self): + + # Block context needs additional checks. + if not self.flow_level: + + # Are we allowed to start a new entry? + if not self.allow_simple_key: + raise ScannerError(None, None, + "sequence entries are not allowed here", + self.get_mark()) + + # We may need to add BLOCK-SEQUENCE-START. + if self.add_indent(self.column): + mark = self.get_mark() + self.tokens.append(BlockSequenceStartToken(mark, mark)) + + # It's an error for the block entry to occur in the flow context, + # but we let the parser detect this. + else: + pass + + # Simple keys are allowed after '-'. + self.allow_simple_key = True + + # Reset possible simple key on the current level. + self.remove_possible_simple_key() + + # Add BLOCK-ENTRY. + start_mark = self.get_mark() + self.forward() + end_mark = self.get_mark() + self.tokens.append(BlockEntryToken(start_mark, end_mark)) + + def fetch_key(self): + + # Block context needs additional checks. + if not self.flow_level: + + # Are we allowed to start a key (not nessesary a simple)? + if not self.allow_simple_key: + raise ScannerError(None, None, + "mapping keys are not allowed here", + self.get_mark()) + + # We may need to add BLOCK-MAPPING-START. + if self.add_indent(self.column): + mark = self.get_mark() + self.tokens.append(BlockMappingStartToken(mark, mark)) + + # Simple keys are allowed after '?' in the block context. + self.allow_simple_key = not self.flow_level + + # Reset possible simple key on the current level. + self.remove_possible_simple_key() + + # Add KEY. + start_mark = self.get_mark() + self.forward() + end_mark = self.get_mark() + self.tokens.append(KeyToken(start_mark, end_mark)) + + def fetch_value(self): + + # Do we determine a simple key? + if self.flow_level in self.possible_simple_keys: + + # Add KEY. + key = self.possible_simple_keys[self.flow_level] + del self.possible_simple_keys[self.flow_level] + self.tokens.insert(key.token_number-self.tokens_taken, + KeyToken(key.mark, key.mark)) + + # If this key starts a new block mapping, we need to add + # BLOCK-MAPPING-START. + if not self.flow_level: + if self.add_indent(key.column): + self.tokens.insert(key.token_number-self.tokens_taken, + BlockMappingStartToken(key.mark, key.mark)) + + # There cannot be two simple keys one after another. + self.allow_simple_key = False + + # It must be a part of a complex key. + else: + + # Block context needs additional checks. + # (Do we really need them? They will be catched by the parser + # anyway.) + if not self.flow_level: + + # We are allowed to start a complex value if and only if + # we can start a simple key. + if not self.allow_simple_key: + raise ScannerError(None, None, + "mapping values are not allowed here", + self.get_mark()) + + # If this value starts a new block mapping, we need to add + # BLOCK-MAPPING-START. It will be detected as an error later by + # the parser. + if not self.flow_level: + if self.add_indent(self.column): + mark = self.get_mark() + self.tokens.append(BlockMappingStartToken(mark, mark)) + + # Simple keys are allowed after ':' in the block context. + self.allow_simple_key = not self.flow_level + + # Reset possible simple key on the current level. + self.remove_possible_simple_key() + + # Add VALUE. + start_mark = self.get_mark() + self.forward() + end_mark = self.get_mark() + self.tokens.append(ValueToken(start_mark, end_mark)) + + def fetch_alias(self): + + # ALIAS could be a simple key. + self.save_possible_simple_key() + + # No simple keys after ALIAS. + self.allow_simple_key = False + + # Scan and add ALIAS. + self.tokens.append(self.scan_anchor(AliasToken)) + + def fetch_anchor(self): + + # ANCHOR could start a simple key. + self.save_possible_simple_key() + + # No simple keys after ANCHOR. + self.allow_simple_key = False + + # Scan and add ANCHOR. + self.tokens.append(self.scan_anchor(AnchorToken)) + + def fetch_tag(self): + + # TAG could start a simple key. + self.save_possible_simple_key() + + # No simple keys after TAG. + self.allow_simple_key = False + + # Scan and add TAG. + self.tokens.append(self.scan_tag()) + + def fetch_literal(self): + self.fetch_block_scalar(style='|') + + def fetch_folded(self): + self.fetch_block_scalar(style='>') + + def fetch_block_scalar(self, style): + + # A simple key may follow a block scalar. + self.allow_simple_key = True + + # Reset possible simple key on the current level. + self.remove_possible_simple_key() + + # Scan and add SCALAR. + self.tokens.append(self.scan_block_scalar(style)) + + def fetch_single(self): + self.fetch_flow_scalar(style='\'') + + def fetch_double(self): + self.fetch_flow_scalar(style='"') + + def fetch_flow_scalar(self, style): + + # A flow scalar could be a simple key. + self.save_possible_simple_key() + + # No simple keys after flow scalars. + self.allow_simple_key = False + + # Scan and add SCALAR. + self.tokens.append(self.scan_flow_scalar(style)) + + def fetch_plain(self): + + # A plain scalar could be a simple key. + self.save_possible_simple_key() + + # No simple keys after plain scalars. But note that `scan_plain` will + # change this flag if the scan is finished at the beginning of the + # line. + self.allow_simple_key = False + + # Scan and add SCALAR. May change `allow_simple_key`. + self.tokens.append(self.scan_plain()) + + # Checkers. + + def check_directive(self): + + # DIRECTIVE: ^ '%' ... + # The '%' indicator is already checked. + if self.column == 0: + return True + + def check_document_start(self): + + # DOCUMENT-START: ^ '---' (' '|'\n') + if self.column == 0: + if self.prefix(3) == u'---' \ + and self.peek(3) in u'\0 \t\r\n\x85\u2028\u2029': + return True + + def check_document_end(self): + + # DOCUMENT-END: ^ '...' (' '|'\n') + if self.column == 0: + if self.prefix(3) == u'...' \ + and self.peek(3) in u'\0 \t\r\n\x85\u2028\u2029': + return True + + def check_block_entry(self): + + # BLOCK-ENTRY: '-' (' '|'\n') + return self.peek(1) in u'\0 \t\r\n\x85\u2028\u2029' + + def check_key(self): + + # KEY(flow context): '?' + if self.flow_level: + return True + + # KEY(block context): '?' (' '|'\n') + else: + return self.peek(1) in u'\0 \t\r\n\x85\u2028\u2029' + + def check_value(self): + + # VALUE(flow context): ':' + if self.flow_level: + return True + + # VALUE(block context): ':' (' '|'\n') + else: + return self.peek(1) in u'\0 \t\r\n\x85\u2028\u2029' + + def check_plain(self): + + # A plain scalar may start with any non-space character except: + # '-', '?', ':', ',', '[', ']', '{', '}', + # '#', '&', '*', '!', '|', '>', '\'', '\"', + # '%', '@', '`'. + # + # It may also start with + # '-', '?', ':' + # if it is followed by a non-space character. + # + # Note that we limit the last rule to the block context (except the + # '-' character) because we want the flow context to be space + # independent. + ch = self.peek() + return ch not in u'\0 \t\r\n\x85\u2028\u2029-?:,[]{}#&*!|>\'\"%@`' \ + or (self.peek(1) not in u'\0 \t\r\n\x85\u2028\u2029' + and (ch == u'-' or (not self.flow_level and ch in u'?:'))) + + # Scanners. + + def scan_to_next_token(self): + # We ignore spaces, line breaks and comments. + # If we find a line break in the block context, we set the flag + # `allow_simple_key` on. + # The byte order mark is stripped if it's the first character in the + # stream. We do not yet support BOM inside the stream as the + # specification requires. Any such mark will be considered as a part + # of the document. + # + # TODO: We need to make tab handling rules more sane. A good rule is + # Tabs cannot precede tokens + # BLOCK-SEQUENCE-START, BLOCK-MAPPING-START, BLOCK-END, + # KEY(block), VALUE(block), BLOCK-ENTRY + # So the checking code is + # if <TAB>: + # self.allow_simple_keys = False + # We also need to add the check for `allow_simple_keys == True` to + # `unwind_indent` before issuing BLOCK-END. + # Scanners for block, flow, and plain scalars need to be modified. + + if self.index == 0 and self.peek() == u'\uFEFF': + self.forward() + found = False + while not found: + while self.peek() == u' ': + self.forward() + if self.peek() == u'#': + while self.peek() not in u'\0\r\n\x85\u2028\u2029': + self.forward() + if self.scan_line_break(): + if not self.flow_level: + self.allow_simple_key = True + else: + found = True + + def scan_directive(self): + # See the specification for details. + start_mark = self.get_mark() + self.forward() + name = self.scan_directive_name(start_mark) + value = None + if name == u'YAML': + value = self.scan_yaml_directive_value(start_mark) + end_mark = self.get_mark() + elif name == u'TAG': + value = self.scan_tag_directive_value(start_mark) + end_mark = self.get_mark() + else: + end_mark = self.get_mark() + while self.peek() not in u'\0\r\n\x85\u2028\u2029': + self.forward() + self.scan_directive_ignored_line(start_mark) + return DirectiveToken(name, value, start_mark, end_mark) + + def scan_directive_name(self, start_mark): + # See the specification for details. + length = 0 + ch = self.peek(length) + while u'0' <= ch <= u'9' or u'A' <= ch <= u'Z' or u'a' <= ch <= u'z' \ + or ch in u'-_': + length += 1 + ch = self.peek(length) + if not length: + raise ScannerError("while scanning a directive", start_mark, + "expected alphabetic or numeric character, but found %r" + % ch.encode('utf-8'), self.get_mark()) + value = self.prefix(length) + self.forward(length) + ch = self.peek() + if ch not in u'\0 \r\n\x85\u2028\u2029': + raise ScannerError("while scanning a directive", start_mark, + "expected alphabetic or numeric character, but found %r" + % ch.encode('utf-8'), self.get_mark()) + return value + + def scan_yaml_directive_value(self, start_mark): + # See the specification for details. + while self.peek() == u' ': + self.forward() + major = self.scan_yaml_directive_number(start_mark) + if self.peek() != '.': + raise ScannerError("while scanning a directive", start_mark, + "expected a digit or '.', but found %r" + % self.peek().encode('utf-8'), + self.get_mark()) + self.forward() + minor = self.scan_yaml_directive_number(start_mark) + if self.peek() not in u'\0 \r\n\x85\u2028\u2029': + raise ScannerError("while scanning a directive", start_mark, + "expected a digit or ' ', but found %r" + % self.peek().encode('utf-8'), + self.get_mark()) + return (major, minor) + + def scan_yaml_directive_number(self, start_mark): + # See the specification for details. + ch = self.peek() + if not (u'0' <= ch <= u'9'): + raise ScannerError("while scanning a directive", start_mark, + "expected a digit, but found %r" % ch.encode('utf-8'), + self.get_mark()) + length = 0 + while u'0' <= self.peek(length) <= u'9': + length += 1 + value = int(self.prefix(length)) + self.forward(length) + return value + + def scan_tag_directive_value(self, start_mark): + # See the specification for details. + while self.peek() == u' ': + self.forward() + handle = self.scan_tag_directive_handle(start_mark) + while self.peek() == u' ': + self.forward() + prefix = self.scan_tag_directive_prefix(start_mark) + return (handle, prefix) + + def scan_tag_directive_handle(self, start_mark): + # See the specification for details. + value = self.scan_tag_handle('directive', start_mark) + ch = self.peek() + if ch != u' ': + raise ScannerError("while scanning a directive", start_mark, + "expected ' ', but found %r" % ch.encode('utf-8'), + self.get_mark()) + return value + + def scan_tag_directive_prefix(self, start_mark): + # See the specification for details. + value = self.scan_tag_uri('directive', start_mark) + ch = self.peek() + if ch not in u'\0 \r\n\x85\u2028\u2029': + raise ScannerError("while scanning a directive", start_mark, + "expected ' ', but found %r" % ch.encode('utf-8'), + self.get_mark()) + return value + + def scan_directive_ignored_line(self, start_mark): + # See the specification for details. + while self.peek() == u' ': + self.forward() + if self.peek() == u'#': + while self.peek() not in u'\0\r\n\x85\u2028\u2029': + self.forward() + ch = self.peek() + if ch not in u'\0\r\n\x85\u2028\u2029': + raise ScannerError("while scanning a directive", start_mark, + "expected a comment or a line break, but found %r" + % ch.encode('utf-8'), self.get_mark()) + self.scan_line_break() + + def scan_anchor(self, TokenClass): + # The specification does not restrict characters for anchors and + # aliases. This may lead to problems, for instance, the document: + # [ *alias, value ] + # can be interpteted in two ways, as + # [ "value" ] + # and + # [ *alias , "value" ] + # Therefore we restrict aliases to numbers and ASCII letters. + start_mark = self.get_mark() + indicator = self.peek() + if indicator == u'*': + name = 'alias' + else: + name = 'anchor' + self.forward() + length = 0 + ch = self.peek(length) + while u'0' <= ch <= u'9' or u'A' <= ch <= u'Z' or u'a' <= ch <= u'z' \ + or ch in u'-_': + length += 1 + ch = self.peek(length) + if not length: + raise ScannerError("while scanning an %s" % name, start_mark, + "expected alphabetic or numeric character, but found %r" + % ch.encode('utf-8'), self.get_mark()) + value = self.prefix(length) + self.forward(length) + ch = self.peek() + if ch not in u'\0 \t\r\n\x85\u2028\u2029?:,]}%@`': + raise ScannerError("while scanning an %s" % name, start_mark, + "expected alphabetic or numeric character, but found %r" + % ch.encode('utf-8'), self.get_mark()) + end_mark = self.get_mark() + return TokenClass(value, start_mark, end_mark) + + def scan_tag(self): + # See the specification for details. + start_mark = self.get_mark() + ch = self.peek(1) + if ch == u'<': + handle = None + self.forward(2) + suffix = self.scan_tag_uri('tag', start_mark) + if self.peek() != u'>': + raise ScannerError("while parsing a tag", start_mark, + "expected '>', but found %r" % self.peek().encode('utf-8'), + self.get_mark()) + self.forward() + elif ch in u'\0 \t\r\n\x85\u2028\u2029': + handle = None + suffix = u'!' + self.forward() + else: + length = 1 + use_handle = False + while ch not in u'\0 \r\n\x85\u2028\u2029': + if ch == u'!': + use_handle = True + break + length += 1 + ch = self.peek(length) + handle = u'!' + if use_handle: + handle = self.scan_tag_handle('tag', start_mark) + else: + handle = u'!' + self.forward() + suffix = self.scan_tag_uri('tag', start_mark) + ch = self.peek() + if ch not in u'\0 \r\n\x85\u2028\u2029': + raise ScannerError("while scanning a tag", start_mark, + "expected ' ', but found %r" % ch.encode('utf-8'), + self.get_mark()) + value = (handle, suffix) + end_mark = self.get_mark() + return TagToken(value, start_mark, end_mark) + + def scan_block_scalar(self, style): + # See the specification for details. + + if style == '>': + folded = True + else: + folded = False + + chunks = [] + start_mark = self.get_mark() + + # Scan the header. + self.forward() + chomping, increment = self.scan_block_scalar_indicators(start_mark) + self.scan_block_scalar_ignored_line(start_mark) + + # Determine the indentation level and go to the first non-empty line. + min_indent = self.indent+1 + if min_indent < 1: + min_indent = 1 + if increment is None: + breaks, max_indent, end_mark = self.scan_block_scalar_indentation() + indent = max(min_indent, max_indent) + else: + indent = min_indent+increment-1 + breaks, end_mark = self.scan_block_scalar_breaks(indent) + line_break = u'' + + # Scan the inner part of the block scalar. + while self.column == indent and self.peek() != u'\0': + chunks.extend(breaks) + leading_non_space = self.peek() not in u' \t' + length = 0 + while self.peek(length) not in u'\0\r\n\x85\u2028\u2029': + length += 1 + chunks.append(self.prefix(length)) + self.forward(length) + line_break = self.scan_line_break() + breaks, end_mark = self.scan_block_scalar_breaks(indent) + if self.column == indent and self.peek() != u'\0': + + # Unfortunately, folding rules are ambiguous. + # + # This is the folding according to the specification: + + if folded and line_break == u'\n' \ + and leading_non_space and self.peek() not in u' \t': + if not breaks: + chunks.append(u' ') + else: + chunks.append(line_break) + + # This is Clark Evans's interpretation (also in the spec + # examples): + # + #if folded and line_break == u'\n': + # if not breaks: + # if self.peek() not in ' \t': + # chunks.append(u' ') + # else: + # chunks.append(line_break) + #else: + # chunks.append(line_break) + else: + break + + # Chomp the tail. + if chomping is not False: + chunks.append(line_break) + if chomping is True: + chunks.extend(breaks) + + # We are done. + return ScalarToken(u''.join(chunks), False, start_mark, end_mark, + style) + + def scan_block_scalar_indicators(self, start_mark): + # See the specification for details. + chomping = None + increment = None + ch = self.peek() + if ch in u'+-': + if ch == '+': + chomping = True + else: + chomping = False + self.forward() + ch = self.peek() + if ch in u'0123456789': + increment = int(ch) + if increment == 0: + raise ScannerError("while scanning a block scalar", start_mark, + "expected indentation indicator in the range 1-9, but found 0", + self.get_mark()) + self.forward() + elif ch in u'0123456789': + increment = int(ch) + if increment == 0: + raise ScannerError("while scanning a block scalar", start_mark, + "expected indentation indicator in the range 1-9, but found 0", + self.get_mark()) + self.forward() + ch = self.peek() + if ch in u'+-': + if ch == '+': + chomping = True + else: + chomping = False + self.forward() + ch = self.peek() + if ch not in u'\0 \r\n\x85\u2028\u2029': + raise ScannerError("while scanning a block scalar", start_mark, + "expected chomping or indentation indicators, but found %r" + % ch.encode('utf-8'), self.get_mark()) + return chomping, increment + + def scan_block_scalar_ignored_line(self, start_mark): + # See the specification for details. + while self.peek() == u' ': + self.forward() + if self.peek() == u'#': + while self.peek() not in u'\0\r\n\x85\u2028\u2029': + self.forward() + ch = self.peek() + if ch not in u'\0\r\n\x85\u2028\u2029': + raise ScannerError("while scanning a block scalar", start_mark, + "expected a comment or a line break, but found %r" + % ch.encode('utf-8'), self.get_mark()) + self.scan_line_break() + + def scan_block_scalar_indentation(self): + # See the specification for details. + chunks = [] + max_indent = 0 + end_mark = self.get_mark() + while self.peek() in u' \r\n\x85\u2028\u2029': + if self.peek() != u' ': + chunks.append(self.scan_line_break()) + end_mark = self.get_mark() + else: + self.forward() + if self.column > max_indent: + max_indent = self.column + return chunks, max_indent, end_mark + + def scan_block_scalar_breaks(self, indent): + # See the specification for details. + chunks = [] + end_mark = self.get_mark() + while self.column < indent and self.peek() == u' ': + self.forward() + while self.peek() in u'\r\n\x85\u2028\u2029': + chunks.append(self.scan_line_break()) + end_mark = self.get_mark() + while self.column < indent and self.peek() == u' ': + self.forward() + return chunks, end_mark + + def scan_flow_scalar(self, style): + # See the specification for details. + # Note that we loose indentation rules for quoted scalars. Quoted + # scalars don't need to adhere indentation because " and ' clearly + # mark the beginning and the end of them. Therefore we are less + # restrictive then the specification requires. We only need to check + # that document separators are not included in scalars. + if style == '"': + double = True + else: + double = False + chunks = [] + start_mark = self.get_mark() + quote = self.peek() + self.forward() + chunks.extend(self.scan_flow_scalar_non_spaces(double, start_mark)) + while self.peek() != quote: + chunks.extend(self.scan_flow_scalar_spaces(double, start_mark)) + chunks.extend(self.scan_flow_scalar_non_spaces(double, start_mark)) + self.forward() + end_mark = self.get_mark() + return ScalarToken(u''.join(chunks), False, start_mark, end_mark, + style) + + ESCAPE_REPLACEMENTS = { + u'0': u'\0', + u'a': u'\x07', + u'b': u'\x08', + u't': u'\x09', + u'\t': u'\x09', + u'n': u'\x0A', + u'v': u'\x0B', + u'f': u'\x0C', + u'r': u'\x0D', + u'e': u'\x1B', + u' ': u'\x20', + u'\"': u'\"', + u'\\': u'\\', + u'N': u'\x85', + u'_': u'\xA0', + u'L': u'\u2028', + u'P': u'\u2029', + } + + ESCAPE_CODES = { + u'x': 2, + u'u': 4, + u'U': 8, + } + + def scan_flow_scalar_non_spaces(self, double, start_mark): + # See the specification for details. + chunks = [] + while True: + length = 0 + while self.peek(length) not in u'\'\"\\\0 \t\r\n\x85\u2028\u2029': + length += 1 + if length: + chunks.append(self.prefix(length)) + self.forward(length) + ch = self.peek() + if not double and ch == u'\'' and self.peek(1) == u'\'': + chunks.append(u'\'') + self.forward(2) + elif (double and ch == u'\'') or (not double and ch in u'\"\\'): + chunks.append(ch) + self.forward() + elif double and ch == u'\\': + self.forward() + ch = self.peek() + if ch in self.ESCAPE_REPLACEMENTS: + chunks.append(self.ESCAPE_REPLACEMENTS[ch]) + self.forward() + elif ch in self.ESCAPE_CODES: + length = self.ESCAPE_CODES[ch] + self.forward() + for k in range(length): + if self.peek(k) not in u'0123456789ABCDEFabcdef': + raise ScannerError("while scanning a double-quoted scalar", start_mark, + "expected escape sequence of %d hexdecimal numbers, but found %r" % + (length, self.peek(k).encode('utf-8')), self.get_mark()) + code = int(self.prefix(length), 16) + chunks.append(unichr(code)) + self.forward(length) + elif ch in u'\r\n\x85\u2028\u2029': + self.scan_line_break() + chunks.extend(self.scan_flow_scalar_breaks(double, start_mark)) + else: + raise ScannerError("while scanning a double-quoted scalar", start_mark, + "found unknown escape character %r" % ch.encode('utf-8'), self.get_mark()) + else: + return chunks + + def scan_flow_scalar_spaces(self, double, start_mark): + # See the specification for details. + chunks = [] + length = 0 + while self.peek(length) in u' \t': + length += 1 + whitespaces = self.prefix(length) + self.forward(length) + ch = self.peek() + if ch == u'\0': + raise ScannerError("while scanning a quoted scalar", start_mark, + "found unexpected end of stream", self.get_mark()) + elif ch in u'\r\n\x85\u2028\u2029': + line_break = self.scan_line_break() + breaks = self.scan_flow_scalar_breaks(double, start_mark) + if line_break != u'\n': + chunks.append(line_break) + elif not breaks: + chunks.append(u' ') + chunks.extend(breaks) + else: + chunks.append(whitespaces) + return chunks + + def scan_flow_scalar_breaks(self, double, start_mark): + # See the specification for details. + chunks = [] + while True: + # Instead of checking indentation, we check for document + # separators. + prefix = self.prefix(3) + if (prefix == u'---' or prefix == u'...') \ + and self.peek(3) in u'\0 \t\r\n\x85\u2028\u2029': + raise ScannerError("while scanning a quoted scalar", start_mark, + "found unexpected document separator", self.get_mark()) + while self.peek() in u' \t': + self.forward() + if self.peek() in u'\r\n\x85\u2028\u2029': + chunks.append(self.scan_line_break()) + else: + return chunks + + def scan_plain(self): + # See the specification for details. + # We add an additional restriction for the flow context: + # plain scalars in the flow context cannot contain ',', ':' and '?'. + # We also keep track of the `allow_simple_key` flag here. + # Indentation rules are loosed for the flow context. + chunks = [] + start_mark = self.get_mark() + end_mark = start_mark + indent = self.indent+1 + # We allow zero indentation for scalars, but then we need to check for + # document separators at the beginning of the line. + #if indent == 0: + # indent = 1 + spaces = [] + while True: + length = 0 + if self.peek() == u'#': + break + while True: + ch = self.peek(length) + if ch in u'\0 \t\r\n\x85\u2028\u2029' \ + or (not self.flow_level and ch == u':' and + self.peek(length+1) in u'\0 \t\r\n\x85\u2028\u2029') \ + or (self.flow_level and ch in u',:?[]{}'): + break + length += 1 + # It's not clear what we should do with ':' in the flow context. + if (self.flow_level and ch == u':' + and self.peek(length+1) not in u'\0 \t\r\n\x85\u2028\u2029,[]{}'): + self.forward(length) + raise ScannerError("while scanning a plain scalar", start_mark, + "found unexpected ':'", self.get_mark(), + "Please check http://pyyaml.org/wiki/YAMLColonInFlowContext for details.") + if length == 0: + break + self.allow_simple_key = False + chunks.extend(spaces) + chunks.append(self.prefix(length)) + self.forward(length) + end_mark = self.get_mark() + spaces = self.scan_plain_spaces(indent, start_mark) + if not spaces or self.peek() == u'#' \ + or (not self.flow_level and self.column < indent): + break + return ScalarToken(u''.join(chunks), True, start_mark, end_mark) + + def scan_plain_spaces(self, indent, start_mark): + # See the specification for details. + # The specification is really confusing about tabs in plain scalars. + # We just forbid them completely. Do not use tabs in YAML! + chunks = [] + length = 0 + while self.peek(length) in u' ': + length += 1 + whitespaces = self.prefix(length) + self.forward(length) + ch = self.peek() + if ch in u'\r\n\x85\u2028\u2029': + line_break = self.scan_line_break() + self.allow_simple_key = True + prefix = self.prefix(3) + if (prefix == u'---' or prefix == u'...') \ + and self.peek(3) in u'\0 \t\r\n\x85\u2028\u2029': + return + breaks = [] + while self.peek() in u' \r\n\x85\u2028\u2029': + if self.peek() == ' ': + self.forward() + else: + breaks.append(self.scan_line_break()) + prefix = self.prefix(3) + if (prefix == u'---' or prefix == u'...') \ + and self.peek(3) in u'\0 \t\r\n\x85\u2028\u2029': + return + if line_break != u'\n': + chunks.append(line_break) + elif not breaks: + chunks.append(u' ') + chunks.extend(breaks) + elif whitespaces: + chunks.append(whitespaces) + return chunks + + def scan_tag_handle(self, name, start_mark): + # See the specification for details. + # For some strange reasons, the specification does not allow '_' in + # tag handles. I have allowed it anyway. + ch = self.peek() + if ch != u'!': + raise ScannerError("while scanning a %s" % name, start_mark, + "expected '!', but found %r" % ch.encode('utf-8'), + self.get_mark()) + length = 1 + ch = self.peek(length) + if ch != u' ': + while u'0' <= ch <= u'9' or u'A' <= ch <= u'Z' or u'a' <= ch <= u'z' \ + or ch in u'-_': + length += 1 + ch = self.peek(length) + if ch != u'!': + self.forward(length) + raise ScannerError("while scanning a %s" % name, start_mark, + "expected '!', but found %r" % ch.encode('utf-8'), + self.get_mark()) + length += 1 + value = self.prefix(length) + self.forward(length) + return value + + def scan_tag_uri(self, name, start_mark): + # See the specification for details. + # Note: we do not check if URI is well-formed. + chunks = [] + length = 0 + ch = self.peek(length) + while u'0' <= ch <= u'9' or u'A' <= ch <= u'Z' or u'a' <= ch <= u'z' \ + or ch in u'-;/?:@&=+$,_.!~*\'()[]%': + if ch == u'%': + chunks.append(self.prefix(length)) + self.forward(length) + length = 0 + chunks.append(self.scan_uri_escapes(name, start_mark)) + else: + length += 1 + ch = self.peek(length) + if length: + chunks.append(self.prefix(length)) + self.forward(length) + length = 0 + if not chunks: + raise ScannerError("while parsing a %s" % name, start_mark, + "expected URI, but found %r" % ch.encode('utf-8'), + self.get_mark()) + return u''.join(chunks) + + def scan_uri_escapes(self, name, start_mark): + # See the specification for details. + bytes = [] + mark = self.get_mark() + while self.peek() == u'%': + self.forward() + for k in range(2): + if self.peek(k) not in u'0123456789ABCDEFabcdef': + raise ScannerError("while scanning a %s" % name, start_mark, + "expected URI escape sequence of 2 hexdecimal numbers, but found %r" % + (self.peek(k).encode('utf-8')), self.get_mark()) + bytes.append(chr(int(self.prefix(2), 16))) + self.forward(2) + try: + value = unicode(''.join(bytes), 'utf-8') + except UnicodeDecodeError, exc: + raise ScannerError("while scanning a %s" % name, start_mark, str(exc), mark) + return value + + def scan_line_break(self): + # Transforms: + # '\r\n' : '\n' + # '\r' : '\n' + # '\n' : '\n' + # '\x85' : '\n' + # '\u2028' : '\u2028' + # '\u2029 : '\u2029' + # default : '' + ch = self.peek() + if ch in u'\r\n\x85': + if self.prefix(2) == u'\r\n': + self.forward(2) + else: + self.forward() + return u'\n' + elif ch in u'\u2028\u2029': + self.forward() + return ch + return u'' + +#try: +# import psyco +# psyco.bind(Scanner) +#except ImportError: +# pass + diff --git a/python.d/python_modules/pyyaml2/serializer.py b/python.d/python_modules/pyyaml2/serializer.py new file mode 100644 index 000000000..0bf1e96dc --- /dev/null +++ b/python.d/python_modules/pyyaml2/serializer.py @@ -0,0 +1,111 @@ + +__all__ = ['Serializer', 'SerializerError'] + +from error import YAMLError +from events import * +from nodes import * + +class SerializerError(YAMLError): + pass + +class Serializer(object): + + ANCHOR_TEMPLATE = u'id%03d' + + def __init__(self, encoding=None, + explicit_start=None, explicit_end=None, version=None, tags=None): + self.use_encoding = encoding + self.use_explicit_start = explicit_start + self.use_explicit_end = explicit_end + self.use_version = version + self.use_tags = tags + self.serialized_nodes = {} + self.anchors = {} + self.last_anchor_id = 0 + self.closed = None + + def open(self): + if self.closed is None: + self.emit(StreamStartEvent(encoding=self.use_encoding)) + self.closed = False + elif self.closed: + raise SerializerError("serializer is closed") + else: + raise SerializerError("serializer is already opened") + + def close(self): + if self.closed is None: + raise SerializerError("serializer is not opened") + elif not self.closed: + self.emit(StreamEndEvent()) + self.closed = True + + #def __del__(self): + # self.close() + + def serialize(self, node): + if self.closed is None: + raise SerializerError("serializer is not opened") + elif self.closed: + raise SerializerError("serializer is closed") + self.emit(DocumentStartEvent(explicit=self.use_explicit_start, + version=self.use_version, tags=self.use_tags)) + self.anchor_node(node) + self.serialize_node(node, None, None) + self.emit(DocumentEndEvent(explicit=self.use_explicit_end)) + self.serialized_nodes = {} + self.anchors = {} + self.last_anchor_id = 0 + + def anchor_node(self, node): + if node in self.anchors: + if self.anchors[node] is None: + self.anchors[node] = self.generate_anchor(node) + else: + self.anchors[node] = None + if isinstance(node, SequenceNode): + for item in node.value: + self.anchor_node(item) + elif isinstance(node, MappingNode): + for key, value in node.value: + self.anchor_node(key) + self.anchor_node(value) + + def generate_anchor(self, node): + self.last_anchor_id += 1 + return self.ANCHOR_TEMPLATE % self.last_anchor_id + + def serialize_node(self, node, parent, index): + alias = self.anchors[node] + if node in self.serialized_nodes: + self.emit(AliasEvent(alias)) + else: + self.serialized_nodes[node] = True + self.descend_resolver(parent, index) + if isinstance(node, ScalarNode): + detected_tag = self.resolve(ScalarNode, node.value, (True, False)) + default_tag = self.resolve(ScalarNode, node.value, (False, True)) + implicit = (node.tag == detected_tag), (node.tag == default_tag) + self.emit(ScalarEvent(alias, node.tag, implicit, node.value, + style=node.style)) + elif isinstance(node, SequenceNode): + implicit = (node.tag + == self.resolve(SequenceNode, node.value, True)) + self.emit(SequenceStartEvent(alias, node.tag, implicit, + flow_style=node.flow_style)) + index = 0 + for item in node.value: + self.serialize_node(item, node, index) + index += 1 + self.emit(SequenceEndEvent()) + elif isinstance(node, MappingNode): + implicit = (node.tag + == self.resolve(MappingNode, node.value, True)) + self.emit(MappingStartEvent(alias, node.tag, implicit, + flow_style=node.flow_style)) + for key, value in node.value: + self.serialize_node(key, node, None) + self.serialize_node(value, node, key) + self.emit(MappingEndEvent()) + self.ascend_resolver() + diff --git a/python.d/python_modules/pyyaml2/tokens.py b/python.d/python_modules/pyyaml2/tokens.py new file mode 100644 index 000000000..4d0b48a39 --- /dev/null +++ b/python.d/python_modules/pyyaml2/tokens.py @@ -0,0 +1,104 @@ + +class Token(object): + def __init__(self, start_mark, end_mark): + self.start_mark = start_mark + self.end_mark = end_mark + def __repr__(self): + attributes = [key for key in self.__dict__ + if not key.endswith('_mark')] + attributes.sort() + arguments = ', '.join(['%s=%r' % (key, getattr(self, key)) + for key in attributes]) + return '%s(%s)' % (self.__class__.__name__, arguments) + +#class BOMToken(Token): +# id = '<byte order mark>' + +class DirectiveToken(Token): + id = '<directive>' + def __init__(self, name, value, start_mark, end_mark): + self.name = name + self.value = value + self.start_mark = start_mark + self.end_mark = end_mark + +class DocumentStartToken(Token): + id = '<document start>' + +class DocumentEndToken(Token): + id = '<document end>' + +class StreamStartToken(Token): + id = '<stream start>' + def __init__(self, start_mark=None, end_mark=None, + encoding=None): + self.start_mark = start_mark + self.end_mark = end_mark + self.encoding = encoding + +class StreamEndToken(Token): + id = '<stream end>' + +class BlockSequenceStartToken(Token): + id = '<block sequence start>' + +class BlockMappingStartToken(Token): + id = '<block mapping start>' + +class BlockEndToken(Token): + id = '<block end>' + +class FlowSequenceStartToken(Token): + id = '[' + +class FlowMappingStartToken(Token): + id = '{' + +class FlowSequenceEndToken(Token): + id = ']' + +class FlowMappingEndToken(Token): + id = '}' + +class KeyToken(Token): + id = '?' + +class ValueToken(Token): + id = ':' + +class BlockEntryToken(Token): + id = '-' + +class FlowEntryToken(Token): + id = ',' + +class AliasToken(Token): + id = '<alias>' + def __init__(self, value, start_mark, end_mark): + self.value = value + self.start_mark = start_mark + self.end_mark = end_mark + +class AnchorToken(Token): + id = '<anchor>' + def __init__(self, value, start_mark, end_mark): + self.value = value + self.start_mark = start_mark + self.end_mark = end_mark + +class TagToken(Token): + id = '<tag>' + def __init__(self, value, start_mark, end_mark): + self.value = value + self.start_mark = start_mark + self.end_mark = end_mark + +class ScalarToken(Token): + id = '<scalar>' + def __init__(self, value, plain, start_mark, end_mark, style=None): + self.value = value + self.plain = plain + self.start_mark = start_mark + self.end_mark = end_mark + self.style = style + diff --git a/python.d/python_modules/pyyaml3/__init__.py b/python.d/python_modules/pyyaml3/__init__.py new file mode 100644 index 000000000..a5e20f94d --- /dev/null +++ b/python.d/python_modules/pyyaml3/__init__.py @@ -0,0 +1,312 @@ + +from .error import * + +from .tokens import * +from .events import * +from .nodes import * + +from .loader import * +from .dumper import * + +__version__ = '3.11' +try: + from .cyaml import * + __with_libyaml__ = True +except ImportError: + __with_libyaml__ = False + +import io + +def scan(stream, Loader=Loader): + """ + Scan a YAML stream and produce scanning tokens. + """ + loader = Loader(stream) + try: + while loader.check_token(): + yield loader.get_token() + finally: + loader.dispose() + +def parse(stream, Loader=Loader): + """ + Parse a YAML stream and produce parsing events. + """ + loader = Loader(stream) + try: + while loader.check_event(): + yield loader.get_event() + finally: + loader.dispose() + +def compose(stream, Loader=Loader): + """ + Parse the first YAML document in a stream + and produce the corresponding representation tree. + """ + loader = Loader(stream) + try: + return loader.get_single_node() + finally: + loader.dispose() + +def compose_all(stream, Loader=Loader): + """ + Parse all YAML documents in a stream + and produce corresponding representation trees. + """ + loader = Loader(stream) + try: + while loader.check_node(): + yield loader.get_node() + finally: + loader.dispose() + +def load(stream, Loader=Loader): + """ + Parse the first YAML document in a stream + and produce the corresponding Python object. + """ + loader = Loader(stream) + try: + return loader.get_single_data() + finally: + loader.dispose() + +def load_all(stream, Loader=Loader): + """ + Parse all YAML documents in a stream + and produce corresponding Python objects. + """ + loader = Loader(stream) + try: + while loader.check_data(): + yield loader.get_data() + finally: + loader.dispose() + +def safe_load(stream): + """ + Parse the first YAML document in a stream + and produce the corresponding Python object. + Resolve only basic YAML tags. + """ + return load(stream, SafeLoader) + +def safe_load_all(stream): + """ + Parse all YAML documents in a stream + and produce corresponding Python objects. + Resolve only basic YAML tags. + """ + return load_all(stream, SafeLoader) + +def emit(events, stream=None, Dumper=Dumper, + canonical=None, indent=None, width=None, + allow_unicode=None, line_break=None): + """ + Emit YAML parsing events into a stream. + If stream is None, return the produced string instead. + """ + getvalue = None + if stream is None: + stream = io.StringIO() + getvalue = stream.getvalue + dumper = Dumper(stream, canonical=canonical, indent=indent, width=width, + allow_unicode=allow_unicode, line_break=line_break) + try: + for event in events: + dumper.emit(event) + finally: + dumper.dispose() + if getvalue: + return getvalue() + +def serialize_all(nodes, stream=None, Dumper=Dumper, + canonical=None, indent=None, width=None, + allow_unicode=None, line_break=None, + encoding=None, explicit_start=None, explicit_end=None, + version=None, tags=None): + """ + Serialize a sequence of representation trees into a YAML stream. + If stream is None, return the produced string instead. + """ + getvalue = None + if stream is None: + if encoding is None: + stream = io.StringIO() + else: + stream = io.BytesIO() + getvalue = stream.getvalue + dumper = Dumper(stream, canonical=canonical, indent=indent, width=width, + allow_unicode=allow_unicode, line_break=line_break, + encoding=encoding, version=version, tags=tags, + explicit_start=explicit_start, explicit_end=explicit_end) + try: + dumper.open() + for node in nodes: + dumper.serialize(node) + dumper.close() + finally: + dumper.dispose() + if getvalue: + return getvalue() + +def serialize(node, stream=None, Dumper=Dumper, **kwds): + """ + Serialize a representation tree into a YAML stream. + If stream is None, return the produced string instead. + """ + return serialize_all([node], stream, Dumper=Dumper, **kwds) + +def dump_all(documents, stream=None, Dumper=Dumper, + default_style=None, default_flow_style=None, + canonical=None, indent=None, width=None, + allow_unicode=None, line_break=None, + encoding=None, explicit_start=None, explicit_end=None, + version=None, tags=None): + """ + Serialize a sequence of Python objects into a YAML stream. + If stream is None, return the produced string instead. + """ + getvalue = None + if stream is None: + if encoding is None: + stream = io.StringIO() + else: + stream = io.BytesIO() + getvalue = stream.getvalue + dumper = Dumper(stream, default_style=default_style, + default_flow_style=default_flow_style, + canonical=canonical, indent=indent, width=width, + allow_unicode=allow_unicode, line_break=line_break, + encoding=encoding, version=version, tags=tags, + explicit_start=explicit_start, explicit_end=explicit_end) + try: + dumper.open() + for data in documents: + dumper.represent(data) + dumper.close() + finally: + dumper.dispose() + if getvalue: + return getvalue() + +def dump(data, stream=None, Dumper=Dumper, **kwds): + """ + Serialize a Python object into a YAML stream. + If stream is None, return the produced string instead. + """ + return dump_all([data], stream, Dumper=Dumper, **kwds) + +def safe_dump_all(documents, stream=None, **kwds): + """ + Serialize a sequence of Python objects into a YAML stream. + Produce only basic YAML tags. + If stream is None, return the produced string instead. + """ + return dump_all(documents, stream, Dumper=SafeDumper, **kwds) + +def safe_dump(data, stream=None, **kwds): + """ + Serialize a Python object into a YAML stream. + Produce only basic YAML tags. + If stream is None, return the produced string instead. + """ + return dump_all([data], stream, Dumper=SafeDumper, **kwds) + +def add_implicit_resolver(tag, regexp, first=None, + Loader=Loader, Dumper=Dumper): + """ + Add an implicit scalar detector. + If an implicit scalar value matches the given regexp, + the corresponding tag is assigned to the scalar. + first is a sequence of possible initial characters or None. + """ + Loader.add_implicit_resolver(tag, regexp, first) + Dumper.add_implicit_resolver(tag, regexp, first) + +def add_path_resolver(tag, path, kind=None, Loader=Loader, Dumper=Dumper): + """ + Add a path based resolver for the given tag. + A path is a list of keys that forms a path + to a node in the representation tree. + Keys can be string values, integers, or None. + """ + Loader.add_path_resolver(tag, path, kind) + Dumper.add_path_resolver(tag, path, kind) + +def add_constructor(tag, constructor, Loader=Loader): + """ + Add a constructor for the given tag. + Constructor is a function that accepts a Loader instance + and a node object and produces the corresponding Python object. + """ + Loader.add_constructor(tag, constructor) + +def add_multi_constructor(tag_prefix, multi_constructor, Loader=Loader): + """ + Add a multi-constructor for the given tag prefix. + Multi-constructor is called for a node if its tag starts with tag_prefix. + Multi-constructor accepts a Loader instance, a tag suffix, + and a node object and produces the corresponding Python object. + """ + Loader.add_multi_constructor(tag_prefix, multi_constructor) + +def add_representer(data_type, representer, Dumper=Dumper): + """ + Add a representer for the given type. + Representer is a function accepting a Dumper instance + and an instance of the given data type + and producing the corresponding representation node. + """ + Dumper.add_representer(data_type, representer) + +def add_multi_representer(data_type, multi_representer, Dumper=Dumper): + """ + Add a representer for the given type. + Multi-representer is a function accepting a Dumper instance + and an instance of the given data type or subtype + and producing the corresponding representation node. + """ + Dumper.add_multi_representer(data_type, multi_representer) + +class YAMLObjectMetaclass(type): + """ + The metaclass for YAMLObject. + """ + def __init__(cls, name, bases, kwds): + super(YAMLObjectMetaclass, cls).__init__(name, bases, kwds) + if 'yaml_tag' in kwds and kwds['yaml_tag'] is not None: + cls.yaml_loader.add_constructor(cls.yaml_tag, cls.from_yaml) + cls.yaml_dumper.add_representer(cls, cls.to_yaml) + +class YAMLObject(metaclass=YAMLObjectMetaclass): + """ + An object that can dump itself to a YAML stream + and load itself from a YAML stream. + """ + + __slots__ = () # no direct instantiation, so allow immutable subclasses + + yaml_loader = Loader + yaml_dumper = Dumper + + yaml_tag = None + yaml_flow_style = None + + @classmethod + def from_yaml(cls, loader, node): + """ + Convert a representation node to a Python object. + """ + return loader.construct_yaml_object(node, cls) + + @classmethod + def to_yaml(cls, dumper, data): + """ + Convert a Python object to a representation node. + """ + return dumper.represent_yaml_object(cls.yaml_tag, data, cls, + flow_style=cls.yaml_flow_style) + diff --git a/python.d/python_modules/pyyaml3/composer.py b/python.d/python_modules/pyyaml3/composer.py new file mode 100644 index 000000000..d5c6a7acd --- /dev/null +++ b/python.d/python_modules/pyyaml3/composer.py @@ -0,0 +1,139 @@ + +__all__ = ['Composer', 'ComposerError'] + +from .error import MarkedYAMLError +from .events import * +from .nodes import * + +class ComposerError(MarkedYAMLError): + pass + +class Composer: + + def __init__(self): + self.anchors = {} + + def check_node(self): + # Drop the STREAM-START event. + if self.check_event(StreamStartEvent): + self.get_event() + + # If there are more documents available? + return not self.check_event(StreamEndEvent) + + def get_node(self): + # Get the root node of the next document. + if not self.check_event(StreamEndEvent): + return self.compose_document() + + def get_single_node(self): + # Drop the STREAM-START event. + self.get_event() + + # Compose a document if the stream is not empty. + document = None + if not self.check_event(StreamEndEvent): + document = self.compose_document() + + # Ensure that the stream contains no more documents. + if not self.check_event(StreamEndEvent): + event = self.get_event() + raise ComposerError("expected a single document in the stream", + document.start_mark, "but found another document", + event.start_mark) + + # Drop the STREAM-END event. + self.get_event() + + return document + + def compose_document(self): + # Drop the DOCUMENT-START event. + self.get_event() + + # Compose the root node. + node = self.compose_node(None, None) + + # Drop the DOCUMENT-END event. + self.get_event() + + self.anchors = {} + return node + + def compose_node(self, parent, index): + if self.check_event(AliasEvent): + event = self.get_event() + anchor = event.anchor + if anchor not in self.anchors: + raise ComposerError(None, None, "found undefined alias %r" + % anchor, event.start_mark) + return self.anchors[anchor] + event = self.peek_event() + anchor = event.anchor + if anchor is not None: + if anchor in self.anchors: + raise ComposerError("found duplicate anchor %r; first occurence" + % anchor, self.anchors[anchor].start_mark, + "second occurence", event.start_mark) + self.descend_resolver(parent, index) + if self.check_event(ScalarEvent): + node = self.compose_scalar_node(anchor) + elif self.check_event(SequenceStartEvent): + node = self.compose_sequence_node(anchor) + elif self.check_event(MappingStartEvent): + node = self.compose_mapping_node(anchor) + self.ascend_resolver() + return node + + def compose_scalar_node(self, anchor): + event = self.get_event() + tag = event.tag + if tag is None or tag == '!': + tag = self.resolve(ScalarNode, event.value, event.implicit) + node = ScalarNode(tag, event.value, + event.start_mark, event.end_mark, style=event.style) + if anchor is not None: + self.anchors[anchor] = node + return node + + def compose_sequence_node(self, anchor): + start_event = self.get_event() + tag = start_event.tag + if tag is None or tag == '!': + tag = self.resolve(SequenceNode, None, start_event.implicit) + node = SequenceNode(tag, [], + start_event.start_mark, None, + flow_style=start_event.flow_style) + if anchor is not None: + self.anchors[anchor] = node + index = 0 + while not self.check_event(SequenceEndEvent): + node.value.append(self.compose_node(node, index)) + index += 1 + end_event = self.get_event() + node.end_mark = end_event.end_mark + return node + + def compose_mapping_node(self, anchor): + start_event = self.get_event() + tag = start_event.tag + if tag is None or tag == '!': + tag = self.resolve(MappingNode, None, start_event.implicit) + node = MappingNode(tag, [], + start_event.start_mark, None, + flow_style=start_event.flow_style) + if anchor is not None: + self.anchors[anchor] = node + while not self.check_event(MappingEndEvent): + #key_event = self.peek_event() + item_key = self.compose_node(node, None) + #if item_key in node.value: + # raise ComposerError("while composing a mapping", start_event.start_mark, + # "found duplicate key", key_event.start_mark) + item_value = self.compose_node(node, item_key) + #node.value[item_key] = item_value + node.value.append((item_key, item_value)) + end_event = self.get_event() + node.end_mark = end_event.end_mark + return node + diff --git a/python.d/python_modules/pyyaml3/constructor.py b/python.d/python_modules/pyyaml3/constructor.py new file mode 100644 index 000000000..981543aeb --- /dev/null +++ b/python.d/python_modules/pyyaml3/constructor.py @@ -0,0 +1,686 @@ + +__all__ = ['BaseConstructor', 'SafeConstructor', 'Constructor', + 'ConstructorError'] + +from .error import * +from .nodes import * + +import collections, datetime, base64, binascii, re, sys, types + +class ConstructorError(MarkedYAMLError): + pass + +class BaseConstructor: + + yaml_constructors = {} + yaml_multi_constructors = {} + + def __init__(self): + self.constructed_objects = {} + self.recursive_objects = {} + self.state_generators = [] + self.deep_construct = False + + def check_data(self): + # If there are more documents available? + return self.check_node() + + def get_data(self): + # Construct and return the next document. + if self.check_node(): + return self.construct_document(self.get_node()) + + def get_single_data(self): + # Ensure that the stream contains a single document and construct it. + node = self.get_single_node() + if node is not None: + return self.construct_document(node) + return None + + def construct_document(self, node): + data = self.construct_object(node) + while self.state_generators: + state_generators = self.state_generators + self.state_generators = [] + for generator in state_generators: + for dummy in generator: + pass + self.constructed_objects = {} + self.recursive_objects = {} + self.deep_construct = False + return data + + def construct_object(self, node, deep=False): + if node in self.constructed_objects: + return self.constructed_objects[node] + if deep: + old_deep = self.deep_construct + self.deep_construct = True + if node in self.recursive_objects: + raise ConstructorError(None, None, + "found unconstructable recursive node", node.start_mark) + self.recursive_objects[node] = None + constructor = None + tag_suffix = None + if node.tag in self.yaml_constructors: + constructor = self.yaml_constructors[node.tag] + else: + for tag_prefix in self.yaml_multi_constructors: + if node.tag.startswith(tag_prefix): + tag_suffix = node.tag[len(tag_prefix):] + constructor = self.yaml_multi_constructors[tag_prefix] + break + else: + if None in self.yaml_multi_constructors: + tag_suffix = node.tag + constructor = self.yaml_multi_constructors[None] + elif None in self.yaml_constructors: + constructor = self.yaml_constructors[None] + elif isinstance(node, ScalarNode): + constructor = self.__class__.construct_scalar + elif isinstance(node, SequenceNode): + constructor = self.__class__.construct_sequence + elif isinstance(node, MappingNode): + constructor = self.__class__.construct_mapping + if tag_suffix is None: + data = constructor(self, node) + else: + data = constructor(self, tag_suffix, node) + if isinstance(data, types.GeneratorType): + generator = data + data = next(generator) + if self.deep_construct: + for dummy in generator: + pass + else: + self.state_generators.append(generator) + self.constructed_objects[node] = data + del self.recursive_objects[node] + if deep: + self.deep_construct = old_deep + return data + + def construct_scalar(self, node): + if not isinstance(node, ScalarNode): + raise ConstructorError(None, None, + "expected a scalar node, but found %s" % node.id, + node.start_mark) + return node.value + + def construct_sequence(self, node, deep=False): + if not isinstance(node, SequenceNode): + raise ConstructorError(None, None, + "expected a sequence node, but found %s" % node.id, + node.start_mark) + return [self.construct_object(child, deep=deep) + for child in node.value] + + def construct_mapping(self, node, deep=False): + if not isinstance(node, MappingNode): + raise ConstructorError(None, None, + "expected a mapping node, but found %s" % node.id, + node.start_mark) + mapping = {} + for key_node, value_node in node.value: + key = self.construct_object(key_node, deep=deep) + if not isinstance(key, collections.Hashable): + raise ConstructorError("while constructing a mapping", node.start_mark, + "found unhashable key", key_node.start_mark) + value = self.construct_object(value_node, deep=deep) + mapping[key] = value + return mapping + + def construct_pairs(self, node, deep=False): + if not isinstance(node, MappingNode): + raise ConstructorError(None, None, + "expected a mapping node, but found %s" % node.id, + node.start_mark) + pairs = [] + for key_node, value_node in node.value: + key = self.construct_object(key_node, deep=deep) + value = self.construct_object(value_node, deep=deep) + pairs.append((key, value)) + return pairs + + @classmethod + def add_constructor(cls, tag, constructor): + if not 'yaml_constructors' in cls.__dict__: + cls.yaml_constructors = cls.yaml_constructors.copy() + cls.yaml_constructors[tag] = constructor + + @classmethod + def add_multi_constructor(cls, tag_prefix, multi_constructor): + if not 'yaml_multi_constructors' in cls.__dict__: + cls.yaml_multi_constructors = cls.yaml_multi_constructors.copy() + cls.yaml_multi_constructors[tag_prefix] = multi_constructor + +class SafeConstructor(BaseConstructor): + + def construct_scalar(self, node): + if isinstance(node, MappingNode): + for key_node, value_node in node.value: + if key_node.tag == 'tag:yaml.org,2002:value': + return self.construct_scalar(value_node) + return super().construct_scalar(node) + + def flatten_mapping(self, node): + merge = [] + index = 0 + while index < len(node.value): + key_node, value_node = node.value[index] + if key_node.tag == 'tag:yaml.org,2002:merge': + del node.value[index] + if isinstance(value_node, MappingNode): + self.flatten_mapping(value_node) + merge.extend(value_node.value) + elif isinstance(value_node, SequenceNode): + submerge = [] + for subnode in value_node.value: + if not isinstance(subnode, MappingNode): + raise ConstructorError("while constructing a mapping", + node.start_mark, + "expected a mapping for merging, but found %s" + % subnode.id, subnode.start_mark) + self.flatten_mapping(subnode) + submerge.append(subnode.value) + submerge.reverse() + for value in submerge: + merge.extend(value) + else: + raise ConstructorError("while constructing a mapping", node.start_mark, + "expected a mapping or list of mappings for merging, but found %s" + % value_node.id, value_node.start_mark) + elif key_node.tag == 'tag:yaml.org,2002:value': + key_node.tag = 'tag:yaml.org,2002:str' + index += 1 + else: + index += 1 + if merge: + node.value = merge + node.value + + def construct_mapping(self, node, deep=False): + if isinstance(node, MappingNode): + self.flatten_mapping(node) + return super().construct_mapping(node, deep=deep) + + def construct_yaml_null(self, node): + self.construct_scalar(node) + return None + + bool_values = { + 'yes': True, + 'no': False, + 'true': True, + 'false': False, + 'on': True, + 'off': False, + } + + def construct_yaml_bool(self, node): + value = self.construct_scalar(node) + return self.bool_values[value.lower()] + + def construct_yaml_int(self, node): + value = self.construct_scalar(node) + value = value.replace('_', '') + sign = +1 + if value[0] == '-': + sign = -1 + if value[0] in '+-': + value = value[1:] + if value == '0': + return 0 + elif value.startswith('0b'): + return sign*int(value[2:], 2) + elif value.startswith('0x'): + return sign*int(value[2:], 16) + elif value[0] == '0': + return sign*int(value, 8) + elif ':' in value: + digits = [int(part) for part in value.split(':')] + digits.reverse() + base = 1 + value = 0 + for digit in digits: + value += digit*base + base *= 60 + return sign*value + else: + return sign*int(value) + + inf_value = 1e300 + while inf_value != inf_value*inf_value: + inf_value *= inf_value + nan_value = -inf_value/inf_value # Trying to make a quiet NaN (like C99). + + def construct_yaml_float(self, node): + value = self.construct_scalar(node) + value = value.replace('_', '').lower() + sign = +1 + if value[0] == '-': + sign = -1 + if value[0] in '+-': + value = value[1:] + if value == '.inf': + return sign*self.inf_value + elif value == '.nan': + return self.nan_value + elif ':' in value: + digits = [float(part) for part in value.split(':')] + digits.reverse() + base = 1 + value = 0.0 + for digit in digits: + value += digit*base + base *= 60 + return sign*value + else: + return sign*float(value) + + def construct_yaml_binary(self, node): + try: + value = self.construct_scalar(node).encode('ascii') + except UnicodeEncodeError as exc: + raise ConstructorError(None, None, + "failed to convert base64 data into ascii: %s" % exc, + node.start_mark) + try: + if hasattr(base64, 'decodebytes'): + return base64.decodebytes(value) + else: + return base64.decodestring(value) + except binascii.Error as exc: + raise ConstructorError(None, None, + "failed to decode base64 data: %s" % exc, node.start_mark) + + timestamp_regexp = re.compile( + r'''^(?P<year>[0-9][0-9][0-9][0-9]) + -(?P<month>[0-9][0-9]?) + -(?P<day>[0-9][0-9]?) + (?:(?:[Tt]|[ \t]+) + (?P<hour>[0-9][0-9]?) + :(?P<minute>[0-9][0-9]) + :(?P<second>[0-9][0-9]) + (?:\.(?P<fraction>[0-9]*))? + (?:[ \t]*(?P<tz>Z|(?P<tz_sign>[-+])(?P<tz_hour>[0-9][0-9]?) + (?::(?P<tz_minute>[0-9][0-9]))?))?)?$''', re.X) + + def construct_yaml_timestamp(self, node): + value = self.construct_scalar(node) + match = self.timestamp_regexp.match(node.value) + values = match.groupdict() + year = int(values['year']) + month = int(values['month']) + day = int(values['day']) + if not values['hour']: + return datetime.date(year, month, day) + hour = int(values['hour']) + minute = int(values['minute']) + second = int(values['second']) + fraction = 0 + if values['fraction']: + fraction = values['fraction'][:6] + while len(fraction) < 6: + fraction += '0' + fraction = int(fraction) + delta = None + if values['tz_sign']: + tz_hour = int(values['tz_hour']) + tz_minute = int(values['tz_minute'] or 0) + delta = datetime.timedelta(hours=tz_hour, minutes=tz_minute) + if values['tz_sign'] == '-': + delta = -delta + data = datetime.datetime(year, month, day, hour, minute, second, fraction) + if delta: + data -= delta + return data + + def construct_yaml_omap(self, node): + # Note: we do not check for duplicate keys, because it's too + # CPU-expensive. + omap = [] + yield omap + if not isinstance(node, SequenceNode): + raise ConstructorError("while constructing an ordered map", node.start_mark, + "expected a sequence, but found %s" % node.id, node.start_mark) + for subnode in node.value: + if not isinstance(subnode, MappingNode): + raise ConstructorError("while constructing an ordered map", node.start_mark, + "expected a mapping of length 1, but found %s" % subnode.id, + subnode.start_mark) + if len(subnode.value) != 1: + raise ConstructorError("while constructing an ordered map", node.start_mark, + "expected a single mapping item, but found %d items" % len(subnode.value), + subnode.start_mark) + key_node, value_node = subnode.value[0] + key = self.construct_object(key_node) + value = self.construct_object(value_node) + omap.append((key, value)) + + def construct_yaml_pairs(self, node): + # Note: the same code as `construct_yaml_omap`. + pairs = [] + yield pairs + if not isinstance(node, SequenceNode): + raise ConstructorError("while constructing pairs", node.start_mark, + "expected a sequence, but found %s" % node.id, node.start_mark) + for subnode in node.value: + if not isinstance(subnode, MappingNode): + raise ConstructorError("while constructing pairs", node.start_mark, + "expected a mapping of length 1, but found %s" % subnode.id, + subnode.start_mark) + if len(subnode.value) != 1: + raise ConstructorError("while constructing pairs", node.start_mark, + "expected a single mapping item, but found %d items" % len(subnode.value), + subnode.start_mark) + key_node, value_node = subnode.value[0] + key = self.construct_object(key_node) + value = self.construct_object(value_node) + pairs.append((key, value)) + + def construct_yaml_set(self, node): + data = set() + yield data + value = self.construct_mapping(node) + data.update(value) + + def construct_yaml_str(self, node): + return self.construct_scalar(node) + + def construct_yaml_seq(self, node): + data = [] + yield data + data.extend(self.construct_sequence(node)) + + def construct_yaml_map(self, node): + data = {} + yield data + value = self.construct_mapping(node) + data.update(value) + + def construct_yaml_object(self, node, cls): + data = cls.__new__(cls) + yield data + if hasattr(data, '__setstate__'): + state = self.construct_mapping(node, deep=True) + data.__setstate__(state) + else: + state = self.construct_mapping(node) + data.__dict__.update(state) + + def construct_undefined(self, node): + raise ConstructorError(None, None, + "could not determine a constructor for the tag %r" % node.tag, + node.start_mark) + +SafeConstructor.add_constructor( + 'tag:yaml.org,2002:null', + SafeConstructor.construct_yaml_null) + +SafeConstructor.add_constructor( + 'tag:yaml.org,2002:bool', + SafeConstructor.construct_yaml_bool) + +SafeConstructor.add_constructor( + 'tag:yaml.org,2002:int', + SafeConstructor.construct_yaml_int) + +SafeConstructor.add_constructor( + 'tag:yaml.org,2002:float', + SafeConstructor.construct_yaml_float) + +SafeConstructor.add_constructor( + 'tag:yaml.org,2002:binary', + SafeConstructor.construct_yaml_binary) + +SafeConstructor.add_constructor( + 'tag:yaml.org,2002:timestamp', + SafeConstructor.construct_yaml_timestamp) + +SafeConstructor.add_constructor( + 'tag:yaml.org,2002:omap', + SafeConstructor.construct_yaml_omap) + +SafeConstructor.add_constructor( + 'tag:yaml.org,2002:pairs', + SafeConstructor.construct_yaml_pairs) + +SafeConstructor.add_constructor( + 'tag:yaml.org,2002:set', + SafeConstructor.construct_yaml_set) + +SafeConstructor.add_constructor( + 'tag:yaml.org,2002:str', + SafeConstructor.construct_yaml_str) + +SafeConstructor.add_constructor( + 'tag:yaml.org,2002:seq', + SafeConstructor.construct_yaml_seq) + +SafeConstructor.add_constructor( + 'tag:yaml.org,2002:map', + SafeConstructor.construct_yaml_map) + +SafeConstructor.add_constructor(None, + SafeConstructor.construct_undefined) + +class Constructor(SafeConstructor): + + def construct_python_str(self, node): + return self.construct_scalar(node) + + def construct_python_unicode(self, node): + return self.construct_scalar(node) + + def construct_python_bytes(self, node): + try: + value = self.construct_scalar(node).encode('ascii') + except UnicodeEncodeError as exc: + raise ConstructorError(None, None, + "failed to convert base64 data into ascii: %s" % exc, + node.start_mark) + try: + if hasattr(base64, 'decodebytes'): + return base64.decodebytes(value) + else: + return base64.decodestring(value) + except binascii.Error as exc: + raise ConstructorError(None, None, + "failed to decode base64 data: %s" % exc, node.start_mark) + + def construct_python_long(self, node): + return self.construct_yaml_int(node) + + def construct_python_complex(self, node): + return complex(self.construct_scalar(node)) + + def construct_python_tuple(self, node): + return tuple(self.construct_sequence(node)) + + def find_python_module(self, name, mark): + if not name: + raise ConstructorError("while constructing a Python module", mark, + "expected non-empty name appended to the tag", mark) + try: + __import__(name) + except ImportError as exc: + raise ConstructorError("while constructing a Python module", mark, + "cannot find module %r (%s)" % (name, exc), mark) + return sys.modules[name] + + def find_python_name(self, name, mark): + if not name: + raise ConstructorError("while constructing a Python object", mark, + "expected non-empty name appended to the tag", mark) + if '.' in name: + module_name, object_name = name.rsplit('.', 1) + else: + module_name = 'builtins' + object_name = name + try: + __import__(module_name) + except ImportError as exc: + raise ConstructorError("while constructing a Python object", mark, + "cannot find module %r (%s)" % (module_name, exc), mark) + module = sys.modules[module_name] + if not hasattr(module, object_name): + raise ConstructorError("while constructing a Python object", mark, + "cannot find %r in the module %r" + % (object_name, module.__name__), mark) + return getattr(module, object_name) + + def construct_python_name(self, suffix, node): + value = self.construct_scalar(node) + if value: + raise ConstructorError("while constructing a Python name", node.start_mark, + "expected the empty value, but found %r" % value, node.start_mark) + return self.find_python_name(suffix, node.start_mark) + + def construct_python_module(self, suffix, node): + value = self.construct_scalar(node) + if value: + raise ConstructorError("while constructing a Python module", node.start_mark, + "expected the empty value, but found %r" % value, node.start_mark) + return self.find_python_module(suffix, node.start_mark) + + def make_python_instance(self, suffix, node, + args=None, kwds=None, newobj=False): + if not args: + args = [] + if not kwds: + kwds = {} + cls = self.find_python_name(suffix, node.start_mark) + if newobj and isinstance(cls, type): + return cls.__new__(cls, *args, **kwds) + else: + return cls(*args, **kwds) + + def set_python_instance_state(self, instance, state): + if hasattr(instance, '__setstate__'): + instance.__setstate__(state) + else: + slotstate = {} + if isinstance(state, tuple) and len(state) == 2: + state, slotstate = state + if hasattr(instance, '__dict__'): + instance.__dict__.update(state) + elif state: + slotstate.update(state) + for key, value in slotstate.items(): + setattr(object, key, value) + + def construct_python_object(self, suffix, node): + # Format: + # !!python/object:module.name { ... state ... } + instance = self.make_python_instance(suffix, node, newobj=True) + yield instance + deep = hasattr(instance, '__setstate__') + state = self.construct_mapping(node, deep=deep) + self.set_python_instance_state(instance, state) + + def construct_python_object_apply(self, suffix, node, newobj=False): + # Format: + # !!python/object/apply # (or !!python/object/new) + # args: [ ... arguments ... ] + # kwds: { ... keywords ... } + # state: ... state ... + # listitems: [ ... listitems ... ] + # dictitems: { ... dictitems ... } + # or short format: + # !!python/object/apply [ ... arguments ... ] + # The difference between !!python/object/apply and !!python/object/new + # is how an object is created, check make_python_instance for details. + if isinstance(node, SequenceNode): + args = self.construct_sequence(node, deep=True) + kwds = {} + state = {} + listitems = [] + dictitems = {} + else: + value = self.construct_mapping(node, deep=True) + args = value.get('args', []) + kwds = value.get('kwds', {}) + state = value.get('state', {}) + listitems = value.get('listitems', []) + dictitems = value.get('dictitems', {}) + instance = self.make_python_instance(suffix, node, args, kwds, newobj) + if state: + self.set_python_instance_state(instance, state) + if listitems: + instance.extend(listitems) + if dictitems: + for key in dictitems: + instance[key] = dictitems[key] + return instance + + def construct_python_object_new(self, suffix, node): + return self.construct_python_object_apply(suffix, node, newobj=True) + +Constructor.add_constructor( + 'tag:yaml.org,2002:python/none', + Constructor.construct_yaml_null) + +Constructor.add_constructor( + 'tag:yaml.org,2002:python/bool', + Constructor.construct_yaml_bool) + +Constructor.add_constructor( + 'tag:yaml.org,2002:python/str', + Constructor.construct_python_str) + +Constructor.add_constructor( + 'tag:yaml.org,2002:python/unicode', + Constructor.construct_python_unicode) + +Constructor.add_constructor( + 'tag:yaml.org,2002:python/bytes', + Constructor.construct_python_bytes) + +Constructor.add_constructor( + 'tag:yaml.org,2002:python/int', + Constructor.construct_yaml_int) + +Constructor.add_constructor( + 'tag:yaml.org,2002:python/long', + Constructor.construct_python_long) + +Constructor.add_constructor( + 'tag:yaml.org,2002:python/float', + Constructor.construct_yaml_float) + +Constructor.add_constructor( + 'tag:yaml.org,2002:python/complex', + Constructor.construct_python_complex) + +Constructor.add_constructor( + 'tag:yaml.org,2002:python/list', + Constructor.construct_yaml_seq) + +Constructor.add_constructor( + 'tag:yaml.org,2002:python/tuple', + Constructor.construct_python_tuple) + +Constructor.add_constructor( + 'tag:yaml.org,2002:python/dict', + Constructor.construct_yaml_map) + +Constructor.add_multi_constructor( + 'tag:yaml.org,2002:python/name:', + Constructor.construct_python_name) + +Constructor.add_multi_constructor( + 'tag:yaml.org,2002:python/module:', + Constructor.construct_python_module) + +Constructor.add_multi_constructor( + 'tag:yaml.org,2002:python/object:', + Constructor.construct_python_object) + +Constructor.add_multi_constructor( + 'tag:yaml.org,2002:python/object/apply:', + Constructor.construct_python_object_apply) + +Constructor.add_multi_constructor( + 'tag:yaml.org,2002:python/object/new:', + Constructor.construct_python_object_new) + diff --git a/python.d/python_modules/pyyaml3/cyaml.py b/python.d/python_modules/pyyaml3/cyaml.py new file mode 100644 index 000000000..d5cb87e99 --- /dev/null +++ b/python.d/python_modules/pyyaml3/cyaml.py @@ -0,0 +1,85 @@ + +__all__ = ['CBaseLoader', 'CSafeLoader', 'CLoader', + 'CBaseDumper', 'CSafeDumper', 'CDumper'] + +from _yaml import CParser, CEmitter + +from .constructor import * + +from .serializer import * +from .representer import * + +from .resolver import * + +class CBaseLoader(CParser, BaseConstructor, BaseResolver): + + def __init__(self, stream): + CParser.__init__(self, stream) + BaseConstructor.__init__(self) + BaseResolver.__init__(self) + +class CSafeLoader(CParser, SafeConstructor, Resolver): + + def __init__(self, stream): + CParser.__init__(self, stream) + SafeConstructor.__init__(self) + Resolver.__init__(self) + +class CLoader(CParser, Constructor, Resolver): + + def __init__(self, stream): + CParser.__init__(self, stream) + Constructor.__init__(self) + Resolver.__init__(self) + +class CBaseDumper(CEmitter, BaseRepresenter, BaseResolver): + + def __init__(self, stream, + default_style=None, default_flow_style=None, + canonical=None, indent=None, width=None, + allow_unicode=None, line_break=None, + encoding=None, explicit_start=None, explicit_end=None, + version=None, tags=None): + CEmitter.__init__(self, stream, canonical=canonical, + indent=indent, width=width, encoding=encoding, + allow_unicode=allow_unicode, line_break=line_break, + explicit_start=explicit_start, explicit_end=explicit_end, + version=version, tags=tags) + Representer.__init__(self, default_style=default_style, + default_flow_style=default_flow_style) + Resolver.__init__(self) + +class CSafeDumper(CEmitter, SafeRepresenter, Resolver): + + def __init__(self, stream, + default_style=None, default_flow_style=None, + canonical=None, indent=None, width=None, + allow_unicode=None, line_break=None, + encoding=None, explicit_start=None, explicit_end=None, + version=None, tags=None): + CEmitter.__init__(self, stream, canonical=canonical, + indent=indent, width=width, encoding=encoding, + allow_unicode=allow_unicode, line_break=line_break, + explicit_start=explicit_start, explicit_end=explicit_end, + version=version, tags=tags) + SafeRepresenter.__init__(self, default_style=default_style, + default_flow_style=default_flow_style) + Resolver.__init__(self) + +class CDumper(CEmitter, Serializer, Representer, Resolver): + + def __init__(self, stream, + default_style=None, default_flow_style=None, + canonical=None, indent=None, width=None, + allow_unicode=None, line_break=None, + encoding=None, explicit_start=None, explicit_end=None, + version=None, tags=None): + CEmitter.__init__(self, stream, canonical=canonical, + indent=indent, width=width, encoding=encoding, + allow_unicode=allow_unicode, line_break=line_break, + explicit_start=explicit_start, explicit_end=explicit_end, + version=version, tags=tags) + Representer.__init__(self, default_style=default_style, + default_flow_style=default_flow_style) + Resolver.__init__(self) + diff --git a/python.d/python_modules/pyyaml3/dumper.py b/python.d/python_modules/pyyaml3/dumper.py new file mode 100644 index 000000000..0b6912877 --- /dev/null +++ b/python.d/python_modules/pyyaml3/dumper.py @@ -0,0 +1,62 @@ + +__all__ = ['BaseDumper', 'SafeDumper', 'Dumper'] + +from .emitter import * +from .serializer import * +from .representer import * +from .resolver import * + +class BaseDumper(Emitter, Serializer, BaseRepresenter, BaseResolver): + + def __init__(self, stream, + default_style=None, default_flow_style=None, + canonical=None, indent=None, width=None, + allow_unicode=None, line_break=None, + encoding=None, explicit_start=None, explicit_end=None, + version=None, tags=None): + Emitter.__init__(self, stream, canonical=canonical, + indent=indent, width=width, + allow_unicode=allow_unicode, line_break=line_break) + Serializer.__init__(self, encoding=encoding, + explicit_start=explicit_start, explicit_end=explicit_end, + version=version, tags=tags) + Representer.__init__(self, default_style=default_style, + default_flow_style=default_flow_style) + Resolver.__init__(self) + +class SafeDumper(Emitter, Serializer, SafeRepresenter, Resolver): + + def __init__(self, stream, + default_style=None, default_flow_style=None, + canonical=None, indent=None, width=None, + allow_unicode=None, line_break=None, + encoding=None, explicit_start=None, explicit_end=None, + version=None, tags=None): + Emitter.__init__(self, stream, canonical=canonical, + indent=indent, width=width, + allow_unicode=allow_unicode, line_break=line_break) + Serializer.__init__(self, encoding=encoding, + explicit_start=explicit_start, explicit_end=explicit_end, + version=version, tags=tags) + SafeRepresenter.__init__(self, default_style=default_style, + default_flow_style=default_flow_style) + Resolver.__init__(self) + +class Dumper(Emitter, Serializer, Representer, Resolver): + + def __init__(self, stream, + default_style=None, default_flow_style=None, + canonical=None, indent=None, width=None, + allow_unicode=None, line_break=None, + encoding=None, explicit_start=None, explicit_end=None, + version=None, tags=None): + Emitter.__init__(self, stream, canonical=canonical, + indent=indent, width=width, + allow_unicode=allow_unicode, line_break=line_break) + Serializer.__init__(self, encoding=encoding, + explicit_start=explicit_start, explicit_end=explicit_end, + version=version, tags=tags) + Representer.__init__(self, default_style=default_style, + default_flow_style=default_flow_style) + Resolver.__init__(self) + diff --git a/python.d/python_modules/pyyaml3/emitter.py b/python.d/python_modules/pyyaml3/emitter.py new file mode 100644 index 000000000..34cb145a5 --- /dev/null +++ b/python.d/python_modules/pyyaml3/emitter.py @@ -0,0 +1,1137 @@ + +# Emitter expects events obeying the following grammar: +# stream ::= STREAM-START document* STREAM-END +# document ::= DOCUMENT-START node DOCUMENT-END +# node ::= SCALAR | sequence | mapping +# sequence ::= SEQUENCE-START node* SEQUENCE-END +# mapping ::= MAPPING-START (node node)* MAPPING-END + +__all__ = ['Emitter', 'EmitterError'] + +from .error import YAMLError +from .events import * + +class EmitterError(YAMLError): + pass + +class ScalarAnalysis: + def __init__(self, scalar, empty, multiline, + allow_flow_plain, allow_block_plain, + allow_single_quoted, allow_double_quoted, + allow_block): + self.scalar = scalar + self.empty = empty + self.multiline = multiline + self.allow_flow_plain = allow_flow_plain + self.allow_block_plain = allow_block_plain + self.allow_single_quoted = allow_single_quoted + self.allow_double_quoted = allow_double_quoted + self.allow_block = allow_block + +class Emitter: + + DEFAULT_TAG_PREFIXES = { + '!' : '!', + 'tag:yaml.org,2002:' : '!!', + } + + def __init__(self, stream, canonical=None, indent=None, width=None, + allow_unicode=None, line_break=None): + + # The stream should have the methods `write` and possibly `flush`. + self.stream = stream + + # Encoding can be overriden by STREAM-START. + self.encoding = None + + # Emitter is a state machine with a stack of states to handle nested + # structures. + self.states = [] + self.state = self.expect_stream_start + + # Current event and the event queue. + self.events = [] + self.event = None + + # The current indentation level and the stack of previous indents. + self.indents = [] + self.indent = None + + # Flow level. + self.flow_level = 0 + + # Contexts. + self.root_context = False + self.sequence_context = False + self.mapping_context = False + self.simple_key_context = False + + # Characteristics of the last emitted character: + # - current position. + # - is it a whitespace? + # - is it an indention character + # (indentation space, '-', '?', or ':')? + self.line = 0 + self.column = 0 + self.whitespace = True + self.indention = True + + # Whether the document requires an explicit document indicator + self.open_ended = False + + # Formatting details. + self.canonical = canonical + self.allow_unicode = allow_unicode + self.best_indent = 2 + if indent and 1 < indent < 10: + self.best_indent = indent + self.best_width = 80 + if width and width > self.best_indent*2: + self.best_width = width + self.best_line_break = '\n' + if line_break in ['\r', '\n', '\r\n']: + self.best_line_break = line_break + + # Tag prefixes. + self.tag_prefixes = None + + # Prepared anchor and tag. + self.prepared_anchor = None + self.prepared_tag = None + + # Scalar analysis and style. + self.analysis = None + self.style = None + + def dispose(self): + # Reset the state attributes (to clear self-references) + self.states = [] + self.state = None + + def emit(self, event): + self.events.append(event) + while not self.need_more_events(): + self.event = self.events.pop(0) + self.state() + self.event = None + + # In some cases, we wait for a few next events before emitting. + + def need_more_events(self): + if not self.events: + return True + event = self.events[0] + if isinstance(event, DocumentStartEvent): + return self.need_events(1) + elif isinstance(event, SequenceStartEvent): + return self.need_events(2) + elif isinstance(event, MappingStartEvent): + return self.need_events(3) + else: + return False + + def need_events(self, count): + level = 0 + for event in self.events[1:]: + if isinstance(event, (DocumentStartEvent, CollectionStartEvent)): + level += 1 + elif isinstance(event, (DocumentEndEvent, CollectionEndEvent)): + level -= 1 + elif isinstance(event, StreamEndEvent): + level = -1 + if level < 0: + return False + return (len(self.events) < count+1) + + def increase_indent(self, flow=False, indentless=False): + self.indents.append(self.indent) + if self.indent is None: + if flow: + self.indent = self.best_indent + else: + self.indent = 0 + elif not indentless: + self.indent += self.best_indent + + # States. + + # Stream handlers. + + def expect_stream_start(self): + if isinstance(self.event, StreamStartEvent): + if self.event.encoding and not hasattr(self.stream, 'encoding'): + self.encoding = self.event.encoding + self.write_stream_start() + self.state = self.expect_first_document_start + else: + raise EmitterError("expected StreamStartEvent, but got %s" + % self.event) + + def expect_nothing(self): + raise EmitterError("expected nothing, but got %s" % self.event) + + # Document handlers. + + def expect_first_document_start(self): + return self.expect_document_start(first=True) + + def expect_document_start(self, first=False): + if isinstance(self.event, DocumentStartEvent): + if (self.event.version or self.event.tags) and self.open_ended: + self.write_indicator('...', True) + self.write_indent() + if self.event.version: + version_text = self.prepare_version(self.event.version) + self.write_version_directive(version_text) + self.tag_prefixes = self.DEFAULT_TAG_PREFIXES.copy() + if self.event.tags: + handles = sorted(self.event.tags.keys()) + for handle in handles: + prefix = self.event.tags[handle] + self.tag_prefixes[prefix] = handle + handle_text = self.prepare_tag_handle(handle) + prefix_text = self.prepare_tag_prefix(prefix) + self.write_tag_directive(handle_text, prefix_text) + implicit = (first and not self.event.explicit and not self.canonical + and not self.event.version and not self.event.tags + and not self.check_empty_document()) + if not implicit: + self.write_indent() + self.write_indicator('---', True) + if self.canonical: + self.write_indent() + self.state = self.expect_document_root + elif isinstance(self.event, StreamEndEvent): + if self.open_ended: + self.write_indicator('...', True) + self.write_indent() + self.write_stream_end() + self.state = self.expect_nothing + else: + raise EmitterError("expected DocumentStartEvent, but got %s" + % self.event) + + def expect_document_end(self): + if isinstance(self.event, DocumentEndEvent): + self.write_indent() + if self.event.explicit: + self.write_indicator('...', True) + self.write_indent() + self.flush_stream() + self.state = self.expect_document_start + else: + raise EmitterError("expected DocumentEndEvent, but got %s" + % self.event) + + def expect_document_root(self): + self.states.append(self.expect_document_end) + self.expect_node(root=True) + + # Node handlers. + + def expect_node(self, root=False, sequence=False, mapping=False, + simple_key=False): + self.root_context = root + self.sequence_context = sequence + self.mapping_context = mapping + self.simple_key_context = simple_key + if isinstance(self.event, AliasEvent): + self.expect_alias() + elif isinstance(self.event, (ScalarEvent, CollectionStartEvent)): + self.process_anchor('&') + self.process_tag() + if isinstance(self.event, ScalarEvent): + self.expect_scalar() + elif isinstance(self.event, SequenceStartEvent): + if self.flow_level or self.canonical or self.event.flow_style \ + or self.check_empty_sequence(): + self.expect_flow_sequence() + else: + self.expect_block_sequence() + elif isinstance(self.event, MappingStartEvent): + if self.flow_level or self.canonical or self.event.flow_style \ + or self.check_empty_mapping(): + self.expect_flow_mapping() + else: + self.expect_block_mapping() + else: + raise EmitterError("expected NodeEvent, but got %s" % self.event) + + def expect_alias(self): + if self.event.anchor is None: + raise EmitterError("anchor is not specified for alias") + self.process_anchor('*') + self.state = self.states.pop() + + def expect_scalar(self): + self.increase_indent(flow=True) + self.process_scalar() + self.indent = self.indents.pop() + self.state = self.states.pop() + + # Flow sequence handlers. + + def expect_flow_sequence(self): + self.write_indicator('[', True, whitespace=True) + self.flow_level += 1 + self.increase_indent(flow=True) + self.state = self.expect_first_flow_sequence_item + + def expect_first_flow_sequence_item(self): + if isinstance(self.event, SequenceEndEvent): + self.indent = self.indents.pop() + self.flow_level -= 1 + self.write_indicator(']', False) + self.state = self.states.pop() + else: + if self.canonical or self.column > self.best_width: + self.write_indent() + self.states.append(self.expect_flow_sequence_item) + self.expect_node(sequence=True) + + def expect_flow_sequence_item(self): + if isinstance(self.event, SequenceEndEvent): + self.indent = self.indents.pop() + self.flow_level -= 1 + if self.canonical: + self.write_indicator(',', False) + self.write_indent() + self.write_indicator(']', False) + self.state = self.states.pop() + else: + self.write_indicator(',', False) + if self.canonical or self.column > self.best_width: + self.write_indent() + self.states.append(self.expect_flow_sequence_item) + self.expect_node(sequence=True) + + # Flow mapping handlers. + + def expect_flow_mapping(self): + self.write_indicator('{', True, whitespace=True) + self.flow_level += 1 + self.increase_indent(flow=True) + self.state = self.expect_first_flow_mapping_key + + def expect_first_flow_mapping_key(self): + if isinstance(self.event, MappingEndEvent): + self.indent = self.indents.pop() + self.flow_level -= 1 + self.write_indicator('}', False) + self.state = self.states.pop() + else: + if self.canonical or self.column > self.best_width: + self.write_indent() + if not self.canonical and self.check_simple_key(): + self.states.append(self.expect_flow_mapping_simple_value) + self.expect_node(mapping=True, simple_key=True) + else: + self.write_indicator('?', True) + self.states.append(self.expect_flow_mapping_value) + self.expect_node(mapping=True) + + def expect_flow_mapping_key(self): + if isinstance(self.event, MappingEndEvent): + self.indent = self.indents.pop() + self.flow_level -= 1 + if self.canonical: + self.write_indicator(',', False) + self.write_indent() + self.write_indicator('}', False) + self.state = self.states.pop() + else: + self.write_indicator(',', False) + if self.canonical or self.column > self.best_width: + self.write_indent() + if not self.canonical and self.check_simple_key(): + self.states.append(self.expect_flow_mapping_simple_value) + self.expect_node(mapping=True, simple_key=True) + else: + self.write_indicator('?', True) + self.states.append(self.expect_flow_mapping_value) + self.expect_node(mapping=True) + + def expect_flow_mapping_simple_value(self): + self.write_indicator(':', False) + self.states.append(self.expect_flow_mapping_key) + self.expect_node(mapping=True) + + def expect_flow_mapping_value(self): + if self.canonical or self.column > self.best_width: + self.write_indent() + self.write_indicator(':', True) + self.states.append(self.expect_flow_mapping_key) + self.expect_node(mapping=True) + + # Block sequence handlers. + + def expect_block_sequence(self): + indentless = (self.mapping_context and not self.indention) + self.increase_indent(flow=False, indentless=indentless) + self.state = self.expect_first_block_sequence_item + + def expect_first_block_sequence_item(self): + return self.expect_block_sequence_item(first=True) + + def expect_block_sequence_item(self, first=False): + if not first and isinstance(self.event, SequenceEndEvent): + self.indent = self.indents.pop() + self.state = self.states.pop() + else: + self.write_indent() + self.write_indicator('-', True, indention=True) + self.states.append(self.expect_block_sequence_item) + self.expect_node(sequence=True) + + # Block mapping handlers. + + def expect_block_mapping(self): + self.increase_indent(flow=False) + self.state = self.expect_first_block_mapping_key + + def expect_first_block_mapping_key(self): + return self.expect_block_mapping_key(first=True) + + def expect_block_mapping_key(self, first=False): + if not first and isinstance(self.event, MappingEndEvent): + self.indent = self.indents.pop() + self.state = self.states.pop() + else: + self.write_indent() + if self.check_simple_key(): + self.states.append(self.expect_block_mapping_simple_value) + self.expect_node(mapping=True, simple_key=True) + else: + self.write_indicator('?', True, indention=True) + self.states.append(self.expect_block_mapping_value) + self.expect_node(mapping=True) + + def expect_block_mapping_simple_value(self): + self.write_indicator(':', False) + self.states.append(self.expect_block_mapping_key) + self.expect_node(mapping=True) + + def expect_block_mapping_value(self): + self.write_indent() + self.write_indicator(':', True, indention=True) + self.states.append(self.expect_block_mapping_key) + self.expect_node(mapping=True) + + # Checkers. + + def check_empty_sequence(self): + return (isinstance(self.event, SequenceStartEvent) and self.events + and isinstance(self.events[0], SequenceEndEvent)) + + def check_empty_mapping(self): + return (isinstance(self.event, MappingStartEvent) and self.events + and isinstance(self.events[0], MappingEndEvent)) + + def check_empty_document(self): + if not isinstance(self.event, DocumentStartEvent) or not self.events: + return False + event = self.events[0] + return (isinstance(event, ScalarEvent) and event.anchor is None + and event.tag is None and event.implicit and event.value == '') + + def check_simple_key(self): + length = 0 + if isinstance(self.event, NodeEvent) and self.event.anchor is not None: + if self.prepared_anchor is None: + self.prepared_anchor = self.prepare_anchor(self.event.anchor) + length += len(self.prepared_anchor) + if isinstance(self.event, (ScalarEvent, CollectionStartEvent)) \ + and self.event.tag is not None: + if self.prepared_tag is None: + self.prepared_tag = self.prepare_tag(self.event.tag) + length += len(self.prepared_tag) + if isinstance(self.event, ScalarEvent): + if self.analysis is None: + self.analysis = self.analyze_scalar(self.event.value) + length += len(self.analysis.scalar) + return (length < 128 and (isinstance(self.event, AliasEvent) + or (isinstance(self.event, ScalarEvent) + and not self.analysis.empty and not self.analysis.multiline) + or self.check_empty_sequence() or self.check_empty_mapping())) + + # Anchor, Tag, and Scalar processors. + + def process_anchor(self, indicator): + if self.event.anchor is None: + self.prepared_anchor = None + return + if self.prepared_anchor is None: + self.prepared_anchor = self.prepare_anchor(self.event.anchor) + if self.prepared_anchor: + self.write_indicator(indicator+self.prepared_anchor, True) + self.prepared_anchor = None + + def process_tag(self): + tag = self.event.tag + if isinstance(self.event, ScalarEvent): + if self.style is None: + self.style = self.choose_scalar_style() + if ((not self.canonical or tag is None) and + ((self.style == '' and self.event.implicit[0]) + or (self.style != '' and self.event.implicit[1]))): + self.prepared_tag = None + return + if self.event.implicit[0] and tag is None: + tag = '!' + self.prepared_tag = None + else: + if (not self.canonical or tag is None) and self.event.implicit: + self.prepared_tag = None + return + if tag is None: + raise EmitterError("tag is not specified") + if self.prepared_tag is None: + self.prepared_tag = self.prepare_tag(tag) + if self.prepared_tag: + self.write_indicator(self.prepared_tag, True) + self.prepared_tag = None + + def choose_scalar_style(self): + if self.analysis is None: + self.analysis = self.analyze_scalar(self.event.value) + if self.event.style == '"' or self.canonical: + return '"' + if not self.event.style and self.event.implicit[0]: + if (not (self.simple_key_context and + (self.analysis.empty or self.analysis.multiline)) + and (self.flow_level and self.analysis.allow_flow_plain + or (not self.flow_level and self.analysis.allow_block_plain))): + return '' + if self.event.style and self.event.style in '|>': + if (not self.flow_level and not self.simple_key_context + and self.analysis.allow_block): + return self.event.style + if not self.event.style or self.event.style == '\'': + if (self.analysis.allow_single_quoted and + not (self.simple_key_context and self.analysis.multiline)): + return '\'' + return '"' + + def process_scalar(self): + if self.analysis is None: + self.analysis = self.analyze_scalar(self.event.value) + if self.style is None: + self.style = self.choose_scalar_style() + split = (not self.simple_key_context) + #if self.analysis.multiline and split \ + # and (not self.style or self.style in '\'\"'): + # self.write_indent() + if self.style == '"': + self.write_double_quoted(self.analysis.scalar, split) + elif self.style == '\'': + self.write_single_quoted(self.analysis.scalar, split) + elif self.style == '>': + self.write_folded(self.analysis.scalar) + elif self.style == '|': + self.write_literal(self.analysis.scalar) + else: + self.write_plain(self.analysis.scalar, split) + self.analysis = None + self.style = None + + # Analyzers. + + def prepare_version(self, version): + major, minor = version + if major != 1: + raise EmitterError("unsupported YAML version: %d.%d" % (major, minor)) + return '%d.%d' % (major, minor) + + def prepare_tag_handle(self, handle): + if not handle: + raise EmitterError("tag handle must not be empty") + if handle[0] != '!' or handle[-1] != '!': + raise EmitterError("tag handle must start and end with '!': %r" % handle) + for ch in handle[1:-1]: + if not ('0' <= ch <= '9' or 'A' <= ch <= 'Z' or 'a' <= ch <= 'z' \ + or ch in '-_'): + raise EmitterError("invalid character %r in the tag handle: %r" + % (ch, handle)) + return handle + + def prepare_tag_prefix(self, prefix): + if not prefix: + raise EmitterError("tag prefix must not be empty") + chunks = [] + start = end = 0 + if prefix[0] == '!': + end = 1 + while end < len(prefix): + ch = prefix[end] + if '0' <= ch <= '9' or 'A' <= ch <= 'Z' or 'a' <= ch <= 'z' \ + or ch in '-;/?!:@&=+$,_.~*\'()[]': + end += 1 + else: + if start < end: + chunks.append(prefix[start:end]) + start = end = end+1 + data = ch.encode('utf-8') + for ch in data: + chunks.append('%%%02X' % ord(ch)) + if start < end: + chunks.append(prefix[start:end]) + return ''.join(chunks) + + def prepare_tag(self, tag): + if not tag: + raise EmitterError("tag must not be empty") + if tag == '!': + return tag + handle = None + suffix = tag + prefixes = sorted(self.tag_prefixes.keys()) + for prefix in prefixes: + if tag.startswith(prefix) \ + and (prefix == '!' or len(prefix) < len(tag)): + handle = self.tag_prefixes[prefix] + suffix = tag[len(prefix):] + chunks = [] + start = end = 0 + while end < len(suffix): + ch = suffix[end] + if '0' <= ch <= '9' or 'A' <= ch <= 'Z' or 'a' <= ch <= 'z' \ + or ch in '-;/?:@&=+$,_.~*\'()[]' \ + or (ch == '!' and handle != '!'): + end += 1 + else: + if start < end: + chunks.append(suffix[start:end]) + start = end = end+1 + data = ch.encode('utf-8') + for ch in data: + chunks.append('%%%02X' % ord(ch)) + if start < end: + chunks.append(suffix[start:end]) + suffix_text = ''.join(chunks) + if handle: + return '%s%s' % (handle, suffix_text) + else: + return '!<%s>' % suffix_text + + def prepare_anchor(self, anchor): + if not anchor: + raise EmitterError("anchor must not be empty") + for ch in anchor: + if not ('0' <= ch <= '9' or 'A' <= ch <= 'Z' or 'a' <= ch <= 'z' \ + or ch in '-_'): + raise EmitterError("invalid character %r in the anchor: %r" + % (ch, anchor)) + return anchor + + def analyze_scalar(self, scalar): + + # Empty scalar is a special case. + if not scalar: + return ScalarAnalysis(scalar=scalar, empty=True, multiline=False, + allow_flow_plain=False, allow_block_plain=True, + allow_single_quoted=True, allow_double_quoted=True, + allow_block=False) + + # Indicators and special characters. + block_indicators = False + flow_indicators = False + line_breaks = False + special_characters = False + + # Important whitespace combinations. + leading_space = False + leading_break = False + trailing_space = False + trailing_break = False + break_space = False + space_break = False + + # Check document indicators. + if scalar.startswith('---') or scalar.startswith('...'): + block_indicators = True + flow_indicators = True + + # First character or preceded by a whitespace. + preceeded_by_whitespace = True + + # Last character or followed by a whitespace. + followed_by_whitespace = (len(scalar) == 1 or + scalar[1] in '\0 \t\r\n\x85\u2028\u2029') + + # The previous character is a space. + previous_space = False + + # The previous character is a break. + previous_break = False + + index = 0 + while index < len(scalar): + ch = scalar[index] + + # Check for indicators. + if index == 0: + # Leading indicators are special characters. + if ch in '#,[]{}&*!|>\'\"%@`': + flow_indicators = True + block_indicators = True + if ch in '?:': + flow_indicators = True + if followed_by_whitespace: + block_indicators = True + if ch == '-' and followed_by_whitespace: + flow_indicators = True + block_indicators = True + else: + # Some indicators cannot appear within a scalar as well. + if ch in ',?[]{}': + flow_indicators = True + if ch == ':': + flow_indicators = True + if followed_by_whitespace: + block_indicators = True + if ch == '#' and preceeded_by_whitespace: + flow_indicators = True + block_indicators = True + + # Check for line breaks, special, and unicode characters. + if ch in '\n\x85\u2028\u2029': + line_breaks = True + if not (ch == '\n' or '\x20' <= ch <= '\x7E'): + if (ch == '\x85' or '\xA0' <= ch <= '\uD7FF' + or '\uE000' <= ch <= '\uFFFD') and ch != '\uFEFF': + unicode_characters = True + if not self.allow_unicode: + special_characters = True + else: + special_characters = True + + # Detect important whitespace combinations. + if ch == ' ': + if index == 0: + leading_space = True + if index == len(scalar)-1: + trailing_space = True + if previous_break: + break_space = True + previous_space = True + previous_break = False + elif ch in '\n\x85\u2028\u2029': + if index == 0: + leading_break = True + if index == len(scalar)-1: + trailing_break = True + if previous_space: + space_break = True + previous_space = False + previous_break = True + else: + previous_space = False + previous_break = False + + # Prepare for the next character. + index += 1 + preceeded_by_whitespace = (ch in '\0 \t\r\n\x85\u2028\u2029') + followed_by_whitespace = (index+1 >= len(scalar) or + scalar[index+1] in '\0 \t\r\n\x85\u2028\u2029') + + # Let's decide what styles are allowed. + allow_flow_plain = True + allow_block_plain = True + allow_single_quoted = True + allow_double_quoted = True + allow_block = True + + # Leading and trailing whitespaces are bad for plain scalars. + if (leading_space or leading_break + or trailing_space or trailing_break): + allow_flow_plain = allow_block_plain = False + + # We do not permit trailing spaces for block scalars. + if trailing_space: + allow_block = False + + # Spaces at the beginning of a new line are only acceptable for block + # scalars. + if break_space: + allow_flow_plain = allow_block_plain = allow_single_quoted = False + + # Spaces followed by breaks, as well as special character are only + # allowed for double quoted scalars. + if space_break or special_characters: + allow_flow_plain = allow_block_plain = \ + allow_single_quoted = allow_block = False + + # Although the plain scalar writer supports breaks, we never emit + # multiline plain scalars. + if line_breaks: + allow_flow_plain = allow_block_plain = False + + # Flow indicators are forbidden for flow plain scalars. + if flow_indicators: + allow_flow_plain = False + + # Block indicators are forbidden for block plain scalars. + if block_indicators: + allow_block_plain = False + + return ScalarAnalysis(scalar=scalar, + empty=False, multiline=line_breaks, + allow_flow_plain=allow_flow_plain, + allow_block_plain=allow_block_plain, + allow_single_quoted=allow_single_quoted, + allow_double_quoted=allow_double_quoted, + allow_block=allow_block) + + # Writers. + + def flush_stream(self): + if hasattr(self.stream, 'flush'): + self.stream.flush() + + def write_stream_start(self): + # Write BOM if needed. + if self.encoding and self.encoding.startswith('utf-16'): + self.stream.write('\uFEFF'.encode(self.encoding)) + + def write_stream_end(self): + self.flush_stream() + + def write_indicator(self, indicator, need_whitespace, + whitespace=False, indention=False): + if self.whitespace or not need_whitespace: + data = indicator + else: + data = ' '+indicator + self.whitespace = whitespace + self.indention = self.indention and indention + self.column += len(data) + self.open_ended = False + if self.encoding: + data = data.encode(self.encoding) + self.stream.write(data) + + def write_indent(self): + indent = self.indent or 0 + if not self.indention or self.column > indent \ + or (self.column == indent and not self.whitespace): + self.write_line_break() + if self.column < indent: + self.whitespace = True + data = ' '*(indent-self.column) + self.column = indent + if self.encoding: + data = data.encode(self.encoding) + self.stream.write(data) + + def write_line_break(self, data=None): + if data is None: + data = self.best_line_break + self.whitespace = True + self.indention = True + self.line += 1 + self.column = 0 + if self.encoding: + data = data.encode(self.encoding) + self.stream.write(data) + + def write_version_directive(self, version_text): + data = '%%YAML %s' % version_text + if self.encoding: + data = data.encode(self.encoding) + self.stream.write(data) + self.write_line_break() + + def write_tag_directive(self, handle_text, prefix_text): + data = '%%TAG %s %s' % (handle_text, prefix_text) + if self.encoding: + data = data.encode(self.encoding) + self.stream.write(data) + self.write_line_break() + + # Scalar streams. + + def write_single_quoted(self, text, split=True): + self.write_indicator('\'', True) + spaces = False + breaks = False + start = end = 0 + while end <= len(text): + ch = None + if end < len(text): + ch = text[end] + if spaces: + if ch is None or ch != ' ': + if start+1 == end and self.column > self.best_width and split \ + and start != 0 and end != len(text): + self.write_indent() + else: + data = text[start:end] + self.column += len(data) + if self.encoding: + data = data.encode(self.encoding) + self.stream.write(data) + start = end + elif breaks: + if ch is None or ch not in '\n\x85\u2028\u2029': + if text[start] == '\n': + self.write_line_break() + for br in text[start:end]: + if br == '\n': + self.write_line_break() + else: + self.write_line_break(br) + self.write_indent() + start = end + else: + if ch is None or ch in ' \n\x85\u2028\u2029' or ch == '\'': + if start < end: + data = text[start:end] + self.column += len(data) + if self.encoding: + data = data.encode(self.encoding) + self.stream.write(data) + start = end + if ch == '\'': + data = '\'\'' + self.column += 2 + if self.encoding: + data = data.encode(self.encoding) + self.stream.write(data) + start = end + 1 + if ch is not None: + spaces = (ch == ' ') + breaks = (ch in '\n\x85\u2028\u2029') + end += 1 + self.write_indicator('\'', False) + + ESCAPE_REPLACEMENTS = { + '\0': '0', + '\x07': 'a', + '\x08': 'b', + '\x09': 't', + '\x0A': 'n', + '\x0B': 'v', + '\x0C': 'f', + '\x0D': 'r', + '\x1B': 'e', + '\"': '\"', + '\\': '\\', + '\x85': 'N', + '\xA0': '_', + '\u2028': 'L', + '\u2029': 'P', + } + + def write_double_quoted(self, text, split=True): + self.write_indicator('"', True) + start = end = 0 + while end <= len(text): + ch = None + if end < len(text): + ch = text[end] + if ch is None or ch in '"\\\x85\u2028\u2029\uFEFF' \ + or not ('\x20' <= ch <= '\x7E' + or (self.allow_unicode + and ('\xA0' <= ch <= '\uD7FF' + or '\uE000' <= ch <= '\uFFFD'))): + if start < end: + data = text[start:end] + self.column += len(data) + if self.encoding: + data = data.encode(self.encoding) + self.stream.write(data) + start = end + if ch is not None: + if ch in self.ESCAPE_REPLACEMENTS: + data = '\\'+self.ESCAPE_REPLACEMENTS[ch] + elif ch <= '\xFF': + data = '\\x%02X' % ord(ch) + elif ch <= '\uFFFF': + data = '\\u%04X' % ord(ch) + else: + data = '\\U%08X' % ord(ch) + self.column += len(data) + if self.encoding: + data = data.encode(self.encoding) + self.stream.write(data) + start = end+1 + if 0 < end < len(text)-1 and (ch == ' ' or start >= end) \ + and self.column+(end-start) > self.best_width and split: + data = text[start:end]+'\\' + if start < end: + start = end + self.column += len(data) + if self.encoding: + data = data.encode(self.encoding) + self.stream.write(data) + self.write_indent() + self.whitespace = False + self.indention = False + if text[start] == ' ': + data = '\\' + self.column += len(data) + if self.encoding: + data = data.encode(self.encoding) + self.stream.write(data) + end += 1 + self.write_indicator('"', False) + + def determine_block_hints(self, text): + hints = '' + if text: + if text[0] in ' \n\x85\u2028\u2029': + hints += str(self.best_indent) + if text[-1] not in '\n\x85\u2028\u2029': + hints += '-' + elif len(text) == 1 or text[-2] in '\n\x85\u2028\u2029': + hints += '+' + return hints + + def write_folded(self, text): + hints = self.determine_block_hints(text) + self.write_indicator('>'+hints, True) + if hints[-1:] == '+': + self.open_ended = True + self.write_line_break() + leading_space = True + spaces = False + breaks = True + start = end = 0 + while end <= len(text): + ch = None + if end < len(text): + ch = text[end] + if breaks: + if ch is None or ch not in '\n\x85\u2028\u2029': + if not leading_space and ch is not None and ch != ' ' \ + and text[start] == '\n': + self.write_line_break() + leading_space = (ch == ' ') + for br in text[start:end]: + if br == '\n': + self.write_line_break() + else: + self.write_line_break(br) + if ch is not None: + self.write_indent() + start = end + elif spaces: + if ch != ' ': + if start+1 == end and self.column > self.best_width: + self.write_indent() + else: + data = text[start:end] + self.column += len(data) + if self.encoding: + data = data.encode(self.encoding) + self.stream.write(data) + start = end + else: + if ch is None or ch in ' \n\x85\u2028\u2029': + data = text[start:end] + self.column += len(data) + if self.encoding: + data = data.encode(self.encoding) + self.stream.write(data) + if ch is None: + self.write_line_break() + start = end + if ch is not None: + breaks = (ch in '\n\x85\u2028\u2029') + spaces = (ch == ' ') + end += 1 + + def write_literal(self, text): + hints = self.determine_block_hints(text) + self.write_indicator('|'+hints, True) + if hints[-1:] == '+': + self.open_ended = True + self.write_line_break() + breaks = True + start = end = 0 + while end <= len(text): + ch = None + if end < len(text): + ch = text[end] + if breaks: + if ch is None or ch not in '\n\x85\u2028\u2029': + for br in text[start:end]: + if br == '\n': + self.write_line_break() + else: + self.write_line_break(br) + if ch is not None: + self.write_indent() + start = end + else: + if ch is None or ch in '\n\x85\u2028\u2029': + data = text[start:end] + if self.encoding: + data = data.encode(self.encoding) + self.stream.write(data) + if ch is None: + self.write_line_break() + start = end + if ch is not None: + breaks = (ch in '\n\x85\u2028\u2029') + end += 1 + + def write_plain(self, text, split=True): + if self.root_context: + self.open_ended = True + if not text: + return + if not self.whitespace: + data = ' ' + self.column += len(data) + if self.encoding: + data = data.encode(self.encoding) + self.stream.write(data) + self.whitespace = False + self.indention = False + spaces = False + breaks = False + start = end = 0 + while end <= len(text): + ch = None + if end < len(text): + ch = text[end] + if spaces: + if ch != ' ': + if start+1 == end and self.column > self.best_width and split: + self.write_indent() + self.whitespace = False + self.indention = False + else: + data = text[start:end] + self.column += len(data) + if self.encoding: + data = data.encode(self.encoding) + self.stream.write(data) + start = end + elif breaks: + if ch not in '\n\x85\u2028\u2029': + if text[start] == '\n': + self.write_line_break() + for br in text[start:end]: + if br == '\n': + self.write_line_break() + else: + self.write_line_break(br) + self.write_indent() + self.whitespace = False + self.indention = False + start = end + else: + if ch is None or ch in ' \n\x85\u2028\u2029': + data = text[start:end] + self.column += len(data) + if self.encoding: + data = data.encode(self.encoding) + self.stream.write(data) + start = end + if ch is not None: + spaces = (ch == ' ') + breaks = (ch in '\n\x85\u2028\u2029') + end += 1 + diff --git a/python.d/python_modules/pyyaml3/error.py b/python.d/python_modules/pyyaml3/error.py new file mode 100644 index 000000000..b796b4dc5 --- /dev/null +++ b/python.d/python_modules/pyyaml3/error.py @@ -0,0 +1,75 @@ + +__all__ = ['Mark', 'YAMLError', 'MarkedYAMLError'] + +class Mark: + + def __init__(self, name, index, line, column, buffer, pointer): + self.name = name + self.index = index + self.line = line + self.column = column + self.buffer = buffer + self.pointer = pointer + + def get_snippet(self, indent=4, max_length=75): + if self.buffer is None: + return None + head = '' + start = self.pointer + while start > 0 and self.buffer[start-1] not in '\0\r\n\x85\u2028\u2029': + start -= 1 + if self.pointer-start > max_length/2-1: + head = ' ... ' + start += 5 + break + tail = '' + end = self.pointer + while end < len(self.buffer) and self.buffer[end] not in '\0\r\n\x85\u2028\u2029': + end += 1 + if end-self.pointer > max_length/2-1: + tail = ' ... ' + end -= 5 + break + snippet = self.buffer[start:end] + return ' '*indent + head + snippet + tail + '\n' \ + + ' '*(indent+self.pointer-start+len(head)) + '^' + + def __str__(self): + snippet = self.get_snippet() + where = " in \"%s\", line %d, column %d" \ + % (self.name, self.line+1, self.column+1) + if snippet is not None: + where += ":\n"+snippet + return where + +class YAMLError(Exception): + pass + +class MarkedYAMLError(YAMLError): + + def __init__(self, context=None, context_mark=None, + problem=None, problem_mark=None, note=None): + self.context = context + self.context_mark = context_mark + self.problem = problem + self.problem_mark = problem_mark + self.note = note + + def __str__(self): + lines = [] + if self.context is not None: + lines.append(self.context) + if self.context_mark is not None \ + and (self.problem is None or self.problem_mark is None + or self.context_mark.name != self.problem_mark.name + or self.context_mark.line != self.problem_mark.line + or self.context_mark.column != self.problem_mark.column): + lines.append(str(self.context_mark)) + if self.problem is not None: + lines.append(self.problem) + if self.problem_mark is not None: + lines.append(str(self.problem_mark)) + if self.note is not None: + lines.append(self.note) + return '\n'.join(lines) + diff --git a/python.d/python_modules/pyyaml3/events.py b/python.d/python_modules/pyyaml3/events.py new file mode 100644 index 000000000..f79ad389c --- /dev/null +++ b/python.d/python_modules/pyyaml3/events.py @@ -0,0 +1,86 @@ + +# Abstract classes. + +class Event(object): + def __init__(self, start_mark=None, end_mark=None): + self.start_mark = start_mark + self.end_mark = end_mark + def __repr__(self): + attributes = [key for key in ['anchor', 'tag', 'implicit', 'value'] + if hasattr(self, key)] + arguments = ', '.join(['%s=%r' % (key, getattr(self, key)) + for key in attributes]) + return '%s(%s)' % (self.__class__.__name__, arguments) + +class NodeEvent(Event): + def __init__(self, anchor, start_mark=None, end_mark=None): + self.anchor = anchor + self.start_mark = start_mark + self.end_mark = end_mark + +class CollectionStartEvent(NodeEvent): + def __init__(self, anchor, tag, implicit, start_mark=None, end_mark=None, + flow_style=None): + self.anchor = anchor + self.tag = tag + self.implicit = implicit + self.start_mark = start_mark + self.end_mark = end_mark + self.flow_style = flow_style + +class CollectionEndEvent(Event): + pass + +# Implementations. + +class StreamStartEvent(Event): + def __init__(self, start_mark=None, end_mark=None, encoding=None): + self.start_mark = start_mark + self.end_mark = end_mark + self.encoding = encoding + +class StreamEndEvent(Event): + pass + +class DocumentStartEvent(Event): + def __init__(self, start_mark=None, end_mark=None, + explicit=None, version=None, tags=None): + self.start_mark = start_mark + self.end_mark = end_mark + self.explicit = explicit + self.version = version + self.tags = tags + +class DocumentEndEvent(Event): + def __init__(self, start_mark=None, end_mark=None, + explicit=None): + self.start_mark = start_mark + self.end_mark = end_mark + self.explicit = explicit + +class AliasEvent(NodeEvent): + pass + +class ScalarEvent(NodeEvent): + def __init__(self, anchor, tag, implicit, value, + start_mark=None, end_mark=None, style=None): + self.anchor = anchor + self.tag = tag + self.implicit = implicit + self.value = value + self.start_mark = start_mark + self.end_mark = end_mark + self.style = style + +class SequenceStartEvent(CollectionStartEvent): + pass + +class SequenceEndEvent(CollectionEndEvent): + pass + +class MappingStartEvent(CollectionStartEvent): + pass + +class MappingEndEvent(CollectionEndEvent): + pass + diff --git a/python.d/python_modules/pyyaml3/loader.py b/python.d/python_modules/pyyaml3/loader.py new file mode 100644 index 000000000..08c8f01b3 --- /dev/null +++ b/python.d/python_modules/pyyaml3/loader.py @@ -0,0 +1,40 @@ + +__all__ = ['BaseLoader', 'SafeLoader', 'Loader'] + +from .reader import * +from .scanner import * +from .parser import * +from .composer import * +from .constructor import * +from .resolver import * + +class BaseLoader(Reader, Scanner, Parser, Composer, BaseConstructor, BaseResolver): + + def __init__(self, stream): + Reader.__init__(self, stream) + Scanner.__init__(self) + Parser.__init__(self) + Composer.__init__(self) + BaseConstructor.__init__(self) + BaseResolver.__init__(self) + +class SafeLoader(Reader, Scanner, Parser, Composer, SafeConstructor, Resolver): + + def __init__(self, stream): + Reader.__init__(self, stream) + Scanner.__init__(self) + Parser.__init__(self) + Composer.__init__(self) + SafeConstructor.__init__(self) + Resolver.__init__(self) + +class Loader(Reader, Scanner, Parser, Composer, Constructor, Resolver): + + def __init__(self, stream): + Reader.__init__(self, stream) + Scanner.__init__(self) + Parser.__init__(self) + Composer.__init__(self) + Constructor.__init__(self) + Resolver.__init__(self) + diff --git a/python.d/python_modules/pyyaml3/nodes.py b/python.d/python_modules/pyyaml3/nodes.py new file mode 100644 index 000000000..c4f070c41 --- /dev/null +++ b/python.d/python_modules/pyyaml3/nodes.py @@ -0,0 +1,49 @@ + +class Node(object): + def __init__(self, tag, value, start_mark, end_mark): + self.tag = tag + self.value = value + self.start_mark = start_mark + self.end_mark = end_mark + def __repr__(self): + value = self.value + #if isinstance(value, list): + # if len(value) == 0: + # value = '<empty>' + # elif len(value) == 1: + # value = '<1 item>' + # else: + # value = '<%d items>' % len(value) + #else: + # if len(value) > 75: + # value = repr(value[:70]+u' ... ') + # else: + # value = repr(value) + value = repr(value) + return '%s(tag=%r, value=%s)' % (self.__class__.__name__, self.tag, value) + +class ScalarNode(Node): + id = 'scalar' + def __init__(self, tag, value, + start_mark=None, end_mark=None, style=None): + self.tag = tag + self.value = value + self.start_mark = start_mark + self.end_mark = end_mark + self.style = style + +class CollectionNode(Node): + def __init__(self, tag, value, + start_mark=None, end_mark=None, flow_style=None): + self.tag = tag + self.value = value + self.start_mark = start_mark + self.end_mark = end_mark + self.flow_style = flow_style + +class SequenceNode(CollectionNode): + id = 'sequence' + +class MappingNode(CollectionNode): + id = 'mapping' + diff --git a/python.d/python_modules/pyyaml3/parser.py b/python.d/python_modules/pyyaml3/parser.py new file mode 100644 index 000000000..13a5995d2 --- /dev/null +++ b/python.d/python_modules/pyyaml3/parser.py @@ -0,0 +1,589 @@ + +# The following YAML grammar is LL(1) and is parsed by a recursive descent +# parser. +# +# stream ::= STREAM-START implicit_document? explicit_document* STREAM-END +# implicit_document ::= block_node DOCUMENT-END* +# explicit_document ::= DIRECTIVE* DOCUMENT-START block_node? DOCUMENT-END* +# block_node_or_indentless_sequence ::= +# ALIAS +# | properties (block_content | indentless_block_sequence)? +# | block_content +# | indentless_block_sequence +# block_node ::= ALIAS +# | properties block_content? +# | block_content +# flow_node ::= ALIAS +# | properties flow_content? +# | flow_content +# properties ::= TAG ANCHOR? | ANCHOR TAG? +# block_content ::= block_collection | flow_collection | SCALAR +# flow_content ::= flow_collection | SCALAR +# block_collection ::= block_sequence | block_mapping +# flow_collection ::= flow_sequence | flow_mapping +# block_sequence ::= BLOCK-SEQUENCE-START (BLOCK-ENTRY block_node?)* BLOCK-END +# indentless_sequence ::= (BLOCK-ENTRY block_node?)+ +# block_mapping ::= BLOCK-MAPPING_START +# ((KEY block_node_or_indentless_sequence?)? +# (VALUE block_node_or_indentless_sequence?)?)* +# BLOCK-END +# flow_sequence ::= FLOW-SEQUENCE-START +# (flow_sequence_entry FLOW-ENTRY)* +# flow_sequence_entry? +# FLOW-SEQUENCE-END +# flow_sequence_entry ::= flow_node | KEY flow_node? (VALUE flow_node?)? +# flow_mapping ::= FLOW-MAPPING-START +# (flow_mapping_entry FLOW-ENTRY)* +# flow_mapping_entry? +# FLOW-MAPPING-END +# flow_mapping_entry ::= flow_node | KEY flow_node? (VALUE flow_node?)? +# +# FIRST sets: +# +# stream: { STREAM-START } +# explicit_document: { DIRECTIVE DOCUMENT-START } +# implicit_document: FIRST(block_node) +# block_node: { ALIAS TAG ANCHOR SCALAR BLOCK-SEQUENCE-START BLOCK-MAPPING-START FLOW-SEQUENCE-START FLOW-MAPPING-START } +# flow_node: { ALIAS ANCHOR TAG SCALAR FLOW-SEQUENCE-START FLOW-MAPPING-START } +# block_content: { BLOCK-SEQUENCE-START BLOCK-MAPPING-START FLOW-SEQUENCE-START FLOW-MAPPING-START SCALAR } +# flow_content: { FLOW-SEQUENCE-START FLOW-MAPPING-START SCALAR } +# block_collection: { BLOCK-SEQUENCE-START BLOCK-MAPPING-START } +# flow_collection: { FLOW-SEQUENCE-START FLOW-MAPPING-START } +# block_sequence: { BLOCK-SEQUENCE-START } +# block_mapping: { BLOCK-MAPPING-START } +# block_node_or_indentless_sequence: { ALIAS ANCHOR TAG SCALAR BLOCK-SEQUENCE-START BLOCK-MAPPING-START FLOW-SEQUENCE-START FLOW-MAPPING-START BLOCK-ENTRY } +# indentless_sequence: { ENTRY } +# flow_collection: { FLOW-SEQUENCE-START FLOW-MAPPING-START } +# flow_sequence: { FLOW-SEQUENCE-START } +# flow_mapping: { FLOW-MAPPING-START } +# flow_sequence_entry: { ALIAS ANCHOR TAG SCALAR FLOW-SEQUENCE-START FLOW-MAPPING-START KEY } +# flow_mapping_entry: { ALIAS ANCHOR TAG SCALAR FLOW-SEQUENCE-START FLOW-MAPPING-START KEY } + +__all__ = ['Parser', 'ParserError'] + +from .error import MarkedYAMLError +from .tokens import * +from .events import * +from .scanner import * + +class ParserError(MarkedYAMLError): + pass + +class Parser: + # Since writing a recursive-descendant parser is a straightforward task, we + # do not give many comments here. + + DEFAULT_TAGS = { + '!': '!', + '!!': 'tag:yaml.org,2002:', + } + + def __init__(self): + self.current_event = None + self.yaml_version = None + self.tag_handles = {} + self.states = [] + self.marks = [] + self.state = self.parse_stream_start + + def dispose(self): + # Reset the state attributes (to clear self-references) + self.states = [] + self.state = None + + def check_event(self, *choices): + # Check the type of the next event. + if self.current_event is None: + if self.state: + self.current_event = self.state() + if self.current_event is not None: + if not choices: + return True + for choice in choices: + if isinstance(self.current_event, choice): + return True + return False + + def peek_event(self): + # Get the next event. + if self.current_event is None: + if self.state: + self.current_event = self.state() + return self.current_event + + def get_event(self): + # Get the next event and proceed further. + if self.current_event is None: + if self.state: + self.current_event = self.state() + value = self.current_event + self.current_event = None + return value + + # stream ::= STREAM-START implicit_document? explicit_document* STREAM-END + # implicit_document ::= block_node DOCUMENT-END* + # explicit_document ::= DIRECTIVE* DOCUMENT-START block_node? DOCUMENT-END* + + def parse_stream_start(self): + + # Parse the stream start. + token = self.get_token() + event = StreamStartEvent(token.start_mark, token.end_mark, + encoding=token.encoding) + + # Prepare the next state. + self.state = self.parse_implicit_document_start + + return event + + def parse_implicit_document_start(self): + + # Parse an implicit document. + if not self.check_token(DirectiveToken, DocumentStartToken, + StreamEndToken): + self.tag_handles = self.DEFAULT_TAGS + token = self.peek_token() + start_mark = end_mark = token.start_mark + event = DocumentStartEvent(start_mark, end_mark, + explicit=False) + + # Prepare the next state. + self.states.append(self.parse_document_end) + self.state = self.parse_block_node + + return event + + else: + return self.parse_document_start() + + def parse_document_start(self): + + # Parse any extra document end indicators. + while self.check_token(DocumentEndToken): + self.get_token() + + # Parse an explicit document. + if not self.check_token(StreamEndToken): + token = self.peek_token() + start_mark = token.start_mark + version, tags = self.process_directives() + if not self.check_token(DocumentStartToken): + raise ParserError(None, None, + "expected '<document start>', but found %r" + % self.peek_token().id, + self.peek_token().start_mark) + token = self.get_token() + end_mark = token.end_mark + event = DocumentStartEvent(start_mark, end_mark, + explicit=True, version=version, tags=tags) + self.states.append(self.parse_document_end) + self.state = self.parse_document_content + else: + # Parse the end of the stream. + token = self.get_token() + event = StreamEndEvent(token.start_mark, token.end_mark) + assert not self.states + assert not self.marks + self.state = None + return event + + def parse_document_end(self): + + # Parse the document end. + token = self.peek_token() + start_mark = end_mark = token.start_mark + explicit = False + if self.check_token(DocumentEndToken): + token = self.get_token() + end_mark = token.end_mark + explicit = True + event = DocumentEndEvent(start_mark, end_mark, + explicit=explicit) + + # Prepare the next state. + self.state = self.parse_document_start + + return event + + def parse_document_content(self): + if self.check_token(DirectiveToken, + DocumentStartToken, DocumentEndToken, StreamEndToken): + event = self.process_empty_scalar(self.peek_token().start_mark) + self.state = self.states.pop() + return event + else: + return self.parse_block_node() + + def process_directives(self): + self.yaml_version = None + self.tag_handles = {} + while self.check_token(DirectiveToken): + token = self.get_token() + if token.name == 'YAML': + if self.yaml_version is not None: + raise ParserError(None, None, + "found duplicate YAML directive", token.start_mark) + major, minor = token.value + if major != 1: + raise ParserError(None, None, + "found incompatible YAML document (version 1.* is required)", + token.start_mark) + self.yaml_version = token.value + elif token.name == 'TAG': + handle, prefix = token.value + if handle in self.tag_handles: + raise ParserError(None, None, + "duplicate tag handle %r" % handle, + token.start_mark) + self.tag_handles[handle] = prefix + if self.tag_handles: + value = self.yaml_version, self.tag_handles.copy() + else: + value = self.yaml_version, None + for key in self.DEFAULT_TAGS: + if key not in self.tag_handles: + self.tag_handles[key] = self.DEFAULT_TAGS[key] + return value + + # block_node_or_indentless_sequence ::= ALIAS + # | properties (block_content | indentless_block_sequence)? + # | block_content + # | indentless_block_sequence + # block_node ::= ALIAS + # | properties block_content? + # | block_content + # flow_node ::= ALIAS + # | properties flow_content? + # | flow_content + # properties ::= TAG ANCHOR? | ANCHOR TAG? + # block_content ::= block_collection | flow_collection | SCALAR + # flow_content ::= flow_collection | SCALAR + # block_collection ::= block_sequence | block_mapping + # flow_collection ::= flow_sequence | flow_mapping + + def parse_block_node(self): + return self.parse_node(block=True) + + def parse_flow_node(self): + return self.parse_node() + + def parse_block_node_or_indentless_sequence(self): + return self.parse_node(block=True, indentless_sequence=True) + + def parse_node(self, block=False, indentless_sequence=False): + if self.check_token(AliasToken): + token = self.get_token() + event = AliasEvent(token.value, token.start_mark, token.end_mark) + self.state = self.states.pop() + else: + anchor = None + tag = None + start_mark = end_mark = tag_mark = None + if self.check_token(AnchorToken): + token = self.get_token() + start_mark = token.start_mark + end_mark = token.end_mark + anchor = token.value + if self.check_token(TagToken): + token = self.get_token() + tag_mark = token.start_mark + end_mark = token.end_mark + tag = token.value + elif self.check_token(TagToken): + token = self.get_token() + start_mark = tag_mark = token.start_mark + end_mark = token.end_mark + tag = token.value + if self.check_token(AnchorToken): + token = self.get_token() + end_mark = token.end_mark + anchor = token.value + if tag is not None: + handle, suffix = tag + if handle is not None: + if handle not in self.tag_handles: + raise ParserError("while parsing a node", start_mark, + "found undefined tag handle %r" % handle, + tag_mark) + tag = self.tag_handles[handle]+suffix + else: + tag = suffix + #if tag == '!': + # raise ParserError("while parsing a node", start_mark, + # "found non-specific tag '!'", tag_mark, + # "Please check 'http://pyyaml.org/wiki/YAMLNonSpecificTag' and share your opinion.") + if start_mark is None: + start_mark = end_mark = self.peek_token().start_mark + event = None + implicit = (tag is None or tag == '!') + if indentless_sequence and self.check_token(BlockEntryToken): + end_mark = self.peek_token().end_mark + event = SequenceStartEvent(anchor, tag, implicit, + start_mark, end_mark) + self.state = self.parse_indentless_sequence_entry + else: + if self.check_token(ScalarToken): + token = self.get_token() + end_mark = token.end_mark + if (token.plain and tag is None) or tag == '!': + implicit = (True, False) + elif tag is None: + implicit = (False, True) + else: + implicit = (False, False) + event = ScalarEvent(anchor, tag, implicit, token.value, + start_mark, end_mark, style=token.style) + self.state = self.states.pop() + elif self.check_token(FlowSequenceStartToken): + end_mark = self.peek_token().end_mark + event = SequenceStartEvent(anchor, tag, implicit, + start_mark, end_mark, flow_style=True) + self.state = self.parse_flow_sequence_first_entry + elif self.check_token(FlowMappingStartToken): + end_mark = self.peek_token().end_mark + event = MappingStartEvent(anchor, tag, implicit, + start_mark, end_mark, flow_style=True) + self.state = self.parse_flow_mapping_first_key + elif block and self.check_token(BlockSequenceStartToken): + end_mark = self.peek_token().start_mark + event = SequenceStartEvent(anchor, tag, implicit, + start_mark, end_mark, flow_style=False) + self.state = self.parse_block_sequence_first_entry + elif block and self.check_token(BlockMappingStartToken): + end_mark = self.peek_token().start_mark + event = MappingStartEvent(anchor, tag, implicit, + start_mark, end_mark, flow_style=False) + self.state = self.parse_block_mapping_first_key + elif anchor is not None or tag is not None: + # Empty scalars are allowed even if a tag or an anchor is + # specified. + event = ScalarEvent(anchor, tag, (implicit, False), '', + start_mark, end_mark) + self.state = self.states.pop() + else: + if block: + node = 'block' + else: + node = 'flow' + token = self.peek_token() + raise ParserError("while parsing a %s node" % node, start_mark, + "expected the node content, but found %r" % token.id, + token.start_mark) + return event + + # block_sequence ::= BLOCK-SEQUENCE-START (BLOCK-ENTRY block_node?)* BLOCK-END + + def parse_block_sequence_first_entry(self): + token = self.get_token() + self.marks.append(token.start_mark) + return self.parse_block_sequence_entry() + + def parse_block_sequence_entry(self): + if self.check_token(BlockEntryToken): + token = self.get_token() + if not self.check_token(BlockEntryToken, BlockEndToken): + self.states.append(self.parse_block_sequence_entry) + return self.parse_block_node() + else: + self.state = self.parse_block_sequence_entry + return self.process_empty_scalar(token.end_mark) + if not self.check_token(BlockEndToken): + token = self.peek_token() + raise ParserError("while parsing a block collection", self.marks[-1], + "expected <block end>, but found %r" % token.id, token.start_mark) + token = self.get_token() + event = SequenceEndEvent(token.start_mark, token.end_mark) + self.state = self.states.pop() + self.marks.pop() + return event + + # indentless_sequence ::= (BLOCK-ENTRY block_node?)+ + + def parse_indentless_sequence_entry(self): + if self.check_token(BlockEntryToken): + token = self.get_token() + if not self.check_token(BlockEntryToken, + KeyToken, ValueToken, BlockEndToken): + self.states.append(self.parse_indentless_sequence_entry) + return self.parse_block_node() + else: + self.state = self.parse_indentless_sequence_entry + return self.process_empty_scalar(token.end_mark) + token = self.peek_token() + event = SequenceEndEvent(token.start_mark, token.start_mark) + self.state = self.states.pop() + return event + + # block_mapping ::= BLOCK-MAPPING_START + # ((KEY block_node_or_indentless_sequence?)? + # (VALUE block_node_or_indentless_sequence?)?)* + # BLOCK-END + + def parse_block_mapping_first_key(self): + token = self.get_token() + self.marks.append(token.start_mark) + return self.parse_block_mapping_key() + + def parse_block_mapping_key(self): + if self.check_token(KeyToken): + token = self.get_token() + if not self.check_token(KeyToken, ValueToken, BlockEndToken): + self.states.append(self.parse_block_mapping_value) + return self.parse_block_node_or_indentless_sequence() + else: + self.state = self.parse_block_mapping_value + return self.process_empty_scalar(token.end_mark) + if not self.check_token(BlockEndToken): + token = self.peek_token() + raise ParserError("while parsing a block mapping", self.marks[-1], + "expected <block end>, but found %r" % token.id, token.start_mark) + token = self.get_token() + event = MappingEndEvent(token.start_mark, token.end_mark) + self.state = self.states.pop() + self.marks.pop() + return event + + def parse_block_mapping_value(self): + if self.check_token(ValueToken): + token = self.get_token() + if not self.check_token(KeyToken, ValueToken, BlockEndToken): + self.states.append(self.parse_block_mapping_key) + return self.parse_block_node_or_indentless_sequence() + else: + self.state = self.parse_block_mapping_key + return self.process_empty_scalar(token.end_mark) + else: + self.state = self.parse_block_mapping_key + token = self.peek_token() + return self.process_empty_scalar(token.start_mark) + + # flow_sequence ::= FLOW-SEQUENCE-START + # (flow_sequence_entry FLOW-ENTRY)* + # flow_sequence_entry? + # FLOW-SEQUENCE-END + # flow_sequence_entry ::= flow_node | KEY flow_node? (VALUE flow_node?)? + # + # Note that while production rules for both flow_sequence_entry and + # flow_mapping_entry are equal, their interpretations are different. + # For `flow_sequence_entry`, the part `KEY flow_node? (VALUE flow_node?)?` + # generate an inline mapping (set syntax). + + def parse_flow_sequence_first_entry(self): + token = self.get_token() + self.marks.append(token.start_mark) + return self.parse_flow_sequence_entry(first=True) + + def parse_flow_sequence_entry(self, first=False): + if not self.check_token(FlowSequenceEndToken): + if not first: + if self.check_token(FlowEntryToken): + self.get_token() + else: + token = self.peek_token() + raise ParserError("while parsing a flow sequence", self.marks[-1], + "expected ',' or ']', but got %r" % token.id, token.start_mark) + + if self.check_token(KeyToken): + token = self.peek_token() + event = MappingStartEvent(None, None, True, + token.start_mark, token.end_mark, + flow_style=True) + self.state = self.parse_flow_sequence_entry_mapping_key + return event + elif not self.check_token(FlowSequenceEndToken): + self.states.append(self.parse_flow_sequence_entry) + return self.parse_flow_node() + token = self.get_token() + event = SequenceEndEvent(token.start_mark, token.end_mark) + self.state = self.states.pop() + self.marks.pop() + return event + + def parse_flow_sequence_entry_mapping_key(self): + token = self.get_token() + if not self.check_token(ValueToken, + FlowEntryToken, FlowSequenceEndToken): + self.states.append(self.parse_flow_sequence_entry_mapping_value) + return self.parse_flow_node() + else: + self.state = self.parse_flow_sequence_entry_mapping_value + return self.process_empty_scalar(token.end_mark) + + def parse_flow_sequence_entry_mapping_value(self): + if self.check_token(ValueToken): + token = self.get_token() + if not self.check_token(FlowEntryToken, FlowSequenceEndToken): + self.states.append(self.parse_flow_sequence_entry_mapping_end) + return self.parse_flow_node() + else: + self.state = self.parse_flow_sequence_entry_mapping_end + return self.process_empty_scalar(token.end_mark) + else: + self.state = self.parse_flow_sequence_entry_mapping_end + token = self.peek_token() + return self.process_empty_scalar(token.start_mark) + + def parse_flow_sequence_entry_mapping_end(self): + self.state = self.parse_flow_sequence_entry + token = self.peek_token() + return MappingEndEvent(token.start_mark, token.start_mark) + + # flow_mapping ::= FLOW-MAPPING-START + # (flow_mapping_entry FLOW-ENTRY)* + # flow_mapping_entry? + # FLOW-MAPPING-END + # flow_mapping_entry ::= flow_node | KEY flow_node? (VALUE flow_node?)? + + def parse_flow_mapping_first_key(self): + token = self.get_token() + self.marks.append(token.start_mark) + return self.parse_flow_mapping_key(first=True) + + def parse_flow_mapping_key(self, first=False): + if not self.check_token(FlowMappingEndToken): + if not first: + if self.check_token(FlowEntryToken): + self.get_token() + else: + token = self.peek_token() + raise ParserError("while parsing a flow mapping", self.marks[-1], + "expected ',' or '}', but got %r" % token.id, token.start_mark) + if self.check_token(KeyToken): + token = self.get_token() + if not self.check_token(ValueToken, + FlowEntryToken, FlowMappingEndToken): + self.states.append(self.parse_flow_mapping_value) + return self.parse_flow_node() + else: + self.state = self.parse_flow_mapping_value + return self.process_empty_scalar(token.end_mark) + elif not self.check_token(FlowMappingEndToken): + self.states.append(self.parse_flow_mapping_empty_value) + return self.parse_flow_node() + token = self.get_token() + event = MappingEndEvent(token.start_mark, token.end_mark) + self.state = self.states.pop() + self.marks.pop() + return event + + def parse_flow_mapping_value(self): + if self.check_token(ValueToken): + token = self.get_token() + if not self.check_token(FlowEntryToken, FlowMappingEndToken): + self.states.append(self.parse_flow_mapping_key) + return self.parse_flow_node() + else: + self.state = self.parse_flow_mapping_key + return self.process_empty_scalar(token.end_mark) + else: + self.state = self.parse_flow_mapping_key + token = self.peek_token() + return self.process_empty_scalar(token.start_mark) + + def parse_flow_mapping_empty_value(self): + self.state = self.parse_flow_mapping_key + return self.process_empty_scalar(self.peek_token().start_mark) + + def process_empty_scalar(self, mark): + return ScalarEvent(None, None, (True, False), '', mark, mark) + diff --git a/python.d/python_modules/pyyaml3/reader.py b/python.d/python_modules/pyyaml3/reader.py new file mode 100644 index 000000000..f70e920f4 --- /dev/null +++ b/python.d/python_modules/pyyaml3/reader.py @@ -0,0 +1,192 @@ +# This module contains abstractions for the input stream. You don't have to +# looks further, there are no pretty code. +# +# We define two classes here. +# +# Mark(source, line, column) +# It's just a record and its only use is producing nice error messages. +# Parser does not use it for any other purposes. +# +# Reader(source, data) +# Reader determines the encoding of `data` and converts it to unicode. +# Reader provides the following methods and attributes: +# reader.peek(length=1) - return the next `length` characters +# reader.forward(length=1) - move the current position to `length` characters. +# reader.index - the number of the current character. +# reader.line, stream.column - the line and the column of the current character. + +__all__ = ['Reader', 'ReaderError'] + +from .error import YAMLError, Mark + +import codecs, re + +class ReaderError(YAMLError): + + def __init__(self, name, position, character, encoding, reason): + self.name = name + self.character = character + self.position = position + self.encoding = encoding + self.reason = reason + + def __str__(self): + if isinstance(self.character, bytes): + return "'%s' codec can't decode byte #x%02x: %s\n" \ + " in \"%s\", position %d" \ + % (self.encoding, ord(self.character), self.reason, + self.name, self.position) + else: + return "unacceptable character #x%04x: %s\n" \ + " in \"%s\", position %d" \ + % (self.character, self.reason, + self.name, self.position) + +class Reader(object): + # Reader: + # - determines the data encoding and converts it to a unicode string, + # - checks if characters are in allowed range, + # - adds '\0' to the end. + + # Reader accepts + # - a `bytes` object, + # - a `str` object, + # - a file-like object with its `read` method returning `str`, + # - a file-like object with its `read` method returning `unicode`. + + # Yeah, it's ugly and slow. + + def __init__(self, stream): + self.name = None + self.stream = None + self.stream_pointer = 0 + self.eof = True + self.buffer = '' + self.pointer = 0 + self.raw_buffer = None + self.raw_decode = None + self.encoding = None + self.index = 0 + self.line = 0 + self.column = 0 + if isinstance(stream, str): + self.name = "<unicode string>" + self.check_printable(stream) + self.buffer = stream+'\0' + elif isinstance(stream, bytes): + self.name = "<byte string>" + self.raw_buffer = stream + self.determine_encoding() + else: + self.stream = stream + self.name = getattr(stream, 'name', "<file>") + self.eof = False + self.raw_buffer = None + self.determine_encoding() + + def peek(self, index=0): + try: + return self.buffer[self.pointer+index] + except IndexError: + self.update(index+1) + return self.buffer[self.pointer+index] + + def prefix(self, length=1): + if self.pointer+length >= len(self.buffer): + self.update(length) + return self.buffer[self.pointer:self.pointer+length] + + def forward(self, length=1): + if self.pointer+length+1 >= len(self.buffer): + self.update(length+1) + while length: + ch = self.buffer[self.pointer] + self.pointer += 1 + self.index += 1 + if ch in '\n\x85\u2028\u2029' \ + or (ch == '\r' and self.buffer[self.pointer] != '\n'): + self.line += 1 + self.column = 0 + elif ch != '\uFEFF': + self.column += 1 + length -= 1 + + def get_mark(self): + if self.stream is None: + return Mark(self.name, self.index, self.line, self.column, + self.buffer, self.pointer) + else: + return Mark(self.name, self.index, self.line, self.column, + None, None) + + def determine_encoding(self): + while not self.eof and (self.raw_buffer is None or len(self.raw_buffer) < 2): + self.update_raw() + if isinstance(self.raw_buffer, bytes): + if self.raw_buffer.startswith(codecs.BOM_UTF16_LE): + self.raw_decode = codecs.utf_16_le_decode + self.encoding = 'utf-16-le' + elif self.raw_buffer.startswith(codecs.BOM_UTF16_BE): + self.raw_decode = codecs.utf_16_be_decode + self.encoding = 'utf-16-be' + else: + self.raw_decode = codecs.utf_8_decode + self.encoding = 'utf-8' + self.update(1) + + NON_PRINTABLE = re.compile('[^\x09\x0A\x0D\x20-\x7E\x85\xA0-\uD7FF\uE000-\uFFFD]') + def check_printable(self, data): + match = self.NON_PRINTABLE.search(data) + if match: + character = match.group() + position = self.index+(len(self.buffer)-self.pointer)+match.start() + raise ReaderError(self.name, position, ord(character), + 'unicode', "special characters are not allowed") + + def update(self, length): + if self.raw_buffer is None: + return + self.buffer = self.buffer[self.pointer:] + self.pointer = 0 + while len(self.buffer) < length: + if not self.eof: + self.update_raw() + if self.raw_decode is not None: + try: + data, converted = self.raw_decode(self.raw_buffer, + 'strict', self.eof) + except UnicodeDecodeError as exc: + character = self.raw_buffer[exc.start] + if self.stream is not None: + position = self.stream_pointer-len(self.raw_buffer)+exc.start + else: + position = exc.start + raise ReaderError(self.name, position, character, + exc.encoding, exc.reason) + else: + data = self.raw_buffer + converted = len(data) + self.check_printable(data) + self.buffer += data + self.raw_buffer = self.raw_buffer[converted:] + if self.eof: + self.buffer += '\0' + self.raw_buffer = None + break + + def update_raw(self, size=4096): + data = self.stream.read(size) + if self.raw_buffer is None: + self.raw_buffer = data + else: + self.raw_buffer += data + self.stream_pointer += len(data) + if not data: + self.eof = True + +#try: +# import psyco +# psyco.bind(Reader) +#except ImportError: +# pass + diff --git a/python.d/python_modules/pyyaml3/representer.py b/python.d/python_modules/pyyaml3/representer.py new file mode 100644 index 000000000..67cd6fd25 --- /dev/null +++ b/python.d/python_modules/pyyaml3/representer.py @@ -0,0 +1,374 @@ + +__all__ = ['BaseRepresenter', 'SafeRepresenter', 'Representer', + 'RepresenterError'] + +from .error import * +from .nodes import * + +import datetime, sys, copyreg, types, base64 + +class RepresenterError(YAMLError): + pass + +class BaseRepresenter: + + yaml_representers = {} + yaml_multi_representers = {} + + def __init__(self, default_style=None, default_flow_style=None): + self.default_style = default_style + self.default_flow_style = default_flow_style + self.represented_objects = {} + self.object_keeper = [] + self.alias_key = None + + def represent(self, data): + node = self.represent_data(data) + self.serialize(node) + self.represented_objects = {} + self.object_keeper = [] + self.alias_key = None + + def represent_data(self, data): + if self.ignore_aliases(data): + self.alias_key = None + else: + self.alias_key = id(data) + if self.alias_key is not None: + if self.alias_key in self.represented_objects: + node = self.represented_objects[self.alias_key] + #if node is None: + # raise RepresenterError("recursive objects are not allowed: %r" % data) + return node + #self.represented_objects[alias_key] = None + self.object_keeper.append(data) + data_types = type(data).__mro__ + if data_types[0] in self.yaml_representers: + node = self.yaml_representers[data_types[0]](self, data) + else: + for data_type in data_types: + if data_type in self.yaml_multi_representers: + node = self.yaml_multi_representers[data_type](self, data) + break + else: + if None in self.yaml_multi_representers: + node = self.yaml_multi_representers[None](self, data) + elif None in self.yaml_representers: + node = self.yaml_representers[None](self, data) + else: + node = ScalarNode(None, str(data)) + #if alias_key is not None: + # self.represented_objects[alias_key] = node + return node + + @classmethod + def add_representer(cls, data_type, representer): + if not 'yaml_representers' in cls.__dict__: + cls.yaml_representers = cls.yaml_representers.copy() + cls.yaml_representers[data_type] = representer + + @classmethod + def add_multi_representer(cls, data_type, representer): + if not 'yaml_multi_representers' in cls.__dict__: + cls.yaml_multi_representers = cls.yaml_multi_representers.copy() + cls.yaml_multi_representers[data_type] = representer + + def represent_scalar(self, tag, value, style=None): + if style is None: + style = self.default_style + node = ScalarNode(tag, value, style=style) + if self.alias_key is not None: + self.represented_objects[self.alias_key] = node + return node + + def represent_sequence(self, tag, sequence, flow_style=None): + value = [] + node = SequenceNode(tag, value, flow_style=flow_style) + if self.alias_key is not None: + self.represented_objects[self.alias_key] = node + best_style = True + for item in sequence: + node_item = self.represent_data(item) + if not (isinstance(node_item, ScalarNode) and not node_item.style): + best_style = False + value.append(node_item) + if flow_style is None: + if self.default_flow_style is not None: + node.flow_style = self.default_flow_style + else: + node.flow_style = best_style + return node + + def represent_mapping(self, tag, mapping, flow_style=None): + value = [] + node = MappingNode(tag, value, flow_style=flow_style) + if self.alias_key is not None: + self.represented_objects[self.alias_key] = node + best_style = True + if hasattr(mapping, 'items'): + mapping = list(mapping.items()) + try: + mapping = sorted(mapping) + except TypeError: + pass + for item_key, item_value in mapping: + node_key = self.represent_data(item_key) + node_value = self.represent_data(item_value) + if not (isinstance(node_key, ScalarNode) and not node_key.style): + best_style = False + if not (isinstance(node_value, ScalarNode) and not node_value.style): + best_style = False + value.append((node_key, node_value)) + if flow_style is None: + if self.default_flow_style is not None: + node.flow_style = self.default_flow_style + else: + node.flow_style = best_style + return node + + def ignore_aliases(self, data): + return False + +class SafeRepresenter(BaseRepresenter): + + def ignore_aliases(self, data): + if data in [None, ()]: + return True + if isinstance(data, (str, bytes, bool, int, float)): + return True + + def represent_none(self, data): + return self.represent_scalar('tag:yaml.org,2002:null', 'null') + + def represent_str(self, data): + return self.represent_scalar('tag:yaml.org,2002:str', data) + + def represent_binary(self, data): + if hasattr(base64, 'encodebytes'): + data = base64.encodebytes(data).decode('ascii') + else: + data = base64.encodestring(data).decode('ascii') + return self.represent_scalar('tag:yaml.org,2002:binary', data, style='|') + + def represent_bool(self, data): + if data: + value = 'true' + else: + value = 'false' + return self.represent_scalar('tag:yaml.org,2002:bool', value) + + def represent_int(self, data): + return self.represent_scalar('tag:yaml.org,2002:int', str(data)) + + inf_value = 1e300 + while repr(inf_value) != repr(inf_value*inf_value): + inf_value *= inf_value + + def represent_float(self, data): + if data != data or (data == 0.0 and data == 1.0): + value = '.nan' + elif data == self.inf_value: + value = '.inf' + elif data == -self.inf_value: + value = '-.inf' + else: + value = repr(data).lower() + # Note that in some cases `repr(data)` represents a float number + # without the decimal parts. For instance: + # >>> repr(1e17) + # '1e17' + # Unfortunately, this is not a valid float representation according + # to the definition of the `!!float` tag. We fix this by adding + # '.0' before the 'e' symbol. + if '.' not in value and 'e' in value: + value = value.replace('e', '.0e', 1) + return self.represent_scalar('tag:yaml.org,2002:float', value) + + def represent_list(self, data): + #pairs = (len(data) > 0 and isinstance(data, list)) + #if pairs: + # for item in data: + # if not isinstance(item, tuple) or len(item) != 2: + # pairs = False + # break + #if not pairs: + return self.represent_sequence('tag:yaml.org,2002:seq', data) + #value = [] + #for item_key, item_value in data: + # value.append(self.represent_mapping(u'tag:yaml.org,2002:map', + # [(item_key, item_value)])) + #return SequenceNode(u'tag:yaml.org,2002:pairs', value) + + def represent_dict(self, data): + return self.represent_mapping('tag:yaml.org,2002:map', data) + + def represent_set(self, data): + value = {} + for key in data: + value[key] = None + return self.represent_mapping('tag:yaml.org,2002:set', value) + + def represent_date(self, data): + value = data.isoformat() + return self.represent_scalar('tag:yaml.org,2002:timestamp', value) + + def represent_datetime(self, data): + value = data.isoformat(' ') + return self.represent_scalar('tag:yaml.org,2002:timestamp', value) + + def represent_yaml_object(self, tag, data, cls, flow_style=None): + if hasattr(data, '__getstate__'): + state = data.__getstate__() + else: + state = data.__dict__.copy() + return self.represent_mapping(tag, state, flow_style=flow_style) + + def represent_undefined(self, data): + raise RepresenterError("cannot represent an object: %s" % data) + +SafeRepresenter.add_representer(type(None), + SafeRepresenter.represent_none) + +SafeRepresenter.add_representer(str, + SafeRepresenter.represent_str) + +SafeRepresenter.add_representer(bytes, + SafeRepresenter.represent_binary) + +SafeRepresenter.add_representer(bool, + SafeRepresenter.represent_bool) + +SafeRepresenter.add_representer(int, + SafeRepresenter.represent_int) + +SafeRepresenter.add_representer(float, + SafeRepresenter.represent_float) + +SafeRepresenter.add_representer(list, + SafeRepresenter.represent_list) + +SafeRepresenter.add_representer(tuple, + SafeRepresenter.represent_list) + +SafeRepresenter.add_representer(dict, + SafeRepresenter.represent_dict) + +SafeRepresenter.add_representer(set, + SafeRepresenter.represent_set) + +SafeRepresenter.add_representer(datetime.date, + SafeRepresenter.represent_date) + +SafeRepresenter.add_representer(datetime.datetime, + SafeRepresenter.represent_datetime) + +SafeRepresenter.add_representer(None, + SafeRepresenter.represent_undefined) + +class Representer(SafeRepresenter): + + def represent_complex(self, data): + if data.imag == 0.0: + data = '%r' % data.real + elif data.real == 0.0: + data = '%rj' % data.imag + elif data.imag > 0: + data = '%r+%rj' % (data.real, data.imag) + else: + data = '%r%rj' % (data.real, data.imag) + return self.represent_scalar('tag:yaml.org,2002:python/complex', data) + + def represent_tuple(self, data): + return self.represent_sequence('tag:yaml.org,2002:python/tuple', data) + + def represent_name(self, data): + name = '%s.%s' % (data.__module__, data.__name__) + return self.represent_scalar('tag:yaml.org,2002:python/name:'+name, '') + + def represent_module(self, data): + return self.represent_scalar( + 'tag:yaml.org,2002:python/module:'+data.__name__, '') + + def represent_object(self, data): + # We use __reduce__ API to save the data. data.__reduce__ returns + # a tuple of length 2-5: + # (function, args, state, listitems, dictitems) + + # For reconstructing, we calls function(*args), then set its state, + # listitems, and dictitems if they are not None. + + # A special case is when function.__name__ == '__newobj__'. In this + # case we create the object with args[0].__new__(*args). + + # Another special case is when __reduce__ returns a string - we don't + # support it. + + # We produce a !!python/object, !!python/object/new or + # !!python/object/apply node. + + cls = type(data) + if cls in copyreg.dispatch_table: + reduce = copyreg.dispatch_table[cls](data) + elif hasattr(data, '__reduce_ex__'): + reduce = data.__reduce_ex__(2) + elif hasattr(data, '__reduce__'): + reduce = data.__reduce__() + else: + raise RepresenterError("cannot represent object: %r" % data) + reduce = (list(reduce)+[None]*5)[:5] + function, args, state, listitems, dictitems = reduce + args = list(args) + if state is None: + state = {} + if listitems is not None: + listitems = list(listitems) + if dictitems is not None: + dictitems = dict(dictitems) + if function.__name__ == '__newobj__': + function = args[0] + args = args[1:] + tag = 'tag:yaml.org,2002:python/object/new:' + newobj = True + else: + tag = 'tag:yaml.org,2002:python/object/apply:' + newobj = False + function_name = '%s.%s' % (function.__module__, function.__name__) + if not args and not listitems and not dictitems \ + and isinstance(state, dict) and newobj: + return self.represent_mapping( + 'tag:yaml.org,2002:python/object:'+function_name, state) + if not listitems and not dictitems \ + and isinstance(state, dict) and not state: + return self.represent_sequence(tag+function_name, args) + value = {} + if args: + value['args'] = args + if state or not isinstance(state, dict): + value['state'] = state + if listitems: + value['listitems'] = listitems + if dictitems: + value['dictitems'] = dictitems + return self.represent_mapping(tag+function_name, value) + +Representer.add_representer(complex, + Representer.represent_complex) + +Representer.add_representer(tuple, + Representer.represent_tuple) + +Representer.add_representer(type, + Representer.represent_name) + +Representer.add_representer(types.FunctionType, + Representer.represent_name) + +Representer.add_representer(types.BuiltinFunctionType, + Representer.represent_name) + +Representer.add_representer(types.ModuleType, + Representer.represent_module) + +Representer.add_multi_representer(object, + Representer.represent_object) + diff --git a/python.d/python_modules/pyyaml3/resolver.py b/python.d/python_modules/pyyaml3/resolver.py new file mode 100644 index 000000000..0eece2582 --- /dev/null +++ b/python.d/python_modules/pyyaml3/resolver.py @@ -0,0 +1,224 @@ + +__all__ = ['BaseResolver', 'Resolver'] + +from .error import * +from .nodes import * + +import re + +class ResolverError(YAMLError): + pass + +class BaseResolver: + + DEFAULT_SCALAR_TAG = 'tag:yaml.org,2002:str' + DEFAULT_SEQUENCE_TAG = 'tag:yaml.org,2002:seq' + DEFAULT_MAPPING_TAG = 'tag:yaml.org,2002:map' + + yaml_implicit_resolvers = {} + yaml_path_resolvers = {} + + def __init__(self): + self.resolver_exact_paths = [] + self.resolver_prefix_paths = [] + + @classmethod + def add_implicit_resolver(cls, tag, regexp, first): + if not 'yaml_implicit_resolvers' in cls.__dict__: + cls.yaml_implicit_resolvers = cls.yaml_implicit_resolvers.copy() + if first is None: + first = [None] + for ch in first: + cls.yaml_implicit_resolvers.setdefault(ch, []).append((tag, regexp)) + + @classmethod + def add_path_resolver(cls, tag, path, kind=None): + # Note: `add_path_resolver` is experimental. The API could be changed. + # `new_path` is a pattern that is matched against the path from the + # root to the node that is being considered. `node_path` elements are + # tuples `(node_check, index_check)`. `node_check` is a node class: + # `ScalarNode`, `SequenceNode`, `MappingNode` or `None`. `None` + # matches any kind of a node. `index_check` could be `None`, a boolean + # value, a string value, or a number. `None` and `False` match against + # any _value_ of sequence and mapping nodes. `True` matches against + # any _key_ of a mapping node. A string `index_check` matches against + # a mapping value that corresponds to a scalar key which content is + # equal to the `index_check` value. An integer `index_check` matches + # against a sequence value with the index equal to `index_check`. + if not 'yaml_path_resolvers' in cls.__dict__: + cls.yaml_path_resolvers = cls.yaml_path_resolvers.copy() + new_path = [] + for element in path: + if isinstance(element, (list, tuple)): + if len(element) == 2: + node_check, index_check = element + elif len(element) == 1: + node_check = element[0] + index_check = True + else: + raise ResolverError("Invalid path element: %s" % element) + else: + node_check = None + index_check = element + if node_check is str: + node_check = ScalarNode + elif node_check is list: + node_check = SequenceNode + elif node_check is dict: + node_check = MappingNode + elif node_check not in [ScalarNode, SequenceNode, MappingNode] \ + and not isinstance(node_check, str) \ + and node_check is not None: + raise ResolverError("Invalid node checker: %s" % node_check) + if not isinstance(index_check, (str, int)) \ + and index_check is not None: + raise ResolverError("Invalid index checker: %s" % index_check) + new_path.append((node_check, index_check)) + if kind is str: + kind = ScalarNode + elif kind is list: + kind = SequenceNode + elif kind is dict: + kind = MappingNode + elif kind not in [ScalarNode, SequenceNode, MappingNode] \ + and kind is not None: + raise ResolverError("Invalid node kind: %s" % kind) + cls.yaml_path_resolvers[tuple(new_path), kind] = tag + + def descend_resolver(self, current_node, current_index): + if not self.yaml_path_resolvers: + return + exact_paths = {} + prefix_paths = [] + if current_node: + depth = len(self.resolver_prefix_paths) + for path, kind in self.resolver_prefix_paths[-1]: + if self.check_resolver_prefix(depth, path, kind, + current_node, current_index): + if len(path) > depth: + prefix_paths.append((path, kind)) + else: + exact_paths[kind] = self.yaml_path_resolvers[path, kind] + else: + for path, kind in self.yaml_path_resolvers: + if not path: + exact_paths[kind] = self.yaml_path_resolvers[path, kind] + else: + prefix_paths.append((path, kind)) + self.resolver_exact_paths.append(exact_paths) + self.resolver_prefix_paths.append(prefix_paths) + + def ascend_resolver(self): + if not self.yaml_path_resolvers: + return + self.resolver_exact_paths.pop() + self.resolver_prefix_paths.pop() + + def check_resolver_prefix(self, depth, path, kind, + current_node, current_index): + node_check, index_check = path[depth-1] + if isinstance(node_check, str): + if current_node.tag != node_check: + return + elif node_check is not None: + if not isinstance(current_node, node_check): + return + if index_check is True and current_index is not None: + return + if (index_check is False or index_check is None) \ + and current_index is None: + return + if isinstance(index_check, str): + if not (isinstance(current_index, ScalarNode) + and index_check == current_index.value): + return + elif isinstance(index_check, int) and not isinstance(index_check, bool): + if index_check != current_index: + return + return True + + def resolve(self, kind, value, implicit): + if kind is ScalarNode and implicit[0]: + if value == '': + resolvers = self.yaml_implicit_resolvers.get('', []) + else: + resolvers = self.yaml_implicit_resolvers.get(value[0], []) + resolvers += self.yaml_implicit_resolvers.get(None, []) + for tag, regexp in resolvers: + if regexp.match(value): + return tag + implicit = implicit[1] + if self.yaml_path_resolvers: + exact_paths = self.resolver_exact_paths[-1] + if kind in exact_paths: + return exact_paths[kind] + if None in exact_paths: + return exact_paths[None] + if kind is ScalarNode: + return self.DEFAULT_SCALAR_TAG + elif kind is SequenceNode: + return self.DEFAULT_SEQUENCE_TAG + elif kind is MappingNode: + return self.DEFAULT_MAPPING_TAG + +class Resolver(BaseResolver): + pass + +Resolver.add_implicit_resolver( + 'tag:yaml.org,2002:bool', + re.compile(r'''^(?:yes|Yes|YES|no|No|NO + |true|True|TRUE|false|False|FALSE + |on|On|ON|off|Off|OFF)$''', re.X), + list('yYnNtTfFoO')) + +Resolver.add_implicit_resolver( + 'tag:yaml.org,2002:float', + re.compile(r'''^(?:[-+]?(?:[0-9][0-9_]*)\.[0-9_]*(?:[eE][-+][0-9]+)? + |\.[0-9_]+(?:[eE][-+][0-9]+)? + |[-+]?[0-9][0-9_]*(?::[0-5]?[0-9])+\.[0-9_]* + |[-+]?\.(?:inf|Inf|INF) + |\.(?:nan|NaN|NAN))$''', re.X), + list('-+0123456789.')) + +Resolver.add_implicit_resolver( + 'tag:yaml.org,2002:int', + re.compile(r'''^(?:[-+]?0b[0-1_]+ + |[-+]?0[0-7_]+ + |[-+]?(?:0|[1-9][0-9_]*) + |[-+]?0x[0-9a-fA-F_]+ + |[-+]?[1-9][0-9_]*(?::[0-5]?[0-9])+)$''', re.X), + list('-+0123456789')) + +Resolver.add_implicit_resolver( + 'tag:yaml.org,2002:merge', + re.compile(r'^(?:<<)$'), + ['<']) + +Resolver.add_implicit_resolver( + 'tag:yaml.org,2002:null', + re.compile(r'''^(?: ~ + |null|Null|NULL + | )$''', re.X), + ['~', 'n', 'N', '']) + +Resolver.add_implicit_resolver( + 'tag:yaml.org,2002:timestamp', + re.compile(r'''^(?:[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9] + |[0-9][0-9][0-9][0-9] -[0-9][0-9]? -[0-9][0-9]? + (?:[Tt]|[ \t]+)[0-9][0-9]? + :[0-9][0-9] :[0-9][0-9] (?:\.[0-9]*)? + (?:[ \t]*(?:Z|[-+][0-9][0-9]?(?::[0-9][0-9])?))?)$''', re.X), + list('0123456789')) + +Resolver.add_implicit_resolver( + 'tag:yaml.org,2002:value', + re.compile(r'^(?:=)$'), + ['=']) + +# The following resolver is only for documentation purposes. It cannot work +# because plain scalars cannot start with '!', '&', or '*'. +Resolver.add_implicit_resolver( + 'tag:yaml.org,2002:yaml', + re.compile(r'^(?:!|&|\*)$'), + list('!&*')) + diff --git a/python.d/python_modules/pyyaml3/scanner.py b/python.d/python_modules/pyyaml3/scanner.py new file mode 100644 index 000000000..494d975ba --- /dev/null +++ b/python.d/python_modules/pyyaml3/scanner.py @@ -0,0 +1,1448 @@ + +# Scanner produces tokens of the following types: +# STREAM-START +# STREAM-END +# DIRECTIVE(name, value) +# DOCUMENT-START +# DOCUMENT-END +# BLOCK-SEQUENCE-START +# BLOCK-MAPPING-START +# BLOCK-END +# FLOW-SEQUENCE-START +# FLOW-MAPPING-START +# FLOW-SEQUENCE-END +# FLOW-MAPPING-END +# BLOCK-ENTRY +# FLOW-ENTRY +# KEY +# VALUE +# ALIAS(value) +# ANCHOR(value) +# TAG(value) +# SCALAR(value, plain, style) +# +# Read comments in the Scanner code for more details. +# + +__all__ = ['Scanner', 'ScannerError'] + +from .error import MarkedYAMLError +from .tokens import * + +class ScannerError(MarkedYAMLError): + pass + +class SimpleKey: + # See below simple keys treatment. + + def __init__(self, token_number, required, index, line, column, mark): + self.token_number = token_number + self.required = required + self.index = index + self.line = line + self.column = column + self.mark = mark + +class Scanner: + + def __init__(self): + """Initialize the scanner.""" + # It is assumed that Scanner and Reader will have a common descendant. + # Reader do the dirty work of checking for BOM and converting the + # input data to Unicode. It also adds NUL to the end. + # + # Reader supports the following methods + # self.peek(i=0) # peek the next i-th character + # self.prefix(l=1) # peek the next l characters + # self.forward(l=1) # read the next l characters and move the pointer. + + # Had we reached the end of the stream? + self.done = False + + # The number of unclosed '{' and '['. `flow_level == 0` means block + # context. + self.flow_level = 0 + + # List of processed tokens that are not yet emitted. + self.tokens = [] + + # Add the STREAM-START token. + self.fetch_stream_start() + + # Number of tokens that were emitted through the `get_token` method. + self.tokens_taken = 0 + + # The current indentation level. + self.indent = -1 + + # Past indentation levels. + self.indents = [] + + # Variables related to simple keys treatment. + + # A simple key is a key that is not denoted by the '?' indicator. + # Example of simple keys: + # --- + # block simple key: value + # ? not a simple key: + # : { flow simple key: value } + # We emit the KEY token before all keys, so when we find a potential + # simple key, we try to locate the corresponding ':' indicator. + # Simple keys should be limited to a single line and 1024 characters. + + # Can a simple key start at the current position? A simple key may + # start: + # - at the beginning of the line, not counting indentation spaces + # (in block context), + # - after '{', '[', ',' (in the flow context), + # - after '?', ':', '-' (in the block context). + # In the block context, this flag also signifies if a block collection + # may start at the current position. + self.allow_simple_key = True + + # Keep track of possible simple keys. This is a dictionary. The key + # is `flow_level`; there can be no more that one possible simple key + # for each level. The value is a SimpleKey record: + # (token_number, required, index, line, column, mark) + # A simple key may start with ALIAS, ANCHOR, TAG, SCALAR(flow), + # '[', or '{' tokens. + self.possible_simple_keys = {} + + # Public methods. + + def check_token(self, *choices): + # Check if the next token is one of the given types. + while self.need_more_tokens(): + self.fetch_more_tokens() + if self.tokens: + if not choices: + return True + for choice in choices: + if isinstance(self.tokens[0], choice): + return True + return False + + def peek_token(self): + # Return the next token, but do not delete if from the queue. + while self.need_more_tokens(): + self.fetch_more_tokens() + if self.tokens: + return self.tokens[0] + + def get_token(self): + # Return the next token. + while self.need_more_tokens(): + self.fetch_more_tokens() + if self.tokens: + self.tokens_taken += 1 + return self.tokens.pop(0) + + # Private methods. + + def need_more_tokens(self): + if self.done: + return False + if not self.tokens: + return True + # The current token may be a potential simple key, so we + # need to look further. + self.stale_possible_simple_keys() + if self.next_possible_simple_key() == self.tokens_taken: + return True + + def fetch_more_tokens(self): + + # Eat whitespaces and comments until we reach the next token. + self.scan_to_next_token() + + # Remove obsolete possible simple keys. + self.stale_possible_simple_keys() + + # Compare the current indentation and column. It may add some tokens + # and decrease the current indentation level. + self.unwind_indent(self.column) + + # Peek the next character. + ch = self.peek() + + # Is it the end of stream? + if ch == '\0': + return self.fetch_stream_end() + + # Is it a directive? + if ch == '%' and self.check_directive(): + return self.fetch_directive() + + # Is it the document start? + if ch == '-' and self.check_document_start(): + return self.fetch_document_start() + + # Is it the document end? + if ch == '.' and self.check_document_end(): + return self.fetch_document_end() + + # TODO: support for BOM within a stream. + #if ch == '\uFEFF': + # return self.fetch_bom() <-- issue BOMToken + + # Note: the order of the following checks is NOT significant. + + # Is it the flow sequence start indicator? + if ch == '[': + return self.fetch_flow_sequence_start() + + # Is it the flow mapping start indicator? + if ch == '{': + return self.fetch_flow_mapping_start() + + # Is it the flow sequence end indicator? + if ch == ']': + return self.fetch_flow_sequence_end() + + # Is it the flow mapping end indicator? + if ch == '}': + return self.fetch_flow_mapping_end() + + # Is it the flow entry indicator? + if ch == ',': + return self.fetch_flow_entry() + + # Is it the block entry indicator? + if ch == '-' and self.check_block_entry(): + return self.fetch_block_entry() + + # Is it the key indicator? + if ch == '?' and self.check_key(): + return self.fetch_key() + + # Is it the value indicator? + if ch == ':' and self.check_value(): + return self.fetch_value() + + # Is it an alias? + if ch == '*': + return self.fetch_alias() + + # Is it an anchor? + if ch == '&': + return self.fetch_anchor() + + # Is it a tag? + if ch == '!': + return self.fetch_tag() + + # Is it a literal scalar? + if ch == '|' and not self.flow_level: + return self.fetch_literal() + + # Is it a folded scalar? + if ch == '>' and not self.flow_level: + return self.fetch_folded() + + # Is it a single quoted scalar? + if ch == '\'': + return self.fetch_single() + + # Is it a double quoted scalar? + if ch == '\"': + return self.fetch_double() + + # It must be a plain scalar then. + if self.check_plain(): + return self.fetch_plain() + + # No? It's an error. Let's produce a nice error message. + raise ScannerError("while scanning for the next token", None, + "found character %r that cannot start any token" % ch, + self.get_mark()) + + # Simple keys treatment. + + def next_possible_simple_key(self): + # Return the number of the nearest possible simple key. Actually we + # don't need to loop through the whole dictionary. We may replace it + # with the following code: + # if not self.possible_simple_keys: + # return None + # return self.possible_simple_keys[ + # min(self.possible_simple_keys.keys())].token_number + min_token_number = None + for level in self.possible_simple_keys: + key = self.possible_simple_keys[level] + if min_token_number is None or key.token_number < min_token_number: + min_token_number = key.token_number + return min_token_number + + def stale_possible_simple_keys(self): + # Remove entries that are no longer possible simple keys. According to + # the YAML specification, simple keys + # - should be limited to a single line, + # - should be no longer than 1024 characters. + # Disabling this procedure will allow simple keys of any length and + # height (may cause problems if indentation is broken though). + for level in list(self.possible_simple_keys): + key = self.possible_simple_keys[level] + if key.line != self.line \ + or self.index-key.index > 1024: + if key.required: + raise ScannerError("while scanning a simple key", key.mark, + "could not found expected ':'", self.get_mark()) + del self.possible_simple_keys[level] + + def save_possible_simple_key(self): + # The next token may start a simple key. We check if it's possible + # and save its position. This function is called for + # ALIAS, ANCHOR, TAG, SCALAR(flow), '[', and '{'. + + # Check if a simple key is required at the current position. + required = not self.flow_level and self.indent == self.column + + # A simple key is required only if it is the first token in the current + # line. Therefore it is always allowed. + assert self.allow_simple_key or not required + + # The next token might be a simple key. Let's save it's number and + # position. + if self.allow_simple_key: + self.remove_possible_simple_key() + token_number = self.tokens_taken+len(self.tokens) + key = SimpleKey(token_number, required, + self.index, self.line, self.column, self.get_mark()) + self.possible_simple_keys[self.flow_level] = key + + def remove_possible_simple_key(self): + # Remove the saved possible key position at the current flow level. + if self.flow_level in self.possible_simple_keys: + key = self.possible_simple_keys[self.flow_level] + + if key.required: + raise ScannerError("while scanning a simple key", key.mark, + "could not found expected ':'", self.get_mark()) + + del self.possible_simple_keys[self.flow_level] + + # Indentation functions. + + def unwind_indent(self, column): + + ## In flow context, tokens should respect indentation. + ## Actually the condition should be `self.indent >= column` according to + ## the spec. But this condition will prohibit intuitively correct + ## constructions such as + ## key : { + ## } + #if self.flow_level and self.indent > column: + # raise ScannerError(None, None, + # "invalid intendation or unclosed '[' or '{'", + # self.get_mark()) + + # In the flow context, indentation is ignored. We make the scanner less + # restrictive then specification requires. + if self.flow_level: + return + + # In block context, we may need to issue the BLOCK-END tokens. + while self.indent > column: + mark = self.get_mark() + self.indent = self.indents.pop() + self.tokens.append(BlockEndToken(mark, mark)) + + def add_indent(self, column): + # Check if we need to increase indentation. + if self.indent < column: + self.indents.append(self.indent) + self.indent = column + return True + return False + + # Fetchers. + + def fetch_stream_start(self): + # We always add STREAM-START as the first token and STREAM-END as the + # last token. + + # Read the token. + mark = self.get_mark() + + # Add STREAM-START. + self.tokens.append(StreamStartToken(mark, mark, + encoding=self.encoding)) + + + def fetch_stream_end(self): + + # Set the current intendation to -1. + self.unwind_indent(-1) + + # Reset simple keys. + self.remove_possible_simple_key() + self.allow_simple_key = False + self.possible_simple_keys = {} + + # Read the token. + mark = self.get_mark() + + # Add STREAM-END. + self.tokens.append(StreamEndToken(mark, mark)) + + # The steam is finished. + self.done = True + + def fetch_directive(self): + + # Set the current intendation to -1. + self.unwind_indent(-1) + + # Reset simple keys. + self.remove_possible_simple_key() + self.allow_simple_key = False + + # Scan and add DIRECTIVE. + self.tokens.append(self.scan_directive()) + + def fetch_document_start(self): + self.fetch_document_indicator(DocumentStartToken) + + def fetch_document_end(self): + self.fetch_document_indicator(DocumentEndToken) + + def fetch_document_indicator(self, TokenClass): + + # Set the current intendation to -1. + self.unwind_indent(-1) + + # Reset simple keys. Note that there could not be a block collection + # after '---'. + self.remove_possible_simple_key() + self.allow_simple_key = False + + # Add DOCUMENT-START or DOCUMENT-END. + start_mark = self.get_mark() + self.forward(3) + end_mark = self.get_mark() + self.tokens.append(TokenClass(start_mark, end_mark)) + + def fetch_flow_sequence_start(self): + self.fetch_flow_collection_start(FlowSequenceStartToken) + + def fetch_flow_mapping_start(self): + self.fetch_flow_collection_start(FlowMappingStartToken) + + def fetch_flow_collection_start(self, TokenClass): + + # '[' and '{' may start a simple key. + self.save_possible_simple_key() + + # Increase the flow level. + self.flow_level += 1 + + # Simple keys are allowed after '[' and '{'. + self.allow_simple_key = True + + # Add FLOW-SEQUENCE-START or FLOW-MAPPING-START. + start_mark = self.get_mark() + self.forward() + end_mark = self.get_mark() + self.tokens.append(TokenClass(start_mark, end_mark)) + + def fetch_flow_sequence_end(self): + self.fetch_flow_collection_end(FlowSequenceEndToken) + + def fetch_flow_mapping_end(self): + self.fetch_flow_collection_end(FlowMappingEndToken) + + def fetch_flow_collection_end(self, TokenClass): + + # Reset possible simple key on the current level. + self.remove_possible_simple_key() + + # Decrease the flow level. + self.flow_level -= 1 + + # No simple keys after ']' or '}'. + self.allow_simple_key = False + + # Add FLOW-SEQUENCE-END or FLOW-MAPPING-END. + start_mark = self.get_mark() + self.forward() + end_mark = self.get_mark() + self.tokens.append(TokenClass(start_mark, end_mark)) + + def fetch_flow_entry(self): + + # Simple keys are allowed after ','. + self.allow_simple_key = True + + # Reset possible simple key on the current level. + self.remove_possible_simple_key() + + # Add FLOW-ENTRY. + start_mark = self.get_mark() + self.forward() + end_mark = self.get_mark() + self.tokens.append(FlowEntryToken(start_mark, end_mark)) + + def fetch_block_entry(self): + + # Block context needs additional checks. + if not self.flow_level: + + # Are we allowed to start a new entry? + if not self.allow_simple_key: + raise ScannerError(None, None, + "sequence entries are not allowed here", + self.get_mark()) + + # We may need to add BLOCK-SEQUENCE-START. + if self.add_indent(self.column): + mark = self.get_mark() + self.tokens.append(BlockSequenceStartToken(mark, mark)) + + # It's an error for the block entry to occur in the flow context, + # but we let the parser detect this. + else: + pass + + # Simple keys are allowed after '-'. + self.allow_simple_key = True + + # Reset possible simple key on the current level. + self.remove_possible_simple_key() + + # Add BLOCK-ENTRY. + start_mark = self.get_mark() + self.forward() + end_mark = self.get_mark() + self.tokens.append(BlockEntryToken(start_mark, end_mark)) + + def fetch_key(self): + + # Block context needs additional checks. + if not self.flow_level: + + # Are we allowed to start a key (not nessesary a simple)? + if not self.allow_simple_key: + raise ScannerError(None, None, + "mapping keys are not allowed here", + self.get_mark()) + + # We may need to add BLOCK-MAPPING-START. + if self.add_indent(self.column): + mark = self.get_mark() + self.tokens.append(BlockMappingStartToken(mark, mark)) + + # Simple keys are allowed after '?' in the block context. + self.allow_simple_key = not self.flow_level + + # Reset possible simple key on the current level. + self.remove_possible_simple_key() + + # Add KEY. + start_mark = self.get_mark() + self.forward() + end_mark = self.get_mark() + self.tokens.append(KeyToken(start_mark, end_mark)) + + def fetch_value(self): + + # Do we determine a simple key? + if self.flow_level in self.possible_simple_keys: + + # Add KEY. + key = self.possible_simple_keys[self.flow_level] + del self.possible_simple_keys[self.flow_level] + self.tokens.insert(key.token_number-self.tokens_taken, + KeyToken(key.mark, key.mark)) + + # If this key starts a new block mapping, we need to add + # BLOCK-MAPPING-START. + if not self.flow_level: + if self.add_indent(key.column): + self.tokens.insert(key.token_number-self.tokens_taken, + BlockMappingStartToken(key.mark, key.mark)) + + # There cannot be two simple keys one after another. + self.allow_simple_key = False + + # It must be a part of a complex key. + else: + + # Block context needs additional checks. + # (Do we really need them? They will be catched by the parser + # anyway.) + if not self.flow_level: + + # We are allowed to start a complex value if and only if + # we can start a simple key. + if not self.allow_simple_key: + raise ScannerError(None, None, + "mapping values are not allowed here", + self.get_mark()) + + # If this value starts a new block mapping, we need to add + # BLOCK-MAPPING-START. It will be detected as an error later by + # the parser. + if not self.flow_level: + if self.add_indent(self.column): + mark = self.get_mark() + self.tokens.append(BlockMappingStartToken(mark, mark)) + + # Simple keys are allowed after ':' in the block context. + self.allow_simple_key = not self.flow_level + + # Reset possible simple key on the current level. + self.remove_possible_simple_key() + + # Add VALUE. + start_mark = self.get_mark() + self.forward() + end_mark = self.get_mark() + self.tokens.append(ValueToken(start_mark, end_mark)) + + def fetch_alias(self): + + # ALIAS could be a simple key. + self.save_possible_simple_key() + + # No simple keys after ALIAS. + self.allow_simple_key = False + + # Scan and add ALIAS. + self.tokens.append(self.scan_anchor(AliasToken)) + + def fetch_anchor(self): + + # ANCHOR could start a simple key. + self.save_possible_simple_key() + + # No simple keys after ANCHOR. + self.allow_simple_key = False + + # Scan and add ANCHOR. + self.tokens.append(self.scan_anchor(AnchorToken)) + + def fetch_tag(self): + + # TAG could start a simple key. + self.save_possible_simple_key() + + # No simple keys after TAG. + self.allow_simple_key = False + + # Scan and add TAG. + self.tokens.append(self.scan_tag()) + + def fetch_literal(self): + self.fetch_block_scalar(style='|') + + def fetch_folded(self): + self.fetch_block_scalar(style='>') + + def fetch_block_scalar(self, style): + + # A simple key may follow a block scalar. + self.allow_simple_key = True + + # Reset possible simple key on the current level. + self.remove_possible_simple_key() + + # Scan and add SCALAR. + self.tokens.append(self.scan_block_scalar(style)) + + def fetch_single(self): + self.fetch_flow_scalar(style='\'') + + def fetch_double(self): + self.fetch_flow_scalar(style='"') + + def fetch_flow_scalar(self, style): + + # A flow scalar could be a simple key. + self.save_possible_simple_key() + + # No simple keys after flow scalars. + self.allow_simple_key = False + + # Scan and add SCALAR. + self.tokens.append(self.scan_flow_scalar(style)) + + def fetch_plain(self): + + # A plain scalar could be a simple key. + self.save_possible_simple_key() + + # No simple keys after plain scalars. But note that `scan_plain` will + # change this flag if the scan is finished at the beginning of the + # line. + self.allow_simple_key = False + + # Scan and add SCALAR. May change `allow_simple_key`. + self.tokens.append(self.scan_plain()) + + # Checkers. + + def check_directive(self): + + # DIRECTIVE: ^ '%' ... + # The '%' indicator is already checked. + if self.column == 0: + return True + + def check_document_start(self): + + # DOCUMENT-START: ^ '---' (' '|'\n') + if self.column == 0: + if self.prefix(3) == '---' \ + and self.peek(3) in '\0 \t\r\n\x85\u2028\u2029': + return True + + def check_document_end(self): + + # DOCUMENT-END: ^ '...' (' '|'\n') + if self.column == 0: + if self.prefix(3) == '...' \ + and self.peek(3) in '\0 \t\r\n\x85\u2028\u2029': + return True + + def check_block_entry(self): + + # BLOCK-ENTRY: '-' (' '|'\n') + return self.peek(1) in '\0 \t\r\n\x85\u2028\u2029' + + def check_key(self): + + # KEY(flow context): '?' + if self.flow_level: + return True + + # KEY(block context): '?' (' '|'\n') + else: + return self.peek(1) in '\0 \t\r\n\x85\u2028\u2029' + + def check_value(self): + + # VALUE(flow context): ':' + if self.flow_level: + return True + + # VALUE(block context): ':' (' '|'\n') + else: + return self.peek(1) in '\0 \t\r\n\x85\u2028\u2029' + + def check_plain(self): + + # A plain scalar may start with any non-space character except: + # '-', '?', ':', ',', '[', ']', '{', '}', + # '#', '&', '*', '!', '|', '>', '\'', '\"', + # '%', '@', '`'. + # + # It may also start with + # '-', '?', ':' + # if it is followed by a non-space character. + # + # Note that we limit the last rule to the block context (except the + # '-' character) because we want the flow context to be space + # independent. + ch = self.peek() + return ch not in '\0 \t\r\n\x85\u2028\u2029-?:,[]{}#&*!|>\'\"%@`' \ + or (self.peek(1) not in '\0 \t\r\n\x85\u2028\u2029' + and (ch == '-' or (not self.flow_level and ch in '?:'))) + + # Scanners. + + def scan_to_next_token(self): + # We ignore spaces, line breaks and comments. + # If we find a line break in the block context, we set the flag + # `allow_simple_key` on. + # The byte order mark is stripped if it's the first character in the + # stream. We do not yet support BOM inside the stream as the + # specification requires. Any such mark will be considered as a part + # of the document. + # + # TODO: We need to make tab handling rules more sane. A good rule is + # Tabs cannot precede tokens + # BLOCK-SEQUENCE-START, BLOCK-MAPPING-START, BLOCK-END, + # KEY(block), VALUE(block), BLOCK-ENTRY + # So the checking code is + # if <TAB>: + # self.allow_simple_keys = False + # We also need to add the check for `allow_simple_keys == True` to + # `unwind_indent` before issuing BLOCK-END. + # Scanners for block, flow, and plain scalars need to be modified. + + if self.index == 0 and self.peek() == '\uFEFF': + self.forward() + found = False + while not found: + while self.peek() == ' ': + self.forward() + if self.peek() == '#': + while self.peek() not in '\0\r\n\x85\u2028\u2029': + self.forward() + if self.scan_line_break(): + if not self.flow_level: + self.allow_simple_key = True + else: + found = True + + def scan_directive(self): + # See the specification for details. + start_mark = self.get_mark() + self.forward() + name = self.scan_directive_name(start_mark) + value = None + if name == 'YAML': + value = self.scan_yaml_directive_value(start_mark) + end_mark = self.get_mark() + elif name == 'TAG': + value = self.scan_tag_directive_value(start_mark) + end_mark = self.get_mark() + else: + end_mark = self.get_mark() + while self.peek() not in '\0\r\n\x85\u2028\u2029': + self.forward() + self.scan_directive_ignored_line(start_mark) + return DirectiveToken(name, value, start_mark, end_mark) + + def scan_directive_name(self, start_mark): + # See the specification for details. + length = 0 + ch = self.peek(length) + while '0' <= ch <= '9' or 'A' <= ch <= 'Z' or 'a' <= ch <= 'z' \ + or ch in '-_': + length += 1 + ch = self.peek(length) + if not length: + raise ScannerError("while scanning a directive", start_mark, + "expected alphabetic or numeric character, but found %r" + % ch, self.get_mark()) + value = self.prefix(length) + self.forward(length) + ch = self.peek() + if ch not in '\0 \r\n\x85\u2028\u2029': + raise ScannerError("while scanning a directive", start_mark, + "expected alphabetic or numeric character, but found %r" + % ch, self.get_mark()) + return value + + def scan_yaml_directive_value(self, start_mark): + # See the specification for details. + while self.peek() == ' ': + self.forward() + major = self.scan_yaml_directive_number(start_mark) + if self.peek() != '.': + raise ScannerError("while scanning a directive", start_mark, + "expected a digit or '.', but found %r" % self.peek(), + self.get_mark()) + self.forward() + minor = self.scan_yaml_directive_number(start_mark) + if self.peek() not in '\0 \r\n\x85\u2028\u2029': + raise ScannerError("while scanning a directive", start_mark, + "expected a digit or ' ', but found %r" % self.peek(), + self.get_mark()) + return (major, minor) + + def scan_yaml_directive_number(self, start_mark): + # See the specification for details. + ch = self.peek() + if not ('0' <= ch <= '9'): + raise ScannerError("while scanning a directive", start_mark, + "expected a digit, but found %r" % ch, self.get_mark()) + length = 0 + while '0' <= self.peek(length) <= '9': + length += 1 + value = int(self.prefix(length)) + self.forward(length) + return value + + def scan_tag_directive_value(self, start_mark): + # See the specification for details. + while self.peek() == ' ': + self.forward() + handle = self.scan_tag_directive_handle(start_mark) + while self.peek() == ' ': + self.forward() + prefix = self.scan_tag_directive_prefix(start_mark) + return (handle, prefix) + + def scan_tag_directive_handle(self, start_mark): + # See the specification for details. + value = self.scan_tag_handle('directive', start_mark) + ch = self.peek() + if ch != ' ': + raise ScannerError("while scanning a directive", start_mark, + "expected ' ', but found %r" % ch, self.get_mark()) + return value + + def scan_tag_directive_prefix(self, start_mark): + # See the specification for details. + value = self.scan_tag_uri('directive', start_mark) + ch = self.peek() + if ch not in '\0 \r\n\x85\u2028\u2029': + raise ScannerError("while scanning a directive", start_mark, + "expected ' ', but found %r" % ch, self.get_mark()) + return value + + def scan_directive_ignored_line(self, start_mark): + # See the specification for details. + while self.peek() == ' ': + self.forward() + if self.peek() == '#': + while self.peek() not in '\0\r\n\x85\u2028\u2029': + self.forward() + ch = self.peek() + if ch not in '\0\r\n\x85\u2028\u2029': + raise ScannerError("while scanning a directive", start_mark, + "expected a comment or a line break, but found %r" + % ch, self.get_mark()) + self.scan_line_break() + + def scan_anchor(self, TokenClass): + # The specification does not restrict characters for anchors and + # aliases. This may lead to problems, for instance, the document: + # [ *alias, value ] + # can be interpteted in two ways, as + # [ "value" ] + # and + # [ *alias , "value" ] + # Therefore we restrict aliases to numbers and ASCII letters. + start_mark = self.get_mark() + indicator = self.peek() + if indicator == '*': + name = 'alias' + else: + name = 'anchor' + self.forward() + length = 0 + ch = self.peek(length) + while '0' <= ch <= '9' or 'A' <= ch <= 'Z' or 'a' <= ch <= 'z' \ + or ch in '-_': + length += 1 + ch = self.peek(length) + if not length: + raise ScannerError("while scanning an %s" % name, start_mark, + "expected alphabetic or numeric character, but found %r" + % ch, self.get_mark()) + value = self.prefix(length) + self.forward(length) + ch = self.peek() + if ch not in '\0 \t\r\n\x85\u2028\u2029?:,]}%@`': + raise ScannerError("while scanning an %s" % name, start_mark, + "expected alphabetic or numeric character, but found %r" + % ch, self.get_mark()) + end_mark = self.get_mark() + return TokenClass(value, start_mark, end_mark) + + def scan_tag(self): + # See the specification for details. + start_mark = self.get_mark() + ch = self.peek(1) + if ch == '<': + handle = None + self.forward(2) + suffix = self.scan_tag_uri('tag', start_mark) + if self.peek() != '>': + raise ScannerError("while parsing a tag", start_mark, + "expected '>', but found %r" % self.peek(), + self.get_mark()) + self.forward() + elif ch in '\0 \t\r\n\x85\u2028\u2029': + handle = None + suffix = '!' + self.forward() + else: + length = 1 + use_handle = False + while ch not in '\0 \r\n\x85\u2028\u2029': + if ch == '!': + use_handle = True + break + length += 1 + ch = self.peek(length) + handle = '!' + if use_handle: + handle = self.scan_tag_handle('tag', start_mark) + else: + handle = '!' + self.forward() + suffix = self.scan_tag_uri('tag', start_mark) + ch = self.peek() + if ch not in '\0 \r\n\x85\u2028\u2029': + raise ScannerError("while scanning a tag", start_mark, + "expected ' ', but found %r" % ch, self.get_mark()) + value = (handle, suffix) + end_mark = self.get_mark() + return TagToken(value, start_mark, end_mark) + + def scan_block_scalar(self, style): + # See the specification for details. + + if style == '>': + folded = True + else: + folded = False + + chunks = [] + start_mark = self.get_mark() + + # Scan the header. + self.forward() + chomping, increment = self.scan_block_scalar_indicators(start_mark) + self.scan_block_scalar_ignored_line(start_mark) + + # Determine the indentation level and go to the first non-empty line. + min_indent = self.indent+1 + if min_indent < 1: + min_indent = 1 + if increment is None: + breaks, max_indent, end_mark = self.scan_block_scalar_indentation() + indent = max(min_indent, max_indent) + else: + indent = min_indent+increment-1 + breaks, end_mark = self.scan_block_scalar_breaks(indent) + line_break = '' + + # Scan the inner part of the block scalar. + while self.column == indent and self.peek() != '\0': + chunks.extend(breaks) + leading_non_space = self.peek() not in ' \t' + length = 0 + while self.peek(length) not in '\0\r\n\x85\u2028\u2029': + length += 1 + chunks.append(self.prefix(length)) + self.forward(length) + line_break = self.scan_line_break() + breaks, end_mark = self.scan_block_scalar_breaks(indent) + if self.column == indent and self.peek() != '\0': + + # Unfortunately, folding rules are ambiguous. + # + # This is the folding according to the specification: + + if folded and line_break == '\n' \ + and leading_non_space and self.peek() not in ' \t': + if not breaks: + chunks.append(' ') + else: + chunks.append(line_break) + + # This is Clark Evans's interpretation (also in the spec + # examples): + # + #if folded and line_break == '\n': + # if not breaks: + # if self.peek() not in ' \t': + # chunks.append(' ') + # else: + # chunks.append(line_break) + #else: + # chunks.append(line_break) + else: + break + + # Chomp the tail. + if chomping is not False: + chunks.append(line_break) + if chomping is True: + chunks.extend(breaks) + + # We are done. + return ScalarToken(''.join(chunks), False, start_mark, end_mark, + style) + + def scan_block_scalar_indicators(self, start_mark): + # See the specification for details. + chomping = None + increment = None + ch = self.peek() + if ch in '+-': + if ch == '+': + chomping = True + else: + chomping = False + self.forward() + ch = self.peek() + if ch in '0123456789': + increment = int(ch) + if increment == 0: + raise ScannerError("while scanning a block scalar", start_mark, + "expected indentation indicator in the range 1-9, but found 0", + self.get_mark()) + self.forward() + elif ch in '0123456789': + increment = int(ch) + if increment == 0: + raise ScannerError("while scanning a block scalar", start_mark, + "expected indentation indicator in the range 1-9, but found 0", + self.get_mark()) + self.forward() + ch = self.peek() + if ch in '+-': + if ch == '+': + chomping = True + else: + chomping = False + self.forward() + ch = self.peek() + if ch not in '\0 \r\n\x85\u2028\u2029': + raise ScannerError("while scanning a block scalar", start_mark, + "expected chomping or indentation indicators, but found %r" + % ch, self.get_mark()) + return chomping, increment + + def scan_block_scalar_ignored_line(self, start_mark): + # See the specification for details. + while self.peek() == ' ': + self.forward() + if self.peek() == '#': + while self.peek() not in '\0\r\n\x85\u2028\u2029': + self.forward() + ch = self.peek() + if ch not in '\0\r\n\x85\u2028\u2029': + raise ScannerError("while scanning a block scalar", start_mark, + "expected a comment or a line break, but found %r" % ch, + self.get_mark()) + self.scan_line_break() + + def scan_block_scalar_indentation(self): + # See the specification for details. + chunks = [] + max_indent = 0 + end_mark = self.get_mark() + while self.peek() in ' \r\n\x85\u2028\u2029': + if self.peek() != ' ': + chunks.append(self.scan_line_break()) + end_mark = self.get_mark() + else: + self.forward() + if self.column > max_indent: + max_indent = self.column + return chunks, max_indent, end_mark + + def scan_block_scalar_breaks(self, indent): + # See the specification for details. + chunks = [] + end_mark = self.get_mark() + while self.column < indent and self.peek() == ' ': + self.forward() + while self.peek() in '\r\n\x85\u2028\u2029': + chunks.append(self.scan_line_break()) + end_mark = self.get_mark() + while self.column < indent and self.peek() == ' ': + self.forward() + return chunks, end_mark + + def scan_flow_scalar(self, style): + # See the specification for details. + # Note that we loose indentation rules for quoted scalars. Quoted + # scalars don't need to adhere indentation because " and ' clearly + # mark the beginning and the end of them. Therefore we are less + # restrictive then the specification requires. We only need to check + # that document separators are not included in scalars. + if style == '"': + double = True + else: + double = False + chunks = [] + start_mark = self.get_mark() + quote = self.peek() + self.forward() + chunks.extend(self.scan_flow_scalar_non_spaces(double, start_mark)) + while self.peek() != quote: + chunks.extend(self.scan_flow_scalar_spaces(double, start_mark)) + chunks.extend(self.scan_flow_scalar_non_spaces(double, start_mark)) + self.forward() + end_mark = self.get_mark() + return ScalarToken(''.join(chunks), False, start_mark, end_mark, + style) + + ESCAPE_REPLACEMENTS = { + '0': '\0', + 'a': '\x07', + 'b': '\x08', + 't': '\x09', + '\t': '\x09', + 'n': '\x0A', + 'v': '\x0B', + 'f': '\x0C', + 'r': '\x0D', + 'e': '\x1B', + ' ': '\x20', + '\"': '\"', + '\\': '\\', + 'N': '\x85', + '_': '\xA0', + 'L': '\u2028', + 'P': '\u2029', + } + + ESCAPE_CODES = { + 'x': 2, + 'u': 4, + 'U': 8, + } + + def scan_flow_scalar_non_spaces(self, double, start_mark): + # See the specification for details. + chunks = [] + while True: + length = 0 + while self.peek(length) not in '\'\"\\\0 \t\r\n\x85\u2028\u2029': + length += 1 + if length: + chunks.append(self.prefix(length)) + self.forward(length) + ch = self.peek() + if not double and ch == '\'' and self.peek(1) == '\'': + chunks.append('\'') + self.forward(2) + elif (double and ch == '\'') or (not double and ch in '\"\\'): + chunks.append(ch) + self.forward() + elif double and ch == '\\': + self.forward() + ch = self.peek() + if ch in self.ESCAPE_REPLACEMENTS: + chunks.append(self.ESCAPE_REPLACEMENTS[ch]) + self.forward() + elif ch in self.ESCAPE_CODES: + length = self.ESCAPE_CODES[ch] + self.forward() + for k in range(length): + if self.peek(k) not in '0123456789ABCDEFabcdef': + raise ScannerError("while scanning a double-quoted scalar", start_mark, + "expected escape sequence of %d hexdecimal numbers, but found %r" % + (length, self.peek(k)), self.get_mark()) + code = int(self.prefix(length), 16) + chunks.append(chr(code)) + self.forward(length) + elif ch in '\r\n\x85\u2028\u2029': + self.scan_line_break() + chunks.extend(self.scan_flow_scalar_breaks(double, start_mark)) + else: + raise ScannerError("while scanning a double-quoted scalar", start_mark, + "found unknown escape character %r" % ch, self.get_mark()) + else: + return chunks + + def scan_flow_scalar_spaces(self, double, start_mark): + # See the specification for details. + chunks = [] + length = 0 + while self.peek(length) in ' \t': + length += 1 + whitespaces = self.prefix(length) + self.forward(length) + ch = self.peek() + if ch == '\0': + raise ScannerError("while scanning a quoted scalar", start_mark, + "found unexpected end of stream", self.get_mark()) + elif ch in '\r\n\x85\u2028\u2029': + line_break = self.scan_line_break() + breaks = self.scan_flow_scalar_breaks(double, start_mark) + if line_break != '\n': + chunks.append(line_break) + elif not breaks: + chunks.append(' ') + chunks.extend(breaks) + else: + chunks.append(whitespaces) + return chunks + + def scan_flow_scalar_breaks(self, double, start_mark): + # See the specification for details. + chunks = [] + while True: + # Instead of checking indentation, we check for document + # separators. + prefix = self.prefix(3) + if (prefix == '---' or prefix == '...') \ + and self.peek(3) in '\0 \t\r\n\x85\u2028\u2029': + raise ScannerError("while scanning a quoted scalar", start_mark, + "found unexpected document separator", self.get_mark()) + while self.peek() in ' \t': + self.forward() + if self.peek() in '\r\n\x85\u2028\u2029': + chunks.append(self.scan_line_break()) + else: + return chunks + + def scan_plain(self): + # See the specification for details. + # We add an additional restriction for the flow context: + # plain scalars in the flow context cannot contain ',', ':' and '?'. + # We also keep track of the `allow_simple_key` flag here. + # Indentation rules are loosed for the flow context. + chunks = [] + start_mark = self.get_mark() + end_mark = start_mark + indent = self.indent+1 + # We allow zero indentation for scalars, but then we need to check for + # document separators at the beginning of the line. + #if indent == 0: + # indent = 1 + spaces = [] + while True: + length = 0 + if self.peek() == '#': + break + while True: + ch = self.peek(length) + if ch in '\0 \t\r\n\x85\u2028\u2029' \ + or (not self.flow_level and ch == ':' and + self.peek(length+1) in '\0 \t\r\n\x85\u2028\u2029') \ + or (self.flow_level and ch in ',:?[]{}'): + break + length += 1 + # It's not clear what we should do with ':' in the flow context. + if (self.flow_level and ch == ':' + and self.peek(length+1) not in '\0 \t\r\n\x85\u2028\u2029,[]{}'): + self.forward(length) + raise ScannerError("while scanning a plain scalar", start_mark, + "found unexpected ':'", self.get_mark(), + "Please check http://pyyaml.org/wiki/YAMLColonInFlowContext for details.") + if length == 0: + break + self.allow_simple_key = False + chunks.extend(spaces) + chunks.append(self.prefix(length)) + self.forward(length) + end_mark = self.get_mark() + spaces = self.scan_plain_spaces(indent, start_mark) + if not spaces or self.peek() == '#' \ + or (not self.flow_level and self.column < indent): + break + return ScalarToken(''.join(chunks), True, start_mark, end_mark) + + def scan_plain_spaces(self, indent, start_mark): + # See the specification for details. + # The specification is really confusing about tabs in plain scalars. + # We just forbid them completely. Do not use tabs in YAML! + chunks = [] + length = 0 + while self.peek(length) in ' ': + length += 1 + whitespaces = self.prefix(length) + self.forward(length) + ch = self.peek() + if ch in '\r\n\x85\u2028\u2029': + line_break = self.scan_line_break() + self.allow_simple_key = True + prefix = self.prefix(3) + if (prefix == '---' or prefix == '...') \ + and self.peek(3) in '\0 \t\r\n\x85\u2028\u2029': + return + breaks = [] + while self.peek() in ' \r\n\x85\u2028\u2029': + if self.peek() == ' ': + self.forward() + else: + breaks.append(self.scan_line_break()) + prefix = self.prefix(3) + if (prefix == '---' or prefix == '...') \ + and self.peek(3) in '\0 \t\r\n\x85\u2028\u2029': + return + if line_break != '\n': + chunks.append(line_break) + elif not breaks: + chunks.append(' ') + chunks.extend(breaks) + elif whitespaces: + chunks.append(whitespaces) + return chunks + + def scan_tag_handle(self, name, start_mark): + # See the specification for details. + # For some strange reasons, the specification does not allow '_' in + # tag handles. I have allowed it anyway. + ch = self.peek() + if ch != '!': + raise ScannerError("while scanning a %s" % name, start_mark, + "expected '!', but found %r" % ch, self.get_mark()) + length = 1 + ch = self.peek(length) + if ch != ' ': + while '0' <= ch <= '9' or 'A' <= ch <= 'Z' or 'a' <= ch <= 'z' \ + or ch in '-_': + length += 1 + ch = self.peek(length) + if ch != '!': + self.forward(length) + raise ScannerError("while scanning a %s" % name, start_mark, + "expected '!', but found %r" % ch, self.get_mark()) + length += 1 + value = self.prefix(length) + self.forward(length) + return value + + def scan_tag_uri(self, name, start_mark): + # See the specification for details. + # Note: we do not check if URI is well-formed. + chunks = [] + length = 0 + ch = self.peek(length) + while '0' <= ch <= '9' or 'A' <= ch <= 'Z' or 'a' <= ch <= 'z' \ + or ch in '-;/?:@&=+$,_.!~*\'()[]%': + if ch == '%': + chunks.append(self.prefix(length)) + self.forward(length) + length = 0 + chunks.append(self.scan_uri_escapes(name, start_mark)) + else: + length += 1 + ch = self.peek(length) + if length: + chunks.append(self.prefix(length)) + self.forward(length) + length = 0 + if not chunks: + raise ScannerError("while parsing a %s" % name, start_mark, + "expected URI, but found %r" % ch, self.get_mark()) + return ''.join(chunks) + + def scan_uri_escapes(self, name, start_mark): + # See the specification for details. + codes = [] + mark = self.get_mark() + while self.peek() == '%': + self.forward() + for k in range(2): + if self.peek(k) not in '0123456789ABCDEFabcdef': + raise ScannerError("while scanning a %s" % name, start_mark, + "expected URI escape sequence of 2 hexdecimal numbers, but found %r" + % self.peek(k), self.get_mark()) + codes.append(int(self.prefix(2), 16)) + self.forward(2) + try: + value = bytes(codes).decode('utf-8') + except UnicodeDecodeError as exc: + raise ScannerError("while scanning a %s" % name, start_mark, str(exc), mark) + return value + + def scan_line_break(self): + # Transforms: + # '\r\n' : '\n' + # '\r' : '\n' + # '\n' : '\n' + # '\x85' : '\n' + # '\u2028' : '\u2028' + # '\u2029 : '\u2029' + # default : '' + ch = self.peek() + if ch in '\r\n\x85': + if self.prefix(2) == '\r\n': + self.forward(2) + else: + self.forward() + return '\n' + elif ch in '\u2028\u2029': + self.forward() + return ch + return '' + +#try: +# import psyco +# psyco.bind(Scanner) +#except ImportError: +# pass + diff --git a/python.d/python_modules/pyyaml3/serializer.py b/python.d/python_modules/pyyaml3/serializer.py new file mode 100644 index 000000000..fe911e67a --- /dev/null +++ b/python.d/python_modules/pyyaml3/serializer.py @@ -0,0 +1,111 @@ + +__all__ = ['Serializer', 'SerializerError'] + +from .error import YAMLError +from .events import * +from .nodes import * + +class SerializerError(YAMLError): + pass + +class Serializer: + + ANCHOR_TEMPLATE = 'id%03d' + + def __init__(self, encoding=None, + explicit_start=None, explicit_end=None, version=None, tags=None): + self.use_encoding = encoding + self.use_explicit_start = explicit_start + self.use_explicit_end = explicit_end + self.use_version = version + self.use_tags = tags + self.serialized_nodes = {} + self.anchors = {} + self.last_anchor_id = 0 + self.closed = None + + def open(self): + if self.closed is None: + self.emit(StreamStartEvent(encoding=self.use_encoding)) + self.closed = False + elif self.closed: + raise SerializerError("serializer is closed") + else: + raise SerializerError("serializer is already opened") + + def close(self): + if self.closed is None: + raise SerializerError("serializer is not opened") + elif not self.closed: + self.emit(StreamEndEvent()) + self.closed = True + + #def __del__(self): + # self.close() + + def serialize(self, node): + if self.closed is None: + raise SerializerError("serializer is not opened") + elif self.closed: + raise SerializerError("serializer is closed") + self.emit(DocumentStartEvent(explicit=self.use_explicit_start, + version=self.use_version, tags=self.use_tags)) + self.anchor_node(node) + self.serialize_node(node, None, None) + self.emit(DocumentEndEvent(explicit=self.use_explicit_end)) + self.serialized_nodes = {} + self.anchors = {} + self.last_anchor_id = 0 + + def anchor_node(self, node): + if node in self.anchors: + if self.anchors[node] is None: + self.anchors[node] = self.generate_anchor(node) + else: + self.anchors[node] = None + if isinstance(node, SequenceNode): + for item in node.value: + self.anchor_node(item) + elif isinstance(node, MappingNode): + for key, value in node.value: + self.anchor_node(key) + self.anchor_node(value) + + def generate_anchor(self, node): + self.last_anchor_id += 1 + return self.ANCHOR_TEMPLATE % self.last_anchor_id + + def serialize_node(self, node, parent, index): + alias = self.anchors[node] + if node in self.serialized_nodes: + self.emit(AliasEvent(alias)) + else: + self.serialized_nodes[node] = True + self.descend_resolver(parent, index) + if isinstance(node, ScalarNode): + detected_tag = self.resolve(ScalarNode, node.value, (True, False)) + default_tag = self.resolve(ScalarNode, node.value, (False, True)) + implicit = (node.tag == detected_tag), (node.tag == default_tag) + self.emit(ScalarEvent(alias, node.tag, implicit, node.value, + style=node.style)) + elif isinstance(node, SequenceNode): + implicit = (node.tag + == self.resolve(SequenceNode, node.value, True)) + self.emit(SequenceStartEvent(alias, node.tag, implicit, + flow_style=node.flow_style)) + index = 0 + for item in node.value: + self.serialize_node(item, node, index) + index += 1 + self.emit(SequenceEndEvent()) + elif isinstance(node, MappingNode): + implicit = (node.tag + == self.resolve(MappingNode, node.value, True)) + self.emit(MappingStartEvent(alias, node.tag, implicit, + flow_style=node.flow_style)) + for key, value in node.value: + self.serialize_node(key, node, None) + self.serialize_node(value, node, key) + self.emit(MappingEndEvent()) + self.ascend_resolver() + diff --git a/python.d/python_modules/pyyaml3/tokens.py b/python.d/python_modules/pyyaml3/tokens.py new file mode 100644 index 000000000..4d0b48a39 --- /dev/null +++ b/python.d/python_modules/pyyaml3/tokens.py @@ -0,0 +1,104 @@ + +class Token(object): + def __init__(self, start_mark, end_mark): + self.start_mark = start_mark + self.end_mark = end_mark + def __repr__(self): + attributes = [key for key in self.__dict__ + if not key.endswith('_mark')] + attributes.sort() + arguments = ', '.join(['%s=%r' % (key, getattr(self, key)) + for key in attributes]) + return '%s(%s)' % (self.__class__.__name__, arguments) + +#class BOMToken(Token): +# id = '<byte order mark>' + +class DirectiveToken(Token): + id = '<directive>' + def __init__(self, name, value, start_mark, end_mark): + self.name = name + self.value = value + self.start_mark = start_mark + self.end_mark = end_mark + +class DocumentStartToken(Token): + id = '<document start>' + +class DocumentEndToken(Token): + id = '<document end>' + +class StreamStartToken(Token): + id = '<stream start>' + def __init__(self, start_mark=None, end_mark=None, + encoding=None): + self.start_mark = start_mark + self.end_mark = end_mark + self.encoding = encoding + +class StreamEndToken(Token): + id = '<stream end>' + +class BlockSequenceStartToken(Token): + id = '<block sequence start>' + +class BlockMappingStartToken(Token): + id = '<block mapping start>' + +class BlockEndToken(Token): + id = '<block end>' + +class FlowSequenceStartToken(Token): + id = '[' + +class FlowMappingStartToken(Token): + id = '{' + +class FlowSequenceEndToken(Token): + id = ']' + +class FlowMappingEndToken(Token): + id = '}' + +class KeyToken(Token): + id = '?' + +class ValueToken(Token): + id = ':' + +class BlockEntryToken(Token): + id = '-' + +class FlowEntryToken(Token): + id = ',' + +class AliasToken(Token): + id = '<alias>' + def __init__(self, value, start_mark, end_mark): + self.value = value + self.start_mark = start_mark + self.end_mark = end_mark + +class AnchorToken(Token): + id = '<anchor>' + def __init__(self, value, start_mark, end_mark): + self.value = value + self.start_mark = start_mark + self.end_mark = end_mark + +class TagToken(Token): + id = '<tag>' + def __init__(self, value, start_mark, end_mark): + self.value = value + self.start_mark = start_mark + self.end_mark = end_mark + +class ScalarToken(Token): + id = '<scalar>' + def __init__(self, value, plain, start_mark, end_mark, style=None): + self.value = value + self.plain = plain + self.start_mark = start_mark + self.end_mark = end_mark + self.style = style + diff --git a/python.d/redis.chart.py b/python.d/redis.chart.py new file mode 100644 index 000000000..218401e12 --- /dev/null +++ b/python.d/redis.chart.py @@ -0,0 +1,140 @@ +# -*- coding: utf-8 -*- +# Description: redis netdata python.d module +# Author: Pawel Krupa (paulfantom) + +from base import SocketService + +# default module values (can be overridden per job in `config`) +#update_every = 2 +priority = 60000 +retries = 60 + +# default job configuration (overridden by python.d.plugin) +# config = {'local': { +# 'update_every': update_every, +# 'retries': retries, +# 'priority': priority, +# 'host': 'localhost', +# 'port': 6379, +# 'unix_socket': None +# }} + +ORDER = ['operations', 'hit_rate', 'memory', 'keys', 'clients', 'slaves'] + +CHARTS = { + 'operations': { + 'options': [None, 'Operations', 'operations/s', 'Statistics', 'redis.operations', 'line'], + 'lines': [ + ['instantaneous_ops_per_sec', 'operations', 'absolute'] + ]}, + 'hit_rate': { + 'options': [None, 'Hit rate', 'percent', 'Statistics', 'redis.hit_rate', 'line'], + 'lines': [ + ['hit_rate', 'rate', 'absolute'] + ]}, + 'memory': { + 'options': [None, 'Memory utilization', 'kilobytes', 'Memory', 'redis.memory', 'line'], + 'lines': [ + ['used_memory', 'total', 'absolute', 1, 1024], + ['used_memory_lua', 'lua', 'absolute', 1, 1024] + ]}, + 'keys': { + 'options': [None, 'Database keys', 'keys', 'Keys', 'redis.keys', 'line'], + 'lines': [ + # lines are created dynamically in `check()` method + ]}, + 'clients': { + 'options': [None, 'Clients', 'clients', 'Clients', 'redis.clients', 'line'], + 'lines': [ + ['connected_clients', 'connected', 'absolute'], + ['blocked_clients', 'blocked', 'absolute'] + ]}, + 'slaves': { + 'options': [None, 'Slaves', 'slaves', 'Replication', 'redis.slaves', 'line'], + 'lines': [ + ['connected_slaves', 'connected', 'absolute'] + ]} +} + + +class Service(SocketService): + def __init__(self, configuration=None, name=None): + SocketService.__init__(self, configuration=configuration, name=name) + self.request = "INFO\r\n" + self.host = "localhost" + self.port = 6379 + self.unix_socket = None + self.order = ORDER + self.definitions = CHARTS + self._keep_alive = True + self.chart_name = "" + + def _get_data(self): + """ + Get data from socket + :return: dict + """ + try: + raw = self._get_raw_data().split("\n") + except AttributeError: + self.error("no data received") + return None + data = {} + for line in raw: + if line.startswith(('instantaneous', 'keyspace', 'used_memory', 'connected', 'blocked')): + try: + t = line.split(':') + data[t[0]] = int(t[1]) + except (IndexError, ValueError): + pass + elif line.startswith('db'): + tmp = line.split(',')[0].replace('keys=', '') + record = tmp.split(':') + data[record[0]] = int(record[1]) + try: + data['hit_rate'] = int((data['keyspace_hits'] / float(data['keyspace_hits'] + data['keyspace_misses'])) * 100) + except: + data['hit_rate'] = 0 + + if len(data) == 0: + self.error("received data doesn't have needed records") + return None + else: + return data + + def _check_raw_data(self, data): + """ + Check if all data has been gathered from socket. + Parse first line containing message length and check against received message + :param data: str + :return: boolean + """ + length = len(data) + supposed = data.split('\n')[0][1:] + offset = len(supposed) + 4 # 1 dollar sing, 1 new line character + 1 ending sequence '\r\n' + supposed = int(supposed) + if length - offset >= supposed: + return True + else: + return False + + return False + + def check(self): + """ + Parse configuration, check if redis is available, and dynamically create chart lines data + :return: boolean + """ + self._parse_config() + if self.name == "": + self.name = "local" + self.chart_name += "_" + self.name + data = self._get_data() + if data is None: + return False + + for name in data: + if name.startswith('db'): + self.definitions['keys']['lines'].append([name, None, 'absolute']) + + return True diff --git a/python.d/sensors.chart.py b/python.d/sensors.chart.py new file mode 100644 index 000000000..7abe7f080 --- /dev/null +++ b/python.d/sensors.chart.py @@ -0,0 +1,134 @@ +# -*- coding: utf-8 -*- +# Description: sensors netdata python.d plugin +# Author: Pawel Krupa (paulfantom) + +from base import SimpleService +import lm_sensors as sensors + +# default module values (can be overridden per job in `config`) +# update_every = 2 + +ORDER = ['temperature', 'fan', 'voltage', 'current', 'power', 'energy', 'humidity'] + +# This is a prototype of chart definition which is used to dynamically create self.definitions +CHARTS = { + 'temperature': { + 'options': [None, ' temperature', 'Celsius', 'temperature', 'sensors.temperature', 'line'], + 'lines': [ + [None, None, 'absolute', 1, 1000] + ]}, + 'voltage': { + 'options': [None, ' voltage', 'Volts', 'voltage', 'sensors.voltage', 'line'], + 'lines': [ + [None, None, 'absolute', 1, 1000] + ]}, + 'current': { + 'options': [None, ' current', 'Ampere', 'current', 'sensors.current', 'line'], + 'lines': [ + [None, None, 'absolute', 1, 1000] + ]}, + 'power': { + 'options': [None, ' power', 'Watt', 'power', 'sensors.power', 'line'], + 'lines': [ + [None, None, 'absolute', 1, 1000000] + ]}, + 'fan': { + 'options': [None, ' fans speed', 'Rotations/min', 'fans', 'sensors.fan', 'line'], + 'lines': [ + [None, None, 'absolute', 1, 1000] + ]}, + 'energy': { + 'options': [None, ' energy', 'Joule', 'energy', 'sensors.energy', 'areastack'], + 'lines': [ + [None, None, 'incremental', 1, 1000000] + ]}, + 'humidity': { + 'options': [None, ' humidity', 'Percent', 'humidity', 'sensors.humidity', 'line'], + 'lines': [ + [None, None, 'absolute', 1, 1000] + ]} +} + +TYPE_MAP = { + 0: 'voltage', + 1: 'fan', + 2: 'temperature', + 3: 'power', + 4: 'energy', + 5: 'current', + 6: 'humidity', + 7: 'max_main', + 16: 'vid', + 17: 'intrusion', + 18: 'max_other', + 24: 'beep_enable' +} + + +class Service(SimpleService): + def __init__(self, configuration=None, name=None): + SimpleService.__init__(self, configuration=configuration, name=name) + self.order = [] + self.definitions = {} + self.chips = [] + + def _get_data(self): + data = {} + try: + for chip in sensors.ChipIterator(): + prefix = sensors.chip_snprintf_name(chip) + for feature in sensors.FeatureIterator(chip): + sfi = sensors.SubFeatureIterator(chip, feature) + for sf in sfi: + val = sensors.get_value(chip, sf.number) + break + data[prefix + "_" + str(feature.name.decode())] = int(val * 1000) + except Exception as e: + self.error(e) + return None + + if len(data) == 0: + return None + return data + + def _create_definitions(self): + prev_chip = "" + for type in ORDER: + for chip in sensors.ChipIterator(): + chip_name = sensors.chip_snprintf_name(chip) + if len(self.chips) != 0 and not any([chip_name.startswith(ex) for ex in self.chips]): + continue + for feature in sensors.FeatureIterator(chip): + sfi = sensors.SubFeatureIterator(chip, feature) + vals = [sensors.get_value(chip, sf.number) for sf in sfi] + if vals[0] == 0: + continue + if TYPE_MAP[feature.type] == type: + # create chart + if chip_name != prev_chip: + name = chip_name + "_" + TYPE_MAP[feature.type] + if name not in self.order: + self.order.append(name) + chart_def = list(CHARTS[type]['options']) + chart_def[1] = chip_name + chart_def[1] + self.definitions[name] = {'options': chart_def} + self.definitions[name]['lines'] = [] + line = list(CHARTS[type]['lines'][0]) + line[0] = chip_name + "_" + str(feature.name.decode()) + line[1] = sensors.get_label(chip, feature) + self.definitions[name]['lines'].append(line) + prev_chip = chip_name + + def check(self): + try: + sensors.init() + except Exception as e: + self.error(e) + return False + + try: + self._create_definitions() + except Exception as e: + self.error(e) + return False + return True diff --git a/python.d/squid.chart.py b/python.d/squid.chart.py new file mode 100644 index 000000000..8300d9bad --- /dev/null +++ b/python.d/squid.chart.py @@ -0,0 +1,112 @@ +# -*- coding: utf-8 -*- +# Description: squid netdata python.d module +# Author: Pawel Krupa (paulfantom) + +from base import SocketService +import select + +# default module values (can be overridden per job in `config`) +# update_every = 2 +priority = 60000 +retries = 60 + +# charts order (can be overridden if you want less charts, or different order) +ORDER = ['clients_net', 'clients_requests', 'servers_net', 'servers_requests'] + +CHARTS = { + 'clients_net': { + 'options': [None, "Squid Client Bandwidth", "kilobits/s", "clients", "squid.clients_net", "area"], + 'lines': [ + ["client_http_kbytes_in", "in", "incremental", 8, 1], + ["client_http_kbytes_out", "out", "incremental", -8, 1], + ["client_http_hit_kbytes_out", "hits", "incremental", -8, 1] + ]}, + 'clients_requests': { + 'options': [None, "Squid Client Requests", "requests/s", "clients", "squid.clients_requests", 'line'], + 'lines': [ + ["client_http_requests", "requests", "incremental"], + ["client_http_hits", "hits", "incremental"], + ["client_http_errors", "errors", "incremental", -1, 1] + ]}, + 'servers_net': { + 'options': [None, "Squid Server Bandwidth", "kilobits/s", "servers", "squid.servers_net", "area"], + 'lines': [ + ["server_all_kbytes_in", "in", "incremental", 8, 1], + ["server_all_kbytes_out", "out", "incremental", -8, 1] + ]}, + 'servers_requests': { + 'options': [None, "Squid Server Requests", "requests/s", "servers", "squid.servers_requests", 'line'], + 'lines': [ + ["server_all_requests", "requests", "incremental"], + ["server_all_errors", "errors", "incremental", -1, 1] + ]} +} + + +class Service(SocketService): + def __init__(self, configuration=None, name=None): + SocketService.__init__(self, configuration=configuration, name=name) + self._keep_alive = True + self.request = "" + self.host = "localhost" + self.port = 3128 + self.order = ORDER + self.definitions = CHARTS + + def _get_data(self): + """ + Get data via http request + :return: dict + """ + data = {} + try: + raw = "" + for tmp in self._get_raw_data().split('\r\n'): + if tmp.startswith("sample_time"): + raw = tmp + break + if raw.startswith('<'): + self.error("invalid data received") + return None + for row in raw.split('\n'): + if row.startswith(("client", "server.all")): + tmp = row.split("=") + data[tmp[0].replace('.', '_').strip(' ')] = int(tmp[1]) + except (ValueError, AttributeError, TypeError): + self.error("invalid data received") + return None + + if len(data) == 0: + self.error("no data received") + return None + else: + return data + + def _check_raw_data(self, data): + if "Connection: keep-alive" in data[:1024]: + self._keep_alive = True + else: + self._keep_alive = False + + if data[-7:] == "\r\n0\r\n\r\n" and "Transfer-Encoding: chunked" in data[:1024]: # HTTP/1.1 response + return True + else: + return False + + def check(self): + """ + Parse essential configuration, autodetect squid configuration (if needed), and check if data is available + :return: boolean + """ + self._parse_config() + # format request + req = self.request.decode() + if not req.startswith("GET"): + req = "GET " + req + if not req.endswith(" HTTP/1.1\r\n\r\n"): + req += " HTTP/1.1\r\n\r\n" + self.request = req.encode() + if self._get_data() is not None: + return True + else: + return False diff --git a/python.d/tomcat.chart.py b/python.d/tomcat.chart.py new file mode 100644 index 000000000..31f6ab248 --- /dev/null +++ b/python.d/tomcat.chart.py @@ -0,0 +1,111 @@ +# -*- coding: utf-8 -*- +# Description: tomcat netdata python.d module +# Author: Pawel Krupa (paulfantom) + +# Python version higher than 2.7 is needed to run this module. + +from base import UrlService +import xml.etree.ElementTree as ET # phone home... +#from xml.parsers.expat import errors + +# default module values (can be overridden per job in `config`) +# update_every = 2 +priority = 60000 +retries = 60 + +# charts order (can be overridden if you want less charts, or different order) +ORDER = ['accesses', 'volume', 'threads', 'jvm'] + +CHARTS = { + 'accesses': { + 'options': [None, "Requests", "requests/s", "statistics", "tomcat.accesses", "area"], + 'lines': [ + ["accesses", None, 'incremental'] + ]}, + 'volume': { + 'options': [None, "Volume", "KB/s", "volume", "tomcat.volume", "area"], + 'lines': [ + ["volume", None, 'incremental', 1, 1024] + ]}, + 'threads': { + 'options': [None, "Threads", "current threads", "statistics", "tomcat.threads", "line"], + 'lines': [ + ["current", None, "absolute"], + ["busy", None, "absolute"] + ]}, + 'jvm': { + 'options': [None, "JVM Free Memory", "MB", "statistics", "tomcat.jvm", "area"], + 'lines': [ + ["jvm", None, "absolute", 1, 1048576] + ]} +} + + +class Service(UrlService): + def __init__(self, configuration=None, name=None): + UrlService.__init__(self, configuration=configuration, name=name) + if len(self.url) == 0: + self.url = "http://localhost:8080/manager/status?XML=true" + self.order = ORDER + self.definitions = CHARTS + self.port = 8080 + + def check(self): + if UrlService.check(self): + return True + + # get port from url + self.port = 0 + for i in self.url.split('/'): + try: + int(i[-1]) + self.port = i.split(':')[-1] + break + except: + pass + if self.port == 0: + self.port = 80 + + test = self._get_data() + if test is None or len(test) == 0: + return False + else: + return True + + def _get_data(self): + """ + Format data received from http request + :return: dict + """ + try: + raw = self._get_raw_data() + try: + data = ET.fromstring(raw) + except ET.ParseError as e: + # if e.code == errors.codes[errors.XML_ERROR_JUNK_AFTER_DOC_ELEMENT]: + if e.code == 9: + end = raw.find('</status>') + end += 9 + raw = raw[:end] + self.debug(raw) + data = ET.fromstring(raw) + else: + raise Exception(e) + + memory = data.find('./jvm/memory') + threads = data.find("./connector[@name='\"http-bio-" + str(self.port) + "\"']/threadInfo") + requests = data.find("./connector[@name='\"http-bio-" + str(self.port) + "\"']/requestInfo") + + return {'accesses': requests.attrib['requestCount'], + 'volume': requests.attrib['bytesSent'], + 'current': threads.attrib['currentThreadCount'], + 'busy': threads.attrib['currentThreadsBusy'], + 'jvm': memory.attrib['free']} + except (ValueError, AttributeError) as e: + self.debug(str(e)) + return None + except SyntaxError as e: + self.error("Tomcat module needs python 2.7 at least. Stopping") + self.debug(str(e)) + except Exception as e: + self.debug(str(e)) diff --git a/src/Makefile.am b/src/Makefile.am index e9fc8f332..8fa6d5bdf 100644 --- a/src/Makefile.am +++ b/src/Makefile.am @@ -21,6 +21,8 @@ AM_CFLAGS = \ sbin_PROGRAMS = netdata dist_cache_DATA = .keep +dist_varlib_DATA = .keep +dist_registry_DATA = .keep dist_log_DATA = .keep plugins_PROGRAMS = apps.plugin @@ -30,7 +32,9 @@ netdata_SOURCES = \ common.c common.h \ daemon.c daemon.h \ dictionary.c dictionary.h \ + eval.c eval.h \ global_statistics.c global_statistics.h \ + health.c health.h \ log.c log.h \ main.c main.h \ plugin_checks.c plugin_checks.h \ @@ -67,6 +71,7 @@ netdata_SOURCES = \ unit_test.c unit_test.h \ url.c url.h \ web_buffer.c web_buffer.h \ + web_buffer_svg.c web_buffer_svg.h \ web_client.c web_client.h \ web_server.c web_server.h \ $(NULL) @@ -83,6 +88,7 @@ apps_plugin_SOURCES = \ common.c common.h \ log.c log.h \ procfile.c procfile.h \ + web_buffer.c web_buffer.h \ $(NULL) install-data-hook: diff --git a/src/Makefile.in b/src/Makefile.in index 11b68946e..1a721a045 100644 --- a/src/Makefile.in +++ b/src/Makefile.in @@ -83,7 +83,8 @@ sbin_PROGRAMS = netdata$(EXEEXT) plugins_PROGRAMS = apps.plugin$(EXEEXT) subdir = src DIST_COMMON = $(srcdir)/Makefile.in $(srcdir)/Makefile.am \ - $(top_srcdir)/depcomp $(dist_cache_DATA) $(dist_log_DATA) + $(top_srcdir)/depcomp $(dist_cache_DATA) $(dist_log_DATA) \ + $(dist_registry_DATA) $(dist_varlib_DATA) ACLOCAL_M4 = $(top_srcdir)/aclocal.m4 am__aclocal_m4_deps = $(top_srcdir)/m4/ax_pthread.m4 \ $(top_srcdir)/configure.ac @@ -94,24 +95,27 @@ CONFIG_HEADER = $(top_builddir)/config.h CONFIG_CLEAN_FILES = CONFIG_CLEAN_VPATH_FILES = am__installdirs = "$(DESTDIR)$(pluginsdir)" "$(DESTDIR)$(sbindir)" \ - "$(DESTDIR)$(cachedir)" "$(DESTDIR)$(logdir)" + "$(DESTDIR)$(cachedir)" "$(DESTDIR)$(logdir)" \ + "$(DESTDIR)$(registrydir)" "$(DESTDIR)$(varlibdir)" PROGRAMS = $(plugins_PROGRAMS) $(sbin_PROGRAMS) am_apps_plugin_OBJECTS = apps_plugin.$(OBJEXT) avl.$(OBJEXT) \ - common.$(OBJEXT) log.$(OBJEXT) procfile.$(OBJEXT) + common.$(OBJEXT) log.$(OBJEXT) procfile.$(OBJEXT) \ + web_buffer.$(OBJEXT) apps_plugin_OBJECTS = $(am_apps_plugin_OBJECTS) apps_plugin_LDADD = $(LDADD) am_netdata_OBJECTS = appconfig.$(OBJEXT) avl.$(OBJEXT) \ common.$(OBJEXT) daemon.$(OBJEXT) dictionary.$(OBJEXT) \ - global_statistics.$(OBJEXT) log.$(OBJEXT) main.$(OBJEXT) \ - plugin_checks.$(OBJEXT) plugin_idlejitter.$(OBJEXT) \ - plugin_nfacct.$(OBJEXT) plugin_proc.$(OBJEXT) \ - plugin_tc.$(OBJEXT) plugins_d.$(OBJEXT) popen.$(OBJEXT) \ - proc_diskstats.$(OBJEXT) proc_interrupts.$(OBJEXT) \ - proc_softirqs.$(OBJEXT) proc_loadavg.$(OBJEXT) \ - proc_meminfo.$(OBJEXT) proc_net_dev.$(OBJEXT) \ - proc_net_ip_vs_stats.$(OBJEXT) proc_net_netstat.$(OBJEXT) \ - proc_net_rpc_nfsd.$(OBJEXT) proc_net_snmp.$(OBJEXT) \ - proc_net_snmp6.$(OBJEXT) proc_net_stat_conntrack.$(OBJEXT) \ + eval.$(OBJEXT) global_statistics.$(OBJEXT) health.$(OBJEXT) \ + log.$(OBJEXT) main.$(OBJEXT) plugin_checks.$(OBJEXT) \ + plugin_idlejitter.$(OBJEXT) plugin_nfacct.$(OBJEXT) \ + plugin_proc.$(OBJEXT) plugin_tc.$(OBJEXT) plugins_d.$(OBJEXT) \ + popen.$(OBJEXT) proc_diskstats.$(OBJEXT) \ + proc_interrupts.$(OBJEXT) proc_softirqs.$(OBJEXT) \ + proc_loadavg.$(OBJEXT) proc_meminfo.$(OBJEXT) \ + proc_net_dev.$(OBJEXT) proc_net_ip_vs_stats.$(OBJEXT) \ + proc_net_netstat.$(OBJEXT) proc_net_rpc_nfsd.$(OBJEXT) \ + proc_net_snmp.$(OBJEXT) proc_net_snmp6.$(OBJEXT) \ + proc_net_stat_conntrack.$(OBJEXT) \ proc_net_stat_synproxy.$(OBJEXT) proc_stat.$(OBJEXT) \ proc_self_mountinfo.$(OBJEXT) \ proc_sys_kernel_random_entropy_avail.$(OBJEXT) \ @@ -119,7 +123,8 @@ am_netdata_OBJECTS = appconfig.$(OBJEXT) avl.$(OBJEXT) \ sys_fs_cgroup.$(OBJEXT) procfile.$(OBJEXT) registry.$(OBJEXT) \ rrd.$(OBJEXT) rrd2json.$(OBJEXT) storage_number.$(OBJEXT) \ unit_test.$(OBJEXT) url.$(OBJEXT) web_buffer.$(OBJEXT) \ - web_client.$(OBJEXT) web_server.$(OBJEXT) + web_buffer_svg.$(OBJEXT) web_client.$(OBJEXT) \ + web_server.$(OBJEXT) netdata_OBJECTS = $(am_netdata_OBJECTS) am__DEPENDENCIES_1 = netdata_DEPENDENCIES = $(am__DEPENDENCIES_1) $(am__DEPENDENCIES_1) \ @@ -186,7 +191,8 @@ am__uninstall_files_from_dir = { \ || { echo " ( cd '$$dir' && rm -f" $$files ")"; \ $(am__cd) "$$dir" && rm -f $$files; }; \ } -DATA = $(dist_cache_DATA) $(dist_log_DATA) +DATA = $(dist_cache_DATA) $(dist_log_DATA) $(dist_registry_DATA) \ + $(dist_varlib_DATA) am__tagged_files = $(HEADERS) $(SOURCES) $(TAGS_FILES) $(LISP) # Read a list of newline-separated strings from the standard input, # and print each of them once, without duplicates. Input order is @@ -328,6 +334,8 @@ pluginsdir = @pluginsdir@ prefix = @prefix@ program_transform_name = @program_transform_name@ psdir = @psdir@ +pythondir = @pythondir@ +registrydir = @registrydir@ sbindir = @sbindir@ sharedstatedir = @sharedstatedir@ srcdir = @srcdir@ @@ -361,6 +369,8 @@ AM_CFLAGS = \ $(NULL) dist_cache_DATA = .keep +dist_varlib_DATA = .keep +dist_registry_DATA = .keep dist_log_DATA = .keep netdata_SOURCES = \ appconfig.c appconfig.h \ @@ -368,7 +378,9 @@ netdata_SOURCES = \ common.c common.h \ daemon.c daemon.h \ dictionary.c dictionary.h \ + eval.c eval.h \ global_statistics.c global_statistics.h \ + health.c health.h \ log.c log.h \ main.c main.h \ plugin_checks.c plugin_checks.h \ @@ -405,6 +417,7 @@ netdata_SOURCES = \ unit_test.c unit_test.h \ url.c url.h \ web_buffer.c web_buffer.h \ + web_buffer_svg.c web_buffer_svg.h \ web_client.c web_client.h \ web_server.c web_server.h \ $(NULL) @@ -422,6 +435,7 @@ apps_plugin_SOURCES = \ common.c common.h \ log.c log.h \ procfile.c procfile.h \ + web_buffer.c web_buffer.h \ $(NULL) all: all-am @@ -563,7 +577,9 @@ distclean-compile: @AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/common.Po@am__quote@ @AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/daemon.Po@am__quote@ @AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/dictionary.Po@am__quote@ +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/eval.Po@am__quote@ @AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/global_statistics.Po@am__quote@ +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/health.Po@am__quote@ @AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/log.Po@am__quote@ @AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/main.Po@am__quote@ @AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/plugin_checks.Po@am__quote@ @@ -600,6 +616,7 @@ distclean-compile: @AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/unit_test.Po@am__quote@ @AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/url.Po@am__quote@ @AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/web_buffer.Po@am__quote@ +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/web_buffer_svg.Po@am__quote@ @AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/web_client.Po@am__quote@ @AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/web_server.Po@am__quote@ @@ -658,6 +675,48 @@ uninstall-dist_logDATA: @list='$(dist_log_DATA)'; test -n "$(logdir)" || list=; \ files=`for p in $$list; do echo $$p; done | sed -e 's|^.*/||'`; \ dir='$(DESTDIR)$(logdir)'; $(am__uninstall_files_from_dir) +install-dist_registryDATA: $(dist_registry_DATA) + @$(NORMAL_INSTALL) + @list='$(dist_registry_DATA)'; test -n "$(registrydir)" || list=; \ + if test -n "$$list"; then \ + echo " $(MKDIR_P) '$(DESTDIR)$(registrydir)'"; \ + $(MKDIR_P) "$(DESTDIR)$(registrydir)" || exit 1; \ + fi; \ + for p in $$list; do \ + if test -f "$$p"; then d=; else d="$(srcdir)/"; fi; \ + echo "$$d$$p"; \ + done | $(am__base_list) | \ + while read files; do \ + echo " $(INSTALL_DATA) $$files '$(DESTDIR)$(registrydir)'"; \ + $(INSTALL_DATA) $$files "$(DESTDIR)$(registrydir)" || exit $$?; \ + done + +uninstall-dist_registryDATA: + @$(NORMAL_UNINSTALL) + @list='$(dist_registry_DATA)'; test -n "$(registrydir)" || list=; \ + files=`for p in $$list; do echo $$p; done | sed -e 's|^.*/||'`; \ + dir='$(DESTDIR)$(registrydir)'; $(am__uninstall_files_from_dir) +install-dist_varlibDATA: $(dist_varlib_DATA) + @$(NORMAL_INSTALL) + @list='$(dist_varlib_DATA)'; test -n "$(varlibdir)" || list=; \ + if test -n "$$list"; then \ + echo " $(MKDIR_P) '$(DESTDIR)$(varlibdir)'"; \ + $(MKDIR_P) "$(DESTDIR)$(varlibdir)" || exit 1; \ + fi; \ + for p in $$list; do \ + if test -f "$$p"; then d=; else d="$(srcdir)/"; fi; \ + echo "$$d$$p"; \ + done | $(am__base_list) | \ + while read files; do \ + echo " $(INSTALL_DATA) $$files '$(DESTDIR)$(varlibdir)'"; \ + $(INSTALL_DATA) $$files "$(DESTDIR)$(varlibdir)" || exit $$?; \ + done + +uninstall-dist_varlibDATA: + @$(NORMAL_UNINSTALL) + @list='$(dist_varlib_DATA)'; test -n "$(varlibdir)" || list=; \ + files=`for p in $$list; do echo $$p; done | sed -e 's|^.*/||'`; \ + dir='$(DESTDIR)$(varlibdir)'; $(am__uninstall_files_from_dir) ID: $(am__tagged_files) $(am__define_uniq_tagged_files); mkid -fID $$unique @@ -745,7 +804,7 @@ check-am: all-am check: check-am all-am: Makefile $(PROGRAMS) $(DATA) installdirs: - for dir in "$(DESTDIR)$(pluginsdir)" "$(DESTDIR)$(sbindir)" "$(DESTDIR)$(cachedir)" "$(DESTDIR)$(logdir)"; do \ + for dir in "$(DESTDIR)$(pluginsdir)" "$(DESTDIR)$(sbindir)" "$(DESTDIR)$(cachedir)" "$(DESTDIR)$(logdir)" "$(DESTDIR)$(registrydir)" "$(DESTDIR)$(varlibdir)"; do \ test -z "$$dir" || $(MKDIR_P) "$$dir"; \ done install: install-am @@ -803,6 +862,7 @@ info: info-am info-am: install-data-am: install-dist_cacheDATA install-dist_logDATA \ + install-dist_registryDATA install-dist_varlibDATA \ install-pluginsPROGRAMS @$(NORMAL_INSTALL) $(MAKE) $(AM_MAKEFLAGS) install-data-hook @@ -850,6 +910,7 @@ ps: ps-am ps-am: uninstall-am: uninstall-dist_cacheDATA uninstall-dist_logDATA \ + uninstall-dist_registryDATA uninstall-dist_varlibDATA \ uninstall-pluginsPROGRAMS uninstall-sbinPROGRAMS .MAKE: install-am install-data-am install-strip @@ -860,14 +921,16 @@ uninstall-am: uninstall-dist_cacheDATA uninstall-dist_logDATA \ distclean-tags distdir dvi dvi-am html html-am info info-am \ install install-am install-data install-data-am \ install-data-hook install-dist_cacheDATA install-dist_logDATA \ - install-dvi install-dvi-am install-exec install-exec-am \ - install-html install-html-am install-info install-info-am \ - install-man install-pdf install-pdf-am install-pluginsPROGRAMS \ - install-ps install-ps-am install-sbinPROGRAMS install-strip \ - installcheck installcheck-am installdirs maintainer-clean \ + install-dist_registryDATA install-dist_varlibDATA install-dvi \ + install-dvi-am install-exec install-exec-am install-html \ + install-html-am install-info install-info-am install-man \ + install-pdf install-pdf-am install-pluginsPROGRAMS install-ps \ + install-ps-am install-sbinPROGRAMS install-strip installcheck \ + installcheck-am installdirs maintainer-clean \ maintainer-clean-generic mostlyclean mostlyclean-compile \ mostlyclean-generic pdf pdf-am ps ps-am tags tags-am uninstall \ uninstall-am uninstall-dist_cacheDATA uninstall-dist_logDATA \ + uninstall-dist_registryDATA uninstall-dist_varlibDATA \ uninstall-pluginsPROGRAMS uninstall-sbinPROGRAMS diff --git a/src/appconfig.c b/src/appconfig.c index 0ec4cad32..34fb6d7d3 100644 --- a/src/appconfig.c +++ b/src/appconfig.c @@ -1,22 +1,4 @@ - -/* - * TODO - * - * 1. Re-write this using DICTIONARY - * - */ - -#ifdef HAVE_CONFIG_H -#include <config.h> -#endif -#include <pthread.h> -#include <stdlib.h> -#include <string.h> - -#include "avl.h" #include "common.h" -#include "appconfig.h" -#include "log.h" #define CONFIG_FILE_LINE_MAX ((CONFIG_MAX_NAME + CONFIG_MAX_VALUE + 1024) * 2) @@ -31,33 +13,33 @@ pthread_mutex_t config_mutex = PTHREAD_MUTEX_INITIALIZER; #define CONFIG_VALUE_CHECKED 0x08 // has been checked if the value is different from the default struct config_value { - avl avl; // the index - this has to be first! + avl avl; // the index - this has to be first! - uint8_t flags; - uint32_t hash; // a simple hash to speed up searching - // we first compare hashes, and only if the hashes are equal we do string comparisons + uint8_t flags; + uint32_t hash; // a simple hash to speed up searching + // we first compare hashes, and only if the hashes are equal we do string comparisons - char *name; - char *value; + char *name; + char *value; - struct config_value *next; // config->mutex protects just this + struct config_value *next; // config->mutex protects just this }; struct config { - avl avl; + avl avl; - uint32_t hash; // a simple hash to speed up searching - // we first compare hashes, and only if the hashes are equal we do string comparisons + uint32_t hash; // a simple hash to speed up searching + // we first compare hashes, and only if the hashes are equal we do string comparisons - char *name; + char *name; - struct config *next; // gloabl config_mutex protects just this + struct config *next; // gloabl config_mutex protects just this - struct config_value *values; - avl_tree_lock values_index; + struct config_value *values; + avl_tree_lock values_index; - pthread_mutex_t mutex; // this locks only the writers, to ensure atomic updates - // readers are protected using the rwlock in avl_tree_lock + pthread_mutex_t mutex; // this locks only the writers, to ensure atomic updates + // readers are protected using the rwlock in avl_tree_lock } *config_root = NULL; @@ -65,19 +47,19 @@ struct config { // locking static inline void config_global_write_lock(void) { - pthread_mutex_lock(&config_mutex); + pthread_mutex_lock(&config_mutex); } static inline void config_global_unlock(void) { - pthread_mutex_unlock(&config_mutex); + pthread_mutex_unlock(&config_mutex); } static inline void config_section_write_lock(struct config *co) { - pthread_mutex_lock(&co->mutex); + pthread_mutex_lock(&co->mutex); } static inline void config_section_unlock(struct config *co) { - pthread_mutex_unlock(&co->mutex); + pthread_mutex_unlock(&co->mutex); } @@ -85,20 +67,20 @@ static inline void config_section_unlock(struct config *co) { // config name-value index static int config_value_compare(void* a, void* b) { - if(((struct config_value *)a)->hash < ((struct config_value *)b)->hash) return -1; - else if(((struct config_value *)a)->hash > ((struct config_value *)b)->hash) return 1; - else return strcmp(((struct config_value *)a)->name, ((struct config_value *)b)->name); + if(((struct config_value *)a)->hash < ((struct config_value *)b)->hash) return -1; + else if(((struct config_value *)a)->hash > ((struct config_value *)b)->hash) return 1; + else return strcmp(((struct config_value *)a)->name, ((struct config_value *)b)->name); } #define config_value_index_add(co, cv) avl_insert_lock(&((co)->values_index), (avl *)(cv)) #define config_value_index_del(co, cv) avl_remove_lock(&((co)->values_index), (avl *)(cv)) static struct config_value *config_value_index_find(struct config *co, const char *name, uint32_t hash) { - struct config_value tmp; - tmp.hash = (hash)?hash:simple_hash(name); - tmp.name = (char *)name; + struct config_value tmp; + tmp.hash = (hash)?hash:simple_hash(name); + tmp.name = (char *)name; - return (struct config_value *)avl_search_lock(&(co->values_index), (avl *) &tmp); + return (struct config_value *)avl_search_lock(&(co->values_index), (avl *) &tmp); } @@ -106,25 +88,25 @@ static struct config_value *config_value_index_find(struct config *co, const cha // config sections index static int config_compare(void* a, void* b) { - if(((struct config *)a)->hash < ((struct config *)b)->hash) return -1; - else if(((struct config *)a)->hash > ((struct config *)b)->hash) return 1; - else return strcmp(((struct config *)a)->name, ((struct config *)b)->name); + if(((struct config *)a)->hash < ((struct config *)b)->hash) return -1; + else if(((struct config *)a)->hash > ((struct config *)b)->hash) return 1; + else return strcmp(((struct config *)a)->name, ((struct config *)b)->name); } avl_tree_lock config_root_index = { - { NULL, config_compare }, - AVL_LOCK_INITIALIZER + { NULL, config_compare }, + AVL_LOCK_INITIALIZER }; #define config_index_add(cfg) avl_insert_lock(&config_root_index, (avl *)(cfg)) #define config_index_del(cfg) avl_remove_lock(&config_root_index, (avl *)(cfg)) static struct config *config_index_find(const char *name, uint32_t hash) { - struct config tmp; - tmp.hash = (hash)?hash:simple_hash(name); - tmp.name = (char *)name; + struct config tmp; + tmp.hash = (hash)?hash:simple_hash(name); + tmp.name = (char *)name; - return (struct config *)avl_search_lock(&config_root_index, (avl *) &tmp); + return (struct config *)avl_search_lock(&config_root_index, (avl *) &tmp); } @@ -132,34 +114,31 @@ static struct config *config_index_find(const char *name, uint32_t hash) { // config section methods static inline struct config *config_section_find(const char *section) { - return config_index_find(section, 0); + return config_index_find(section, 0); } static inline struct config *config_section_create(const char *section) { - debug(D_CONFIG, "Creating section '%s'.", section); - - struct config *co = calloc(1, sizeof(struct config)); - if(!co) fatal("Cannot allocate config"); + debug(D_CONFIG, "Creating section '%s'.", section); - co->name = strdup(section); - if(!co->name) fatal("Cannot allocate config.name"); - co->hash = simple_hash(co->name); + struct config *co = callocz(1, sizeof(struct config)); + co->name = strdupz(section); + co->hash = simple_hash(co->name); - avl_init_lock(&co->values_index, config_value_compare); + avl_init_lock(&co->values_index, config_value_compare); - config_index_add(co); + config_index_add(co); - config_global_write_lock(); - struct config *co2 = config_root; - if(co2) { - while (co2->next) co2 = co2->next; - co2->next = co; - } - else config_root = co; - config_global_unlock(); + config_global_write_lock(); + struct config *co2 = config_root; + if(co2) { + while (co2->next) co2 = co2->next; + co2->next = co; + } + else config_root = co; + config_global_unlock(); - return co; + return co; } @@ -168,181 +147,221 @@ static inline struct config *config_section_create(const char *section) static inline struct config_value *config_value_create(struct config *co, const char *name, const char *value) { - debug(D_CONFIG, "Creating config entry for name '%s', value '%s', in section '%s'.", name, value, co->name); + debug(D_CONFIG, "Creating config entry for name '%s', value '%s', in section '%s'.", name, value, co->name); + + struct config_value *cv = callocz(1, sizeof(struct config_value)); + cv->name = strdupz(name); + cv->hash = simple_hash(cv->name); + cv->value = strdupz(value); + + config_value_index_add(co, cv); + + config_section_write_lock(co); + struct config_value *cv2 = co->values; + if(cv2) { + while (cv2->next) cv2 = cv2->next; + cv2->next = cv; + } + else co->values = cv; + config_section_unlock(co); + + return cv; +} + +int config_exists(const char *section, const char *name) { + struct config_value *cv; + + debug(D_CONFIG, "request to get config in section '%s', name '%s'", section, name); + + struct config *co = config_section_find(section); + if(!co) return 0; + + cv = config_value_index_find(co, name, 0); + if(!cv) return 0; + + return 1; +} + +int config_rename(const char *section, const char *old, const char *new) { + struct config_value *cv, *cv2; + + debug(D_CONFIG, "request to rename config in section '%s', old name '%s', new name '%s'", section, old, new); + + struct config *co = config_section_find(section); + if(!co) return -1; + + config_section_write_lock(co); + + cv = config_value_index_find(co, old, 0); + if(!cv) goto cleanup; + + cv2 = config_value_index_find(co, new, 0); + if(cv2) goto cleanup; - struct config_value *cv = calloc(1, sizeof(struct config_value)); - if(!cv) fatal("Cannot allocate config_value"); + config_value_index_del(co, cv); - cv->name = strdup(name); - if(!cv->name) fatal("Cannot allocate config.name"); - cv->hash = simple_hash(cv->name); + freez(cv->name); + cv->name = strdupz(new); - cv->value = strdup(value); - if(!cv->value) fatal("Cannot allocate config.value"); + cv->hash = simple_hash(cv->name); - config_value_index_add(co, cv); + config_value_index_add(co, cv); + config_section_unlock(co); - config_section_write_lock(co); - struct config_value *cv2 = co->values; - if(cv2) { - while (cv2->next) cv2 = cv2->next; - cv2->next = cv; - } - else co->values = cv; - config_section_unlock(co); + return 0; - return cv; +cleanup: + config_section_unlock(co); + return -1; } char *config_get(const char *section, const char *name, const char *default_value) { - struct config_value *cv; - - debug(D_CONFIG, "request to get config in section '%s', name '%s', default_value '%s'", section, name, default_value); - - struct config *co = config_section_find(section); - if(!co) co = config_section_create(section); - - cv = config_value_index_find(co, name, 0); - if(!cv) { - cv = config_value_create(co, name, default_value); - if(!cv) return NULL; - } - cv->flags |= CONFIG_VALUE_USED; - - if((cv->flags & CONFIG_VALUE_LOADED) || (cv->flags & CONFIG_VALUE_CHANGED)) { - // this is a loaded value from the config file - // if it is different that the default, mark it - if(!(cv->flags & CONFIG_VALUE_CHECKED)) { - if(strcmp(cv->value, default_value) != 0) cv->flags |= CONFIG_VALUE_CHANGED; - cv->flags |= CONFIG_VALUE_CHECKED; - } - } - - return(cv->value); + struct config_value *cv; + + debug(D_CONFIG, "request to get config in section '%s', name '%s', default_value '%s'", section, name, default_value); + + struct config *co = config_section_find(section); + if(!co) co = config_section_create(section); + + cv = config_value_index_find(co, name, 0); + if(!cv) { + cv = config_value_create(co, name, default_value); + if(!cv) return NULL; + } + cv->flags |= CONFIG_VALUE_USED; + + if((cv->flags & CONFIG_VALUE_LOADED) || (cv->flags & CONFIG_VALUE_CHANGED)) { + // this is a loaded value from the config file + // if it is different that the default, mark it + if(!(cv->flags & CONFIG_VALUE_CHECKED)) { + if(strcmp(cv->value, default_value) != 0) cv->flags |= CONFIG_VALUE_CHANGED; + cv->flags |= CONFIG_VALUE_CHECKED; + } + } + + return(cv->value); } long long config_get_number(const char *section, const char *name, long long value) { - char buffer[100], *s; - sprintf(buffer, "%lld", value); + char buffer[100], *s; + sprintf(buffer, "%lld", value); - s = config_get(section, name, buffer); - if(!s) return value; + s = config_get(section, name, buffer); + if(!s) return value; - return strtoll(s, NULL, 0); + return strtoll(s, NULL, 0); } int config_get_boolean(const char *section, const char *name, int value) { - char *s; - if(value) s = "yes"; - else s = "no"; + char *s; + if(value) s = "yes"; + else s = "no"; - s = config_get(section, name, s); - if(!s) return value; + s = config_get(section, name, s); + if(!s) return value; - if(!strcmp(s, "yes")) return 1; - else return 0; + if(!strcmp(s, "yes") || !strcmp(s, "auto") || !strcmp(s, "on demand")) return 1; + return 0; } int config_get_boolean_ondemand(const char *section, const char *name, int value) { - char *s; + char *s; - if(value == CONFIG_ONDEMAND_ONDEMAND) - s = "on demand"; + if(value == CONFIG_ONDEMAND_ONDEMAND) + s = "auto"; - else if(value == CONFIG_ONDEMAND_NO) - s = "no"; + else if(value == CONFIG_ONDEMAND_NO) + s = "no"; - else - s = "yes"; + else + s = "yes"; - s = config_get(section, name, s); - if(!s) return value; + s = config_get(section, name, s); + if(!s) return value; - if(!strcmp(s, "yes")) - return CONFIG_ONDEMAND_YES; - else if(!strcmp(s, "no")) - return CONFIG_ONDEMAND_NO; - else if(!strcmp(s, "on demand")) - return CONFIG_ONDEMAND_ONDEMAND; + if(!strcmp(s, "yes")) + return CONFIG_ONDEMAND_YES; + else if(!strcmp(s, "no")) + return CONFIG_ONDEMAND_NO; + else if(!strcmp(s, "auto") || !strcmp(s, "on demand")) + return CONFIG_ONDEMAND_ONDEMAND; - return value; + return value; } const char *config_set_default(const char *section, const char *name, const char *value) { - struct config_value *cv; + struct config_value *cv; - debug(D_CONFIG, "request to set config in section '%s', name '%s', value '%s'", section, name, value); + debug(D_CONFIG, "request to set config in section '%s', name '%s', value '%s'", section, name, value); - struct config *co = config_section_find(section); - if(!co) return config_set(section, name, value); + struct config *co = config_section_find(section); + if(!co) return config_set(section, name, value); - cv = config_value_index_find(co, name, 0); - if(!cv) return config_set(section, name, value); + cv = config_value_index_find(co, name, 0); + if(!cv) return config_set(section, name, value); - cv->flags |= CONFIG_VALUE_USED; + cv->flags |= CONFIG_VALUE_USED; - if(cv->flags & CONFIG_VALUE_LOADED) - return cv->value; + if(cv->flags & CONFIG_VALUE_LOADED) + return cv->value; - if(strcmp(cv->value, value) != 0) { - cv->flags |= CONFIG_VALUE_CHANGED; + if(strcmp(cv->value, value) != 0) { + cv->flags |= CONFIG_VALUE_CHANGED; - free(cv->value); - cv->value = strdup(value); - if(!cv->value) fatal("Cannot allocate config.value"); - } + freez(cv->value); + cv->value = strdupz(value); + } - return cv->value; + return cv->value; } const char *config_set(const char *section, const char *name, const char *value) { - struct config_value *cv; + struct config_value *cv; - debug(D_CONFIG, "request to set config in section '%s', name '%s', value '%s'", section, name, value); + debug(D_CONFIG, "request to set config in section '%s', name '%s', value '%s'", section, name, value); - struct config *co = config_section_find(section); - if(!co) co = config_section_create(section); + struct config *co = config_section_find(section); + if(!co) co = config_section_create(section); - cv = config_value_index_find(co, name, 0); - if(!cv) cv = config_value_create(co, name, value); - cv->flags |= CONFIG_VALUE_USED; + cv = config_value_index_find(co, name, 0); + if(!cv) cv = config_value_create(co, name, value); + cv->flags |= CONFIG_VALUE_USED; - if(strcmp(cv->value, value) != 0) { - cv->flags |= CONFIG_VALUE_CHANGED; + if(strcmp(cv->value, value) != 0) { + cv->flags |= CONFIG_VALUE_CHANGED; - free(cv->value); - cv->value = strdup(value); - if(!cv->value) fatal("Cannot allocate config.value"); - } + freez(cv->value); + cv->value = strdupz(value); + } - return value; + return value; } long long config_set_number(const char *section, const char *name, long long value) { - char buffer[100]; - sprintf(buffer, "%lld", value); + char buffer[100]; + sprintf(buffer, "%lld", value); - config_set(section, name, buffer); + config_set(section, name, buffer); - return value; + return value; } int config_set_boolean(const char *section, const char *name, int value) { - char *s; - if(value) s = "yes"; - else s = "no"; + char *s; + if(value) s = "yes"; + else s = "no"; - config_set(section, name, s); + config_set(section, name, s); - return value; + return value; } @@ -351,152 +370,158 @@ int config_set_boolean(const char *section, const char *name, int value) int load_config(char *filename, int overwrite_used) { - int line = 0; - struct config *co = NULL; - - char buffer[CONFIG_FILE_LINE_MAX + 1], *s; - - if(!filename) filename = CONFIG_DIR "/" CONFIG_FILENAME; - FILE *fp = fopen(filename, "r"); - if(!fp) { - error("Cannot open file '%s'", filename); - return 0; - } - - while(fgets(buffer, CONFIG_FILE_LINE_MAX, fp) != NULL) { - buffer[CONFIG_FILE_LINE_MAX] = '\0'; - line++; - - s = trim(buffer); - if(!s) { - debug(D_CONFIG, "Ignoring line %d, it is empty.", line); - continue; - } - - int len = (int) strlen(s); - if(*s == '[' && s[len - 1] == ']') { - // new section - s[len - 1] = '\0'; - s++; - - co = config_section_find(s); - if(!co) co = config_section_create(s); - - continue; - } - - if(!co) { - // line outside a section - error("Ignoring line %d ('%s'), it is outside all sections.", line, s); - continue; - } - - char *name = s; - char *value = strchr(s, '='); - if(!value) { - error("Ignoring line %d ('%s'), there is no = in it.", line, s); - continue; - } - *value = '\0'; - value++; - - name = trim(name); - value = trim(value); - - if(!name) { - error("Ignoring line %d, name is empty.", line); - continue; - } - if(!value) { - debug(D_CONFIG, "Ignoring line %d, value is empty.", line); - continue; - } - - struct config_value *cv = config_value_index_find(co, name, 0); - - if(!cv) cv = config_value_create(co, name, value); - else { - if(((cv->flags & CONFIG_VALUE_USED) && overwrite_used) || !(cv->flags & CONFIG_VALUE_USED)) { - debug(D_CONFIG, "Overwriting '%s/%s'.", line, co->name, cv->name); - free(cv->value); - cv->value = strdup(value); - if(!cv->value) fatal("Cannot allocate config.value"); - } - else - debug(D_CONFIG, "Ignoring line %d, '%s/%s' is already present and used.", line, co->name, cv->name); - } - cv->flags |= CONFIG_VALUE_LOADED; - } - - fclose(fp); - - return 1; + int line = 0; + struct config *co = NULL; + + char buffer[CONFIG_FILE_LINE_MAX + 1], *s; + + if(!filename) filename = CONFIG_DIR "/" CONFIG_FILENAME; + + debug(D_CONFIG, "Opening config file '%s'", filename); + + FILE *fp = fopen(filename, "r"); + if(!fp) { + error("Cannot open file '%s'", filename); + return 0; + } + + while(fgets(buffer, CONFIG_FILE_LINE_MAX, fp) != NULL) { + buffer[CONFIG_FILE_LINE_MAX] = '\0'; + line++; + + s = trim(buffer); + if(!s) { + debug(D_CONFIG, "Ignoring line %d, it is empty.", line); + continue; + } + + int len = (int) strlen(s); + if(*s == '[' && s[len - 1] == ']') { + // new section + s[len - 1] = '\0'; + s++; + + co = config_section_find(s); + if(!co) co = config_section_create(s); + + continue; + } + + if(!co) { + // line outside a section + error("Ignoring line %d ('%s'), it is outside all sections.", line, s); + continue; + } + + char *name = s; + char *value = strchr(s, '='); + if(!value) { + error("Ignoring line %d ('%s'), there is no = in it.", line, s); + continue; + } + *value = '\0'; + value++; + + name = trim(name); + value = trim(value); + + if(!name) { + error("Ignoring line %d, name is empty.", line); + continue; + } + if(!value) { + debug(D_CONFIG, "Ignoring line %d, value is empty.", line); + continue; + } + + struct config_value *cv = config_value_index_find(co, name, 0); + + if(!cv) cv = config_value_create(co, name, value); + else { + if(((cv->flags & CONFIG_VALUE_USED) && overwrite_used) || !(cv->flags & CONFIG_VALUE_USED)) { + debug(D_CONFIG, "Line %d, overwriting '%s/%s'.", line, co->name, cv->name); + freez(cv->value); + cv->value = strdupz(value); + } + else + debug(D_CONFIG, "Ignoring line %d, '%s/%s' is already present and used.", line, co->name, cv->name); + } + cv->flags |= CONFIG_VALUE_LOADED; + } + + fclose(fp); + + return 1; } void generate_config(BUFFER *wb, int only_changed) { - int i, pri; - struct config *co; - struct config_value *cv; - - for(i = 0; i < 3 ;i++) { - switch(i) { - case 0: - buffer_strcat(wb, - "# NetData Configuration\n" - "# You can uncomment and change any of the options below.\n" - "# The value shown in the commented settings, is the default value.\n" - "\n# global netdata configuration\n"); - break; - - case 1: - buffer_strcat(wb, "\n\n# per plugin configuration\n"); - break; - - case 2: - buffer_strcat(wb, "\n\n# per chart configuration\n"); - break; - } - - config_global_write_lock(); - for(co = config_root; co ; co = co->next) { - if(strcmp(co->name, "global") == 0 || strcmp(co->name, "plugins") == 0 || strcmp(co->name, "registry") == 0) pri = 0; - else if(strncmp(co->name, "plugin:", 7) == 0) pri = 1; - else pri = 2; - - if(i == pri) { - int used = 0; - int changed = 0; - int count = 0; - - config_section_write_lock(co); - for(cv = co->values; cv ; cv = cv->next) { - used += (cv->flags & CONFIG_VALUE_USED)?1:0; - changed += (cv->flags & CONFIG_VALUE_CHANGED)?1:0; - count++; - } - config_section_unlock(co); - - if(!count) continue; - if(only_changed && !changed) continue; - - if(!used) { - buffer_sprintf(wb, "\n# node '%s' is not used.", co->name); - } - - buffer_sprintf(wb, "\n[%s]\n", co->name); - - config_section_write_lock(co); - for(cv = co->values; cv ; cv = cv->next) { - - if(used && !(cv->flags & CONFIG_VALUE_USED)) { - buffer_sprintf(wb, "\n\t# option '%s' is not used.\n", cv->name); - } - buffer_sprintf(wb, "\t%s%s = %s\n", ((!(cv->flags & CONFIG_VALUE_CHANGED)) && (cv->flags & CONFIG_VALUE_USED))?"# ":"", cv->name, cv->value); - } - config_section_unlock(co); - } - } - config_global_unlock(); - } + int i, pri; + struct config *co; + struct config_value *cv; + + for(i = 0; i < 3 ;i++) { + switch(i) { + case 0: + buffer_strcat(wb, + "# NetData Configuration\n" + "# You can uncomment and change any of the options below.\n" + "# The value shown in the commented settings, is the default value.\n" + "\n# global netdata configuration\n"); + break; + + case 1: + buffer_strcat(wb, "\n\n# per plugin configuration\n"); + break; + + case 2: + buffer_strcat(wb, "\n\n# per chart configuration\n"); + break; + } + + config_global_write_lock(); + for(co = config_root; co ; co = co->next) { + if(!strcmp(co->name, "global") || + !strcmp(co->name, "plugins") || + !strcmp(co->name, "registry") || + !strcmp(co->name, "health")) + pri = 0; + else if(!strncmp(co->name, "plugin:", 7)) pri = 1; + else pri = 2; + + if(i == pri) { + int used = 0; + int changed = 0; + int count = 0; + + config_section_write_lock(co); + for(cv = co->values; cv ; cv = cv->next) { + used += (cv->flags & CONFIG_VALUE_USED)?1:0; + changed += (cv->flags & CONFIG_VALUE_CHANGED)?1:0; + count++; + } + config_section_unlock(co); + + if(!count) continue; + if(only_changed && !changed) continue; + + if(!used) { + buffer_sprintf(wb, "\n# node '%s' is not used.", co->name); + } + + buffer_sprintf(wb, "\n[%s]\n", co->name); + + config_section_write_lock(co); + for(cv = co->values; cv ; cv = cv->next) { + + if(used && !(cv->flags & CONFIG_VALUE_USED)) { + buffer_sprintf(wb, "\n\t# option '%s' is not used.\n", cv->name); + } + buffer_sprintf(wb, "\t%s%s = %s\n", ((!(cv->flags & CONFIG_VALUE_CHANGED)) && (cv->flags & CONFIG_VALUE_USED))?"# ":"", cv->name, cv->value); + } + config_section_unlock(co); + } + } + config_global_unlock(); + } } diff --git a/src/appconfig.h b/src/appconfig.h index 41d1e19bb..08aae8348 100644 --- a/src/appconfig.h +++ b/src/appconfig.h @@ -1,5 +1,3 @@ -#include "web_buffer.h" - #ifndef NETDATA_CONFIG_H #define NETDATA_CONFIG_H 1 @@ -26,6 +24,9 @@ extern const char *config_set_default(const char *section, const char *name, con extern long long config_set_number(const char *section, const char *name, long long value); extern int config_set_boolean(const char *section, const char *name, int value); +extern int config_exists(const char *section, const char *name); +extern int config_rename(const char *section, const char *old, const char *new); + extern void generate_config(BUFFER *wb, int only_changed); #endif /* NETDATA_CONFIG_H */ diff --git a/src/apps_plugin.c b/src/apps_plugin.c index 0bcdfcf50..ee400b72e 100644 --- a/src/apps_plugin.c +++ b/src/apps_plugin.c @@ -1,272 +1,96 @@ -// TODO -// -// 1. disable RESET_OR_OVERFLOW check in charts - -#ifdef HAVE_CONFIG_H -#include <config.h> -#endif -#include <stdio.h> -#include <stdlib.h> -#include <string.h> -#include <time.h> -#include <unistd.h> -#include <sys/types.h> -#include <sys/time.h> -#include <sys/wait.h> -#include <sys/stat.h> - -#include <sys/resource.h> -#include <sys/stat.h> - -#include <errno.h> -#include <stdarg.h> -#include <locale.h> -#include <ctype.h> -#include <fcntl.h> - -#include <malloc.h> -#include <dirent.h> -#include <arpa/inet.h> - -#include <sys/types.h> -#include <pwd.h> -#include <grp.h> - -#include "avl.h" - #include "common.h" -#include "log.h" -#include "procfile.h" -#include "../config.h" - -#ifdef NETDATA_INTERNAL_CHECKS -#include <sys/prctl.h> -#endif #define MAX_COMPARE_NAME 100 #define MAX_NAME 100 #define MAX_CMDLINE 1024 -long processors = 1; -long pid_max = 32768; +// the rates we are going to send to netdata +// will have this detail +// a value of: +// 1 will send just integer parts to netdata +// 100 will send 2 decimal points +// 1000 will send 3 decimal points +// etc. +#define RATES_DETAIL 10000ULL + +int processors = 1; +pid_t pid_max = 32768; int debug = 0; int update_every = 1; +unsigned long long global_iterations_counter = 1; unsigned long long file_counter = 0; int proc_pid_cmdline_is_needed = 0; - +int include_exited_childs = 1; char *host_prefix = ""; char *config_dir = CONFIG_DIR; -#ifdef NETDATA_INTERNAL_CHECKS -// ---------------------------------------------------------------------------- -// memory debugger -// do not use in production systems - it mis-aligns allocated memory - -struct allocations { - size_t allocations; - size_t allocated; - size_t allocated_max; -} allocations = { 0, 0, 0 }; - -#define MALLOC_MARK (uint32_t)(0x0BADCAFE) -#define MALLOC_PREFIX (sizeof(uint32_t) * 2) -#define MALLOC_SUFFIX (sizeof(uint32_t)) -#define MALLOC_OVERHEAD (MALLOC_PREFIX + MALLOC_SUFFIX) - -void *mark_allocation(void *allocated_ptr, size_t size_without_overheads) { - uint32_t *real_ptr = (uint32_t *)allocated_ptr; - real_ptr[0] = MALLOC_MARK; - real_ptr[1] = (uint32_t) size_without_overheads; - - uint32_t *end_ptr = (uint32_t *)(allocated_ptr + MALLOC_PREFIX + size_without_overheads); - end_ptr[0] = MALLOC_MARK; - - // fprintf(stderr, "MEMORY_POINTER: Allocated at %p, returning %p.\n", allocated_ptr, (void *)(allocated_ptr + MALLOC_PREFIX)); - - return allocated_ptr + MALLOC_PREFIX; -} - -void *check_allocation(const char *file, int line, const char *function, void *marked_ptr, size_t *size_without_overheads_ptr) { - uint32_t *real_ptr = (uint32_t *)(marked_ptr - MALLOC_PREFIX); - - // fprintf(stderr, "MEMORY_POINTER: Checking pointer at %p, real %p for %s/%u@%s.\n", marked_ptr, (void *)(marked_ptr - MALLOC_PREFIX), function, line, file); - - if(real_ptr[0] != MALLOC_MARK) fatal("MEMORY: prefix MARK is not valid for %s/%u@%s.", function, line, file); - - size_t size = real_ptr[1]; - - uint32_t *end_ptr = (uint32_t *)(marked_ptr + size); - if(end_ptr[0] != MALLOC_MARK) fatal("MEMORY: suffix MARK of allocation with size %zu is not valid for %s/%u@%s.", size, function, line, file); - - if(size_without_overheads_ptr) *size_without_overheads_ptr = size; - - return real_ptr; -} - -void *malloc_debug(const char *file, int line, const char *function, size_t size) { - void *ptr = malloc(size + MALLOC_OVERHEAD); - if(!ptr) fatal("MEMORY: Cannot allocate %zu bytes for %s/%u@%s.", size, function, line, file); - - allocations.allocated += size; - allocations.allocations++; - - debug(D_MEMORY, "MEMORY: Allocated %zu bytes for %s/%u@%s." - " Status: allocated %zu in %zu allocs." - , size - , function, line, file - , allocations.allocated - , allocations.allocations - ); - - if(allocations.allocated > allocations.allocated_max) { - debug(D_MEMORY, "MEMORY: total allocation peak increased from %zu to %zu", allocations.allocated_max, allocations.allocated); - allocations.allocated_max = allocations.allocated; - } - - size_t csize; - check_allocation(file, line, function, mark_allocation(ptr, size), &csize); - if(size != csize) { - fatal("Invalid size."); - } - - return mark_allocation(ptr, size); -} - -void *calloc_debug(const char *file, int line, const char *function, size_t nmemb, size_t size) { - void *ptr = malloc_debug(file, line, function, (nmemb * size)); - bzero(ptr, nmemb * size); - return ptr; -} - -void free_debug(const char *file, int line, const char *function, void *ptr) { - size_t size; - void *real_ptr = check_allocation(file, line, function, ptr, &size); - - bzero(real_ptr, size + MALLOC_OVERHEAD); - - free(real_ptr); - allocations.allocated -= size; - allocations.allocations--; - - debug(D_MEMORY, "MEMORY: freed %zu bytes for %s/%u@%s." - " Status: allocated %zu in %zu allocs." - , size - , function, line, file - , allocations.allocated - , allocations.allocations - ); -} - -void *realloc_debug(const char *file, int line, const char *function, void *ptr, size_t size) { - if(!ptr) return malloc_debug(file, line, function, size); - if(!size) { free_debug(file, line, function, ptr); return NULL; } - - size_t old_size; - void *real_ptr = check_allocation(file, line, function, ptr, &old_size); - - void *new_ptr = realloc(real_ptr, size + MALLOC_OVERHEAD); - if(!new_ptr) fatal("MEMORY: Cannot allocate %zu bytes for %s/%u@%s.", size, function, line, file); +pid_t *all_pids_sortlist = NULL; - allocations.allocated += size; - allocations.allocated -= old_size; +// will be automatically set to 1, if guest values are collected +int show_guest_time = 0; +int show_guest_time_old = 0; - debug(D_MEMORY, "MEMORY: Re-allocated from %zu to %zu bytes for %s/%u@%s." - " Status: allocated %z in %zu allocs." - , old_size, size - , function, line, file - , allocations.allocated - , allocations.allocations - ); +int enable_guest_charts = 0; +int enable_file_charts = 1; - if(allocations.allocated > allocations.allocated_max) { - debug(D_MEMORY, "MEMORY: total allocation peak increased from %zu to %zu", allocations.allocated_max, allocations.allocated); - allocations.allocated_max = allocations.allocated; - } - - return mark_allocation(new_ptr, size); -} - -char *strdup_debug(const char *file, int line, const char *function, const char *ptr) { - size_t size = 0; - const char *s = ptr; - - while(*s++) size++; - size++; - - char *p = malloc_debug(file, line, function, size); - if(!p) fatal("Cannot allocate %zu bytes.", size); +// ---------------------------------------------------------------------------- - memcpy(p, ptr, size); - return p; +void netdata_cleanup_and_exit(int ret) { + exit(ret); } -#define malloc(size) malloc_debug(__FILE__, __LINE__, __FUNCTION__, (size)) -#define calloc(nmemb, size) calloc_debug(__FILE__, __LINE__, __FUNCTION__, (nmemb), (size)) -#define realloc(ptr, size) realloc_debug(__FILE__, __LINE__, __FUNCTION__, (ptr), (size)) -#define free(ptr) free_debug(__FILE__, __LINE__, __FUNCTION__, (ptr)) - -#ifdef strdup -#undef strdup -#endif -#define strdup(ptr) strdup_debug(__FILE__, __LINE__, __FUNCTION__, (ptr)) - -#endif /* NETDATA_INTERNAL_CHECKS */ // ---------------------------------------------------------------------------- // system functions // to retrieve settings of the system long get_system_cpus(void) { - procfile *ff = NULL; + procfile *ff = NULL; - int processors = 0; + int processors = 0; - char filename[FILENAME_MAX + 1]; - snprintfz(filename, FILENAME_MAX, "%s/proc/stat", host_prefix); + char filename[FILENAME_MAX + 1]; + snprintfz(filename, FILENAME_MAX, "%s/proc/stat", host_prefix); - ff = procfile_open(filename, NULL, PROCFILE_FLAG_DEFAULT); - if(!ff) return 1; + ff = procfile_open(filename, NULL, PROCFILE_FLAG_DEFAULT); + if(!ff) return 1; - ff = procfile_readall(ff); - if(!ff) { - procfile_close(ff); - return 1; - } + ff = procfile_readall(ff); + if(!ff) + return 1; - unsigned int i; - for(i = 0; i < procfile_lines(ff); i++) { - if(!procfile_linewords(ff, i)) continue; + unsigned int i; + for(i = 0; i < procfile_lines(ff); i++) { + if(!procfile_linewords(ff, i)) continue; - if(strncmp(procfile_lineword(ff, i, 0), "cpu", 3) == 0) processors++; - } - processors--; - if(processors < 1) processors = 1; + if(strncmp(procfile_lineword(ff, i, 0), "cpu", 3) == 0) processors++; + } + processors--; + if(processors < 1) processors = 1; - procfile_close(ff); - return processors; + procfile_close(ff); + return processors; } -long get_system_pid_max(void) { - procfile *ff = NULL; - long mpid = 32768; +pid_t get_system_pid_max(void) { + procfile *ff = NULL; + pid_t mpid = 32768; - char filename[FILENAME_MAX + 1]; - snprintfz(filename, FILENAME_MAX, "%s/proc/sys/kernel/pid_max", host_prefix); - ff = procfile_open(filename, NULL, PROCFILE_FLAG_DEFAULT); - if(!ff) return mpid; + char filename[FILENAME_MAX + 1]; + snprintfz(filename, FILENAME_MAX, "%s/proc/sys/kernel/pid_max", host_prefix); + ff = procfile_open(filename, NULL, PROCFILE_FLAG_DEFAULT); + if(!ff) return mpid; - ff = procfile_readall(ff); - if(!ff) { - procfile_close(ff); - return mpid; - } + ff = procfile_readall(ff); + if(!ff) + return mpid; - mpid = atol(procfile_lineword(ff, 0, 0)); - if(!mpid) mpid = 32768; + mpid = (pid_t)atoi(procfile_lineword(ff, 0, 0)); + if(!mpid) mpid = 32768; - procfile_close(ff); - return mpid; + procfile_close(ff); + return mpid; } // ---------------------------------------------------------------------------- @@ -274,83 +98,68 @@ long get_system_pid_max(void) { // target is the structure that process data are aggregated struct target { - char compare[MAX_COMPARE_NAME + 1]; - uint32_t comparehash; - size_t comparelen; - - char id[MAX_NAME + 1]; - uint32_t idhash; - - char name[MAX_NAME + 1]; - - uid_t uid; - gid_t gid; - - unsigned long long minflt; - unsigned long long cminflt; - unsigned long long majflt; - unsigned long long cmajflt; - unsigned long long utime; - unsigned long long stime; - unsigned long long cutime; - unsigned long long cstime; - unsigned long long num_threads; - unsigned long long rss; - - unsigned long long fix_minflt; - unsigned long long fix_cminflt; - unsigned long long fix_majflt; - unsigned long long fix_cmajflt; - unsigned long long fix_utime; - unsigned long long fix_stime; - unsigned long long fix_cutime; - unsigned long long fix_cstime; - - unsigned long long statm_size; - unsigned long long statm_resident; - unsigned long long statm_share; - unsigned long long statm_text; - unsigned long long statm_lib; - unsigned long long statm_data; - unsigned long long statm_dirty; - - unsigned long long io_logical_bytes_read; - unsigned long long io_logical_bytes_written; - unsigned long long io_read_calls; - unsigned long long io_write_calls; - unsigned long long io_storage_bytes_read; - unsigned long long io_storage_bytes_written; - unsigned long long io_cancelled_write_bytes; - - unsigned long long fix_io_logical_bytes_read; - unsigned long long fix_io_logical_bytes_written; - unsigned long long fix_io_read_calls; - unsigned long long fix_io_write_calls; - unsigned long long fix_io_storage_bytes_read; - unsigned long long fix_io_storage_bytes_written; - unsigned long long fix_io_cancelled_write_bytes; - - int *fds; - unsigned long long openfiles; - unsigned long long openpipes; - unsigned long long opensockets; - unsigned long long openinotifies; - unsigned long long openeventfds; - unsigned long long opentimerfds; - unsigned long long opensignalfds; - unsigned long long openeventpolls; - unsigned long long openother; - - unsigned long processes; // how many processes have been merged to this - int exposed; // if set, we have sent this to netdata - int hidden; // if set, we set the hidden flag on the dimension - int debug; - int ends_with; - int starts_with; // if set, the compare string matches only the - // beginning of the command - - struct target *target; // the one that will be reported to netdata - struct target *next; + char compare[MAX_COMPARE_NAME + 1]; + uint32_t comparehash; + size_t comparelen; + + char id[MAX_NAME + 1]; + uint32_t idhash; + + char name[MAX_NAME + 1]; + + uid_t uid; + gid_t gid; + + unsigned long long minflt; + unsigned long long cminflt; + unsigned long long majflt; + unsigned long long cmajflt; + unsigned long long utime; + unsigned long long stime; + unsigned long long gtime; + unsigned long long cutime; + unsigned long long cstime; + unsigned long long cgtime; + unsigned long long num_threads; + unsigned long long rss; + + unsigned long long statm_size; + unsigned long long statm_resident; + unsigned long long statm_share; + unsigned long long statm_text; + unsigned long long statm_lib; + unsigned long long statm_data; + unsigned long long statm_dirty; + + unsigned long long io_logical_bytes_read; + unsigned long long io_logical_bytes_written; + unsigned long long io_read_calls; + unsigned long long io_write_calls; + unsigned long long io_storage_bytes_read; + unsigned long long io_storage_bytes_written; + unsigned long long io_cancelled_write_bytes; + + int *fds; + unsigned long long openfiles; + unsigned long long openpipes; + unsigned long long opensockets; + unsigned long long openinotifies; + unsigned long long openeventfds; + unsigned long long opentimerfds; + unsigned long long opensignalfds; + unsigned long long openeventpolls; + unsigned long long openother; + + unsigned long processes; // how many processes have been merged to this + int exposed; // if set, we have sent this to netdata + int hidden; // if set, we set the hidden flag on the dimension + int debug; + int ends_with; + int starts_with; // if set, the compare string matches only the + // beginning of the command + + struct target *target; // the one that will be reported to netdata + struct target *next; }; @@ -367,227 +176,212 @@ struct target *groups_root_target = NULL; struct target *get_users_target(uid_t uid) { - struct target *w; - for(w = users_root_target ; w ; w = w->next) - if(w->uid == uid) return w; + struct target *w; + for(w = users_root_target ; w ; w = w->next) + if(w->uid == uid) return w; - w = calloc(sizeof(struct target), 1); - if(unlikely(!w)) { - error("Cannot allocate %lu bytes of memory", (unsigned long)sizeof(struct target)); - return NULL; - } + w = callocz(sizeof(struct target), 1); + snprintfz(w->compare, MAX_COMPARE_NAME, "%u", uid); + w->comparehash = simple_hash(w->compare); + w->comparelen = strlen(w->compare); - snprintfz(w->compare, MAX_COMPARE_NAME, "%d", uid); - w->comparehash = simple_hash(w->compare); - w->comparelen = strlen(w->compare); + snprintfz(w->id, MAX_NAME, "%u", uid); + w->idhash = simple_hash(w->id); - snprintfz(w->id, MAX_NAME, "%d", uid); - w->idhash = simple_hash(w->id); + struct passwd *pw = getpwuid(uid); + if(!pw) + snprintfz(w->name, MAX_NAME, "%u", uid); + else + snprintfz(w->name, MAX_NAME, "%s", pw->pw_name); - struct passwd *pw = getpwuid(uid); - if(!pw) - snprintfz(w->name, MAX_NAME, "%d", uid); - else - snprintfz(w->name, MAX_NAME, "%s", pw->pw_name); + netdata_fix_chart_name(w->name); - netdata_fix_chart_name(w->name); + w->uid = uid; - w->uid = uid; + w->next = users_root_target; + users_root_target = w; - w->next = users_root_target; - users_root_target = w; + if(unlikely(debug)) + fprintf(stderr, "apps.plugin: added uid %u ('%s') target\n", w->uid, w->name); - if(unlikely(debug)) - fprintf(stderr, "apps.plugin: added uid %d ('%s') target\n", w->uid, w->name); - - return w; + return w; } struct target *get_groups_target(gid_t gid) { - struct target *w; - for(w = groups_root_target ; w ; w = w->next) - if(w->gid == gid) return w; - - w = calloc(sizeof(struct target), 1); - if(unlikely(!w)) { - error("Cannot allocate %lu bytes of memory", (unsigned long)sizeof(struct target)); - return NULL; - } + struct target *w; + for(w = groups_root_target ; w ; w = w->next) + if(w->gid == gid) return w; - snprintfz(w->compare, MAX_COMPARE_NAME, "%d", gid); - w->comparehash = simple_hash(w->compare); - w->comparelen = strlen(w->compare); + w = callocz(sizeof(struct target), 1); + snprintfz(w->compare, MAX_COMPARE_NAME, "%u", gid); + w->comparehash = simple_hash(w->compare); + w->comparelen = strlen(w->compare); - snprintfz(w->id, MAX_NAME, "%d", gid); - w->idhash = simple_hash(w->id); + snprintfz(w->id, MAX_NAME, "%u", gid); + w->idhash = simple_hash(w->id); - struct group *gr = getgrgid(gid); - if(!gr) - snprintfz(w->name, MAX_NAME, "%d", gid); - else - snprintfz(w->name, MAX_NAME, "%s", gr->gr_name); + struct group *gr = getgrgid(gid); + if(!gr) + snprintfz(w->name, MAX_NAME, "%u", gid); + else + snprintfz(w->name, MAX_NAME, "%s", gr->gr_name); - netdata_fix_chart_name(w->name); + netdata_fix_chart_name(w->name); - w->gid = gid; + w->gid = gid; - w->next = groups_root_target; - groups_root_target = w; + w->next = groups_root_target; + groups_root_target = w; - if(unlikely(debug)) - fprintf(stderr, "apps.plugin: added gid %d ('%s') target\n", w->gid, w->name); + if(unlikely(debug)) + fprintf(stderr, "apps.plugin: added gid %u ('%s') target\n", w->gid, w->name); - return w; + return w; } // find or create a new target // there are targets that are just aggregated to other target (the second argument) -struct target *get_apps_groups_target(const char *id, struct target *target) -{ - int tdebug = 0, thidden = 0, ends_with = 0; - const char *nid = id; - - while(nid[0] == '-' || nid[0] == '+' || nid[0] == '*') { - if(nid[0] == '-') thidden = 1; - if(nid[0] == '+') tdebug = 1; - if(nid[0] == '*') ends_with = 1; - nid++; - } - uint32_t hash = simple_hash(id); - - struct target *w; - for(w = apps_groups_root_target ; w ; w = w->next) { - if(w->idhash == hash && strncmp(nid, w->id, MAX_NAME) == 0) - return w; - } - - w = calloc(sizeof(struct target), 1); - if(unlikely(!w)) { - error("Cannot allocate %lu bytes of memory", (unsigned long)sizeof(struct target)); - return NULL; - } - - strncpyz(w->id, nid, MAX_NAME); - w->idhash = simple_hash(w->id); - - strncpyz(w->name, nid, MAX_NAME); - - strncpyz(w->compare, nid, MAX_COMPARE_NAME); - int len = strlen(w->compare); - if(w->compare[len - 1] == '*') { - w->compare[len - 1] = '\0'; - w->starts_with = 1; - } - w->ends_with = ends_with; - - if(w->starts_with && w->ends_with) - proc_pid_cmdline_is_needed = 1; - - w->comparehash = simple_hash(w->compare); - w->comparelen = strlen(w->compare); - - w->hidden = thidden; - w->debug = tdebug; - w->target = target; - - w->next = apps_groups_root_target; - apps_groups_root_target = w; - - if(unlikely(debug)) - fprintf(stderr, "apps.plugin: ADDING TARGET ID '%s', process name '%s' (%s), aggregated on target '%s', options: %s %s\n" - , w->id - , w->compare, (w->starts_with && w->ends_with)?"substring":((w->starts_with)?"prefix":((w->ends_with)?"suffix":"exact")) - , w->target?w->target->id:w->id - , (w->hidden)?"hidden":"-" - , (w->debug)?"debug":"-" - ); - - return w; +struct target *get_apps_groups_target(const char *id, struct target *target) { + int tdebug = 0, thidden = 0, ends_with = 0; + const char *nid = id; + + while(nid[0] == '-' || nid[0] == '+' || nid[0] == '*') { + if(nid[0] == '-') thidden = 1; + if(nid[0] == '+') tdebug = 1; + if(nid[0] == '*') ends_with = 1; + nid++; + } + uint32_t hash = simple_hash(id); + + struct target *w, *last = apps_groups_root_target; + for(w = apps_groups_root_target ; w ; w = w->next) { + if(w->idhash == hash && strncmp(nid, w->id, MAX_NAME) == 0) + return w; + + last = w; + } + + w = callocz(sizeof(struct target), 1); + strncpyz(w->id, nid, MAX_NAME); + w->idhash = simple_hash(w->id); + + strncpyz(w->name, nid, MAX_NAME); + + strncpyz(w->compare, nid, MAX_COMPARE_NAME); + size_t len = strlen(w->compare); + if(w->compare[len - 1] == '*') { + w->compare[len - 1] = '\0'; + w->starts_with = 1; + } + w->ends_with = ends_with; + + if(w->starts_with && w->ends_with) + proc_pid_cmdline_is_needed = 1; + + w->comparehash = simple_hash(w->compare); + w->comparelen = strlen(w->compare); + + w->hidden = thidden; + w->debug = tdebug; + w->target = target; + + // append it, to maintain the order in apps_groups.conf + if(last) last->next = w; + else apps_groups_root_target = w; + + if(unlikely(debug)) + fprintf(stderr, "apps.plugin: ADDING TARGET ID '%s', process name '%s' (%s), aggregated on target '%s', options: %s %s\n" + , w->id + , w->compare, (w->starts_with && w->ends_with)?"substring":((w->starts_with)?"prefix":((w->ends_with)?"suffix":"exact")) + , w->target?w->target->id:w->id + , (w->hidden)?"hidden":"-" + , (w->debug)?"debug":"-" + ); + + return w; } // read the apps_groups.conf file int read_apps_groups_conf(const char *name) { - char filename[FILENAME_MAX + 1]; + char filename[FILENAME_MAX + 1]; - snprintfz(filename, FILENAME_MAX, "%s/apps_%s.conf", config_dir, name); + snprintfz(filename, FILENAME_MAX, "%s/apps_%s.conf", config_dir, name); - if(unlikely(debug)) - fprintf(stderr, "apps.plugin: process groups file: '%s'\n", filename); + if(unlikely(debug)) + fprintf(stderr, "apps.plugin: process groups file: '%s'\n", filename); - // ---------------------------------------- + // ---------------------------------------- - procfile *ff = procfile_open(filename, " :\t", PROCFILE_FLAG_DEFAULT); - if(!ff) return 1; + procfile *ff = procfile_open(filename, " :\t", PROCFILE_FLAG_DEFAULT); + if(!ff) return 1; - procfile_set_quotes(ff, "'\""); + procfile_set_quotes(ff, "'\""); - ff = procfile_readall(ff); - if(!ff) { - procfile_close(ff); - return 1; - } + ff = procfile_readall(ff); + if(!ff) + return 1; - unsigned long line, lines = procfile_lines(ff); + unsigned long line, lines = procfile_lines(ff); - for(line = 0; line < lines ;line++) { - unsigned long word, words = procfile_linewords(ff, line); - struct target *w = NULL; + for(line = 0; line < lines ;line++) { + unsigned long word, words = procfile_linewords(ff, line); + struct target *w = NULL; - char *t = procfile_lineword(ff, line, 0); - if(!t || !*t) continue; + char *t = procfile_lineword(ff, line, 0); + if(!t || !*t) continue; - for(word = 0; word < words ;word++) { - char *s = procfile_lineword(ff, line, word); - if(!s || !*s) continue; - if(*s == '#') break; + for(word = 0; word < words ;word++) { + char *s = procfile_lineword(ff, line, word); + if(!s || !*s) continue; + if(*s == '#') break; - if(t == s) continue; + if(t == s) continue; - struct target *n = get_apps_groups_target(s, w); - if(!n) { - error("Cannot create target '%s' (line %d, word %d)", s, line, word); - continue; - } + struct target *n = get_apps_groups_target(s, w); + if(!n) { + error("Cannot create target '%s' (line %lu, word %lu)", s, line, word); + continue; + } - if(!w) w = n; - } + if(!w) w = n; + } - if(w) { - int tdebug = 0, thidden = 0; + if(w) { + int tdebug = 0, thidden = 0; - while(t[0] == '-' || t[0] == '+') { - if(t[0] == '-') thidden = 1; - if(t[0] == '+') tdebug = 1; - t++; - } + while(t[0] == '-' || t[0] == '+') { + if(t[0] == '-') thidden = 1; + if(t[0] == '+') tdebug = 1; + t++; + } - strncpyz(w->name, t, MAX_NAME); - w->hidden = thidden; - w->debug = tdebug; + strncpyz(w->name, t, MAX_NAME); + w->hidden = thidden; + w->debug = tdebug; - if(unlikely(debug)) - fprintf(stderr, "apps.plugin: AGGREGATION TARGET NAME '%s' on ID '%s', process name '%s' (%s), aggregated on target '%s', options: %s %s\n" - , w->name - , w->id - , w->compare, (w->starts_with && w->ends_with)?"substring":((w->starts_with)?"prefix":((w->ends_with)?"suffix":"exact")) - , w->target?w->target->id:w->id - , (w->hidden)?"hidden":"-" - , (w->debug)?"debug":"-" - ); - } - } + if(unlikely(debug)) + fprintf(stderr, "apps.plugin: AGGREGATION TARGET NAME '%s' on ID '%s', process name '%s' (%s), aggregated on target '%s', options: %s %s\n" + , w->name + , w->id + , w->compare, (w->starts_with && w->ends_with)?"substring":((w->starts_with)?"prefix":((w->ends_with)?"suffix":"exact")) + , w->target?w->target->id:w->id + , (w->hidden)?"hidden":"-" + , (w->debug)?"debug":"-" + ); + } + } - procfile_close(ff); + procfile_close(ff); - apps_groups_default_target = get_apps_groups_target("p+!o@w#e$i^r&7*5(-i)l-o_", NULL); // match nothing - if(!apps_groups_default_target) - error("Cannot create default target"); - else - strncpyz(apps_groups_default_target->name, "other", MAX_NAME); + apps_groups_default_target = get_apps_groups_target("p+!o@w#e$i^r&7*5(-i)l-o_", NULL); // match nothing + if(!apps_groups_default_target) + error("Cannot create default target"); + else + strncpyz(apps_groups_default_target->name, "other", MAX_NAME); - return 0; + return 0; } @@ -595,178 +389,183 @@ int read_apps_groups_conf(const char *name) // data to store for each pid // see: man proc -struct pid_stat { - int32_t pid; - char comm[MAX_COMPARE_NAME + 1]; - char cmdline[MAX_CMDLINE + 1]; - - // char state; - int32_t ppid; - // int32_t pgrp; - // int32_t session; - // int32_t tty_nr; - // int32_t tpgid; - // uint64_t flags; - unsigned long long minflt; - unsigned long long cminflt; - unsigned long long majflt; - unsigned long long cmajflt; - unsigned long long utime; - unsigned long long stime; - unsigned long long cutime; - unsigned long long cstime; - // int64_t priority; - // int64_t nice; - int32_t num_threads; - // int64_t itrealvalue; - // unsigned long long starttime; - // unsigned long long vsize; - unsigned long long rss; - // unsigned long long rsslim; - // unsigned long long starcode; - // unsigned long long endcode; - // unsigned long long startstack; - // unsigned long long kstkesp; - // unsigned long long kstkeip; - // uint64_t signal; - // uint64_t blocked; - // uint64_t sigignore; - // uint64_t sigcatch; - // uint64_t wchan; - // uint64_t nswap; - // uint64_t cnswap; - // int32_t exit_signal; - // int32_t processor; - // uint32_t rt_priority; - // uint32_t policy; - // unsigned long long delayacct_blkio_ticks; - // uint64_t guest_time; - // int64_t cguest_time; - - uid_t uid; - gid_t gid; - - unsigned long long statm_size; - unsigned long long statm_resident; - unsigned long long statm_share; - unsigned long long statm_text; - unsigned long long statm_lib; - unsigned long long statm_data; - unsigned long long statm_dirty; - - unsigned long long io_logical_bytes_read; - unsigned long long io_logical_bytes_written; - unsigned long long io_read_calls; - unsigned long long io_write_calls; - unsigned long long io_storage_bytes_read; - unsigned long long io_storage_bytes_written; - unsigned long long io_cancelled_write_bytes; - - // we need the last values - // for all incremental counters - // so that when a process switches users/groups - // we will subtract these values from the old - // target - unsigned long long last_minflt; - unsigned long long last_cminflt; - unsigned long long last_majflt; - unsigned long long last_cmajflt; - unsigned long long last_utime; - unsigned long long last_stime; - unsigned long long last_cutime; - unsigned long long last_cstime; - - unsigned long long last_io_logical_bytes_read; - unsigned long long last_io_logical_bytes_written; - unsigned long long last_io_read_calls; - unsigned long long last_io_write_calls; - unsigned long long last_io_storage_bytes_read; - unsigned long long last_io_storage_bytes_written; - unsigned long long last_io_cancelled_write_bytes; - -#ifdef AGGREGATE_CHILDREN_TO_PARENTS - unsigned long long old_utime; - unsigned long long old_stime; - unsigned long long old_minflt; - unsigned long long old_majflt; - - unsigned long long old_cutime; - unsigned long long old_cstime; - unsigned long long old_cminflt; - unsigned long long old_cmajflt; - - unsigned long long fix_cutime; - unsigned long long fix_cstime; - unsigned long long fix_cminflt; - unsigned long long fix_cmajflt; - - unsigned long long diff_cutime; - unsigned long long diff_cstime; - unsigned long long diff_cminflt; - unsigned long long diff_cmajflt; -#endif /* AGGREGATE_CHILDREN_TO_PARENTS */ - - int *fds; // array of fds it uses - int fds_size; // the size of the fds array - - int children_count; // number of processes directly referencing this - int updated; // 1 when update - int merged; // 1 when it has been merged to its parent - int new_entry; - - struct target *target; // app_groups.conf targets - struct target *user_target; // uid based targets - struct target *group_target; // gid based targets - - struct pid_stat *parent; - struct pid_stat *prev; - struct pid_stat *next; +#define PID_LOG_IO 0x00000001 +#define PID_LOG_STATM 0x00000002 +#define PID_LOG_CMDLINE 0x00000004 +#define PID_LOG_FDS 0x00000008 +#define PID_LOG_STAT 0x00000010 +struct pid_stat { + int32_t pid; + char comm[MAX_COMPARE_NAME + 1]; + char cmdline[MAX_CMDLINE + 1]; + + uint32_t log_thrown; + + // char state; + int32_t ppid; + // int32_t pgrp; + // int32_t session; + // int32_t tty_nr; + // int32_t tpgid; + // uint64_t flags; + + // these are raw values collected + unsigned long long minflt_raw; + unsigned long long cminflt_raw; + unsigned long long majflt_raw; + unsigned long long cmajflt_raw; + unsigned long long utime_raw; + unsigned long long stime_raw; + unsigned long long gtime_raw; // guest_time + unsigned long long cutime_raw; + unsigned long long cstime_raw; + unsigned long long cgtime_raw; // cguest_time + + // these are rates + unsigned long long minflt; + unsigned long long cminflt; + unsigned long long majflt; + unsigned long long cmajflt; + unsigned long long utime; + unsigned long long stime; + unsigned long long gtime; + unsigned long long cutime; + unsigned long long cstime; + unsigned long long cgtime; + + // int64_t priority; + // int64_t nice; + int32_t num_threads; + // int64_t itrealvalue; + // unsigned long long starttime; + // unsigned long long vsize; + unsigned long long rss; + // unsigned long long rsslim; + // unsigned long long starcode; + // unsigned long long endcode; + // unsigned long long startstack; + // unsigned long long kstkesp; + // unsigned long long kstkeip; + // uint64_t signal; + // uint64_t blocked; + // uint64_t sigignore; + // uint64_t sigcatch; + // uint64_t wchan; + // uint64_t nswap; + // uint64_t cnswap; + // int32_t exit_signal; + // int32_t processor; + // uint32_t rt_priority; + // uint32_t policy; + // unsigned long long delayacct_blkio_ticks; + + uid_t uid; + gid_t gid; + + unsigned long long statm_size; + unsigned long long statm_resident; + unsigned long long statm_share; + unsigned long long statm_text; + unsigned long long statm_lib; + unsigned long long statm_data; + unsigned long long statm_dirty; + + unsigned long long io_logical_bytes_read_raw; + unsigned long long io_logical_bytes_written_raw; + unsigned long long io_read_calls_raw; + unsigned long long io_write_calls_raw; + unsigned long long io_storage_bytes_read_raw; + unsigned long long io_storage_bytes_written_raw; + unsigned long long io_cancelled_write_bytes_raw; + + unsigned long long io_logical_bytes_read; + unsigned long long io_logical_bytes_written; + unsigned long long io_read_calls; + unsigned long long io_write_calls; + unsigned long long io_storage_bytes_read; + unsigned long long io_storage_bytes_written; + unsigned long long io_cancelled_write_bytes; + + int *fds; // array of fds it uses + int fds_size; // the size of the fds array + + int children_count; // number of processes directly referencing this + int keep; // 1 when we need to keep this process in memory even after it exited + int keeploops; // increases by 1 every time keep is 1 and updated 0 + int updated; // 1 when the process is currently running + int merged; // 1 when it has been merged to its parent + int new_entry; // 1 when this is a new process, just saw for the first time + int read; // 1 when we have already read this process for this iteration + int sortlist; // higher numbers = top on the process tree + // each process gets a unique number + + struct target *target; // app_groups.conf targets + struct target *user_target; // uid based targets + struct target *group_target; // gid based targets + + unsigned long long stat_collected_usec; + unsigned long long last_stat_collected_usec; + + unsigned long long io_collected_usec; + unsigned long long last_io_collected_usec; + + char *stat_filename; + char *statm_filename; + char *io_filename; + char *cmdline_filename; + + struct pid_stat *parent; + struct pid_stat *prev; + struct pid_stat *next; } *root_of_pids = NULL, **all_pids; long all_pids_count = 0; -struct pid_stat *get_pid_entry(pid_t pid) -{ - if(all_pids[pid]) { - all_pids[pid]->new_entry = 0; - return all_pids[pid]; - } - - all_pids[pid] = calloc(sizeof(struct pid_stat), 1); - if(!all_pids[pid]) { - error("Cannot allocate %lu bytes of memory", (unsigned long)sizeof(struct pid_stat)); - return NULL; - } - - all_pids[pid]->fds = calloc(sizeof(int), 100); - if(!all_pids[pid]->fds) - error("Cannot allocate %ld bytes of memory", (unsigned long)(sizeof(int) * 100)); - else all_pids[pid]->fds_size = 100; - - if(root_of_pids) root_of_pids->prev = all_pids[pid]; - all_pids[pid]->next = root_of_pids; - root_of_pids = all_pids[pid]; - - all_pids[pid]->pid = pid; - all_pids[pid]->new_entry = 1; - - return all_pids[pid]; +struct pid_stat *get_pid_entry(pid_t pid) { + if(all_pids[pid]) { + all_pids[pid]->new_entry = 0; + return all_pids[pid]; + } + + all_pids[pid] = callocz(sizeof(struct pid_stat), 1); + all_pids[pid]->fds = callocz(sizeof(int), 100); + all_pids[pid]->fds_size = 100; + + if(root_of_pids) root_of_pids->prev = all_pids[pid]; + all_pids[pid]->next = root_of_pids; + root_of_pids = all_pids[pid]; + + all_pids[pid]->pid = pid; + all_pids[pid]->new_entry = 1; + + all_pids_count++; + + return all_pids[pid]; } -void del_pid_entry(pid_t pid) -{ - if(!all_pids[pid]) return; +void del_pid_entry(pid_t pid) { + if(!all_pids[pid]) { + error("attempted to free pid %d that is not allocated.", pid); + return; + } + + if(unlikely(debug)) + fprintf(stderr, "apps.plugin: process %d %s exited, deleting it.\n", pid, all_pids[pid]->comm); - if(debug) fprintf(stderr, "apps.plugin: process %d %s exited, deleting it.\n", pid, all_pids[pid]->comm); + if(root_of_pids == all_pids[pid]) root_of_pids = all_pids[pid]->next; + if(all_pids[pid]->next) all_pids[pid]->next->prev = all_pids[pid]->prev; + if(all_pids[pid]->prev) all_pids[pid]->prev->next = all_pids[pid]->next; - if(root_of_pids == all_pids[pid]) root_of_pids = all_pids[pid]->next; - if(all_pids[pid]->next) all_pids[pid]->next->prev = all_pids[pid]->prev; - if(all_pids[pid]->prev) all_pids[pid]->prev->next = all_pids[pid]->next; + if(all_pids[pid]->fds) freez(all_pids[pid]->fds); + if(all_pids[pid]->stat_filename) freez(all_pids[pid]->stat_filename); + if(all_pids[pid]->statm_filename) freez(all_pids[pid]->statm_filename); + if(all_pids[pid]->io_filename) freez(all_pids[pid]->io_filename); + if(all_pids[pid]->cmdline_filename) freez(all_pids[pid]->cmdline_filename); + freez(all_pids[pid]); - if(all_pids[pid]->fds) free(all_pids[pid]->fds); - free(all_pids[pid]); - all_pids[pid] = NULL; + all_pids[pid] = NULL; + all_pids_count--; } @@ -774,188 +573,380 @@ void del_pid_entry(pid_t pid) // update pids from proc int read_proc_pid_cmdline(struct pid_stat *p) { - char filename[FILENAME_MAX + 1]; - snprintfz(filename, FILENAME_MAX, "%s/proc/%d/cmdline", host_prefix, p->pid); + + if(unlikely(!p->cmdline_filename)) { + char filename[FILENAME_MAX + 1]; + snprintfz(filename, FILENAME_MAX, "%s/proc/%d/cmdline", host_prefix, p->pid); + p->cmdline_filename = strdupz(filename); + } - int fd = open(filename, O_RDONLY, 0666); - if(unlikely(fd == -1)) return 1; + int fd = open(p->cmdline_filename, O_RDONLY, 0666); + if(unlikely(fd == -1)) goto cleanup; - int i, bytes = read(fd, p->cmdline, MAX_CMDLINE); - close(fd); + ssize_t i, bytes = read(fd, p->cmdline, MAX_CMDLINE); + close(fd); - if(bytes <= 0) { - // copy the command to the command line - strncpyz(p->cmdline, p->comm, MAX_CMDLINE); - return 0; - } + if(unlikely(bytes < 0)) goto cleanup; - p->cmdline[bytes] = '\0'; - for(i = 0; i < bytes ; i++) - if(!p->cmdline[i]) p->cmdline[i] = ' '; + p->cmdline[bytes] = '\0'; + for(i = 0; i < bytes ; i++) + if(unlikely(!p->cmdline[i])) p->cmdline[i] = ' '; - if(unlikely(debug)) - fprintf(stderr, "Read file '%s' contents: %s\n", filename, p->cmdline); + if(unlikely(debug)) + fprintf(stderr, "Read file '%s' contents: %s\n", p->cmdline_filename, p->cmdline); - return 0; + return 1; + +cleanup: + // copy the command to the command line + strncpyz(p->cmdline, p->comm, MAX_CMDLINE); + return 0; } int read_proc_pid_ownership(struct pid_stat *p) { - char filename[FILENAME_MAX + 1]; - - snprintfz(filename, FILENAME_MAX, "%s/proc/%d", host_prefix, p->pid); + if(unlikely(!p->stat_filename)) { + error("pid %d does not have a stat_filename", p->pid); + return 0; + } - // ---------------------------------------- - // read uid and gid + // ---------------------------------------- + // read uid and gid - struct stat st; - if(stat(filename, &st) != 0) - return 1; + struct stat st; + if(stat(p->stat_filename, &st) != 0) { + error("Cannot stat file '%s'", p->stat_filename); + return 1; + } - p->uid = st.st_uid; - p->gid = st.st_gid; + p->uid = st.st_uid; + p->gid = st.st_gid; - return 0; + return 1; } int read_proc_pid_stat(struct pid_stat *p) { - static procfile *ff = NULL; - - char filename[FILENAME_MAX + 1]; - - snprintfz(filename, FILENAME_MAX, "%s/proc/%d/stat", host_prefix, p->pid); - - // ---------------------------------------- - - int set_quotes = (!ff)?1:0; - - ff = procfile_reopen(ff, filename, NULL, PROCFILE_FLAG_NO_ERROR_ON_FILE_IO); - if(!ff) return 1; - - // if(set_quotes) procfile_set_quotes(ff, "()"); - if(set_quotes) procfile_set_open_close(ff, "(", ")"); - - ff = procfile_readall(ff); - if(!ff) { - // procfile_close(ff); - return 1; - } - - file_counter++; - - // parse the process name - unsigned int i = 0; - strncpyz(p->comm, procfile_lineword(ff, 0, 1), MAX_COMPARE_NAME); - - // p->pid = atol(procfile_lineword(ff, 0, 0+i)); - // comm is at 1 - // p->state = *(procfile_lineword(ff, 0, 2+i)); - p->ppid = (int32_t) atol(procfile_lineword(ff, 0, 3 + i)); - // p->pgrp = atol(procfile_lineword(ff, 0, 4+i)); - // p->session = atol(procfile_lineword(ff, 0, 5+i)); - // p->tty_nr = atol(procfile_lineword(ff, 0, 6+i)); - // p->tpgid = atol(procfile_lineword(ff, 0, 7+i)); - // p->flags = strtoull(procfile_lineword(ff, 0, 8+i), NULL, 10); - p->minflt = strtoull(procfile_lineword(ff, 0, 9+i), NULL, 10); - p->cminflt = strtoull(procfile_lineword(ff, 0, 10+i), NULL, 10); - p->majflt = strtoull(procfile_lineword(ff, 0, 11+i), NULL, 10); - p->cmajflt = strtoull(procfile_lineword(ff, 0, 12+i), NULL, 10); - p->utime = strtoull(procfile_lineword(ff, 0, 13+i), NULL, 10); - p->stime = strtoull(procfile_lineword(ff, 0, 14+i), NULL, 10); - p->cutime = strtoull(procfile_lineword(ff, 0, 15+i), NULL, 10); - p->cstime = strtoull(procfile_lineword(ff, 0, 16+i), NULL, 10); - // p->priority = strtoull(procfile_lineword(ff, 0, 17+i), NULL, 10); - // p->nice = strtoull(procfile_lineword(ff, 0, 18+i), NULL, 10); - p->num_threads = (int32_t) atol(procfile_lineword(ff, 0, 19 + i)); - // p->itrealvalue = strtoull(procfile_lineword(ff, 0, 20+i), NULL, 10); - // p->starttime = strtoull(procfile_lineword(ff, 0, 21+i), NULL, 10); - // p->vsize = strtoull(procfile_lineword(ff, 0, 22+i), NULL, 10); - p->rss = strtoull(procfile_lineword(ff, 0, 23+i), NULL, 10); - // p->rsslim = strtoull(procfile_lineword(ff, 0, 24+i), NULL, 10); - // p->starcode = strtoull(procfile_lineword(ff, 0, 25+i), NULL, 10); - // p->endcode = strtoull(procfile_lineword(ff, 0, 26+i), NULL, 10); - // p->startstack = strtoull(procfile_lineword(ff, 0, 27+i), NULL, 10); - // p->kstkesp = strtoull(procfile_lineword(ff, 0, 28+i), NULL, 10); - // p->kstkeip = strtoull(procfile_lineword(ff, 0, 29+i), NULL, 10); - // p->signal = strtoull(procfile_lineword(ff, 0, 30+i), NULL, 10); - // p->blocked = strtoull(procfile_lineword(ff, 0, 31+i), NULL, 10); - // p->sigignore = strtoull(procfile_lineword(ff, 0, 32+i), NULL, 10); - // p->sigcatch = strtoull(procfile_lineword(ff, 0, 33+i), NULL, 10); - // p->wchan = strtoull(procfile_lineword(ff, 0, 34+i), NULL, 10); - // p->nswap = strtoull(procfile_lineword(ff, 0, 35+i), NULL, 10); - // p->cnswap = strtoull(procfile_lineword(ff, 0, 36+i), NULL, 10); - // p->exit_signal = atol(procfile_lineword(ff, 0, 37+i)); - // p->processor = atol(procfile_lineword(ff, 0, 38+i)); - // p->rt_priority = strtoul(procfile_lineword(ff, 0, 39+i), NULL, 10); - // p->policy = strtoul(procfile_lineword(ff, 0, 40+i), NULL, 10); - // p->delayacct_blkio_ticks = strtoull(procfile_lineword(ff, 0, 41+i), NULL, 10); - // p->guest_time = strtoull(procfile_lineword(ff, 0, 42+i), NULL, 10); - // p->cguest_time = strtoull(procfile_lineword(ff, 0, 43), NULL, 10); - - if(debug || (p->target && p->target->debug)) - fprintf(stderr, "apps.plugin: READ PROC/PID/STAT: %s/proc/%d/stat, process: '%s' VALUES: utime=%llu, stime=%llu, cutime=%llu, cstime=%llu, minflt=%llu, majflt=%llu, cminflt=%llu, cmajflt=%llu, threads=%d\n", host_prefix, p->pid, p->comm, p->utime, p->stime, p->cutime, p->cstime, p->minflt, p->majflt, p->cminflt, p->cmajflt, p->num_threads); - - // procfile_close(ff); - return 0; + static procfile *ff = NULL; + + if(unlikely(!p->stat_filename)) { + char filename[FILENAME_MAX + 1]; + snprintfz(filename, FILENAME_MAX, "%s/proc/%d/stat", host_prefix, p->pid); + p->stat_filename = strdupz(filename); + } + + int set_quotes = (!ff)?1:0; + + ff = procfile_reopen(ff, p->stat_filename, NULL, PROCFILE_FLAG_NO_ERROR_ON_FILE_IO); + if(unlikely(!ff)) goto cleanup; + + // if(set_quotes) procfile_set_quotes(ff, "()"); + if(set_quotes) procfile_set_open_close(ff, "(", ")"); + + ff = procfile_readall(ff); + if(unlikely(!ff)) goto cleanup; + + p->last_stat_collected_usec = p->stat_collected_usec; + p->stat_collected_usec = time_usec(); + file_counter++; + + // p->pid = atol(procfile_lineword(ff, 0, 0+i)); + + strncpyz(p->comm, procfile_lineword(ff, 0, 1), MAX_COMPARE_NAME); + + // p->state = *(procfile_lineword(ff, 0, 2)); + p->ppid = (int32_t) atol(procfile_lineword(ff, 0, 3)); + // p->pgrp = atol(procfile_lineword(ff, 0, 4)); + // p->session = atol(procfile_lineword(ff, 0, 5)); + // p->tty_nr = atol(procfile_lineword(ff, 0, 6)); + // p->tpgid = atol(procfile_lineword(ff, 0, 7)); + // p->flags = strtoull(procfile_lineword(ff, 0, 8), NULL, 10); + + unsigned long long last; + + last = p->minflt_raw; + p->minflt_raw = strtoull(procfile_lineword(ff, 0, 9), NULL, 10); + p->minflt = (p->minflt_raw - last) * (1000000ULL * RATES_DETAIL) / (p->stat_collected_usec - p->last_stat_collected_usec); + + last = p->cminflt_raw; + p->cminflt_raw = strtoull(procfile_lineword(ff, 0, 10), NULL, 10); + p->cminflt = (p->cminflt_raw - last) * (1000000ULL * RATES_DETAIL) / (p->stat_collected_usec - p->last_stat_collected_usec); + + last = p->majflt_raw; + p->majflt_raw = strtoull(procfile_lineword(ff, 0, 11), NULL, 10); + p->majflt = (p->majflt_raw - last) * (1000000ULL * RATES_DETAIL) / (p->stat_collected_usec - p->last_stat_collected_usec); + + last = p->cmajflt_raw; + p->cmajflt_raw = strtoull(procfile_lineword(ff, 0, 12), NULL, 10); + p->cmajflt = (p->cmajflt_raw - last) * (1000000ULL * RATES_DETAIL) / (p->stat_collected_usec - p->last_stat_collected_usec); + + last = p->utime_raw; + p->utime_raw = strtoull(procfile_lineword(ff, 0, 13), NULL, 10); + p->utime = (p->utime_raw - last) * (1000000ULL * RATES_DETAIL) / (p->stat_collected_usec - p->last_stat_collected_usec); + + last = p->stime_raw; + p->stime_raw = strtoull(procfile_lineword(ff, 0, 14), NULL, 10); + p->stime = (p->stime_raw - last) * (1000000ULL * RATES_DETAIL) / (p->stat_collected_usec - p->last_stat_collected_usec); + + last = p->cutime_raw; + p->cutime_raw = strtoull(procfile_lineword(ff, 0, 15), NULL, 10); + p->cutime = (p->cutime_raw - last) * (1000000ULL * RATES_DETAIL) / (p->stat_collected_usec - p->last_stat_collected_usec); + + last = p->cstime_raw; + p->cstime_raw = strtoull(procfile_lineword(ff, 0, 16), NULL, 10); + p->cstime = (p->cstime_raw - last) * (1000000ULL * RATES_DETAIL) / (p->stat_collected_usec - p->last_stat_collected_usec); + + // p->priority = strtoull(procfile_lineword(ff, 0, 17), NULL, 10); + // p->nice = strtoull(procfile_lineword(ff, 0, 18), NULL, 10); + p->num_threads = (int32_t) atol(procfile_lineword(ff, 0, 19)); + // p->itrealvalue = strtoull(procfile_lineword(ff, 0, 20), NULL, 10); + // p->starttime = strtoull(procfile_lineword(ff, 0, 21), NULL, 10); + // p->vsize = strtoull(procfile_lineword(ff, 0, 22), NULL, 10); + p->rss = strtoull(procfile_lineword(ff, 0, 23), NULL, 10); + // p->rsslim = strtoull(procfile_lineword(ff, 0, 24), NULL, 10); + // p->starcode = strtoull(procfile_lineword(ff, 0, 25), NULL, 10); + // p->endcode = strtoull(procfile_lineword(ff, 0, 26), NULL, 10); + // p->startstack = strtoull(procfile_lineword(ff, 0, 27), NULL, 10); + // p->kstkesp = strtoull(procfile_lineword(ff, 0, 28), NULL, 10); + // p->kstkeip = strtoull(procfile_lineword(ff, 0, 29), NULL, 10); + // p->signal = strtoull(procfile_lineword(ff, 0, 30), NULL, 10); + // p->blocked = strtoull(procfile_lineword(ff, 0, 31), NULL, 10); + // p->sigignore = strtoull(procfile_lineword(ff, 0, 32), NULL, 10); + // p->sigcatch = strtoull(procfile_lineword(ff, 0, 33), NULL, 10); + // p->wchan = strtoull(procfile_lineword(ff, 0, 34), NULL, 10); + // p->nswap = strtoull(procfile_lineword(ff, 0, 35), NULL, 10); + // p->cnswap = strtoull(procfile_lineword(ff, 0, 36), NULL, 10); + // p->exit_signal = atol(procfile_lineword(ff, 0, 37)); + // p->processor = atol(procfile_lineword(ff, 0, 38)); + // p->rt_priority = strtoul(procfile_lineword(ff, 0, 39), NULL, 10); + // p->policy = strtoul(procfile_lineword(ff, 0, 40), NULL, 10); + // p->delayacct_blkio_ticks = strtoull(procfile_lineword(ff, 0, 41), NULL, 10); + + if(enable_guest_charts) { + last = p->gtime_raw; + p->gtime_raw = strtoull(procfile_lineword(ff, 0, 42), NULL, 10); + p->gtime = (p->gtime_raw - last) * (1000000ULL * RATES_DETAIL) / (p->stat_collected_usec - p->last_stat_collected_usec); + + last = p->cgtime_raw; + p->cgtime_raw = strtoull(procfile_lineword(ff, 0, 43), NULL, 10); + p->cgtime = (p->cgtime_raw - last) * (1000000ULL * RATES_DETAIL) / (p->stat_collected_usec - p->last_stat_collected_usec); + + if (show_guest_time || p->gtime || p->cgtime) { + p->utime -= (p->utime >= p->gtime) ? p->gtime : p->utime; + p->cutime -= (p->cutime >= p->cgtime) ? p->cgtime : p->cutime; + show_guest_time = 1; + } + } + + if(unlikely(debug || (p->target && p->target->debug))) + fprintf(stderr, "apps.plugin: READ PROC/PID/STAT: %s/proc/%d/stat, process: '%s' on target '%s' (dt=%llu) VALUES: utime=%llu, stime=%llu, cutime=%llu, cstime=%llu, minflt=%llu, majflt=%llu, cminflt=%llu, cmajflt=%llu, threads=%d\n", host_prefix, p->pid, p->comm, (p->target)?p->target->name:"UNSET", p->stat_collected_usec - p->last_stat_collected_usec, p->utime, p->stime, p->cutime, p->cstime, p->minflt, p->majflt, p->cminflt, p->cmajflt, p->num_threads); + + if(unlikely(global_iterations_counter == 1)) { + p->minflt = 0; + p->cminflt = 0; + p->majflt = 0; + p->cmajflt = 0; + p->utime = 0; + p->stime = 0; + p->gtime = 0; + p->cutime = 0; + p->cstime = 0; + p->cgtime = 0; + } + + return 1; + +cleanup: + p->minflt = 0; + p->cminflt = 0; + p->majflt = 0; + p->cmajflt = 0; + p->utime = 0; + p->stime = 0; + p->gtime = 0; + p->cutime = 0; + p->cstime = 0; + p->cgtime = 0; + p->num_threads = 0; + p->rss = 0; + return 0; } int read_proc_pid_statm(struct pid_stat *p) { - static procfile *ff = NULL; + static procfile *ff = NULL; + + if(unlikely(!p->statm_filename)) { + char filename[FILENAME_MAX + 1]; + snprintfz(filename, FILENAME_MAX, "%s/proc/%d/statm", host_prefix, p->pid); + p->statm_filename = strdupz(filename); + } + + ff = procfile_reopen(ff, p->statm_filename, NULL, PROCFILE_FLAG_NO_ERROR_ON_FILE_IO); + if(unlikely(!ff)) goto cleanup; + + ff = procfile_readall(ff); + if(unlikely(!ff)) goto cleanup; + + file_counter++; + + p->statm_size = strtoull(procfile_lineword(ff, 0, 0), NULL, 10); + p->statm_resident = strtoull(procfile_lineword(ff, 0, 1), NULL, 10); + p->statm_share = strtoull(procfile_lineword(ff, 0, 2), NULL, 10); + p->statm_text = strtoull(procfile_lineword(ff, 0, 3), NULL, 10); + p->statm_lib = strtoull(procfile_lineword(ff, 0, 4), NULL, 10); + p->statm_data = strtoull(procfile_lineword(ff, 0, 5), NULL, 10); + p->statm_dirty = strtoull(procfile_lineword(ff, 0, 6), NULL, 10); + + return 1; + +cleanup: + p->statm_size = 0; + p->statm_resident = 0; + p->statm_share = 0; + p->statm_text = 0; + p->statm_lib = 0; + p->statm_data = 0; + p->statm_dirty = 0; + return 0; +} - char filename[FILENAME_MAX + 1]; +int read_proc_pid_io(struct pid_stat *p) { + static procfile *ff = NULL; + + if(unlikely(!p->io_filename)) { + char filename[FILENAME_MAX + 1]; + snprintfz(filename, FILENAME_MAX, "%s/proc/%d/io", host_prefix, p->pid); + p->io_filename = strdupz(filename); + } + + // open the file + ff = procfile_reopen(ff, p->io_filename, NULL, PROCFILE_FLAG_NO_ERROR_ON_FILE_IO); + if(unlikely(!ff)) goto cleanup; + + ff = procfile_readall(ff); + if(unlikely(!ff)) goto cleanup; + + file_counter++; + + p->last_io_collected_usec = p->io_collected_usec; + p->io_collected_usec = time_usec(); + + unsigned long long last; + + last = p->io_logical_bytes_read_raw; + p->io_logical_bytes_read_raw = strtoull(procfile_lineword(ff, 0, 1), NULL, 10); + p->io_logical_bytes_read = (p->io_logical_bytes_read_raw - last) * (1000000ULL * RATES_DETAIL) / (p->io_collected_usec - p->last_io_collected_usec); + + last = p->io_logical_bytes_written_raw; + p->io_logical_bytes_written_raw = strtoull(procfile_lineword(ff, 1, 1), NULL, 10); + p->io_logical_bytes_written = (p->io_logical_bytes_written_raw - last) * (1000000ULL * RATES_DETAIL) / (p->io_collected_usec - p->last_io_collected_usec); + + last = p->io_read_calls_raw; + p->io_read_calls_raw = strtoull(procfile_lineword(ff, 2, 1), NULL, 10); + p->io_read_calls = (p->io_read_calls_raw - last) * (1000000ULL * RATES_DETAIL) / (p->io_collected_usec - p->last_io_collected_usec); + + last = p->io_write_calls_raw; + p->io_write_calls_raw = strtoull(procfile_lineword(ff, 3, 1), NULL, 10); + p->io_write_calls = (p->io_write_calls_raw - last) * (1000000ULL * RATES_DETAIL) / (p->io_collected_usec - p->last_io_collected_usec); + + last = p->io_storage_bytes_read_raw; + p->io_storage_bytes_read_raw = strtoull(procfile_lineword(ff, 4, 1), NULL, 10); + p->io_storage_bytes_read = (p->io_storage_bytes_read_raw - last) * (1000000ULL * RATES_DETAIL) / (p->io_collected_usec - p->last_io_collected_usec); + + last = p->io_storage_bytes_written_raw; + p->io_storage_bytes_written_raw = strtoull(procfile_lineword(ff, 5, 1), NULL, 10); + p->io_storage_bytes_written = (p->io_storage_bytes_written_raw - last) * (1000000ULL * RATES_DETAIL) / (p->io_collected_usec - p->last_io_collected_usec); + + last = p->io_cancelled_write_bytes_raw; + p->io_cancelled_write_bytes_raw = strtoull(procfile_lineword(ff, 6, 1), NULL, 10); + p->io_cancelled_write_bytes = (p->io_cancelled_write_bytes_raw - last) * (1000000ULL * RATES_DETAIL) / (p->io_collected_usec - p->last_io_collected_usec); + + if(unlikely(global_iterations_counter == 1)) { + p->io_logical_bytes_read = 0; + p->io_logical_bytes_written = 0; + p->io_read_calls = 0; + p->io_write_calls = 0; + p->io_storage_bytes_read = 0; + p->io_storage_bytes_written = 0; + p->io_cancelled_write_bytes = 0; + } + + return 1; + +cleanup: + p->io_logical_bytes_read = 0; + p->io_logical_bytes_written = 0; + p->io_read_calls = 0; + p->io_write_calls = 0; + p->io_storage_bytes_read = 0; + p->io_storage_bytes_written = 0; + p->io_cancelled_write_bytes = 0; + return 0; +} - snprintfz(filename, FILENAME_MAX, "%s/proc/%d/statm", host_prefix, p->pid); +unsigned long long global_utime = 0; +unsigned long long global_stime = 0; +unsigned long long global_gtime = 0; - ff = procfile_reopen(ff, filename, NULL, PROCFILE_FLAG_NO_ERROR_ON_FILE_IO); - if(!ff) return 1; +int read_proc_stat() { + static char filename[FILENAME_MAX + 1] = ""; + static procfile *ff = NULL; + static unsigned long long utime_raw = 0, stime_raw = 0, gtime_raw = 0, gntime_raw = 0, ntime_raw = 0, collected_usec = 0, last_collected_usec = 0; - ff = procfile_readall(ff); - if(!ff) { - // procfile_close(ff); - return 1; - } + if(unlikely(!ff)) { + snprintfz(filename, FILENAME_MAX, "%s/proc/stat", host_prefix); + ff = procfile_open(filename, " \t:", PROCFILE_FLAG_DEFAULT); + if(unlikely(!ff)) goto cleanup; + } - file_counter++; + ff = procfile_readall(ff); + if(unlikely(!ff)) goto cleanup; - p->statm_size = strtoull(procfile_lineword(ff, 0, 0), NULL, 10); - p->statm_resident = strtoull(procfile_lineword(ff, 0, 1), NULL, 10); - p->statm_share = strtoull(procfile_lineword(ff, 0, 2), NULL, 10); - p->statm_text = strtoull(procfile_lineword(ff, 0, 3), NULL, 10); - p->statm_lib = strtoull(procfile_lineword(ff, 0, 4), NULL, 10); - p->statm_data = strtoull(procfile_lineword(ff, 0, 5), NULL, 10); - p->statm_dirty = strtoull(procfile_lineword(ff, 0, 6), NULL, 10); + last_collected_usec = collected_usec; + collected_usec = time_usec(); - // procfile_close(ff); - return 0; -} + file_counter++; -int read_proc_pid_io(struct pid_stat *p) { - static procfile *ff = NULL; + unsigned long long last; + + last = utime_raw; + utime_raw = strtoull(procfile_lineword(ff, 0, 1), NULL, 10); + global_utime = (utime_raw - last) * (1000000ULL * RATES_DETAIL) / (collected_usec - last_collected_usec); + + // nice time, on user time + last = ntime_raw; + ntime_raw = strtoull(procfile_lineword(ff, 0, 2), NULL, 10); + global_utime += (ntime_raw - last) * (1000000ULL * RATES_DETAIL) / (collected_usec - last_collected_usec); - char filename[FILENAME_MAX + 1]; + last = stime_raw; + stime_raw = strtoull(procfile_lineword(ff, 0, 3), NULL, 10); + global_stime = (stime_raw - last) * (1000000ULL * RATES_DETAIL) / (collected_usec - last_collected_usec); - snprintfz(filename, FILENAME_MAX, "%s/proc/%d/io", host_prefix, p->pid); + last = gtime_raw; + gtime_raw = strtoull(procfile_lineword(ff, 0, 10), NULL, 10); + global_gtime = (gtime_raw - last) * (1000000ULL * RATES_DETAIL) / (collected_usec - last_collected_usec); - ff = procfile_reopen(ff, filename, NULL, PROCFILE_FLAG_NO_ERROR_ON_FILE_IO); - if(!ff) return 1; + if(enable_guest_charts) { + // guest nice time, on guest time + last = gntime_raw; + gntime_raw = strtoull(procfile_lineword(ff, 0, 11), NULL, 10); + global_gtime += (gntime_raw - last) * (1000000ULL * RATES_DETAIL) / (collected_usec - last_collected_usec); - ff = procfile_readall(ff); - if(!ff) { - // procfile_close(ff); - return 1; - } + // remove guest time from user time + global_utime -= (global_utime > global_gtime) ? global_gtime : global_utime; + } - file_counter++; + if(unlikely(global_iterations_counter == 1)) { + global_utime = 0; + global_stime = 0; + global_gtime = 0; + } - p->io_logical_bytes_read = strtoull(procfile_lineword(ff, 0, 1), NULL, 10); - p->io_logical_bytes_written = strtoull(procfile_lineword(ff, 1, 1), NULL, 10); - p->io_read_calls = strtoull(procfile_lineword(ff, 2, 1), NULL, 10); - p->io_write_calls = strtoull(procfile_lineword(ff, 3, 1), NULL, 10); - p->io_storage_bytes_read = strtoull(procfile_lineword(ff, 4, 1), NULL, 10); - p->io_storage_bytes_written = strtoull(procfile_lineword(ff, 5, 1), NULL, 10); - p->io_cancelled_write_bytes = strtoull(procfile_lineword(ff, 6, 1), NULL, 10); + return 1; - // procfile_close(ff); - return 0; +cleanup: + global_utime = 0; + global_stime = 0; + global_gtime = 0; + return 0; } @@ -967,15 +958,15 @@ int read_proc_pid_io(struct pid_stat *p) { #define FILE_DESCRIPTORS_INCREASE_STEP 100 struct file_descriptor { - avl avl; + avl avl; #ifdef NETDATA_INTERNAL_CHECKS - uint32_t magic; + uint32_t magic; #endif /* NETDATA_INTERNAL_CHECKS */ - uint32_t hash; - const char *name; - int type; - int count; - int pos; + uint32_t hash; + const char *name; + int type; + int count; + int pos; } *all_files = NULL; int all_files_len = 0; @@ -983,38 +974,38 @@ int all_files_size = 0; int file_descriptor_compare(void* a, void* b) { #ifdef NETDATA_INTERNAL_CHECKS - if(((struct file_descriptor *)a)->magic != 0x0BADCAFE || ((struct file_descriptor *)b)->magic != 0x0BADCAFE) - error("Corrupted index data detected. Please report this."); + if(((struct file_descriptor *)a)->magic != 0x0BADCAFE || ((struct file_descriptor *)b)->magic != 0x0BADCAFE) + error("Corrupted index data detected. Please report this."); #endif /* NETDATA_INTERNAL_CHECKS */ - if(((struct file_descriptor *)a)->hash < ((struct file_descriptor *)b)->hash) - return -1; + if(((struct file_descriptor *)a)->hash < ((struct file_descriptor *)b)->hash) + return -1; - else if(((struct file_descriptor *)a)->hash > ((struct file_descriptor *)b)->hash) - return 1; + else if(((struct file_descriptor *)a)->hash > ((struct file_descriptor *)b)->hash) + return 1; - else - return strcmp(((struct file_descriptor *)a)->name, ((struct file_descriptor *)b)->name); + else + return strcmp(((struct file_descriptor *)a)->name, ((struct file_descriptor *)b)->name); } int file_descriptor_iterator(avl *a) { if(a) {}; return 0; } avl_tree all_files_index = { - NULL, - file_descriptor_compare + NULL, + file_descriptor_compare }; static struct file_descriptor *file_descriptor_find(const char *name, uint32_t hash) { - struct file_descriptor tmp; - tmp.hash = (hash)?hash:simple_hash(name); - tmp.name = name; - tmp.count = 0; - tmp.pos = 0; + struct file_descriptor tmp; + tmp.hash = (hash)?hash:simple_hash(name); + tmp.name = name; + tmp.count = 0; + tmp.pos = 0; #ifdef NETDATA_INTERNAL_CHECKS - tmp.magic = 0x0BADCAFE; + tmp.magic = 0x0BADCAFE; #endif /* NETDATA_INTERNAL_CHECKS */ - return (struct file_descriptor *)avl_search(&all_files_index, (avl *) &tmp); + return (struct file_descriptor *)avl_search(&all_files_index, (avl *) &tmp); } #define file_descriptor_add(fd) avl_insert(&all_files_index, (avl *)(fd)) @@ -1032,223 +1023,526 @@ static struct file_descriptor *file_descriptor_find(const char *name, uint32_t h void file_descriptor_not_used(int id) { - if(id > 0 && id < all_files_size) { + if(id > 0 && id < all_files_size) { #ifdef NETDATA_INTERNAL_CHECKS - if(all_files[id].magic != 0x0BADCAFE) { - error("Ignoring request to remove empty file id %d.", id); - return; - } + if(all_files[id].magic != 0x0BADCAFE) { + error("Ignoring request to remove empty file id %d.", id); + return; + } #endif /* NETDATA_INTERNAL_CHECKS */ - if(debug) fprintf(stderr, "apps.plugin: decreasing slot %d (count = %d).\n", id, all_files[id].count); + if(unlikely(debug)) + fprintf(stderr, "apps.plugin: decreasing slot %d (count = %d).\n", id, all_files[id].count); - if(all_files[id].count > 0) { - all_files[id].count--; + if(all_files[id].count > 0) { + all_files[id].count--; - if(!all_files[id].count) { - if(debug) fprintf(stderr, "apps.plugin: >> slot %d is empty.\n", id); - file_descriptor_remove(&all_files[id]); + if(!all_files[id].count) { + if(unlikely(debug)) + fprintf(stderr, "apps.plugin: >> slot %d is empty.\n", id); + + file_descriptor_remove(&all_files[id]); #ifdef NETDATA_INTERNAL_CHECKS - all_files[id].magic = 0x00000000; + all_files[id].magic = 0x00000000; #endif /* NETDATA_INTERNAL_CHECKS */ - all_files_len--; - } - } - else - error("Request to decrease counter of fd %d (%s), while the use counter is 0", id, all_files[id].name); - } - else error("Request to decrease counter of fd %d, which is outside the array size (1 to %d)", id, all_files_size); + all_files_len--; + } + } + else + error("Request to decrease counter of fd %d (%s), while the use counter is 0", id, all_files[id].name); + } + else error("Request to decrease counter of fd %d, which is outside the array size (1 to %d)", id, all_files_size); } int file_descriptor_find_or_add(const char *name) { - static int last_pos = 0; - uint32_t hash = simple_hash(name); - - if(debug) fprintf(stderr, "apps.plugin: adding or finding name '%s' with hash %u\n", name, hash); - - struct file_descriptor *fd = file_descriptor_find(name, hash); - if(fd) { - // found - if(debug) fprintf(stderr, "apps.plugin: >> found on slot %d\n", fd->pos); - fd->count++; - return fd->pos; - } - // not found - - // check we have enough memory to add it - if(!all_files || all_files_len == all_files_size) { - void *old = all_files; - int i; - - // there is no empty slot - if(debug) fprintf(stderr, "apps.plugin: extending fd array to %d entries\n", all_files_size + FILE_DESCRIPTORS_INCREASE_STEP); - all_files = realloc(all_files, (all_files_size + FILE_DESCRIPTORS_INCREASE_STEP) * sizeof(struct file_descriptor)); - - // if the address changed, we have to rebuild the index - // since all pointers are now invalid - if(old && old != (void *)all_files) { - if(debug) fprintf(stderr, "apps.plugin: >> re-indexing.\n"); - all_files_index.root = NULL; - for(i = 0; i < all_files_size; i++) { - if(!all_files[i].count) continue; - file_descriptor_add(&all_files[i]); - } - if(debug) fprintf(stderr, "apps.plugin: >> re-indexing done.\n"); - } - - for(i = all_files_size; i < (all_files_size + FILE_DESCRIPTORS_INCREASE_STEP); i++) { - all_files[i].count = 0; - all_files[i].name = NULL; + static int last_pos = 0; + uint32_t hash = simple_hash(name); + + if(unlikely(debug)) + fprintf(stderr, "apps.plugin: adding or finding name '%s' with hash %u\n", name, hash); + + struct file_descriptor *fd = file_descriptor_find(name, hash); + if(fd) { + // found + if(unlikely(debug)) + fprintf(stderr, "apps.plugin: >> found on slot %d\n", fd->pos); + + fd->count++; + return fd->pos; + } + // not found + + // check we have enough memory to add it + if(!all_files || all_files_len == all_files_size) { + void *old = all_files; + int i; + + // there is no empty slot + if(unlikely(debug)) + fprintf(stderr, "apps.plugin: extending fd array to %d entries\n", all_files_size + FILE_DESCRIPTORS_INCREASE_STEP); + + all_files = reallocz(all_files, (all_files_size + FILE_DESCRIPTORS_INCREASE_STEP) * sizeof(struct file_descriptor)); + + // if the address changed, we have to rebuild the index + // since all pointers are now invalid + if(old && old != (void *)all_files) { + if(unlikely(debug)) + fprintf(stderr, "apps.plugin: >> re-indexing.\n"); + + all_files_index.root = NULL; + for(i = 0; i < all_files_size; i++) { + if(!all_files[i].count) continue; + file_descriptor_add(&all_files[i]); + } + + if(unlikely(debug)) + fprintf(stderr, "apps.plugin: >> re-indexing done.\n"); + } + + for(i = all_files_size; i < (all_files_size + FILE_DESCRIPTORS_INCREASE_STEP); i++) { + all_files[i].count = 0; + all_files[i].name = NULL; #ifdef NETDATA_INTERNAL_CHECKS - all_files[i].magic = 0x00000000; + all_files[i].magic = 0x00000000; #endif /* NETDATA_INTERNAL_CHECKS */ - all_files[i].pos = i; - } + all_files[i].pos = i; + } - if(!all_files_size) all_files_len = 1; - all_files_size += FILE_DESCRIPTORS_INCREASE_STEP; - } + if(!all_files_size) all_files_len = 1; + all_files_size += FILE_DESCRIPTORS_INCREASE_STEP; + } - if(debug) fprintf(stderr, "apps.plugin: >> searching for empty slot.\n"); + if(unlikely(debug)) + fprintf(stderr, "apps.plugin: >> searching for empty slot.\n"); - // search for an empty slot - int i, c; - for(i = 0, c = last_pos ; i < all_files_size ; i++, c++) { - if(c >= all_files_size) c = 0; - if(c == 0) continue; + // search for an empty slot + int i, c; + for(i = 0, c = last_pos ; i < all_files_size ; i++, c++) { + if(c >= all_files_size) c = 0; + if(c == 0) continue; - if(!all_files[c].count) { - if(debug) fprintf(stderr, "apps.plugin: >> Examining slot %d.\n", c); + if(!all_files[c].count) { + if(unlikely(debug)) + fprintf(stderr, "apps.plugin: >> Examining slot %d.\n", c); #ifdef NETDATA_INTERNAL_CHECKS - if(all_files[c].magic == 0x0BADCAFE && all_files[c].name && file_descriptor_find(all_files[c].name, all_files[c].hash)) - error("fd on position %d is not cleared properly. It still has %s in it.\n", c, all_files[c].name); + if(all_files[c].magic == 0x0BADCAFE && all_files[c].name && file_descriptor_find(all_files[c].name, all_files[c].hash)) + error("fd on position %d is not cleared properly. It still has %s in it.\n", c, all_files[c].name); #endif /* NETDATA_INTERNAL_CHECKS */ - if(debug) fprintf(stderr, "apps.plugin: >> %s fd position %d for %s (last name: %s)\n", all_files[c].name?"re-using":"using", c, name, all_files[c].name); - if(all_files[c].name) free((void *)all_files[c].name); - all_files[c].name = NULL; - last_pos = c; - break; - } - } - if(i == all_files_size) { - fatal("We should find an empty slot, but there isn't any"); - exit(1); - } - if(debug) fprintf(stderr, "apps.plugin: >> updating slot %d.\n", c); - - all_files_len++; - - // else we have an empty slot in 'c' - - int type; - if(name[0] == '/') type = FILETYPE_FILE; - else if(strncmp(name, "pipe:", 5) == 0) type = FILETYPE_PIPE; - else if(strncmp(name, "socket:", 7) == 0) type = FILETYPE_SOCKET; - else if(strcmp(name, "anon_inode:inotify") == 0 || strcmp(name, "inotify") == 0) type = FILETYPE_INOTIFY; - else if(strcmp(name, "anon_inode:[eventfd]") == 0) type = FILETYPE_EVENTFD; - else if(strcmp(name, "anon_inode:[eventpoll]") == 0) type = FILETYPE_EVENTPOLL; - else if(strcmp(name, "anon_inode:[timerfd]") == 0) type = FILETYPE_TIMERFD; - else if(strcmp(name, "anon_inode:[signalfd]") == 0) type = FILETYPE_SIGNALFD; - else if(strncmp(name, "anon_inode:", 11) == 0) { - if(debug) fprintf(stderr, "apps.plugin: FIXME: unknown anonymous inode: %s\n", name); - type = FILETYPE_OTHER; - } - else { - if(debug) fprintf(stderr, "apps.plugin: FIXME: cannot understand linkname: %s\n", name); - type = FILETYPE_OTHER; - } - - all_files[c].name = strdup(name); - all_files[c].hash = hash; - all_files[c].type = type; - all_files[c].pos = c; - all_files[c].count = 1; + if(unlikely(debug)) + fprintf(stderr, "apps.plugin: >> %s fd position %d for %s (last name: %s)\n", all_files[c].name?"re-using":"using", c, name, all_files[c].name); + + if(all_files[c].name) freez((void *)all_files[c].name); + all_files[c].name = NULL; + last_pos = c; + break; + } + } + if(i == all_files_size) { + fatal("We should find an empty slot, but there isn't any"); + exit(1); + } + + if(unlikely(debug)) + fprintf(stderr, "apps.plugin: >> updating slot %d.\n", c); + + all_files_len++; + + // else we have an empty slot in 'c' + + int type; + if(name[0] == '/') type = FILETYPE_FILE; + else if(strncmp(name, "pipe:", 5) == 0) type = FILETYPE_PIPE; + else if(strncmp(name, "socket:", 7) == 0) type = FILETYPE_SOCKET; + else if(strcmp(name, "anon_inode:inotify") == 0 || strcmp(name, "inotify") == 0) type = FILETYPE_INOTIFY; + else if(strcmp(name, "anon_inode:[eventfd]") == 0) type = FILETYPE_EVENTFD; + else if(strcmp(name, "anon_inode:[eventpoll]") == 0) type = FILETYPE_EVENTPOLL; + else if(strcmp(name, "anon_inode:[timerfd]") == 0) type = FILETYPE_TIMERFD; + else if(strcmp(name, "anon_inode:[signalfd]") == 0) type = FILETYPE_SIGNALFD; + else if(strncmp(name, "anon_inode:", 11) == 0) { + if(unlikely(debug)) + fprintf(stderr, "apps.plugin: FIXME: unknown anonymous inode: %s\n", name); + + type = FILETYPE_OTHER; + } + else { + if(unlikely(debug)) + fprintf(stderr, "apps.plugin: FIXME: cannot understand linkname: %s\n", name); + + type = FILETYPE_OTHER; + } + + all_files[c].name = strdupz(name); + all_files[c].hash = hash; + all_files[c].type = type; + all_files[c].pos = c; + all_files[c].count = 1; #ifdef NETDATA_INTERNAL_CHECKS - all_files[c].magic = 0x0BADCAFE; + all_files[c].magic = 0x0BADCAFE; #endif /* NETDATA_INTERNAL_CHECKS */ - file_descriptor_add(&all_files[c]); + file_descriptor_add(&all_files[c]); - if(debug) fprintf(stderr, "apps.plugin: using fd position %d (name: %s)\n", c, all_files[c].name); + if(unlikely(debug)) + fprintf(stderr, "apps.plugin: using fd position %d (name: %s)\n", c, all_files[c].name); - return c; + return c; } int read_pid_file_descriptors(struct pid_stat *p) { - char dirname[FILENAME_MAX+1]; - - snprintfz(dirname, FILENAME_MAX, "%s/proc/%d/fd", host_prefix, p->pid); - DIR *fds = opendir(dirname); - if(fds) { - int c; - struct dirent *de; - char fdname[FILENAME_MAX + 1]; - char linkname[FILENAME_MAX + 1]; - - // make the array negative - for(c = 0 ; c < p->fds_size ; c++) - p->fds[c] = -p->fds[c]; - - while((de = readdir(fds))) { - if(strcmp(de->d_name, ".") == 0 || strcmp(de->d_name, "..") == 0) - continue; - - // check if the fds array is small - int fdid = atoi(de->d_name); - if(fdid < 0) continue; - if(fdid >= p->fds_size) { - // it is small, extend it - if(debug) fprintf(stderr, "apps.plugin: extending fd memory slots for %s from %d to %d\n", p->comm, p->fds_size, fdid + 100); - p->fds = realloc(p->fds, (fdid + 100) * sizeof(int)); - if(!p->fds) { - fatal("Cannot re-allocate fds for %s", p->comm); - break; - } - - // and initialize it - for(c = p->fds_size ; c < (fdid + 100) ; c++) p->fds[c] = 0; - p->fds_size = fdid + 100; - } - - if(p->fds[fdid] == 0) { - // we don't know this fd, get it - - sprintf(fdname, "%s/proc/%d/fd/%s", host_prefix, p->pid, de->d_name); - ssize_t l = readlink(fdname, linkname, FILENAME_MAX); - if(l == -1) { - if(debug || (p->target && p->target->debug)) { - if(debug || (p->target && p->target->debug)) - error("Cannot read link %s", fdname); - } - continue; - } - linkname[l] = '\0'; - file_counter++; - - // if another process already has this, we will get - // the same id - p->fds[fdid] = file_descriptor_find_or_add(linkname); - } - - // else make it positive again, we need it - // of course, the actual file may have changed, but we don't care so much - // FIXME: we could compare the inode as returned by readdir direct structure - else p->fds[fdid] = -p->fds[fdid]; - } - closedir(fds); - - // remove all the negative file descriptors - for(c = 0 ; c < p->fds_size ; c++) if(p->fds[c] < 0) { - file_descriptor_not_used(-p->fds[c]); - p->fds[c] = 0; - } - } - else return 1; - - return 0; + char dirname[FILENAME_MAX+1]; + + snprintfz(dirname, FILENAME_MAX, "%s/proc/%d/fd", host_prefix, p->pid); + DIR *fds = opendir(dirname); + if(fds) { + int c; + struct dirent *de; + char fdname[FILENAME_MAX + 1]; + char linkname[FILENAME_MAX + 1]; + + // make the array negative + for(c = 0 ; c < p->fds_size ; c++) + p->fds[c] = -p->fds[c]; + + while((de = readdir(fds))) { + if(strcmp(de->d_name, ".") == 0 || strcmp(de->d_name, "..") == 0) + continue; + + // check if the fds array is small + int fdid = atoi(de->d_name); + if(fdid < 0) continue; + if(fdid >= p->fds_size) { + // it is small, extend it + if(unlikely(debug)) + fprintf(stderr, "apps.plugin: extending fd memory slots for %s from %d to %d\n", p->comm, p->fds_size, fdid + 100); + + p->fds = reallocz(p->fds, (fdid + 100) * sizeof(int)); + if(!p->fds) { + fatal("Cannot re-allocate fds for %s", p->comm); + break; + } + + // and initialize it + for(c = p->fds_size ; c < (fdid + 100) ; c++) p->fds[c] = 0; + p->fds_size = fdid + 100; + } + + if(p->fds[fdid] == 0) { + // we don't know this fd, get it + + sprintf(fdname, "%s/proc/%d/fd/%s", host_prefix, p->pid, de->d_name); + ssize_t l = readlink(fdname, linkname, FILENAME_MAX); + if(l == -1) { + if(debug || (p->target && p->target->debug)) { + if(debug || (p->target && p->target->debug)) + error("Cannot read link %s", fdname); + } + continue; + } + linkname[l] = '\0'; + file_counter++; + + // if another process already has this, we will get + // the same id + p->fds[fdid] = file_descriptor_find_or_add(linkname); + } + + // else make it positive again, we need it + // of course, the actual file may have changed, but we don't care so much + // FIXME: we could compare the inode as returned by readdir dirent structure + else p->fds[fdid] = -p->fds[fdid]; + } + closedir(fds); + + // remove all the negative file descriptors + for(c = 0 ; c < p->fds_size ; c++) if(p->fds[c] < 0) { + file_descriptor_not_used(-p->fds[c]); + p->fds[c] = 0; + } + } + else return 0; + + return 1; +} + +// ---------------------------------------------------------------------------- + +int print_process_and_parents(struct pid_stat *p, unsigned long long time) { + char *prefix = "\\_ "; + int indent = 0; + + if(p->parent) + indent = print_process_and_parents(p->parent, p->stat_collected_usec); + else + prefix = " > "; + + char buffer[indent + 1]; + int i; + + for(i = 0; i < indent ;i++) buffer[i] = ' '; + buffer[i] = '\0'; + + fprintf(stderr, " %s %s%s (%d %s %lld" + , buffer + , prefix + , p->comm + , p->pid + , p->updated?"running":"exited" + , (long long)p->stat_collected_usec - (long long)time + ); + + if(p->utime) fprintf(stderr, " utime=%llu", p->utime); + if(p->stime) fprintf(stderr, " stime=%llu", p->stime); + if(p->gtime) fprintf(stderr, " gtime=%llu", p->gtime); + if(p->cutime) fprintf(stderr, " cutime=%llu", p->cutime); + if(p->cstime) fprintf(stderr, " cstime=%llu", p->cstime); + if(p->cgtime) fprintf(stderr, " cgtime=%llu", p->cgtime); + if(p->minflt) fprintf(stderr, " minflt=%llu", p->minflt); + if(p->cminflt) fprintf(stderr, " cminflt=%llu", p->cminflt); + if(p->majflt) fprintf(stderr, " majflt=%llu", p->majflt); + if(p->cmajflt) fprintf(stderr, " cmajflt=%llu", p->cmajflt); + fprintf(stderr, ")\n"); + + return indent + 1; +} + +void print_process_tree(struct pid_stat *p, char *msg) { + log_date(stderr); + fprintf(stderr, "%s: process %s (%d, %s) with parents:\n", msg, p->comm, p->pid, p->updated?"running":"exited"); + print_process_and_parents(p, p->stat_collected_usec); +} + +void find_lost_child_debug(struct pid_stat *pe, unsigned long long lost, int type) { + int found = 0; + struct pid_stat *p = NULL; + + for(p = root_of_pids; p ; p = p->next) { + if(p == pe) continue; + + switch(type) { + case 1: + if(p->cminflt > lost) { + fprintf(stderr, " > process %d (%s) could use the lost exited child minflt %llu of process %d (%s)\n", p->pid, p->comm, lost, pe->pid, pe->comm); + found++; + } + break; + + case 2: + if(p->cmajflt > lost) { + fprintf(stderr, " > process %d (%s) could use the lost exited child majflt %llu of process %d (%s)\n", p->pid, p->comm, lost, pe->pid, pe->comm); + found++; + } + break; + + case 3: + if(p->cutime > lost) { + fprintf(stderr, " > process %d (%s) could use the lost exited child utime %llu of process %d (%s)\n", p->pid, p->comm, lost, pe->pid, pe->comm); + found++; + } + break; + + case 4: + if(p->cstime > lost) { + fprintf(stderr, " > process %d (%s) could use the lost exited child stime %llu of process %d (%s)\n", p->pid, p->comm, lost, pe->pid, pe->comm); + found++; + } + break; + + case 5: + if(p->cgtime > lost) { + fprintf(stderr, " > process %d (%s) could use the lost exited child gtime %llu of process %d (%s)\n", p->pid, p->comm, lost, pe->pid, pe->comm); + found++; + } + break; + } + } + + if(!found) { + switch(type) { + case 1: + fprintf(stderr, " > cannot find any process to use the lost exited child minflt %llu of process %d (%s)\n", lost, pe->pid, pe->comm); + break; + + case 2: + fprintf(stderr, " > cannot find any process to use the lost exited child majflt %llu of process %d (%s)\n", lost, pe->pid, pe->comm); + break; + + case 3: + fprintf(stderr, " > cannot find any process to use the lost exited child utime %llu of process %d (%s)\n", lost, pe->pid, pe->comm); + break; + + case 4: + fprintf(stderr, " > cannot find any process to use the lost exited child stime %llu of process %d (%s)\n", lost, pe->pid, pe->comm); + break; + + case 5: + fprintf(stderr, " > cannot find any process to use the lost exited child gtime %llu of process %d (%s)\n", lost, pe->pid, pe->comm); + break; + } + } +} + +unsigned long long remove_exited_child_from_parent(unsigned long long *field, unsigned long long *pfield) { + unsigned long long absorbed = 0; + + if(*field > *pfield) { + absorbed += *pfield; + *field -= *pfield; + *pfield = 0; + } + else { + absorbed += *field; + *pfield -= *field; + *field = 0; + } + + return absorbed; +} + +void process_exited_processes() { + struct pid_stat *p; + + for(p = root_of_pids; p ; p = p->next) { + if(p->updated || !p->stat_collected_usec) + continue; + + struct pid_stat *pp = p->parent; + + unsigned long long utime = (p->utime_raw + p->cutime_raw) * (1000000ULL * RATES_DETAIL) / (p->stat_collected_usec - p->last_stat_collected_usec); + unsigned long long stime = (p->stime_raw + p->cstime_raw) * (1000000ULL * RATES_DETAIL) / (p->stat_collected_usec - p->last_stat_collected_usec); + unsigned long long gtime = (p->gtime_raw + p->cgtime_raw) * (1000000ULL * RATES_DETAIL) / (p->stat_collected_usec - p->last_stat_collected_usec); + unsigned long long minflt = (p->minflt_raw + p->cminflt_raw) * (1000000ULL * RATES_DETAIL) / (p->stat_collected_usec - p->last_stat_collected_usec); + unsigned long long majflt = (p->majflt_raw + p->cmajflt_raw) * (1000000ULL * RATES_DETAIL) / (p->stat_collected_usec - p->last_stat_collected_usec); + + if(utime + stime + gtime + minflt + majflt == 0) + continue; + + if(unlikely(debug)) { + log_date(stderr); + fprintf(stderr, "Absorb %s (%d %s total resources: utime=%llu stime=%llu gtime=%llu minflt=%llu majflt=%llu)\n" + , p->comm + , p->pid + , p->updated?"running":"exited" + , utime + , stime + , gtime + , minflt + , majflt + ); + print_process_tree(p, "Searching parents"); + } + + for(pp = p->parent; pp ; pp = pp->parent) { + if(!pp->updated) continue; + + unsigned long long absorbed; + absorbed = remove_exited_child_from_parent(&utime, &pp->cutime); + if(unlikely(debug && absorbed)) + fprintf(stderr, " > process %s (%d %s) absorbed %llu utime (remaining: %llu)\n", pp->comm, pp->pid, pp->updated?"running":"exited", absorbed, utime); + + absorbed = remove_exited_child_from_parent(&stime, &pp->cstime); + if(unlikely(debug && absorbed)) + fprintf(stderr, " > process %s (%d %s) absorbed %llu stime (remaining: %llu)\n", pp->comm, pp->pid, pp->updated?"running":"exited", absorbed, stime); + + absorbed = remove_exited_child_from_parent(>ime, &pp->cgtime); + if(unlikely(debug && absorbed)) + fprintf(stderr, " > process %s (%d %s) absorbed %llu gtime (remaining: %llu)\n", pp->comm, pp->pid, pp->updated?"running":"exited", absorbed, gtime); + + absorbed = remove_exited_child_from_parent(&minflt, &pp->cminflt); + if(unlikely(debug && absorbed)) + fprintf(stderr, " > process %s (%d %s) absorbed %llu minflt (remaining: %llu)\n", pp->comm, pp->pid, pp->updated?"running":"exited", absorbed, minflt); + + absorbed = remove_exited_child_from_parent(&majflt, &pp->cmajflt); + if(unlikely(debug && absorbed)) + fprintf(stderr, " > process %s (%d %s) absorbed %llu majflt (remaining: %llu)\n", pp->comm, pp->pid, pp->updated?"running":"exited", absorbed, majflt); + } + + if(unlikely(utime + stime + gtime + minflt + majflt > 0)) { + if(unlikely(debug)) { + if(utime) find_lost_child_debug(p, utime, 3); + if(stime) find_lost_child_debug(p, stime, 4); + if(gtime) find_lost_child_debug(p, gtime, 5); + if(minflt) find_lost_child_debug(p, minflt, 1); + if(majflt) find_lost_child_debug(p, majflt, 2); + } + + p->keep = 1; + + if(unlikely(debug)) + fprintf(stderr, " > remaining resources - KEEP - for another loop: %s (%d %s total resources: utime=%llu stime=%llu gtime=%llu minflt=%llu majflt=%llu)\n" + , p->comm + , p->pid + , p->updated?"running":"exited" + , utime + , stime + , gtime + , minflt + , majflt + ); + + for(pp = p->parent; pp ; pp = pp->parent) { + if(pp->updated) break; + pp->keep = 1; + + if(unlikely(debug)) + fprintf(stderr, " > - KEEP - parent for another loop: %s (%d %s)\n" + , pp->comm + , pp->pid + , pp->updated?"running":"exited" + ); + } + + p->utime_raw = utime * (p->stat_collected_usec - p->last_stat_collected_usec) / (1000000ULL * RATES_DETAIL); + p->stime_raw = stime * (p->stat_collected_usec - p->last_stat_collected_usec) / (1000000ULL * RATES_DETAIL); + p->gtime_raw = gtime * (p->stat_collected_usec - p->last_stat_collected_usec) / (1000000ULL * RATES_DETAIL); + p->minflt_raw = minflt * (p->stat_collected_usec - p->last_stat_collected_usec) / (1000000ULL * RATES_DETAIL); + p->majflt_raw = majflt * (p->stat_collected_usec - p->last_stat_collected_usec) / (1000000ULL * RATES_DETAIL); + p->cutime_raw = p->cstime_raw = p->cgtime_raw = p->cminflt_raw = p->cmajflt_raw = 0; + + if(unlikely(debug)) + fprintf(stderr, "\n"); + } + else if(unlikely(debug)) { + fprintf(stderr, " > totally absorbed - DONE - %s (%d %s)\n" + , p->comm + , p->pid + , p->updated?"running":"exited" + ); + } + } +} + +void link_all_processes_to_their_parents(void) { + struct pid_stat *p, *pp; + + // link all children to their parents + // and update children count on parents + for(p = root_of_pids; p ; p = p->next) { + // for each process found + + p->sortlist = 0; + p->parent = NULL; + + if(unlikely(!p->ppid)) { + p->parent = NULL; + continue; + } + + pp = all_pids[p->ppid]; + if(likely(pp)) { + p->parent = pp; + pp->children_count++; + + if(unlikely(debug || (p->target && p->target->debug))) + fprintf(stderr, "apps.plugin: \tchild %d (%s, %s) on target '%s' has parent %d (%s, %s). Parent: utime=%llu, stime=%llu, gtime=%llu, minflt=%llu, majflt=%llu, cutime=%llu, cstime=%llu, cgtime=%llu, cminflt=%llu, cmajflt=%llu\n", p->pid, p->comm, p->updated?"running":"exited", (p->target)?p->target->name:"UNSET", pp->pid, pp->comm, pp->updated?"running":"exited", pp->utime, pp->stime, pp->gtime, pp->minflt, pp->majflt, pp->cutime, pp->cstime, pp->cgtime, pp->cminflt, pp->cmajflt); + } + else { + p->parent = NULL; + error("pid %d %s states parent %d, but the later does not exist.", p->pid, p->comm, p->ppid); + } + } } // ---------------------------------------------------------------------------- @@ -1269,206 +1563,224 @@ int read_pid_file_descriptors(struct pid_stat *p) { // to avoid filling up all disk space // if debug is enabled, all errors are printed -int collect_data_for_all_processes_from_proc(void) -{ - char dirname[FILENAME_MAX + 1]; +static int compar_pid(const void *pid1, const void *pid2) { + + struct pid_stat *p1 = all_pids[*((pid_t *)pid1)]; + struct pid_stat *p2 = all_pids[*((pid_t *)pid2)]; - snprintfz(dirname, FILENAME_MAX, "%s/proc", host_prefix); - DIR *dir = opendir(dirname); - if(!dir) return 0; + if(p1->sortlist > p2->sortlist) + return -1; + else + return 1; +} - struct dirent *file = NULL; - struct pid_stat *p = NULL; +static inline int managed_log(struct pid_stat *p, uint32_t log, int status) { + if(unlikely(!status)) { + // error("command failed log %u, errno %d", log, errno); + + if(unlikely(debug || errno != ENOENT)) { + if(unlikely(debug || !(p->log_thrown & log))) { + p->log_thrown |= log; + switch(log) { + case PID_LOG_IO: + error("Cannot process %s/proc/%d/io (command '%s')", host_prefix, p->pid, p->comm); + break; + + case PID_LOG_STATM: + error("Cannot process %s/proc/%d/statm (command '%s')", host_prefix, p->pid, p->comm); + break; + + case PID_LOG_CMDLINE: + error("Cannot process %s/proc/%d/cmdline (command '%s')", host_prefix, p->pid, p->comm); + break; + + case PID_LOG_FDS: + error("Cannot process entries in %s/proc/%d/fd (command '%s')", host_prefix, p->pid, p->comm); + break; + + case PID_LOG_STAT: + break; + + default: + error("unhandled error for pid %d, command '%s'", p->pid, p->comm); + break; + } + } + } + errno = 0; + } + else if(unlikely(p->log_thrown & log)) { + // error("unsetting log %u on pid %d", log, p->pid); + p->log_thrown &= ~log; + } + + return status; +} - // mark them all as un-updated - all_pids_count = 0; - for(p = root_of_pids; p ; p = p->next) { - all_pids_count++; - p->parent = NULL; - p->updated = 0; - p->children_count = 0; - p->merged = 0; - p->new_entry = 0; +void collect_data_for_pid(pid_t pid) { + if(unlikely(pid <= 0 || pid > pid_max)) { + error("Invalid pid %d read (expected 1 to %d). Ignoring process.", pid, pid_max); + return; + } - p->last_minflt = p->minflt; - p->last_cminflt = p->cminflt; - p->last_majflt = p->majflt; - p->last_cmajflt = p->cmajflt; - p->last_utime = p->utime; - p->last_stime = p->stime; - p->last_cutime = p->cutime; - p->last_cstime = p->cstime; - - p->last_io_logical_bytes_read = p->io_logical_bytes_read; - p->last_io_logical_bytes_written = p->io_logical_bytes_written; - p->last_io_read_calls = p->io_read_calls; - p->last_io_write_calls = p->io_write_calls; - p->last_io_storage_bytes_read = p->io_storage_bytes_read; - p->last_io_storage_bytes_written = p->io_storage_bytes_written; - p->last_io_cancelled_write_bytes = p->io_cancelled_write_bytes; - } - - while((file = readdir(dir))) { - char *endptr = file->d_name; - pid_t pid = (pid_t) strtoul(file->d_name, &endptr, 10); - - // make sure we read a valid number - if(unlikely(pid <= 0 || pid > pid_max || endptr == file->d_name || *endptr != '\0')) - continue; - - p = get_pid_entry(pid); - if(unlikely(!p)) continue; - - - // -------------------------------------------------------------------- - // /proc/<pid>/stat - - if(unlikely(read_proc_pid_stat(p))) { - error("Cannot process %s/proc/%d/stat", host_prefix, pid); - - // there is no reason to proceed if we cannot get its status - continue; - } - - // check its parent pid - if(unlikely(p->ppid < 0 || p->ppid > pid_max)) { - error("Pid %d states invalid parent pid %d. Using 0.", pid, p->ppid); - - p->ppid = 0; - } - - // -------------------------------------------------------------------- - // /proc/<pid>/cmdline - - if(proc_pid_cmdline_is_needed) { - if(unlikely(read_proc_pid_cmdline(p))) { - error("Cannot process %s/proc/%d/cmdline", host_prefix, pid); - } - } - - // -------------------------------------------------------------------- - // /proc/<pid>/statm - - if(unlikely(read_proc_pid_statm(p))) { - error("Cannot process %s/proc/%d/statm", host_prefix, pid); - - // there is no reason to proceed if we cannot get its memory status - continue; - } - - - // -------------------------------------------------------------------- - // /proc/<pid>/io - - if(unlikely(read_proc_pid_io(p))) { - error("Cannot process %s/proc/%d/io", host_prefix, pid); - - // on systems without /proc/X/io - // allow proceeding without I/O information - // continue; - } - - // -------------------------------------------------------------------- - // <pid> ownership - - if(unlikely(read_proc_pid_ownership(p))) { - error("Cannot stat %s/proc/%d", host_prefix, pid); - } - - // -------------------------------------------------------------------- - // link it - - // check if it is target - // we do this only once, the first time this pid is loaded - if(unlikely(p->new_entry)) { - if(debug) fprintf(stderr, "apps.plugin: \tJust added %s\n", p->comm); - uint32_t hash = simple_hash(p->comm); - size_t pclen = strlen(p->comm); - - struct target *w; - for(w = apps_groups_root_target; w ; w = w->next) { - // if(debug || (p->target && p->target->debug)) fprintf(stderr, "apps.plugin: \t\tcomparing '%s' with '%s'\n", w->compare, p->comm); - - // find it - 4 cases: - // 1. the target is not a pattern - // 2. the target has the prefix - // 3. the target has the suffix - // 4. the target is something inside cmdline - if( (!w->starts_with && !w->ends_with && w->comparehash == hash && !strcmp(w->compare, p->comm)) - || (w->starts_with && !w->ends_with && !strncmp(w->compare, p->comm, w->comparelen)) - || (!w->starts_with && w->ends_with && pclen >= w->comparelen && !strcmp(w->compare, &p->comm[pclen - w->comparelen])) - || (proc_pid_cmdline_is_needed && w->starts_with && w->ends_with && strstr(p->cmdline, w->compare)) - ) { - if(w->target) p->target = w->target; - else p->target = w; + struct pid_stat *p = get_pid_entry(pid); + if(unlikely(!p || p->read)) return; + p->read = 1; - if(debug || (p->target && p->target->debug)) - fprintf(stderr, "apps.plugin: \t\t%s linked to target %s\n", p->comm, p->target->name); - } - } - } + // fprintf(stderr, "Reading process %d (%s), sortlist %d\n", p->pid, p->comm, p->sortlist); - // -------------------------------------------------------------------- - // /proc/<pid>/fd + // -------------------------------------------------------------------- + // /proc/<pid>/stat - if(unlikely(read_pid_file_descriptors(p))) { - error("Cannot process entries in %s/proc/%d/fd", host_prefix, pid); - } + if(unlikely(!managed_log(p, PID_LOG_STAT, read_proc_pid_stat(p)))) + // there is no reason to proceed if we cannot get its status + return; - // -------------------------------------------------------------------- - // done! + read_proc_pid_ownership(p); - // mark it as updated - p->updated = 1; - } + // check its parent pid + if(unlikely(p->ppid < 0 || p->ppid > pid_max)) { + error("Pid %d (command '%s') states invalid parent pid %d. Using 0.", pid, p->comm, p->ppid); + p->ppid = 0; + } - closedir(dir); + // -------------------------------------------------------------------- + // /proc/<pid>/io - return 1; -} + managed_log(p, PID_LOG_IO, read_proc_pid_io(p)); + // -------------------------------------------------------------------- + // /proc/<pid>/statm -// ---------------------------------------------------------------------------- + if(unlikely(!managed_log(p, PID_LOG_STATM, read_proc_pid_statm(p)))) + // there is no reason to proceed if we cannot get its memory status + return; -#ifdef AGGREGATE_CHILDREN_TO_PARENTS -// print a tree view of all processes -int debug_childrens_aggregations(pid_t pid, int level) { - struct pid_stat *p = NULL; - char b[level+3]; - int i, ret = 0; - - for(i = 0; i < level; i++) b[i] = '\t'; - b[level] = '|'; - b[level+1] = '-'; - b[level+2] = '\0'; - - for(p = root_of_pids; p ; p = p->next) { - if(p->ppid == pid) { - ret += debug_childrens_aggregations(p->pid, level+1); - } - } - - p = all_pids[pid]; - if(p) { - if(!p->updated) ret += 1; - if(ret) fprintf(stderr, "%s %s %d [%s, %s] c=%d u=%llu+%llu, s=%llu+%llu, cu=%llu+%llu, cs=%llu+%llu, n=%llu+%llu, j=%llu+%llu, cn=%llu+%llu, cj=%llu+%llu\n" - , b, p->comm, p->pid, p->updated?"OK":"KILLED", p->target->name, p->children_count - , p->utime, p->utime - p->old_utime - , p->stime, p->stime - p->old_stime - , p->cutime, p->cutime - p->old_cutime - , p->cstime, p->cstime - p->old_cstime - , p->minflt, p->minflt - p->old_minflt - , p->majflt, p->majflt - p->old_majflt - , p->cminflt, p->cminflt - p->old_cminflt - , p->cmajflt, p->cmajflt - p->old_cmajflt - ); - } - - return ret; -} -#endif /* AGGREGATE_CHILDREN_TO_PARENTS */ + // -------------------------------------------------------------------- + // link it + // check if it is target + // we do this only once, the first time this pid is loaded + if(unlikely(p->new_entry)) { + // /proc/<pid>/cmdline + if(likely(proc_pid_cmdline_is_needed)) + managed_log(p, PID_LOG_CMDLINE, read_proc_pid_cmdline(p)); + if(unlikely(debug)) + fprintf(stderr, "apps.plugin: \tJust added %d (%s)\n", pid, p->comm); + + uint32_t hash = simple_hash(p->comm); + size_t pclen = strlen(p->comm); + + struct target *w; + for(w = apps_groups_root_target; w ; w = w->next) { + // if(debug || (p->target && p->target->debug)) fprintf(stderr, "apps.plugin: \t\tcomparing '%s' with '%s'\n", w->compare, p->comm); + + // find it - 4 cases: + // 1. the target is not a pattern + // 2. the target has the prefix + // 3. the target has the suffix + // 4. the target is something inside cmdline + if( (!w->starts_with && !w->ends_with && w->comparehash == hash && !strcmp(w->compare, p->comm)) + || (w->starts_with && !w->ends_with && !strncmp(w->compare, p->comm, w->comparelen)) + || (!w->starts_with && w->ends_with && pclen >= w->comparelen && !strcmp(w->compare, &p->comm[pclen - w->comparelen])) + || (proc_pid_cmdline_is_needed && w->starts_with && w->ends_with && strstr(p->cmdline, w->compare)) + ) { + if(w->target) p->target = w->target; + else p->target = w; + + if(debug || (p->target && p->target->debug)) + fprintf(stderr, "apps.plugin: \t\t%s linked to target %s\n", p->comm, p->target->name); + + break; + } + } + } + + // -------------------------------------------------------------------- + // /proc/<pid>/fd + + if(enable_file_charts) + managed_log(p, PID_LOG_FDS, read_pid_file_descriptors(p)); + + // -------------------------------------------------------------------- + // done! + + if(unlikely(debug && include_exited_childs && all_pids_count && p->ppid && all_pids[p->ppid] && !all_pids[p->ppid]->read)) + fprintf(stderr, "Read process %d (%s) sortlisted %d, but its parent %d (%s) sortlisted %d, is not read\n", p->pid, p->comm, p->sortlist, all_pids[p->ppid]->pid, all_pids[p->ppid]->comm, all_pids[p->ppid]->sortlist); + + // mark it as updated + p->updated = 1; + p->keep = 0; + p->keeploops = 0; +} + +int collect_data_for_all_processes_from_proc(void) { + struct pid_stat *p = NULL; + + if(all_pids_count) { + // read parents before childs + // this is needed to prevent a situation where + // a child is found running, but until we read + // its parent, it has exited and its parent + // has accumulated its resources + + long slc = 0; + for(p = root_of_pids; p ; p = p->next) { + p->read = 0; + p->updated = 0; + p->new_entry = 0; + p->merged = 0; + p->children_count = 0; + p->parent = NULL; + + all_pids_sortlist[slc++] = p->pid; + } + + if(unlikely(slc != all_pids_count)) { + error("Internal error: I was thinking I had %ld processes in my arrays, but it seems there are more.", all_pids_count); + all_pids_count = slc; + } + + if(include_exited_childs) { + qsort((void *)all_pids_sortlist, all_pids_count, sizeof(pid_t), compar_pid); + for(slc = 0; slc < all_pids_count; slc++) + collect_data_for_pid(all_pids_sortlist[slc]); + } + } + + char dirname[FILENAME_MAX + 1]; + + snprintfz(dirname, FILENAME_MAX, "%s/proc", host_prefix); + DIR *dir = opendir(dirname); + if(!dir) return 0; + + struct dirent *file = NULL; + + while((file = readdir(dir))) { + char *endptr = file->d_name; + pid_t pid = (pid_t) strtoul(file->d_name, &endptr, 10); + + // make sure we read a valid number + if(unlikely(endptr == file->d_name || *endptr != '\0')) + continue; + + collect_data_for_pid(pid); + } + closedir(dir); + + // normally this is done + // however we may have processes exited while we collected values + // so let's find the exited ones + // we do this by collecting the ownership of process + // if we manage to get the ownership, the process still runs + + read_proc_stat(); + link_all_processes_to_their_parents(); + process_exited_processes(); + + return 1; +} // ---------------------------------------------------------------------------- // update statistics on the targets @@ -1486,729 +1798,760 @@ int debug_childrens_aggregations(pid_t pid, int level) { // 9. find the unique file count for each target // check: update_apps_groups_statistics() -void link_all_processes_to_their_parents(void) { - struct pid_stat *p = NULL; - - // link all children to their parents - // and update children count on parents - for(p = root_of_pids; p ; p = p->next) { - // for each process found running - - if(p->ppid > 0 - && p->ppid <= pid_max - && all_pids[p->ppid] - ) { - // for valid processes - - if(debug || (p->target && p->target->debug)) - fprintf(stderr, "apps.plugin: \tparent of %d (%s) is %d (%s)\n", p->pid, p->comm, p->ppid, all_pids[p->ppid]->comm); - - p->parent = all_pids[p->ppid]; - p->parent->children_count++; - } - else if(p->ppid != 0) - error("pid %d %s states parent %d, but the later does not exist.", p->pid, p->comm, p->ppid); - } -} - -#ifdef AGGREGATE_CHILDREN_TO_PARENTS -void aggregate_children_to_parents(void) { - struct pid_stat *p = NULL; - - // for each killed process, remove its values from the parents - // sums (we had already added them in a previous loop) - for(p = root_of_pids; p ; p = p->next) { - if(p->updated) continue; - - if(debug) fprintf(stderr, "apps.plugin: UNMERGING %d %s\n", p->pid, p->comm); - - unsigned long long diff_utime = p->utime + p->cutime + p->fix_cutime; - unsigned long long diff_stime = p->stime + p->cstime + p->fix_cstime; - unsigned long long diff_minflt = p->minflt + p->cminflt + p->fix_cminflt; - unsigned long long diff_majflt = p->majflt + p->cmajflt + p->fix_cmajflt; - - struct pid_stat *t = p; - while((t = t->parent)) { - if(!t->updated) continue; - - unsigned long long x; - if(diff_utime && t->diff_cutime) { - x = (t->diff_cutime < diff_utime)?t->diff_cutime:diff_utime; - diff_utime -= x; - t->diff_cutime -= x; - t->fix_cutime += x; - if(debug) fprintf(stderr, "apps.plugin: \t cutime %llu from %d %s %s\n", x, t->pid, t->comm, t->target->name); - } - if(diff_stime && t->diff_cstime) { - x = (t->diff_cstime < diff_stime)?t->diff_cstime:diff_stime; - diff_stime -= x; - t->diff_cstime -= x; - t->fix_cstime += x; - if(debug) fprintf(stderr, "apps.plugin: \t cstime %llu from %d %s %s\n", x, t->pid, t->comm, t->target->name); - } - if(diff_minflt && t->diff_cminflt) { - x = (t->diff_cminflt < diff_minflt)?t->diff_cminflt:diff_minflt; - diff_minflt -= x; - t->diff_cminflt -= x; - t->fix_cminflt += x; - if(debug) fprintf(stderr, "apps.plugin: \t cminflt %llu from %d %s %s\n", x, t->pid, t->comm, t->target->name); - } - if(diff_majflt && t->diff_cmajflt) { - x = (t->diff_cmajflt < diff_majflt)?t->diff_cmajflt:diff_majflt; - diff_majflt -= x; - t->diff_cmajflt -= x; - t->fix_cmajflt += x; - if(debug) fprintf(stderr, "apps.plugin: \t cmajflt %llu from %d %s %s\n", x, t->pid, t->comm, t->target->name); - } - } - - if(diff_utime) error("Cannot fix up utime %llu", diff_utime); - if(diff_stime) error("Cannot fix up stime %llu", diff_stime); - if(diff_minflt) error("Cannot fix up minflt %llu", diff_minflt); - if(diff_majflt) error("Cannot fix up majflt %llu", diff_majflt); - } -} -#endif /* AGGREGATE_CHILDREN_TO_PARENTS */ - -void cleanup_non_existing_pids(void) { - int c; - struct pid_stat *p = NULL; - - for(p = root_of_pids; p ;) { - if(!p->updated) { -// fprintf(stderr, "\tEXITED %d %s [parent %d %s, target %s] utime=%llu, stime=%llu, cutime=%llu, cstime=%llu, minflt=%llu, majflt=%llu, cminflt=%llu, cmajflt=%llu\n", p->pid, p->comm, p->parent->pid, p->parent->comm, p->target->name, p->utime, p->stime, p->cutime, p->cstime, p->minflt, p->majflt, p->cminflt, p->cmajflt); - - for(c = 0 ; c < p->fds_size ; c++) if(p->fds[c] > 0) { - file_descriptor_not_used(p->fds[c]); - p->fds[c] = 0; - } - - pid_t r = p->pid; - p = p->next; - del_pid_entry(r); - } - else p = p->next; - } +void cleanup_exited_pids(void) { + int c; + struct pid_stat *p = NULL; + + for(p = root_of_pids; p ;) { + if(!p->updated && (!p->keep || p->keeploops > 0)) { +// fprintf(stderr, "\tEXITED %d %s [parent %d %s, target %s] utime=%llu, stime=%llu, gtime=%llu, cutime=%llu, cstime=%llu, cgtime=%llu, minflt=%llu, majflt=%llu, cminflt=%llu, cmajflt=%llu\n", p->pid, p->comm, p->parent->pid, p->parent->comm, p->target->name, p->utime, p->stime, p->gtime, p->cutime, p->cstime, p->cgtime, p->minflt, p->majflt, p->cminflt, p->cmajflt); + + if(unlikely(debug && (p->keep || p->keeploops))) + fprintf(stderr, " > CLEANUP cannot keep exited process %d (%s) anymore - removing it.\n", p->pid, p->comm); + + for(c = 0 ; c < p->fds_size ; c++) if(p->fds[c] > 0) { + file_descriptor_not_used(p->fds[c]); + p->fds[c] = 0; + } + + pid_t r = p->pid; + p = p->next; + del_pid_entry(r); + } + else { + if(unlikely(p->keep)) p->keeploops++; + p->keep = 0; + p = p->next; + } + } } void apply_apps_groups_targets_inheritance(void) { - struct pid_stat *p = NULL; - - // children that do not have a target - // inherit their target from their parent - int found = 1; - while(found) { - found = 0; - for(p = root_of_pids; p ; p = p->next) { - // if this process does not have a target - // and it has a parent - // and its parent has a target - // then, set the parent's target to this process - if(unlikely(!p->target && p->parent && p->parent->target)) { - p->target = p->parent->target; - found++; - - if(debug || (p->target && p->target->debug)) - fprintf(stderr, "apps.plugin: \t\tTARGET INHERITANCE: %s is inherited by %d (%s) from its parent %d (%s).\n", p->target->name, p->pid, p->comm, p->parent->pid, p->parent->comm); - } - } - } - - - // find all the procs with 0 childs and merge them to their parents - // repeat, until nothing more can be done. - found = 1; - while(found) { - found = 0; - for(p = root_of_pids; p ; p = p->next) { - // if this process does not have any children - // and is not already merged - // and has a parent - // and its parent has children - // and the target of this process and its parent is the same, or the parent does not have a target - // and its parent is not init - // then, mark them as merged. - if(unlikely( - !p->children_count - && !p->merged - && p->parent - && p->parent->children_count - && (p->target == p->parent->target || !p->parent->target) - && p->ppid != 1 - )) { - p->parent->children_count--; - p->merged = 1; - - // the parent inherits the child's target, if it does not have a target itself - if(unlikely(p->target && !p->parent->target)) { - p->parent->target = p->target; - - if(debug || (p->target && p->target->debug)) - fprintf(stderr, "apps.plugin: \t\tTARGET INHERITANCE: %s is inherited by %d (%s) from its child %d (%s).\n", p->target->name, p->parent->pid, p->parent->comm, p->pid, p->comm); - } - - found++; - } - } - - if(debug) - fprintf(stderr, "apps.plugin: merged %d processes\n", found); - } - - // init goes always to default target - if(all_pids[1]) - all_pids[1]->target = apps_groups_default_target; - - // give a default target on all top level processes - for(p = root_of_pids; p ; p = p->next) { - // if the process is not merged itself - // then is is a top level process - if(!p->merged && !p->target) - p->target = apps_groups_default_target; - -#ifdef AGGREGATE_CHILDREN_TO_PARENTS - // by the way, update the diffs - // will be used later for subtracting killed process times - p->diff_cutime = p->utime - p->cutime; - p->diff_cstime = p->stime - p->cstime; - p->diff_cminflt = p->minflt - p->cminflt; - p->diff_cmajflt = p->majflt - p->cmajflt; -#endif /* AGGREGATE_CHILDREN_TO_PARENTS */ - } - - // give a target to all merged child processes - found = 1; - while(found) { - found = 0; - for(p = root_of_pids; p ; p = p->next) { - if(unlikely(!p->target && p->merged && p->parent && p->parent->target)) { - p->target = p->parent->target; - found++; - - if(debug || (p->target && p->target->debug)) - fprintf(stderr, "apps.plugin: \t\tTARGET INHERITANCE: %s is inherited by %d (%s) from its parent %d (%s) at phase 2.\n", p->target->name, p->pid, p->comm, p->parent->pid, p->parent->comm); - } - } - } + struct pid_stat *p = NULL; + + // children that do not have a target + // inherit their target from their parent + int found = 1, loops = 0; + while(found) { + if(unlikely(debug)) loops++; + found = 0; + for(p = root_of_pids; p ; p = p->next) { + // if this process does not have a target + // and it has a parent + // and its parent has a target + // then, set the parent's target to this process + if(unlikely(!p->target && p->parent && p->parent->target)) { + p->target = p->parent->target; + found++; + + if(debug || (p->target && p->target->debug)) + fprintf(stderr, "apps.plugin: \t\tTARGET INHERITANCE: %s is inherited by %d (%s) from its parent %d (%s).\n", p->target->name, p->pid, p->comm, p->parent->pid, p->parent->comm); + } + } + } + + // find all the procs with 0 childs and merge them to their parents + // repeat, until nothing more can be done. + int sortlist = 1; + found = 1; + while(found) { + if(unlikely(debug)) loops++; + found = 0; + + for(p = root_of_pids; p ; p = p->next) { + if(unlikely(!p->sortlist && !p->children_count)) + p->sortlist = sortlist++; + + // if this process does not have any children + // and is not already merged + // and has a parent + // and its parent has children + // and the target of this process and its parent is the same, or the parent does not have a target + // and its parent is not init + // then, mark them as merged. + if(unlikely( + !p->children_count + && !p->merged + && p->parent + && p->parent->children_count + && (p->target == p->parent->target || !p->parent->target) + && p->ppid != 1 + )) { + p->parent->children_count--; + p->merged = 1; + + // the parent inherits the child's target, if it does not have a target itself + if(unlikely(p->target && !p->parent->target)) { + p->parent->target = p->target; + + if(debug || (p->target && p->target->debug)) + fprintf(stderr, "apps.plugin: \t\tTARGET INHERITANCE: %s is inherited by %d (%s) from its child %d (%s).\n", p->target->name, p->parent->pid, p->parent->comm, p->pid, p->comm); + } + + found++; + } + } + + if(unlikely(debug)) + fprintf(stderr, "apps.plugin: TARGET INHERITANCE: merged %d processes\n", found); + } + + // init goes always to default target + if(all_pids[1]) + all_pids[1]->target = apps_groups_default_target; + + // give a default target on all top level processes + if(unlikely(debug)) loops++; + for(p = root_of_pids; p ; p = p->next) { + // if the process is not merged itself + // then is is a top level process + if(unlikely(!p->merged && !p->target)) + p->target = apps_groups_default_target; + + // make sure all processes have a sortlist + if(unlikely(!p->sortlist)) + p->sortlist = sortlist++; + } + + if(all_pids[1]) + all_pids[1]->sortlist = sortlist++; + + // give a target to all merged child processes + found = 1; + while(found) { + if(unlikely(debug)) loops++; + found = 0; + for(p = root_of_pids; p ; p = p->next) { + if(unlikely(!p->target && p->merged && p->parent && p->parent->target)) { + p->target = p->parent->target; + found++; + + if(debug || (p->target && p->target->debug)) + fprintf(stderr, "apps.plugin: \t\tTARGET INHERITANCE: %s is inherited by %d (%s) from its parent %d (%s) at phase 2.\n", p->target->name, p->pid, p->comm, p->parent->pid, p->parent->comm); + } + } + } + + if(unlikely(debug)) + fprintf(stderr, "apps.plugin: apply_apps_groups_targets_inheritance() made %d loops on the process tree\n", loops); } long zero_all_targets(struct target *root) { - struct target *w; - long count = 0; - - for (w = root; w ; w = w->next) { - count++; - - if(w->fds) free(w->fds); - w->fds = NULL; - - w->minflt = 0; - w->majflt = 0; - w->utime = 0; - w->stime = 0; - w->cminflt = 0; - w->cmajflt = 0; - w->cutime = 0; - w->cstime = 0; - w->num_threads = 0; - w->rss = 0; - w->processes = 0; - - w->statm_size = 0; - w->statm_resident = 0; - w->statm_share = 0; - w->statm_text = 0; - w->statm_lib = 0; - w->statm_data = 0; - w->statm_dirty = 0; - - w->io_logical_bytes_read = 0; - w->io_logical_bytes_written = 0; - w->io_read_calls = 0; - w->io_write_calls = 0; - w->io_storage_bytes_read = 0; - w->io_storage_bytes_written = 0; - w->io_cancelled_write_bytes = 0; - } - - return count; + struct target *w; + long count = 0; + + for (w = root; w ; w = w->next) { + count++; + + if(w->fds) freez(w->fds); + w->fds = NULL; + + w->minflt = 0; + w->majflt = 0; + w->utime = 0; + w->stime = 0; + w->gtime = 0; + w->cminflt = 0; + w->cmajflt = 0; + w->cutime = 0; + w->cstime = 0; + w->cgtime = 0; + w->num_threads = 0; + w->rss = 0; + w->processes = 0; + + w->statm_size = 0; + w->statm_resident = 0; + w->statm_share = 0; + w->statm_text = 0; + w->statm_lib = 0; + w->statm_data = 0; + w->statm_dirty = 0; + + w->io_logical_bytes_read = 0; + w->io_logical_bytes_written = 0; + w->io_read_calls = 0; + w->io_write_calls = 0; + w->io_storage_bytes_read = 0; + w->io_storage_bytes_written = 0; + w->io_cancelled_write_bytes = 0; + } + + return count; } void aggregate_pid_on_target(struct target *w, struct pid_stat *p, struct target *o) { - if(unlikely(!w->fds)) { - w->fds = calloc(sizeof(int), (size_t) all_files_size); - if(unlikely(!w->fds)) - error("Cannot allocate memory for fds in %s", w->name); - } - - if(likely(p->updated)) { - w->cutime += p->cutime; // - p->fix_cutime; - w->cstime += p->cstime; // - p->fix_cstime; - w->cminflt += p->cminflt; // - p->fix_cminflt; - w->cmajflt += p->cmajflt; // - p->fix_cmajflt; - - w->utime += p->utime; //+ (p->pid != 1)?(p->cutime - p->fix_cutime):0; - w->stime += p->stime; //+ (p->pid != 1)?(p->cstime - p->fix_cstime):0; - w->minflt += p->minflt; //+ (p->pid != 1)?(p->cminflt - p->fix_cminflt):0; - w->majflt += p->majflt; //+ (p->pid != 1)?(p->cmajflt - p->fix_cmajflt):0; - - //if(p->num_threads < 0) - // error("Negative threads number for pid '%s' (%d): %d", p->comm, p->pid, p->num_threads); - - //if(p->num_threads > 10000) - // error("Excessive threads number for pid '%s' (%d): %d", p->comm, p->pid, p->num_threads); - - w->num_threads += p->num_threads; - w->rss += p->rss; - - w->statm_size += p->statm_size; - w->statm_resident += p->statm_resident; - w->statm_share += p->statm_share; - w->statm_text += p->statm_text; - w->statm_lib += p->statm_lib; - w->statm_data += p->statm_data; - w->statm_dirty += p->statm_dirty; - - w->io_logical_bytes_read += p->io_logical_bytes_read; - w->io_logical_bytes_written += p->io_logical_bytes_written; - w->io_read_calls += p->io_read_calls; - w->io_write_calls += p->io_write_calls; - w->io_storage_bytes_read += p->io_storage_bytes_read; - w->io_storage_bytes_written += p->io_storage_bytes_written; - w->io_cancelled_write_bytes += p->io_cancelled_write_bytes; - - w->processes++; - - if(likely(w->fds)) { - int c; - for(c = 0; c < p->fds_size ;c++) { - if(p->fds[c] == 0) continue; - - if(likely(p->fds[c] < all_files_size)) { - if(w->fds) w->fds[p->fds[c]]++; - } - else - error("Invalid fd number %d", p->fds[c]); - } - } - - if(unlikely(debug || w->debug)) - fprintf(stderr, "apps.plugin: \tAgregating %s pid %d on %s utime=%llu, stime=%llu, cutime=%llu, cstime=%llu, minflt=%llu, majflt=%llu, cminflt=%llu, cmajflt=%llu\n", p->comm, p->pid, w->name, p->utime, p->stime, p->cutime, p->cstime, p->minflt, p->majflt, p->cminflt, p->cmajflt); - -/* if(p->utime - p->old_utime > 100) fprintf(stderr, "BIG CHANGE: %d %s utime increased by %llu from %llu to %llu\n", p->pid, p->comm, p->utime - p->old_utime, p->old_utime, p->utime); - if(p->cutime - p->old_cutime > 100) fprintf(stderr, "BIG CHANGE: %d %s cutime increased by %llu from %llu to %llu\n", p->pid, p->comm, p->cutime - p->old_cutime, p->old_cutime, p->cutime); - if(p->stime - p->old_stime > 100) fprintf(stderr, "BIG CHANGE: %d %s stime increased by %llu from %llu to %llu\n", p->pid, p->comm, p->stime - p->old_stime, p->old_stime, p->stime); - if(p->cstime - p->old_cstime > 100) fprintf(stderr, "BIG CHANGE: %d %s cstime increased by %llu from %llu to %llu\n", p->pid, p->comm, p->cstime - p->old_cstime, p->old_cstime, p->cstime); - if(p->minflt - p->old_minflt > 5000) fprintf(stderr, "BIG CHANGE: %d %s minflt increased by %llu from %llu to %llu\n", p->pid, p->comm, p->minflt - p->old_minflt, p->old_minflt, p->minflt); - if(p->majflt - p->old_majflt > 5000) fprintf(stderr, "BIG CHANGE: %d %s majflt increased by %llu from %llu to %llu\n", p->pid, p->comm, p->majflt - p->old_majflt, p->old_majflt, p->majflt); - if(p->cminflt - p->old_cminflt > 15000) fprintf(stderr, "BIG CHANGE: %d %s cminflt increased by %llu from %llu to %llu\n", p->pid, p->comm, p->cminflt - p->old_cminflt, p->old_cminflt, p->cminflt); - if(p->cmajflt - p->old_cmajflt > 15000) fprintf(stderr, "BIG CHANGE: %d %s cmajflt increased by %llu from %llu to %llu\n", p->pid, p->comm, p->cmajflt - p->old_cmajflt, p->old_cmajflt, p->cmajflt); -*/ -#ifdef AGGREGATE_CHILDREN_TO_PARENTS - p->old_utime = p->utime; - p->old_cutime = p->cutime; - p->old_stime = p->stime; - p->old_cstime = p->cstime; - p->old_minflt = p->minflt; - p->old_majflt = p->majflt; - p->old_cminflt = p->cminflt; - p->old_cmajflt = p->cmajflt; -#endif /* AGGREGATE_CHILDREN_TO_PARENTS */ - - if(o) { - // since the process switched target - // for all incremental values - // we have to subtract its OLD values from the new target - // and add its OLD values to the old target - - // IMPORTANT - // We add/subtract the last/OLD values we added to the target - - w->fix_cutime -= p->last_cutime; - w->fix_cstime -= p->last_cstime; - w->fix_cminflt -= p->last_cminflt; - w->fix_cmajflt -= p->last_cmajflt; - - w->fix_utime -= p->last_utime; - w->fix_stime -= p->last_stime; - w->fix_minflt -= p->last_minflt; - w->fix_majflt -= p->last_majflt; - - - w->fix_io_logical_bytes_read -= p->last_io_logical_bytes_read; - w->fix_io_logical_bytes_written -= p->last_io_logical_bytes_written; - w->fix_io_read_calls -= p->last_io_read_calls; - w->fix_io_write_calls -= p->last_io_write_calls; - w->fix_io_storage_bytes_read -= p->last_io_storage_bytes_read; - w->fix_io_storage_bytes_written -= p->last_io_storage_bytes_written; - w->fix_io_cancelled_write_bytes -= p->last_io_cancelled_write_bytes; - - // --- - - o->fix_cutime += p->last_cutime; - o->fix_cstime += p->last_cstime; - o->fix_cminflt += p->last_cminflt; - o->fix_cmajflt += p->last_cmajflt; - - o->fix_utime += p->last_utime; - o->fix_stime += p->last_stime; - o->fix_minflt += p->last_minflt; - o->fix_majflt += p->last_majflt; - - o->fix_io_logical_bytes_read += p->last_io_logical_bytes_read; - o->fix_io_logical_bytes_written += p->last_io_logical_bytes_written; - o->fix_io_read_calls += p->last_io_read_calls; - o->fix_io_write_calls += p->last_io_write_calls; - o->fix_io_storage_bytes_read += p->last_io_storage_bytes_read; - o->fix_io_storage_bytes_written += p->last_io_storage_bytes_written; - o->fix_io_cancelled_write_bytes += p->last_io_cancelled_write_bytes; - } - } - else { - // if(o) fprintf(stderr, "apps.plugin: \t\tpid %d (%s) is not updated by OLD target %s (%s) is present.\n", p->pid, p->comm, o->id, o->name); - - // since the process has exited, the user - // will see a drop in our charts, because the incremental - // values of this process will not be there - - // add them to the fix_* values and they will be added to - // the reported values, so that the report goes steady - w->fix_minflt += p->minflt; - w->fix_majflt += p->majflt; - w->fix_utime += p->utime; - w->fix_stime += p->stime; - w->fix_cminflt += p->cminflt; - w->fix_cmajflt += p->cmajflt; - w->fix_cutime += p->cutime; - w->fix_cstime += p->cstime; - - w->fix_io_logical_bytes_read += p->io_logical_bytes_read; - w->fix_io_logical_bytes_written += p->io_logical_bytes_written; - w->fix_io_read_calls += p->io_read_calls; - w->fix_io_write_calls += p->io_write_calls; - w->fix_io_storage_bytes_read += p->io_storage_bytes_read; - w->fix_io_storage_bytes_written += p->io_storage_bytes_written; - w->fix_io_cancelled_write_bytes += p->io_cancelled_write_bytes; - } - + (void)o; + + if(unlikely(!w->fds)) + w->fds = callocz(sizeof(int), (size_t) all_files_size); + + if(likely(p->updated)) { + w->cutime += p->cutime; + w->cstime += p->cstime; + w->cgtime += p->cgtime; + w->cminflt += p->cminflt; + w->cmajflt += p->cmajflt; + + w->utime += p->utime; + w->stime += p->stime; + w->gtime += p->gtime; + w->minflt += p->minflt; + w->majflt += p->majflt; + + w->rss += p->rss; + + w->statm_size += p->statm_size; + w->statm_resident += p->statm_resident; + w->statm_share += p->statm_share; + w->statm_text += p->statm_text; + w->statm_lib += p->statm_lib; + w->statm_data += p->statm_data; + w->statm_dirty += p->statm_dirty; + + w->io_logical_bytes_read += p->io_logical_bytes_read; + w->io_logical_bytes_written += p->io_logical_bytes_written; + w->io_read_calls += p->io_read_calls; + w->io_write_calls += p->io_write_calls; + w->io_storage_bytes_read += p->io_storage_bytes_read; + w->io_storage_bytes_written += p->io_storage_bytes_written; + w->io_cancelled_write_bytes += p->io_cancelled_write_bytes; + + w->processes++; + w->num_threads += p->num_threads; + + if(likely(w->fds)) { + int c; + for(c = 0; c < p->fds_size ;c++) { + if(p->fds[c] == 0) continue; + + if(likely(p->fds[c] < all_files_size)) { + if(w->fds) w->fds[p->fds[c]]++; + } + else + error("Invalid fd number %d", p->fds[c]); + } + } + + if(unlikely(debug || w->debug)) + fprintf(stderr, "apps.plugin: \taggregating '%s' pid %d on target '%s' utime=%llu, stime=%llu, gtime=%llu, cutime=%llu, cstime=%llu, cgtime=%llu, minflt=%llu, majflt=%llu, cminflt=%llu, cmajflt=%llu\n", p->comm, p->pid, w->name, p->utime, p->stime, p->gtime, p->cutime, p->cstime, p->cgtime, p->minflt, p->majflt, p->cminflt, p->cmajflt); + } } void count_targets_fds(struct target *root) { - int c; - struct target *w; - - for (w = root; w ; w = w->next) { - if(!w->fds) continue; - - w->openfiles = 0; - w->openpipes = 0; - w->opensockets = 0; - w->openinotifies = 0; - w->openeventfds = 0; - w->opentimerfds = 0; - w->opensignalfds = 0; - w->openeventpolls = 0; - w->openother = 0; - - for(c = 1; c < all_files_size ;c++) { - if(w->fds[c] > 0) - switch(all_files[c].type) { - case FILETYPE_FILE: - w->openfiles++; - break; - - case FILETYPE_PIPE: - w->openpipes++; - break; - - case FILETYPE_SOCKET: - w->opensockets++; - break; - - case FILETYPE_INOTIFY: - w->openinotifies++; - break; - - case FILETYPE_EVENTFD: - w->openeventfds++; - break; - - case FILETYPE_TIMERFD: - w->opentimerfds++; - break; - - case FILETYPE_SIGNALFD: - w->opensignalfds++; - break; - - case FILETYPE_EVENTPOLL: - w->openeventpolls++; - break; - - default: - w->openother++; - } - } - - free(w->fds); - w->fds = NULL; - } + int c; + struct target *w; + + for (w = root; w ; w = w->next) { + if(!w->fds) continue; + + w->openfiles = 0; + w->openpipes = 0; + w->opensockets = 0; + w->openinotifies = 0; + w->openeventfds = 0; + w->opentimerfds = 0; + w->opensignalfds = 0; + w->openeventpolls = 0; + w->openother = 0; + + for(c = 1; c < all_files_size ;c++) { + if(w->fds[c] > 0) + switch(all_files[c].type) { + case FILETYPE_FILE: + w->openfiles++; + break; + + case FILETYPE_PIPE: + w->openpipes++; + break; + + case FILETYPE_SOCKET: + w->opensockets++; + break; + + case FILETYPE_INOTIFY: + w->openinotifies++; + break; + + case FILETYPE_EVENTFD: + w->openeventfds++; + break; + + case FILETYPE_TIMERFD: + w->opentimerfds++; + break; + + case FILETYPE_SIGNALFD: + w->opensignalfds++; + break; + + case FILETYPE_EVENTPOLL: + w->openeventpolls++; + break; + + default: + w->openother++; + } + } + + freez(w->fds); + w->fds = NULL; + } } -void calculate_netdata_statistics(void) -{ - link_all_processes_to_their_parents(); - apply_apps_groups_targets_inheritance(); - -#ifdef AGGREGATE_CHILDREN_TO_PARENTS - aggregate_children_to_parents(); -#endif /* AGGREGATE_CHILDREN_TO_PARENTS */ - - zero_all_targets(users_root_target); - zero_all_targets(groups_root_target); - apps_groups_targets = zero_all_targets(apps_groups_root_target); - -#ifdef AGGREGATE_CHILDREN_TO_PARENTS - if(debug) - debug_childrens_aggregations(0, 1); -#endif /* AGGREGATE_CHILDREN_TO_PARENTS */ - - // this has to be done, before the cleanup - struct pid_stat *p = NULL; - struct target *w = NULL, *o = NULL; - - // concentrate everything on the apps_groups_targets - for(p = root_of_pids; p ; p = p->next) { - - // -------------------------------------------------------------------- - // apps_groups targets - if(likely(p->target)) - aggregate_pid_on_target(p->target, p, NULL); - else - error("pid %d %s was left without a target!", p->pid, p->comm); - - - // -------------------------------------------------------------------- - // user targets - o = p->user_target; - if(likely(p->user_target && p->user_target->uid == p->uid)) - w = p->user_target; - else { - if(unlikely(debug && p->user_target)) - fprintf(stderr, "apps.plugin: \t\tpid %d (%s) switched user from %d (%s) to %d.\n", p->pid, p->comm, p->user_target->uid, p->user_target->name, p->uid); - - w = p->user_target = get_users_target(p->uid); - } - - if(likely(w)) - aggregate_pid_on_target(w, p, o); - else - error("pid %d %s was left without a user target!", p->pid, p->comm); - - - // -------------------------------------------------------------------- - // group targets - o = p->group_target; - if(likely(p->group_target && p->group_target->gid == p->gid)) - w = p->group_target; - else { - if(unlikely(debug && p->group_target)) - fprintf(stderr, "apps.plugin: \t\tpid %d (%s) switched group from %d (%s) to %d.\n", p->pid, p->comm, p->group_target->gid, p->group_target->name, p->gid); - - w = p->group_target = get_groups_target(p->gid); - } - - if(likely(w)) - aggregate_pid_on_target(w, p, o); - else - error("pid %d %s was left without a group target!", p->pid, p->comm); - - } - - count_targets_fds(apps_groups_root_target); - count_targets_fds(users_root_target); - count_targets_fds(groups_root_target); - - cleanup_non_existing_pids(); -} +void calculate_netdata_statistics(void) { + apply_apps_groups_targets_inheritance(); -// ---------------------------------------------------------------------------- -// update chart dimensions + zero_all_targets(users_root_target); + zero_all_targets(groups_root_target); + apps_groups_targets = zero_all_targets(apps_groups_root_target); -unsigned long long send_resource_usage_to_netdata() { - static struct timeval last = { 0, 0 }; - static struct rusage me_last; - - struct timeval now; - struct rusage me; - - unsigned long long usec; - unsigned long long cpuuser; - unsigned long long cpusyst; - - if(!last.tv_sec) { - gettimeofday(&last, NULL); - getrusage(RUSAGE_SELF, &me_last); - - // the first time, give a zero to allow - // netdata calibrate to the current time - // usec = update_every * 1000000ULL; - usec = 0ULL; - cpuuser = 0; - cpusyst = 0; - } - else { - gettimeofday(&now, NULL); - getrusage(RUSAGE_SELF, &me); - - usec = usecdiff(&now, &last); - cpuuser = me.ru_utime.tv_sec * 1000000ULL + me.ru_utime.tv_usec; - cpusyst = me.ru_stime.tv_sec * 1000000ULL + me.ru_stime.tv_usec; - - bcopy(&now, &last, sizeof(struct timeval)); - bcopy(&me, &me_last, sizeof(struct rusage)); - } - - fprintf(stdout, "BEGIN netdata.apps_cpu %llu\n", usec); - fprintf(stdout, "SET user = %llu\n", cpuuser); - fprintf(stdout, "SET system = %llu\n", cpusyst); - fprintf(stdout, "END\n"); - - fprintf(stdout, "BEGIN netdata.apps_files %llu\n", usec); - fprintf(stdout, "SET files = %llu\n", file_counter); - fprintf(stdout, "SET pids = %ld\n", all_pids_count); - fprintf(stdout, "SET fds = %d\n", all_files_len); - fprintf(stdout, "SET targets = %ld\n", apps_groups_targets); - fprintf(stdout, "END\n"); - - return usec; -} - -void send_collected_data_to_netdata(struct target *root, const char *type, unsigned long long usec) -{ - struct target *w; - - fprintf(stdout, "BEGIN %s.cpu %llu\n", type, usec); - for (w = root; w ; w = w->next) { - if(w->target || (!w->processes && !w->exposed)) continue; - - fprintf(stdout, "SET %s = %llu\n", w->name, w->utime + w->stime + w->fix_utime + w->fix_stime); - } - fprintf(stdout, "END\n"); - - fprintf(stdout, "BEGIN %s.cpu_user %llu\n", type, usec); - for (w = root; w ; w = w->next) { - if(w->target || (!w->processes && !w->exposed)) continue; - - fprintf(stdout, "SET %s = %llu\n", w->name, w->utime + w->fix_utime); - } - fprintf(stdout, "END\n"); - - fprintf(stdout, "BEGIN %s.cpu_system %llu\n", type, usec); - for (w = root; w ; w = w->next) { - if(w->target || (!w->processes && !w->exposed)) continue; - - fprintf(stdout, "SET %s = %llu\n", w->name, w->stime + w->fix_stime); - } - fprintf(stdout, "END\n"); - - fprintf(stdout, "BEGIN %s.threads %llu\n", type, usec); - for (w = root; w ; w = w->next) { - if(w->target || (!w->processes && !w->exposed)) continue; + // this has to be done, before the cleanup + struct pid_stat *p = NULL; + struct target *w = NULL, *o = NULL; - fprintf(stdout, "SET %s = %llu\n", w->name, w->num_threads); - } - fprintf(stdout, "END\n"); + // concentrate everything on the apps_groups_targets + for(p = root_of_pids; p ; p = p->next) { - fprintf(stdout, "BEGIN %s.processes %llu\n", type, usec); - for (w = root; w ; w = w->next) { - if(w->target || (!w->processes && !w->exposed)) continue; + // -------------------------------------------------------------------- + // apps_groups targets + if(likely(p->target)) + aggregate_pid_on_target(p->target, p, NULL); + else + error("pid %d %s was left without a target!", p->pid, p->comm); - fprintf(stdout, "SET %s = %lu\n", w->name, w->processes); - } - fprintf(stdout, "END\n"); - fprintf(stdout, "BEGIN %s.mem %llu\n", type, usec); - for (w = root; w ; w = w->next) { - if(w->target || (!w->processes && !w->exposed)) continue; + // -------------------------------------------------------------------- + // user targets + o = p->user_target; + if(likely(p->user_target && p->user_target->uid == p->uid)) + w = p->user_target; + else { + if(unlikely(debug && p->user_target)) + fprintf(stderr, "apps.plugin: \t\tpid %d (%s) switched user from %u (%s) to %u.\n", p->pid, p->comm, p->user_target->uid, p->user_target->name, p->uid); - fprintf(stdout, "SET %s = %lld\n", w->name, (long long)w->statm_resident - (long long)w->statm_share); - } - fprintf(stdout, "END\n"); + w = p->user_target = get_users_target(p->uid); + } - fprintf(stdout, "BEGIN %s.minor_faults %llu\n", type, usec); - for (w = root; w ; w = w->next) { - if(w->target || (!w->processes && !w->exposed)) continue; + if(likely(w)) + aggregate_pid_on_target(w, p, o); + else + error("pid %d %s was left without a user target!", p->pid, p->comm); - fprintf(stdout, "SET %s = %llu\n", w->name, w->minflt + w->fix_minflt); - } - fprintf(stdout, "END\n"); - fprintf(stdout, "BEGIN %s.major_faults %llu\n", type, usec); - for (w = root; w ; w = w->next) { - if(w->target || (!w->processes && !w->exposed)) continue; + // -------------------------------------------------------------------- + // group targets + o = p->group_target; + if(likely(p->group_target && p->group_target->gid == p->gid)) + w = p->group_target; + else { + if(unlikely(debug && p->group_target)) + fprintf(stderr, "apps.plugin: \t\tpid %d (%s) switched group from %u (%s) to %u.\n", p->pid, p->comm, p->group_target->gid, p->group_target->name, p->gid); - fprintf(stdout, "SET %s = %llu\n", w->name, w->majflt + w->fix_majflt); - } - fprintf(stdout, "END\n"); + w = p->group_target = get_groups_target(p->gid); + } - fprintf(stdout, "BEGIN %s.lreads %llu\n", type, usec); - for (w = root; w ; w = w->next) { - if(w->target || (!w->processes && !w->exposed)) continue; + if(likely(w)) + aggregate_pid_on_target(w, p, o); + else + error("pid %d %s was left without a group target!", p->pid, p->comm); - fprintf(stdout, "SET %s = %llu\n", w->name, w->io_logical_bytes_read + w->fix_io_logical_bytes_read); - } - fprintf(stdout, "END\n"); + } - fprintf(stdout, "BEGIN %s.lwrites %llu\n", type, usec); - for (w = root; w ; w = w->next) { - if(w->target || (!w->processes && !w->exposed)) continue; + count_targets_fds(apps_groups_root_target); + count_targets_fds(users_root_target); + count_targets_fds(groups_root_target); - fprintf(stdout, "SET %s = %llu\n", w->name, w->io_logical_bytes_written + w->fix_io_logical_bytes_written); - } - fprintf(stdout, "END\n"); - - fprintf(stdout, "BEGIN %s.preads %llu\n", type, usec); - for (w = root; w ; w = w->next) { - if(w->target || (!w->processes && !w->exposed)) continue; - - fprintf(stdout, "SET %s = %llu\n", w->name, w->io_storage_bytes_read + w->fix_io_storage_bytes_read); - } - fprintf(stdout, "END\n"); - - fprintf(stdout, "BEGIN %s.pwrites %llu\n", type, usec); - for (w = root; w ; w = w->next) { - if(w->target || (!w->processes && !w->exposed)) continue; + cleanup_exited_pids(); +} - fprintf(stdout, "SET %s = %llu\n", w->name, w->io_storage_bytes_written + w->fix_io_storage_bytes_written); - } - fprintf(stdout, "END\n"); +// ---------------------------------------------------------------------------- +// update chart dimensions - fprintf(stdout, "BEGIN %s.files %llu\n", type, usec); - for (w = root; w ; w = w->next) { - if(w->target || (!w->processes && !w->exposed)) continue; +BUFFER *output = NULL; +int print_calculated_number(char *str, calculated_number value) { (void)str; (void)value; return 0; } + +static inline void send_BEGIN(const char *type, const char *id, unsigned long long usec) { + // fprintf(stdout, "BEGIN %s.%s %llu\n", type, id, usec); + buffer_strcat(output, "BEGIN "); + buffer_strcat(output, type); + buffer_strcat(output, "."); + buffer_strcat(output, id); + buffer_strcat(output, " "); + buffer_print_llu(output, usec); + buffer_strcat(output, "\n"); +} - fprintf(stdout, "SET %s = %llu\n", w->name, w->openfiles); - } - fprintf(stdout, "END\n"); +static inline void send_SET(const char *name, unsigned long long value) { + // fprintf(stdout, "SET %s = %llu\n", name, value); + buffer_strcat(output, "SET "); + buffer_strcat(output, name); + buffer_strcat(output, " = "); + buffer_print_llu(output, value); + buffer_strcat(output, "\n"); +} - fprintf(stdout, "BEGIN %s.sockets %llu\n", type, usec); - for (w = root; w ; w = w->next) { - if(w->target || (!w->processes && !w->exposed)) continue; +static inline void send_END(void) { + // fprintf(stdout, "END\n"); + buffer_strcat(output, "END\n"); +} - fprintf(stdout, "SET %s = %llu\n", w->name, w->opensockets); - } - fprintf(stdout, "END\n"); +double utime_fix_ratio = 1.0, stime_fix_ratio = 1.0, gtime_fix_ratio = 1.0, cutime_fix_ratio = 1.0, cstime_fix_ratio = 1.0, cgtime_fix_ratio = 1.0; +double minflt_fix_ratio = 1.0, majflt_fix_ratio = 1.0, cminflt_fix_ratio = 1.0, cmajflt_fix_ratio = 1.0; - fprintf(stdout, "BEGIN %s.pipes %llu\n", type, usec); - for (w = root; w ; w = w->next) { - if(w->target || (!w->processes && !w->exposed)) continue; +unsigned long long send_resource_usage_to_netdata() { + static struct timeval last = { 0, 0 }; + static struct rusage me_last; + + struct timeval now; + struct rusage me; + + unsigned long long usec; + unsigned long long cpuuser; + unsigned long long cpusyst; + + if(!last.tv_sec) { + gettimeofday(&last, NULL); + getrusage(RUSAGE_SELF, &me_last); + + // the first time, give a zero to allow + // netdata calibrate to the current time + // usec = update_every * 1000000ULL; + usec = 0ULL; + cpuuser = 0; + cpusyst = 0; + } + else { + gettimeofday(&now, NULL); + getrusage(RUSAGE_SELF, &me); + + usec = usec_dt(&now, &last); + cpuuser = me.ru_utime.tv_sec * 1000000ULL + me.ru_utime.tv_usec; + cpusyst = me.ru_stime.tv_sec * 1000000ULL + me.ru_stime.tv_usec; + + bcopy(&now, &last, sizeof(struct timeval)); + bcopy(&me, &me_last, sizeof(struct rusage)); + } + + buffer_sprintf(output, + "BEGIN netdata.apps_cpu %llu\n" + "SET user = %llu\n" + "SET system = %llu\n" + "END\n" + "BEGIN netdata.apps_files %llu\n" + "SET files = %llu\n" + "SET pids = %ld\n" + "SET fds = %d\n" + "SET targets = %ld\n" + "END\n" + "BEGIN netdata.apps_fix %llu\n" + "SET utime = %llu\n" + "SET stime = %llu\n" + "SET gtime = %llu\n" + "SET minflt = %llu\n" + "SET majflt = %llu\n" + "END\n" + , usec + , cpuuser + , cpusyst + , usec + , file_counter + , all_pids_count + , all_files_len + , apps_groups_targets + , usec + , (unsigned long long)(utime_fix_ratio * 100 * RATES_DETAIL) + , (unsigned long long)(stime_fix_ratio * 100 * RATES_DETAIL) + , (unsigned long long)(gtime_fix_ratio * 100 * RATES_DETAIL) + , (unsigned long long)(minflt_fix_ratio * 100 * RATES_DETAIL) + , (unsigned long long)(majflt_fix_ratio * 100 * RATES_DETAIL) + ); + + if(include_exited_childs) + buffer_sprintf(output, + "BEGIN netdata.apps_children_fix %llu\n" + "SET cutime = %llu\n" + "SET cstime = %llu\n" + "SET cgtime = %llu\n" + "SET cminflt = %llu\n" + "SET cmajflt = %llu\n" + "END\n" + , usec + , (unsigned long long)(cutime_fix_ratio * 100 * RATES_DETAIL) + , (unsigned long long)(cstime_fix_ratio * 100 * RATES_DETAIL) + , (unsigned long long)(cgtime_fix_ratio * 100 * RATES_DETAIL) + , (unsigned long long)(cminflt_fix_ratio * 100 * RATES_DETAIL) + , (unsigned long long)(cmajflt_fix_ratio * 100 * RATES_DETAIL) + ); + + return usec; +} - fprintf(stdout, "SET %s = %llu\n", w->name, w->openpipes); - } - fprintf(stdout, "END\n"); +void normalize_data(struct target *root) { + struct target *w; + + // childs processing introduces spikes + // here we try to eliminate them by disabling childs processing either for specific dimensions + // or entirely. Of course, either way, we disable it just a single iteration. + + unsigned long long max = processors * hz * RATES_DETAIL; + unsigned long long utime = 0, cutime = 0, stime = 0, cstime = 0, gtime = 0, cgtime = 0, minflt = 0, cminflt = 0, majflt = 0, cmajflt = 0; + + if(global_utime > max) global_utime = max; + if(global_stime > max) global_stime = max; + if(global_gtime > max) global_gtime = max; + + for(w = root; w ; w = w->next) { + if(w->target || (!w->processes && !w->exposed)) continue; + + utime += w->utime; + stime += w->stime; + gtime += w->gtime; + cutime += w->cutime; + cstime += w->cstime; + cgtime += w->cgtime; + + minflt += w->minflt; + majflt += w->majflt; + cminflt += w->cminflt; + cmajflt += w->cmajflt; + } + + if((global_utime || global_stime || global_gtime) && (utime || stime || gtime)) { + if(global_utime + global_stime + global_gtime > utime + cutime + stime + cstime + gtime + cgtime) { + // everything we collected fits + utime_fix_ratio = + stime_fix_ratio = + gtime_fix_ratio = + cutime_fix_ratio = + cstime_fix_ratio = + cgtime_fix_ratio = 1.0; //(double)(global_utime + global_stime) / (double)(utime + cutime + stime + cstime); + } + else if(global_utime + global_stime > utime + stime) { + // childrens resources are too high + // lower only the children resources + utime_fix_ratio = + stime_fix_ratio = + gtime_fix_ratio = 1.0; + cutime_fix_ratio = + cstime_fix_ratio = + cgtime_fix_ratio = (double)((global_utime + global_stime) - (utime + stime)) / (double)(cutime + cstime); + } + else { + // even running processes are unrealistic + // zero the children resources + // lower the running processes resources + utime_fix_ratio = + stime_fix_ratio = + gtime_fix_ratio = (double)(global_utime + global_stime) / (double)(utime + stime); + cutime_fix_ratio = + cstime_fix_ratio = + cgtime_fix_ratio = 0.0; + } + } + else { + utime_fix_ratio = + stime_fix_ratio = + gtime_fix_ratio = + cutime_fix_ratio = + cstime_fix_ratio = + cgtime_fix_ratio = 0.0; + } + + if(utime_fix_ratio > 1.0) utime_fix_ratio = 1.0; + if(cutime_fix_ratio > 1.0) cutime_fix_ratio = 1.0; + if(stime_fix_ratio > 1.0) stime_fix_ratio = 1.0; + if(cstime_fix_ratio > 1.0) cstime_fix_ratio = 1.0; + if(gtime_fix_ratio > 1.0) gtime_fix_ratio = 1.0; + if(cgtime_fix_ratio > 1.0) cgtime_fix_ratio = 1.0; + + // if(utime_fix_ratio < 0.0) utime_fix_ratio = 0.0; + // if(cutime_fix_ratio < 0.0) cutime_fix_ratio = 0.0; + // if(stime_fix_ratio < 0.0) stime_fix_ratio = 0.0; + // if(cstime_fix_ratio < 0.0) cstime_fix_ratio = 0.0; + // if(gtime_fix_ratio < 0.0) gtime_fix_ratio = 0.0; + // if(cgtime_fix_ratio < 0.0) cgtime_fix_ratio = 0.0; + + // FIXME + // we use cpu time to normalize page faults + // the problem is that to find the proper max values + // for page faults we have to parse /proc/vmstat + // which is quite big to do it again (netdata does it already) + // + // a better solution could be to somehow have netdata + // do this normalization for us + + if(utime || stime || gtime) + majflt_fix_ratio = + minflt_fix_ratio = (double)(utime * utime_fix_ratio + stime * stime_fix_ratio + gtime * gtime_fix_ratio) / (double)(utime + stime + gtime); + else + minflt_fix_ratio = + majflt_fix_ratio = 1.0; + + if(cutime || cstime || cgtime) + cmajflt_fix_ratio = + cminflt_fix_ratio = (double)(cutime * cutime_fix_ratio + cstime * cstime_fix_ratio + cgtime * cgtime_fix_ratio) / (double)(cutime + cstime + cgtime); + else + cminflt_fix_ratio = + cmajflt_fix_ratio = 1.0; + + // the report + + if(unlikely(debug)) { + fprintf(stderr, + "SYSTEM: u=%llu s=%llu g=%llu " + "COLLECTED: u=%llu s=%llu g=%llu cu=%llu cs=%llu cg=%llu " + "DELTA: u=%lld s=%lld g=%lld " + "FIX: u=%0.2f s=%0.2f g=%0.2f cu=%0.2f cs=%0.2f cg=%0.2f " + "FINALLY: u=%llu s=%llu g=%llu cu=%llu cs=%llu cg=%llu " + "\n" + , global_utime + , global_stime + , global_gtime + , utime + , stime + , gtime + , cutime + , cstime + , cgtime + , (long long)utime + (long long)cutime - (long long)global_utime + , (long long)stime + (long long)cstime - (long long)global_stime + , (long long)gtime + (long long)cgtime - (long long)global_gtime + , utime_fix_ratio + , stime_fix_ratio + , gtime_fix_ratio + , cutime_fix_ratio + , cstime_fix_ratio + , cgtime_fix_ratio + , (unsigned long long)(utime * utime_fix_ratio) + , (unsigned long long)(stime * stime_fix_ratio) + , (unsigned long long)(gtime * gtime_fix_ratio) + , (unsigned long long)(cutime * cutime_fix_ratio) + , (unsigned long long)(cstime * cstime_fix_ratio) + , (unsigned long long)(cgtime * cgtime_fix_ratio) + ); + } +} - fflush(stdout); +void send_collected_data_to_netdata(struct target *root, const char *type, unsigned long long usec) { + struct target *w; + + send_BEGIN(type, "cpu", usec); + for (w = root; w ; w = w->next) { + if(unlikely(w->exposed)) + send_SET(w->name, (unsigned long long)(w->utime * utime_fix_ratio) + (unsigned long long)(w->stime * stime_fix_ratio) + (unsigned long long)(w->gtime * gtime_fix_ratio) + (include_exited_childs?((unsigned long long)(w->cutime * cutime_fix_ratio) + (unsigned long long)(w->cstime * cstime_fix_ratio) + (unsigned long long)(w->cgtime * cgtime_fix_ratio)):0ULL)); + } + send_END(); + + send_BEGIN(type, "cpu_user", usec); + for (w = root; w ; w = w->next) { + if(unlikely(w->exposed)) + send_SET(w->name, (unsigned long long)(w->utime * utime_fix_ratio) + (include_exited_childs?((unsigned long long)(w->cutime * cutime_fix_ratio)):0ULL)); + } + send_END(); + + send_BEGIN(type, "cpu_system", usec); + for (w = root; w ; w = w->next) { + if(unlikely(w->exposed)) + send_SET(w->name, (unsigned long long)(w->stime * stime_fix_ratio) + (include_exited_childs?((unsigned long long)(w->cstime * cstime_fix_ratio)):0ULL)); + } + send_END(); + + if(show_guest_time) { + send_BEGIN(type, "cpu_guest", usec); + for (w = root; w ; w = w->next) { + if(unlikely(w->exposed)) + send_SET(w->name, (unsigned long long)(w->gtime * gtime_fix_ratio) + (include_exited_childs?((unsigned long long)(w->cgtime * cgtime_fix_ratio)):0ULL)); + } + send_END(); + } + + send_BEGIN(type, "threads", usec); + for (w = root; w ; w = w->next) { + if(unlikely(w->exposed)) + send_SET(w->name, w->num_threads); + } + send_END(); + + send_BEGIN(type, "processes", usec); + for (w = root; w ; w = w->next) { + if(unlikely(w->exposed)) + send_SET(w->name, w->processes); + } + send_END(); + + send_BEGIN(type, "mem", usec); + for (w = root; w ; w = w->next) { + if(unlikely(w->exposed)) + send_SET(w->name, (w->statm_resident > w->statm_share)?(w->statm_resident - w->statm_share):0ULL); + } + send_END(); + + send_BEGIN(type, "minor_faults", usec); + for (w = root; w ; w = w->next) { + if(unlikely(w->exposed)) + send_SET(w->name, (unsigned long long)(w->minflt * minflt_fix_ratio) + (include_exited_childs?((unsigned long long)(w->cminflt * cminflt_fix_ratio)):0ULL)); + } + send_END(); + + send_BEGIN(type, "major_faults", usec); + for (w = root; w ; w = w->next) { + if(unlikely(w->exposed)) + send_SET(w->name, (unsigned long long)(w->majflt * majflt_fix_ratio) + (include_exited_childs?((unsigned long long)(w->cmajflt * cmajflt_fix_ratio)):0ULL)); + } + send_END(); + + send_BEGIN(type, "lreads", usec); + for (w = root; w ; w = w->next) { + if(unlikely(w->exposed)) + send_SET(w->name, w->io_logical_bytes_read); + } + send_END(); + + send_BEGIN(type, "lwrites", usec); + for (w = root; w ; w = w->next) { + if(unlikely(w->exposed)) + send_SET(w->name, w->io_logical_bytes_written); + } + send_END(); + + send_BEGIN(type, "preads", usec); + for (w = root; w ; w = w->next) { + if(unlikely(w->exposed)) + send_SET(w->name, w->io_storage_bytes_read); + } + send_END(); + + send_BEGIN(type, "pwrites", usec); + for (w = root; w ; w = w->next) { + if(unlikely(w->exposed)) + send_SET(w->name, w->io_storage_bytes_written); + } + send_END(); + + if(enable_file_charts) { + send_BEGIN(type, "files", usec); + for (w = root; w; w = w->next) { + if (unlikely(w->exposed)) + send_SET(w->name, w->openfiles); + } + send_END(); + + send_BEGIN(type, "sockets", usec); + for (w = root; w; w = w->next) { + if (unlikely(w->exposed)) + send_SET(w->name, w->opensockets); + } + send_END(); + + send_BEGIN(type, "pipes", usec); + for (w = root; w; w = w->next) { + if (unlikely(w->exposed)) + send_SET(w->name, w->openpipes); + } + send_END(); + } } @@ -2217,125 +2560,126 @@ void send_collected_data_to_netdata(struct target *root, const char *type, unsig void send_charts_updates_to_netdata(struct target *root, const char *type, const char *title) { - struct target *w; - int newly_added = 0; - - for(w = root ; w ; w = w->next) - if(!w->exposed && w->processes) { - newly_added++; - w->exposed = 1; - if(debug || w->debug) fprintf(stderr, "apps.plugin: %s just added - regenerating charts.\n", w->name); - } - - // nothing more to show - if(!newly_added) return; - - // we have something new to show - // update the charts - fprintf(stdout, "CHART %s.cpu '' '%s CPU Time (%ld%% = %ld core%s)' 'cpu time %%' cpu %s.cpu stacked 20001 %d\n", type, title, (processors * 100), processors, (processors>1)?"s":"", type, update_every); - for (w = root; w ; w = w->next) { - if(w->target || (!w->processes && !w->exposed)) continue; - - fprintf(stdout, "DIMENSION %s '' incremental 100 %u %s\n", w->name, hz, w->hidden ? "hidden,noreset" : "noreset"); - } - - fprintf(stdout, "CHART %s.mem '' '%s Dedicated Memory (w/o shared)' 'MB' mem %s.mem stacked 20003 %d\n", type, title, type, update_every); - for (w = root; w ; w = w->next) { - if(w->target || (!w->processes && !w->exposed)) continue; - - fprintf(stdout, "DIMENSION %s '' absolute %ld %ld noreset\n", w->name, sysconf(_SC_PAGESIZE), 1024L*1024L); - } - - fprintf(stdout, "CHART %s.threads '' '%s Threads' 'threads' processes %s.threads stacked 20005 %d\n", type, title, type, update_every); - for (w = root; w ; w = w->next) { - if(w->target || (!w->processes && !w->exposed)) continue; - - fprintf(stdout, "DIMENSION %s '' absolute 1 1 noreset\n", w->name); - } - - fprintf(stdout, "CHART %s.processes '' '%s Processes' 'processes' processes %s.processes stacked 20004 %d\n", type, title, type, update_every); - for (w = root; w ; w = w->next) { - if(w->target || (!w->processes && !w->exposed)) continue; - - fprintf(stdout, "DIMENSION %s '' absolute 1 1 noreset\n", w->name); - } - - fprintf(stdout, "CHART %s.cpu_user '' '%s CPU User Time (%ld%% = %ld core%s)' 'cpu time %%' cpu %s.cpu_user stacked 20020 %d\n", type, title, (processors * 100), processors, (processors>1)?"s":"", type, update_every); - for (w = root; w ; w = w->next) { - if(w->target || (!w->processes && !w->exposed)) continue; - - fprintf(stdout, "DIMENSION %s '' incremental 100 %ld noreset\n", w->name, hz * processors); - } - - fprintf(stdout, "CHART %s.cpu_system '' '%s CPU System Time (%ld%% = %ld core%s)' 'cpu time %%' cpu %s.cpu_system stacked 20021 %d\n", type, title, (processors * 100), processors, (processors>1)?"s":"", type, update_every); - for (w = root; w ; w = w->next) { - if(w->target || (!w->processes && !w->exposed)) continue; - - fprintf(stdout, "DIMENSION %s '' incremental 100 %ld noreset\n", w->name, hz * processors); - } - - fprintf(stdout, "CHART %s.major_faults '' '%s Major Page Faults (swap read)' 'page faults/s' swap %s.major_faults stacked 20010 %d\n", type, title, type, update_every); - for (w = root; w ; w = w->next) { - if(w->target || (!w->processes && !w->exposed)) continue; - - fprintf(stdout, "DIMENSION %s '' incremental 1 1 noreset\n", w->name); - } - - fprintf(stdout, "CHART %s.minor_faults '' '%s Minor Page Faults' 'page faults/s' mem %s.minor_faults stacked 20011 %d\n", type, title, type, update_every); - for (w = root; w ; w = w->next) { - if(w->target || (!w->processes && !w->exposed)) continue; - - fprintf(stdout, "DIMENSION %s '' incremental 1 1 noreset\n", w->name); - } - - fprintf(stdout, "CHART %s.lreads '' '%s Disk Logical Reads' 'kilobytes/s' disk %s.lreads stacked 20042 %d\n", type, title, type, update_every); - for (w = root; w ; w = w->next) { - if(w->target || (!w->processes && !w->exposed)) continue; - - fprintf(stdout, "DIMENSION %s '' incremental 1 %d noreset\n", w->name, 1024); - } - - fprintf(stdout, "CHART %s.lwrites '' '%s I/O Logical Writes' 'kilobytes/s' disk %s.lwrites stacked 20042 %d\n", type, title, type, update_every); - for (w = root; w ; w = w->next) { - if(w->target || (!w->processes && !w->exposed)) continue; - - fprintf(stdout, "DIMENSION %s '' incremental 1 %d noreset\n", w->name, 1024); - } - - fprintf(stdout, "CHART %s.preads '' '%s Disk Reads' 'kilobytes/s' disk %s.preads stacked 20002 %d\n", type, title, type, update_every); - for (w = root; w ; w = w->next) { - if(w->target || (!w->processes && !w->exposed)) continue; - - fprintf(stdout, "DIMENSION %s '' incremental 1 %d noreset\n", w->name, 1024); - } - - fprintf(stdout, "CHART %s.pwrites '' '%s Disk Writes' 'kilobytes/s' disk %s.pwrites stacked 20002 %d\n", type, title, type, update_every); - for (w = root; w ; w = w->next) { - if(w->target || (!w->processes && !w->exposed)) continue; - - fprintf(stdout, "DIMENSION %s '' incremental 1 %d noreset\n", w->name, 1024); - } - - fprintf(stdout, "CHART %s.files '' '%s Open Files' 'open files' disk %s.files stacked 20050 %d\n", type, title, type, update_every); - for (w = root; w ; w = w->next) { - if(w->target || (!w->processes && !w->exposed)) continue; - - fprintf(stdout, "DIMENSION %s '' absolute 1 1 noreset\n", w->name); - } - - fprintf(stdout, "CHART %s.sockets '' '%s Open Sockets' 'open sockets' net %s.sockets stacked 20051 %d\n", type, title, type, update_every); - for (w = root; w ; w = w->next) { - if(w->target || (!w->processes && !w->exposed)) continue; - - fprintf(stdout, "DIMENSION %s '' absolute 1 1 noreset\n", w->name); - } - - fprintf(stdout, "CHART %s.pipes '' '%s Pipes' 'open pipes' processes %s.pipes stacked 20053 %d\n", type, title, type, update_every); - for (w = root; w ; w = w->next) { - if(w->target || (!w->processes && !w->exposed)) continue; - - fprintf(stdout, "DIMENSION %s '' absolute 1 1 noreset\n", w->name); - } + struct target *w; + int newly_added = 0; + + for(w = root ; w ; w = w->next) { + if (w->target) continue; + + if (!w->exposed && w->processes) { + newly_added++; + w->exposed = 1; + if (debug || w->debug) fprintf(stderr, "apps.plugin: %s just added - regenerating charts.\n", w->name); + } + } + + // nothing more to show + if(!newly_added && show_guest_time == show_guest_time_old) return; + + // we have something new to show + // update the charts + buffer_sprintf(output, "CHART %s.cpu '' '%s CPU Time (%d%% = %d core%s)' 'cpu time %%' cpu %s.cpu stacked 20001 %d\n", type, title, (processors * 100), processors, (processors>1)?"s":"", type, update_every); + for (w = root; w ; w = w->next) { + if(unlikely(w->exposed)) + buffer_sprintf(output, "DIMENSION %s '' absolute 1 %llu %s\n", w->name, hz * RATES_DETAIL / 100, w->hidden ? "hidden" : ""); + } + + buffer_sprintf(output, "CHART %s.mem '' '%s Dedicated Memory (w/o shared)' 'MB' mem %s.mem stacked 20003 %d\n", type, title, type, update_every); + for (w = root; w ; w = w->next) { + if(unlikely(w->exposed)) + buffer_sprintf(output, "DIMENSION %s '' absolute %ld %ld\n", w->name, sysconf(_SC_PAGESIZE), 1024L*1024L); + } + + buffer_sprintf(output, "CHART %s.threads '' '%s Threads' 'threads' processes %s.threads stacked 20005 %d\n", type, title, type, update_every); + for (w = root; w ; w = w->next) { + if(unlikely(w->exposed)) + buffer_sprintf(output, "DIMENSION %s '' absolute 1 1\n", w->name); + } + + buffer_sprintf(output, "CHART %s.processes '' '%s Processes' 'processes' processes %s.processes stacked 20004 %d\n", type, title, type, update_every); + for (w = root; w ; w = w->next) { + if(unlikely(w->exposed)) + buffer_sprintf(output, "DIMENSION %s '' absolute 1 1\n", w->name); + } + + buffer_sprintf(output, "CHART %s.cpu_user '' '%s CPU User Time (%d%% = %d core%s)' 'cpu time %%' cpu %s.cpu_user stacked 20020 %d\n", type, title, (processors * 100), processors, (processors>1)?"s":"", type, update_every); + for (w = root; w ; w = w->next) { + if(unlikely(w->exposed)) + buffer_sprintf(output, "DIMENSION %s '' absolute 1 %llu\n", w->name, hz * RATES_DETAIL / 100LLU); + } + + buffer_sprintf(output, "CHART %s.cpu_system '' '%s CPU System Time (%d%% = %d core%s)' 'cpu time %%' cpu %s.cpu_system stacked 20021 %d\n", type, title, (processors * 100), processors, (processors>1)?"s":"", type, update_every); + for (w = root; w ; w = w->next) { + if(unlikely(w->exposed)) + buffer_sprintf(output, "DIMENSION %s '' absolute 1 %llu\n", w->name, hz * RATES_DETAIL / 100LLU); + } + + if(show_guest_time) { + buffer_sprintf(output, "CHART %s.cpu_guest '' '%s CPU Guest Time (%d%% = %d core%s)' 'cpu time %%' cpu %s.cpu_system stacked 20022 %d\n", type, title, (processors * 100), processors, (processors > 1) ? "s" : "", type, update_every); + for (w = root; w; w = w->next) { + if(unlikely(w->exposed)) + buffer_sprintf(output, "DIMENSION %s '' absolute 1 %llu\n", w->name, hz * RATES_DETAIL / 100LLU); + } + } + + buffer_sprintf(output, "CHART %s.major_faults '' '%s Major Page Faults (swap read)' 'page faults/s' swap %s.major_faults stacked 20010 %d\n", type, title, type, update_every); + for (w = root; w ; w = w->next) { + if(unlikely(w->exposed)) + buffer_sprintf(output, "DIMENSION %s '' absolute 1 %llu\n", w->name, RATES_DETAIL); + } + + buffer_sprintf(output, "CHART %s.minor_faults '' '%s Minor Page Faults' 'page faults/s' mem %s.minor_faults stacked 20011 %d\n", type, title, type, update_every); + for (w = root; w ; w = w->next) { + if(unlikely(w->exposed)) + buffer_sprintf(output, "DIMENSION %s '' absolute 1 %llu\n", w->name, RATES_DETAIL); + } + + buffer_sprintf(output, "CHART %s.lreads '' '%s Disk Logical Reads' 'kilobytes/s' disk %s.lreads stacked 20042 %d\n", type, title, type, update_every); + for (w = root; w ; w = w->next) { + if(unlikely(w->exposed)) + buffer_sprintf(output, "DIMENSION %s '' absolute 1 %llu\n", w->name, 1024LLU * RATES_DETAIL); + } + + buffer_sprintf(output, "CHART %s.lwrites '' '%s I/O Logical Writes' 'kilobytes/s' disk %s.lwrites stacked 20042 %d\n", type, title, type, update_every); + for (w = root; w ; w = w->next) { + if(unlikely(w->exposed)) + buffer_sprintf(output, "DIMENSION %s '' absolute 1 %llu\n", w->name, 1024LLU * RATES_DETAIL); + } + + buffer_sprintf(output, "CHART %s.preads '' '%s Disk Reads' 'kilobytes/s' disk %s.preads stacked 20002 %d\n", type, title, type, update_every); + for (w = root; w ; w = w->next) { + if(unlikely(w->exposed)) + buffer_sprintf(output, "DIMENSION %s '' absolute 1 %llu\n", w->name, 1024LLU * RATES_DETAIL); + } + + buffer_sprintf(output, "CHART %s.pwrites '' '%s Disk Writes' 'kilobytes/s' disk %s.pwrites stacked 20002 %d\n", type, title, type, update_every); + for (w = root; w ; w = w->next) { + if(unlikely(w->exposed)) + buffer_sprintf(output, "DIMENSION %s '' absolute 1 %llu\n", w->name, 1024LLU * RATES_DETAIL); + } + + if(enable_file_charts) { + buffer_sprintf(output, "CHART %s.files '' '%s Open Files' 'open files' disk %s.files stacked 20050 %d\n", type, + title, type, update_every); + for (w = root; w; w = w->next) { + if (unlikely(w->exposed)) + buffer_sprintf(output, "DIMENSION %s '' absolute 1 1\n", w->name); + } + + buffer_sprintf(output, "CHART %s.sockets '' '%s Open Sockets' 'open sockets' net %s.sockets stacked 20051 %d\n", + type, title, type, update_every); + for (w = root; w; w = w->next) { + if (unlikely(w->exposed)) + buffer_sprintf(output, "DIMENSION %s '' absolute 1 1\n", w->name); + } + + buffer_sprintf(output, "CHART %s.pipes '' '%s Pipes' 'open pipes' processes %s.pipes stacked 20053 %d\n", type, + title, type, update_every); + for (w = root; w; w = w->next) { + if (unlikely(w->exposed)) + buffer_sprintf(output, "DIMENSION %s '' absolute 1 1\n", w->name); + } + } } @@ -2344,152 +2688,245 @@ void send_charts_updates_to_netdata(struct target *root, const char *type, const void parse_args(int argc, char **argv) { - int i, freq = 0; - char *name = NULL; - - for(i = 1; i < argc; i++) { - if(!freq) { - int n = atoi(argv[i]); - if(n > 0) { - freq = n; - continue; - } - } - - if(strcmp("debug", argv[i]) == 0) { - debug = 1; - debug_flags = 0xffffffff; - continue; - } - - if(!name) { - name = argv[i]; - continue; - } - - error("Cannot understand option %s", argv[i]); - exit(1); - } - - if(freq > 0) update_every = freq; - if(!name) name = "groups"; - - if(read_apps_groups_conf(name)) { - error("Cannot read process groups %s", name); - exit(1); - } + int i, freq = 0; + char *name = NULL; + + for(i = 1; i < argc; i++) { + if(!freq) { + int n = atoi(argv[i]); + if(n > 0) { + freq = n; + continue; + } + } + + if(strcmp("debug", argv[i]) == 0) { + debug = 1; + // debug_flags = 0xffffffff; + continue; + } + + if(strcmp("no-childs", argv[i]) == 0 || strcmp("without-childs", argv[i]) == 0) { + include_exited_childs = 0; + continue; + } + + if(strcmp("with-childs", argv[i]) == 0) { + include_exited_childs = 1; + continue; + } + + if(strcmp("with-guest", argv[i]) == 0) { + enable_guest_charts = 1; + continue; + } + + if(strcmp("no-guest", argv[i]) == 0 || strcmp("without-guest", argv[i]) == 0) { + enable_guest_charts = 0; + continue; + } + + if(strcmp("with-files", argv[i]) == 0) { + enable_file_charts = 1; + continue; + } + + if(strcmp("no-files", argv[i]) == 0 || strcmp("without-files", argv[i]) == 0) { + enable_file_charts = 0; + continue; + } + + if(strcmp("-h", argv[i]) == 0 || strcmp("--help", argv[i]) == 0) { + fprintf(stderr, + "apps.plugin\n" + "(C) 2016 Costa Tsaousis" + "GPL v3+\n" + "This program is a data collector plugin for netdata.\n" + "\n" + "Valid command line options:\n" + "\n" + "SECONDS set the data collection frequency\n" + "\n" + "debug enable debugging (lot of output)\n" + "\n" + "with-childs\n" + "without-childs enable / disable aggregating exited\n" + " children resources into parents\n" + " (default is enabled)\n" + "\n" + "with-guest\n" + "without-guest enable / disable reporting guest charts\n" + " (default is disabled)\n" + "\n" + "with-files\n" + "without-files enable / disable reporting files, sockets, pipes\n" + " (default is enabled)\n" + "\n" + "NAME read apps_NAME.conf instead of\n" + " apps_groups.conf\n" + " (default NAME=groups)\n" + ); + exit(1); + } + + if(!name) { + name = argv[i]; + continue; + } + + error("Cannot understand option %s", argv[i]); + exit(1); + } + + if(freq > 0) update_every = freq; + if(!name) name = "groups"; + + if(read_apps_groups_conf(name)) { + error("Cannot read process groups %s", name); + exit(1); + } } int main(int argc, char **argv) { - // debug_flags = D_PROCFILE; + // debug_flags = D_PROCFILE; + + // set the name for logging + program_name = "apps.plugin"; - // set the name for logging - program_name = "apps.plugin"; + info("started on pid %d", getpid()); - // disable syslog for apps.plugin - error_log_syslog = 0; + // disable syslog for apps.plugin + error_log_syslog = 0; - // set errors flood protection to 100 logs per hour - error_log_errors_per_period = 100; - error_log_throttle_period = 3600; + // set errors flood protection to 100 logs per hour + error_log_errors_per_period = 100; + error_log_throttle_period = 3600; - host_prefix = getenv("NETDATA_HOST_PREFIX"); - if(host_prefix == NULL) { - info("NETDATA_HOST_PREFIX is not passed from netdata"); - host_prefix = ""; - } - else info("Found NETDATA_HOST_PREFIX='%s'", host_prefix); + host_prefix = getenv("NETDATA_HOST_PREFIX"); + if(host_prefix == NULL) { + // info("NETDATA_HOST_PREFIX is not passed from netdata"); + host_prefix = ""; + } + // else info("Found NETDATA_HOST_PREFIX='%s'", host_prefix); - config_dir = getenv("NETDATA_CONFIG_DIR"); - if(config_dir == NULL) { - info("NETDATA_CONFIG_DIR is not passed from netdata"); - config_dir = CONFIG_DIR; - } - else info("Found NETDATA_CONFIG_DIR='%s'", config_dir); + config_dir = getenv("NETDATA_CONFIG_DIR"); + if(config_dir == NULL) { + // info("NETDATA_CONFIG_DIR is not passed from netdata"); + config_dir = CONFIG_DIR; + } + // else info("Found NETDATA_CONFIG_DIR='%s'", config_dir); #ifdef NETDATA_INTERNAL_CHECKS - if(debug_flags != 0) { - struct rlimit rl = { RLIM_INFINITY, RLIM_INFINITY }; - if(setrlimit(RLIMIT_CORE, &rl) != 0) - info("Cannot request unlimited core dumps for debugging... Proceeding anyway..."); - prctl(PR_SET_DUMPABLE, 1, 0, 0, 0); - } + if(debug_flags != 0) { + struct rlimit rl = { RLIM_INFINITY, RLIM_INFINITY }; + if(setrlimit(RLIMIT_CORE, &rl) != 0) + info("Cannot request unlimited core dumps for debugging... Proceeding anyway..."); + prctl(PR_SET_DUMPABLE, 1, 0, 0, 0); + } #endif /* NETDATA_INTERNAL_CHECKS */ - info("starting..."); - - procfile_adaptive_initial_allocation = 1; - - time_t started_t = time(NULL); - time_t current_t; - get_HZ(); - pid_max = get_system_pid_max(); - processors = get_system_cpus(); - - parse_args(argc, argv); - - all_pids = calloc(sizeof(struct pid_stat *), (size_t) pid_max); - if(!all_pids) { - error("Cannot allocate %lu bytes of memory.", sizeof(struct pid_stat *) * pid_max); - printf("DISABLE\n"); - exit(1); - } - - fprintf(stdout, "CHART netdata.apps_cpu '' 'Apps Plugin CPU' 'milliseconds/s' apps.plugin netdata.apps_cpu stacked 140000 %1$d\n" - "DIMENSION user '' incremental 1 1000\n" - "DIMENSION system '' incremental 1 1000\n" - "CHART netdata.apps_files '' 'Apps Plugin Files' 'files/s' apps.plugin netdata.apps_files line 140001 %1$d\n" - "DIMENSION files '' incremental 1 1\n" - "DIMENSION pids '' absolute 1 1\n" - "DIMENSION fds '' absolute 1 1\n" - "DIMENSION targets '' absolute 1 1\n", update_every); + procfile_adaptive_initial_allocation = 1; + + time_t started_t = time(NULL); + time_t current_t; + get_HZ(); + pid_max = get_system_pid_max(); + processors = get_system_cpus(); + + parse_args(argc, argv); + + all_pids_sortlist = callocz(sizeof(pid_t), (size_t)pid_max); + all_pids = callocz(sizeof(struct pid_stat *), (size_t) pid_max); + + output = buffer_create(1024); + buffer_sprintf(output, + "CHART netdata.apps_cpu '' 'Apps Plugin CPU' 'milliseconds/s' apps.plugin netdata.apps_cpu stacked 140000 %1$d\n" + "DIMENSION user '' incremental 1 1000\n" + "DIMENSION system '' incremental 1 1000\n" + "CHART netdata.apps_files '' 'Apps Plugin Files' 'files/s' apps.plugin netdata.apps_files line 140001 %1$d\n" + "DIMENSION files '' incremental 1 1\n" + "DIMENSION pids '' absolute 1 1\n" + "DIMENSION fds '' absolute 1 1\n" + "DIMENSION targets '' absolute 1 1\n" + "CHART netdata.apps_fix '' 'Apps Plugin Normalization Ratios' 'percentage' apps.plugin netdata.apps_fix line 140002 %1$d\n" + "DIMENSION utime '' absolute 1 %2$llu\n" + "DIMENSION stime '' absolute 1 %2$llu\n" + "DIMENSION gtime '' absolute 1 %2$llu\n" + "DIMENSION minflt '' absolute 1 %2$llu\n" + "DIMENSION majflt '' absolute 1 %2$llu\n" + , update_every + , RATES_DETAIL + ); + + if(include_exited_childs) + buffer_sprintf(output, + "CHART netdata.apps_children_fix '' 'Apps Plugin Exited Children Normalization Ratios' 'percentage' apps.plugin netdata.apps_children_fix line 140003 %1$d\n" + "DIMENSION cutime '' absolute 1 %2$llu\n" + "DIMENSION cstime '' absolute 1 %2$llu\n" + "DIMENSION cgtime '' absolute 1 %2$llu\n" + "DIMENSION cminflt '' absolute 1 %2$llu\n" + "DIMENSION cmajflt '' absolute 1 %2$llu\n" + , update_every + , RATES_DETAIL + ); #ifndef PROFILING_MODE - unsigned long long sunext = (time(NULL) - (time(NULL) % update_every) + update_every) * 1000000ULL; - unsigned long long sunow; + unsigned long long sunext = (time(NULL) - (time(NULL) % update_every) + update_every) * 1000000ULL; + unsigned long long sunow; #endif /* PROFILING_MODE */ - unsigned long long counter = 1; - for(;1; counter++) { + global_iterations_counter = 1; + for(;1; global_iterations_counter++) { #ifndef PROFILING_MODE - // delay until it is our time to run - while((sunow = timems()) < sunext) - usleep((useconds_t)(sunext - sunow)); + // delay until it is our time to run + while((sunow = time_usec()) < sunext) + sleep_usec(sunext - sunow); - // find the next time we need to run - while(timems() > sunext) - sunext += update_every * 1000000ULL; + // find the next time we need to run + while(time_usec() > sunext) + sunext += update_every * 1000000ULL; #endif /* PROFILING_MODE */ - if(!collect_data_for_all_processes_from_proc()) { - error("Cannot collect /proc data for running processes. Disabling apps.plugin..."); - printf("DISABLE\n"); - exit(1); - } + if(!collect_data_for_all_processes_from_proc()) { + error("Cannot collect /proc data for running processes. Disabling apps.plugin..."); + printf("DISABLE\n"); + exit(1); + } + + calculate_netdata_statistics(); + normalize_data(apps_groups_root_target); + + unsigned long long dt = send_resource_usage_to_netdata(); + + // this is smart enough to show only newly added apps, when needed + send_charts_updates_to_netdata(apps_groups_root_target, "apps", "Apps"); + send_charts_updates_to_netdata(users_root_target, "users", "Users"); + send_charts_updates_to_netdata(groups_root_target, "groups", "User Groups"); - calculate_netdata_statistics(); + send_collected_data_to_netdata(apps_groups_root_target, "apps", dt); + send_collected_data_to_netdata(users_root_target, "users", dt); + send_collected_data_to_netdata(groups_root_target, "groups", dt); - unsigned long long dt = send_resource_usage_to_netdata(); + show_guest_time_old = show_guest_time; - // this is smart enough to show only newly added apps, when needed - send_charts_updates_to_netdata(apps_groups_root_target, "apps", "Apps"); - send_charts_updates_to_netdata(users_root_target, "users", "Users"); - send_charts_updates_to_netdata(groups_root_target, "groups", "User Groups"); + //if(puts(buffer_tostring(output)) == EOF) + if(write(STDOUT_FILENO, buffer_tostring(output), buffer_strlen(output)) == -1) + fatal("Cannot send chart values to netdata."); - send_collected_data_to_netdata(apps_groups_root_target, "apps", dt); - send_collected_data_to_netdata(users_root_target, "users", dt); - send_collected_data_to_netdata(groups_root_target, "groups", dt); + // fflush(stdout); + buffer_flush(output); - if(debug) fprintf(stderr, "apps.plugin: done Loop No %llu\n", counter); + if(unlikely(debug)) + fprintf(stderr, "apps.plugin: done Loop No %llu\n", global_iterations_counter); - current_t = time(NULL); + current_t = time(NULL); #ifndef PROFILING_MODE - // restart check (14400 seconds) - if(current_t - started_t > 14400) exit(0); + // restart check (14400 seconds) + if(current_t - started_t > 14400) exit(0); #else - if(current_t - started_t > 10) exit(0); + if(current_t - started_t > 10) exit(0); #endif /* PROFILING_MODE */ - } + } } @@ -1,350 +1,311 @@ +#include "common.h" + +/* ------------------------------------------------------------------------- */ /* - * ANSI C Library for maintainance of AVL Balanced Trees - * - * ref.: - * G. M. Adelson-Velskij & E. M. Landis - * Doklady Akad. Nauk SSSR 146 (1962), 263-266 - * - * see also: - * D. E. Knuth: The Art of Computer Programming Vol.3 (Sorting and Searching) + * avl_insert(), avl_remove() and avl_search() + * are adaptations (by Costa Tsaousis) of the AVL algorithm found in libavl + * v2.0.3, so that they do not use any memory allocations and their memory + * footprint is optimized (by eliminating non-necessary data members). * - * (C) 2000 Daniel Nagy, Budapest University of Technology and Economics - * Released under GNU General Public License (GPL) version 2 - * - */ + * libavl - library for manipulation of binary trees. + * Copyright (C) 1998, 1999, 2000, 2001, 2002, 2004 Free Software + * Foundation, Inc. + * GNU Lesser General Public License +*/ -#ifdef HAVE_CONFIG_H -#include <config.h> -#endif -#include "avl.h" -#include "log.h" -/* Swing to the left - * Warning: no balance maintainance - */ -void avl_swl(avl** root) { - avl* a = *root; - avl* b = a->right; - *root = b; - a->right = b->left; - b->left = a; -} +/* Search |tree| for an item matching |item|, and return it if found. + Otherwise return |NULL|. */ +avl *avl_search(avl_tree *tree, avl *item) { + avl *p; -/* Swing to the right - * Warning: no balance maintainance - */ -void avl_swr(avl** root) { - avl* a = *root; - avl* b = a->left; - *root = b; - a->left = b->right; - b->right = a; -} + // assert (tree != NULL && item != NULL); -/* Balance maintainance after especially nasty swings - */ -void avl_nasty(avl* root) { - switch (root->balance) { - case -1: - root->left->balance = 0; - root->right->balance = 1; - break; - case 1: - root->left->balance = -1; - root->right->balance = 0; - break; - case 0: - root->left->balance = 0; - root->right->balance = 0; - } - root->balance = 0; -} + for (p = tree->root; p != NULL; ) { + int cmp = tree->compar(item, p); -/* Public methods */ + if (cmp < 0) + p = p->avl_link[0]; + else if (cmp > 0) + p = p->avl_link[1]; + else /* |cmp == 0| */ + return p; + } -/* Insert element a into the AVL tree t - * returns 1 if the depth of the tree has grown - * Warning: do not insert elements already present - */ -int avl_insert(avl_tree* t, avl* a) { - /* initialize */ - a->left = 0; - a->right = 0; - a->balance = 0; - /* insert into an empty tree */ - if (!t->root) { - t->root = a; - return 1; - } - - if (t->compar(t->root, a) > 0) { - /* insert into the left subtree */ - if (t->root->left) { - avl_tree left_subtree; - left_subtree.root = t->root->left; - left_subtree.compar = t->compar; - if (avl_insert(&left_subtree, a)) { - switch (t->root->balance--) { - case 1: - return 0; - case 0: - return 1; - } - if (t->root->left->balance < 0) { - avl_swr(&(t->root)); - t->root->balance = 0; - t->root->right->balance = 0; - } else { - avl_swl(&(t->root->left)); - avl_swr(&(t->root)); - avl_nasty(t->root); - } - } else - t->root->left = left_subtree.root; - return 0; - } else { - t->root->left = a; - if (t->root->balance--) - return 0; - return 1; - } - } else { - /* insert into the right subtree */ - if (t->root->right) { - avl_tree right_subtree; - right_subtree.root = t->root->right; - right_subtree.compar = t->compar; - if (avl_insert(&right_subtree, a)) { - switch (t->root->balance++) { - case -1: - return 0; - case 0: - return 1; - } - if (t->root->right->balance > 0) { - avl_swl(&(t->root)); - t->root->balance = 0; - t->root->left->balance = 0; - } else { - avl_swr(&(t->root->right)); - avl_swl(&(t->root)); - avl_nasty(t->root); - } - } else - t->root->right = right_subtree.root; - return 0; - } else { - t->root->right = a; - if (t->root->balance++) - return 0; - return 1; - } - } -} - -/* Remove an element a from the AVL tree t - * returns -1 if the depth of the tree has shrunk - * Warning: if the element is not present in the tree, - * returns 0 as if it had been removed succesfully. - */ -int avl_remove(avl_tree* t, avl* a) { - int b; - if (t->root == a) - return avl_removeroot(t); - b = t->compar(t->root, a); - if (b >= 0) { - /* remove from the left subtree */ - int ch; - avl_tree left_subtree; - if ((left_subtree.root = t->root->left)) { - left_subtree.compar = t->compar; - ch = avl_remove(&left_subtree, a); - t->root->left = left_subtree.root; - if (ch) { - switch (t->root->balance++) { - case -1: - return -1; - case 0: - return 0; - } - switch (t->root->right->balance) { - case 0: - avl_swl(&(t->root)); - t->root->balance = -1; - t->root->left->balance = 1; - return 0; - case 1: - avl_swl(&(t->root)); - t->root->balance = 0; - t->root->left->balance = 0; - return -1; - } - avl_swr(&(t->root->right)); - avl_swl(&(t->root)); - avl_nasty(t->root); - return -1; - } - } - } - if (b <= 0) { - /* remove from the right subtree */ - int ch; - avl_tree right_subtree; - if ((right_subtree.root = t->root->right)) { - right_subtree.compar = t->compar; - ch = avl_remove(&right_subtree, a); - t->root->right = right_subtree.root; - if (ch) { - switch (t->root->balance--) { - case 1: - return -1; - case 0: - return 0; - } - switch (t->root->left->balance) { - case 0: - avl_swr(&(t->root)); - t->root->balance = 1; - t->root->right->balance = -1; - return 0; - case -1: - avl_swr(&(t->root)); - t->root->balance = 0; - t->root->right->balance = 0; - return -1; - } - avl_swl(&(t->root->left)); - avl_swr(&(t->root)); - avl_nasty(t->root); - return -1; - } - } - } - return 0; + return NULL; } -/* Remove the root of the AVL tree t - * Warning: dumps core if t is empty +/* Inserts |item| into |tree| and returns a pointer to |item|'s address. + If a duplicate item is found in the tree, + returns a pointer to the duplicate without inserting |item|. */ -int avl_removeroot(avl_tree* t) { - int ch; - avl* a; - if (!t->root->left) { - if (!t->root->right) { - t->root = 0; - return -1; - } - t->root = t->root->right; - return -1; - } - if (!t->root->right) { - t->root = t->root->left; - return -1; - } - if (t->root->balance < 0) { - /* remove from the left subtree */ - a = t->root->left; - while (a->right) - a = a->right; - } else { - /* remove from the right subtree */ - a = t->root->right; - while (a->left) - a = a->left; - } - ch = avl_remove(t, a); - a->left = t->root->left; - a->right = t->root->right; - a->balance = t->root->balance; - t->root = a; - if (a->balance == 0) - return ch; - return 0; +avl *avl_insert(avl_tree *tree, avl *item) { + avl *y, *z; /* Top node to update balance factor, and parent. */ + avl *p, *q; /* Iterator, and parent. */ + avl *n; /* Newly inserted node. */ + avl *w; /* New root of rebalanced subtree. */ + int dir; /* Direction to descend. */ + + unsigned char da[AVL_MAX_HEIGHT]; /* Cached comparison results. */ + int k = 0; /* Number of cached results. */ + + // assert(tree != NULL && item != NULL); + + z = (avl *) &tree->root; + y = tree->root; + dir = 0; + for (q = z, p = y; p != NULL; q = p, p = p->avl_link[dir]) { + int cmp = tree->compar(item, p); + if (cmp == 0) + return p; + + if (p->avl_balance != 0) + z = q, y = p, k = 0; + da[k++] = dir = (cmp > 0); + } + + n = q->avl_link[dir] = item; + + // tree->avl_count++; + n->avl_link[0] = n->avl_link[1] = NULL; + n->avl_balance = 0; + if (y == NULL) return n; + + for (p = y, k = 0; p != n; p = p->avl_link[da[k]], k++) + if (da[k] == 0) + p->avl_balance--; + else + p->avl_balance++; + + if (y->avl_balance == -2) { + avl *x = y->avl_link[0]; + if (x->avl_balance == -1) { + w = x; + y->avl_link[0] = x->avl_link[1]; + x->avl_link[1] = y; + x->avl_balance = y->avl_balance = 0; + } + else { + // assert (x->avl_balance == +1); + w = x->avl_link[1]; + x->avl_link[1] = w->avl_link[0]; + w->avl_link[0] = x; + y->avl_link[0] = w->avl_link[1]; + w->avl_link[1] = y; + if (w->avl_balance == -1) + x->avl_balance = 0, y->avl_balance = +1; + else if (w->avl_balance == 0) + x->avl_balance = y->avl_balance = 0; + else /* |w->avl_balance == +1| */ + x->avl_balance = -1, y->avl_balance = 0; + w->avl_balance = 0; + } + } + else if (y->avl_balance == +2) { + avl *x = y->avl_link[1]; + if (x->avl_balance == +1) { + w = x; + y->avl_link[1] = x->avl_link[0]; + x->avl_link[0] = y; + x->avl_balance = y->avl_balance = 0; + } + else { + // assert (x->avl_balance == -1); + w = x->avl_link[0]; + x->avl_link[0] = w->avl_link[1]; + w->avl_link[1] = x; + y->avl_link[1] = w->avl_link[0]; + w->avl_link[0] = y; + if (w->avl_balance == +1) + x->avl_balance = 0, y->avl_balance = -1; + else if (w->avl_balance == 0) + x->avl_balance = y->avl_balance = 0; + else /* |w->avl_balance == -1| */ + x->avl_balance = +1, y->avl_balance = 0; + w->avl_balance = 0; + } + } + else return n; + + z->avl_link[y != z->avl_link[0]] = w; + + // tree->avl_generation++; + return n; } -/* Iterate through elements in t from a range between a and b (inclusive) - * for each element calls iter(a) until it returns 0 - * returns the last value returned by iterator or 0 if there were no calls - * Warning: a<=b must hold - */ -int avl_range(avl_tree* t, avl* a, avl* b, int (*iter)(avl*), avl** ret) { - int x, c = 0; - if (!t->root) - return 0; - x = t->compar(t->root, a); - if (a != b) { - if (x < 0) { - x = t->compar(t->root, b); - if (x > 0) - x = 0; - } - } - if (x >= 0) { - /* search in the left subtree */ - avl_tree left_subtree; - if ((left_subtree.root = t->root->left)) { - left_subtree.compar = t->compar; - if (!(c = avl_range(&left_subtree, a, b, iter, ret))) - if (x > 0) - return 0; - } - } - if (x == 0) { - if (!(c = iter(t->root))) { - if (ret) - *ret = t->root; - return 0; - } - } - if (x <= 0) { - /* search in the right subtree */ - avl_tree right_subtree; - if ((right_subtree.root = t->root->right)) { - right_subtree.compar = t->compar; - if (!(c = avl_range(&right_subtree, a, b, iter, ret))) - if (x < 0) - return 0; - } - } - return c; +/* Deletes from |tree| and returns an item matching |item|. + Returns a null pointer if no matching item found. */ +avl *avl_remove(avl_tree *tree, avl *item) { + /* Stack of nodes. */ + avl *pa[AVL_MAX_HEIGHT]; /* Nodes. */ + unsigned char da[AVL_MAX_HEIGHT]; /* |avl_link[]| indexes. */ + int k; /* Stack pointer. */ + + avl *p; /* Traverses tree to find node to delete. */ + int cmp; /* Result of comparison between |item| and |p|. */ + + // assert (tree != NULL && item != NULL); + + k = 0; + p = (avl *) &tree->root; + for(cmp = -1; cmp != 0; cmp = tree->compar(item, p)) { + int dir = (cmp > 0); + + pa[k] = p; + da[k++] = dir; + + p = p->avl_link[dir]; + if(p == NULL) return NULL; + } + + item = p; + + if (p->avl_link[1] == NULL) + pa[k - 1]->avl_link[da[k - 1]] = p->avl_link[0]; + else { + avl *r = p->avl_link[1]; + if (r->avl_link[0] == NULL) { + r->avl_link[0] = p->avl_link[0]; + r->avl_balance = p->avl_balance; + pa[k - 1]->avl_link[da[k - 1]] = r; + da[k] = 1; + pa[k++] = r; + } + else { + avl *s; + int j = k++; + + for (;;) { + da[k] = 0; + pa[k++] = r; + s = r->avl_link[0]; + if (s->avl_link[0] == NULL) break; + + r = s; + } + + s->avl_link[0] = p->avl_link[0]; + r->avl_link[0] = s->avl_link[1]; + s->avl_link[1] = p->avl_link[1]; + s->avl_balance = p->avl_balance; + + pa[j - 1]->avl_link[da[j - 1]] = s; + da[j] = 1; + pa[j] = s; + } + } + + // assert (k > 0); + while (--k > 0) { + avl *y = pa[k]; + + if (da[k] == 0) { + y->avl_balance++; + if (y->avl_balance == +1) break; + else if (y->avl_balance == +2) { + avl *x = y->avl_link[1]; + if (x->avl_balance == -1) { + avl *w; + // assert (x->avl_balance == -1); + w = x->avl_link[0]; + x->avl_link[0] = w->avl_link[1]; + w->avl_link[1] = x; + y->avl_link[1] = w->avl_link[0]; + w->avl_link[0] = y; + if (w->avl_balance == +1) + x->avl_balance = 0, y->avl_balance = -1; + else if (w->avl_balance == 0) + x->avl_balance = y->avl_balance = 0; + else /* |w->avl_balance == -1| */ + x->avl_balance = +1, y->avl_balance = 0; + w->avl_balance = 0; + pa[k - 1]->avl_link[da[k - 1]] = w; + } + else { + y->avl_link[1] = x->avl_link[0]; + x->avl_link[0] = y; + pa[k - 1]->avl_link[da[k - 1]] = x; + if (x->avl_balance == 0) { + x->avl_balance = -1; + y->avl_balance = +1; + break; + } + else x->avl_balance = y->avl_balance = 0; + } + } + } + else + { + y->avl_balance--; + if (y->avl_balance == -1) break; + else if (y->avl_balance == -2) { + avl *x = y->avl_link[0]; + if (x->avl_balance == +1) { + avl *w; + // assert (x->avl_balance == +1); + w = x->avl_link[1]; + x->avl_link[1] = w->avl_link[0]; + w->avl_link[0] = x; + y->avl_link[0] = w->avl_link[1]; + w->avl_link[1] = y; + if (w->avl_balance == -1) + x->avl_balance = 0, y->avl_balance = +1; + else if (w->avl_balance == 0) + x->avl_balance = y->avl_balance = 0; + else /* |w->avl_balance == +1| */ + x->avl_balance = -1, y->avl_balance = 0; + w->avl_balance = 0; + pa[k - 1]->avl_link[da[k - 1]] = w; + } + else { + y->avl_link[0] = x->avl_link[1]; + x->avl_link[1] = y; + pa[k - 1]->avl_link[da[k - 1]] = x; + if (x->avl_balance == 0) { + x->avl_balance = +1; + y->avl_balance = -1; + break; + } + else x->avl_balance = y->avl_balance = 0; + } + } + } + } + + // tree->avl_count--; + // tree->avl_generation++; + return item; } -/* high performance searching - by ktsaou */ -avl *avl_search(avl_tree *t, avl *a) { - avl *root = t->root; - - while(root) { - int x = t->compar(root, a); +/* ------------------------------------------------------------------------- */ +// below are functions by (C) Costa Tsaousis - if(x > 0) { - root = root->left; - continue; - } +// --------------------------- +// traversing - if(x < 0) { - root = root->right; - continue; - } +void avl_walker(avl *node, void (*callback)(void *)) { + if(node->avl_link[0]) + avl_walker(node->avl_link[0], callback); - return root; - } + callback(node); - return NULL; + if(node->avl_link[1]) + avl_walker(node->avl_link[1], callback); } -void avl_init(avl_tree *t, int (*compar)(void *a, void *b)) { - t->root = NULL; - t->compar = compar; +void avl_traverse(avl_tree *t, void (*callback)(void *)) { + avl_walker(t->root, callback); } -/* ------------------------------------------------------------------------- */ +// --------------------------- +// locks void avl_read_lock(avl_tree_lock *t) { #ifndef AVL_WITHOUT_PTHREADS #ifdef AVL_LOCK_WITH_MUTEX - pthread_mutex_lock(&t->mutex); + pthread_mutex_lock(&t->mutex); #else - pthread_rwlock_rdlock(&t->rwlock); + pthread_rwlock_rdlock(&t->rwlock); #endif #endif /* AVL_WITHOUT_PTHREADS */ } @@ -352,9 +313,9 @@ void avl_read_lock(avl_tree_lock *t) { void avl_write_lock(avl_tree_lock *t) { #ifndef AVL_WITHOUT_PTHREADS #ifdef AVL_LOCK_WITH_MUTEX - pthread_mutex_lock(&t->mutex); + pthread_mutex_lock(&t->mutex); #else - pthread_rwlock_wrlock(&t->rwlock); + pthread_rwlock_wrlock(&t->rwlock); #endif #endif /* AVL_WITHOUT_PTHREADS */ } @@ -362,64 +323,64 @@ void avl_write_lock(avl_tree_lock *t) { void avl_unlock(avl_tree_lock *t) { #ifndef AVL_WITHOUT_PTHREADS #ifdef AVL_LOCK_WITH_MUTEX - pthread_mutex_unlock(&t->mutex); + pthread_mutex_unlock(&t->mutex); #else - pthread_rwlock_unlock(&t->rwlock); + pthread_rwlock_unlock(&t->rwlock); #endif #endif /* AVL_WITHOUT_PTHREADS */ } -/* ------------------------------------------------------------------------- */ +// --------------------------- +// operations with locking void avl_init_lock(avl_tree_lock *t, int (*compar)(void *a, void *b)) { - avl_init(&t->avl_tree, compar); + avl_init(&t->avl_tree, compar); #ifndef AVL_WITHOUT_PTHREADS - int lock; + int lock; #ifdef AVL_LOCK_WITH_MUTEX - lock = pthread_mutex_init(&t->mutex, NULL); + lock = pthread_mutex_init(&t->mutex, NULL); #else - lock = pthread_rwlock_init(&t->rwlock, NULL); + lock = pthread_rwlock_init(&t->rwlock, NULL); #endif - if(lock != 0) - fatal("Failed to initialize AVL mutex/rwlock, error: %d", lock); + if(lock != 0) + fatal("Failed to initialize AVL mutex/rwlock, error: %d", lock); #endif /* AVL_WITHOUT_PTHREADS */ } avl *avl_search_lock(avl_tree_lock *t, avl *a) { - avl_read_lock(t); - avl *ret = avl_search(&t->avl_tree, a); - avl_unlock(t); - return ret; + avl_read_lock(t); + avl *ret = avl_search(&t->avl_tree, a); + avl_unlock(t); + return ret; } -int avl_range_lock(avl_tree_lock *t, avl *a, avl *b, int (*iter)(avl *), avl **ret) { - avl_read_lock(t); - int ret2 = avl_range(&t->avl_tree, a, b, iter, ret); - avl_unlock(t); - return ret2; +avl * avl_remove_lock(avl_tree_lock *t, avl *a) { + avl_write_lock(t); + avl *ret = avl_remove(&t->avl_tree, a); + avl_unlock(t); + return ret; } -int avl_removeroot_lock(avl_tree_lock *t) { - avl_write_lock(t); - int ret = avl_removeroot(&t->avl_tree); - avl_unlock(t); - return ret; +avl *avl_insert_lock(avl_tree_lock *t, avl *a) { + avl_write_lock(t); + avl * ret = avl_insert(&t->avl_tree, a); + avl_unlock(t); + return ret; } -int avl_remove_lock(avl_tree_lock *t, avl *a) { - avl_write_lock(t); - int ret = avl_remove(&t->avl_tree, a); - avl_unlock(t); - return ret; +void avl_traverse_lock(avl_tree_lock *t, void (*callback)(void *)) { + avl_read_lock(t); + avl_traverse(&t->avl_tree, callback); + avl_unlock(t); } -int avl_insert_lock(avl_tree_lock *t, avl *a) { - avl_write_lock(t); - int ret = avl_insert(&t->avl_tree, a); - avl_unlock(t); - return ret; +void avl_init(avl_tree *t, int (*compar)(void *a, void *b)) { + t->root = NULL; + t->compar = compar; } + +// ------------------
\ No newline at end of file @@ -1,20 +1,12 @@ -/* - * ANSI C Library for maintainance of AVL Balanced Trees - * - * ref.: - * G. M. Adelson-Velskij & E. M. Landis - * Doklady Akad. Nauk SSSR 146 (1962), 263-266 - * - * see also: - * D. E. Knuth: The Art of Computer Programming Vol.3 (Sorting and Searching) - * - * (C) 2000 Daniel Nagy, Budapest University of Technology and Economics - * Released under GNU General Public License (GPL) version 2 - * - */ + #ifndef _AVL_H #define _AVL_H 1 +/* Maximum AVL tree height. */ +#ifndef AVL_MAX_HEIGHT +#define AVL_MAX_HEIGHT 92 +#endif + #ifndef AVL_WITHOUT_PTHREADS #include <pthread.h> @@ -34,26 +26,24 @@ /* One element of the AVL tree */ typedef struct avl { - struct avl* left; - struct avl* right; - signed char balance; + struct avl *avl_link[2]; /* Subtrees. */ + signed char avl_balance; /* Balance factor. */ } avl; /* An AVL tree */ typedef struct avl_tree { - avl *root; - - int (*compar)(void *a, void *b); + avl *root; + int (*compar)(void *a, void *b); } avl_tree; typedef struct avl_tree_lock { - avl_tree avl_tree; + avl_tree avl_tree; #ifndef AVL_WITHOUT_PTHREADS #ifdef AVL_LOCK_WITH_MUTEX - pthread_mutex_t mutex; + pthread_mutex_t mutex; #else /* AVL_LOCK_WITH_MUTEX */ - pthread_rwlock_t rwlock; + pthread_rwlock_t rwlock; #endif /* AVL_LOCK_WITH_MUTEX */ #endif /* AVL_WITHOUT_PTHREADS */ } avl_tree_lock; @@ -61,37 +51,25 @@ typedef struct avl_tree_lock { /* Public methods */ /* Insert element a into the AVL tree t - * returns 1 if the depth of the tree has grown - * Warning: do not insert elements already present + * returns the added element a, or a pointer the + * element that is equal to a (as returned by t->compar()) + * a is linked directly to the tree, so it has to + * be properly allocated by the caller. */ -int avl_insert_lock(avl_tree_lock *t, avl *a); -int avl_insert(avl_tree *t, avl *a); +avl *avl_insert_lock(avl_tree_lock *t, avl *a); +avl *avl_insert(avl_tree *t, avl *a); /* Remove an element a from the AVL tree t - * returns -1 if the depth of the tree has shrunk - * Warning: if the element is not present in the tree, - * returns 0 as if it had been removed succesfully. - */ -int avl_remove_lock(avl_tree_lock *t, avl *a); -int avl_remove(avl_tree *t, avl *a); - -/* Remove the root of the AVL tree t - * Warning: dumps core if t is empty - */ -int avl_removeroot_lock(avl_tree_lock *t); -int avl_removeroot(avl_tree *t); - -/* Iterate through elements in t from a range between a and b (inclusive) - * for each element calls iter(a) until it returns 0 - * returns the last value returned by iterator or 0 if there were no calls - * Warning: a<=b must hold + * returns a pointer to the removed element + * or NULL if an element equal to a is not found + * (equal as returned by t->compar()) */ -int avl_range_lock(avl_tree_lock *t, avl *a, avl *b, int (*iter)(avl *), avl **ret); -int avl_range(avl_tree *t, avl *a, avl *b, int (*iter)(avl *), avl **ret); +avl *avl_remove_lock(avl_tree_lock *t, avl *a); +avl *avl_remove(avl_tree *t, avl *a); -/* Iterate through elements in t equal to a - * for each element calls iter(a) until it returns 0 - * returns the last value returned by iterator or 0 if there were no calls +/* Find the element into the tree that equal to a + * (equal as returned by t->compar()) + * returns NULL is no element is equal to a */ avl *avl_search_lock(avl_tree_lock *t, avl *a); avl *avl_search(avl_tree *t, avl *a); @@ -101,4 +79,8 @@ avl *avl_search(avl_tree *t, avl *a); void avl_init_lock(avl_tree_lock *t, int (*compar)(void *a, void *b)); void avl_init(avl_tree *t, int (*compar)(void *a, void *b)); + +void avl_traverse_lock(avl_tree_lock *t, void (*callback)(void *)); +void avl_traverse(avl_tree *t, void (*callback)(void *)); + #endif /* avl.h */ diff --git a/src/common.c b/src/common.c index a2b0d940f..7d0fac9aa 100644 --- a/src/common.c +++ b/src/common.c @@ -1,752 +1,859 @@ -#ifdef HAVE_CONFIG_H -#include <config.h> -#endif -#include <sys/syscall.h> -#include <string.h> -#include <ctype.h> -#include <errno.h> -#include <unistd.h> -#include <sys/types.h> -#include <sys/stat.h> -#include <fcntl.h> -#include <sys/mman.h> - - -#include "log.h" #include "common.h" -#include "appconfig.h" -#include "../config.h" char *global_host_prefix = ""; int enable_ksm = 1; -// time(NULL) in milliseconds -unsigned long long timems(void) { - struct timeval now; - gettimeofday(&now, NULL); - return now.tv_sec * 1000000ULL + now.tv_usec; +volatile sig_atomic_t netdata_exit = 0; + +// ---------------------------------------------------------------------------- +// memory allocation functions that handle failures + +// although netdata does not use memory allocations too often (netdata tries to +// maintain its memory footprint stable during runtime, i.e. all buffers are +// allocated during initialization and are adapted to current use throughout +// its lifetime), these can be used to override the default system allocation +// routines. + +char *strdupz(const char *s) { + char *t = strdup(s); + if (unlikely(!t)) fatal("Cannot strdup() string '%s'", s); + return t; +} + +void *mallocz(size_t size) { + void *p = malloc(size); + if (unlikely(!p)) fatal("Cannot allocate %zu bytes of memory.", size); + return p; +} + +void *callocz(size_t nmemb, size_t size) { + void *p = calloc(nmemb, size); + if (unlikely(!p)) fatal("Cannot allocate %zu bytes of memory.", nmemb * size); + return p; +} + +void *reallocz(void *ptr, size_t size) { + void *p = realloc(ptr, size); + if (unlikely(!p)) fatal("Cannot re-allocate memory to %zu bytes.", size); + return p; +} + +void freez(void *ptr) { + free(ptr); +} + +// ---------------------------------------------------------------------------- +// time functions + +inline unsigned long long timeval_usec(struct timeval *tv) { + return tv->tv_sec * 1000000ULL + tv->tv_usec; +} + +// time(NULL) in nanoseconds +inline unsigned long long time_usec(void) { + struct timeval now; + gettimeofday(&now, NULL); + return timeval_usec(&now); +} + +inline unsigned long long usec_dt(struct timeval *now, struct timeval *old) { + unsigned long long tv1 = timeval_usec(now); + unsigned long long tv2 = timeval_usec(old); + return (tv1 > tv2) ? (tv1 - tv2) : (tv2 - tv1); +} + +int sleep_usec(unsigned long long usec) { + +#ifndef NETDATA_WITH_USLEEP + // we expect microseconds (1.000.000 per second) + // but timespec is nanoseconds (1.000.000.000 per second) + struct timespec rem, req = { + .tv_sec = (time_t) (usec / 1000000), + .tv_nsec = (suseconds_t) ((usec % 1000000) * 1000) + }; + + while (nanosleep(&req, &rem) == -1) { + if (likely(errno == EINTR)) { + info("nanosleep() interrupted (while sleeping for %llu microseconds).", usec); + req.tv_sec = rem.tv_sec; + req.tv_nsec = rem.tv_nsec; + } else { + error("Cannot nanosleep() for %llu microseconds.", usec); + break; + } + } + + return 0; +#else + int ret = usleep(usec); + if(unlikely(ret == -1 && errno == EINVAL)) { + // on certain systems, usec has to be up to 999999 + if(usec > 999999) { + int counter = usec / 999999; + while(counter--) + usleep(999999); + + usleep(usec % 999999); + } + else { + error("Cannot usleep() for %llu microseconds.", usec); + return ret; + } + } + + if(ret != 0) + error("usleep() failed for %llu microseconds.", usec); + + return ret; +#endif } unsigned char netdata_map_chart_names[256] = { - [0] = '\0', // - [1] = '_', // - [2] = '_', // - [3] = '_', // - [4] = '_', // - [5] = '_', // - [6] = '_', // - [7] = '_', // - [8] = '_', // - [9] = '_', // - [10] = '_', // - [11] = '_', // - [12] = '_', // - [13] = '_', // - [14] = '_', // - [15] = '_', // - [16] = '_', // - [17] = '_', // - [18] = '_', // - [19] = '_', // - [20] = '_', // - [21] = '_', // - [22] = '_', // - [23] = '_', // - [24] = '_', // - [25] = '_', // - [26] = '_', // - [27] = '_', // - [28] = '_', // - [29] = '_', // - [30] = '_', // - [31] = '_', // - [32] = '_', // - [33] = '_', // ! - [34] = '_', // " - [35] = '_', // # - [36] = '_', // $ - [37] = '_', // % - [38] = '_', // & - [39] = '_', // ' - [40] = '_', // ( - [41] = '_', // ) - [42] = '_', // * - [43] = '_', // + - [44] = '.', // , - [45] = '-', // - - [46] = '.', // . - [47] = '/', // / - [48] = '0', // 0 - [49] = '1', // 1 - [50] = '2', // 2 - [51] = '3', // 3 - [52] = '4', // 4 - [53] = '5', // 5 - [54] = '6', // 6 - [55] = '7', // 7 - [56] = '8', // 8 - [57] = '9', // 9 - [58] = '_', // : - [59] = '_', // ; - [60] = '_', // < - [61] = '_', // = - [62] = '_', // > - [63] = '_', // ? - [64] = '_', // @ - [65] = 'a', // A - [66] = 'b', // B - [67] = 'c', // C - [68] = 'd', // D - [69] = 'e', // E - [70] = 'f', // F - [71] = 'g', // G - [72] = 'h', // H - [73] = 'i', // I - [74] = 'j', // J - [75] = 'k', // K - [76] = 'l', // L - [77] = 'm', // M - [78] = 'n', // N - [79] = 'o', // O - [80] = 'p', // P - [81] = 'q', // Q - [82] = 'r', // R - [83] = 's', // S - [84] = 't', // T - [85] = 'u', // U - [86] = 'v', // V - [87] = 'w', // W - [88] = 'x', // X - [89] = 'y', // Y - [90] = 'z', // Z - [91] = '_', // [ - [92] = '/', // backslash - [93] = '_', // ] - [94] = '_', // ^ - [95] = '_', // _ - [96] = '_', // ` - [97] = 'a', // a - [98] = 'b', // b - [99] = 'c', // c - [100] = 'd', // d - [101] = 'e', // e - [102] = 'f', // f - [103] = 'g', // g - [104] = 'h', // h - [105] = 'i', // i - [106] = 'j', // j - [107] = 'k', // k - [108] = 'l', // l - [109] = 'm', // m - [110] = 'n', // n - [111] = 'o', // o - [112] = 'p', // p - [113] = 'q', // q - [114] = 'r', // r - [115] = 's', // s - [116] = 't', // t - [117] = 'u', // u - [118] = 'v', // v - [119] = 'w', // w - [120] = 'x', // x - [121] = 'y', // y - [122] = 'z', // z - [123] = '_', // { - [124] = '_', // | - [125] = '_', // } - [126] = '_', // ~ - [127] = '_', // - [128] = '_', // - [129] = '_', // - [130] = '_', // - [131] = '_', // - [132] = '_', // - [133] = '_', // - [134] = '_', // - [135] = '_', // - [136] = '_', // - [137] = '_', // - [138] = '_', // - [139] = '_', // - [140] = '_', // - [141] = '_', // - [142] = '_', // - [143] = '_', // - [144] = '_', // - [145] = '_', // - [146] = '_', // - [147] = '_', // - [148] = '_', // - [149] = '_', // - [150] = '_', // - [151] = '_', // - [152] = '_', // - [153] = '_', // - [154] = '_', // - [155] = '_', // - [156] = '_', // - [157] = '_', // - [158] = '_', // - [159] = '_', // - [160] = '_', // - [161] = '_', // - [162] = '_', // - [163] = '_', // - [164] = '_', // - [165] = '_', // - [166] = '_', // - [167] = '_', // - [168] = '_', // - [169] = '_', // - [170] = '_', // - [171] = '_', // - [172] = '_', // - [173] = '_', // - [174] = '_', // - [175] = '_', // - [176] = '_', // - [177] = '_', // - [178] = '_', // - [179] = '_', // - [180] = '_', // - [181] = '_', // - [182] = '_', // - [183] = '_', // - [184] = '_', // - [185] = '_', // - [186] = '_', // - [187] = '_', // - [188] = '_', // - [189] = '_', // - [190] = '_', // - [191] = '_', // - [192] = '_', // - [193] = '_', // - [194] = '_', // - [195] = '_', // - [196] = '_', // - [197] = '_', // - [198] = '_', // - [199] = '_', // - [200] = '_', // - [201] = '_', // - [202] = '_', // - [203] = '_', // - [204] = '_', // - [205] = '_', // - [206] = '_', // - [207] = '_', // - [208] = '_', // - [209] = '_', // - [210] = '_', // - [211] = '_', // - [212] = '_', // - [213] = '_', // - [214] = '_', // - [215] = '_', // - [216] = '_', // - [217] = '_', // - [218] = '_', // - [219] = '_', // - [220] = '_', // - [221] = '_', // - [222] = '_', // - [223] = '_', // - [224] = '_', // - [225] = '_', // - [226] = '_', // - [227] = '_', // - [228] = '_', // - [229] = '_', // - [230] = '_', // - [231] = '_', // - [232] = '_', // - [233] = '_', // - [234] = '_', // - [235] = '_', // - [236] = '_', // - [237] = '_', // - [238] = '_', // - [239] = '_', // - [240] = '_', // - [241] = '_', // - [242] = '_', // - [243] = '_', // - [244] = '_', // - [245] = '_', // - [246] = '_', // - [247] = '_', // - [248] = '_', // - [249] = '_', // - [250] = '_', // - [251] = '_', // - [252] = '_', // - [253] = '_', // - [254] = '_', // - [255] = '_' // + [0] = '\0', // + [1] = '_', // + [2] = '_', // + [3] = '_', // + [4] = '_', // + [5] = '_', // + [6] = '_', // + [7] = '_', // + [8] = '_', // + [9] = '_', // + [10] = '_', // + [11] = '_', // + [12] = '_', // + [13] = '_', // + [14] = '_', // + [15] = '_', // + [16] = '_', // + [17] = '_', // + [18] = '_', // + [19] = '_', // + [20] = '_', // + [21] = '_', // + [22] = '_', // + [23] = '_', // + [24] = '_', // + [25] = '_', // + [26] = '_', // + [27] = '_', // + [28] = '_', // + [29] = '_', // + [30] = '_', // + [31] = '_', // + [32] = '_', // + [33] = '_', // ! + [34] = '_', // " + [35] = '_', // # + [36] = '_', // $ + [37] = '_', // % + [38] = '_', // & + [39] = '_', // ' + [40] = '_', // ( + [41] = '_', // ) + [42] = '_', // * + [43] = '_', // + + [44] = '.', // , + [45] = '-', // - + [46] = '.', // . + [47] = '/', // / + [48] = '0', // 0 + [49] = '1', // 1 + [50] = '2', // 2 + [51] = '3', // 3 + [52] = '4', // 4 + [53] = '5', // 5 + [54] = '6', // 6 + [55] = '7', // 7 + [56] = '8', // 8 + [57] = '9', // 9 + [58] = '_', // : + [59] = '_', // ; + [60] = '_', // < + [61] = '_', // = + [62] = '_', // > + [63] = '_', // ? + [64] = '_', // @ + [65] = 'a', // A + [66] = 'b', // B + [67] = 'c', // C + [68] = 'd', // D + [69] = 'e', // E + [70] = 'f', // F + [71] = 'g', // G + [72] = 'h', // H + [73] = 'i', // I + [74] = 'j', // J + [75] = 'k', // K + [76] = 'l', // L + [77] = 'm', // M + [78] = 'n', // N + [79] = 'o', // O + [80] = 'p', // P + [81] = 'q', // Q + [82] = 'r', // R + [83] = 's', // S + [84] = 't', // T + [85] = 'u', // U + [86] = 'v', // V + [87] = 'w', // W + [88] = 'x', // X + [89] = 'y', // Y + [90] = 'z', // Z + [91] = '_', // [ + [92] = '/', // backslash + [93] = '_', // ] + [94] = '_', // ^ + [95] = '_', // _ + [96] = '_', // ` + [97] = 'a', // a + [98] = 'b', // b + [99] = 'c', // c + [100] = 'd', // d + [101] = 'e', // e + [102] = 'f', // f + [103] = 'g', // g + [104] = 'h', // h + [105] = 'i', // i + [106] = 'j', // j + [107] = 'k', // k + [108] = 'l', // l + [109] = 'm', // m + [110] = 'n', // n + [111] = 'o', // o + [112] = 'p', // p + [113] = 'q', // q + [114] = 'r', // r + [115] = 's', // s + [116] = 't', // t + [117] = 'u', // u + [118] = 'v', // v + [119] = 'w', // w + [120] = 'x', // x + [121] = 'y', // y + [122] = 'z', // z + [123] = '_', // { + [124] = '_', // | + [125] = '_', // } + [126] = '_', // ~ + [127] = '_', // + [128] = '_', // + [129] = '_', // + [130] = '_', // + [131] = '_', // + [132] = '_', // + [133] = '_', // + [134] = '_', // + [135] = '_', // + [136] = '_', // + [137] = '_', // + [138] = '_', // + [139] = '_', // + [140] = '_', // + [141] = '_', // + [142] = '_', // + [143] = '_', // + [144] = '_', // + [145] = '_', // + [146] = '_', // + [147] = '_', // + [148] = '_', // + [149] = '_', // + [150] = '_', // + [151] = '_', // + [152] = '_', // + [153] = '_', // + [154] = '_', // + [155] = '_', // + [156] = '_', // + [157] = '_', // + [158] = '_', // + [159] = '_', // + [160] = '_', // + [161] = '_', // + [162] = '_', // + [163] = '_', // + [164] = '_', // + [165] = '_', // + [166] = '_', // + [167] = '_', // + [168] = '_', // + [169] = '_', // + [170] = '_', // + [171] = '_', // + [172] = '_', // + [173] = '_', // + [174] = '_', // + [175] = '_', // + [176] = '_', // + [177] = '_', // + [178] = '_', // + [179] = '_', // + [180] = '_', // + [181] = '_', // + [182] = '_', // + [183] = '_', // + [184] = '_', // + [185] = '_', // + [186] = '_', // + [187] = '_', // + [188] = '_', // + [189] = '_', // + [190] = '_', // + [191] = '_', // + [192] = '_', // + [193] = '_', // + [194] = '_', // + [195] = '_', // + [196] = '_', // + [197] = '_', // + [198] = '_', // + [199] = '_', // + [200] = '_', // + [201] = '_', // + [202] = '_', // + [203] = '_', // + [204] = '_', // + [205] = '_', // + [206] = '_', // + [207] = '_', // + [208] = '_', // + [209] = '_', // + [210] = '_', // + [211] = '_', // + [212] = '_', // + [213] = '_', // + [214] = '_', // + [215] = '_', // + [216] = '_', // + [217] = '_', // + [218] = '_', // + [219] = '_', // + [220] = '_', // + [221] = '_', // + [222] = '_', // + [223] = '_', // + [224] = '_', // + [225] = '_', // + [226] = '_', // + [227] = '_', // + [228] = '_', // + [229] = '_', // + [230] = '_', // + [231] = '_', // + [232] = '_', // + [233] = '_', // + [234] = '_', // + [235] = '_', // + [236] = '_', // + [237] = '_', // + [238] = '_', // + [239] = '_', // + [240] = '_', // + [241] = '_', // + [242] = '_', // + [243] = '_', // + [244] = '_', // + [245] = '_', // + [246] = '_', // + [247] = '_', // + [248] = '_', // + [249] = '_', // + [250] = '_', // + [251] = '_', // + [252] = '_', // + [253] = '_', // + [254] = '_', // + [255] = '_' // }; // make sure the supplied string // is good for a netdata chart/dimension ID/NAME void netdata_fix_chart_name(char *s) { - while((*s = netdata_map_chart_names[(unsigned char)*s])) s++; + while ((*s = netdata_map_chart_names[(unsigned char) *s])) s++; } unsigned char netdata_map_chart_ids[256] = { - [0] = '\0', // - [1] = '_', // - [2] = '_', // - [3] = '_', // - [4] = '_', // - [5] = '_', // - [6] = '_', // - [7] = '_', // - [8] = '_', // - [9] = '_', // - [10] = '_', // - [11] = '_', // - [12] = '_', // - [13] = '_', // - [14] = '_', // - [15] = '_', // - [16] = '_', // - [17] = '_', // - [18] = '_', // - [19] = '_', // - [20] = '_', // - [21] = '_', // - [22] = '_', // - [23] = '_', // - [24] = '_', // - [25] = '_', // - [26] = '_', // - [27] = '_', // - [28] = '_', // - [29] = '_', // - [30] = '_', // - [31] = '_', // - [32] = '_', // - [33] = '_', // ! - [34] = '_', // " - [35] = '_', // # - [36] = '_', // $ - [37] = '_', // % - [38] = '_', // & - [39] = '_', // ' - [40] = '_', // ( - [41] = '_', // ) - [42] = '_', // * - [43] = '_', // + - [44] = '.', // , - [45] = '-', // - - [46] = '.', // . - [47] = '_', // / - [48] = '0', // 0 - [49] = '1', // 1 - [50] = '2', // 2 - [51] = '3', // 3 - [52] = '4', // 4 - [53] = '5', // 5 - [54] = '6', // 6 - [55] = '7', // 7 - [56] = '8', // 8 - [57] = '9', // 9 - [58] = '_', // : - [59] = '_', // ; - [60] = '_', // < - [61] = '_', // = - [62] = '_', // > - [63] = '_', // ? - [64] = '_', // @ - [65] = 'a', // A - [66] = 'b', // B - [67] = 'c', // C - [68] = 'd', // D - [69] = 'e', // E - [70] = 'f', // F - [71] = 'g', // G - [72] = 'h', // H - [73] = 'i', // I - [74] = 'j', // J - [75] = 'k', // K - [76] = 'l', // L - [77] = 'm', // M - [78] = 'n', // N - [79] = 'o', // O - [80] = 'p', // P - [81] = 'q', // Q - [82] = 'r', // R - [83] = 's', // S - [84] = 't', // T - [85] = 'u', // U - [86] = 'v', // V - [87] = 'w', // W - [88] = 'x', // X - [89] = 'y', // Y - [90] = 'z', // Z - [91] = '_', // [ - [92] = '/', // backslash - [93] = '_', // ] - [94] = '_', // ^ - [95] = '_', // _ - [96] = '_', // ` - [97] = 'a', // a - [98] = 'b', // b - [99] = 'c', // c - [100] = 'd', // d - [101] = 'e', // e - [102] = 'f', // f - [103] = 'g', // g - [104] = 'h', // h - [105] = 'i', // i - [106] = 'j', // j - [107] = 'k', // k - [108] = 'l', // l - [109] = 'm', // m - [110] = 'n', // n - [111] = 'o', // o - [112] = 'p', // p - [113] = 'q', // q - [114] = 'r', // r - [115] = 's', // s - [116] = 't', // t - [117] = 'u', // u - [118] = 'v', // v - [119] = 'w', // w - [120] = 'x', // x - [121] = 'y', // y - [122] = 'z', // z - [123] = '_', // { - [124] = '_', // | - [125] = '_', // } - [126] = '_', // ~ - [127] = '_', // - [128] = '_', // - [129] = '_', // - [130] = '_', // - [131] = '_', // - [132] = '_', // - [133] = '_', // - [134] = '_', // - [135] = '_', // - [136] = '_', // - [137] = '_', // - [138] = '_', // - [139] = '_', // - [140] = '_', // - [141] = '_', // - [142] = '_', // - [143] = '_', // - [144] = '_', // - [145] = '_', // - [146] = '_', // - [147] = '_', // - [148] = '_', // - [149] = '_', // - [150] = '_', // - [151] = '_', // - [152] = '_', // - [153] = '_', // - [154] = '_', // - [155] = '_', // - [156] = '_', // - [157] = '_', // - [158] = '_', // - [159] = '_', // - [160] = '_', // - [161] = '_', // - [162] = '_', // - [163] = '_', // - [164] = '_', // - [165] = '_', // - [166] = '_', // - [167] = '_', // - [168] = '_', // - [169] = '_', // - [170] = '_', // - [171] = '_', // - [172] = '_', // - [173] = '_', // - [174] = '_', // - [175] = '_', // - [176] = '_', // - [177] = '_', // - [178] = '_', // - [179] = '_', // - [180] = '_', // - [181] = '_', // - [182] = '_', // - [183] = '_', // - [184] = '_', // - [185] = '_', // - [186] = '_', // - [187] = '_', // - [188] = '_', // - [189] = '_', // - [190] = '_', // - [191] = '_', // - [192] = '_', // - [193] = '_', // - [194] = '_', // - [195] = '_', // - [196] = '_', // - [197] = '_', // - [198] = '_', // - [199] = '_', // - [200] = '_', // - [201] = '_', // - [202] = '_', // - [203] = '_', // - [204] = '_', // - [205] = '_', // - [206] = '_', // - [207] = '_', // - [208] = '_', // - [209] = '_', // - [210] = '_', // - [211] = '_', // - [212] = '_', // - [213] = '_', // - [214] = '_', // - [215] = '_', // - [216] = '_', // - [217] = '_', // - [218] = '_', // - [219] = '_', // - [220] = '_', // - [221] = '_', // - [222] = '_', // - [223] = '_', // - [224] = '_', // - [225] = '_', // - [226] = '_', // - [227] = '_', // - [228] = '_', // - [229] = '_', // - [230] = '_', // - [231] = '_', // - [232] = '_', // - [233] = '_', // - [234] = '_', // - [235] = '_', // - [236] = '_', // - [237] = '_', // - [238] = '_', // - [239] = '_', // - [240] = '_', // - [241] = '_', // - [242] = '_', // - [243] = '_', // - [244] = '_', // - [245] = '_', // - [246] = '_', // - [247] = '_', // - [248] = '_', // - [249] = '_', // - [250] = '_', // - [251] = '_', // - [252] = '_', // - [253] = '_', // - [254] = '_', // - [255] = '_' // + [0] = '\0', // + [1] = '_', // + [2] = '_', // + [3] = '_', // + [4] = '_', // + [5] = '_', // + [6] = '_', // + [7] = '_', // + [8] = '_', // + [9] = '_', // + [10] = '_', // + [11] = '_', // + [12] = '_', // + [13] = '_', // + [14] = '_', // + [15] = '_', // + [16] = '_', // + [17] = '_', // + [18] = '_', // + [19] = '_', // + [20] = '_', // + [21] = '_', // + [22] = '_', // + [23] = '_', // + [24] = '_', // + [25] = '_', // + [26] = '_', // + [27] = '_', // + [28] = '_', // + [29] = '_', // + [30] = '_', // + [31] = '_', // + [32] = '_', // + [33] = '_', // ! + [34] = '_', // " + [35] = '_', // # + [36] = '_', // $ + [37] = '_', // % + [38] = '_', // & + [39] = '_', // ' + [40] = '_', // ( + [41] = '_', // ) + [42] = '_', // * + [43] = '_', // + + [44] = '.', // , + [45] = '-', // - + [46] = '.', // . + [47] = '_', // / + [48] = '0', // 0 + [49] = '1', // 1 + [50] = '2', // 2 + [51] = '3', // 3 + [52] = '4', // 4 + [53] = '5', // 5 + [54] = '6', // 6 + [55] = '7', // 7 + [56] = '8', // 8 + [57] = '9', // 9 + [58] = '_', // : + [59] = '_', // ; + [60] = '_', // < + [61] = '_', // = + [62] = '_', // > + [63] = '_', // ? + [64] = '_', // @ + [65] = 'a', // A + [66] = 'b', // B + [67] = 'c', // C + [68] = 'd', // D + [69] = 'e', // E + [70] = 'f', // F + [71] = 'g', // G + [72] = 'h', // H + [73] = 'i', // I + [74] = 'j', // J + [75] = 'k', // K + [76] = 'l', // L + [77] = 'm', // M + [78] = 'n', // N + [79] = 'o', // O + [80] = 'p', // P + [81] = 'q', // Q + [82] = 'r', // R + [83] = 's', // S + [84] = 't', // T + [85] = 'u', // U + [86] = 'v', // V + [87] = 'w', // W + [88] = 'x', // X + [89] = 'y', // Y + [90] = 'z', // Z + [91] = '_', // [ + [92] = '/', // backslash + [93] = '_', // ] + [94] = '_', // ^ + [95] = '_', // _ + [96] = '_', // ` + [97] = 'a', // a + [98] = 'b', // b + [99] = 'c', // c + [100] = 'd', // d + [101] = 'e', // e + [102] = 'f', // f + [103] = 'g', // g + [104] = 'h', // h + [105] = 'i', // i + [106] = 'j', // j + [107] = 'k', // k + [108] = 'l', // l + [109] = 'm', // m + [110] = 'n', // n + [111] = 'o', // o + [112] = 'p', // p + [113] = 'q', // q + [114] = 'r', // r + [115] = 's', // s + [116] = 't', // t + [117] = 'u', // u + [118] = 'v', // v + [119] = 'w', // w + [120] = 'x', // x + [121] = 'y', // y + [122] = 'z', // z + [123] = '_', // { + [124] = '_', // | + [125] = '_', // } + [126] = '_', // ~ + [127] = '_', // + [128] = '_', // + [129] = '_', // + [130] = '_', // + [131] = '_', // + [132] = '_', // + [133] = '_', // + [134] = '_', // + [135] = '_', // + [136] = '_', // + [137] = '_', // + [138] = '_', // + [139] = '_', // + [140] = '_', // + [141] = '_', // + [142] = '_', // + [143] = '_', // + [144] = '_', // + [145] = '_', // + [146] = '_', // + [147] = '_', // + [148] = '_', // + [149] = '_', // + [150] = '_', // + [151] = '_', // + [152] = '_', // + [153] = '_', // + [154] = '_', // + [155] = '_', // + [156] = '_', // + [157] = '_', // + [158] = '_', // + [159] = '_', // + [160] = '_', // + [161] = '_', // + [162] = '_', // + [163] = '_', // + [164] = '_', // + [165] = '_', // + [166] = '_', // + [167] = '_', // + [168] = '_', // + [169] = '_', // + [170] = '_', // + [171] = '_', // + [172] = '_', // + [173] = '_', // + [174] = '_', // + [175] = '_', // + [176] = '_', // + [177] = '_', // + [178] = '_', // + [179] = '_', // + [180] = '_', // + [181] = '_', // + [182] = '_', // + [183] = '_', // + [184] = '_', // + [185] = '_', // + [186] = '_', // + [187] = '_', // + [188] = '_', // + [189] = '_', // + [190] = '_', // + [191] = '_', // + [192] = '_', // + [193] = '_', // + [194] = '_', // + [195] = '_', // + [196] = '_', // + [197] = '_', // + [198] = '_', // + [199] = '_', // + [200] = '_', // + [201] = '_', // + [202] = '_', // + [203] = '_', // + [204] = '_', // + [205] = '_', // + [206] = '_', // + [207] = '_', // + [208] = '_', // + [209] = '_', // + [210] = '_', // + [211] = '_', // + [212] = '_', // + [213] = '_', // + [214] = '_', // + [215] = '_', // + [216] = '_', // + [217] = '_', // + [218] = '_', // + [219] = '_', // + [220] = '_', // + [221] = '_', // + [222] = '_', // + [223] = '_', // + [224] = '_', // + [225] = '_', // + [226] = '_', // + [227] = '_', // + [228] = '_', // + [229] = '_', // + [230] = '_', // + [231] = '_', // + [232] = '_', // + [233] = '_', // + [234] = '_', // + [235] = '_', // + [236] = '_', // + [237] = '_', // + [238] = '_', // + [239] = '_', // + [240] = '_', // + [241] = '_', // + [242] = '_', // + [243] = '_', // + [244] = '_', // + [245] = '_', // + [246] = '_', // + [247] = '_', // + [248] = '_', // + [249] = '_', // + [250] = '_', // + [251] = '_', // + [252] = '_', // + [253] = '_', // + [254] = '_', // + [255] = '_' // }; // make sure the supplied string // is good for a netdata chart/dimension ID/NAME void netdata_fix_chart_id(char *s) { - while((*s = netdata_map_chart_ids[(unsigned char)*s])) s++; + while ((*s = netdata_map_chart_ids[(unsigned char) *s])) s++; } /* // http://stackoverflow.com/questions/7666509/hash-function-for-string uint32_t simple_hash(const char *name) { - const char *s = name; - uint32_t hash = 5381; - int i; + const char *s = name; + uint32_t hash = 5381; + int i; - while((i = *s++)) hash = ((hash << 5) + hash) + i; + while((i = *s++)) hash = ((hash << 5) + hash) + i; - // fprintf(stderr, "HASH: %lu %s\n", hash, name); + // fprintf(stderr, "HASH: %lu %s\n", hash, name); - return hash; + return hash; } */ // http://isthe.com/chongo/tech/comp/fnv/#FNV-1a uint32_t simple_hash(const char *name) { - unsigned char *s = (unsigned char *)name; - uint32_t hval = 0x811c9dc5; - - // FNV-1a algorithm - while (*s) { - // multiply by the 32 bit FNV magic prime mod 2^32 - // NOTE: No need to optimize with left shifts. - // GCC will use imul instruction anyway. - // Tested with 'gcc -O3 -S' - //hval += (hval<<1) + (hval<<4) + (hval<<7) + (hval<<8) + (hval<<24); - hval *= 16777619; - - // xor the bottom with the current octet - hval ^= (uint32_t)*s++; - } - - // fprintf(stderr, "HASH: %u = %s\n", hval, name); - return hval; + unsigned char *s = (unsigned char *) name; + uint32_t hval = 0x811c9dc5; + + // FNV-1a algorithm + while (*s) { + // multiply by the 32 bit FNV magic prime mod 2^32 + // NOTE: No need to optimize with left shifts. + // GCC will use imul instruction anyway. + // Tested with 'gcc -O3 -S' + //hval += (hval<<1) + (hval<<4) + (hval<<7) + (hval<<8) + (hval<<24); + hval *= 16777619; + + // xor the bottom with the current octet + hval ^= (uint32_t) *s++; + } + + // fprintf(stderr, "HASH: %u = %s\n", hval, name); + return hval; +} + +uint32_t simple_uhash(const char *name) { + unsigned char *s = (unsigned char *) name; + uint32_t hval = 0x811c9dc5, c; + + // FNV-1a algorithm + while ((c = *s++)) { + if (unlikely(c >= 'A' && c <= 'Z')) c += 'a' - 'A'; + hval *= 16777619; + hval ^= c; + } + return hval; } /* // http://eternallyconfuzzled.com/tuts/algorithms/jsw_tut_hashing.aspx // one at a time hash uint32_t simple_hash(const char *name) { - unsigned char *s = (unsigned char *)name; - uint32_t h = 0; + unsigned char *s = (unsigned char *)name; + uint32_t h = 0; - while(*s) { - h += *s++; - h += (h << 10); - h ^= (h >> 6); - } + while(*s) { + h += *s++; + h += (h << 10); + h ^= (h >> 6); + } - h += (h << 3); - h ^= (h >> 11); - h += (h << 15); + h += (h << 3); + h ^= (h >> 11); + h += (h << 15); - // fprintf(stderr, "HASH: %u = %s\n", h, name); + // fprintf(stderr, "HASH: %u = %s\n", h, name); - return h; + return h; } */ -void strreverse(char* begin, char* end) -{ - char aux; - - while (end > begin) - { - // clearer code. - aux = *end; - *end-- = *begin; - *begin++ = aux; - } +void strreverse(char *begin, char *end) { + char aux; + + while (end > begin) { + // clearer code. + aux = *end; + *end-- = *begin; + *begin++ = aux; + } } -char *mystrsep(char **ptr, char *s) -{ - char *p = ""; - while ( p && !p[0] && *ptr ) p = strsep(ptr, s); - return(p); +char *mystrsep(char **ptr, char *s) { + char *p = ""; + while (p && !p[0] && *ptr) p = strsep(ptr, s); + return (p); } -char *trim(char *s) -{ - // skip leading spaces - // and 'comments' as well!? - while(*s && isspace(*s)) s++; - if(!*s || *s == '#') return NULL; - - // skip tailing spaces - // this way is way faster. Writes only one NUL char. - ssize_t l = strlen(s); - if (--l >= 0) - { - char *p = s + l; - while (p > s && isspace(*p)) p--; - *++p = '\0'; - } - - if(!*s) return NULL; - - return s; +char *trim(char *s) { + // skip leading spaces + // and 'comments' as well!? + while (*s && isspace(*s)) s++; + if (!*s || *s == '#') return NULL; + + // skip tailing spaces + // this way is way faster. Writes only one NUL char. + ssize_t l = strlen(s); + if (--l >= 0) { + char *p = s + l; + while (p > s && isspace(*p)) p--; + *++p = '\0'; + } + + if (!*s) return NULL; + + return s; } -void *mymmap(const char *filename, size_t size, int flags, int ksm) -{ - int fd; - void *mem = NULL; +void *mymmap(const char *filename, size_t size, int flags, int ksm) { + static int log_madvise_1 = 1; +#ifdef MADV_MERGEABLE + static int log_madvise_2 = 1, log_madvise_3 = 1; +#endif + int fd; + void *mem = NULL; - errno = 0; - fd = open(filename, O_RDWR|O_CREAT|O_NOATIME, 0664); - if(fd != -1) { - if(lseek(fd, size, SEEK_SET) == (long)size) { - if(write(fd, "", 1) == 1) { - if(ftruncate(fd, size)) - error("Cannot truncate file '%s' to size %ld. Will use the larger file.", filename, size); + errno = 0; + fd = open(filename, O_RDWR | O_CREAT | O_NOATIME, 0664); + if (fd != -1) { + if (lseek(fd, size, SEEK_SET) == (off_t) size) { + if (write(fd, "", 1) == 1) { + if (ftruncate(fd, size)) + error("Cannot truncate file '%s' to size %zu. Will use the larger file.", filename, size); #ifdef MADV_MERGEABLE - if(flags & MAP_SHARED || !enable_ksm || !ksm) { + if (flags & MAP_SHARED || !enable_ksm || !ksm) { #endif - mem = mmap(NULL, size, PROT_READ|PROT_WRITE, flags, fd, 0); - if(mem) { - int advise = MADV_SEQUENTIAL|MADV_DONTFORK; - if(flags & MAP_SHARED) advise |= MADV_WILLNEED; - - if(madvise(mem, size, advise) != 0) - error("Cannot advise the kernel about the memory usage of file '%s'.", filename); - } + mem = mmap(NULL, size, PROT_READ | PROT_WRITE, flags, fd, 0); + if (mem != MAP_FAILED) { + int advise = MADV_SEQUENTIAL | MADV_DONTFORK; + if (flags & MAP_SHARED) advise |= MADV_WILLNEED; + + if (madvise(mem, size, advise) != 0 && log_madvise_1) { + error("Cannot advise the kernel about the memory usage of file '%s'.", filename); + log_madvise_1--; + } + } #ifdef MADV_MERGEABLE - } - else { - mem = mmap(NULL, size, PROT_READ|PROT_WRITE, flags|MAP_ANONYMOUS, -1, 0); - if(mem) { - if(lseek(fd, 0, SEEK_SET) == 0) { - if(read(fd, mem, size) != (ssize_t)size) - error("Cannot read from file '%s'", filename); - } - else - error("Cannot seek to beginning of file '%s'.", filename); - - // don't use MADV_SEQUENTIAL|MADV_DONTFORK, they disable MADV_MERGEABLE - if(madvise(mem, size, MADV_SEQUENTIAL|MADV_DONTFORK) != 0) - error("Cannot advise the kernel about the memory usage (MADV_SEQUENTIAL|MADV_DONTFORK) of file '%s'.", filename); - - if(madvise(mem, size, MADV_MERGEABLE) != 0) - error("Cannot advise the kernel about the memory usage (MADV_MERGEABLE) of file '%s'.", filename); - } - else - error("Cannot allocate PRIVATE ANONYMOUS memory for KSM for file '%s'.", filename); - } + } else { +/* + // test - load the file into memory + mem = calloc(1, size); + if(mem) { + if(lseek(fd, 0, SEEK_SET) == 0) { + if(read(fd, mem, size) != (ssize_t)size) + error("Cannot read from file '%s'", filename); + } + else + error("Cannot seek to beginning of file '%s'.", filename); + } +*/ + mem = mmap(NULL, size, PROT_READ | PROT_WRITE, flags | MAP_ANONYMOUS, -1, 0); + if (mem != MAP_FAILED) { + if (lseek(fd, 0, SEEK_SET) == 0) { + if (read(fd, mem, size) != (ssize_t) size) + error("Cannot read from file '%s'", filename); + } else + error("Cannot seek to beginning of file '%s'.", filename); + + // don't use MADV_SEQUENTIAL|MADV_DONTFORK, they disable MADV_MERGEABLE + if (madvise(mem, size, MADV_SEQUENTIAL | MADV_DONTFORK) != 0 && log_madvise_2) { + error("Cannot advise the kernel about the memory usage (MADV_SEQUENTIAL|MADV_DONTFORK) of file '%s'.", + filename); + log_madvise_2--; + } + + if (madvise(mem, size, MADV_MERGEABLE) != 0 && log_madvise_3) { + error("Cannot advise the kernel about the memory usage (MADV_MERGEABLE) of file '%s'.", + filename); + log_madvise_3--; + } + } else + error("Cannot allocate PRIVATE ANONYMOUS memory for KSM for file '%s'.", filename); + } #endif - } - else error("Cannot write to file '%s' at position %ld.", filename, size); - } - else error("Cannot seek file '%s' to size %ld.", filename, size); + } else + error("Cannot write to file '%s' at position %zu.", filename, size); + } else + error("Cannot seek file '%s' to size %zu.", filename, size); - close(fd); - } - else error("Cannot create/open file '%s'.", filename); + close(fd); + } else + error("Cannot create/open file '%s'.", filename); - return mem; + return mem; } -int savememory(const char *filename, void *mem, unsigned long size) -{ - char tmpfilename[FILENAME_MAX + 1]; +int savememory(const char *filename, void *mem, size_t size) { + char tmpfilename[FILENAME_MAX + 1]; - snprintfz(tmpfilename, FILENAME_MAX, "%s.%ld.tmp", filename, (long)getpid()); + snprintfz(tmpfilename, FILENAME_MAX, "%s.%ld.tmp", filename, (long) getpid()); - int fd = open(tmpfilename, O_RDWR|O_CREAT|O_NOATIME, 0664); - if(fd < 0) { - error("Cannot create/open file '%s'.", filename); - return -1; - } + int fd = open(tmpfilename, O_RDWR | O_CREAT | O_NOATIME, 0664); + if (fd < 0) { + error("Cannot create/open file '%s'.", filename); + return -1; + } - if(write(fd, mem, size) != (long)size) { - error("Cannot write to file '%s' %ld bytes.", filename, (long)size); - close(fd); - return -1; - } + if (write(fd, mem, size) != (ssize_t) size) { + error("Cannot write to file '%s' %ld bytes.", filename, (long) size); + close(fd); + return -1; + } - close(fd); + close(fd); - int ret = 0; - if(rename(tmpfilename, filename)) { - error("Cannot rename '%s' to '%s'", tmpfilename, filename); - ret = -1; - } + if (rename(tmpfilename, filename)) { + error("Cannot rename '%s' to '%s'", tmpfilename, filename); + return -1; + } - return ret; + return 0; } int fd_is_valid(int fd) { @@ -760,73 +867,69 @@ int fd_is_valid(int fd) { */ unsigned int hz; -void get_HZ(void) -{ - long ticks; +void get_HZ(void) { + long ticks; - if ((ticks = sysconf(_SC_CLK_TCK)) == -1) { - perror("sysconf"); - } + if ((ticks = sysconf(_SC_CLK_TCK)) == -1) { + perror("sysconf"); + } - hz = (unsigned int) ticks; + hz = (unsigned int) ticks; } -pid_t gettid(void) -{ - return syscall(SYS_gettid); +pid_t gettid(void) { + return (pid_t)syscall(SYS_gettid); } char *fgets_trim_len(char *buf, size_t buf_size, FILE *fp, size_t *len) { - char *s = fgets(buf, buf_size, fp); - if(!s) return NULL; + char *s = fgets(buf, (int)buf_size, fp); + if (!s) return NULL; - char *t = s; - if(*t != '\0') { - // find the string end - while (*++t != '\0'); + char *t = s; + if (*t != '\0') { + // find the string end + while (*++t != '\0'); - // trim trailing spaces/newlines/tabs - while (--t > s && *t == '\n') - *t = '\0'; - } + // trim trailing spaces/newlines/tabs + while (--t > s && *t == '\n') + *t = '\0'; + } - if(len) - *len = t - s + 1; + if (len) + *len = t - s + 1; - return s; + return s; } char *strncpyz(char *dst, const char *src, size_t n) { - char *p = dst; + char *p = dst; - while(*src && n--) - *dst++ = *src++; + while (*src && n--) + *dst++ = *src++; - *dst = '\0'; + *dst = '\0'; - return p; + return p; } int vsnprintfz(char *dst, size_t n, const char *fmt, va_list args) { - int size; + int size = vsnprintf(dst, n, fmt, args); - size = vsnprintf(dst, n, fmt, args); + if (unlikely((size_t) size > n)) { + // truncated + size = (int)n; + } - if(unlikely((size_t)size > n)) { - // there is bug in vsnprintf() and it returns - // a number higher to len, but it does not - // overflow the buffer. - size = n; - } - - dst[size] = '\0'; - return size; + dst[size] = '\0'; + return size; } int snprintfz(char *dst, size_t n, const char *fmt, ...) { - va_list args; + va_list args; + + va_start(args, fmt); + int ret = vsnprintfz(dst, n, fmt, args); + va_end(args); - va_start(args, fmt); - return vsnprintfz(dst, n, fmt, args); - va_end(args); + return ret; } diff --git a/src/common.h b/src/common.h index c94f1cde5..a6e85034b 100644 --- a/src/common.h +++ b/src/common.h @@ -1,36 +1,152 @@ +#ifndef NETDATA_COMMON_H +#define NETDATA_COMMON_H 1 + +#ifdef HAVE_CONFIG_H +#include <config.h> +#endif + +#include <pthread.h> +#include <errno.h> + +#include <stdio.h> +#include <stdlib.h> #include <stdarg.h> -#include <sys/time.h> +#include <stddef.h> + +#include <ctype.h> +#include <string.h> +#include <strings.h> + +#include <arpa/inet.h> +#include <netinet/in.h> +#include <netinet/tcp.h> + +#include <dirent.h> +#include <fcntl.h> +#include <getopt.h> +#include <grp.h> +#include <pwd.h> +#include <locale.h> +#include <malloc.h> +#include <netdb.h> +#include <poll.h> +#include <signal.h> +#include <syslog.h> +#include <sys/mman.h> +#include <sys/prctl.h> #include <sys/resource.h> -#include <stdio.h> +#include <sys/socket.h> +#include <sys/stat.h> +#include <sys/statvfs.h> +#include <sys/syscall.h> +#include <sys/time.h> +#include <sys/types.h> +#include <sys/wait.h> +#include <time.h> +#include <unistd.h> +#include <uuid/uuid.h> -#ifndef NETDATA_COMMON_H -#define NETDATA_COMMON_H 1 +#ifdef STORAGE_WITH_MATH +#include <math.h> +#endif #if defined(HAVE_INTTYPES_H) #include <inttypes.h> #elif defined(HAVE_STDINT_H) #include <stdint.h> #endif -#include <sys/types.h> -#include <unistd.h> +#ifdef NETDATA_WITH_ZLIB +#include <zlib.h> +#endif + +#ifndef __ATOMIC_SEQ_CST +#define NETDATA_NO_ATOMIC_INSTRUCTIONS 1 +#endif + +#ifdef __GNUC__ +#define GCC_VERSION (__GNUC__ * 10000 \ + + __GNUC_MINOR__ * 100 \ + + __GNUC_PATCHLEVEL__) + +#if __x86_64__ || __ppc64__ +#define ENVIRONMENT64 +#else +#define ENVIRONMENT32 +#endif + +#else // !__GNUC__ +#define NETDATA_NO_ATOMIC_INSTRUCTIONS 1 +#define ENVIRONMENT32 +#endif // __GNUC__ + +#include "avl.h" +#include "log.h" +#include "global_statistics.h" +#include "storage_number.h" +#include "web_buffer.h" +#include "web_buffer_svg.h" +#include "url.h" +#include "popen.h" + +#include "procfile.h" +#include "appconfig.h" +#include "dictionary.h" +#include "proc_self_mountinfo.h" +#include "plugin_checks.h" +#include "plugin_idlejitter.h" +#include "plugin_nfacct.h" +#include "plugin_proc.h" +#include "plugin_tc.h" +#include "plugins_d.h" + +#include "eval.h" +#include "health.h" + +#include "rrd.h" +#include "rrd2json.h" + +#include "web_client.h" +#include "web_server.h" + +#include "registry.h" +#include "daemon.h" +#include "main.h" +#include "unit_test.h" + +#ifdef abs +#undef abs +#endif #define abs(x) ((x < 0)? -x : x) -#define usecdiff(now, last) (((((now)->tv_sec * 1000000ULL) + (now)->tv_usec) - (((last)->tv_sec * 1000000ULL) + (last)->tv_usec))) + +extern unsigned long long usec_dt(struct timeval *now, struct timeval *old); +extern unsigned long long timeval_usec(struct timeval *tv); + +// #define usec_dt(now, last) (((((now)->tv_sec * 1000000ULL) + (now)->tv_usec) - (((last)->tv_sec * 1000000ULL) + (last)->tv_usec))) extern void netdata_fix_chart_id(char *s); extern void netdata_fix_chart_name(char *s); extern uint32_t simple_hash(const char *name); +extern uint32_t simple_uhash(const char *name); + extern void strreverse(char* begin, char* end); extern char *mystrsep(char **ptr, char *s); extern char *trim(char *s); extern char *strncpyz(char *dst, const char *src, size_t n); extern int vsnprintfz(char *dst, size_t n, const char *fmt, va_list args); -extern int snprintfz(char *dst, size_t n, const char *fmt, ...); +extern int snprintfz(char *dst, size_t n, const char *fmt, ...) __attribute__ (( format (printf, 3, 4))); + +// memory allocation functions that handle failures +extern char *strdupz(const char *s); +extern void *callocz(size_t nmemb, size_t size); +extern void *mallocz(size_t size); +extern void freez(void *ptr); +extern void *reallocz(void *ptr, size_t size); extern void *mymmap(const char *filename, size_t size, int flags, int ksm); -extern int savememory(const char *filename, void *mem, unsigned long size); +extern int savememory(const char *filename, void *mem, size_t size); extern int fd_is_valid(int fd); @@ -43,7 +159,8 @@ extern void get_HZ(void); extern pid_t gettid(void); -extern unsigned long long timems(void); +extern unsigned long long time_usec(void); +extern int sleep_usec(unsigned long long usec); extern char *fgets_trim_len(char *buf, size_t buf_size, FILE *fp, size_t *len); diff --git a/src/daemon.c b/src/daemon.c index 2a56ae0cc..bc4614f07 100644 --- a/src/daemon.c +++ b/src/daemon.c @@ -1,315 +1,230 @@ -#ifdef HAVE_CONFIG_H -#include <config.h> -#endif -#include <stdio.h> -#include <stdlib.h> -#include <unistd.h> -#include <fcntl.h> -#include <errno.h> -#include <signal.h> -#include <string.h> -#include <sys/types.h> -#include <pwd.h> -#include <grp.h> -#include <pthread.h> -#include <sys/wait.h> -#include <sys/stat.h> - #include "common.h" -#include "appconfig.h" -#include "log.h" -#include "web_client.h" -#include "plugins_d.h" -#include "rrd.h" -#include "popen.h" -#include "main.h" -#include "daemon.h" +#include <sched.h> char pidfile[FILENAME_MAX + 1] = ""; -int pidfd = -1; -void sig_handler(int signo) +void sig_handler_exit(int signo) +{ + if(signo) { + error_log_limit_unlimited(); + error("Received signal %d. Exiting...", signo); + netdata_exit = 1; + } +} + +void sig_handler_logrotate(int signo) { - if(signo) - netdata_exit = 1; + if(signo) { + error_log_limit_reset(); + info("Received signal %d to re-open the log files", signo); + reopen_all_log_files(); + } } -int become_user(const char *username) +void sig_handler_save(int signo) { - struct passwd *pw = getpwnam(username); - if(!pw) { - error("User %s is not present.", username); - return -1; - } - - uid_t uid = pw->pw_uid; - gid_t gid = pw->pw_gid; - - int ngroups = sysconf(_SC_NGROUPS_MAX); - gid_t *supplementary_groups = NULL; - if(ngroups) { - supplementary_groups = malloc(sizeof(gid_t) * ngroups); - if(supplementary_groups) { - if(getgrouplist(username, gid, supplementary_groups, &ngroups) == -1) { - error("Cannot get supplementary groups of user '%s'.", username); - free(supplementary_groups); - supplementary_groups = NULL; - ngroups = 0; - } - } - else fatal("Cannot allocate memory for %d supplementary groups", ngroups); - } - - if(pidfile[0] && getuid() != uid) { - // we are dropping privileges - if(chown(pidfile, uid, gid) != 0) - error("Cannot chown pidfile '%s' to user '%s'", pidfile, username); - - else if(pidfd != -1) { - // not need to keep it open - close(pidfd); - pidfd = -1; - } - } - else if(pidfd != -1) { - // not need to keep it open - close(pidfd); - pidfd = -1; - } - - if(supplementary_groups && ngroups) { - if(setgroups(ngroups, supplementary_groups) == -1) - error("Cannot set supplementary groups for user '%s'", username); - - free(supplementary_groups); - supplementary_groups = NULL; - ngroups = 0; - } - - if(setresgid(gid, gid, gid) != 0) { - error("Cannot switch to user's %s group (gid: %d).", username, gid); - return -1; - } - - if(setresuid(uid, uid, uid) != 0) { - error("Cannot switch to user %s (uid: %d).", username, uid); - return -1; - } - - if(setgid(gid) != 0) { - error("Cannot switch to user's %s group (gid: %d).", username, gid); - return -1; - } - if(setegid(gid) != 0) { - error("Cannot effectively switch to user's %s group (gid: %d).", username, gid); - return -1; - } - if(setuid(uid) != 0) { - error("Cannot switch to user %s (uid: %d).", username, uid); - return -1; - } - if(seteuid(uid) != 0) { - error("Cannot effectively switch to user %s (uid: %d).", username, uid); - return -1; - } - - return(0); + if(signo) { + error_log_limit_reset(); + info("Received signal %d to save the database...", signo); + rrdset_save_all(); + } +} + +void sig_handler_reload_health(int signo) +{ + if(signo) { + error_log_limit_reset(); + info("Received signal %d to reload health configuration...", signo); + health_reload(); + } +} + +static void chown_open_file(int fd, uid_t uid, gid_t gid) { + if(fd == -1) return; + + struct stat buf; + + if(fstat(fd, &buf) == -1) { + error("Cannot fstat() fd %d", fd); + return; + } + + if((buf.st_uid != uid || buf.st_gid != gid) && S_ISREG(buf.st_mode)) { + if(fchown(fd, uid, gid) == -1) + error("Cannot fchown() fd %d.", fd); + } +} + +int become_user(const char *username, int pid_fd) +{ + struct passwd *pw = getpwnam(username); + if(!pw) { + error("User %s is not present.", username); + return -1; + } + + uid_t uid = pw->pw_uid; + gid_t gid = pw->pw_gid; + + int ngroups = (int)sysconf(_SC_NGROUPS_MAX); + gid_t *supplementary_groups = NULL; + if(ngroups) { + supplementary_groups = mallocz(sizeof(gid_t) * ngroups); + if(getgrouplist(username, gid, supplementary_groups, &ngroups) == -1) { + error("Cannot get supplementary groups of user '%s'.", username); + freez(supplementary_groups); + supplementary_groups = NULL; + ngroups = 0; + } + } + + chown_open_file(STDOUT_FILENO, uid, gid); + chown_open_file(STDERR_FILENO, uid, gid); + chown_open_file(stdaccess_fd, uid, gid); + chown_open_file(pid_fd, uid, gid); + + if(supplementary_groups && ngroups) { + if(setgroups(ngroups, supplementary_groups) == -1) + error("Cannot set supplementary groups for user '%s'", username); + + freez(supplementary_groups); + supplementary_groups = NULL; + ngroups = 0; + } + + if(setresgid(gid, gid, gid) != 0) { + error("Cannot switch to user's %s group (gid: %u).", username, gid); + return -1; + } + + if(setresuid(uid, uid, uid) != 0) { + error("Cannot switch to user %s (uid: %u).", username, uid); + return -1; + } + + if(setgid(gid) != 0) { + error("Cannot switch to user's %s group (gid: %u).", username, gid); + return -1; + } + if(setegid(gid) != 0) { + error("Cannot effectively switch to user's %s group (gid: %u).", username, gid); + return -1; + } + if(setuid(uid) != 0) { + error("Cannot switch to user %s (uid: %u).", username, uid); + return -1; + } + if(seteuid(uid) != 0) { + error("Cannot effectively switch to user %s (uid: %u).", username, uid); + return -1; + } + + return(0); +} + +void oom_score_adj(int score) { + int done = 0; + int fd = open("/proc/self/oom_score_adj", O_WRONLY); + if(fd != -1) { + char buf[10 + 1]; + ssize_t len = snprintfz(buf, 10, "%d", score); + if(write(fd, buf, len) == len) done = 1; + close(fd); + } + + if(!done) + error("Cannot adjust my Out-Of-Memory score to %d.", score); + else + info("Adjusted my Out-Of-Memory score to %d.", score); +} + +int sched_setscheduler_idle(void) { +#ifdef SCHED_IDLE + const struct sched_param param = { + .sched_priority = 0 + }; + + int i = sched_setscheduler(0, SCHED_IDLE, ¶m); + if(i != 0) + error("Cannot adjust my scheduling priority to IDLE."); + else + info("Adjusted my scheduling priority to IDLE."); + + return i; +#else + return -1; +#endif } -int become_daemon(int dont_fork, int close_all_files, const char *user, const char *input, const char *output, const char *error, const char *access, int *access_fd, FILE **access_fp) +int become_daemon(int dont_fork, const char *user) { - fflush(NULL); - - // open the files before forking - int input_fd = -1, output_fd = -1, error_fd = -1, dev_null; - - if(input && *input) { - if((input_fd = open(input, O_RDONLY, 0666)) == -1) { - error("Cannot open input file '%s'.", input); - return -1; - } - } - - if(output && *output) { - if((output_fd = open(output, O_RDWR | O_APPEND | O_CREAT, 0666)) == -1) { - error("Cannot open output log file '%s'", output); - if(input_fd != -1) close(input_fd); - return -1; - } - } - - if(error && *error) { - if((error_fd = open(error, O_RDWR | O_APPEND | O_CREAT, 0666)) == -1) { - error("Cannot open error log file '%s'.", error); - if(input_fd != -1) close(input_fd); - if(output_fd != -1) close(output_fd); - return -1; - } - } - - if(access && *access && access_fd) { - if((*access_fd = open(access, O_RDWR | O_APPEND | O_CREAT, 0666)) == -1) { - error("Cannot open access log file '%s'", access); - if(input_fd != -1) close(input_fd); - if(output_fd != -1) close(output_fd); - if(error_fd != -1) close(error_fd); - return -1; - } - - if(access_fp) { - *access_fp = fdopen(*access_fd, "w"); - if(!*access_fp) { - error("Cannot migrate file's '%s' fd %d.", access, *access_fd); - if(input_fd != -1) close(input_fd); - if(output_fd != -1) close(output_fd); - if(error_fd != -1) close(error_fd); - close(*access_fd); - *access_fd = -1; - return -1; - } - if(setvbuf(*access_fp, NULL, _IOLBF, 0) != 0) - error("Cannot set line buffering on access.log"); - } - } - - if((dev_null = open("/dev/null", O_RDWR, 0666)) == -1) { - perror("Cannot open /dev/null"); - if(input_fd != -1) close(input_fd); - if(output_fd != -1) close(output_fd); - if(error_fd != -1) close(error_fd); - if(access && access_fd && *access_fd != -1) { - close(*access_fd); - *access_fd = -1; - if(access_fp) { - fclose(*access_fp); - *access_fp = NULL; - } - } - return -1; - } - - // all files opened - // lets do it - - if(!dont_fork) { - int i = fork(); - if(i == -1) { - perror("cannot fork"); - exit(1); - } - if(i != 0) { - exit(0); // the parent - } - - // become session leader - if (setsid() < 0) { - perror("Cannot become session leader."); - exit(2); - } - } - - // fork() again - if(!dont_fork) { - int i = fork(); - if(i == -1) { - perror("cannot fork"); - exit(1); - } - if(i != 0) { - exit(0); // the parent - } - } - - // Set new file permissions - umask(0); - - // close all files - if(close_all_files) { - int i; - for(i = (int) (sysconf(_SC_OPEN_MAX) - 1); i > 0; i--) - if( - ((access_fd && i != *access_fd) || !access_fd) - && i != dev_null - && i != input_fd - && i != output_fd - && i != error_fd - && fd_is_valid(i) - ) close(i); - } - else { - close(STDIN_FILENO); - close(STDOUT_FILENO); - close(STDERR_FILENO); - } - - // put the opened files - // to our standard file descriptors - if(input_fd != -1) { - if(input_fd != STDIN_FILENO) { - dup2(input_fd, STDIN_FILENO); - close(input_fd); - } - input_fd = -1; - } - else dup2(dev_null, STDIN_FILENO); - - if(output_fd != -1) { - if(output_fd != STDOUT_FILENO) { - dup2(output_fd, STDOUT_FILENO); - close(output_fd); - } - - if(setvbuf(stdout, NULL, _IOLBF, 0) != 0) - error("Cannot set line buffering on debug.log"); - - output_fd = -1; - } - else dup2(dev_null, STDOUT_FILENO); - - if(error_fd != -1) { - if(error_fd != STDERR_FILENO) { - dup2(error_fd, STDERR_FILENO); - close(error_fd); - } - - if(setvbuf(stderr, NULL, _IOLBF, 0) != 0) - error("Cannot set line buffering on error.log"); - - error_fd = -1; - } - else dup2(dev_null, STDERR_FILENO); - - // close /dev/null - if(dev_null != STDIN_FILENO && dev_null != STDOUT_FILENO && dev_null != STDERR_FILENO) - close(dev_null); - - // generate our pid file - if(pidfile[0]) { - pidfd = open(pidfile, O_RDWR | O_CREAT, 0644); - if(pidfd >= 0) { - if(ftruncate(pidfd, 0) != 0) - error("Cannot truncate pidfile '%s'.", pidfile); - - char b[100]; - sprintf(b, "%d\n", getpid()); - ssize_t i = write(pidfd, b, strlen(b)); - if(i <= 0) - error("Cannot write pidfile '%s'.", pidfile); - - // don't close it, we might need it at exit - // close(pidfd); - } - else error("Failed to open pidfile '%s'.", pidfile); - } - - if(user && *user) { - if(become_user(user) != 0) { - error("Cannot become user '%s'. Continuing as we are.", user); - } - else info("Successfully became user '%s'.", user); - } - else if(pidfd != -1) - close(pidfd); - - return(0); + if(!dont_fork) { + int i = fork(); + if(i == -1) { + perror("cannot fork"); + exit(1); + } + if(i != 0) { + exit(0); // the parent + } + + // become session leader + if (setsid() < 0) { + perror("Cannot become session leader."); + exit(2); + } + + // fork() again + i = fork(); + if(i == -1) { + perror("cannot fork"); + exit(1); + } + if(i != 0) { + exit(0); // the parent + } + } + + // generate our pid file + int pidfd = -1; + if(pidfile[0]) { + pidfd = open(pidfile, O_WRONLY | O_CREAT, 0644); + if(pidfd >= 0) { + if(ftruncate(pidfd, 0) != 0) + error("Cannot truncate pidfile '%s'.", pidfile); + + char b[100]; + sprintf(b, "%d\n", getpid()); + ssize_t i = write(pidfd, b, strlen(b)); + if(i <= 0) + error("Cannot write pidfile '%s'.", pidfile); + } + else error("Failed to open pidfile '%s'.", pidfile); + } + + // Set new file permissions + umask(0002); + + // adjust my Out-Of-Memory score + oom_score_adj(1000); + + // never become a problem + if(sched_setscheduler_idle() != 0) { + if(nice(19) == -1) error("Cannot lower my CPU priority."); + else info("Set my nice value to 19."); + } + + if(user && *user) { + if(become_user(user, pidfd) != 0) { + error("Cannot become user '%s'. Continuing as we are.", user); + } + else info("Successfully became user '%s'.", user); + } + + if(pidfd != -1) { + close(pidfd); + pidfd = -1; + } + + return(0); } diff --git a/src/daemon.h b/src/daemon.h index 0642be3c0..b193602d6 100644 --- a/src/daemon.h +++ b/src/daemon.h @@ -1,15 +1,17 @@ #ifndef NETDATA_DAEMON_H #define NETDATA_DAEMON_H 1 -extern void sig_handler(int signo); +extern void sig_handler_exit(int signo); +extern void sig_handler_save(int signo); +extern void sig_handler_logrotate(int signo); +extern void sig_handler_reload_health(int signo); -extern int become_user(const char *username); +extern int become_user(const char *username, int pid_fd); -extern int become_daemon(int dont_fork, int close_all_files, const char *user, const char *input, const char *output, const char *error, const char *access, int *access_fd, FILE **access_fp); +extern int become_daemon(int dont_fork, const char *user); extern void netdata_cleanup_and_exit(int i); extern char pidfile[]; -extern int pidfd; #endif /* NETDATA_DAEMON_H */ diff --git a/src/dictionary.c b/src/dictionary.c index 1543f4d0e..91d3b45f1 100644 --- a/src/dictionary.c +++ b/src/dictionary.c @@ -1,39 +1,52 @@ -#ifdef HAVE_CONFIG_H -#include <config.h> -#endif +#include "common.h" -#include <pthread.h> -#include <stdlib.h> -#include <string.h> +// ---------------------------------------------------------------------------- +// dictionary statistics -#include "avl.h" -#include "common.h" -#include "log.h" +static inline void NETDATA_DICTIONARY_STATS_INSERTS_PLUS1(DICTIONARY *dict) { + if(likely(dict->stats)) + dict->stats->inserts++; +} +static inline void NETDATA_DICTIONARY_STATS_DELETES_PLUS1(DICTIONARY *dict) { + if(likely(dict->stats)) + dict->stats->deletes++; +} +static inline void NETDATA_DICTIONARY_STATS_SEARCHES_PLUS1(DICTIONARY *dict) { + if(likely(dict->stats)) + dict->stats->searches++; +} +static inline void NETDATA_DICTIONARY_STATS_ENTRIES_PLUS1(DICTIONARY *dict) { + if(likely(dict->stats)) + dict->stats->entries++; +} +static inline void NETDATA_DICTIONARY_STATS_ENTRIES_MINUS1(DICTIONARY *dict) { + if(likely(dict->stats)) + dict->stats->entries--; +} -#include "dictionary.h" // ---------------------------------------------------------------------------- // dictionary locks static inline void dictionary_read_lock(DICTIONARY *dict) { - if(likely(!(dict->flags & DICTIONARY_FLAG_SINGLE_THREADED))) { - // debug(D_DICTIONARY, "Dictionary READ lock"); - pthread_rwlock_rdlock(&dict->rwlock); - } + if(likely(dict->rwlock)) { + // debug(D_DICTIONARY, "Dictionary READ lock"); + pthread_rwlock_rdlock(dict->rwlock); + } } static inline void dictionary_write_lock(DICTIONARY *dict) { - if(likely(!(dict->flags & DICTIONARY_FLAG_SINGLE_THREADED))) { - // debug(D_DICTIONARY, "Dictionary WRITE lock"); - pthread_rwlock_wrlock(&dict->rwlock); - } + if(likely(dict->rwlock)) { + // debug(D_DICTIONARY, "Dictionary WRITE lock"); + pthread_rwlock_wrlock(dict->rwlock); + } } static inline void dictionary_unlock(DICTIONARY *dict) { - if(likely(!(dict->flags & DICTIONARY_FLAG_SINGLE_THREADED))) { - // debug(D_DICTIONARY, "Dictionary UNLOCK lock"); - pthread_rwlock_unlock(&dict->rwlock); - } + if(likely(dict->rwlock)) { + // debug(D_DICTIONARY, "Dictionary UNLOCK lock"); + pthread_rwlock_unlock(dict->rwlock); + } } @@ -41,192 +54,196 @@ static inline void dictionary_unlock(DICTIONARY *dict) { // avl index static int name_value_compare(void* a, void* b) { - if(((NAME_VALUE *)a)->hash < ((NAME_VALUE *)b)->hash) return -1; - else if(((NAME_VALUE *)a)->hash > ((NAME_VALUE *)b)->hash) return 1; - else return strcmp(((NAME_VALUE *)a)->name, ((NAME_VALUE *)b)->name); + if(((NAME_VALUE *)a)->hash < ((NAME_VALUE *)b)->hash) return -1; + else if(((NAME_VALUE *)a)->hash > ((NAME_VALUE *)b)->hash) return 1; + else return strcmp(((NAME_VALUE *)a)->name, ((NAME_VALUE *)b)->name); } -#define dictionary_name_value_index_add_nolock(dict, nv) do { (dict)->inserts++; avl_insert(&((dict)->values_index), (avl *)(nv)); } while(0) -#define dictionary_name_value_index_del_nolock(dict, nv) do { (dict)->deletes++; avl_remove(&(dict->values_index), (avl *)(nv)); } while(0) +#define dictionary_name_value_index_add_nolock(dict, nv) do { NETDATA_DICTIONARY_STATS_INSERTS_PLUS1(dict); avl_insert(&((dict)->values_index), (avl *)(nv)); } while(0) +#define dictionary_name_value_index_del_nolock(dict, nv) do { NETDATA_DICTIONARY_STATS_DELETES_PLUS1(dict); avl_remove(&(dict->values_index), (avl *)(nv)); } while(0) static inline NAME_VALUE *dictionary_name_value_index_find_nolock(DICTIONARY *dict, const char *name, uint32_t hash) { - NAME_VALUE tmp; - tmp.hash = (hash)?hash:simple_hash(name); - tmp.name = (char *)name; + NAME_VALUE tmp; + tmp.hash = (hash)?hash:simple_hash(name); + tmp.name = (char *)name; - dict->searches++; - return (NAME_VALUE *)avl_search(&(dict->values_index), (avl *) &tmp); + NETDATA_DICTIONARY_STATS_SEARCHES_PLUS1(dict); + return (NAME_VALUE *)avl_search(&(dict->values_index), (avl *) &tmp); } // ---------------------------------------------------------------------------- // internal methods static NAME_VALUE *dictionary_name_value_create_nolock(DICTIONARY *dict, const char *name, void *value, size_t value_len, uint32_t hash) { - debug(D_DICTIONARY, "Creating name value entry for name '%s'.", name); - - NAME_VALUE *nv = calloc(1, sizeof(NAME_VALUE)); - if(unlikely(!nv)) fatal("Cannot allocate name_value of size %z", sizeof(NAME_VALUE)); + debug(D_DICTIONARY, "Creating name value entry for name '%s'.", name); - if(dict->flags & DICTIONARY_FLAG_NAME_LINK_DONT_CLONE) - nv->name = (char *)name; - else { - nv->name = strdup(name); - if (unlikely(!nv->name)) - fatal("Cannot allocate name_value.name of size %z", strlen(name)); - } + NAME_VALUE *nv = callocz(1, sizeof(NAME_VALUE)); - nv->hash = (hash)?hash:simple_hash(nv->name); + if(dict->flags & DICTIONARY_FLAG_NAME_LINK_DONT_CLONE) + nv->name = (char *)name; + else { + nv->name = strdupz(name); + } - if(dict->flags & DICTIONARY_FLAG_VALUE_LINK_DONT_CLONE) - nv->value = value; - else { - nv->value = malloc(value_len); - if (unlikely(!nv->value)) - fatal("Cannot allocate name_value.value of size %z", value_len); + nv->hash = (hash)?hash:simple_hash(nv->name); - memcpy(nv->value, value, value_len); - } + if(dict->flags & DICTIONARY_FLAG_VALUE_LINK_DONT_CLONE) + nv->value = value; + else { + nv->value = mallocz(value_len); + memcpy(nv->value, value, value_len); + } - // index it - dictionary_name_value_index_add_nolock(dict, nv); - dict->entries++; + // index it + dictionary_name_value_index_add_nolock(dict, nv); + NETDATA_DICTIONARY_STATS_ENTRIES_PLUS1(dict); - return nv; + return nv; } static void dictionary_name_value_destroy_nolock(DICTIONARY *dict, NAME_VALUE *nv) { - debug(D_DICTIONARY, "Destroying name value entry for name '%s'.", nv->name); + debug(D_DICTIONARY, "Destroying name value entry for name '%s'.", nv->name); - dictionary_name_value_index_del_nolock(dict, nv); + dictionary_name_value_index_del_nolock(dict, nv); - dict->entries--; + NETDATA_DICTIONARY_STATS_ENTRIES_MINUS1(dict); - if(!(dict->flags & DICTIONARY_FLAG_VALUE_LINK_DONT_CLONE)) { - debug(D_REGISTRY, "Dictionary freeing value of '%s'", nv->name); - free(nv->value); - } + if(!(dict->flags & DICTIONARY_FLAG_VALUE_LINK_DONT_CLONE)) { + debug(D_REGISTRY, "Dictionary freeing value of '%s'", nv->name); + freez(nv->value); + } - if(!(dict->flags & DICTIONARY_FLAG_NAME_LINK_DONT_CLONE)) { - debug(D_REGISTRY, "Dictionary freeing name '%s'", nv->name); - free(nv->name); - } + if(!(dict->flags & DICTIONARY_FLAG_NAME_LINK_DONT_CLONE)) { + debug(D_REGISTRY, "Dictionary freeing name '%s'", nv->name); + freez(nv->name); + } - free(nv); + freez(nv); } // ---------------------------------------------------------------------------- // API - basic methods -DICTIONARY *dictionary_create(uint32_t flags) { - debug(D_DICTIONARY, "Creating dictionary."); +DICTIONARY *dictionary_create(uint8_t flags) { + debug(D_DICTIONARY, "Creating dictionary."); + + DICTIONARY *dict = callocz(1, sizeof(DICTIONARY)); - DICTIONARY *dict = calloc(1, sizeof(DICTIONARY)); - if(unlikely(!dict)) fatal("Cannot allocate DICTIONARY"); + if(flags & DICTIONARY_FLAG_WITH_STATISTICS) + dict->stats = callocz(1, sizeof(struct dictionary_stats)); - avl_init(&dict->values_index, name_value_compare); - pthread_rwlock_init(&dict->rwlock, NULL); + if(!(flags & DICTIONARY_FLAG_SINGLE_THREADED)) { + dict->rwlock = callocz(1, sizeof(pthread_rwlock_t)); + pthread_rwlock_init(dict->rwlock, NULL); + } - dict->flags = flags; + avl_init(&dict->values_index, name_value_compare); + dict->flags = flags; - return dict; + return dict; } void dictionary_destroy(DICTIONARY *dict) { - debug(D_DICTIONARY, "Destroying dictionary."); + debug(D_DICTIONARY, "Destroying dictionary."); - dictionary_write_lock(dict); + dictionary_write_lock(dict); - while(dict->values_index.root) - dictionary_name_value_destroy_nolock(dict, (NAME_VALUE *)dict->values_index.root); + while(dict->values_index.root) + dictionary_name_value_destroy_nolock(dict, (NAME_VALUE *)dict->values_index.root); - dictionary_unlock(dict); + dictionary_unlock(dict); - free(dict); + if(dict->stats) + freez(dict->stats); + + if(dict->rwlock) + freez(dict->rwlock); + + freez(dict); } // ---------------------------------------------------------------------------- void *dictionary_set(DICTIONARY *dict, const char *name, void *value, size_t value_len) { - debug(D_DICTIONARY, "SET dictionary entry with name '%s'.", name); - - uint32_t hash = simple_hash(name); + debug(D_DICTIONARY, "SET dictionary entry with name '%s'.", name); - dictionary_write_lock(dict); + uint32_t hash = simple_hash(name); - NAME_VALUE *nv = dictionary_name_value_index_find_nolock(dict, name, hash); - if(unlikely(!nv)) { - debug(D_DICTIONARY, "Dictionary entry with name '%s' not found. Creating a new one.", name); + dictionary_write_lock(dict); - nv = dictionary_name_value_create_nolock(dict, name, value, value_len, hash); - if(unlikely(!nv)) - fatal("Cannot create name_value."); - } - else { - debug(D_DICTIONARY, "Dictionary entry with name '%s' found. Changing its value.", name); + NAME_VALUE *nv = dictionary_name_value_index_find_nolock(dict, name, hash); + if(unlikely(!nv)) { + debug(D_DICTIONARY, "Dictionary entry with name '%s' not found. Creating a new one.", name); - if(dict->flags & DICTIONARY_FLAG_VALUE_LINK_DONT_CLONE) { - debug(D_REGISTRY, "Dictionary: linking value to '%s'", name); - nv->value = value; - } - else { - debug(D_REGISTRY, "Dictionary: cloning value to '%s'", name); + nv = dictionary_name_value_create_nolock(dict, name, value, value_len, hash); + if(unlikely(!nv)) + fatal("Cannot create name_value."); + } + else { + debug(D_DICTIONARY, "Dictionary entry with name '%s' found. Changing its value.", name); - void *value = malloc(value_len), - *old = nv->value; + if(dict->flags & DICTIONARY_FLAG_VALUE_LINK_DONT_CLONE) { + debug(D_REGISTRY, "Dictionary: linking value to '%s'", name); + nv->value = value; + } + else { + debug(D_REGISTRY, "Dictionary: cloning value to '%s'", name); - if(unlikely(!nv->value)) - fatal("Cannot allocate value of size %z", value_len); + // copy the new value without breaking + // any other thread accessing the same entry + void *new = mallocz(value_len), + *old = nv->value; - memcpy(value, value, value_len); - nv->value = value; + memcpy(new, value, value_len); + nv->value = new; - debug(D_REGISTRY, "Dictionary: freeing old value of '%s'", name); - free(old); - } - } + debug(D_REGISTRY, "Dictionary: freeing old value of '%s'", name); + freez(old); + } + } - dictionary_unlock(dict); + dictionary_unlock(dict); - return nv->value; + return nv->value; } void *dictionary_get(DICTIONARY *dict, const char *name) { - debug(D_DICTIONARY, "GET dictionary entry with name '%s'.", name); + debug(D_DICTIONARY, "GET dictionary entry with name '%s'.", name); - dictionary_read_lock(dict); - NAME_VALUE *nv = dictionary_name_value_index_find_nolock(dict, name, 0); - dictionary_unlock(dict); + dictionary_read_lock(dict); + NAME_VALUE *nv = dictionary_name_value_index_find_nolock(dict, name, 0); + dictionary_unlock(dict); - if(unlikely(!nv)) { - debug(D_DICTIONARY, "Not found dictionary entry with name '%s'.", name); - return NULL; - } + if(unlikely(!nv)) { + debug(D_DICTIONARY, "Not found dictionary entry with name '%s'.", name); + return NULL; + } - debug(D_DICTIONARY, "Found dictionary entry with name '%s'.", name); - return nv->value; + debug(D_DICTIONARY, "Found dictionary entry with name '%s'.", name); + return nv->value; } int dictionary_del(DICTIONARY *dict, const char *name) { - int ret; + int ret; - debug(D_DICTIONARY, "DEL dictionary entry with name '%s'.", name); + debug(D_DICTIONARY, "DEL dictionary entry with name '%s'.", name); - dictionary_write_lock(dict); + dictionary_write_lock(dict); - NAME_VALUE *nv = dictionary_name_value_index_find_nolock(dict, name, 0); - if(unlikely(!nv)) { - debug(D_DICTIONARY, "Not found dictionary entry with name '%s'.", name); - ret = -1; - } - else { - debug(D_DICTIONARY, "Found dictionary entry with name '%s'.", name); - dictionary_name_value_destroy_nolock(dict, nv); - ret = 0; - } + NAME_VALUE *nv = dictionary_name_value_index_find_nolock(dict, name, 0); + if(unlikely(!nv)) { + debug(D_DICTIONARY, "Not found dictionary entry with name '%s'.", name); + ret = -1; + } + else { + debug(D_DICTIONARY, "Found dictionary entry with name '%s'.", name); + dictionary_name_value_destroy_nolock(dict, nv); + ret = 0; + } - dictionary_unlock(dict); + dictionary_unlock(dict); - return ret; + return ret; } @@ -236,36 +253,36 @@ int dictionary_del(DICTIONARY *dict, const char *name) { // do not user other dictionary calls while walking the dictionary - deadlock! static int dictionary_walker(avl *a, int (*callback)(void *entry, void *data), void *data) { - int total = 0, ret = 0; + int total = 0, ret = 0; - if(a->right) { - ret = dictionary_walker(a->right, callback, data); - if(ret < 0) return ret; - total += ret; - } + if(a->avl_link[0]) { + ret = dictionary_walker(a->avl_link[0], callback, data); + if(ret < 0) return ret; + total += ret; + } - ret = callback(((NAME_VALUE *)a)->value, data); - if(ret < 0) return ret; - total += ret; + ret = callback(((NAME_VALUE *)a)->value, data); + if(ret < 0) return ret; + total += ret; - if(a->left) { - dictionary_walker(a->left, callback, data); - if (ret < 0) return ret; - total += ret; - } + if(a->avl_link[1]) { + ret = dictionary_walker(a->avl_link[1], callback, data); + if (ret < 0) return ret; + total += ret; + } - return total; + return total; } int dictionary_get_all(DICTIONARY *dict, int (*callback)(void *entry, void *data), void *data) { - int ret = 0; + int ret = 0; - dictionary_read_lock(dict); + dictionary_read_lock(dict); - if(likely(dict->values_index.root)) - ret = dictionary_walker(dict->values_index.root, callback, data); + if(likely(dict->values_index.root)) + ret = dictionary_walker(dict->values_index.root, callback, data); - dictionary_unlock(dict); + dictionary_unlock(dict); - return ret; + return ret; } diff --git a/src/dictionary.h b/src/dictionary.h index 575f28271..6bebbfa85 100644 --- a/src/dictionary.h +++ b/src/dictionary.h @@ -1,45 +1,44 @@ -#include <pthread.h> - -#include "web_buffer.h" -#include "avl.h" - #ifndef NETDATA_DICTIONARY_H #define NETDATA_DICTIONARY_H 1 +struct dictionary_stats { + unsigned long long inserts; + unsigned long long deletes; + unsigned long long searches; + unsigned long long entries; +}; + typedef struct name_value { - avl avl; // the index - this has to be first! + avl avl; // the index - this has to be first! - uint32_t hash; // a simple hash to speed up searching - // we first compare hashes, and only if the hashes are equal we do string comparisons + uint32_t hash; // a simple hash to speed up searching + // we first compare hashes, and only if the hashes are equal we do string comparisons - char *name; - void *value; + char *name; + void *value; } NAME_VALUE; typedef struct dictionary { - avl_tree values_index; - - uint8_t flags; + avl_tree values_index; - unsigned long long inserts; - unsigned long long deletes; - unsigned long long searches; - unsigned long long entries; + uint8_t flags; - pthread_rwlock_t rwlock; + struct dictionary_stats *stats; + pthread_rwlock_t *rwlock; } DICTIONARY; -#define DICTIONARY_FLAG_DEFAULT 0x00000000 -#define DICTIONARY_FLAG_SINGLE_THREADED 0x00000001 -#define DICTIONARY_FLAG_VALUE_LINK_DONT_CLONE 0x00000002 -#define DICTIONARY_FLAG_NAME_LINK_DONT_CLONE 0x00000004 +#define DICTIONARY_FLAG_DEFAULT 0x00000000 +#define DICTIONARY_FLAG_SINGLE_THREADED 0x00000001 +#define DICTIONARY_FLAG_VALUE_LINK_DONT_CLONE 0x00000002 +#define DICTIONARY_FLAG_NAME_LINK_DONT_CLONE 0x00000004 +#define DICTIONARY_FLAG_WITH_STATISTICS 0x00000008 -extern DICTIONARY *dictionary_create(uint32_t flags); +extern DICTIONARY *dictionary_create(uint8_t flags); extern void dictionary_destroy(DICTIONARY *dict); extern void *dictionary_set(DICTIONARY *dict, const char *name, void *value, size_t value_len); extern void *dictionary_get(DICTIONARY *dict, const char *name); extern int dictionary_del(DICTIONARY *dict, const char *name); -extern int dictionary_get_all(DICTIONARY *dict, int (*callback)(void *entry, void *data), void *data); +extern int dictionary_get_all(DICTIONARY *dict, int (*callback)(void *entry, void *d), void *data); #endif /* NETDATA_DICTIONARY_H */ diff --git a/src/eval.c b/src/eval.c new file mode 100644 index 000000000..8866ee95d --- /dev/null +++ b/src/eval.c @@ -0,0 +1,1119 @@ +#include "common.h" + +// ---------------------------------------------------------------------------- +// data structures for storing the parsed expression in memory + +typedef struct eval_value { + int type; + + union { + calculated_number number; + EVAL_VARIABLE *variable; + struct eval_node *expression; + }; +} EVAL_VALUE; + +typedef struct eval_node { + int id; + unsigned char operator; + int precedence; + + int count; + EVAL_VALUE ops[]; +} EVAL_NODE; + +// these are used for EVAL_NODE.operator +// they are used as internal IDs to identify an operator +// THEY ARE NOT USED FOR PARSING OPERATORS LIKE THAT +#define EVAL_OPERATOR_NOP '\0' +#define EVAL_OPERATOR_EXPRESSION_OPEN '(' +#define EVAL_OPERATOR_EXPRESSION_CLOSE ')' +#define EVAL_OPERATOR_NOT '!' +#define EVAL_OPERATOR_PLUS '+' +#define EVAL_OPERATOR_MINUS '-' +#define EVAL_OPERATOR_AND '&' +#define EVAL_OPERATOR_OR '|' +#define EVAL_OPERATOR_GREATER_THAN_OR_EQUAL 'G' +#define EVAL_OPERATOR_LESS_THAN_OR_EQUAL 'L' +#define EVAL_OPERATOR_NOT_EQUAL '~' +#define EVAL_OPERATOR_EQUAL '=' +#define EVAL_OPERATOR_LESS '<' +#define EVAL_OPERATOR_GREATER '>' +#define EVAL_OPERATOR_MULTIPLY '*' +#define EVAL_OPERATOR_DIVIDE '/' +#define EVAL_OPERATOR_SIGN_PLUS 'P' +#define EVAL_OPERATOR_SIGN_MINUS 'M' +#define EVAL_OPERATOR_ABS 'A' +#define EVAL_OPERATOR_IF_THEN_ELSE '?' + +// ---------------------------------------------------------------------------- +// forward function definitions + +static inline void eval_node_free(EVAL_NODE *op); +static inline EVAL_NODE *parse_full_expression(const char **string, int *error); +static inline EVAL_NODE *parse_one_full_operand(const char **string, int *error); +static inline calculated_number eval_node(EVAL_EXPRESSION *exp, EVAL_NODE *op, int *error); +static inline void print_parsed_as_node(BUFFER *out, EVAL_NODE *op, int *error); +static inline void print_parsed_as_constant(BUFFER *out, calculated_number n); + +// ---------------------------------------------------------------------------- +// evaluation of expressions + +static inline calculated_number eval_check_number(calculated_number n, int *error) { + if(unlikely(isnan(n))) { + *error = EVAL_ERROR_VALUE_IS_NAN; + return 0; + } + + if(unlikely(isinf(n))) { + *error = EVAL_ERROR_VALUE_IS_INFINITE; + return 0; + } + + return n; +} + +static inline calculated_number eval_variable(EVAL_EXPRESSION *exp, EVAL_VARIABLE *v, int *error) { + static uint32_t this_hash = 0, now_hash = 0, after_hash = 0, before_hash = 0; + calculated_number n; + + if(unlikely(this_hash == 0)) { + this_hash = simple_hash("this"); + now_hash = simple_hash("now"); + after_hash = simple_hash("after"); + before_hash = simple_hash("before"); + } + + if(v->hash == this_hash && !strcmp(v->name, "this")) { + n = (exp->this)?*exp->this:NAN; + buffer_strcat(exp->error_msg, "[ $this = "); + print_parsed_as_constant(exp->error_msg, n); + buffer_strcat(exp->error_msg, " ] "); + return n; + } + + if(v->hash == after_hash && !strcmp(v->name, "after")) { + n = (exp->after && *exp->after)?*exp->after:NAN; + buffer_strcat(exp->error_msg, "[ $after = "); + print_parsed_as_constant(exp->error_msg, n); + buffer_strcat(exp->error_msg, " ] "); + return n; + } + + if(v->hash == before_hash && !strcmp(v->name, "before")) { + n = (exp->before && *exp->before)?*exp->before:NAN; + buffer_strcat(exp->error_msg, "[ $before = "); + print_parsed_as_constant(exp->error_msg, n); + buffer_strcat(exp->error_msg, " ] "); + return n; + } + + if(v->hash == now_hash && !strcmp(v->name, "now")) { + n = time(NULL); + buffer_strcat(exp->error_msg, "[ $now = "); + print_parsed_as_constant(exp->error_msg, n); + buffer_strcat(exp->error_msg, " ] "); + return n; + } + + if(exp->rrdcalc && health_variable_lookup(v->name, v->hash, exp->rrdcalc, &n)) { + buffer_sprintf(exp->error_msg, "[ $%s = ", v->name); + print_parsed_as_constant(exp->error_msg, n); + buffer_strcat(exp->error_msg, " ] "); + return n; + } + + *error = EVAL_ERROR_UNKNOWN_VARIABLE; + buffer_sprintf(exp->error_msg, "unknown variable '%s'", v->name); + return 0; +} + +static inline calculated_number eval_value(EVAL_EXPRESSION *exp, EVAL_VALUE *v, int *error) { + calculated_number n; + + switch(v->type) { + case EVAL_VALUE_EXPRESSION: + n = eval_node(exp, v->expression, error); + break; + + case EVAL_VALUE_NUMBER: + n = v->number; + break; + + case EVAL_VALUE_VARIABLE: + n = eval_variable(exp, v->variable, error); + break; + + default: + *error = EVAL_ERROR_INVALID_VALUE; + n = 0; + break; + } + + // return eval_check_number(n, error); + return n; +} + +static inline int is_true(calculated_number n) { + if(isnan(n)) return 0; + if(isinf(n)) return 1; + if(n == 0) return 0; + return 1; +} + +calculated_number eval_and(EVAL_EXPRESSION *exp, EVAL_NODE *op, int *error) { + return is_true(eval_value(exp, &op->ops[0], error)) && is_true(eval_value(exp, &op->ops[1], error)); +} +calculated_number eval_or(EVAL_EXPRESSION *exp, EVAL_NODE *op, int *error) { + return is_true(eval_value(exp, &op->ops[0], error)) || is_true(eval_value(exp, &op->ops[1], error)); +} +calculated_number eval_greater_than_or_equal(EVAL_EXPRESSION *exp, EVAL_NODE *op, int *error) { + calculated_number n1 = eval_value(exp, &op->ops[0], error); + calculated_number n2 = eval_value(exp, &op->ops[1], error); + return isgreaterequal(n1, n2); +} +calculated_number eval_less_than_or_equal(EVAL_EXPRESSION *exp, EVAL_NODE *op, int *error) { + calculated_number n1 = eval_value(exp, &op->ops[0], error); + calculated_number n2 = eval_value(exp, &op->ops[1], error); + return islessequal(n1, n2); +} +calculated_number eval_equal(EVAL_EXPRESSION *exp, EVAL_NODE *op, int *error) { + calculated_number n1 = eval_value(exp, &op->ops[0], error); + calculated_number n2 = eval_value(exp, &op->ops[1], error); + if(isnan(n1) && isnan(n2)) return 1; + if(isinf(n1) && isinf(n2)) return 1; + if(isnan(n1) || isnan(n2)) return 0; + if(isinf(n1) || isinf(n2)) return 0; + return n1 == n2; +} +calculated_number eval_not_equal(EVAL_EXPRESSION *exp, EVAL_NODE *op, int *error) { + return !eval_equal(exp, op, error); +} +calculated_number eval_less(EVAL_EXPRESSION *exp, EVAL_NODE *op, int *error) { + calculated_number n1 = eval_value(exp, &op->ops[0], error); + calculated_number n2 = eval_value(exp, &op->ops[1], error); + return isless(n1, n2); +} +calculated_number eval_greater(EVAL_EXPRESSION *exp, EVAL_NODE *op, int *error) { + calculated_number n1 = eval_value(exp, &op->ops[0], error); + calculated_number n2 = eval_value(exp, &op->ops[1], error); + return isgreater(n1, n2); +} +calculated_number eval_plus(EVAL_EXPRESSION *exp, EVAL_NODE *op, int *error) { + calculated_number n1 = eval_value(exp, &op->ops[0], error); + calculated_number n2 = eval_value(exp, &op->ops[1], error); + if(isnan(n1) || isnan(n2)) return NAN; + if(isinf(n1) || isinf(n2)) return INFINITY; + return n1 + n2; +} +calculated_number eval_minus(EVAL_EXPRESSION *exp, EVAL_NODE *op, int *error) { + calculated_number n1 = eval_value(exp, &op->ops[0], error); + calculated_number n2 = eval_value(exp, &op->ops[1], error); + if(isnan(n1) || isnan(n2)) return NAN; + if(isinf(n1) || isinf(n2)) return INFINITY; + return n1 - n2; +} +calculated_number eval_multiply(EVAL_EXPRESSION *exp, EVAL_NODE *op, int *error) { + calculated_number n1 = eval_value(exp, &op->ops[0], error); + calculated_number n2 = eval_value(exp, &op->ops[1], error); + if(isnan(n1) || isnan(n2)) return NAN; + if(isinf(n1) || isinf(n2)) return INFINITY; + return n1 * n2; +} +calculated_number eval_divide(EVAL_EXPRESSION *exp, EVAL_NODE *op, int *error) { + calculated_number n1 = eval_value(exp, &op->ops[0], error); + calculated_number n2 = eval_value(exp, &op->ops[1], error); + if(isnan(n1) || isnan(n2)) return NAN; + if(isinf(n1) || isinf(n2)) return INFINITY; + return n1 / n2; +} +calculated_number eval_nop(EVAL_EXPRESSION *exp, EVAL_NODE *op, int *error) { + return eval_value(exp, &op->ops[0], error); +} +calculated_number eval_not(EVAL_EXPRESSION *exp, EVAL_NODE *op, int *error) { + return !is_true(eval_value(exp, &op->ops[0], error)); +} +calculated_number eval_sign_plus(EVAL_EXPRESSION *exp, EVAL_NODE *op, int *error) { + return eval_value(exp, &op->ops[0], error); +} +calculated_number eval_sign_minus(EVAL_EXPRESSION *exp, EVAL_NODE *op, int *error) { + calculated_number n1 = eval_value(exp, &op->ops[0], error); + if(isnan(n1)) return NAN; + if(isinf(n1)) return INFINITY; + return -n1; +} +calculated_number eval_abs(EVAL_EXPRESSION *exp, EVAL_NODE *op, int *error) { + calculated_number n1 = eval_value(exp, &op->ops[0], error); + if(isnan(n1)) return NAN; + if(isinf(n1)) return INFINITY; + return abs(n1); +} +calculated_number eval_if_then_else(EVAL_EXPRESSION *exp, EVAL_NODE *op, int *error) { + if(is_true(eval_value(exp, &op->ops[0], error))) + return eval_value(exp, &op->ops[1], error); + else + return eval_value(exp, &op->ops[2], error); +} + +static struct operator { + const char *print_as; + char precedence; + char parameters; + char isfunction; + calculated_number (*eval)(EVAL_EXPRESSION *exp, EVAL_NODE *op, int *error); +} operators[256] = { + // this is a random access array + // we always access it with a known EVAL_OPERATOR_X + + [EVAL_OPERATOR_IF_THEN_ELSE] = { "?", 1, 3, 0, eval_if_then_else }, + [EVAL_OPERATOR_AND] = { "&&", 2, 2, 0, eval_and }, + [EVAL_OPERATOR_OR] = { "||", 2, 2, 0, eval_or }, + [EVAL_OPERATOR_GREATER_THAN_OR_EQUAL] = { ">=", 3, 2, 0, eval_greater_than_or_equal }, + [EVAL_OPERATOR_LESS_THAN_OR_EQUAL] = { "<=", 3, 2, 0, eval_less_than_or_equal }, + [EVAL_OPERATOR_NOT_EQUAL] = { "!=", 3, 2, 0, eval_not_equal }, + [EVAL_OPERATOR_EQUAL] = { "==", 3, 2, 0, eval_equal }, + [EVAL_OPERATOR_LESS] = { "<", 3, 2, 0, eval_less }, + [EVAL_OPERATOR_GREATER] = { ">", 3, 2, 0, eval_greater }, + [EVAL_OPERATOR_PLUS] = { "+", 4, 2, 0, eval_plus }, + [EVAL_OPERATOR_MINUS] = { "-", 4, 2, 0, eval_minus }, + [EVAL_OPERATOR_MULTIPLY] = { "*", 5, 2, 0, eval_multiply }, + [EVAL_OPERATOR_DIVIDE] = { "/", 5, 2, 0, eval_divide }, + [EVAL_OPERATOR_NOT] = { "!", 6, 1, 0, eval_not }, + [EVAL_OPERATOR_SIGN_PLUS] = { "+", 6, 1, 0, eval_sign_plus }, + [EVAL_OPERATOR_SIGN_MINUS] = { "-", 6, 1, 0, eval_sign_minus }, + [EVAL_OPERATOR_ABS] = { "abs(",6,1, 1, eval_abs }, + [EVAL_OPERATOR_NOP] = { NULL, 7, 1, 0, eval_nop }, + [EVAL_OPERATOR_EXPRESSION_OPEN] = { NULL, 7, 1, 0, eval_nop }, + + // this should exist in our evaluation list + [EVAL_OPERATOR_EXPRESSION_CLOSE] = { NULL, 99, 1, 0, eval_nop } +}; + +#define eval_precedence(operator) (operators[(unsigned char)(operator)].precedence) + +static inline calculated_number eval_node(EVAL_EXPRESSION *exp, EVAL_NODE *op, int *error) { + if(unlikely(op->count != operators[op->operator].parameters)) { + *error = EVAL_ERROR_INVALID_NUMBER_OF_OPERANDS; + return 0; + } + + calculated_number n = operators[op->operator].eval(exp, op, error); + + // return eval_check_number(n, error); + return n; +} + +// ---------------------------------------------------------------------------- +// parsed-as generation + +static inline void print_parsed_as_variable(BUFFER *out, EVAL_VARIABLE *v, int *error) { + (void)error; + buffer_sprintf(out, "$%s", v->name); +} + +static inline void print_parsed_as_constant(BUFFER *out, calculated_number n) { + if(unlikely(isnan(n))) { + buffer_strcat(out, "nan"); + return; + } + + if(unlikely(isinf(n))) { + buffer_strcat(out, "inf"); + return; + } + + char b[100+1], *s; + snprintfz(b, 100, CALCULATED_NUMBER_FORMAT, n); + + s = &b[strlen(b) - 1]; + while(s > b && *s == '0') { + *s ='\0'; + s--; + } + + if(s > b && *s == '.') + *s = '\0'; + + buffer_strcat(out, b); +} + +static inline void print_parsed_as_value(BUFFER *out, EVAL_VALUE *v, int *error) { + switch(v->type) { + case EVAL_VALUE_EXPRESSION: + print_parsed_as_node(out, v->expression, error); + break; + + case EVAL_VALUE_NUMBER: + print_parsed_as_constant(out, v->number); + break; + + case EVAL_VALUE_VARIABLE: + print_parsed_as_variable(out, v->variable, error); + break; + + default: + *error = EVAL_ERROR_INVALID_VALUE; + break; + } +} + +static inline void print_parsed_as_node(BUFFER *out, EVAL_NODE *op, int *error) { + if(unlikely(op->count != operators[op->operator].parameters)) { + *error = EVAL_ERROR_INVALID_NUMBER_OF_OPERANDS; + return; + } + + if(operators[op->operator].parameters == 1) { + + if(operators[op->operator].print_as) + buffer_sprintf(out, "%s", operators[op->operator].print_as); + + //if(op->operator == EVAL_OPERATOR_EXPRESSION_OPEN) + // buffer_strcat(out, "("); + + print_parsed_as_value(out, &op->ops[0], error); + + //if(op->operator == EVAL_OPERATOR_EXPRESSION_OPEN) + // buffer_strcat(out, ")"); + } + + else if(operators[op->operator].parameters == 2) { + buffer_strcat(out, "("); + print_parsed_as_value(out, &op->ops[0], error); + + if(operators[op->operator].print_as) + buffer_sprintf(out, " %s ", operators[op->operator].print_as); + + print_parsed_as_value(out, &op->ops[1], error); + buffer_strcat(out, ")"); + } + else if(op->operator == EVAL_OPERATOR_IF_THEN_ELSE && operators[op->operator].parameters == 3) { + buffer_strcat(out, "("); + print_parsed_as_value(out, &op->ops[0], error); + + if(operators[op->operator].print_as) + buffer_sprintf(out, " %s ", operators[op->operator].print_as); + + print_parsed_as_value(out, &op->ops[1], error); + buffer_strcat(out, " : "); + print_parsed_as_value(out, &op->ops[2], error); + buffer_strcat(out, ")"); + } + + if(operators[op->operator].isfunction) + buffer_strcat(out, ")"); +} + +// ---------------------------------------------------------------------------- +// parsing expressions + +// skip spaces +static inline void skip_spaces(const char **string) { + const char *s = *string; + while(isspace(*s)) s++; + *string = s; +} + +// what character can appear just after an operator keyword +// like NOT AND OR ? +static inline int isoperatorterm_word(const char s) { + if(isspace(s) || s == '(' || s == '$' || s == '!' || s == '-' || s == '+' || isdigit(s) || !s) + return 1; + + return 0; +} + +// what character can appear just after an operator symbol? +static inline int isoperatorterm_symbol(const char s) { + if(isoperatorterm_word(s) || isalpha(s)) + return 1; + + return 0; +} + +// return 1 if the character should never appear in a variable +static inline int isvariableterm(const char s) { + if(isalnum(s) || s == '.' || s == '_') + return 0; + + return 1; +} + +// ---------------------------------------------------------------------------- +// parse operators + +static inline int parse_and(const char **string) { + const char *s = *string; + + // AND + if((s[0] == 'A' || s[0] == 'a') && (s[1] == 'N' || s[1] == 'n') && (s[2] == 'D' || s[2] == 'd') && isoperatorterm_word(s[3])) { + *string = &s[4]; + return 1; + } + + // && + if(s[0] == '&' && s[1] == '&' && isoperatorterm_symbol(s[2])) { + *string = &s[2]; + return 1; + } + + return 0; +} + +static inline int parse_or(const char **string) { + const char *s = *string; + + // OR + if((s[0] == 'O' || s[0] == 'o') && (s[1] == 'R' || s[1] == 'r') && isoperatorterm_word(s[2])) { + *string = &s[3]; + return 1; + } + + // || + if(s[0] == '|' && s[1] == '|' && isoperatorterm_symbol(s[2])) { + *string = &s[2]; + return 1; + } + + return 0; +} + +static inline int parse_greater_than_or_equal(const char **string) { + const char *s = *string; + + // >= + if(s[0] == '>' && s[1] == '=' && isoperatorterm_symbol(s[2])) { + *string = &s[2]; + return 1; + } + + return 0; +} + +static inline int parse_less_than_or_equal(const char **string) { + const char *s = *string; + + // <= + if (s[0] == '<' && s[1] == '=' && isoperatorterm_symbol(s[2])) { + *string = &s[2]; + return 1; + } + + return 0; +} + +static inline int parse_greater(const char **string) { + const char *s = *string; + + // > + if(s[0] == '>' && isoperatorterm_symbol(s[1])) { + *string = &s[1]; + return 1; + } + + return 0; +} + +static inline int parse_less(const char **string) { + const char *s = *string; + + // < + if(s[0] == '<' && isoperatorterm_symbol(s[1])) { + *string = &s[1]; + return 1; + } + + return 0; +} + +static inline int parse_equal(const char **string) { + const char *s = *string; + + // == + if(s[0] == '=' && s[1] == '=' && isoperatorterm_symbol(s[2])) { + *string = &s[2]; + return 1; + } + + // = + if(s[0] == '=' && isoperatorterm_symbol(s[1])) { + *string = &s[1]; + return 1; + } + + return 0; +} + +static inline int parse_not_equal(const char **string) { + const char *s = *string; + + // != + if(s[0] == '!' && s[1] == '=' && isoperatorterm_symbol(s[2])) { + *string = &s[2]; + return 1; + } + + // <> + if(s[0] == '<' && s[1] == '>' && isoperatorterm_symbol(s[2])) { + *string = &s[2]; + } + + return 0; +} + +static inline int parse_not(const char **string) { + const char *s = *string; + + // NOT + if((s[0] == 'N' || s[0] == 'n') && (s[1] == 'O' || s[1] == 'o') && (s[2] == 'T' || s[2] == 't') && isoperatorterm_word(s[3])) { + *string = &s[3]; + return 1; + } + + if(s[0] == '!') { + *string = &s[1]; + return 1; + } + + return 0; +} + +static inline int parse_multiply(const char **string) { + const char *s = *string; + + // * + if(s[0] == '*' && isoperatorterm_symbol(s[1])) { + *string = &s[1]; + return 1; + } + + return 0; +} + +static inline int parse_divide(const char **string) { + const char *s = *string; + + // / + if(s[0] == '/' && isoperatorterm_symbol(s[1])) { + *string = &s[1]; + return 1; + } + + return 0; +} + +static inline int parse_minus(const char **string) { + const char *s = *string; + + // - + if(s[0] == '-' && isoperatorterm_symbol(s[1])) { + *string = &s[1]; + return 1; + } + + return 0; +} + +static inline int parse_plus(const char **string) { + const char *s = *string; + + // + + if(s[0] == '+' && isoperatorterm_symbol(s[1])) { + *string = &s[1]; + return 1; + } + + return 0; +} + +static inline int parse_open_subexpression(const char **string) { + const char *s = *string; + + // ( + if(s[0] == '(') { + *string = &s[1]; + return 1; + } + + return 0; +} + +#define parse_close_function(x) parse_close_subexpression(x) + +static inline int parse_close_subexpression(const char **string) { + const char *s = *string; + + // ) + if(s[0] == ')') { + *string = &s[1]; + return 1; + } + + return 0; +} + +static inline int parse_variable(const char **string, char *buffer, size_t len) { + const char *s = *string; + + // $ + if(s[0] == '$') { + size_t i = 0; + s++; + + while(*s && !isvariableterm(*s) && i < len) + buffer[i++] = *s++; + + buffer[i] = '\0'; + + if(buffer[0]) { + *string = &s[0]; + return 1; + } + } + + return 0; +} + +static inline int parse_constant(const char **string, calculated_number *number) { + char *end = NULL; + calculated_number n = strtold(*string, &end); + if(unlikely(!end || *string == end)) { + *number = 0; + return 0; + } + *number = n; + *string = end; + return 1; +} + +static inline int parse_abs(const char **string) { + const char *s = *string; + + // ABS + if((s[0] == 'A' || s[0] == 'a') && (s[1] == 'B' || s[1] == 'b') && (s[2] == 'S' || s[2] == 's') && s[3] == '(') { + *string = &s[3]; + return 1; + } + + return 0; +} + +static inline int parse_if_then_else(const char **string) { + const char *s = *string; + + // ? + if(s[0] == '?') { + *string = &s[1]; + return 1; + } + + return 0; +} + +static struct operator_parser { + unsigned char id; + int (*parse)(const char **); +} operator_parsers[] = { + // the order in this list is important! + // the first matching will be used + // so place the longer of overlapping ones + // at the top + + { EVAL_OPERATOR_AND, parse_and }, + { EVAL_OPERATOR_OR, parse_or }, + { EVAL_OPERATOR_GREATER_THAN_OR_EQUAL, parse_greater_than_or_equal }, + { EVAL_OPERATOR_LESS_THAN_OR_EQUAL, parse_less_than_or_equal }, + { EVAL_OPERATOR_NOT_EQUAL, parse_not_equal }, + { EVAL_OPERATOR_EQUAL, parse_equal }, + { EVAL_OPERATOR_LESS, parse_less }, + { EVAL_OPERATOR_GREATER, parse_greater }, + { EVAL_OPERATOR_PLUS, parse_plus }, + { EVAL_OPERATOR_MINUS, parse_minus }, + { EVAL_OPERATOR_MULTIPLY, parse_multiply }, + { EVAL_OPERATOR_DIVIDE, parse_divide }, + { EVAL_OPERATOR_IF_THEN_ELSE, parse_if_then_else }, + + /* we should not put in this list the following: + * + * - NOT + * - ( + * - ) + * + * these are handled in code + */ + + // termination + { EVAL_OPERATOR_NOP, NULL } +}; + +static inline unsigned char parse_operator(const char **string, int *precedence) { + skip_spaces(string); + + int i; + for(i = 0 ; operator_parsers[i].parse != NULL ; i++) + if(operator_parsers[i].parse(string)) { + if(precedence) *precedence = eval_precedence(operator_parsers[i].id); + return operator_parsers[i].id; + } + + return EVAL_OPERATOR_NOP; +} + +// ---------------------------------------------------------------------------- +// memory management + +static inline EVAL_NODE *eval_node_alloc(int count) { + static int id = 1; + + EVAL_NODE *op = callocz(1, sizeof(EVAL_NODE) + (sizeof(EVAL_VALUE) * count)); + + op->id = id++; + op->operator = EVAL_OPERATOR_NOP; + op->precedence = eval_precedence(EVAL_OPERATOR_NOP); + op->count = count; + return op; +} + +static inline void eval_node_set_value_to_node(EVAL_NODE *op, int pos, EVAL_NODE *value) { + if(pos >= op->count) + fatal("Invalid request to set position %d of OPERAND that has only %d values", pos + 1, op->count + 1); + + op->ops[pos].type = EVAL_VALUE_EXPRESSION; + op->ops[pos].expression = value; +} + +static inline void eval_node_set_value_to_constant(EVAL_NODE *op, int pos, calculated_number value) { + if(pos >= op->count) + fatal("Invalid request to set position %d of OPERAND that has only %d values", pos + 1, op->count + 1); + + op->ops[pos].type = EVAL_VALUE_NUMBER; + op->ops[pos].number = value; +} + +static inline void eval_node_set_value_to_variable(EVAL_NODE *op, int pos, const char *variable) { + if(pos >= op->count) + fatal("Invalid request to set position %d of OPERAND that has only %d values", pos + 1, op->count + 1); + + op->ops[pos].type = EVAL_VALUE_VARIABLE; + op->ops[pos].variable = callocz(1, sizeof(EVAL_VARIABLE)); + op->ops[pos].variable->name = strdupz(variable); + op->ops[pos].variable->hash = simple_hash(op->ops[pos].variable->name); +} + +static inline void eval_variable_free(EVAL_VARIABLE *v) { + freez(v->name); + freez(v); +} + +static inline void eval_value_free(EVAL_VALUE *v) { + switch(v->type) { + case EVAL_VALUE_EXPRESSION: + eval_node_free(v->expression); + break; + + case EVAL_VALUE_VARIABLE: + eval_variable_free(v->variable); + break; + + default: + break; + } +} + +static inline void eval_node_free(EVAL_NODE *op) { + if(op->count) { + int i; + for(i = op->count - 1; i >= 0 ;i--) + eval_value_free(&op->ops[i]); + } + + freez(op); +} + +// ---------------------------------------------------------------------------- +// the parsing logic + +// helper function to avoid allocations all over the place +static inline EVAL_NODE *parse_next_operand_given_its_operator(const char **string, unsigned char operator_type, int *error) { + EVAL_NODE *sub = parse_one_full_operand(string, error); + if(!sub) return NULL; + + EVAL_NODE *op = eval_node_alloc(1); + op->operator = operator_type; + eval_node_set_value_to_node(op, 0, sub); + return op; +} + +// parse a full operand, including its sign or other associative operator (e.g. NOT) +static inline EVAL_NODE *parse_one_full_operand(const char **string, int *error) { + char variable_buffer[EVAL_MAX_VARIABLE_NAME_LENGTH + 1]; + EVAL_NODE *op1 = NULL; + calculated_number number; + + *error = EVAL_ERROR_OK; + + skip_spaces(string); + if(!(**string)) { + *error = EVAL_ERROR_MISSING_OPERAND; + return NULL; + } + + if(parse_not(string)) { + op1 = parse_next_operand_given_its_operator(string, EVAL_OPERATOR_NOT, error); + op1->precedence = eval_precedence(EVAL_OPERATOR_NOT); + } + else if(parse_plus(string)) { + op1 = parse_next_operand_given_its_operator(string, EVAL_OPERATOR_SIGN_PLUS, error); + op1->precedence = eval_precedence(EVAL_OPERATOR_SIGN_PLUS); + } + else if(parse_minus(string)) { + op1 = parse_next_operand_given_its_operator(string, EVAL_OPERATOR_SIGN_MINUS, error); + op1->precedence = eval_precedence(EVAL_OPERATOR_SIGN_MINUS); + } + else if(parse_abs(string)) { + op1 = parse_next_operand_given_its_operator(string, EVAL_OPERATOR_ABS, error); + op1->precedence = eval_precedence(EVAL_OPERATOR_ABS); + } + else if(parse_open_subexpression(string)) { + EVAL_NODE *sub = parse_full_expression(string, error); + if(sub) { + op1 = eval_node_alloc(1); + op1->operator = EVAL_OPERATOR_EXPRESSION_OPEN; + op1->precedence = eval_precedence(EVAL_OPERATOR_EXPRESSION_OPEN); + eval_node_set_value_to_node(op1, 0, sub); + if(!parse_close_subexpression(string)) { + *error = EVAL_ERROR_MISSING_CLOSE_SUBEXPRESSION; + eval_node_free(op1); + return NULL; + } + } + } + else if(parse_variable(string, variable_buffer, EVAL_MAX_VARIABLE_NAME_LENGTH)) { + op1 = eval_node_alloc(1); + op1->operator = EVAL_OPERATOR_NOP; + eval_node_set_value_to_variable(op1, 0, variable_buffer); + } + else if(parse_constant(string, &number)) { + op1 = eval_node_alloc(1); + op1->operator = EVAL_OPERATOR_NOP; + eval_node_set_value_to_constant(op1, 0, number); + } + else if(**string) + *error = EVAL_ERROR_UNKNOWN_OPERAND; + else + *error = EVAL_ERROR_MISSING_OPERAND; + + return op1; +} + +// parse an operator and the rest of the expression +// precedence processing is handled here +static inline EVAL_NODE *parse_rest_of_expression(const char **string, int *error, EVAL_NODE *op1) { + EVAL_NODE *op2 = NULL; + unsigned char operator; + int precedence; + + operator = parse_operator(string, &precedence); + skip_spaces(string); + + if(operator != EVAL_OPERATOR_NOP) { + op2 = parse_one_full_operand(string, error); + if(!op2) { + // error is already reported + eval_node_free(op1); + return NULL; + } + + EVAL_NODE *op = eval_node_alloc(operators[operator].parameters); + op->operator = operator; + op->precedence = precedence; + + if(operator == EVAL_OPERATOR_IF_THEN_ELSE && op->count == 3) { + skip_spaces(string); + + if(**string != ':') { + eval_node_free(op); + eval_node_free(op1); + eval_node_free(op2); + *error = EVAL_ERROR_IF_THEN_ELSE_MISSING_ELSE; + return NULL; + } + (*string)++; + + skip_spaces(string); + + EVAL_NODE *op3 = parse_one_full_operand(string, error); + if(!op3) { + eval_node_free(op); + eval_node_free(op1); + eval_node_free(op2); + // error is already reported + return NULL; + } + + eval_node_set_value_to_node(op, 2, op3); + } + + eval_node_set_value_to_node(op, 1, op2); + + // precedence processing + // if this operator has a higher precedence compared to its next + // put the next operator on top of us (top = evaluated later) + // function recursion does the rest... + if(op->precedence > op1->precedence && op1->count == 2 && op1->operator != '(' && op1->ops[1].type == EVAL_VALUE_EXPRESSION) { + eval_node_set_value_to_node(op, 0, op1->ops[1].expression); + op1->ops[1].expression = op; + op = op1; + } + else + eval_node_set_value_to_node(op, 0, op1); + + return parse_rest_of_expression(string, error, op); + } + else if(**string == ')') { + ; + } + else if(**string) { + eval_node_free(op1); + op1 = NULL; + *error = EVAL_ERROR_MISSING_OPERATOR; + } + + return op1; +} + +// high level function to parse an expression or a sub-expression +static inline EVAL_NODE *parse_full_expression(const char **string, int *error) { + EVAL_NODE *op1 = NULL; + + op1 = parse_one_full_operand(string, error); + if(!op1) { + *error = EVAL_ERROR_MISSING_OPERAND; + return NULL; + } + + return parse_rest_of_expression(string, error, op1); +} + +// ---------------------------------------------------------------------------- +// public API + +int expression_evaluate(EVAL_EXPRESSION *exp) { + exp->error = EVAL_ERROR_OK; + + buffer_reset(exp->error_msg); + exp->result = eval_node(exp, (EVAL_NODE *)exp->nodes, &exp->error); + + if(exp->error == EVAL_ERROR_OK) + exp->result = eval_check_number(exp->result, &exp->error); + + if(exp->error != EVAL_ERROR_OK) { + exp->result = NAN; + + if(buffer_strlen(exp->error_msg)) + buffer_strcat(exp->error_msg, "; "); + + buffer_sprintf(exp->error_msg, "failed to evaluate expression with error %d (%s)", exp->error, expression_strerror(exp->error)); + return 0; + } + + return 1; +} + +EVAL_EXPRESSION *expression_parse(const char *string, const char **failed_at, int *error) { + const char *s = string; + int err = EVAL_ERROR_OK; + unsigned long pos = 0; + + EVAL_NODE *op = parse_full_expression(&s, &err); + + if(*s) { + if(op) { + eval_node_free(op); + op = NULL; + } + err = EVAL_ERROR_REMAINING_GARBAGE; + } + + if (failed_at) *failed_at = s; + if (error) *error = err; + + if(!op) { + pos = s - string + 1; + error("failed to parse expression '%s': %s at character %lu (i.e.: '%s').", string, expression_strerror(err), pos, s); + return NULL; + } + + BUFFER *out = buffer_create(1024); + print_parsed_as_node(out, op, &err); + if(err != EVAL_ERROR_OK) { + error("failed to re-generate expression '%s' with reason: %s", string, expression_strerror(err)); + eval_node_free(op); + buffer_free(out); + return NULL; + } + + EVAL_EXPRESSION *exp = callocz(1, sizeof(EVAL_EXPRESSION)); + + exp->source = strdupz(string); + exp->parsed_as = strdupz(buffer_tostring(out)); + buffer_free(out); + + exp->error_msg = buffer_create(100); + exp->nodes = (void *)op; + + return exp; +} + +void expression_free(EVAL_EXPRESSION *exp) { + if(!exp) return; + + if(exp->nodes) eval_node_free((EVAL_NODE *)exp->nodes); + freez((void *)exp->source); + freez((void *)exp->parsed_as); + buffer_free(exp->error_msg); + freez(exp); +} + +const char *expression_strerror(int error) { + switch(error) { + case EVAL_ERROR_OK: + return "success"; + + case EVAL_ERROR_MISSING_CLOSE_SUBEXPRESSION: + return "missing closing parenthesis"; + + case EVAL_ERROR_UNKNOWN_OPERAND: + return "unknown operand"; + + case EVAL_ERROR_MISSING_OPERAND: + return "expected operand"; + + case EVAL_ERROR_MISSING_OPERATOR: + return "expected operator"; + + case EVAL_ERROR_REMAINING_GARBAGE: + return "remaining characters after expression"; + + case EVAL_ERROR_INVALID_VALUE: + return "invalid value structure - internal error"; + + case EVAL_ERROR_INVALID_NUMBER_OF_OPERANDS: + return "wrong number of operands for operation - internal error"; + + case EVAL_ERROR_VALUE_IS_NAN: + return "value is unset"; + + case EVAL_ERROR_VALUE_IS_INFINITE: + return "computed value is infinite"; + + case EVAL_ERROR_UNKNOWN_VARIABLE: + return "undefined variable"; + + case EVAL_ERROR_IF_THEN_ELSE_MISSING_ELSE: + return "missing second sub-expression of inline conditional"; + + default: + return "unknown error"; + } +} diff --git a/src/eval.h b/src/eval.h new file mode 100644 index 000000000..b509a9fa8 --- /dev/null +++ b/src/eval.h @@ -0,0 +1,72 @@ +#ifndef NETDATA_EVAL_H +#define NETDATA_EVAL_H + +#define EVAL_MAX_VARIABLE_NAME_LENGTH 300 + +typedef struct eval_variable { + char *name; + uint32_t hash; + struct rrdvar *rrdvar; + struct eval_variable *next; +} EVAL_VARIABLE; + +typedef struct eval_expression { + const char *source; + const char *parsed_as; + + calculated_number *this; + time_t *after; + time_t *before; + + calculated_number result; + + int error; + BUFFER *error_msg; + + // hidden EVAL_NODE * + void *nodes; + + // custom data to be used for looking up variables + struct rrdcalc *rrdcalc; +} EVAL_EXPRESSION; + +#define EVAL_VALUE_INVALID 0 +#define EVAL_VALUE_NUMBER 1 +#define EVAL_VALUE_VARIABLE 2 +#define EVAL_VALUE_EXPRESSION 3 + +// parsing and evaluation +#define EVAL_ERROR_OK 0 + +// parsing errors +#define EVAL_ERROR_MISSING_CLOSE_SUBEXPRESSION 1 +#define EVAL_ERROR_UNKNOWN_OPERAND 2 +#define EVAL_ERROR_MISSING_OPERAND 3 +#define EVAL_ERROR_MISSING_OPERATOR 4 +#define EVAL_ERROR_REMAINING_GARBAGE 5 +#define EVAL_ERROR_IF_THEN_ELSE_MISSING_ELSE 6 + +// evaluation errors +#define EVAL_ERROR_INVALID_VALUE 101 +#define EVAL_ERROR_INVALID_NUMBER_OF_OPERANDS 102 +#define EVAL_ERROR_VALUE_IS_NAN 103 +#define EVAL_ERROR_VALUE_IS_INFINITE 104 +#define EVAL_ERROR_UNKNOWN_VARIABLE 105 + +// parse the given string as an expression and return: +// a pointer to an expression if it parsed OK +// NULL in which case the pointer to error has the error code +extern EVAL_EXPRESSION *expression_parse(const char *string, const char **failed_at, int *error); + +// free all resources allocated for an expression +extern void expression_free(EVAL_EXPRESSION *op); + +// convert an error code to a message +extern const char *expression_strerror(int error); + +// evaluate an expression and return +// 1 = OK, the result is in: expression->result +// 2 = FAILED, the error message is in: buffer_tostring(expression->error_msg) +extern int expression_evaluate(EVAL_EXPRESSION *expression); + +#endif //NETDATA_EVAL_H diff --git a/src/global_statistics.c b/src/global_statistics.c index d4a04efd2..f39a4cf25 100644 --- a/src/global_statistics.c +++ b/src/global_statistics.c @@ -1,18 +1,262 @@ -#ifdef HAVE_CONFIG_H -#include <config.h> +#include "common.h" + +volatile struct global_statistics global_statistics = { + .connected_clients = 0, + .web_requests = 0, + .web_usec = 0, + .bytes_received = 0, + .bytes_sent = 0, + .content_size = 0, + .compressed_content_size = 0 +}; + +pthread_mutex_t global_statistics_mutex = PTHREAD_MUTEX_INITIALIZER; + +inline void global_statistics_lock(void) { + pthread_mutex_lock(&global_statistics_mutex); +} + +inline void global_statistics_unlock(void) { + pthread_mutex_unlock(&global_statistics_mutex); +} + +void finished_web_request_statistics(uint64_t dt, + uint64_t bytes_received, + uint64_t bytes_sent, + uint64_t content_size, + uint64_t compressed_content_size) { +#ifndef NETDATA_NO_ATOMIC_INSTRUCTIONS + uint64_t old_web_usec_max = global_statistics.web_usec_max; + while(dt > old_web_usec_max) + __atomic_compare_exchange(&global_statistics.web_usec_max, &old_web_usec_max, &dt, 1, __ATOMIC_SEQ_CST, __ATOMIC_SEQ_CST); + + __atomic_fetch_add(&global_statistics.web_requests, 1, __ATOMIC_SEQ_CST); + __atomic_fetch_add(&global_statistics.web_usec, dt, __ATOMIC_SEQ_CST); + __atomic_fetch_add(&global_statistics.bytes_received, bytes_received, __ATOMIC_SEQ_CST); + __atomic_fetch_add(&global_statistics.bytes_sent, bytes_sent, __ATOMIC_SEQ_CST); + __atomic_fetch_add(&global_statistics.content_size, content_size, __ATOMIC_SEQ_CST); + __atomic_fetch_add(&global_statistics.compressed_content_size, compressed_content_size, __ATOMIC_SEQ_CST); +#else +#warning NOT using atomic operations - using locks for global statistics + if (web_server_mode == WEB_SERVER_MODE_MULTI_THREADED) + global_statistics_lock(); + + if (dt > global_statistics.web_usec_max) + global_statistics.web_usec_max = dt; + + global_statistics.web_requests++; + global_statistics.web_usec += dt; + global_statistics.bytes_received += bytes_received; + global_statistics.bytes_sent += bytes_sent; + global_statistics.content_size += content_size; + global_statistics.compressed_content_size += compressed_content_size; + + if (web_server_mode == WEB_SERVER_MODE_MULTI_THREADED) + global_statistics_unlock(); #endif -#include <pthread.h> +} -#include "global_statistics.h" +void web_client_connected(void) { +#ifndef NETDATA_NO_ATOMIC_INSTRUCTIONS + __atomic_fetch_add(&global_statistics.connected_clients, 1, __ATOMIC_SEQ_CST); +#else + if (web_server_mode == WEB_SERVER_MODE_MULTI_THREADED) + global_statistics_lock(); -struct global_statistics global_statistics = { 0ULL, 0ULL, 0ULL, 0ULL }; + global_statistics.connected_clients++; -pthread_mutex_t global_statistics_mutex = PTHREAD_MUTEX_INITIALIZER; + if (web_server_mode == WEB_SERVER_MODE_MULTI_THREADED) + global_statistics_unlock(); +#endif +} + +void web_client_disconnected(void) { +#ifndef NETDATA_NO_ATOMIC_INSTRUCTIONS + __atomic_fetch_sub(&global_statistics.connected_clients, 1, __ATOMIC_SEQ_CST); +#else + if (web_server_mode == WEB_SERVER_MODE_MULTI_THREADED) + global_statistics_lock(); -void global_statistics_lock(void) { - pthread_mutex_lock(&global_statistics_mutex); + global_statistics.connected_clients--; + + if (web_server_mode == WEB_SERVER_MODE_MULTI_THREADED) + global_statistics_unlock(); +#endif } -void global_statistics_unlock(void) { - pthread_mutex_unlock(&global_statistics_mutex); + +inline void global_statistics_copy(struct global_statistics *gs, uint8_t options) { +#ifndef NETDATA_NO_ATOMIC_INSTRUCTIONS + gs->connected_clients = __atomic_fetch_add(&global_statistics.connected_clients, 0, __ATOMIC_SEQ_CST); + gs->web_requests = __atomic_fetch_add(&global_statistics.web_requests, 0, __ATOMIC_SEQ_CST); + gs->web_usec = __atomic_fetch_add(&global_statistics.web_usec, 0, __ATOMIC_SEQ_CST); + gs->web_usec_max = __atomic_fetch_add(&global_statistics.web_usec_max, 0, __ATOMIC_SEQ_CST); + gs->bytes_received = __atomic_fetch_add(&global_statistics.bytes_received, 0, __ATOMIC_SEQ_CST); + gs->bytes_sent = __atomic_fetch_add(&global_statistics.bytes_sent, 0, __ATOMIC_SEQ_CST); + gs->content_size = __atomic_fetch_add(&global_statistics.content_size, 0, __ATOMIC_SEQ_CST); + gs->compressed_content_size = __atomic_fetch_add(&global_statistics.compressed_content_size, 0, __ATOMIC_SEQ_CST); + + if(options & GLOBAL_STATS_RESET_WEB_USEC_MAX) { + uint64_t n = 0; + __atomic_compare_exchange(&global_statistics.web_usec_max, &gs->web_usec_max, &n, 1, __ATOMIC_SEQ_CST, + __ATOMIC_SEQ_CST); + } +#else + global_statistics_lock(); + + memcpy(gs, (const void *)&global_statistics, sizeof(struct global_statistics)); + + if (options & GLOBAL_STATS_RESET_WEB_USEC_MAX) + global_statistics.web_usec_max = 0; + + global_statistics_unlock(); +#endif } + +void global_statistics_charts(void) { + static unsigned long long old_web_requests = 0, old_web_usec = 0, + old_content_size = 0, old_compressed_content_size = 0; + + static collected_number compression_ratio = -1, average_response_time = -1; + + static RRDSET *stcpu = NULL, *stcpu_thread = NULL, *stclients = NULL, *streqs = NULL, *stbytes = NULL, *stduration = NULL, + *stcompression = NULL; + + struct global_statistics gs; + struct rusage me, thread; + + global_statistics_copy(&gs, GLOBAL_STATS_RESET_WEB_USEC_MAX); + getrusage(RUSAGE_THREAD, &thread); + getrusage(RUSAGE_SELF, &me); + + if (!stcpu_thread) stcpu_thread = rrdset_find("netdata.plugin_proc_cpu"); + if (!stcpu_thread) { + stcpu_thread = rrdset_create("netdata", "plugin_proc_cpu", NULL, "proc.internal", NULL, + "NetData Proc Plugin CPU usage", "milliseconds/s", 132000, rrd_update_every, + RRDSET_TYPE_STACKED); + + rrddim_add(stcpu_thread, "user", NULL, 1, 1000, RRDDIM_INCREMENTAL); + rrddim_add(stcpu_thread, "system", NULL, 1, 1000, RRDDIM_INCREMENTAL); + } else rrdset_next(stcpu_thread); + + rrddim_set(stcpu_thread, "user", thread.ru_utime.tv_sec * 1000000ULL + thread.ru_utime.tv_usec); + rrddim_set(stcpu_thread, "system", thread.ru_stime.tv_sec * 1000000ULL + thread.ru_stime.tv_usec); + rrdset_done(stcpu_thread); + + // ---------------------------------------------------------------- + + if (!stcpu) stcpu = rrdset_find("netdata.server_cpu"); + if (!stcpu) { + stcpu = rrdset_create("netdata", "server_cpu", NULL, "netdata", NULL, "NetData CPU usage", "milliseconds/s", + 130000, rrd_update_every, RRDSET_TYPE_STACKED); + + rrddim_add(stcpu, "user", NULL, 1, 1000, RRDDIM_INCREMENTAL); + rrddim_add(stcpu, "system", NULL, 1, 1000, RRDDIM_INCREMENTAL); + } else rrdset_next(stcpu); + + rrddim_set(stcpu, "user", me.ru_utime.tv_sec * 1000000ULL + me.ru_utime.tv_usec); + rrddim_set(stcpu, "system", me.ru_stime.tv_sec * 1000000ULL + me.ru_stime.tv_usec); + rrdset_done(stcpu); + + // ---------------------------------------------------------------- + + if (!stclients) stclients = rrdset_find("netdata.clients"); + if (!stclients) { + stclients = rrdset_create("netdata", "clients", NULL, "netdata", NULL, "NetData Web Clients", + "connected clients", 130200, rrd_update_every, RRDSET_TYPE_LINE); + + rrddim_add(stclients, "clients", NULL, 1, 1, RRDDIM_ABSOLUTE); + } else rrdset_next(stclients); + + rrddim_set(stclients, "clients", gs.connected_clients); + rrdset_done(stclients); + + // ---------------------------------------------------------------- + + if (!streqs) streqs = rrdset_find("netdata.requests"); + if (!streqs) { + streqs = rrdset_create("netdata", "requests", NULL, "netdata", NULL, "NetData Web Requests", "requests/s", + 130300, rrd_update_every, RRDSET_TYPE_LINE); + + rrddim_add(streqs, "requests", NULL, 1, 1, RRDDIM_INCREMENTAL); + } else rrdset_next(streqs); + + rrddim_set(streqs, "requests", (collected_number) gs.web_requests); + rrdset_done(streqs); + + // ---------------------------------------------------------------- + + if (!stbytes) stbytes = rrdset_find("netdata.net"); + if (!stbytes) { + stbytes = rrdset_create("netdata", "net", NULL, "netdata", NULL, "NetData Network Traffic", "kilobits/s", + 130000, rrd_update_every, RRDSET_TYPE_AREA); + + rrddim_add(stbytes, "in", NULL, 8, 1024, RRDDIM_INCREMENTAL); + rrddim_add(stbytes, "out", NULL, -8, 1024, RRDDIM_INCREMENTAL); + } else rrdset_next(stbytes); + + rrddim_set(stbytes, "in", (collected_number) gs.bytes_received); + rrddim_set(stbytes, "out", (collected_number) gs.bytes_sent); + rrdset_done(stbytes); + + // ---------------------------------------------------------------- + + if (!stduration) stduration = rrdset_find("netdata.response_time"); + if (!stduration) { + stduration = rrdset_create("netdata", "response_time", NULL, "netdata", NULL, "NetData API Response Time", + "ms/request", 130400, rrd_update_every, RRDSET_TYPE_LINE); + + rrddim_add(stduration, "average", NULL, 1, 1000, RRDDIM_ABSOLUTE); + rrddim_add(stduration, "max", NULL, 1, 1000, RRDDIM_ABSOLUTE); + } else rrdset_next(stduration); + + uint64_t gweb_usec = gs.web_usec; + uint64_t gweb_requests = gs.web_requests; + + uint64_t web_usec = (gweb_usec >= old_web_usec) ? gweb_usec - old_web_usec : 0; + uint64_t web_requests = (gweb_requests >= old_web_requests) ? gweb_requests - old_web_requests : 0; + + old_web_usec = gweb_usec; + old_web_requests = gweb_requests; + + if (web_requests) + average_response_time = (collected_number) (web_usec / web_requests); + + if (unlikely(average_response_time != -1)) + rrddim_set(stduration, "average", average_response_time); + else + rrddim_set(stduration, "average", 0); + + rrddim_set(stduration, "max", ((gs.web_usec_max)?(collected_number)gs.web_usec_max:average_response_time)); + rrdset_done(stduration); + + // ---------------------------------------------------------------- + + if (!stcompression) stcompression = rrdset_find("netdata.compression_ratio"); + if (!stcompression) { + stcompression = rrdset_create("netdata", "compression_ratio", NULL, "netdata", NULL, + "NetData API Responses Compression Savings Ratio", "percentage", 130500, + rrd_update_every, RRDSET_TYPE_LINE); + + rrddim_add(stcompression, "savings", NULL, 1, 1000, RRDDIM_ABSOLUTE); + } else rrdset_next(stcompression); + + // since we don't lock here to read the global statistics + // read the smaller value first + unsigned long long gcompressed_content_size = gs.compressed_content_size; + unsigned long long gcontent_size = gs.content_size; + + unsigned long long compressed_content_size = gcompressed_content_size - old_compressed_content_size; + unsigned long long content_size = gcontent_size - old_content_size; + + old_compressed_content_size = gcompressed_content_size; + old_content_size = gcontent_size; + + if (content_size && content_size >= compressed_content_size) + compression_ratio = ((content_size - compressed_content_size) * 100 * 1000) / content_size; + + if (compression_ratio != -1) + rrddim_set(stcompression, "savings", compression_ratio); + + rrdset_done(stcompression); +}
\ No newline at end of file diff --git a/src/global_statistics.h b/src/global_statistics.h index ce3c3490e..d28aa4401 100644 --- a/src/global_statistics.h +++ b/src/global_statistics.h @@ -5,16 +5,32 @@ // global statistics struct global_statistics { - unsigned long long connected_clients; - unsigned long long web_requests; - unsigned long long bytes_received; - unsigned long long bytes_sent; + volatile uint16_t connected_clients; + volatile uint64_t web_requests; + volatile uint64_t web_usec; + volatile uint64_t web_usec_max; + volatile uint64_t bytes_received; + volatile uint64_t bytes_sent; + volatile uint64_t content_size; + volatile uint64_t compressed_content_size; }; -extern struct global_statistics global_statistics; +extern volatile struct global_statistics global_statistics; extern void global_statistics_lock(void); extern void global_statistics_unlock(void); +extern void finished_web_request_statistics(uint64_t dt, + uint64_t bytes_received, + uint64_t bytes_sent, + uint64_t content_size, + uint64_t compressed_content_size); + +extern void web_client_connected(void); +extern void web_client_disconnected(void); + +#define GLOBAL_STATS_RESET_WEB_USEC_MAX 0x01 +extern void global_statistics_copy(struct global_statistics *gs, uint8_t options); +extern void global_statistics_charts(void); #endif /* NETDATA_GLOBAL_STATISTICS_H */ diff --git a/src/health.c b/src/health.c new file mode 100644 index 000000000..3156cd080 --- /dev/null +++ b/src/health.c @@ -0,0 +1,2194 @@ +#include "common.h" + +#define RRDVAR_MAX_LENGTH 1024 + +static const char *health_default_exec = PLUGINS_DIR "/alarm-email.sh"; +int health_enabled = 1; + +// ---------------------------------------------------------------------------- +// RRDVAR management + +static inline int rrdvar_fix_name(char *variable) { + int fixed = 0; + while(*variable) { + if (!isalnum(*variable) && *variable != '.' && *variable != '_') { + *variable++ = '_'; + fixed++; + } + else + variable++; + } + + return fixed; +} + +int rrdvar_compare(void* a, void* b) { + if(((RRDVAR *)a)->hash < ((RRDVAR *)b)->hash) return -1; + else if(((RRDVAR *)a)->hash > ((RRDVAR *)b)->hash) return 1; + else return strcmp(((RRDVAR *)a)->name, ((RRDVAR *)b)->name); +} + +static inline RRDVAR *rrdvar_index_add(avl_tree_lock *tree, RRDVAR *rv) { + RRDVAR *ret = (RRDVAR *)avl_insert_lock(tree, (avl *)(rv)); + if(ret != rv) + debug(D_VARIABLES, "Request to insert RRDVAR '%s' into index failed. Already exists.", rv->name); + + return ret; +} + +static inline RRDVAR *rrdvar_index_del(avl_tree_lock *tree, RRDVAR *rv) { + RRDVAR *ret = (RRDVAR *)avl_remove_lock(tree, (avl *)(rv)); + if(!ret) + error("Request to remove RRDVAR '%s' from index failed. Not Found.", rv->name); + + return ret; +} + +static inline RRDVAR *rrdvar_index_find(avl_tree_lock *tree, const char *name, uint32_t hash) { + RRDVAR tmp; + tmp.name = (char *)name; + tmp.hash = (hash)?hash:simple_hash(tmp.name); + + return (RRDVAR *)avl_search_lock(tree, (avl *)&tmp); +} + +static inline void rrdvar_free(RRDHOST *host, avl_tree_lock *tree, RRDVAR *rv) { + (void)host; + + if(!rv) return; + + if(tree) + rrdvar_index_del(tree, rv); + + freez(rv->name); + freez(rv); +} + +static inline RRDVAR *rrdvar_create_and_index(const char *scope, avl_tree_lock *tree, const char *name, int type, calculated_number *value) { + char *variable = strdupz(name); + rrdvar_fix_name(variable); + uint32_t hash = simple_hash(variable); + + RRDVAR *rv = rrdvar_index_find(tree, variable, hash); + if(unlikely(!rv)) { + debug(D_VARIABLES, "Variable '%s' not found in scope '%s'. Creating a new one.", variable, scope); + + rv = callocz(1, sizeof(RRDVAR)); + rv->name = variable; + rv->hash = hash; + rv->type = type; + rv->value = value; + + RRDVAR *ret = rrdvar_index_add(tree, rv); + if(unlikely(ret != rv)) { + debug(D_VARIABLES, "Variable '%s' in scope '%s' already exists", variable, scope); + rrdvar_free(NULL, NULL, rv); + rv = NULL; + } + else + debug(D_VARIABLES, "Variable '%s' created in scope '%s'", variable, scope); + } + else { + // already exists + freez(variable); + rv = NULL; + } + + return rv; +} + +// ---------------------------------------------------------------------------- +// RRDVAR lookup + +calculated_number rrdvar2number(RRDVAR *rv) { + switch(rv->type) { + case RRDVAR_TYPE_CALCULATED: { + calculated_number *n = (calculated_number *)rv->value; + return *n; + } + + case RRDVAR_TYPE_TIME_T: { + time_t *n = (time_t *)rv->value; + return *n; + } + + case RRDVAR_TYPE_COLLECTED: { + collected_number *n = (collected_number *)rv->value; + return *n; + } + + case RRDVAR_TYPE_TOTAL: { + total_number *n = (total_number *)rv->value; + return *n; + } + + case RRDVAR_TYPE_INT: { + int *n = (int *)rv->value; + return *n; + } + + default: + error("I don't know how to convert RRDVAR type %d to calculated_number", rv->type); + return NAN; + } +} + +void dump_variable(void *data) { + RRDVAR *rv = (RRDVAR *)data; + debug(D_HEALTH, "%50s : %20.5Lf", rv->name, rrdvar2number(rv)); +} + +int health_variable_lookup(const char *variable, uint32_t hash, RRDCALC *rc, calculated_number *result) { + RRDSET *st = rc->rrdset; + RRDVAR *rv; + + if(!st) return 0; + + rv = rrdvar_index_find(&st->variables_root_index, variable, hash); + if(rv) { + *result = rrdvar2number(rv); + return 1; + } + + rv = rrdvar_index_find(&st->rrdfamily->variables_root_index, variable, hash); + if(rv) { + *result = rrdvar2number(rv); + return 1; + } + + rv = rrdvar_index_find(&st->rrdhost->variables_root_index, variable, hash); + if(rv) { + *result = rrdvar2number(rv); + return 1; + } + + debug(D_HEALTH, "Available local chart '%s' variables:", st->id); + avl_traverse_lock(&st->variables_root_index, dump_variable); + + debug(D_HEALTH, "Available family '%s' variables:", st->rrdfamily->family); + avl_traverse_lock(&st->rrdfamily->variables_root_index, dump_variable); + + debug(D_HEALTH, "Available host '%s' variables:", st->rrdhost->hostname); + avl_traverse_lock(&st->rrdhost->variables_root_index, dump_variable); + + return 0; +} + +// ---------------------------------------------------------------------------- +// RRDSETVAR management + +RRDSETVAR *rrdsetvar_create(RRDSET *st, const char *variable, int type, void *value, uint32_t options) { + debug(D_VARIABLES, "RRDVARSET create for chart id '%s' name '%s' with variable name '%s'", st->id, st->name, variable); + RRDSETVAR *rs = (RRDSETVAR *)callocz(1, sizeof(RRDSETVAR)); + + char buffer[RRDVAR_MAX_LENGTH + 1]; + snprintfz(buffer, RRDVAR_MAX_LENGTH, "%s.%s", st->id, variable); + rs->fullid = strdupz(buffer); + + snprintfz(buffer, RRDVAR_MAX_LENGTH, "%s.%s", st->name, variable); + rs->fullname = strdupz(buffer); + + rs->variable = strdupz(variable); + + rs->type = type; + rs->value = value; + rs->options = options; + rs->rrdset = st; + + rs->local = rrdvar_create_and_index("local", &st->variables_root_index, rs->variable, rs->type, rs->value); + rs->family = rrdvar_create_and_index("family", &st->rrdfamily->variables_root_index, rs->fullid, rs->type, rs->value); + rs->host = rrdvar_create_and_index("host", &st->rrdhost->variables_root_index, rs->fullid, rs->type, rs->value); + rs->family_name = rrdvar_create_and_index("family", &st->rrdfamily->variables_root_index, rs->fullname, rs->type, rs->value); + rs->host_name = rrdvar_create_and_index("host", &st->rrdhost->variables_root_index, rs->fullname, rs->type, rs->value); + + rs->next = st->variables; + st->variables = rs; + + return rs; +} + +void rrdsetvar_rename_all(RRDSET *st) { + debug(D_VARIABLES, "RRDSETVAR rename for chart id '%s' name '%s'", st->id, st->name); + + // only these 2 can change name + // rs->family_name + // rs->host_name + + char buffer[RRDVAR_MAX_LENGTH + 1]; + RRDSETVAR *rs, *next = st->variables; + while((rs = next)) { + next = rs->next; + + snprintfz(buffer, RRDVAR_MAX_LENGTH, "%s.%s", st->name, rs->variable); + + if (strcmp(buffer, rs->fullname)) { + // name changed + rrdvar_free(st->rrdhost, &st->rrdfamily->variables_root_index, rs->family_name); + rrdvar_free(st->rrdhost, &st->rrdhost->variables_root_index, rs->host_name); + + freez(rs->fullname); + rs->fullname = strdupz(st->name); + rs->family_name = rrdvar_create_and_index("family", &st->rrdfamily->variables_root_index, rs->fullname, rs->type, rs->value); + rs->host_name = rrdvar_create_and_index("host", &st->rrdhost->variables_root_index, rs->fullname, rs->type, rs->value); + } + } + + rrdsetcalc_link_matching(st); +} + +void rrdsetvar_free(RRDSETVAR *rs) { + RRDSET *st = rs->rrdset; + debug(D_VARIABLES, "RRDSETVAR free for chart id '%s' name '%s', variable '%s'", st->id, st->name, rs->variable); + + rrdvar_free(st->rrdhost, &st->variables_root_index, rs->local); + rrdvar_free(st->rrdhost, &st->rrdfamily->variables_root_index, rs->family); + rrdvar_free(st->rrdhost, &st->rrdhost->variables_root_index, rs->host); + rrdvar_free(st->rrdhost, &st->rrdfamily->variables_root_index, rs->family_name); + rrdvar_free(st->rrdhost, &st->rrdhost->variables_root_index, rs->host_name); + + if(st->variables == rs) { + st->variables = rs->next; + } + else { + RRDSETVAR *t; + for (t = st->variables; t && t->next != rs; t = t->next); + if(!t) error("RRDSETVAR '%s' not found in chart '%s' variables linked list", rs->fullname, st->id); + else t->next = rs->next; + } + + freez(rs->fullid); + freez(rs->fullname); + freez(rs->variable); + freez(rs); +} + +// ---------------------------------------------------------------------------- +// RRDDIMVAR management + +#define RRDDIMVAR_ID_MAX 1024 + +RRDDIMVAR *rrddimvar_create(RRDDIM *rd, int type, const char *prefix, const char *suffix, void *value, uint32_t options) { + RRDSET *st = rd->rrdset; + + debug(D_VARIABLES, "RRDDIMSET create for chart id '%s' name '%s', dimension id '%s', name '%s%s%s'", st->id, st->name, rd->id, (prefix)?prefix:"", rd->name, (suffix)?suffix:""); + + if(!prefix) prefix = ""; + if(!suffix) suffix = ""; + + char buffer[RRDDIMVAR_ID_MAX + 1]; + RRDDIMVAR *rs = (RRDDIMVAR *)callocz(1, sizeof(RRDDIMVAR)); + + rs->prefix = strdupz(prefix); + rs->suffix = strdupz(suffix); + + snprintfz(buffer, RRDDIMVAR_ID_MAX, "%s%s%s", rs->prefix, rd->id, rs->suffix); + rs->id = strdupz(buffer); + + snprintfz(buffer, RRDDIMVAR_ID_MAX, "%s%s%s", rs->prefix, rd->name, rs->suffix); + rs->name = strdupz(buffer); + + snprintfz(buffer, RRDDIMVAR_ID_MAX, "%s.%s", rd->rrdset->id, rs->id); + rs->fullidid = strdupz(buffer); + + snprintfz(buffer, RRDDIMVAR_ID_MAX, "%s.%s", rd->rrdset->id, rs->name); + rs->fullidname = strdupz(buffer); + + snprintfz(buffer, RRDDIMVAR_ID_MAX, "%s.%s", rd->rrdset->name, rs->id); + rs->fullnameid = strdupz(buffer); + + snprintfz(buffer, RRDDIMVAR_ID_MAX, "%s.%s", rd->rrdset->name, rs->name); + rs->fullnamename = strdupz(buffer); + + rs->type = type; + rs->value = value; + rs->options = options; + rs->rrddim = rd; + + rs->local_id = rrdvar_create_and_index("local", &st->variables_root_index, rs->id, rs->type, rs->value); + rs->local_name = rrdvar_create_and_index("local", &st->variables_root_index, rs->name, rs->type, rs->value); + + rs->family_id = rrdvar_create_and_index("family", &st->rrdfamily->variables_root_index, rs->id, rs->type, rs->value); + rs->family_name = rrdvar_create_and_index("family", &st->rrdfamily->variables_root_index, rs->name, rs->type, rs->value); + + rs->host_fullidid = rrdvar_create_and_index("host", &st->rrdhost->variables_root_index, rs->fullidid, rs->type, rs->value); + rs->host_fullidname = rrdvar_create_and_index("host", &st->rrdhost->variables_root_index, rs->fullidname, rs->type, rs->value); + rs->host_fullnameid = rrdvar_create_and_index("host", &st->rrdhost->variables_root_index, rs->fullnameid, rs->type, rs->value); + rs->host_fullnamename = rrdvar_create_and_index("host", &st->rrdhost->variables_root_index, rs->fullnamename, rs->type, rs->value); + + rs->next = rd->variables; + rd->variables = rs; + + return rs; +} + +void rrddimvar_rename_all(RRDDIM *rd) { + RRDSET *st = rd->rrdset; + debug(D_VARIABLES, "RRDDIMSET rename for chart id '%s' name '%s', dimension id '%s', name '%s'", st->id, st->name, rd->id, rd->name); + + RRDDIMVAR *rs, *next = rd->variables; + while((rs = next)) { + next = rs->next; + + if (strcmp(rd->name, rs->name)) { + char buffer[RRDDIMVAR_ID_MAX + 1]; + // name changed + + // name + rrdvar_free(st->rrdhost, &st->variables_root_index, rs->local_name); + freez(rs->name); + snprintfz(buffer, RRDDIMVAR_ID_MAX, "%s%s%s", rs->prefix, rd->name, rs->suffix); + rs->name = strdupz(buffer); + rs->local_name = rrdvar_create_and_index("local", &st->variables_root_index, rs->name, rs->type, rs->value); + + rrdvar_free(st->rrdhost, &st->rrdhost->variables_root_index, rs->host_fullidname); + freez(rs->fullidname); + snprintfz(buffer, RRDDIMVAR_ID_MAX, "%s.%s", st->id, rs->name); + rs->fullidname = strdupz(buffer); + rs->host_fullidname = rrdvar_create_and_index("host", &st->rrdhost->variables_root_index, + rs->fullidname, rs->type, rs->value); + + // fullnameid + rrdvar_free(st->rrdhost, &st->rrdhost->variables_root_index, rs->host_fullnameid); + freez(rs->fullnameid); + snprintfz(buffer, RRDDIMVAR_ID_MAX, "%s.%s", st->name, rs->id); + rs->fullnameid = strdupz(buffer); + rs->host_fullnameid = rrdvar_create_and_index("host", &st->rrdhost->variables_root_index, + rs->fullnameid, rs->type, rs->value); + + // fullnamename + rrdvar_free(st->rrdhost, &st->rrdhost->variables_root_index, rs->host_fullnamename); + freez(rs->fullnamename); + snprintfz(buffer, RRDDIMVAR_ID_MAX, "%s.%s", st->name, rs->name); + rs->fullnamename = strdupz(buffer); + rs->host_fullnamename = rrdvar_create_and_index("host", &st->rrdhost->variables_root_index, + rs->fullnamename, rs->type, rs->value); + } + } +} + +void rrddimvar_free(RRDDIMVAR *rs) { + RRDDIM *rd = rs->rrddim; + RRDSET *st = rd->rrdset; + debug(D_VARIABLES, "RRDDIMSET free for chart id '%s' name '%s', dimension id '%s', name '%s', prefix='%s', suffix='%s'", st->id, st->name, rd->id, rd->name, rs->prefix, rs->suffix); + + rrdvar_free(st->rrdhost, &st->variables_root_index, rs->local_id); + rrdvar_free(st->rrdhost, &st->variables_root_index, rs->local_name); + + rrdvar_free(st->rrdhost, &st->rrdfamily->variables_root_index, rs->family_id); + rrdvar_free(st->rrdhost, &st->rrdfamily->variables_root_index, rs->family_name); + + rrdvar_free(st->rrdhost, &st->rrdhost->variables_root_index, rs->host_fullidid); + rrdvar_free(st->rrdhost, &st->rrdhost->variables_root_index, rs->host_fullidname); + rrdvar_free(st->rrdhost, &st->rrdhost->variables_root_index, rs->host_fullnameid); + rrdvar_free(st->rrdhost, &st->rrdhost->variables_root_index, rs->host_fullnamename); + + if(rd->variables == rs) { + debug(D_VARIABLES, "RRDDIMSET removing first entry for chart id '%s' name '%s', dimension id '%s', name '%s'", st->id, st->name, rd->id, rd->name); + rd->variables = rs->next; + } + else { + debug(D_VARIABLES, "RRDDIMSET removing non-first entry for chart id '%s' name '%s', dimension id '%s', name '%s'", st->id, st->name, rd->id, rd->name); + RRDDIMVAR *t; + for (t = rd->variables; t && t->next != rs; t = t->next) ; + if(!t) error("RRDDIMVAR '%s' not found in dimension '%s/%s' variables linked list", rs->name, st->id, rd->id); + else t->next = rs->next; + } + + freez(rs->prefix); + freez(rs->suffix); + freez(rs->id); + freez(rs->name); + freez(rs->fullidid); + freez(rs->fullidname); + freez(rs->fullnameid); + freez(rs->fullnamename); + freez(rs); +} + +// ---------------------------------------------------------------------------- +// RRDCALC management + +static inline const char *rrdcalc_status2string(int status) { + switch(status) { + case RRDCALC_STATUS_UNINITIALIZED: + return "UNINITIALIZED"; + + case RRDCALC_STATUS_UNDEFINED: + return "UNDEFINED"; + + case RRDCALC_STATUS_CLEAR: + return "CLEAR"; + + case RRDCALC_STATUS_RAISED: + return "RAISED"; + + case RRDCALC_STATUS_WARNING: + return "WARNING"; + + case RRDCALC_STATUS_CRITICAL: + return "CRITICAL"; + + default: + return "UNKNOWN"; + } +} + +static void rrdsetcalc_link(RRDSET *st, RRDCALC *rc) { + debug(D_HEALTH, "Health linking alarm '%s.%s' to chart '%s' of host '%s'", rc->chart?rc->chart:"NOCHART", rc->name, st->id, st->rrdhost->hostname); + + rc->last_status_change = time(NULL); + rc->rrdset = st; + + rc->rrdset_next = st->alarms; + rc->rrdset_prev = NULL; + st->alarms = rc; + + if(rc->update_every < rc->rrdset->update_every) { + error("Health alarm '%s.%s' has update every %d, less than chart update every %d. Setting alarm update frequency to %d.", rc->rrdset->id, rc->name, rc->update_every, rc->rrdset->update_every, rc->rrdset->update_every); + rc->update_every = rc->rrdset->update_every; + } + + if(!isnan(rc->green) && isnan(st->green)) { + debug(D_HEALTH, "Health alarm '%s.%s' green threshold set from %Lf to %Lf.", rc->rrdset->id, rc->name, rc->rrdset->green, rc->green); + st->green = rc->green; + } + + if(!isnan(rc->red) && isnan(st->red)) { + debug(D_HEALTH, "Health alarm '%s.%s' red threshold set from %Lf to %Lf.", rc->rrdset->id, rc->name, rc->rrdset->red, rc->red); + st->red = rc->red; + } + + rc->local = rrdvar_create_and_index("local", &st->variables_root_index, rc->name, RRDVAR_TYPE_CALCULATED, &rc->value); + rc->family = rrdvar_create_and_index("family", &st->rrdfamily->variables_root_index, rc->name, RRDVAR_TYPE_CALCULATED, &rc->value); + + char fullname[RRDVAR_MAX_LENGTH + 1]; + snprintfz(fullname, RRDVAR_MAX_LENGTH, "%s.%s", st->id, rc->name); + rc->hostid = rrdvar_create_and_index("host", &st->rrdhost->variables_root_index, fullname, RRDVAR_TYPE_CALCULATED, &rc->value); + + snprintfz(fullname, RRDVAR_MAX_LENGTH, "%s.%s", st->name, rc->name); + rc->hostname = rrdvar_create_and_index("host", &st->rrdhost->variables_root_index, fullname, RRDVAR_TYPE_CALCULATED, &rc->value); + + if(!rc->units) rc->units = strdupz(st->units); +} + +static inline int rrdcalc_is_matching_this_rrdset(RRDCALC *rc, RRDSET *st) { + if( (rc->hash_chart == st->hash && !strcmp(rc->chart, st->id)) || + (rc->hash_chart == st->hash_name && !strcmp(rc->chart, st->name))) + return 1; + + return 0; +} + +// this has to be called while the RRDHOST is locked +inline void rrdsetcalc_link_matching(RRDSET *st) { + // debug(D_HEALTH, "find matching alarms for chart '%s'", st->id); + + RRDCALC *rc; + for(rc = st->rrdhost->alarms; rc ; rc = rc->next) { + if(rc->rrdset) continue; + + if(rrdcalc_is_matching_this_rrdset(rc, st)) + rrdsetcalc_link(st, rc); + } +} + +// this has to be called while the RRDHOST is locked +inline void rrdsetcalc_unlink(RRDCALC *rc) { + RRDSET *st = rc->rrdset; + + if(!st) { + error("Requested to unlink RRDCALC '%s.%s' which is not linked to any RRDSET", rc->chart?rc->chart:"NOCHART", rc->name); + return; + } + + RRDHOST *host = st->rrdhost; + + debug(D_HEALTH, "Health unlinking alarm '%s.%s' from chart '%s' of host '%s'", rc->chart?rc->chart:"NOCHART", rc->name, st->id, host->hostname); + + // unlink it + if(rc->rrdset_prev) + rc->rrdset_prev->rrdset_next = rc->rrdset_next; + + if(rc->rrdset_next) + rc->rrdset_next->rrdset_prev = rc->rrdset_prev; + + if(st->alarms == rc) + st->alarms = rc->rrdset_next; + + rc->rrdset_prev = rc->rrdset_next = NULL; + + rrdvar_free(st->rrdhost, &st->variables_root_index, rc->local); + rc->local = NULL; + + rrdvar_free(st->rrdhost, &st->rrdfamily->variables_root_index, rc->family); + rc->family = NULL; + + rrdvar_free(st->rrdhost, &st->rrdhost->variables_root_index, rc->hostid); + rc->hostid = NULL; + + rrdvar_free(st->rrdhost, &st->rrdhost->variables_root_index, rc->hostname); + rc->hostname = NULL; + + rc->rrdset = NULL; + + // RRDCALC will remain in RRDHOST + // so that if the matching chart is found in the future + // it will be applied automatically +} + +RRDCALC *rrdcalc_find(RRDSET *st, const char *name) { + RRDCALC *rc; + uint32_t hash = simple_hash(name); + + for( rc = st->alarms; rc ; rc = rc->rrdset_next ) { + if(rc->hash == hash && !strcmp(rc->name, name)) + return rc; + } + + return NULL; +} + +static inline int rrdcalc_exists(RRDHOST *host, const char *name, uint32_t hash) { + RRDCALC *rc; + + // make sure it does not already exist + for(rc = host->alarms; rc ; rc = rc->next) { + if (rc->hash == hash && !strcmp(name, rc->name)) { + error("Health alarm '%s' already exists in host '%s'.", name, host->hostname); + return 1; + } + } + + return 0; +} + +static inline void rrdcalc_create_part2(RRDHOST *host, RRDCALC *rc) { + rrdhost_check_rdlock(host); + + if(rc->calculation) { + rc->calculation->this = &rc->value; + rc->calculation->after = &rc->db_after; + rc->calculation->before = &rc->db_before; + rc->calculation->rrdcalc = rc; + } + + if(rc->warning) { + rc->warning->this = &rc->value; + rc->warning->after = &rc->db_after; + rc->warning->before = &rc->db_before; + rc->warning->rrdcalc = rc; + } + + if(rc->critical) { + rc->critical->this = &rc->value; + rc->critical->after = &rc->db_after; + rc->critical->before = &rc->db_before; + rc->critical->rrdcalc = rc; + } + + // link it to the host + rc->next = host->alarms; + host->alarms = rc; + + // link it to its chart + RRDSET *st; + for(st = host->rrdset_root; st ; st = st->next) { + if(rrdcalc_is_matching_this_rrdset(rc, st)) { + rrdsetcalc_link(st, rc); + break; + } + } +} + +static inline uint32_t rrdcalc_fullname(char *fullname, size_t len, const char *chart, const char *name) { + snprintfz(fullname, len - 1, "%s%s%s", chart?chart:"", chart?".":"", name); + rrdvar_fix_name(fullname); + return simple_hash(fullname); +} + +static inline RRDCALC *rrdcalc_create(RRDHOST *host, const char *name, const char *chart, const char *dimensions, + const char *units, const char *info, + int group_method, int after, int before, int update_every, uint32_t options, + calculated_number green, calculated_number red, + const char *exec, const char *source, + const char *calc, const char *warn, const char *crit) { + + char fullname[RRDVAR_MAX_LENGTH + 1]; + uint32_t hash = rrdcalc_fullname(fullname, RRDVAR_MAX_LENGTH + 1, chart, name); + + if(rrdcalc_exists(host, fullname, hash)) + return NULL; + + RRDCALC *rc = callocz(1, sizeof(RRDCALC)); + + rc->name = strdupz(name); + rc->hash = simple_hash(rc->name); + + rc->chart = strdupz(chart); + rc->hash_chart = simple_hash(rc->chart); + + if(dimensions) rc->dimensions = strdupz(dimensions); + + rc->green = green; + rc->red = red; + rc->value = NAN; + rc->old_value = NAN; + + rc->group = group_method; + rc->after = after; + rc->before = before; + rc->update_every = update_every; + rc->options = options; + + if(exec) rc->exec = strdupz(exec); + if(source) rc->source = strdupz(source); + if(units) rc->units = strdupz(units); + if(info) rc->info = strdupz(info); + + if(calc) { + rc->calculation = expression_parse(calc, NULL, NULL); + if(!rc->calculation) + error("Health alarm '%s.%s': failed to parse calculation expression '%s'", chart, name, calc); + } + if(warn) { + rc->warning = expression_parse(warn, NULL, NULL); + if(!rc->warning) + error("Health alarm '%s.%s': failed to re-parse warning expression '%s'", chart, name, warn); + } + if(crit) { + rc->critical = expression_parse(crit, NULL, NULL); + if(!rc->critical) + error("Health alarm '%s.%s': failed to re-parse critical expression '%s'", chart, name, crit); + } + + debug(D_HEALTH, "Health runtime added alarm '%s.%s': exec '%s', green %Lf, red %Lf, lookup: group %d, after %d, before %d, options %u, dimensions '%s', update every %d, calculation '%s', warning '%s', critical '%s', source '%s", + (rc->chart)?rc->chart:"NOCHART", + rc->name, + (rc->exec)?rc->exec:"DEFAULT", + rc->green, + rc->red, + rc->group, + rc->after, + rc->before, + rc->options, + (rc->dimensions)?rc->dimensions:"NONE", + rc->update_every, + (rc->calculation)?rc->calculation->parsed_as:"NONE", + (rc->warning)?rc->warning->parsed_as:"NONE", + (rc->critical)?rc->critical->parsed_as:"NONE", + rc->source + ); + + rrdcalc_create_part2(host, rc); + return rc; +} + +void rrdcalc_free(RRDHOST *host, RRDCALC *rc) { + if(!rc) return; + + debug(D_HEALTH, "Health removing alarm '%s.%s' of host '%s'", rc->chart?rc->chart:"NOCHART", rc->name, host->hostname); + + // unlink it from RRDSET + if(rc->rrdset) rrdsetcalc_unlink(rc); + + // unlink it from RRDHOST + if(rc == host->alarms) + host->alarms = rc->next; + + else if(host->alarms) { + RRDCALC *t, *last = host->alarms; + + for(t = last->next; t && t != rc; last = t, t = t->next) ; + if(last && last->next == rc) + last->next = rc->next; + else + error("Cannot unlink alarm '%s.%s' from host '%s': not found", rc->chart?rc->chart:"NOCHART", rc->name, host->hostname); + } + else + error("Cannot unlink unlink '%s.%s' from host '%s': This host does not have any calculations", rc->chart?rc->chart:"NOCHART", rc->name, host->hostname); + + expression_free(rc->calculation); + expression_free(rc->warning); + expression_free(rc->critical); + + freez(rc->name); + freez(rc->chart); + freez(rc->family); + freez(rc->dimensions); + freez(rc->exec); + freez(rc->source); + freez(rc->units); + freez(rc->info); + freez(rc); +} + +// ---------------------------------------------------------------------------- +// RRDCALCTEMPLATE management + +void rrdcalctemplate_link_matching(RRDSET *st) { + RRDCALCTEMPLATE *rt; + + for(rt = st->rrdhost->templates; rt ; rt = rt->next) { + if(rt->hash_context == st->hash_context && !strcmp(rt->context, st->context)) { + + RRDCALC *rc = rrdcalc_create(st->rrdhost, rt->name, st->id, + rt->dimensions, rt->units, rt->info, rt->group, rt->after, rt->before, rt->update_every, rt->options, + rt->green, rt->red, rt->exec, rt->source, + (rt->calculation)?rt->calculation->source:NULL, + (rt->warning)?rt->warning->source:NULL, + (rt->critical)?rt->critical->source:NULL); + + if(!rc) + error("Health tried to create alarm from template '%s', but it failed", rt->name); + +#ifdef NETDATA_INTERNAL_CHECKS + else if(rc->rrdset != st) + error("Health alarm '%s.%s' should be linked to chart '%s', but it is not", rc->chart?rc->chart:"NOCHART", rc->name, st->id); +#else + (void)rc; +#endif + } + } +} + +static inline void rrdcalctemplate_free(RRDHOST *host, RRDCALCTEMPLATE *rt) { + debug(D_HEALTH, "Health removing template '%s' of host '%s'", rt->name, host->hostname); + + if(host->templates) { + if(host->templates == rt) { + host->templates = rt->next; + } + else { + RRDCALCTEMPLATE *t, *last = host->templates; + for (t = last->next; t && t != rt; last = t, t = t->next ) ; + if(last && last->next == rt) { + last->next = rt->next; + rt->next = NULL; + } + else + error("Cannot find RRDCALCTEMPLATE '%s' linked in host '%s'", rt->name, host->hostname); + } + } + + expression_free(rt->calculation); + expression_free(rt->warning); + expression_free(rt->critical); + + freez(rt->name); + freez(rt->exec); + freez(rt->context); + freez(rt->source); + freez(rt->units); + freez(rt->info); + freez(rt->dimensions); + freez(rt); +} + +// ---------------------------------------------------------------------------- +// load health configuration + +#define HEALTH_CONF_MAX_LINE 4096 + +#define HEALTH_ALARM_KEY "alarm" +#define HEALTH_TEMPLATE_KEY "template" +#define HEALTH_ON_KEY "on" +#define HEALTH_LOOKUP_KEY "lookup" +#define HEALTH_CALC_KEY "calc" +#define HEALTH_EVERY_KEY "every" +#define HEALTH_GREEN_KEY "green" +#define HEALTH_RED_KEY "red" +#define HEALTH_WARN_KEY "warn" +#define HEALTH_CRIT_KEY "crit" +#define HEALTH_EXEC_KEY "exec" +#define HEALTH_UNITS_KEY "units" +#define HEALTH_INFO_KEY "info" + +static inline int rrdcalc_add_alarm_from_config(RRDHOST *host, RRDCALC *rc) { + { + char fullname[RRDVAR_MAX_LENGTH + 1]; + uint32_t hash = rrdcalc_fullname(fullname, RRDVAR_MAX_LENGTH + 1, rc->chart, rc->name); + + if (rrdcalc_exists(host, fullname, hash)) + return 0; + } + + if(!rc->chart) { + error("Health configuration for alarm '%s' does not have a chart", rc->name); + return 0; + } + + if(!rc->update_every) { + error("Health configuration for alarm '%s.%s' has no frequency (parameter 'every'). Ignoring it.", rc->chart?rc->chart:"NOCHART", rc->name); + return 0; + } + + if(!RRDCALC_HAS_DB_LOOKUP(rc) && !rc->warning && !rc->critical) { + error("Health configuration for alarm '%s.%s' is useless (no calculation, no warning and no critical evaluation)", rc->chart?rc->chart:"NOCHART", rc->name); + return 0; + } + + debug(D_HEALTH, "Health configuration adding alarm '%s.%s': exec '%s', green %Lf, red %Lf, lookup: group %d, after %d, before %d, options %u, dimensions '%s', update every %d, calculation '%s', warning '%s', critical '%s', source '%s", + rc->chart?rc->chart:"NOCHART", + rc->name, + (rc->exec)?rc->exec:"DEFAULT", + rc->green, + rc->red, + rc->group, + rc->after, + rc->before, + rc->options, + (rc->dimensions)?rc->dimensions:"NONE", + rc->update_every, + (rc->calculation)?rc->calculation->parsed_as:"NONE", + (rc->warning)?rc->warning->parsed_as:"NONE", + (rc->critical)?rc->critical->parsed_as:"NONE", + rc->source + ); + + rrdcalc_create_part2(host, rc); + return 1; +} + +static inline int rrdcalctemplate_add_template_from_config(RRDHOST *host, RRDCALCTEMPLATE *rt) { + if(!rt->context) { + error("Health configuration for template '%s' does not have a context", rt->name); + return 0; + } + + if(!rt->update_every) { + error("Health configuration for template '%s' has no frequency (parameter 'every'). Ignoring it.", rt->name); + return 0; + } + + if(!RRDCALCTEMPLATE_HAS_CALCULATION(rt) && !rt->warning && !rt->critical) { + error("Health configuration for template '%s' is useless (no calculation, no warning and no critical evaluation)", rt->name); + return 0; + } + + RRDCALCTEMPLATE *t; + for (t = host->templates; t ; t = t->next) { + if(t->hash_name == rt->hash_name && !strcmp(t->name, rt->name)) { + error("Health configuration template '%s' already exists for host '%s'.", rt->name, host->hostname); + return 0; + } + } + + debug(D_HEALTH, "Health configuration adding template '%s': context '%s', exec '%s', green %Lf, red %Lf, lookup: group %d, after %d, before %d, options %u, dimensions '%s', update every %d, calculation '%s', warning '%s', critical '%s', source '%s'", + rt->name, + (rt->context)?rt->context:"NONE", + (rt->exec)?rt->exec:"DEFAULT", + rt->green, + rt->red, + rt->group, + rt->after, + rt->before, + rt->options, + (rt->dimensions)?rt->dimensions:"NONE", + rt->update_every, + (rt->calculation)?rt->calculation->parsed_as:"NONE", + (rt->warning)?rt->warning->parsed_as:"NONE", + (rt->critical)?rt->critical->parsed_as:"NONE", + rt->source + ); + + rt->next = host->templates; + host->templates = rt; + return 1; +} + +static inline int health_parse_duration(char *string, int *result) { + // make sure it is a number + if(!*string || !(isdigit(*string) || *string == '+' || *string == '-')) { + *result = 0; + return 0; + } + + char *e = NULL; + calculated_number n = strtold(string, &e); + if(e && *e) { + switch (*e) { + case 'Y': + *result = (int) (n * 86400 * 365); + break; + case 'M': + *result = (int) (n * 86400 * 30); + break; + case 'w': + *result = (int) (n * 86400 * 7); + break; + case 'd': + *result = (int) (n * 86400); + break; + case 'h': + *result = (int) (n * 3600); + break; + case 'm': + *result = (int) (n * 60); + break; + + default: + case 's': + *result = (int) (n); + break; + } + } + else + *result = (int)(n); + + return 1; +} + +static inline int health_parse_db_lookup( + size_t line, const char *path, const char *file, char *string, + int *group_method, int *after, int *before, int *every, + uint32_t *options, char **dimensions +) { + debug(D_HEALTH, "Health configuration parsing database lookup %zu@%s/%s: %s", line, path, file, string); + + if(*dimensions) freez(*dimensions); + *dimensions = NULL; + *after = 0; + *before = 0; + *every = 0; + *options = 0; + + char *s = string, *key; + + // first is the group method + key = s; + while(*s && !isspace(*s)) s++; + while(*s && isspace(*s)) *s++ = '\0'; + if(!*s) { + error("Health configuration invalid chart calculation at line %zu of file '%s/%s': expected group method followed by the 'after' time, but got '%s'", + line, path, file, key); + return 0; + } + + if((*group_method = web_client_api_request_v1_data_group(key, -1)) == -1) { + error("Health configuration at line %zu of file '%s/%s': invalid group method '%s'", + line, path, file, key); + return 0; + } + + // then is the 'after' time + key = s; + while(*s && !isspace(*s)) s++; + while(*s && isspace(*s)) *s++ = '\0'; + + if(!health_parse_duration(key, after)) { + error("Health configuration at line %zu of file '%s/%s': invalid duration '%s' after group method", + line, path, file, key); + return 0; + } + + // sane defaults + *every = abs(*after); + + // now we may have optional parameters + while(*s) { + key = s; + while(*s && !isspace(*s)) s++; + while(*s && isspace(*s)) *s++ = '\0'; + if(!*key) break; + + if(!strcasecmp(key, "at")) { + char *value = s; + while(*s && !isspace(*s)) s++; + while(*s && isspace(*s)) *s++ = '\0'; + + if (!health_parse_duration(value, before)) { + error("Health configuration at line %zu of file '%s/%s': invalid duration '%s' for '%s' keyword", + line, path, file, value, key); + } + } + else if(!strcasecmp(key, HEALTH_EVERY_KEY)) { + char *value = s; + while(*s && !isspace(*s)) s++; + while(*s && isspace(*s)) *s++ = '\0'; + + if (!health_parse_duration(value, every)) { + error("Health configuration at line %zu of file '%s/%s': invalid duration '%s' for '%s' keyword", + line, path, file, value, key); + } + } + else if(!strcasecmp(key, "absolute") || !strcasecmp(key, "abs") || !strcasecmp(key, "absolute_sum")) { + *options |= RRDR_OPTION_ABSOLUTE; + } + else if(!strcasecmp(key, "min2max")) { + *options |= RRDR_OPTION_MIN2MAX; + } + else if(!strcasecmp(key, "null2zero")) { + *options |= RRDR_OPTION_NULL2ZERO; + } + else if(!strcasecmp(key, "percentage")) { + *options |= RRDR_OPTION_PERCENTAGE; + } + else if(!strcasecmp(key, "unaligned")) { + *options |= RRDR_OPTION_NOT_ALIGNED; + } + else if(!strcasecmp(key, "of")) { + if(*s && strcasecmp(s, "all")) + *dimensions = strdupz(s); + break; + } + else { + error("Health configuration at line %zu of file '%s/%s': unknown keyword '%s'", + line, path, file, key); + } + } + + return 1; +} + +static inline char *health_source_file(size_t line, const char *path, const char *filename) { + char buffer[FILENAME_MAX + 1]; + snprintfz(buffer, FILENAME_MAX, "%zu@%s/%s", line, path, filename); + return strdupz(buffer); +} + +static inline void strip_quotes(char *s) { + while(*s) { + if(*s == '\'' || *s == '"') *s = ' '; + s++; + } +} + +int health_readfile(const char *path, const char *filename) { + debug(D_HEALTH, "Health configuration reading file '%s/%s'", path, filename); + + static uint32_t hash_alarm = 0, hash_template = 0, hash_on = 0, hash_calc = 0, hash_green = 0, hash_red = 0, hash_warn = 0, hash_crit = 0, hash_exec = 0, hash_every = 0, hash_lookup = 0, hash_units = 0, hash_info = 0; + char buffer[HEALTH_CONF_MAX_LINE + 1]; + + if(unlikely(!hash_alarm)) { + hash_alarm = simple_uhash(HEALTH_ALARM_KEY); + hash_template = simple_uhash(HEALTH_TEMPLATE_KEY); + hash_on = simple_uhash(HEALTH_ON_KEY); + hash_calc = simple_uhash(HEALTH_CALC_KEY); + hash_lookup = simple_uhash(HEALTH_LOOKUP_KEY); + hash_green = simple_uhash(HEALTH_GREEN_KEY); + hash_red = simple_uhash(HEALTH_RED_KEY); + hash_warn = simple_uhash(HEALTH_WARN_KEY); + hash_crit = simple_uhash(HEALTH_CRIT_KEY); + hash_exec = simple_uhash(HEALTH_EXEC_KEY); + hash_every = simple_uhash(HEALTH_EVERY_KEY); + hash_units = simple_hash(HEALTH_UNITS_KEY); + hash_info = simple_hash(HEALTH_INFO_KEY); + } + + snprintfz(buffer, HEALTH_CONF_MAX_LINE, "%s/%s", path, filename); + FILE *fp = fopen(buffer, "r"); + if(!fp) { + error("Health configuration cannot read file '%s'.", buffer); + return 0; + } + + RRDCALC *rc = NULL; + RRDCALCTEMPLATE *rt = NULL; + + size_t line = 0, append = 0; + char *s; + while((s = fgets(&buffer[append], (int)(HEALTH_CONF_MAX_LINE - append), fp)) || append) { + int stop_appending = !s; + line++; + // info("Line %zu of file '%s/%s': '%s'", line, path, filename, s); + s = trim(buffer); + if(!s) continue; + // info("Trimmed line %zu of file '%s/%s': '%s'", line, path, filename, s); + + append = strlen(s); + if(!stop_appending && s[append - 1] == '\\') { + s[append - 1] = ' '; + append = &s[append] - buffer; + if(append < HEALTH_CONF_MAX_LINE) + continue; + else { + error("Health configuration has too long muli-line at line %zu of file '%s/%s'.", line, path, filename); + } + } + append = 0; + + char *key = s; + while(*s && *s != ':') s++; + if(!*s) { + error("Health configuration has invalid line %zu of file '%s/%s'. It does not contain a ':'. Ignoring it.", line, path, filename); + continue; + } + *s = '\0'; + s++; + + char *value = s; + key = trim(key); + value = trim(value); + + if(!key) { + error("Health configuration has invalid line %zu of file '%s/%s'. Keyword is empty. Ignoring it.", line, path, filename); + continue; + } + + if(!value) { + error("Health configuration has invalid line %zu of file '%s/%s'. value is empty. Ignoring it.", line, path, filename); + continue; + } + + // info("Health file '%s/%s', key '%s', value '%s'", path, filename, key, value); + uint32_t hash = simple_uhash(key); + + if(hash == hash_alarm && !strcasecmp(key, HEALTH_ALARM_KEY)) { + if(rc && !rrdcalc_add_alarm_from_config(&localhost, rc)) + rrdcalc_free(&localhost, rc); + + if(rt) { + if (!rrdcalctemplate_add_template_from_config(&localhost, rt)) + rrdcalctemplate_free(&localhost, rt); + rt = NULL; + } + + rc = callocz(1, sizeof(RRDCALC)); + rc->name = strdupz(value); + rc->hash = simple_hash(rc->name); + rc->source = health_source_file(line, path, filename); + rc->green = NAN; + rc->red = NAN; + rc->value = NAN; + rc->old_value = NAN; + + if(rrdvar_fix_name(rc->name)) + error("Health configuration renamed alarm '%s' to '%s'", value, rc->name); + } + else if(hash == hash_template && !strcasecmp(key, HEALTH_TEMPLATE_KEY)) { + if(rc) { + if(!rrdcalc_add_alarm_from_config(&localhost, rc)) + rrdcalc_free(&localhost, rc); + rc = NULL; + } + + if(rt && !rrdcalctemplate_add_template_from_config(&localhost, rt)) + rrdcalctemplate_free(&localhost, rt); + + rt = callocz(1, sizeof(RRDCALCTEMPLATE)); + rt->name = strdupz(value); + rt->hash_name = simple_hash(rt->name); + rt->source = health_source_file(line, path, filename); + rt->green = NAN; + rt->red = NAN; + + if(rrdvar_fix_name(rt->name)) + error("Health configuration renamed template '%s' to '%s'", value, rt->name); + } + else if(rc) { + if(hash == hash_on && !strcasecmp(key, HEALTH_ON_KEY)) { + if(rc->chart) { + if(strcmp(rc->chart, value)) + info("Health configuration at line %zu of file '%s/%s' for alarm '%s' has key '%s' twice, once with value '%s' and later with value '%s'. Using ('%s').", + line, path, filename, rc->name, key, rc->chart, value, value); + + freez(rc->chart); + } + rc->chart = strdupz(value); + rc->hash_chart = simple_hash(rc->chart); + } + else if(hash == hash_lookup && !strcasecmp(key, HEALTH_LOOKUP_KEY)) { + health_parse_db_lookup(line, path, filename, value, &rc->group, &rc->after, &rc->before, + &rc->update_every, + &rc->options, &rc->dimensions); + } + else if(hash == hash_every && !strcasecmp(key, HEALTH_EVERY_KEY)) { + if(!health_parse_duration(value, &rc->update_every)) + info("Health configuration at line %zu of file '%s/%s' for alarm '%s' at key '%s' cannot parse duration: '%s'.", + line, path, filename, rc->name, key, value); + } + else if(hash == hash_green && !strcasecmp(key, HEALTH_GREEN_KEY)) { + char *e; + rc->green = strtold(value, &e); + if(e && *e) { + info("Health configuration at line %zu of file '%s/%s' for alarm '%s' at key '%s' leaves this string unmatched: '%s'.", + line, path, filename, rc->name, key, e); + } + } + else if(hash == hash_red && !strcasecmp(key, HEALTH_RED_KEY)) { + char *e; + rc->red = strtold(value, &e); + if(e && *e) { + info("Health configuration at line %zu of file '%s/%s' for alarm '%s' at key '%s' leaves this string unmatched: '%s'.", + line, path, filename, rc->name, key, e); + } + } + else if(hash == hash_calc && !strcasecmp(key, HEALTH_CALC_KEY)) { + const char *failed_at = NULL; + int error = 0; + rc->calculation = expression_parse(value, &failed_at, &error); + if(!rc->calculation) { + error("Health configuration at line %zu of file '%s/%s' for alarm '%s' at key '%s' has unparse-able expression '%s': %s at '%s'", + line, path, filename, rc->name, key, value, expression_strerror(error), failed_at); + } + } + else if(hash == hash_warn && !strcasecmp(key, HEALTH_WARN_KEY)) { + const char *failed_at = NULL; + int error = 0; + rc->warning = expression_parse(value, &failed_at, &error); + if(!rc->warning) { + error("Health configuration at line %zu of file '%s/%s' for alarm '%s' at key '%s' has unparse-able expression '%s': %s at '%s'", + line, path, filename, rc->name, key, value, expression_strerror(error), failed_at); + } + } + else if(hash == hash_crit && !strcasecmp(key, HEALTH_CRIT_KEY)) { + const char *failed_at = NULL; + int error = 0; + rc->critical = expression_parse(value, &failed_at, &error); + if(!rc->critical) { + error("Health configuration at line %zu of file '%s/%s' for alarm '%s' at key '%s' has unparse-able expression '%s': %s at '%s'", + line, path, filename, rc->name, key, value, expression_strerror(error), failed_at); + } + } + else if(hash == hash_exec && !strcasecmp(key, HEALTH_EXEC_KEY)) { + if(rc->exec) { + if(strcmp(rc->exec, value)) + info("Health configuration at line %zu of file '%s/%s' for alarm '%s' has key '%s' twice, once with value '%s' and later with value '%s'. Using ('%s').", + line, path, filename, rc->name, key, rc->exec, value, value); + + freez(rc->exec); + } + rc->exec = strdupz(value); + } + else if(hash == hash_units && !strcasecmp(key, HEALTH_UNITS_KEY)) { + if(rc->units) { + if(strcmp(rc->units, value)) + info("Health configuration at line %zu of file '%s/%s' for alarm '%s' has key '%s' twice, once with value '%s' and later with value '%s'. Using ('%s').", + line, path, filename, rc->name, key, rc->units, value, value); + + freez(rc->units); + } + rc->units = strdupz(value); + strip_quotes(rc->units); + } + else if(hash == hash_info && !strcasecmp(key, HEALTH_INFO_KEY)) { + if(rc->info) { + if(strcmp(rc->info, value)) + info("Health configuration at line %zu of file '%s/%s' for alarm '%s' has key '%s' twice, once with value '%s' and later with value '%s'. Using ('%s').", + line, path, filename, rc->name, key, rc->info, value, value); + + freez(rc->info); + } + rc->info = strdupz(value); + strip_quotes(rc->info); + } + else { + error("Health configuration at line %zu of file '%s/%s' for alarm '%s' has unknown key '%s'.", + line, path, filename, rc->name, key); + } + } + else if(rt) { + if(hash == hash_on && !strcasecmp(key, HEALTH_ON_KEY)) { + if(rt->context) { + if(strcmp(rt->context, value)) + info("Health configuration at line %zu of file '%s/%s' for template '%s' has key '%s' twice, once with value '%s' and later with value '%s'. Using ('%s').", + line, path, filename, rt->name, key, rt->context, value, value); + + freez(rt->context); + } + rt->context = strdupz(value); + rt->hash_context = simple_hash(rt->context); + } + else if(hash == hash_lookup && !strcasecmp(key, HEALTH_LOOKUP_KEY)) { + health_parse_db_lookup(line, path, filename, value, &rt->group, &rt->after, &rt->before, + &rt->update_every, + &rt->options, &rt->dimensions); + } + else if(hash == hash_every && !strcasecmp(key, HEALTH_EVERY_KEY)) { + if(!health_parse_duration(value, &rt->update_every)) + info("Health configuration at line %zu of file '%s/%s' for template '%s' at key '%s' cannot parse duration: '%s'.", + line, path, filename, rt->name, key, value); + } + else if(hash == hash_green && !strcasecmp(key, HEALTH_GREEN_KEY)) { + char *e; + rt->green = strtold(value, &e); + if(e && *e) { + info("Health configuration at line %zu of file '%s/%s' for template '%s' at key '%s' leaves this string unmatched: '%s'.", + line, path, filename, rt->name, key, e); + } + } + else if(hash == hash_red && !strcasecmp(key, HEALTH_RED_KEY)) { + char *e; + rt->red = strtold(value, &e); + if(e && *e) { + info("Health configuration at line %zu of file '%s/%s' for template '%s' at key '%s' leaves this string unmatched: '%s'.", + line, path, filename, rt->name, key, e); + } + } + else if(hash == hash_calc && !strcasecmp(key, HEALTH_CALC_KEY)) { + const char *failed_at = NULL; + int error = 0; + rt->calculation = expression_parse(value, &failed_at, &error); + if(!rt->calculation) { + error("Health configuration at line %zu of file '%s/%s' for template '%s' at key '%s' has unparse-able expression '%s': %s at '%s'", + line, path, filename, rt->name, key, value, expression_strerror(error), failed_at); + } + } + else if(hash == hash_warn && !strcasecmp(key, HEALTH_WARN_KEY)) { + const char *failed_at = NULL; + int error = 0; + rt->warning = expression_parse(value, &failed_at, &error); + if(!rt->warning) { + error("Health configuration at line %zu of file '%s/%s' for template '%s' at key '%s' has unparse-able expression '%s': %s at '%s'", + line, path, filename, rt->name, key, value, expression_strerror(error), failed_at); + } + } + else if(hash == hash_crit && !strcasecmp(key, HEALTH_CRIT_KEY)) { + const char *failed_at = NULL; + int error = 0; + rt->critical = expression_parse(value, &failed_at, &error); + if(!rt->critical) { + error("Health configuration at line %zu of file '%s/%s' for template '%s' at key '%s' has unparse-able expression '%s': %s at '%s'", + line, path, filename, rt->name, key, value, expression_strerror(error), failed_at); + } + } + else if(hash == hash_exec && !strcasecmp(key, HEALTH_EXEC_KEY)) { + if(rt->exec) { + if(strcmp(rt->exec, value)) + info("Health configuration at line %zu of file '%s/%s' for template '%s' has key '%s' twice, once with value '%s' and later with value '%s'. Using ('%s').", + line, path, filename, rt->name, key, rt->exec, value, value); + + freez(rt->exec); + } + rt->exec = strdupz(value); + } + else if(hash == hash_units && !strcasecmp(key, HEALTH_UNITS_KEY)) { + if(rt->units) { + if(strcmp(rt->units, value)) + info("Health configuration at line %zu of file '%s/%s' for template '%s' has key '%s' twice, once with value '%s' and later with value '%s'. Using ('%s').", + line, path, filename, rt->name, key, rt->units, value, value); + + freez(rt->units); + } + rt->units = strdupz(value); + strip_quotes(rt->units); + } + else if(hash == hash_info && !strcasecmp(key, HEALTH_INFO_KEY)) { + if(rt->info) { + if(strcmp(rt->info, value)) + info("Health configuration at line %zu of file '%s/%s' for template '%s' has key '%s' twice, once with value '%s' and later with value '%s'. Using ('%s').", + line, path, filename, rt->name, key, rt->info, value, value); + + freez(rt->info); + } + rt->info = strdupz(value); + strip_quotes(rt->info); + } + else { + error("Health configuration at line %zu of file '%s/%s' for template '%s' has unknown key '%s'.", + line, path, filename, rt->name, key); + } + } + else { + error("Health configuration at line %zu of file '%s/%s' has unknown key '%s'. Expected either '" HEALTH_ALARM_KEY "' or '" HEALTH_TEMPLATE_KEY "'.", + line, path, filename, key); + } + } + + if(rc && !rrdcalc_add_alarm_from_config(&localhost, rc)) + rrdcalc_free(&localhost, rc); + + if(rt && !rrdcalctemplate_add_template_from_config(&localhost, rt)) + rrdcalctemplate_free(&localhost, rt); + + fclose(fp); + return 1; +} + +void health_readdir(const char *path) { + size_t pathlen = strlen(path); + + debug(D_HEALTH, "Health configuration reading directory '%s'", path); + + DIR *dir = opendir(path); + if (!dir) { + error("Health configuration cannot open directory '%s'.", path); + return; + } + + struct dirent *de = NULL; + while ((de = readdir(dir))) { + size_t len = strlen(de->d_name); + + if(de->d_type == DT_DIR + && ( + (de->d_name[0] == '.' && de->d_name[1] == '\0') + || (de->d_name[0] == '.' && de->d_name[1] == '.' && de->d_name[2] == '\0') + )) + continue; + + else if(de->d_type == DT_DIR) { + char *s = mallocz(pathlen + strlen(de->d_name) + 2); + strcpy(s, path); + strcat(s, "/"); + strcat(s, de->d_name); + health_readdir(s); + freez(s); + continue; + } + + else if((de->d_type == DT_LNK || de->d_type == DT_REG) && + len > 5 && !strcmp(&de->d_name[len - 5], ".conf")) { + health_readfile(path, de->d_name); + } + } + + closedir(dir); +} + +static inline char *health_config_dir(void) { + char buffer[FILENAME_MAX + 1]; + snprintfz(buffer, FILENAME_MAX, "%s/health.d", config_get("global", "config directory", CONFIG_DIR)); + return config_get("health", "health configuration directory", buffer); +} + +void health_init(void) { + debug(D_HEALTH, "Health configuration initializing"); + + if(!(health_enabled = config_get_boolean("health", "enabled", 1))) { + debug(D_HEALTH, "Health is disabled."); + return; + } + + char *path = health_config_dir(); + + { + char buffer[FILENAME_MAX + 1]; + snprintfz(buffer, FILENAME_MAX, "%s/alarm-email.sh", config_get("global", "plugins directory", PLUGINS_DIR)); + health_default_exec = config_get("health", "script to execute on alarm", buffer); + } + + long n = config_get_number("health", "in memory max health log entries", (long)localhost.health_log.max); + if(n < 2) { + error("Health configuration has invalid max log entries %ld. Using default %u", n, localhost.health_log.max); + config_set_number("health", "in memory max health log entries", (long)localhost.health_log.max); + } + else localhost.health_log.max = (unsigned int)n; + + rrdhost_rwlock(&localhost); + health_readdir(path); + rrdhost_unlock(&localhost); +} + +// ---------------------------------------------------------------------------- +// JSON generation + +static inline void health_string2json(BUFFER *wb, const char *prefix, const char *label, const char *value, const char *suffix) { + if(value && *value) + buffer_sprintf(wb, "%s\"%s\":\"%s\"%s", prefix, label, value, suffix); + else + buffer_sprintf(wb, "%s\"%s\":null%s", prefix, label, suffix); +} + +static inline void health_alarm_entry2json_nolock(BUFFER *wb, ALARM_ENTRY *ae) { + buffer_sprintf(wb, "\n\t{\n" + "\t\t\"id\":%u,\n" + "\t\t\"name\":\"%s\",\n" + "\t\t\"chart\":\"%s\",\n" + "\t\t\"family\":\"%s\",\n" + "\t\t\"processed\":%s,\n" + "\t\t\"updated\":%s,\n" + "\t\t\"exec_run\":%s,\n" + "\t\t\"exec_failed\":%s,\n" + "\t\t\"exec\":\"%s\",\n" + "\t\t\"exec_code\":%d,\n" + "\t\t\"source\":\"%s\",\n" + "\t\t\"units\":\"%s\",\n" + "\t\t\"info\":\"%s\",\n" + "\t\t\"when\":%lu,\n" + "\t\t\"duration\":%lu,\n" + "\t\t\"non_clear_duration\":%lu,\n" + "\t\t\"status\":\"%s\",\n" + "\t\t\"old_status\":\"%s\",\n", + ae->id, + ae->name, + ae->chart, + ae->family, + (ae->notifications & HEALTH_ENTRY_NOTIFICATIONS_PROCESSED)?"true":"false", + (ae->notifications & HEALTH_ENTRY_NOTIFICATIONS_UPDATED)?"true":"false", + (ae->notifications & HEALTH_ENTRY_NOTIFICATIONS_EXEC_RUN)?"true":"false", + (ae->notifications & HEALTH_ENTRY_NOTIFICATIONS_EXEC_FAILED)?"true":"false", + ae->exec?ae->exec:health_default_exec, + ae->exec_code, + ae->source, + ae->units?ae->units:"", + ae->info?ae->info:"", + (unsigned long)ae->when, + (unsigned long)ae->duration, + (unsigned long)ae->non_clear_duration, + rrdcalc_status2string(ae->new_status), + rrdcalc_status2string(ae->old_status) + ); + + buffer_strcat(wb, "\t\t\"value\":"); + buffer_rrd_value(wb, ae->new_value); + buffer_strcat(wb, ",\n"); + + buffer_strcat(wb, "\t\t\"old_value\":"); + buffer_rrd_value(wb, ae->old_value); + buffer_strcat(wb, "\n"); + + buffer_strcat(wb, "\t}"); +} + +void health_alarm_log2json(RRDHOST *host, BUFFER *wb) { + pthread_rwlock_rdlock(&host->health_log.alarm_log_rwlock); + + buffer_strcat(wb, "["); + + unsigned int max = host->health_log.max; + unsigned int count = 0; + ALARM_ENTRY *ae; + for(ae = host->health_log.alarms; ae && count < max ; count++, ae = ae->next) { + if(likely(count)) buffer_strcat(wb, ","); + health_alarm_entry2json_nolock(wb, ae); + } + + buffer_strcat(wb, "\n]\n"); + + pthread_rwlock_unlock(&host->health_log.alarm_log_rwlock); +} + +static inline void health_rrdcalc2json_nolock(BUFFER *wb, RRDCALC *rc) { + buffer_sprintf(wb, + "\t\t\"%s.%s\": {\n" + "\t\t\t\"name\": \"%s\",\n" + "\t\t\t\"chart\": \"%s\",\n" + "\t\t\t\"family\": \"%s\",\n" + "\t\t\t\"active\": %s,\n" + "\t\t\t\"exec\": \"%s\",\n" + "\t\t\t\"source\": \"%s\",\n" + "\t\t\t\"units\": \"%s\",\n" + "\t\t\t\"info\": \"%s\",\n" + "\t\t\t\"status\": \"%s\",\n" + "\t\t\t\"last_status_change\": %lu,\n" + "\t\t\t\"last_updated\": %lu,\n" + "\t\t\t\"next_update\": %lu,\n" + "\t\t\t\"update_every\": %d,\n" + , rc->chart, rc->name + , rc->name + , rc->chart + , (rc->rrdset && rc->rrdset->family)?rc->rrdset->family:"" + , (rc->rrdset)?"true":"false" + , rc->exec?rc->exec:health_default_exec + , rc->source + , rc->units?rc->units:"" + , rc->info?rc->info:"" + , rrdcalc_status2string(rc->status) + , (unsigned long)rc->last_status_change + , (unsigned long)rc->last_updated + , (unsigned long)rc->next_update + , rc->update_every + ); + + if(RRDCALC_HAS_DB_LOOKUP(rc)) { + if(rc->dimensions && *rc->dimensions) + health_string2json(wb, "\t\t\t", "lookup_dimensions", rc->dimensions, ",\n"); + + buffer_sprintf(wb, + "\t\t\t\"db_after\": %lu,\n" + "\t\t\t\"db_before\": %lu,\n" + "\t\t\t\"lookup_method\": \"%s\",\n" + "\t\t\t\"lookup_after\": %d,\n" + "\t\t\t\"lookup_before\": %d,\n" + "\t\t\t\"lookup_options\": \"", + (unsigned long) rc->db_after, + (unsigned long) rc->db_before, + group_method2string(rc->group), + rc->after, + rc->before + ); + buffer_data_options2string(wb, rc->options); + buffer_strcat(wb, "\",\n"); + } + + if(rc->calculation) { + health_string2json(wb, "\t\t\t", "calc", rc->calculation->source, ",\n"); + health_string2json(wb, "\t\t\t", "calc_parsed", rc->calculation->parsed_as, ",\n"); + } + + if(rc->warning) { + health_string2json(wb, "\t\t\t", "warn", rc->warning->source, ",\n"); + health_string2json(wb, "\t\t\t", "warn_parsed", rc->warning->parsed_as, ",\n"); + } + + if(rc->critical) { + health_string2json(wb, "\t\t\t", "crit", rc->critical->source, ",\n"); + health_string2json(wb, "\t\t\t", "crit_parsed", rc->critical->parsed_as, ",\n"); + } + + buffer_strcat(wb, "\t\t\t\"green\":"); + buffer_rrd_value(wb, rc->green); + buffer_strcat(wb, ",\n"); + + buffer_strcat(wb, "\t\t\t\"red\":"); + buffer_rrd_value(wb, rc->red); + buffer_strcat(wb, ",\n"); + + buffer_strcat(wb, "\t\t\t\"value\":"); + buffer_rrd_value(wb, rc->value); + buffer_strcat(wb, "\n"); + + buffer_strcat(wb, "\t\t}"); +} + +//void health_rrdcalctemplate2json_nolock(BUFFER *wb, RRDCALCTEMPLATE *rt) { +// +//} + +void health_alarms2json(RRDHOST *host, BUFFER *wb, int all) { + int i; + rrdhost_rdlock(&localhost); + + buffer_strcat(wb, "{\n\t\"alarms\": {\n"); + RRDCALC *rc; + for(i = 0, rc = host->alarms; rc ; rc = rc->next) { + if(!rc->rrdset) + continue; + + if(!all && !(rc->status == RRDCALC_STATUS_WARNING || rc->status == RRDCALC_STATUS_CRITICAL)) + continue; + + if(likely(i)) buffer_strcat(wb, ",\n"); + health_rrdcalc2json_nolock(wb, rc); + i++; + } + +// buffer_strcat(wb, "\n\t},\n\t\"templates\": {"); + +// RRDCALCTEMPLATE *rt; +// for(rt = host->templates; rt ; rt = rt->next) +// health_rrdcalctemplate2json_nolock(wb, rt); + + buffer_sprintf(wb, "\n\t},\n\t\"now\": %lu\n}\n", (unsigned long)time(NULL)); + rrdhost_unlock(&localhost); +} + + +// ---------------------------------------------------------------------------- +// re-load health configuration + +static inline void health_free_all_nolock(RRDHOST *host) { + while(host->templates) + rrdcalctemplate_free(host, host->templates); + + while(host->alarms) + rrdcalc_free(host, host->alarms); +} + +void health_reload(void) { + if(!health_enabled) { + error("Health reload is requested, but health is not enabled."); + return; + } + + char *path = health_config_dir(); + + rrdhost_rwlock(&localhost); + health_free_all_nolock(&localhost); + rrdhost_unlock(&localhost); + + RRDSET *st; + for(st = localhost.rrdset_root; st ; st = st->next) { + st->green = NAN; + st->red = NAN; + } + + rrdhost_rwlock(&localhost); + health_readdir(path); + rrdhost_unlock(&localhost); + + for(st = localhost.rrdset_root; st ; st = st->next) { + rrdhost_rwlock(&localhost); + + rrdsetcalc_link_matching(st); + rrdcalctemplate_link_matching(st); + + rrdhost_unlock(&localhost); + } +} + + +// ---------------------------------------------------------------------------- +// health main thread and friends + +static inline int rrdcalc_isrunnable(RRDCALC *rc, time_t now, time_t *next_run) { + if (unlikely(!rc->rrdset)) { + debug(D_HEALTH, "Health not running alarm '%s.%s'. It is not linked to a chart.", rc->chart?rc->chart:"NOCHART", rc->name); + return 0; + } + + if (unlikely(!rc->update_every)) { + debug(D_HEALTH, "Health not running alarm '%s.%s'. It does not have an update frequency", rc->chart?rc->chart:"NOCHART", rc->name); + return 0; + } + + if (unlikely(rc->next_update > now)) { + if (*next_run > rc->next_update) + *next_run = rc->next_update; + + debug(D_HEALTH, "Health not examining alarm '%s.%s' yet (will do in %d secs).", rc->chart?rc->chart:"NOCHART", rc->name, (int) (rc->next_update - now)); + return 0; + } + + return 1; +} + +static inline int rrdcalc_value2status(calculated_number n) { + if(isnan(n)) return RRDCALC_STATUS_UNDEFINED; + if(n) return RRDCALC_STATUS_RAISED; + return RRDCALC_STATUS_CLEAR; +} + +static inline void health_alarm_execute(ALARM_ENTRY *ae) { + if(ae->old_status == RRDCALC_STATUS_UNINITIALIZED && ae->new_status == RRDCALC_STATUS_CLEAR) + return; + + char buffer[FILENAME_MAX + 1]; + pid_t command_pid; + + const char *exec = ae->exec; + if(!exec) exec = health_default_exec; + + snprintfz(buffer, FILENAME_MAX, "exec %s '%s' '%s' '%s' '%s' '%s' '%0.0Lf' '%0.0Lf' '%s' '%u' '%u' '%s' '%s'", + exec, + ae->name, + ae->chart?ae->chart:"NOCAHRT", + ae->family?ae->family:"NOFAMILY", + rrdcalc_status2string(ae->new_status), + rrdcalc_status2string(ae->old_status), + ae->new_value, + ae->old_value, + ae->source?ae->source:"UNKNOWN", + (uint32_t)ae->duration, + (uint32_t)ae->non_clear_duration, + ae->units?ae->units:"", + ae->info?ae->info:"" + ); + + ae->notifications |= HEALTH_ENTRY_NOTIFICATIONS_EXEC_RUN; + + debug(D_HEALTH, "executing command '%s'", buffer); + FILE *fp = mypopen(buffer, &command_pid); + if(!fp) { + error("HEALTH: Cannot popen(\"%s\", \"r\").", buffer); + return; + } + debug(D_HEALTH, "HEALTH reading from command"); + char *s = fgets(buffer, FILENAME_MAX, fp); + (void)s; + debug(D_HEALTH, "HEALTH closing command"); + ae->exec_code = mypclose(fp, command_pid); + debug(D_HEALTH, "done executing command - returned with code %d", ae->exec_code); + + if(ae->exec_code != 0) + ae->notifications |= HEALTH_ENTRY_NOTIFICATIONS_EXEC_FAILED; +} + +static inline void health_process_notifications(ALARM_ENTRY *ae) { + info("Health alarm '%s.%s' = %0.2Lf - changed status from %s to %s", + ae->chart?ae->chart:"NOCHART", ae->name, + ae->new_value, + rrdcalc_status2string(ae->old_status), + rrdcalc_status2string(ae->new_status) + ); + + health_alarm_execute(ae); +} + +static inline void health_alarm_log(RRDHOST *host, time_t when, + const char *name, const char *chart, const char *family, + const char *exec, time_t duration, + calculated_number old_value, calculated_number new_value, + int old_status, int new_status, + const char *source, + const char *units, + const char *info +) { + ALARM_ENTRY *ae = callocz(1, sizeof(ALARM_ENTRY)); + ae->name = strdupz(name); + ae->hash_name = simple_hash(ae->name); + + if(chart) { + ae->chart = strdupz(chart); + ae->hash_chart = simple_hash(ae->chart); + } + + if(family) + ae->family = strdupz(family); + + if(exec) ae->exec = strdupz(exec); + if(source) ae->source = strdupz(source); + if(units) ae->units = strdupz(units); + if(info) ae->info = strdupz(info); + + ae->id = host->health_log.nextid++; + ae->when = when; + ae->old_value = old_value; + ae->new_value = new_value; + ae->old_status = old_status; + ae->new_status = new_status; + ae->duration = duration; + + if(ae->old_status == RRDCALC_STATUS_WARNING || ae->old_status == RRDCALC_STATUS_CRITICAL) + ae->non_clear_duration += ae->duration; + + // link it + pthread_rwlock_wrlock(&host->health_log.alarm_log_rwlock); + ae->next = host->health_log.alarms; + host->health_log.alarms = ae; + host->health_log.count++; + pthread_rwlock_unlock(&host->health_log.alarm_log_rwlock); + + // match previous alarms + pthread_rwlock_rdlock(&host->health_log.alarm_log_rwlock); + ALARM_ENTRY *t; + for(t = host->health_log.alarms ; t ; t = t->next) { + if(t != ae && + t->hash_name == ae->hash_name && + t->hash_chart == ae->hash_chart && + !strcmp(t->name, ae->name) && + t->chart && ae->chart && !strcmp(t->chart, ae->chart)) { + + if(!(t->notifications & HEALTH_ENTRY_NOTIFICATIONS_UPDATED) && !t->updated_by) { + t->notifications |= HEALTH_ENTRY_NOTIFICATIONS_UPDATED; + t->updated_by = ae; + + if((t->new_status == RRDCALC_STATUS_WARNING || t->new_status == RRDCALC_STATUS_CRITICAL) && + (t->old_status == RRDCALC_STATUS_WARNING || t->old_status == RRDCALC_STATUS_CRITICAL)) + ae->non_clear_duration += t->non_clear_duration; + } + else { + // no need to continue + break; + } + } + } + pthread_rwlock_unlock(&host->health_log.alarm_log_rwlock); +} + +static inline void health_alarm_log_process(RRDHOST *host) { + static uint32_t last_processed = 0; + ALARM_ENTRY *ae; + + pthread_rwlock_rdlock(&host->health_log.alarm_log_rwlock); + + for(ae = host->health_log.alarms; ae ;ae = ae->next) { + if(last_processed >= ae->id) break; + + if(!(ae->notifications & HEALTH_ENTRY_NOTIFICATIONS_PROCESSED) && + !(ae->notifications & HEALTH_ENTRY_NOTIFICATIONS_UPDATED)) { + ae->notifications |= HEALTH_ENTRY_NOTIFICATIONS_PROCESSED; + health_process_notifications(ae); + } + } + + if(host->health_log.alarms) + last_processed = host->health_log.alarms->id; + + pthread_rwlock_unlock(&host->health_log.alarm_log_rwlock); + + if(host->health_log.count <= host->health_log.max) + return; + + // cleanup excess entries in the log + pthread_rwlock_wrlock(&host->health_log.alarm_log_rwlock); + + ALARM_ENTRY *last = NULL; + unsigned int count = host->health_log.max; + for(ae = host->health_log.alarms; ae && count ; count--, last = ae, ae = ae->next) ; + + if(ae && last && last->next == ae) + last->next = NULL; + else + ae = NULL; + + while(ae) { + ALARM_ENTRY *t = ae->next; + + freez(ae->name); + freez(ae->chart); + freez(ae->family); + freez(ae->exec); + freez(ae->source); + freez(ae->units); + freez(ae->info); + freez(ae); + + ae = t; + } + + pthread_rwlock_unlock(&host->health_log.alarm_log_rwlock); +} + +void *health_main(void *ptr) { + (void)ptr; + + info("HEALTH thread created with task id %d", gettid()); + + if(pthread_setcanceltype(PTHREAD_CANCEL_DEFERRED, NULL) != 0) + error("Cannot set pthread cancel type to DEFERRED."); + + if(pthread_setcancelstate(PTHREAD_CANCEL_ENABLE, NULL) != 0) + error("Cannot set pthread cancel state to ENABLE."); + + int min_run_every = (int)config_get_number("health", "run at least every seconds", 10); + if(min_run_every < 1) min_run_every = 1; + + BUFFER *wb = buffer_create(100); + + unsigned int loop = 0; + while(health_enabled) { + loop++; + debug(D_HEALTH, "Health monitoring iteration no %u started", loop); + + int oldstate, runnable = 0; + time_t now = time(NULL); + time_t next_run = now + min_run_every; + RRDCALC *rc; + + if (unlikely(pthread_setcancelstate(PTHREAD_CANCEL_DISABLE, &oldstate) != 0)) + error("Cannot set pthread cancel state to DISABLE."); + + rrdhost_rdlock(&localhost); + + // the first loop is to lookup values from the db + for (rc = localhost.alarms; rc; rc = rc->next) { + if (unlikely(!rrdcalc_isrunnable(rc, now, &next_run))) + continue; + + runnable++; + rc->old_value = rc->value; + + // 1. if there is database lookup, do it + // 2. if there is calculation expression, run it + + if (unlikely(RRDCALC_HAS_DB_LOOKUP(rc))) { + time_t old_db_timestamp = rc->db_before; + int value_is_null = 0; + + int ret = rrd2value(rc->rrdset, wb, &rc->value, + rc->dimensions, 1, rc->after, rc->before, rc->group, + rc->options, &rc->db_after, &rc->db_before, &value_is_null); + + if (unlikely(ret != 200)) { + // database lookup failed + rc->value = NAN; + + debug(D_HEALTH, "Health alarm '%s.%s': database lookup returned error %d", rc->chart?rc->chart:"NOCHART", rc->name, ret); + + if (unlikely(!(rc->rrdcalc_flags & RRDCALC_FLAG_DB_ERROR))) { + rc->rrdcalc_flags |= RRDCALC_FLAG_DB_ERROR; + error("Health alarm '%s.%s': database lookup returned error %d", rc->chart?rc->chart:"NOCHART", rc->name, ret); + } + } + else if (unlikely(rc->rrdcalc_flags & RRDCALC_FLAG_DB_ERROR)) + rc->rrdcalc_flags &= ~RRDCALC_FLAG_DB_ERROR; + + if (unlikely(old_db_timestamp == rc->db_before)) { + // database is stale + + debug(D_HEALTH, "Health alarm '%s.%s': database is stale", rc->chart?rc->chart:"NOCHART", rc->name); + + if (unlikely(!(rc->rrdcalc_flags & RRDCALC_FLAG_DB_STALE))) { + rc->rrdcalc_flags |= RRDCALC_FLAG_DB_STALE; + error("Health alarm '%s.%s': database is stale", rc->chart?rc->chart:"NOCHART", rc->name); + } + } + else if (unlikely(rc->rrdcalc_flags & RRDCALC_FLAG_DB_STALE)) + rc->rrdcalc_flags &= ~RRDCALC_FLAG_DB_STALE; + + if (unlikely(value_is_null)) { + // collected value is null + + rc->value = NAN; + + debug(D_HEALTH, "Health alarm '%s.%s': database lookup returned empty value (possibly value is not collected yet)", + rc->chart?rc->chart:"NOCHART", rc->name); + + if (unlikely(!(rc->rrdcalc_flags & RRDCALC_FLAG_DB_NAN))) { + rc->rrdcalc_flags |= RRDCALC_FLAG_DB_NAN; + error("Health alarm '%s.%s': database lookup returned empty value (possibly value is not collected yet)", + rc->chart?rc->chart:"NOCHART", rc->name); + } + } + else if (unlikely(rc->rrdcalc_flags & RRDCALC_FLAG_DB_NAN)) + rc->rrdcalc_flags &= ~RRDCALC_FLAG_DB_NAN; + + debug(D_HEALTH, "Health alarm '%s.%s': database lookup gave value " + CALCULATED_NUMBER_FORMAT, rc->chart?rc->chart:"NOCHART", rc->name, rc->value); + } + + if(unlikely(rc->calculation)) { + if (unlikely(!expression_evaluate(rc->calculation))) { + // calculation failed + + rc->value = NAN; + + debug(D_HEALTH, "Health alarm '%s.%s': failed to evaluate calculation with error: %s", + rc->chart?rc->chart:"NOCHART", rc->name, buffer_tostring(rc->calculation->error_msg)); + + if (unlikely(!(rc->rrdcalc_flags & RRDCALC_FLAG_CALC_ERROR))) { + rc->rrdcalc_flags |= RRDCALC_FLAG_CALC_ERROR; + error("Health alarm '%s.%s': failed to evaluate calculation with error: %s", + rc->chart?rc->chart:"NOCHART", rc->name, buffer_tostring(rc->calculation->error_msg)); + } + } + else { + if (unlikely(rc->rrdcalc_flags & RRDCALC_FLAG_CALC_ERROR)) + rc->rrdcalc_flags &= ~RRDCALC_FLAG_CALC_ERROR; + + debug(D_HEALTH, "Health alarm '%s.%s': calculation expression gave value " + CALCULATED_NUMBER_FORMAT + ": %s (source: %s)", + rc->chart?rc->chart:"NOCHART", rc->name, + rc->calculation->result, + buffer_tostring(rc->calculation->error_msg), + rc->source + ); + + rc->value = rc->calculation->result; + } + } + } + rrdhost_unlock(&localhost); + + if (runnable) { + rrdhost_rdlock(&localhost); + + for (rc = localhost.alarms; rc; rc = rc->next) { + if (unlikely(!rrdcalc_isrunnable(rc, now, &next_run))) + continue; + + int warning_status = RRDCALC_STATUS_UNDEFINED; + int critical_status = RRDCALC_STATUS_UNDEFINED; + + if(unlikely(rc->warning)) { + if(unlikely(!expression_evaluate(rc->warning))) { + // calculation failed + + debug(D_HEALTH, "Health alarm '%s.%s': warning expression failed with error: %s", + rc->chart?rc->chart:"NOCHART", rc->name, buffer_tostring(rc->warning->error_msg)); + + if (unlikely(!(rc->rrdcalc_flags & RRDCALC_FLAG_WARN_ERROR))) { + rc->rrdcalc_flags |= RRDCALC_FLAG_WARN_ERROR; + error("Health alarm '%s.%s': warning expression failed with error: %s", + rc->chart?rc->chart:"NOCHART", rc->name, buffer_tostring(rc->warning->error_msg)); + } + } + else { + if(unlikely(rc->rrdcalc_flags & RRDCALC_FLAG_WARN_ERROR)) + rc->rrdcalc_flags &= ~RRDCALC_FLAG_WARN_ERROR; + + debug(D_HEALTH, "Health alarm '%s.%s': warning expression gave value " + CALCULATED_NUMBER_FORMAT + ": %s (source: %s)", + rc->chart?rc->chart:"NOCHART", rc->name, + rc->warning->result, + buffer_tostring(rc->warning->error_msg), + rc->source + ); + + warning_status = rrdcalc_value2status(rc->warning->result); + } + } + + if(unlikely(rc->critical)) { + if(unlikely(!expression_evaluate(rc->critical))) { + // calculation failed + + debug(D_HEALTH, "Health alarm '%s.%s': critical expression failed with error: %s", + rc->chart?rc->chart:"NOCHART", rc->name, buffer_tostring(rc->critical->error_msg)); + + if (unlikely(!(rc->rrdcalc_flags & RRDCALC_FLAG_CRIT_ERROR))) { + rc->rrdcalc_flags |= RRDCALC_FLAG_CRIT_ERROR; + error("Health alarm '%s.%s': critical expression failed with error: %s", + rc->chart?rc->chart:"NOCHART", rc->name, buffer_tostring(rc->critical->error_msg)); + } + } + else { + if(unlikely(rc->rrdcalc_flags & RRDCALC_FLAG_CRIT_ERROR)) + rc->rrdcalc_flags &= ~RRDCALC_FLAG_CRIT_ERROR; + + debug(D_HEALTH, "Health alarm '%s.%s': critical expression gave value " + CALCULATED_NUMBER_FORMAT + ": %s (source: %s)", + rc->chart?rc->chart:"NOCHART", rc->name, + rc->critical->result, + buffer_tostring(rc->critical->error_msg), + rc->source + ); + + critical_status = rrdcalc_value2status(rc->critical->result); + } + } + + int status = RRDCALC_STATUS_UNDEFINED; + + switch(warning_status) { + case RRDCALC_STATUS_CLEAR: + status = RRDCALC_STATUS_CLEAR; + break; + + case RRDCALC_STATUS_RAISED: + status = RRDCALC_STATUS_WARNING; + break; + + default: + break; + } + + switch(critical_status) { + case RRDCALC_STATUS_CLEAR: + if(status == RRDCALC_STATUS_UNDEFINED) + status = RRDCALC_STATUS_CLEAR; + break; + + case RRDCALC_STATUS_RAISED: + status = RRDCALC_STATUS_CRITICAL; + break; + + default: + break; + } + + if(status != rc->status) { + health_alarm_log(&localhost, time(NULL), rc->name, rc->rrdset->id, rc->rrdset->family, rc->exec, now - rc->last_status_change, rc->old_value, rc->value, rc->status, status, rc->source, rc->units, rc->info); + rc->last_status_change = now; + rc->status = status; + } + + rc->last_updated = now; + rc->next_update = now + rc->update_every; + + if (next_run > rc->next_update) + next_run = rc->next_update; + } + + rrdhost_unlock(&localhost); + } + + if (unlikely(pthread_setcancelstate(oldstate, NULL) != 0)) + error("Cannot set pthread cancel state to RESTORE (%d).", oldstate); + + // execute notifications + // and cleanup + health_alarm_log_process(&localhost); + + now = time(NULL); + if(now < next_run) { + debug(D_HEALTH, "Health monitoring iteration no %u done. Next iteration in %d secs", + loop, (int) (next_run - now)); + sleep_usec(1000000 * (unsigned long long) (next_run - now)); + } + else { + debug(D_HEALTH, "Health monitoring iteration no %u done. Next iteration now", loop); + } + } + + buffer_free(wb); + + info("HEALTH thread exiting"); + pthread_exit(NULL); + return NULL; +} diff --git a/src/health.h b/src/health.h new file mode 100644 index 000000000..ef1158a29 --- /dev/null +++ b/src/health.h @@ -0,0 +1,283 @@ +#ifndef NETDATA_HEALTH_H +#define NETDATA_HEALTH_H + +extern int health_enabled; + +extern int rrdvar_compare(void *a, void *b); + +#define RRDVAR_TYPE_CALCULATED 1 +#define RRDVAR_TYPE_TIME_T 2 +#define RRDVAR_TYPE_COLLECTED 3 +#define RRDVAR_TYPE_TOTAL 4 +#define RRDVAR_TYPE_INT 5 + +// the variables as stored in the variables indexes +// there are 3 indexes: +// 1. at each chart (RRDSET.variables_root_index) +// 2. at each context (RRDFAMILY.variables_root_index) +// 3. at each host (RRDHOST.variables_root_index) +typedef struct rrdvar { + avl avl; + + char *name; + uint32_t hash; + + int type; + void *value; + + time_t last_updated; +} RRDVAR; + +// variables linked to charts +// We link variables to point to the values that are already +// calculated / processed by the normal data collection process +// This means, there will be no speed penalty for using +// these variables +typedef struct rrdsetvar { + char *fullid; // chart type.chart id.variable + char *fullname; // chart type.chart name.variable + char *variable; // variable + + int type; + void *value; + + uint32_t options; + + RRDVAR *local; + RRDVAR *family; + RRDVAR *host; + RRDVAR *family_name; + RRDVAR *host_name; + + struct rrdset *rrdset; + + struct rrdsetvar *next; +} RRDSETVAR; + + +// variables linked to individual dimensions +// We link variables to point the values that are already +// calculated / processed by the normal data collection process +// This means, there will be no speed penalty for using +// these variables +typedef struct rrddimvar { + char *prefix; + char *suffix; + + char *id; // dimension id + char *name; // dimension name + char *fullidid; // chart type.chart id.dimension id + char *fullidname; // chart type.chart id.dimension name + char *fullnameid; // chart type.chart name.dimension id + char *fullnamename; // chart type.chart name.dimension name + + int type; + void *value; + + uint32_t options; + + RRDVAR *local_id; + RRDVAR *local_name; + + RRDVAR *family_id; + RRDVAR *family_name; + + RRDVAR *host_fullidid; + RRDVAR *host_fullidname; + RRDVAR *host_fullnameid; + RRDVAR *host_fullnamename; + + struct rrddim *rrddim; + + struct rrddimvar *next; +} RRDDIMVAR; + +// calculated variables (defined in health configuration) +// These aggregate time-series data at fixed intervals +// (defined in their update_every member below) +// These increase the overhead of netdata. +// +// These calculations are allocated and linked (->next) +// under RRDHOST. +// Then are also linked to RRDSET (of course only when the +// chart is found, via ->rrdset_next and ->rrdset_prev). +// This double-linked list is maintained sorted at all times +// having as RRDSET.calculations the RRDCALC to be processed +// next. + +#define RRDCALC_STATUS_UNINITIALIZED 0 +#define RRDCALC_STATUS_UNDEFINED -1 +#define RRDCALC_STATUS_CLEAR 1 +#define RRDCALC_STATUS_RAISED 2 +#define RRDCALC_STATUS_WARNING 3 +#define RRDCALC_STATUS_CRITICAL 4 + +#define RRDCALC_FLAG_DB_ERROR 0x00000001 +#define RRDCALC_FLAG_DB_NAN 0x00000002 +#define RRDCALC_FLAG_DB_STALE 0x00000004 +#define RRDCALC_FLAG_CALC_ERROR 0x00000008 +#define RRDCALC_FLAG_WARN_ERROR 0x00000010 +#define RRDCALC_FLAG_CRIT_ERROR 0x00000020 + +typedef struct rrdcalc { + char *name; + uint32_t hash; + + char *exec; + + char *chart; // the chart id this should be linked to + uint32_t hash_chart; + + char *source; // the source of this calculation + char *units; + char *info; + + char *dimensions; // the chart dimensions + + int group; // grouping method: average, max, etc. + int before; // ending point in time-series + int after; // starting point in time-series + uint32_t options; // calculation options + int update_every; // update frequency for the calculation + + time_t last_updated; + time_t next_update; + + EVAL_EXPRESSION *calculation; + EVAL_EXPRESSION *warning; + EVAL_EXPRESSION *critical; + + uint32_t rrdcalc_flags; + int status; + + time_t db_after; + time_t db_before; + time_t last_status_change; + + calculated_number value; + calculated_number old_value; + + calculated_number green; + calculated_number red; + + RRDVAR *local; + RRDVAR *family; + RRDVAR *hostid; + RRDVAR *hostname; + + struct rrdset *rrdset; + struct rrdcalc *rrdset_next; + struct rrdcalc *rrdset_prev; + + struct rrdcalc *next; +} RRDCALC; + +#define RRDCALC_HAS_DB_LOOKUP(rc) ((rc)->after) + +// RRDCALCTEMPLATE +// these are to be applied to charts found dynamically +// based on their context. +typedef struct rrdcalctemplate { + char *name; + uint32_t hash_name; + + char *exec; + + char *context; + uint32_t hash_context; + + char *source; // the source of this template + char *units; + char *info; + + char *dimensions; + + int group; // grouping method: average, max, etc. + int before; // ending point in time-series + int after; // starting point in time-series + uint32_t options; // calculation options + int update_every; // update frequency for the calculation + + EVAL_EXPRESSION *calculation; + EVAL_EXPRESSION *warning; + EVAL_EXPRESSION *critical; + + calculated_number green; + calculated_number red; + + struct rrdcalctemplate *next; +} RRDCALCTEMPLATE; + +#define RRDCALCTEMPLATE_HAS_CALCULATION(rt) ((rt)->after) + +#define HEALTH_ENTRY_NOTIFICATIONS_PROCESSED 0x00000001 +#define HEALTH_ENTRY_NOTIFICATIONS_UPDATED 0x00000002 +#define HEALTH_ENTRY_NOTIFICATIONS_EXEC_RUN 0x00000004 +#define HEALTH_ENTRY_NOTIFICATIONS_EXEC_FAILED 0x00000008 + +typedef struct alarm_entry { + uint32_t id; + + time_t when; + time_t duration; + time_t non_clear_duration; + + char *name; + uint32_t hash_name; + + char *chart; + uint32_t hash_chart; + + char *family; + + char *exec; + int exec_code; + + char *source; + char *units; + char *info; + + calculated_number old_value; + calculated_number new_value; + int old_status; + int new_status; + + uint32_t notifications; + + struct alarm_entry *updated_by; + struct alarm_entry *next; +} ALARM_ENTRY; + +typedef struct alarm_log { + uint32_t nextid; + unsigned int count; + unsigned int max; + ALARM_ENTRY *alarms; + pthread_rwlock_t alarm_log_rwlock; +} ALARM_LOG; + +#include "rrd.h" + +extern void rrdsetvar_rename_all(RRDSET *st); +extern RRDSETVAR *rrdsetvar_create(RRDSET *st, const char *variable, int type, void *value, uint32_t options); +extern void rrdsetvar_free(RRDSETVAR *rs); + +extern void rrddimvar_rename_all(RRDDIM *rd); +extern RRDDIMVAR *rrddimvar_create(RRDDIM *rd, int type, const char *prefix, const char *suffix, void *value, uint32_t options); +extern void rrddimvar_free(RRDDIMVAR *rs); + +extern void rrdsetcalc_link_matching(RRDSET *st); +extern void rrdsetcalc_unlink(RRDCALC *rc); +extern void rrdcalctemplate_link_matching(RRDSET *st); +extern RRDCALC *rrdcalc_find(RRDSET *st, const char *name); + +extern void health_init(void); +extern void *health_main(void *ptr); + +extern void health_reload(void); + +extern int health_variable_lookup(const char *variable, uint32_t hash, RRDCALC *rc, calculated_number *result); +extern void health_alarms2json(RRDHOST *host, BUFFER *wb, int all); +extern void health_alarm_log2json(RRDHOST *host, BUFFER *wb); + +#endif //NETDATA_HEALTH_H @@ -1,241 +1,344 @@ -#ifdef HAVE_CONFIG_H -#include <config.h> -#endif -#include <time.h> -#include <syslog.h> -#include <errno.h> -#include <string.h> -#include <unistd.h> -#include <stdlib.h> - -#include "log.h" #include "common.h" - -// ---------------------------------------------------------------------------- -// LOG - const char *program_name = ""; unsigned long long debug_flags = DEBUG; -int silent = 0; +int access_log_syslog = 1; +int error_log_syslog = 1; +int output_log_syslog = 1; // debug log -int access_fd = -1; +int stdaccess_fd = -1; FILE *stdaccess = NULL; -int access_log_syslog = 1; -int error_log_syslog = 1; -int output_log_syslog = 1; // debug log +const char *stdaccess_filename = NULL; +const char *stderr_filename = NULL; +const char *stdout_filename = NULL; + +void syslog_init(void) { + static int i = 0; + + if(!i) { + openlog(program_name, LOG_PID, LOG_DAEMON); + i = 1; + } +} + +int open_log_file(int fd, FILE **fp, const char *filename, int *enabled_syslog) { + int f, t; + + if(!filename || !*filename || !strcmp(filename, "none")) + filename = "/dev/null"; + + if(!strcmp(filename, "syslog")) { + filename = "/dev/null"; + syslog_init(); + if(enabled_syslog) *enabled_syslog = 1; + } + else if(enabled_syslog) *enabled_syslog = 0; + + // don't do anything if the user is willing + // to have the standard one + if(!strcmp(filename, "system")) { + if(fd != -1) return fd; + filename = "stdout"; + } + + if(!strcmp(filename, "stdout")) + f = STDOUT_FILENO; + + else if(!strcmp(filename, "stderr")) + f = STDERR_FILENO; + + else { + f = open(filename, O_WRONLY | O_APPEND | O_CREAT, 0664); + if(f == -1) { + error("Cannot open file '%s'. Leaving %d to its default.", filename, fd); + return fd; + } + } + + // if there is a level-2 file pointer + // flush it before switching the level-1 fds + if(fp && *fp) + fflush(*fp); + + if(fd != f && fd != -1) { + // it automatically closes + t = dup2(f, fd); + if (t == -1) { + error("Cannot dup2() new fd %d to old fd %d for '%s'", f, fd, filename); + close(f); + return fd; + } + // info("dup2() new fd %d to old fd %d for '%s'", f, fd, filename); + close(f); + } + else fd = f; + + if(fp && !*fp) { + *fp = fdopen(fd, "a"); + if (!*fp) + error("Cannot fdopen() fd %d ('%s')", fd, filename); + else { + if (setvbuf(*fp, NULL, _IOLBF, 0) != 0) + error("Cannot set line buffering on fd %d ('%s')", fd, filename); + } + } + + return fd; +} + +void reopen_all_log_files() { + if(stdout_filename) + open_log_file(STDOUT_FILENO, &stdout, stdout_filename, &output_log_syslog); + + if(stderr_filename) + open_log_file(STDERR_FILENO, &stderr, stderr_filename, &error_log_syslog); + + if(stdaccess_filename) + stdaccess_fd = open_log_file(stdaccess_fd, &stdaccess, stdaccess_filename, &access_log_syslog); +} + +void open_all_log_files() { + // disable stdin + open_log_file(STDIN_FILENO, &stdin, "/dev/null", NULL); + + open_log_file(STDOUT_FILENO, &stdout, stdout_filename, &output_log_syslog); + open_log_file(STDERR_FILENO, &stderr, stderr_filename, &error_log_syslog); + stdaccess_fd = open_log_file(stdaccess_fd, &stdaccess, stdaccess_filename, &access_log_syslog); +} + +// ---------------------------------------------------------------------------- +// error log throttling time_t error_log_throttle_period = 1200; unsigned long error_log_errors_per_period = 200; int error_log_limit(int reset) { - static time_t start = 0; - static unsigned long counter = 0, prevented = 0; - - // do not throttle if the period is 0 - if(error_log_throttle_period == 0) - return 0; - - // prevent all logs if the errors per period is 0 - if(error_log_errors_per_period == 0) - return 1; - - time_t now = time(NULL); - if(!start) start = now; - - if(reset) { - if(prevented) { - log_date(stderr); - fprintf(stderr, "%s: Resetting logging for process '%s' (prevented %lu logs in the last %ld seconds).\n" - , program_name - , program_name - , prevented - , now - start - ); - } - - start = now; - counter = 0; - prevented = 0; - } - - // detect if we log too much - counter++; - - if(now - start > error_log_throttle_period) { - if(prevented) { - log_date(stderr); - fprintf(stderr, "%s: Resuming logging from process '%s' (prevented %lu logs in the last %ld seconds).\n" - , program_name - , program_name - , prevented - , error_log_throttle_period - ); - } - - // restart the period accounting - start = now; - counter = 1; - prevented = 0; - - // log this error - return 0; - } - - if(counter > error_log_errors_per_period) { - if(!prevented) { - log_date(stderr); - fprintf(stderr, "%s: Too many logs (%lu logs in %ld seconds, threshold is set to %lu logs in %ld seconds). Preventing more logs from process '%s' for %ld seconds.\n" - , program_name - , counter - , now - start - , error_log_errors_per_period - , error_log_throttle_period - , program_name - , start + error_log_throttle_period - now); - } - - prevented++; - - // prevent logging this error - return 1; - } - - return 0; + static time_t start = 0; + static unsigned long counter = 0, prevented = 0; + + // do not throttle if the period is 0 + if(error_log_throttle_period == 0) + return 0; + + // prevent all logs if the errors per period is 0 + if(error_log_errors_per_period == 0) + return 1; + + time_t now = time(NULL); + if(!start) start = now; + + if(reset) { + if(prevented) { + log_date(stderr); + fprintf(stderr, "%s: Resetting logging for process '%s' (prevented %lu logs in the last %ld seconds).\n" + , program_name + , program_name + , prevented + , now - start + ); + } + + start = now; + counter = 0; + prevented = 0; + } + + // detect if we log too much + counter++; + + if(now - start > error_log_throttle_period) { + if(prevented) { + log_date(stderr); + fprintf(stderr, "%s: Resuming logging from process '%s' (prevented %lu logs in the last %ld seconds).\n" + , program_name + , program_name + , prevented + , error_log_throttle_period + ); + } + + // restart the period accounting + start = now; + counter = 1; + prevented = 0; + + // log this error + return 0; + } + + if(counter > error_log_errors_per_period) { + if(!prevented) { + log_date(stderr); + fprintf(stderr, "%s: Too many logs (%lu logs in %ld seconds, threshold is set to %lu logs in %ld seconds). Preventing more logs from process '%s' for %ld seconds.\n" + , program_name + , counter + , now - start + , error_log_errors_per_period + , error_log_throttle_period + , program_name + , start + error_log_throttle_period - now); + } + + prevented++; + + // prevent logging this error + return 1; + } + + return 0; } +// ---------------------------------------------------------------------------- +// print the date + +// FIXME +// this should print the date in a buffer the way it +// is now, logs from multiple threads may be multiplexed + void log_date(FILE *out) { - char outstr[200]; - time_t t; - struct tm *tmp, tmbuf; + char outstr[24]; + time_t t; + struct tm *tmp, tmbuf; - t = time(NULL); - tmp = localtime_r(&t, &tmbuf); + t = time(NULL); + tmp = localtime_r(&t, &tmbuf); - if (tmp == NULL) return; - if (strftime(outstr, sizeof(outstr), "%y-%m-%d %H:%M:%S", tmp) == 0) return; + if (tmp == NULL) return; + if (unlikely(strftime(outstr, sizeof(outstr), "%y-%m-%d %H:%M:%S", tmp) == 0)) return; - fprintf(out, "%s: ", outstr); + fprintf(out, "%s: ", outstr); } +// ---------------------------------------------------------------------------- +// debug log + void debug_int( const char *file, const char *function, const unsigned long line, const char *fmt, ... ) { - va_list args; - - log_date(stdout); - va_start( args, fmt ); - fprintf(stdout, "DEBUG (%04lu@%-10.10s:%-15.15s): %s: ", line, file, function, program_name); - vfprintf( stdout, fmt, args ); - va_end( args ); - fprintf(stdout, "\n"); - // fflush( stdout ); - - if(output_log_syslog) { - va_start( args, fmt ); - vsyslog(LOG_ERR, fmt, args ); - va_end( args ); - } + va_list args; + + log_date(stdout); + va_start( args, fmt ); + printf("DEBUG (%04lu@%-10.10s:%-15.15s): %s: ", line, file, function, program_name); + vprintf(fmt, args); + va_end( args ); + putchar('\n'); + + if(output_log_syslog) { + va_start( args, fmt ); + vsyslog(LOG_ERR, fmt, args ); + va_end( args ); + } + + fflush(stdout); } +// ---------------------------------------------------------------------------- +// info log + void info_int( const char *file, const char *function, const unsigned long line, const char *fmt, ... ) { - va_list args; + va_list args; - // prevent logging too much - if(error_log_limit(0)) return; + // prevent logging too much + if(error_log_limit(0)) return; - log_date(stderr); + log_date(stderr); - va_start( args, fmt ); - if(debug_flags) fprintf(stderr, "INFO (%04lu@%-10.10s:%-15.15s): %s: ", line, file, function, program_name); - else fprintf(stderr, "INFO: %s: ", program_name); - vfprintf( stderr, fmt, args ); - va_end( args ); + va_start( args, fmt ); + if(debug_flags) fprintf(stderr, "INFO (%04lu@%-10.10s:%-15.15s): %s: ", line, file, function, program_name); + else fprintf(stderr, "INFO: %s: ", program_name); + vfprintf( stderr, fmt, args ); + va_end( args ); - fprintf(stderr, "\n"); + fputc('\n', stderr); - if(error_log_syslog) { - va_start( args, fmt ); - vsyslog(LOG_INFO, fmt, args ); - va_end( args ); - } + if(error_log_syslog) { + va_start( args, fmt ); + vsyslog(LOG_INFO, fmt, args ); + va_end( args ); + } } +// ---------------------------------------------------------------------------- +// error log + void error_int( const char *prefix, const char *file, const char *function, const unsigned long line, const char *fmt, ... ) { - va_list args; - - // prevent logging too much - if(error_log_limit(0)) return; - - log_date(stderr); - - va_start( args, fmt ); - if(debug_flags) fprintf(stderr, "%s (%04lu@%-10.10s:%-15.15s): %s: ", prefix, line, file, function, program_name); - else fprintf(stderr, "%s: %s: ", prefix, program_name); - vfprintf( stderr, fmt, args ); - va_end( args ); - - if(errno) { - char buf[200]; - char *s = strerror_r(errno, buf, 200); - fprintf(stderr, " (errno %d, %s)\n", errno, s); - errno = 0; - } - else fprintf(stderr, "\n"); - - if(error_log_syslog) { - va_start( args, fmt ); - vsyslog(LOG_ERR, fmt, args ); - va_end( args ); - } + va_list args; + + // prevent logging too much + if(error_log_limit(0)) return; + + log_date(stderr); + + va_start( args, fmt ); + if(debug_flags) fprintf(stderr, "%s (%04lu@%-10.10s:%-15.15s): %s: ", prefix, line, file, function, program_name); + else fprintf(stderr, "%s: %s: ", prefix, program_name); + vfprintf( stderr, fmt, args ); + va_end( args ); + + if(errno) { + char buf[1024]; + fprintf(stderr, " (errno %d, %s)\n", errno, strerror_r(errno, buf, 1023)); + errno = 0; + } + else + fputc('\n', stderr); + + if(error_log_syslog) { + va_start( args, fmt ); + vsyslog(LOG_ERR, fmt, args ); + va_end( args ); + } } void fatal_int( const char *file, const char *function, const unsigned long line, const char *fmt, ... ) { - va_list args; + va_list args; - log_date(stderr); + log_date(stderr); - va_start( args, fmt ); - if(debug_flags) fprintf(stderr, "FATAL (%04lu@%-10.10s:%-15.15s): %s: ", line, file, function, program_name); - else fprintf(stderr, "FATAL: %s: ", program_name); - vfprintf( stderr, fmt, args ); - va_end( args ); + va_start( args, fmt ); + if(debug_flags) fprintf(stderr, "FATAL (%04lu@%-10.10s:%-15.15s): %s: ", line, file, function, program_name); + else fprintf(stderr, "FATAL: %s: ", program_name); + vfprintf( stderr, fmt, args ); + va_end( args ); - perror(" # "); - fprintf(stderr, "\n"); + perror(" # "); + fputc('\n', stderr); - if(error_log_syslog) { - va_start( args, fmt ); - vsyslog(LOG_CRIT, fmt, args ); - va_end( args ); - } + if(error_log_syslog) { + va_start( args, fmt ); + vsyslog(LOG_CRIT, fmt, args ); + va_end( args ); + } - exit(1); + netdata_cleanup_and_exit(1); } +// ---------------------------------------------------------------------------- +// access log + void log_access( const char *fmt, ... ) { - va_list args; - - if(stdaccess) { - log_date(stdaccess); - - va_start( args, fmt ); - vfprintf( stdaccess, fmt, args ); - va_end( args ); - fprintf( stdaccess, "\n"); - // fflush( stdaccess ); - } - - if(access_log_syslog) { - va_start( args, fmt ); - vsyslog(LOG_INFO, fmt, args ); - va_end( args ); - } + va_list args; + + if(stdaccess) { + log_date(stdaccess); + + va_start( args, fmt ); + vfprintf( stdaccess, fmt, args ); + va_end( args ); + fputc('\n', stdaccess); + } + + if(access_log_syslog) { + va_start( args, fmt ); + vsyslog(LOG_INFO, fmt, args ); + va_end( args ); + } } @@ -1,18 +1,14 @@ -#include <stdio.h> -#include <stdarg.h> -#include <time.h> - #ifndef NETDATA_LOG_H #define NETDATA_LOG_H 1 -#define D_WEB_BUFFER 0x00000001 -#define D_WEB_CLIENT 0x00000002 -#define D_LISTENER 0x00000004 -#define D_WEB_DATA 0x00000008 -#define D_OPTIONS 0x00000010 +#define D_WEB_BUFFER 0x00000001 +#define D_WEB_CLIENT 0x00000002 +#define D_LISTENER 0x00000004 +#define D_WEB_DATA 0x00000008 +#define D_OPTIONS 0x00000010 #define D_PROCNETDEV_LOOP 0x00000020 -#define D_RRD_STATS 0x00000040 -#define D_WEB_CLIENT_ACCESS 0x00000080 +#define D_RRD_STATS 0x00000040 +#define D_WEB_CLIENT_ACCESS 0x00000080 #define D_TC_LOOP 0x00000100 #define D_DEFLATE 0x00000200 #define D_CONFIG 0x00000400 @@ -20,13 +16,15 @@ #define D_CHILDS 0x00001000 #define D_EXIT 0x00002000 #define D_CHECKS 0x00004000 -#define D_NFACCT_LOOP 0x00008000 -#define D_PROCFILE 0x00010000 -#define D_RRD_CALLS 0x00020000 -#define D_DICTIONARY 0x00040000 -#define D_MEMORY 0x00080000 +#define D_NFACCT_LOOP 0x00008000 +#define D_PROCFILE 0x00010000 +#define D_RRD_CALLS 0x00020000 +#define D_DICTIONARY 0x00040000 +#define D_MEMORY 0x00080000 #define D_CGROUP 0x00100000 -#define D_REGISTRY 0x00200000 +#define D_REGISTRY 0x00200000 +#define D_VARIABLES 0x00400000 +#define D_HEALTH 0x00800000 //#define DEBUG (D_WEB_CLIENT_ACCESS|D_LISTENER|D_RRD_STATS) //#define DEBUG 0xffffffff @@ -36,11 +34,13 @@ extern unsigned long long debug_flags; extern const char *program_name; -extern int silent; - -extern int access_fd; +extern int stdaccess_fd; extern FILE *stdaccess; +extern const char *stdaccess_filename; +extern const char *stderr_filename; +extern const char *stdout_filename; + extern int access_log_syslog; extern int error_log_syslog; extern int output_log_syslog; @@ -49,19 +49,23 @@ extern time_t error_log_throttle_period; extern unsigned long error_log_errors_per_period; extern int error_log_limit(int reset); +extern void open_all_log_files(); +extern void reopen_all_log_files(); + #define error_log_limit_reset() do { error_log_limit(1); } while(0) +#define error_log_limit_unlimited() do { error_log_throttle_period = 0; } while(0) -#define debug(type, args...) do { if(unlikely(!silent && (debug_flags & type))) debug_int(__FILE__, __FUNCTION__, __LINE__, ##args); } while(0) +#define debug(type, args...) do { if(unlikely(debug_flags & type)) debug_int(__FILE__, __FUNCTION__, __LINE__, ##args); } while(0) #define info(args...) info_int(__FILE__, __FUNCTION__, __LINE__, ##args) #define infoerr(args...) error_int("INFO", __FILE__, __FUNCTION__, __LINE__, ##args) #define error(args...) error_int("ERROR", __FILE__, __FUNCTION__, __LINE__, ##args) #define fatal(args...) fatal_int(__FILE__, __FUNCTION__, __LINE__, ##args) extern void log_date(FILE *out); -extern void debug_int( const char *file, const char *function, const unsigned long line, const char *fmt, ... ); -extern void info_int( const char *file, const char *function, const unsigned long line, const char *fmt, ... ); -extern void error_int( const char *prefix, const char *file, const char *function, const unsigned long line, const char *fmt, ... ); -extern void fatal_int( const char *file, const char *function, const unsigned long line, const char *fmt, ... ) __attribute__ ((noreturn)); -extern void log_access( const char *fmt, ... ); +extern void debug_int( const char *file, const char *function, const unsigned long line, const char *fmt, ... ) __attribute__ (( format (printf, 4, 5))); +extern void info_int( const char *file, const char *function, const unsigned long line, const char *fmt, ... ) __attribute__ (( format (printf, 4, 5))); +extern void error_int( const char *prefix, const char *file, const char *function, const unsigned long line, const char *fmt, ... ) __attribute__ (( format (printf, 5, 6))); +extern void fatal_int( const char *file, const char *function, const unsigned long line, const char *fmt, ... ) __attribute__ ((noreturn, format (printf, 4, 5))); +extern void log_access( const char *fmt, ... ) __attribute__ (( format (printf, 1, 2))); #endif /* NETDATA_LOG_H */ diff --git a/src/main.c b/src/main.c index ec3c59ca7..4d55a1425 100644 --- a/src/main.c +++ b/src/main.c @@ -1,573 +1,702 @@ -#ifdef HAVE_CONFIG_H -#include <config.h> -#endif -#include <stdio.h> -#include <stdlib.h> -#include <unistd.h> -#include <signal.h> -#include <syslog.h> -#include <pthread.h> -#include <string.h> -#include <errno.h> -#include <sys/types.h> -#include <sys/wait.h> -#include <sys/time.h> -#include <sys/resource.h> -#include <sys/mman.h> -#include <sys/prctl.h> - #include "common.h" -#include "log.h" -#include "daemon.h" -#include "web_server.h" -#include "popen.h" -#include "appconfig.h" -#include "web_client.h" -#include "rrd.h" -#include "rrd2json.h" - -#include "unit_test.h" - -#include "plugins_d.h" -#include "plugin_idlejitter.h" -#include "plugin_tc.h" -#include "plugin_checks.h" -#include "plugin_proc.h" -#include "plugin_nfacct.h" -#include "registry.h" - -#include "main.h" extern void *cgroups_main(void *ptr); -volatile sig_atomic_t netdata_exit = 0; - -void netdata_cleanup_and_exit(int ret) -{ - netdata_exit = 1; - rrdset_save_all(); - // kill_childs(); - - // let it log a few more error messages - error_log_limit_reset(); +void netdata_cleanup_and_exit(int ret) { + netdata_exit = 1; - if(pidfd != -1) { - if(ftruncate(pidfd, 0) != 0) - error("Cannot truncate pidfile '%s'.", pidfile); + error_log_limit_unlimited(); - close(pidfd); - pidfd = -1; - } + info("Called: netdata_cleanup_and_exit()"); +#ifdef NETDATA_INTERNAL_CHECKS + rrdset_free_all(); +#else + rrdset_save_all(); +#endif + // kill_childs(); - if(pidfile[0]) { - if(unlink(pidfile) != 0) - error("Cannot unlink pidfile '%s'.", pidfile); - } + if(pidfile[0]) { + if(unlink(pidfile) != 0) + error("Cannot unlink pidfile '%s'.", pidfile); + } - info("NetData exiting. Bye bye..."); - exit(ret); + info("NetData exiting. Bye bye..."); + exit(ret); } struct netdata_static_thread { - char *name; + char *name; - char *config_section; - char *config_name; + char *config_section; + char *config_name; - int enabled; + int enabled; - pthread_t *thread; - - void (*init_routine) (void); - void *(*start_routine) (void *); -}; - -struct netdata_static_thread static_threads[] = { - {"tc", "plugins", "tc", 1, NULL, NULL, tc_main}, - {"idlejitter", "plugins", "idlejitter", 1, NULL, NULL, cpuidlejitter_main}, - {"proc", "plugins", "proc", 1, NULL, NULL, proc_main}, - {"cgroups", "plugins", "cgroups", 1, NULL, NULL, cgroups_main}, + pthread_t *thread; + void (*init_routine) (void); + void *(*start_routine) (void *); +} static_threads[] = { #ifdef INTERNAL_PLUGIN_NFACCT - // nfacct requires root access - // so, we build it as an external plugin with setuid to root - {"nfacct", "plugins", "nfacct", 1, NULL, NULL, nfacct_main}, +// nfacct requires root access + // so, we build it as an external plugin with setuid to root + {"nfacct", "plugins", "nfacct", 1, NULL, NULL, nfacct_main}, #endif - {"plugins.d", NULL, NULL, 1, NULL, NULL, pluginsd_main}, - {"check", "plugins", "checks", 0, NULL, NULL, checks_main}, - {"web", NULL, NULL, 1, NULL, NULL, socket_listen_main}, - {NULL, NULL, NULL, 0, NULL, NULL, NULL} + {"tc", "plugins", "tc", 1, NULL, NULL, tc_main}, + {"idlejitter", "plugins", "idlejitter", 1, NULL, NULL, cpuidlejitter_main}, + {"proc", "plugins", "proc", 1, NULL, NULL, proc_main}, + {"cgroups", "plugins", "cgroups", 1, NULL, NULL, cgroups_main}, + {"check", "plugins", "checks", 0, NULL, NULL, checks_main}, + {"health", NULL, NULL, 1, NULL, NULL, health_main}, + {"plugins.d", NULL, NULL, 1, NULL, NULL, pluginsd_main}, + {"web", NULL, NULL, 1, NULL, NULL, socket_listen_main_multi_threaded}, + {"web-single-threaded", NULL, NULL, 0, NULL, NULL, socket_listen_main_single_threaded}, + {NULL, NULL, NULL, 0, NULL, NULL, NULL} }; +void web_server_threading_selection(void) { + int threaded = config_get_boolean("global", "multi threaded web server", 1); + + int i; + for(i = 0; static_threads[i].name ; i++) { + if(static_threads[i].start_routine == socket_listen_main_multi_threaded) + static_threads[i].enabled = threaded?1:0; + + if(static_threads[i].start_routine == socket_listen_main_single_threaded) + static_threads[i].enabled = threaded?0:1; + } + + web_client_timeout = (int) config_get_number("global", "disconnect idle web clients after seconds", DEFAULT_DISCONNECT_IDLE_WEB_CLIENTS_AFTER_SECONDS); + + web_donotrack_comply = config_get_boolean("global", "respect web browser do not track policy", web_donotrack_comply); + +#ifdef NETDATA_WITH_ZLIB + web_enable_gzip = config_get_boolean("global", "enable web responses gzip compression", web_enable_gzip); + + char *s = config_get("global", "web compression strategy", "default"); + if(!strcmp(s, "default")) + web_gzip_strategy = Z_DEFAULT_STRATEGY; + else if(!strcmp(s, "filtered")) + web_gzip_strategy = Z_FILTERED; + else if(!strcmp(s, "huffman only")) + web_gzip_strategy = Z_HUFFMAN_ONLY; + else if(!strcmp(s, "rle")) + web_gzip_strategy = Z_RLE; + else if(!strcmp(s, "fixed")) + web_gzip_strategy = Z_FIXED; + else { + error("Invalid compression strategy '%s'. Valid strategies are 'default', 'filtered', 'huffman only', 'rle' and 'fixed'. Proceeding with 'default'.", s); + web_gzip_strategy = Z_DEFAULT_STRATEGY; + } + + web_gzip_level = (int)config_get_number("global", "web compression level", 3); + if(web_gzip_level < 1) { + error("Invalid compression level %d. Valid levels are 1 (fastest) to 9 (best ratio). Proceeding with level 1 (fastest compression).", web_gzip_level); + web_gzip_level = 1; + } + else if(web_gzip_level > 9) { + error("Invalid compression level %d. Valid levels are 1 (fastest) to 9 (best ratio). Proceeding with level 9 (best compression).", web_gzip_level); + web_gzip_level = 9; + } +#endif /* NETDATA_WITH_ZLIB */ +} + + int killpid(pid_t pid, int sig) { - int ret = -1; - debug(D_EXIT, "Request to kill pid %d", pid); - - errno = 0; - if(kill(pid, 0) == -1) { - switch(errno) { - case ESRCH: - error("Request to kill pid %d, but it is not running.", pid); - break; - - case EPERM: - error("Request to kill pid %d, but I do not have enough permissions.", pid); - break; - - default: - error("Request to kill pid %d, but I received an error.", pid); - break; - } - } - else { - errno = 0; - ret = kill(pid, sig); - if(ret == -1) { - switch(errno) { - case ESRCH: - error("Cannot kill pid %d, but it is not running.", pid); - break; - - case EPERM: - error("Cannot kill pid %d, but I do not have enough permissions.", pid); - break; - - default: - error("Cannot kill pid %d, but I received an error.", pid); - break; - } - } - } - - return ret; + int ret = -1; + debug(D_EXIT, "Request to kill pid %d", pid); + + errno = 0; + if(kill(pid, 0) == -1) { + switch(errno) { + case ESRCH: + error("Request to kill pid %d, but it is not running.", pid); + break; + + case EPERM: + error("Request to kill pid %d, but I do not have enough permissions.", pid); + break; + + default: + error("Request to kill pid %d, but I received an error.", pid); + break; + } + } + else { + errno = 0; + ret = kill(pid, sig); + if(ret == -1) { + switch(errno) { + case ESRCH: + error("Cannot kill pid %d, but it is not running.", pid); + break; + + case EPERM: + error("Cannot kill pid %d, but I do not have enough permissions.", pid); + break; + + default: + error("Cannot kill pid %d, but I received an error.", pid); + break; + } + } + } + + return ret; } void kill_childs() { - siginfo_t info; - - struct web_client *w; - for(w = web_clients; w ; w = w->next) { - debug(D_EXIT, "Stopping web client %s", w->client_ip); - pthread_cancel(w->thread); - pthread_join(w->thread, NULL); - } - - int i; - for (i = 0; static_threads[i].name != NULL ; i++) { - if(static_threads[i].thread) { - debug(D_EXIT, "Stopping %s thread", static_threads[i].name); - pthread_cancel(*static_threads[i].thread); - pthread_join(*static_threads[i].thread, NULL); - static_threads[i].thread = NULL; - } - } - - if(tc_child_pid) { - info("Killing tc-qos-helper procees"); - if(killpid(tc_child_pid, SIGTERM) != -1) - waitid(P_PID, (id_t) tc_child_pid, &info, WEXITED); - } - tc_child_pid = 0; - - struct plugind *cd; - for(cd = pluginsd_root ; cd ; cd = cd->next) { - debug(D_EXIT, "Stopping %s plugin thread", cd->id); - pthread_cancel(cd->thread); - pthread_join(cd->thread, NULL); - - if(cd->pid && !cd->obsolete) { - debug(D_EXIT, "killing %s plugin process", cd->id); - if(killpid(cd->pid, SIGTERM) != -1) - waitid(P_PID, (id_t) cd->pid, &info, WEXITED); - } - } - - // if, for any reason there is any child exited - // catch it here - waitid(P_PID, 0, &info, WEXITED|WNOHANG); - - debug(D_EXIT, "All threads/childs stopped."); + siginfo_t info; + + struct web_client *w; + for(w = web_clients; w ; w = w->next) { + debug(D_EXIT, "Stopping web client %s", w->client_ip); + pthread_cancel(w->thread); + pthread_join(w->thread, NULL); + } + + int i; + for (i = 0; static_threads[i].name != NULL ; i++) { + if(static_threads[i].thread) { + debug(D_EXIT, "Stopping %s thread", static_threads[i].name); + pthread_cancel(*static_threads[i].thread); + pthread_join(*static_threads[i].thread, NULL); + static_threads[i].thread = NULL; + } + } + + if(tc_child_pid) { + info("Killing tc-qos-helper procees"); + if(killpid(tc_child_pid, SIGTERM) != -1) + waitid(P_PID, (id_t) tc_child_pid, &info, WEXITED); + } + tc_child_pid = 0; + + struct plugind *cd; + for(cd = pluginsd_root ; cd ; cd = cd->next) { + debug(D_EXIT, "Stopping %s plugin thread", cd->id); + pthread_cancel(cd->thread); + pthread_join(cd->thread, NULL); + + if(cd->pid && !cd->obsolete) { + debug(D_EXIT, "killing %s plugin process", cd->id); + if(killpid(cd->pid, SIGTERM) != -1) + waitid(P_PID, (id_t) cd->pid, &info, WEXITED); + } + } + + // if, for any reason there is any child exited + // catch it here + waitid(P_PID, 0, &info, WEXITED|WNOHANG); + + debug(D_EXIT, "All threads/childs stopped."); } +struct option_def options[] = { + // opt description arg name default value + {'c', "Load alternate configuration file", "config_file", CONFIG_DIR "/" CONFIG_FILENAME}, + {'D', "Disable fork into background", NULL, NULL}, + {'h', "Display help message", NULL, NULL}, + {'P', "File to save a pid while running", "FILE", NULL}, + {'i', "The IP address to listen to.", "address", "All addresses"}, + {'k', "Check daemon configuration.", NULL, NULL}, + {'p', "Port to listen. Can be from 1 to 65535.", "port_number", "19999"}, + {'s', "Path to access host /proc and /sys when running in a container.", "PATH", NULL}, + {'t', "The frequency in seconds, for data collection. \ +Same as 'update every' config file option.", "seconds", "1"}, + {'u', "System username to run as.", "username", "netdata"}, + {'v', "Version of the program", NULL, NULL}, + {'W', "vendor options.", "stacksize=N|unittest|debug_flags=N", NULL}, +}; + +void help(int exitcode) { + FILE *stream; + if(exitcode == 0) + stream = stdout; + else + stream = stderr; + + int num_opts = sizeof(options) / sizeof(struct option_def); + int i; + int max_len_arg = 0; + + // Compute maximum argument length + for( i = 0; i < num_opts; i++ ) { + if(options[i].arg_name) { + int len_arg = (int)strlen(options[i].arg_name); + if(len_arg > max_len_arg) max_len_arg = len_arg; + } + } + + fprintf(stream, "SYNOPSIS: netdata [options]\n"); + fprintf(stream, "\n"); + fprintf(stream, "Options:\n"); + + // Output options description. + for( i = 0; i < num_opts; i++ ) { + fprintf(stream, " -%c %-*s %s", options[i].val, max_len_arg, options[i].arg_name ? options[i].arg_name : "", options[i].description); + if(options[i].default_value) { + fprintf(stream, " Default: %s\n", options[i].default_value); + } else { + fprintf(stream, "\n"); + } + } + + fflush(stream); + exit(exitcode); +} + +// TODO: Remove this function with the nix major release. +void remove_option(int opt_index, int *argc, char **argv) { + int i = opt_index; + // remove the options. + do { + *argc = *argc - 1; + for(i = opt_index; i < *argc; i++) { + argv[i] = argv[i+1]; + } + i = opt_index; + } while(argv[i][0] != '-' && opt_index >= *argc); +} + +static const char *verify_required_directory(const char *dir) { + if(chdir(dir) == -1) + fatal("Cannot cd to directory '%s'", dir); + + DIR *d = opendir(dir); + if(!d) + fatal("Cannot examine the contents of directory '%s'", dir); + closedir(d); + + return dir; +} int main(int argc, char **argv) { - int i; - int config_loaded = 0; - int dont_fork = 0; - size_t wanted_stacksize = 0, stacksize = 0; - pthread_attr_t attr; - - // global initialization - get_HZ(); - - // set the name for logging - program_name = "netdata"; - - // parse the arguments - for(i = 1; i < argc ; i++) { - if(strcmp(argv[i], "-c") == 0 && (i+1) < argc) { - if(load_config(argv[i+1], 1) != 1) { - error("Cannot load configuration file %s.", argv[i+1]); - exit(1); - } - else { - debug(D_OPTIONS, "Configuration loaded from %s.", argv[i+1]); - config_loaded = 1; - } - i++; - } - else if(strcmp(argv[i], "-df") == 0 && (i+1) < argc) { config_set("global", "debug flags", argv[i+1]); debug_flags = strtoull(argv[i+1], NULL, 0); i++; } - else if(strcmp(argv[i], "-p") == 0 && (i+1) < argc) { config_set("global", "port", argv[i+1]); i++; } - else if(strcmp(argv[i], "-u") == 0 && (i+1) < argc) { config_set("global", "run as user", argv[i+1]); i++; } - else if(strcmp(argv[i], "-l") == 0 && (i+1) < argc) { config_set("global", "history", argv[i+1]); i++; } - else if(strcmp(argv[i], "-t") == 0 && (i+1) < argc) { config_set("global", "update every", argv[i+1]); i++; } - else if(strcmp(argv[i], "-ch") == 0 && (i+1) < argc) { config_set("global", "host access prefix", argv[i+1]); i++; } - else if(strcmp(argv[i], "-stacksize") == 0 && (i+1) < argc) { config_set("global", "pthread stack size", argv[i+1]); i++; } - else if(strcmp(argv[i], "-nodaemon") == 0 || strcmp(argv[i], "-nd") == 0) dont_fork = 1; - else if(strcmp(argv[i], "-pidfile") == 0 && (i+1) < argc) { - i++; - strncpyz(pidfile, argv[i], FILENAME_MAX); - } - else if(strcmp(argv[i], "--unittest") == 0) { - rrd_update_every = 1; - if(run_all_mockup_tests()) exit(1); - if(unit_test_storage()) exit(1); - fprintf(stderr, "\n\nALL TESTS PASSED\n\n"); - exit(0); - } - else { - fprintf(stderr, "Cannot understand option '%s'.\n", argv[i]); - fprintf(stderr, "\nUSAGE: %s [-d] [-l LINES_TO_SAVE] [-u UPDATE_TIMER] [-p LISTEN_PORT] [-df debug flags].\n\n", argv[0]); - fprintf(stderr, " -c CONFIG FILE the configuration file to load. Default: %s.\n", CONFIG_DIR "/" CONFIG_FILENAME); - fprintf(stderr, " -l LINES_TO_SAVE can be from 5 to %d lines in JSON data. Default: %d.\n", RRD_HISTORY_ENTRIES_MAX, RRD_DEFAULT_HISTORY_ENTRIES); - fprintf(stderr, " -t UPDATE_TIMER can be from 1 to %d seconds. Default: %d.\n", UPDATE_EVERY_MAX, UPDATE_EVERY); - fprintf(stderr, " -p LISTEN_PORT can be from 1 to %d. Default: %d.\n", 65535, LISTEN_PORT); - fprintf(stderr, " -u USERNAME can be any system username to run as. Default: none.\n"); - fprintf(stderr, " -ch path to access host /proc and /sys when running in a container. Default: empty.\n"); - fprintf(stderr, " -nd or -nodeamon to disable forking in the background. Default: unset.\n"); - fprintf(stderr, " -df FLAGS debug options. Default: 0x%08llx.\n", debug_flags); - fprintf(stderr, " -stacksize BYTES to overwrite the pthread stack size.\n"); - fprintf(stderr, " -pidfile FILENAME to save a pid while running.\n"); - exit(1); - } - } - - if(!config_loaded) load_config(NULL, 0); - - // prepare configuration environment variables for the plugins - setenv("NETDATA_CONFIG_DIR" , config_get("global", "config directory" , CONFIG_DIR) , 1); - setenv("NETDATA_PLUGINS_DIR", config_get("global", "plugins directory" , PLUGINS_DIR), 1); - setenv("NETDATA_WEB_DIR" , config_get("global", "web files directory", WEB_DIR) , 1); - setenv("NETDATA_CACHE_DIR" , config_get("global", "cache directory" , CACHE_DIR) , 1); - setenv("NETDATA_LIB_DIR" , config_get("global", "lib directory" , VARLIB_DIR) , 1); - setenv("NETDATA_LOG_DIR" , config_get("global", "log directory" , LOG_DIR) , 1); - setenv("NETDATA_HOST_PREFIX", config_get("global", "host access prefix" , "") , 1); - setenv("HOME" , config_get("global", "home directory" , CACHE_DIR) , 1); - - // avoid extended to stat(/etc/localtime) - // http://stackoverflow.com/questions/4554271/how-to-avoid-excessive-stat-etc-localtime-calls-in-strftime-on-linux - setenv("TZ", ":/etc/localtime", 0); - - // cd to /tmp to avoid any plugins writing files at random places - if(chdir("/tmp")) error("netdata: ERROR: Cannot cd to /tmp"); - - char *input_log_file = NULL; - char *output_log_file = NULL; - char *error_log_file = NULL; - char *access_log_file = NULL; - char *user = NULL; - { - char buffer[1024]; - - // -------------------------------------------------------------------- - - sprintf(buffer, "0x%08llx", 0ULL); - char *flags = config_get("global", "debug flags", buffer); - setenv("NETDATA_DEBUG_FLAGS", flags, 1); - - debug_flags = strtoull(flags, NULL, 0); - debug(D_OPTIONS, "Debug flags set to '0x%8llx'.", debug_flags); - - if(debug_flags != 0) { - struct rlimit rl = { RLIM_INFINITY, RLIM_INFINITY }; - if(setrlimit(RLIMIT_CORE, &rl) != 0) - info("Cannot request unlimited core dumps for debugging... Proceeding anyway..."); - prctl(PR_SET_DUMPABLE, 1, 0, 0, 0); - } - - // -------------------------------------------------------------------- + int i, check_config = 0; + int config_loaded = 0; + int dont_fork = 0; + size_t wanted_stacksize = 0, stacksize = 0; + pthread_attr_t attr; + + // global initialization + get_HZ(); + + // set the name for logging + program_name = "netdata"; + + // parse command line. + + // parse depercated options + // TODO: Remove this block with the next major release. + { + i = 1; + while(i < argc) { + if(strcmp(argv[i], "-pidfile") == 0 && (i+1) < argc) { + strncpyz(pidfile, argv[i+1], FILENAME_MAX); + fprintf(stderr, "%s: deprecated option -- %s -- please use -P instead.\n", argv[0], argv[i]); + remove_option(i, &argc, argv); + } + else if(strcmp(argv[i], "-nodaemon") == 0 || strcmp(argv[i], "-nd") == 0) { + dont_fork = 1; + fprintf(stderr, "%s: deprecated option -- %s -- please use -D instead.\n ", argv[0], argv[i]); + remove_option(i, &argc, argv); + } + else if(strcmp(argv[i], "-ch") == 0 && (i+1) < argc) { + config_set("global", "host access prefix", argv[i+1]); + fprintf(stderr, "%s: deprecated option -- %s -- please use -s instead.\n", argv[0], argv[i]); + remove_option(i, &argc, argv); + } + else if(strcmp(argv[i], "-l") == 0 && (i+1) < argc) { + config_set("global", "history", argv[i+1]); + fprintf(stderr, "%s: deprecated option -- %s -- This option will be removed with V2.*.\n", argv[0], argv[i]); + remove_option(i, &argc, argv); + } + else i++; + } + } + + // parse options + { + int num_opts = sizeof(options) / sizeof(struct option_def); + char optstring[(num_opts * 2) + 1]; + + int string_i = 0; + for( i = 0; i < num_opts; i++ ) { + optstring[string_i] = options[i].val; + string_i++; + if(options[i].arg_name) { + optstring[string_i] = ':'; + string_i++; + } + } + + int opt; + while( (opt = getopt(argc, argv, optstring)) != -1 ) { + switch(opt) { + case 'c': + if(load_config(optarg, 1) != 1) { + error("Cannot load configuration file %s.", optarg); + exit(1); + } + else { + debug(D_OPTIONS, "Configuration loaded from %s.", optarg); + config_loaded = 1; + } + break; + case 'D': + dont_fork = 1; + break; + case 'h': + help(0); + break; + case 'i': + config_set("global", "bind to", optarg); + break; + case 'k': + dont_fork = 1; + check_config = 1; + break; + case 'P': + strncpy(pidfile, optarg, FILENAME_MAX); + pidfile[FILENAME_MAX] = '\0'; + break; + case 'p': + config_set("global", "default port", optarg); + break; + case 's': + config_set("global", "host access prefix", optarg); + break; + case 't': + config_set("global", "update every", optarg); + break; + case 'u': + config_set("global", "run as user", optarg); + break; + case 'v': + // TODO: Outsource version to makefile which can compute version from git. + printf("netdata 1.3.0\n"); + return 0; + case 'W': + { + char* stacksize_string = "stacksize="; + char* debug_flags_string = "debug_flags="; + if(strcmp(optarg, "unittest") == 0) { + rrd_update_every = 1; + if(run_all_mockup_tests()) exit(1); + if(unit_test_storage()) exit(1); + fprintf(stderr, "\n\nALL TESTS PASSED\n\n"); + exit(0); + } else if(strncmp(optarg, stacksize_string, strlen(stacksize_string)) == 0) { + optarg += strlen(stacksize_string); + config_set("global", "pthread stack size", optarg); + } else if(strncmp(optarg, debug_flags_string, strlen(debug_flags_string)) == 0) { + optarg += strlen(debug_flags_string); + config_set("global", "debug flags", optarg); + debug_flags = strtoull(optarg, NULL, 0); + } + } + break; + default: /* ? */ + help(1); + break; + } + } + } + + if(!config_loaded) + load_config(NULL, 0); + + { + char *config_dir = config_get("global", "config directory", CONFIG_DIR); + + // prepare configuration environment variables for the plugins + setenv("NETDATA_CONFIG_DIR" , verify_required_directory(config_dir) , 1); + setenv("NETDATA_PLUGINS_DIR", verify_required_directory(config_get("global", "plugins directory" , PLUGINS_DIR)), 1); + setenv("NETDATA_WEB_DIR" , verify_required_directory(config_get("global", "web files directory", WEB_DIR)) , 1); + setenv("NETDATA_CACHE_DIR" , verify_required_directory(config_get("global", "cache directory" , CACHE_DIR)) , 1); + setenv("NETDATA_LIB_DIR" , verify_required_directory(config_get("global", "lib directory" , VARLIB_DIR)) , 1); + setenv("NETDATA_LOG_DIR" , verify_required_directory(config_get("global", "log directory" , LOG_DIR)) , 1); + + setenv("NETDATA_HOST_PREFIX", config_get("global", "host access prefix" , "") , 1); + setenv("HOME" , config_get("global", "home directory" , CACHE_DIR) , 1); + + // disable buffering for python plugins + setenv("PYTHONUNBUFFERED", "1", 1); + + // avoid flood calls to stat(/etc/localtime) + // http://stackoverflow.com/questions/4554271/how-to-avoid-excessive-stat-etc-localtime-calls-in-strftime-on-linux + setenv("TZ", ":/etc/localtime", 0); + + // work while we are cd into config_dir + // to allow the plugins refer to their config + // files using relative filenames + if(chdir(config_dir) == -1) + fatal("Cannot cd to '%s'", config_dir); + + char path[1024 + 1], *p = getenv("PATH"); + if(!p) p = "/bin:/usr/bin"; + snprintfz(path, 1024, "%s:%s", p, "/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin"); + setenv("PATH", config_get("plugins", "PATH environment variable", path), 1); + } + + char *user = NULL; + { + char *flags = config_get("global", "debug flags", "0x00000000"); + setenv("NETDATA_DEBUG_FLAGS", flags, 1); + + debug_flags = strtoull(flags, NULL, 0); + debug(D_OPTIONS, "Debug flags set to '0x%8llx'.", debug_flags); + + if(debug_flags != 0) { + struct rlimit rl = { RLIM_INFINITY, RLIM_INFINITY }; + if(setrlimit(RLIMIT_CORE, &rl) != 0) + info("Cannot request unlimited core dumps for debugging... Proceeding anyway..."); + prctl(PR_SET_DUMPABLE, 1, 0, 0, 0); + } + + // -------------------------------------------------------------------- #ifdef MADV_MERGEABLE - enable_ksm = config_get_boolean("global", "memory deduplication (ksm)", enable_ksm); + enable_ksm = config_get_boolean("global", "memory deduplication (ksm)", enable_ksm); #else #warning "Kernel memory deduplication (KSM) is not available" #endif - // -------------------------------------------------------------------- - - - global_host_prefix = config_get("global", "host access prefix", ""); - setenv("NETDATA_HOST_PREFIX", global_host_prefix, 1); - - // -------------------------------------------------------------------- - - output_log_file = config_get("global", "debug log", LOG_DIR "/debug.log"); - if(strcmp(output_log_file, "syslog") == 0) { - output_log_syslog = 1; - output_log_file = NULL; - } - else if(strcmp(output_log_file, "none") == 0) { - output_log_syslog = 0; - output_log_file = NULL; - } - else output_log_syslog = 0; - - // -------------------------------------------------------------------- - - error_log_file = config_get("global", "error log", LOG_DIR "/error.log"); - if(strcmp(error_log_file, "syslog") == 0) { - error_log_syslog = 1; - error_log_file = NULL; - } - else if(strcmp(error_log_file, "none") == 0) { - error_log_syslog = 0; - error_log_file = NULL; - // optimization - do not even generate debug log entries - } - else error_log_syslog = 0; - - error_log_throttle_period = config_get_number("global", "errors flood protection period", error_log_throttle_period); - setenv("NETDATA_ERRORS_THROTTLE_PERIOD", config_get("global", "errors flood protection period" , ""), 1); - - error_log_errors_per_period = config_get_number("global", "errors to trigger flood protection", error_log_errors_per_period); - setenv("NETDATA_ERRORS_PER_PERIOD" , config_get("global", "errors to trigger flood protection", ""), 1); - - // -------------------------------------------------------------------- - - access_log_file = config_get("global", "access log", LOG_DIR "/access.log"); - if(strcmp(access_log_file, "syslog") == 0) { - access_log_syslog = 1; - access_log_file = NULL; - } - else if(strcmp(access_log_file, "none") == 0) { - access_log_syslog = 0; - access_log_file = NULL; - } - else access_log_syslog = 0; - - // -------------------------------------------------------------------- - - rrd_memory_mode = rrd_memory_mode_id(config_get("global", "memory mode", rrd_memory_mode_name(rrd_memory_mode))); - - // -------------------------------------------------------------------- - - if(gethostname(buffer, HOSTNAME_MAX) == -1) - error("WARNING: Cannot get machine hostname."); - hostname = config_get("global", "hostname", buffer); - debug(D_OPTIONS, "hostname set to '%s'", hostname); - - // -------------------------------------------------------------------- - - rrd_default_history_entries = (int) config_get_number("global", "history", RRD_DEFAULT_HISTORY_ENTRIES); - if(rrd_default_history_entries < 5 || rrd_default_history_entries > RRD_HISTORY_ENTRIES_MAX) { - info("Invalid save lines %d given. Defaulting to %d.", rrd_default_history_entries, RRD_DEFAULT_HISTORY_ENTRIES); - rrd_default_history_entries = RRD_DEFAULT_HISTORY_ENTRIES; - } - else { - debug(D_OPTIONS, "save lines set to %d.", rrd_default_history_entries); - } - - // -------------------------------------------------------------------- - - rrd_update_every = (int) config_get_number("global", "update every", UPDATE_EVERY); - if(rrd_update_every < 1 || rrd_update_every > 600) { - info("Invalid update timer %d given. Defaulting to %d.", rrd_update_every, UPDATE_EVERY_MAX); - rrd_update_every = UPDATE_EVERY; - } - else debug(D_OPTIONS, "update timer set to %d.", rrd_update_every); - - // let the plugins know the min update_every - { - char buf[51]; - snprintfz(buf, 50, "%d", rrd_update_every); - setenv("NETDATA_UPDATE_EVERY", buf, 1); - } - - // -------------------------------------------------------------------- - - // block signals while initializing threads. - // this causes the threads to block signals. - sigset_t sigset; - sigfillset(&sigset); - - if(pthread_sigmask(SIG_BLOCK, &sigset, NULL) == -1) { - error("Could not block signals for threads"); - } - - // Catch signals which we want to use to quit savely - struct sigaction sa; - sigemptyset(&sa.sa_mask); - sigaddset(&sa.sa_mask, SIGHUP); - sigaddset(&sa.sa_mask, SIGINT); - sigaddset(&sa.sa_mask, SIGTERM); - sa.sa_handler = sig_handler; - sa.sa_flags = 0; - if(sigaction(SIGHUP, &sa, NULL) == -1) { - error("Failed to change signal handler for SIGHUP"); - } - if(sigaction(SIGINT, &sa, NULL) == -1) { - error("Failed to change signal handler for SIGINT"); - } - if(sigaction(SIGTERM, &sa, NULL) == -1) { - error("Failed to change signal handler for SIGTERM"); - } - // Ignore SIGPIPE completely. - // INFO: If we add signals here we have to unblock them - // at popen.c when running a external plugin. - sa.sa_handler = SIG_IGN; - if(sigaction(SIGPIPE, &sa, NULL) == -1) { - error("Failed to change signal handler for SIGTERM"); - } - - // -------------------------------------------------------------------- - - i = pthread_attr_init(&attr); - if(i != 0) - fatal("pthread_attr_init() failed with code %d.", i); - - i = pthread_attr_getstacksize(&attr, &stacksize); - if(i != 0) - fatal("pthread_attr_getstacksize() failed with code %d.", i); - else - debug(D_OPTIONS, "initial pthread stack size is %zu bytes", stacksize); - - wanted_stacksize = config_get_number("global", "pthread stack size", stacksize); - - // -------------------------------------------------------------------- - - for (i = 0; static_threads[i].name != NULL ; i++) { - struct netdata_static_thread *st = &static_threads[i]; - - if(st->config_name) st->enabled = config_get_boolean(st->config_section, st->config_name, st->enabled); - if(st->enabled && st->init_routine) st->init_routine(); - } - - // -------------------------------------------------------------------- - - // get the user we should run - // IMPORTANT: this is required before web_files_uid() - user = config_get("global", "run as user" , (getuid() == 0)?NETDATA_USER:""); - - // IMPORTANT: these have to run once, while single threaded - web_files_uid(); // IMPORTANT: web_files_uid() before web_files_gid() - web_files_gid(); - - // -------------------------------------------------------------------- - - listen_backlog = (int) config_get_number("global", "http port listen backlog", LISTEN_BACKLOG); - - listen_port = (int) config_get_number("global", "port", LISTEN_PORT); - if(listen_port < 1 || listen_port > 65535) { - info("Invalid listen port %d given. Defaulting to %d.", listen_port, LISTEN_PORT); - listen_port = LISTEN_PORT; - } - else debug(D_OPTIONS, "Listen port set to %d.", listen_port); - - int ip = 0; - char *ipv = config_get("global", "ip version", "any"); - if(!strcmp(ipv, "any") || !strcmp(ipv, "both") || !strcmp(ipv, "all")) ip = 0; - else if(!strcmp(ipv, "ipv4") || !strcmp(ipv, "IPV4") || !strcmp(ipv, "IPv4") || !strcmp(ipv, "4")) ip = 4; - else if(!strcmp(ipv, "ipv6") || !strcmp(ipv, "IPV6") || !strcmp(ipv, "IPv6") || !strcmp(ipv, "6")) ip = 6; - else info("Cannot understand ip version '%s'. Assuming 'any'.", ipv); - - if(ip == 0 || ip == 6) listen_fd = create_listen_socket6(config_get("global", "bind socket to IP", "*"), listen_port, listen_backlog); - if(listen_fd < 0) { - listen_fd = create_listen_socket4(config_get("global", "bind socket to IP", "*"), listen_port, listen_backlog); - if(listen_fd >= 0 && ip != 4) info("Managed to open an IPv4 socket on port %d.", listen_port); - } - - if(listen_fd < 0) fatal("Cannot listen socket."); - } - - // never become a problem - if(nice(20) == -1) error("Cannot lower my CPU priority."); - - if(become_daemon(dont_fork, 0, user, input_log_file, output_log_file, error_log_file, access_log_file, &access_fd, &stdaccess) == -1) - fatal("Cannot demonize myself."); + // -------------------------------------------------------------------- + + global_host_prefix = config_get("global", "host access prefix", ""); + setenv("NETDATA_HOST_PREFIX", global_host_prefix, 1); + + // -------------------------------------------------------------------- + + stdout_filename = config_get("global", "debug log", LOG_DIR "/debug.log"); + stderr_filename = config_get("global", "error log", LOG_DIR "/error.log"); + stdaccess_filename = config_get("global", "access log", LOG_DIR "/access.log"); + + error_log_throttle_period = config_get_number("global", "errors flood protection period", error_log_throttle_period); + setenv("NETDATA_ERRORS_THROTTLE_PERIOD", config_get("global", "errors flood protection period" , ""), 1); + + error_log_errors_per_period = (unsigned long)config_get_number("global", "errors to trigger flood protection", error_log_errors_per_period); + setenv("NETDATA_ERRORS_PER_PERIOD" , config_get("global", "errors to trigger flood protection", ""), 1); + + if(check_config) { + stdout_filename = stderr_filename = stdaccess_filename = "system"; + error_log_throttle_period = 0; + error_log_errors_per_period = 0; + } + + // -------------------------------------------------------------------- + + rrd_memory_mode = rrd_memory_mode_id(config_get("global", "memory mode", rrd_memory_mode_name(rrd_memory_mode))); + + // -------------------------------------------------------------------- + + { + char hostnamebuf[HOSTNAME_MAX + 1]; + if(gethostname(hostnamebuf, HOSTNAME_MAX) == -1) + error("WARNING: Cannot get machine hostname."); + hostname = config_get("global", "hostname", hostnamebuf); + debug(D_OPTIONS, "hostname set to '%s'", hostname); + setenv("NETDATA_HOSTNAME", hostname, 1); + } + + // -------------------------------------------------------------------- + + rrd_default_history_entries = (int) config_get_number("global", "history", RRD_DEFAULT_HISTORY_ENTRIES); + if(rrd_default_history_entries < 5 || rrd_default_history_entries > RRD_HISTORY_ENTRIES_MAX) { + info("Invalid save lines %d given. Defaulting to %d.", rrd_default_history_entries, RRD_DEFAULT_HISTORY_ENTRIES); + rrd_default_history_entries = RRD_DEFAULT_HISTORY_ENTRIES; + } + else { + debug(D_OPTIONS, "save lines set to %d.", rrd_default_history_entries); + } + + // -------------------------------------------------------------------- + + rrd_update_every = (int) config_get_number("global", "update every", UPDATE_EVERY); + if(rrd_update_every < 1 || rrd_update_every > 600) { + info("Invalid update timer %d given. Defaulting to %d.", rrd_update_every, UPDATE_EVERY_MAX); + rrd_update_every = UPDATE_EVERY; + } + else debug(D_OPTIONS, "update timer set to %d.", rrd_update_every); + + // let the plugins know the min update_every + { + char buf[16]; + snprintfz(buf, 15, "%d", rrd_update_every); + setenv("NETDATA_UPDATE_EVERY", buf, 1); + } + + // -------------------------------------------------------------------- + + // block signals while initializing threads. + // this causes the threads to block signals. + sigset_t sigset; + sigfillset(&sigset); + if(pthread_sigmask(SIG_BLOCK, &sigset, NULL) == -1) + error("Could not block signals for threads"); + + // Catch signals which we want to use + struct sigaction sa; + sa.sa_flags = 0; + + // ingore all signals while we run in a signal handler + sigfillset(&sa.sa_mask); + + // INFO: If we add signals here we have to unblock them + // at popen.c when running a external plugin. + + // Ignore SIGPIPE completely. + sa.sa_handler = SIG_IGN; + if(sigaction(SIGPIPE, &sa, NULL) == -1) + error("Failed to change signal handler for SIGPIPE"); + + sa.sa_handler = sig_handler_exit; + if(sigaction(SIGINT, &sa, NULL) == -1) + error("Failed to change signal handler for SIGINT"); + + sa.sa_handler = sig_handler_exit; + if(sigaction(SIGTERM, &sa, NULL) == -1) + error("Failed to change signal handler for SIGTERM"); + + sa.sa_handler = sig_handler_logrotate; + if(sigaction(SIGHUP, &sa, NULL) == -1) + error("Failed to change signal handler for SIGHUP"); + + // save database on SIGUSR1 + sa.sa_handler = sig_handler_save; + if(sigaction(SIGUSR1, &sa, NULL) == -1) + error("Failed to change signal handler for SIGUSR1"); + + // reload health configuration on SIGUSR2 + sa.sa_handler = sig_handler_reload_health; + if(sigaction(SIGUSR2, &sa, NULL) == -1) + error("Failed to change signal handler for SIGUSR2"); + + // -------------------------------------------------------------------- + + i = pthread_attr_init(&attr); + if(i != 0) + fatal("pthread_attr_init() failed with code %d.", i); + + i = pthread_attr_getstacksize(&attr, &stacksize); + if(i != 0) + fatal("pthread_attr_getstacksize() failed with code %d.", i); + else + debug(D_OPTIONS, "initial pthread stack size is %zu bytes", stacksize); + + wanted_stacksize = (size_t)config_get_number("global", "pthread stack size", (long)stacksize); + + // -------------------------------------------------------------------- + + for (i = 0; static_threads[i].name != NULL ; i++) { + struct netdata_static_thread *st = &static_threads[i]; + + if(st->config_name) st->enabled = config_get_boolean(st->config_section, st->config_name, st->enabled); + if(st->enabled && st->init_routine) st->init_routine(); + } + + // -------------------------------------------------------------------- + + // get the user we should run + // IMPORTANT: this is required before web_files_uid() + user = config_get("global", "run as user" , (getuid() == 0)?NETDATA_USER:""); + + // IMPORTANT: these have to run once, while single threaded + web_files_uid(); // IMPORTANT: web_files_uid() before web_files_gid() + web_files_gid(); + + // -------------------------------------------------------------------- + + if(!check_config) + create_listen_sockets(); + } + + // initialize the log files + open_all_log_files(); #ifdef NETDATA_INTERNAL_CHECKS - if(debug_flags != 0) { - struct rlimit rl = { RLIM_INFINITY, RLIM_INFINITY }; - if(setrlimit(RLIMIT_CORE, &rl) != 0) - info("Cannot request unlimited core dumps for debugging... Proceeding anyway..."); - prctl(PR_SET_DUMPABLE, 1, 0, 0, 0); - } + if(debug_flags != 0) { + struct rlimit rl = { RLIM_INFINITY, RLIM_INFINITY }; + if(setrlimit(RLIMIT_CORE, &rl) != 0) + info("Cannot request unlimited core dumps for debugging... Proceeding anyway..."); + prctl(PR_SET_DUMPABLE, 1, 0, 0, 0); + } #endif /* NETDATA_INTERNAL_CHECKS */ - if(output_log_syslog || error_log_syslog || access_log_syslog) - openlog("netdata", LOG_PID, LOG_DAEMON); + // fork, switch user, create pid file, set process priority + if(become_daemon(dont_fork, user) == -1) + fatal("Cannot demonize myself."); + + info("NetData started on pid %d", getpid()); + + + // ------------------------------------------------------------------------ + // get default pthread stack size + + if(stacksize < wanted_stacksize) { + i = pthread_attr_setstacksize(&attr, wanted_stacksize); + if(i != 0) + fatal("pthread_attr_setstacksize() to %zu bytes, failed with code %d.", wanted_stacksize, i); + else + info("Successfully set pthread stacksize to %zu bytes", wanted_stacksize); + } - info("NetData started on pid %d", getpid()); + // ------------------------------------------------------------------------ + // initialize the registry + registry_init(); - // ------------------------------------------------------------------------ - // get default pthread stack size + // ------------------------------------------------------------------------ + // initialize health monitoring - if(stacksize < wanted_stacksize) { - i = pthread_attr_setstacksize(&attr, wanted_stacksize); - if(i != 0) - fatal("pthread_attr_setstacksize() to %zu bytes, failed with code %d.", wanted_stacksize, i); - else - info("Successfully set pthread stacksize to %zu bytes", wanted_stacksize); - } + health_init(); - // -------------------------------------------------------------------- - // initialize the registry + if(check_config) + exit(1); - registry_init(); + // ------------------------------------------------------------------------ + // spawn the threads - // ------------------------------------------------------------------------ - // spawn the threads + web_server_threading_selection(); - for (i = 0; static_threads[i].name != NULL ; i++) { - struct netdata_static_thread *st = &static_threads[i]; + for (i = 0; static_threads[i].name != NULL ; i++) { + struct netdata_static_thread *st = &static_threads[i]; - if(st->enabled) { - st->thread = malloc(sizeof(pthread_t)); - if(!st->thread) - fatal("Cannot allocate pthread_t memory"); + if(st->enabled) { + st->thread = mallocz(sizeof(pthread_t)); - info("Starting thread %s.", st->name); + info("Starting thread %s.", st->name); - if(pthread_create(st->thread, &attr, st->start_routine, NULL)) - error("failed to create new thread for %s.", st->name); + if(pthread_create(st->thread, &attr, st->start_routine, NULL)) + error("failed to create new thread for %s.", st->name); - else if(pthread_detach(*st->thread)) - error("Cannot request detach of newly created %s thread.", st->name); - } - else info("Not starting thread %s.", st->name); - } + else if(pthread_detach(*st->thread)) + error("Cannot request detach of newly created %s thread.", st->name); + } + else info("Not starting thread %s.", st->name); + } - // ------------------------------------------------------------------------ - // block signals while initializing threads. - sigset_t sigset; - sigfillset(&sigset); + // ------------------------------------------------------------------------ + // block signals while initializing threads. + sigset_t sigset; + sigfillset(&sigset); - if(pthread_sigmask(SIG_UNBLOCK, &sigset, NULL) == -1) { - error("Could not unblock signals for threads"); - } + if(pthread_sigmask(SIG_UNBLOCK, &sigset, NULL) == -1) { + error("Could not unblock signals for threads"); + } - // Handle flags set in the signal handler. - while(1) { - pause(); - if(netdata_exit) { - info("Exit main loop of netdata."); - netdata_cleanup_and_exit(0); - exit(0); - } - } + // Handle flags set in the signal handler. + while(1) { + pause(); + if(netdata_exit) { + info("Exit main loop of netdata."); + netdata_cleanup_and_exit(0); + exit(0); + } + } } diff --git a/src/main.h b/src/main.h index d9edda58e..646827fbd 100644 --- a/src/main.h +++ b/src/main.h @@ -1,12 +1,31 @@ #ifndef NETDATA_MAIN_H #define NETDATA_MAIN_H 1 -#include <signal.h> - extern volatile sig_atomic_t netdata_exit; +/** + * This struct contains information about command line options. + */ +struct option_def { + /** The option character */ + const char val; + /** The name of the long option. */ + const char *description; + /** Short descripton what the option does */ + /** Name of the argument displayed in SYNOPSIS */ + const char *arg_name; + /** Default value if not set */ + const char *default_value; +}; + +/** + * List of command line options. + * This can be used to compute manpage, help messages, ect. + */ +extern struct option_def options[]; + extern void kill_childs(void); extern int killpid(pid_t pid, int signal); -extern void netdata_cleanup_and_exit(int ret); +extern void netdata_cleanup_and_exit(int ret) __attribute__ ((noreturn)); #endif /* NETDATA_MAIN_H */ diff --git a/src/plugin_checks.c b/src/plugin_checks.c index 379fb9a84..007d6565f 100644 --- a/src/plugin_checks.c +++ b/src/plugin_checks.c @@ -1,97 +1,84 @@ -#ifdef HAVE_CONFIG_H -#include <config.h> -#endif -#include <pthread.h> -#include <sys/time.h> -#include <sys/resource.h> -#include <strings.h> -#include <unistd.h> - #include "common.h" -#include "appconfig.h" -#include "log.h" -#include "rrd.h" -#include "plugin_checks.h" void *checks_main(void *ptr) { - if(ptr) { ; } + if(ptr) { ; } - info("CHECKS thread created with task id %d", gettid()); + info("CHECKS thread created with task id %d", gettid()); - if(pthread_setcanceltype(PTHREAD_CANCEL_DEFERRED, NULL) != 0) - error("Cannot set pthread cancel type to DEFERRED."); + if(pthread_setcanceltype(PTHREAD_CANCEL_DEFERRED, NULL) != 0) + error("Cannot set pthread cancel type to DEFERRED."); - if(pthread_setcancelstate(PTHREAD_CANCEL_ENABLE, NULL) != 0) - error("Cannot set pthread cancel state to ENABLE."); + if(pthread_setcancelstate(PTHREAD_CANCEL_ENABLE, NULL) != 0) + error("Cannot set pthread cancel state to ENABLE."); - unsigned long long usec = 0, susec = rrd_update_every * 1000000ULL, loop_usec = 0, total_susec = 0; - struct timeval now, last, loop; + unsigned long long usec = 0, susec = rrd_update_every * 1000000ULL, loop_usec = 0, total_susec = 0; + struct timeval now, last, loop; - RRDSET *check1, *check2, *check3, *apps_cpu = NULL; + RRDSET *check1, *check2, *check3, *apps_cpu = NULL; - check1 = rrdset_create("netdata", "check1", NULL, "netdata", NULL, "Caller gives microseconds", "a million !", 99999, rrd_update_every, RRDSET_TYPE_LINE); - rrddim_add(check1, "absolute", NULL, -1, 1, RRDDIM_ABSOLUTE); - rrddim_add(check1, "incremental", NULL, 1, 1, RRDDIM_INCREMENTAL); + check1 = rrdset_create("netdata", "check1", NULL, "netdata", NULL, "Caller gives microseconds", "a million !", 99999, rrd_update_every, RRDSET_TYPE_LINE); + rrddim_add(check1, "absolute", NULL, -1, 1, RRDDIM_ABSOLUTE); + rrddim_add(check1, "incremental", NULL, 1, 1, RRDDIM_INCREMENTAL); - check2 = rrdset_create("netdata", "check2", NULL, "netdata", NULL, "Netdata calcs microseconds", "a million !", 99999, rrd_update_every, RRDSET_TYPE_LINE); - rrddim_add(check2, "absolute", NULL, -1, 1, RRDDIM_ABSOLUTE); - rrddim_add(check2, "incremental", NULL, 1, 1, RRDDIM_INCREMENTAL); + check2 = rrdset_create("netdata", "check2", NULL, "netdata", NULL, "Netdata calcs microseconds", "a million !", 99999, rrd_update_every, RRDSET_TYPE_LINE); + rrddim_add(check2, "absolute", NULL, -1, 1, RRDDIM_ABSOLUTE); + rrddim_add(check2, "incremental", NULL, 1, 1, RRDDIM_INCREMENTAL); - check3 = rrdset_create("netdata", "checkdt", NULL, "netdata", NULL, "Clock difference", "microseconds diff", 99999, rrd_update_every, RRDSET_TYPE_LINE); - rrddim_add(check3, "caller", NULL, 1, 1, RRDDIM_ABSOLUTE); - rrddim_add(check3, "netdata", NULL, 1, 1, RRDDIM_ABSOLUTE); - rrddim_add(check3, "apps.plugin", NULL, 1, 1, RRDDIM_ABSOLUTE); + check3 = rrdset_create("netdata", "checkdt", NULL, "netdata", NULL, "Clock difference", "microseconds diff", 99999, rrd_update_every, RRDSET_TYPE_LINE); + rrddim_add(check3, "caller", NULL, 1, 1, RRDDIM_ABSOLUTE); + rrddim_add(check3, "netdata", NULL, 1, 1, RRDDIM_ABSOLUTE); + rrddim_add(check3, "apps.plugin", NULL, 1, 1, RRDDIM_ABSOLUTE); - gettimeofday(&last, NULL); - while(1) { - usleep(susec); + gettimeofday(&last, NULL); + while(1) { + usleep(susec); - // find the time to sleep in order to wait exactly update_every seconds - gettimeofday(&now, NULL); - loop_usec = usecdiff(&now, &last); - usec = loop_usec - susec; - debug(D_PROCNETDEV_LOOP, "CHECK: last loop took %llu usec (worked for %llu, sleeped for %llu).", loop_usec, usec, susec); + // find the time to sleep in order to wait exactly update_every seconds + gettimeofday(&now, NULL); + loop_usec = usec_dt(&now, &last); + usec = loop_usec - susec; + debug(D_PROCNETDEV_LOOP, "CHECK: last loop took %llu usec (worked for %llu, sleeped for %llu).", loop_usec, usec, susec); - if(usec < (rrd_update_every * 1000000ULL / 2ULL)) susec = (rrd_update_every * 1000000ULL) - usec; - else susec = rrd_update_every * 1000000ULL / 2ULL; + if(usec < (rrd_update_every * 1000000ULL / 2ULL)) susec = (rrd_update_every * 1000000ULL) - usec; + else susec = rrd_update_every * 1000000ULL / 2ULL; - // -------------------------------------------------------------------- - // Calculate loop time + // -------------------------------------------------------------------- + // Calculate loop time - last.tv_sec = now.tv_sec; - last.tv_usec = now.tv_usec; - total_susec += loop_usec; + last.tv_sec = now.tv_sec; + last.tv_usec = now.tv_usec; + total_susec += loop_usec; - // -------------------------------------------------------------------- - // check chart 1 + // -------------------------------------------------------------------- + // check chart 1 - if(check1->counter_done) rrdset_next_usec(check1, loop_usec); - rrddim_set(check1, "absolute", 1000000); - rrddim_set(check1, "incremental", total_susec); - rrdset_done(check1); + if(check1->counter_done) rrdset_next_usec(check1, loop_usec); + rrddim_set(check1, "absolute", 1000000); + rrddim_set(check1, "incremental", total_susec); + rrdset_done(check1); - // -------------------------------------------------------------------- - // check chart 2 + // -------------------------------------------------------------------- + // check chart 2 - if(check2->counter_done) rrdset_next(check2); - rrddim_set(check2, "absolute", 1000000); - rrddim_set(check2, "incremental", total_susec); - rrdset_done(check2); + if(check2->counter_done) rrdset_next(check2); + rrddim_set(check2, "absolute", 1000000); + rrddim_set(check2, "incremental", total_susec); + rrdset_done(check2); - // -------------------------------------------------------------------- - // check chart 3 + // -------------------------------------------------------------------- + // check chart 3 - if(!apps_cpu) apps_cpu = rrdset_find("apps.cpu"); - if(check3->counter_done) rrdset_next_usec(check3, loop_usec); - gettimeofday(&loop, NULL); - rrddim_set(check3, "caller", (long long)usecdiff(&loop, &check1->last_collected_time)); - rrddim_set(check3, "netdata", (long long)usecdiff(&loop, &check2->last_collected_time)); - if(apps_cpu) rrddim_set(check3, "apps.plugin", (long long)usecdiff(&loop, &apps_cpu->last_collected_time)); - rrdset_done(check3); - } + if(!apps_cpu) apps_cpu = rrdset_find("apps.cpu"); + if(check3->counter_done) rrdset_next_usec(check3, loop_usec); + gettimeofday(&loop, NULL); + rrddim_set(check3, "caller", (long long) usec_dt(&loop, &check1->last_collected_time)); + rrddim_set(check3, "netdata", (long long) usec_dt(&loop, &check2->last_collected_time)); + if(apps_cpu) rrddim_set(check3, "apps.plugin", (long long) usec_dt(&loop, &apps_cpu->last_collected_time)); + rrdset_done(check3); + } - pthread_exit(NULL); - return NULL; + pthread_exit(NULL); + return NULL; } diff --git a/src/plugin_idlejitter.c b/src/plugin_idlejitter.c index 56c22a160..30c6d8708 100644 --- a/src/plugin_idlejitter.c +++ b/src/plugin_idlejitter.c @@ -1,68 +1,54 @@ -#ifdef HAVE_CONFIG_H -#include <config.h> -#endif -#include <pthread.h> -#include <sys/time.h> -#include <sys/resource.h> -#include <strings.h> -#include <unistd.h> - -#include "global_statistics.h" #include "common.h" -#include "appconfig.h" -#include "log.h" -#include "rrd.h" -#include "plugin_idlejitter.h" #define CPU_IDLEJITTER_SLEEP_TIME_MS 20 void *cpuidlejitter_main(void *ptr) { - if(ptr) { ; } + if(ptr) { ; } - info("CPU Idle Jitter thread created with task id %d", gettid()); + info("CPU Idle Jitter thread created with task id %d", gettid()); - if(pthread_setcanceltype(PTHREAD_CANCEL_DEFERRED, NULL) != 0) - error("Cannot set pthread cancel type to DEFERRED."); + if(pthread_setcanceltype(PTHREAD_CANCEL_DEFERRED, NULL) != 0) + error("Cannot set pthread cancel type to DEFERRED."); - if(pthread_setcancelstate(PTHREAD_CANCEL_ENABLE, NULL) != 0) - error("Cannot set pthread cancel state to ENABLE."); + if(pthread_setcancelstate(PTHREAD_CANCEL_ENABLE, NULL) != 0) + error("Cannot set pthread cancel state to ENABLE."); - int sleep_ms = (int) config_get_number("plugin:idlejitter", "loop time in ms", CPU_IDLEJITTER_SLEEP_TIME_MS); - if(sleep_ms <= 0) { - config_set_number("plugin:idlejitter", "loop time in ms", CPU_IDLEJITTER_SLEEP_TIME_MS); - sleep_ms = CPU_IDLEJITTER_SLEEP_TIME_MS; - } + int sleep_ms = (int) config_get_number("plugin:idlejitter", "loop time in ms", CPU_IDLEJITTER_SLEEP_TIME_MS); + if(sleep_ms <= 0) { + config_set_number("plugin:idlejitter", "loop time in ms", CPU_IDLEJITTER_SLEEP_TIME_MS); + sleep_ms = CPU_IDLEJITTER_SLEEP_TIME_MS; + } - RRDSET *st = rrdset_find("system.idlejitter"); - if(!st) { - st = rrdset_create("system", "idlejitter", NULL, "processes", NULL, "CPU Idle Jitter", "microseconds lost/s", 9999, rrd_update_every, RRDSET_TYPE_LINE); - rrddim_add(st, "jitter", NULL, 1, 1, RRDDIM_ABSOLUTE); - } + RRDSET *st = rrdset_find("system.idlejitter"); + if(!st) { + st = rrdset_create("system", "idlejitter", NULL, "processes", NULL, "CPU Idle Jitter", "microseconds lost/s", 9999, rrd_update_every, RRDSET_TYPE_LINE); + rrddim_add(st, "jitter", NULL, 1, 1, RRDDIM_ABSOLUTE); + } - struct timeval before, after; - unsigned long long counter; - for(counter = 0; 1 ;counter++) { - unsigned long long usec = 0, susec = 0; + struct timeval before, after; + unsigned long long counter; + for(counter = 0; 1 ;counter++) { + unsigned long long usec = 0, susec = 0; - while(susec < (rrd_update_every * 1000000ULL)) { + while(susec < (rrd_update_every * 1000000ULL)) { - gettimeofday(&before, NULL); - usleep(sleep_ms * 1000); - gettimeofday(&after, NULL); + gettimeofday(&before, NULL); + usleep(sleep_ms * 1000); + gettimeofday(&after, NULL); - // calculate the time it took for a full loop - usec = usecdiff(&after, &before); - susec += usec; - } - usec -= (sleep_ms * 1000); + // calculate the time it took for a full loop + usec = usec_dt(&after, &before); + susec += usec; + } + usec -= (sleep_ms * 1000); - if(counter) rrdset_next(st); - rrddim_set(st, "jitter", usec); - rrdset_done(st); - } + if(counter) rrdset_next(st); + rrddim_set(st, "jitter", usec); + rrdset_done(st); + } - pthread_exit(NULL); - return NULL; + pthread_exit(NULL); + return NULL; } diff --git a/src/plugin_nfacct.c b/src/plugin_nfacct.c index 6cde66e0c..b2396fac1 100644 --- a/src/plugin_nfacct.c +++ b/src/plugin_nfacct.c @@ -1,222 +1,202 @@ -#ifdef HAVE_CONFIG_H -#include <config.h> -#endif -#ifdef INTERNAL_PLUGIN_NFACCT -#include <stdio.h> -#include <stdlib.h> -#include <string.h> -#include <sys/types.h> -#include <dirent.h> -#include <sys/stat.h> -#include <fcntl.h> -#include <unistd.h> -#include <time.h> -#include <errno.h> +#include "common.h" +#ifdef INTERNAL_PLUGIN_NFACCT #include <libmnl/libmnl.h> #include <libnetfilter_acct/libnetfilter_acct.h> -#include "main.h" -#include "global_statistics.h" -#include "common.h" -#include "appconfig.h" -#include "log.h" -#include "rrd.h" -#include "plugin_proc.h" - struct mynfacct { - const char *name; - uint64_t pkts; - uint64_t bytes; - struct nfacct *nfacct; + const char *name; + uint64_t pkts; + uint64_t bytes; + struct nfacct *nfacct; }; struct nfacct_list { - int size; - int len; - struct mynfacct data[]; + int size; + int len; + struct mynfacct data[]; } *nfacct_list = NULL; static int nfacct_callback(const struct nlmsghdr *nlh, void *data) { - if(data) {}; - - if(!nfacct_list || nfacct_list->len == nfacct_list->size) { - int size = (nfacct_list) ? nfacct_list->size : 0; - int len = (nfacct_list) ? nfacct_list->len : 0; - size++; - - info("nfacct.plugin: increasing nfacct_list to size %d", size); - - nfacct_list = realloc(nfacct_list, sizeof(struct nfacct_list) + (sizeof(struct mynfacct) * size)); - if(!nfacct_list) { - error("nfacct.plugin: cannot allocate nfacct_list."); - return MNL_CB_OK; - } - - nfacct_list->data[len].nfacct = nfacct_alloc(); - if(!nfacct_list->data[size - 1].nfacct) { - error("nfacct.plugin: nfacct_alloc() failed."); - free(nfacct_list); - nfacct_list = NULL; - return MNL_CB_OK; - } - - nfacct_list->size = size; - nfacct_list->len = len; - } - - if(nfacct_nlmsg_parse_payload(nlh, nfacct_list->data[nfacct_list->len].nfacct) < 0) { - error("nfacct.plugin: nfacct_nlmsg_parse_payload() failed."); - return MNL_CB_OK; - } - - nfacct_list->data[nfacct_list->len].name = nfacct_attr_get_str(nfacct_list->data[nfacct_list->len].nfacct, NFACCT_ATTR_NAME); - nfacct_list->data[nfacct_list->len].pkts = nfacct_attr_get_u64(nfacct_list->data[nfacct_list->len].nfacct, NFACCT_ATTR_PKTS); - nfacct_list->data[nfacct_list->len].bytes = nfacct_attr_get_u64(nfacct_list->data[nfacct_list->len].nfacct, NFACCT_ATTR_BYTES); - - nfacct_list->len++; - return MNL_CB_OK; + if(data) {}; + + if(!nfacct_list || nfacct_list->len == nfacct_list->size) { + int size = (nfacct_list) ? nfacct_list->size : 0; + int len = (nfacct_list) ? nfacct_list->len : 0; + size++; + + info("nfacct.plugin: increasing nfacct_list to size %d", size); + + nfacct_list = realloc(nfacct_list, sizeof(struct nfacct_list) + (sizeof(struct mynfacct) * size)); + if(!nfacct_list) { + error("nfacct.plugin: cannot allocate nfacct_list."); + return MNL_CB_OK; + } + + nfacct_list->data[len].nfacct = nfacct_alloc(); + if(!nfacct_list->data[size - 1].nfacct) { + error("nfacct.plugin: nfacct_alloc() failed."); + free(nfacct_list); + nfacct_list = NULL; + return MNL_CB_OK; + } + + nfacct_list->size = size; + nfacct_list->len = len; + } + + if(nfacct_nlmsg_parse_payload(nlh, nfacct_list->data[nfacct_list->len].nfacct) < 0) { + error("nfacct.plugin: nfacct_nlmsg_parse_payload() failed."); + return MNL_CB_OK; + } + + nfacct_list->data[nfacct_list->len].name = nfacct_attr_get_str(nfacct_list->data[nfacct_list->len].nfacct, NFACCT_ATTR_NAME); + nfacct_list->data[nfacct_list->len].pkts = nfacct_attr_get_u64(nfacct_list->data[nfacct_list->len].nfacct, NFACCT_ATTR_PKTS); + nfacct_list->data[nfacct_list->len].bytes = nfacct_attr_get_u64(nfacct_list->data[nfacct_list->len].nfacct, NFACCT_ATTR_BYTES); + + nfacct_list->len++; + return MNL_CB_OK; } void *nfacct_main(void *ptr) { - if(ptr) { ; } + if(ptr) { ; } - info("NFACCT thread created with task id %d", gettid()); + info("NFACCT thread created with task id %d", gettid()); - if(pthread_setcanceltype(PTHREAD_CANCEL_DEFERRED, NULL) != 0) - error("nfacct.plugin: Cannot set pthread cancel type to DEFERRED."); + if(pthread_setcanceltype(PTHREAD_CANCEL_DEFERRED, NULL) != 0) + error("nfacct.plugin: Cannot set pthread cancel type to DEFERRED."); - if(pthread_setcancelstate(PTHREAD_CANCEL_ENABLE, NULL) != 0) - error("nfacct.plugin: Cannot set pthread cancel state to ENABLE."); + if(pthread_setcancelstate(PTHREAD_CANCEL_ENABLE, NULL) != 0) + error("nfacct.plugin: Cannot set pthread cancel state to ENABLE."); - char buf[MNL_SOCKET_BUFFER_SIZE]; - struct mnl_socket *nl = NULL; - struct nlmsghdr *nlh = NULL; - unsigned int seq = 0, portid = 0; + char buf[MNL_SOCKET_BUFFER_SIZE]; + struct mnl_socket *nl = NULL; + struct nlmsghdr *nlh = NULL; + unsigned int seq = 0, portid = 0; - seq = time(NULL) - 1; + seq = time(NULL) - 1; - nl = mnl_socket_open(NETLINK_NETFILTER); - if(!nl) { - error("nfacct.plugin: mnl_socket_open() failed"); - pthread_exit(NULL); - return NULL; - } + nl = mnl_socket_open(NETLINK_NETFILTER); + if(!nl) { + error("nfacct.plugin: mnl_socket_open() failed"); + pthread_exit(NULL); + return NULL; + } - if(mnl_socket_bind(nl, 0, MNL_SOCKET_AUTOPID) < 0) { - mnl_socket_close(nl); - error("nfacct.plugin: mnl_socket_bind() failed"); - pthread_exit(NULL); - return NULL; - } - portid = mnl_socket_get_portid(nl); + if(mnl_socket_bind(nl, 0, MNL_SOCKET_AUTOPID) < 0) { + mnl_socket_close(nl); + error("nfacct.plugin: mnl_socket_bind() failed"); + pthread_exit(NULL); + return NULL; + } + portid = mnl_socket_get_portid(nl); - // ------------------------------------------------------------------------ + // ------------------------------------------------------------------------ - struct timeval last, now; - unsigned long long usec = 0, susec = 0; - RRDSET *st = NULL; + struct timeval last, now; + unsigned long long usec = 0, susec = 0; + RRDSET *st = NULL; - gettimeofday(&last, NULL); + gettimeofday(&last, NULL); - // ------------------------------------------------------------------------ + // ------------------------------------------------------------------------ - while(1) { - if(unlikely(netdata_exit)) break; + while(1) { + if(unlikely(netdata_exit)) break; - seq++; + seq++; - nlh = nfacct_nlmsg_build_hdr(buf, NFNL_MSG_ACCT_GET, NLM_F_DUMP, seq); - if(!nlh) { - mnl_socket_close(nl); - error("nfacct.plugin: nfacct_nlmsg_build_hdr() failed"); - pthread_exit(NULL); - return NULL; - } + nlh = nfacct_nlmsg_build_hdr(buf, NFNL_MSG_ACCT_GET, NLM_F_DUMP, seq); + if(!nlh) { + mnl_socket_close(nl); + error("nfacct.plugin: nfacct_nlmsg_build_hdr() failed"); + pthread_exit(NULL); + return NULL; + } - if(mnl_socket_sendto(nl, nlh, nlh->nlmsg_len) < 0) { - error("nfacct.plugin: mnl_socket_send"); - pthread_exit(NULL); - return NULL; - } + if(mnl_socket_sendto(nl, nlh, nlh->nlmsg_len) < 0) { + error("nfacct.plugin: mnl_socket_send"); + pthread_exit(NULL); + return NULL; + } - if(nfacct_list) nfacct_list->len = 0; + if(nfacct_list) nfacct_list->len = 0; - int ret; - while((ret = mnl_socket_recvfrom(nl, buf, sizeof(buf))) > 0) { - if((ret = mnl_cb_run(buf, ret, seq, portid, nfacct_callback, NULL)) <= 0) break; - } + int ret; + while((ret = mnl_socket_recvfrom(nl, buf, sizeof(buf))) > 0) { + if((ret = mnl_cb_run(buf, ret, seq, portid, nfacct_callback, NULL)) <= 0) break; + } - if (ret == -1) { - error("nfacct.plugin: error communicating with kernel."); - pthread_exit(NULL); - return NULL; - } + if (ret == -1) { + error("nfacct.plugin: error communicating with kernel."); + pthread_exit(NULL); + return NULL; + } - // -------------------------------------------------------------------- + // -------------------------------------------------------------------- - gettimeofday(&now, NULL); - usec = usecdiff(&now, &last) - susec; - debug(D_NFACCT_LOOP, "nfacct.plugin: last loop took %llu usec (worked for %llu, sleeped for %llu).", usec + susec, usec, susec); + gettimeofday(&now, NULL); + usec = usec_dt(&now, &last) - susec; + debug(D_NFACCT_LOOP, "nfacct.plugin: last loop took %llu usec (worked for %llu, sleeped for %llu).", usec + susec, usec, susec); - if(usec < (rrd_update_every * 1000000ULL / 2ULL)) susec = (rrd_update_every * 1000000ULL) - usec; - else susec = rrd_update_every * 1000000ULL / 2ULL; + if(usec < (rrd_update_every * 1000000ULL / 2ULL)) susec = (rrd_update_every * 1000000ULL) - usec; + else susec = rrd_update_every * 1000000ULL / 2ULL; - // -------------------------------------------------------------------- + // -------------------------------------------------------------------- - if(nfacct_list && nfacct_list->len) { - int i; + if(nfacct_list && nfacct_list->len) { + int i; - st = rrdset_find_bytype("netfilter", "nfacct_packets"); - if(!st) { - st = rrdset_create("netfilter", "nfacct_packets", NULL, "nfacct", NULL, "Netfilter Accounting Packets", "packets/s", 1006, rrd_update_every, RRDSET_TYPE_STACKED); + st = rrdset_find_bytype("netfilter", "nfacct_packets"); + if(!st) { + st = rrdset_create("netfilter", "nfacct_packets", NULL, "nfacct", NULL, "Netfilter Accounting Packets", "packets/s", 1006, rrd_update_every, RRDSET_TYPE_STACKED); - for(i = 0; i < nfacct_list->len ; i++) - rrddim_add(st, nfacct_list->data[i].name, NULL, 1, rrd_update_every, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); + for(i = 0; i < nfacct_list->len ; i++) + rrddim_add(st, nfacct_list->data[i].name, NULL, 1, rrd_update_every, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); - for(i = 0; i < nfacct_list->len ; i++) { - RRDDIM *rd = rrddim_find(st, nfacct_list->data[i].name); + for(i = 0; i < nfacct_list->len ; i++) { + RRDDIM *rd = rrddim_find(st, nfacct_list->data[i].name); - if(!rd) rd = rrddim_add(st, nfacct_list->data[i].name, NULL, 1, rrd_update_every, RRDDIM_INCREMENTAL); - if(rd) rrddim_set_by_pointer(st, rd, nfacct_list->data[i].pkts); - } + if(!rd) rd = rrddim_add(st, nfacct_list->data[i].name, NULL, 1, rrd_update_every, RRDDIM_INCREMENTAL); + if(rd) rrddim_set_by_pointer(st, rd, nfacct_list->data[i].pkts); + } - rrdset_done(st); + rrdset_done(st); - // ---------------------------------------------------------------- + // ---------------------------------------------------------------- - st = rrdset_find_bytype("netfilter", "nfacct_bytes"); - if(!st) { - st = rrdset_create("netfilter", "nfacct_bytes", NULL, "nfacct", NULL, "Netfilter Accounting Bandwidth", "kilobytes/s", 1007, rrd_update_every, RRDSET_TYPE_STACKED); + st = rrdset_find_bytype("netfilter", "nfacct_bytes"); + if(!st) { + st = rrdset_create("netfilter", "nfacct_bytes", NULL, "nfacct", NULL, "Netfilter Accounting Bandwidth", "kilobytes/s", 1007, rrd_update_every, RRDSET_TYPE_STACKED); - for(i = 0; i < nfacct_list->len ; i++) - rrddim_add(st, nfacct_list->data[i].name, NULL, 1, 1000 * rrd_update_every, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); + for(i = 0; i < nfacct_list->len ; i++) + rrddim_add(st, nfacct_list->data[i].name, NULL, 1, 1000 * rrd_update_every, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); - for(i = 0; i < nfacct_list->len ; i++) { - RRDDIM *rd = rrddim_find(st, nfacct_list->data[i].name); + for(i = 0; i < nfacct_list->len ; i++) { + RRDDIM *rd = rrddim_find(st, nfacct_list->data[i].name); - if(!rd) rd = rrddim_add(st, nfacct_list->data[i].name, NULL, 1, 1000 * rrd_update_every, RRDDIM_INCREMENTAL); - if(rd) rrddim_set_by_pointer(st, rd, nfacct_list->data[i].bytes); - } + if(!rd) rd = rrddim_add(st, nfacct_list->data[i].name, NULL, 1, 1000 * rrd_update_every, RRDDIM_INCREMENTAL); + if(rd) rrddim_set_by_pointer(st, rd, nfacct_list->data[i].bytes); + } - rrdset_done(st); - } + rrdset_done(st); + } - // -------------------------------------------------------------------- + // -------------------------------------------------------------------- - usleep(susec); + usleep(susec); - // copy current to last - bcopy(&now, &last, sizeof(struct timeval)); - } + // copy current to last + bcopy(&now, &last, sizeof(struct timeval)); + } - mnl_socket_close(nl); - pthread_exit(NULL); - return NULL; + mnl_socket_close(nl); + pthread_exit(NULL); + return NULL; } #endif diff --git a/src/plugin_proc.c b/src/plugin_proc.c index a147d971f..a1bf314de 100644 --- a/src/plugin_proc.c +++ b/src/plugin_proc.c @@ -1,319 +1,226 @@ -#ifdef HAVE_CONFIG_H -#include <config.h> -#endif -#include <pthread.h> -#include <sys/time.h> -#include <sys/resource.h> -#include <strings.h> -#include <unistd.h> - -#include "global_statistics.h" #include "common.h" -#include "appconfig.h" -#include "log.h" -#include "rrd.h" -#include "plugin_proc.h" -#include "main.h" -#include "registry.h" void *proc_main(void *ptr) { - if(ptr) { ; } - - info("PROC Plugin thread created with task id %d", gettid()); - - if(pthread_setcanceltype(PTHREAD_CANCEL_DEFERRED, NULL) != 0) - error("Cannot set pthread cancel type to DEFERRED."); - - if(pthread_setcancelstate(PTHREAD_CANCEL_ENABLE, NULL) != 0) - error("Cannot set pthread cancel state to ENABLE."); - - struct rusage me, thread; - - // disable (by default) various interface that are not needed - config_get_boolean("plugin:proc:/proc/net/dev:lo", "enabled", 0); - config_get_boolean("plugin:proc:/proc/net/dev:fireqos_monitor", "enabled", 0); - - // when ZERO, attempt to do it - int vdo_proc_net_dev = !config_get_boolean("plugin:proc", "/proc/net/dev", 1); - int vdo_proc_diskstats = !config_get_boolean("plugin:proc", "/proc/diskstats", 1); - int vdo_proc_net_snmp = !config_get_boolean("plugin:proc", "/proc/net/snmp", 1); - int vdo_proc_net_snmp6 = !config_get_boolean("plugin:proc", "/proc/net/snmp6", 1); - int vdo_proc_net_netstat = !config_get_boolean("plugin:proc", "/proc/net/netstat", 1); - int vdo_proc_net_stat_conntrack = !config_get_boolean("plugin:proc", "/proc/net/stat/conntrack", 1); - int vdo_proc_net_ip_vs_stats = !config_get_boolean("plugin:proc", "/proc/net/ip_vs/stats", 1); - int vdo_proc_net_stat_synproxy = !config_get_boolean("plugin:proc", "/proc/net/stat/synproxy", 1); - int vdo_proc_stat = !config_get_boolean("plugin:proc", "/proc/stat", 1); - int vdo_proc_meminfo = !config_get_boolean("plugin:proc", "/proc/meminfo", 1); - int vdo_proc_vmstat = !config_get_boolean("plugin:proc", "/proc/vmstat", 1); - int vdo_proc_net_rpc_nfsd = !config_get_boolean("plugin:proc", "/proc/net/rpc/nfsd", 1); - int vdo_proc_sys_kernel_random_entropy_avail = !config_get_boolean("plugin:proc", "/proc/sys/kernel/random/entropy_avail", 1); - int vdo_proc_interrupts = !config_get_boolean("plugin:proc", "/proc/interrupts", 1); - int vdo_proc_softirqs = !config_get_boolean("plugin:proc", "/proc/softirqs", 1); - int vdo_proc_loadavg = !config_get_boolean("plugin:proc", "/proc/loadavg", 1); - int vdo_sys_kernel_mm_ksm = !config_get_boolean("plugin:proc", "/sys/kernel/mm/ksm", 1); - int vdo_cpu_netdata = !config_get_boolean("plugin:proc", "netdata server resources", 1); - - // keep track of the time each module was called - unsigned long long sutime_proc_net_dev = 0ULL; - unsigned long long sutime_proc_diskstats = 0ULL; - unsigned long long sutime_proc_net_snmp = 0ULL; - unsigned long long sutime_proc_net_snmp6 = 0ULL; - unsigned long long sutime_proc_net_netstat = 0ULL; - unsigned long long sutime_proc_net_stat_conntrack = 0ULL; - unsigned long long sutime_proc_net_ip_vs_stats = 0ULL; - unsigned long long sutime_proc_net_stat_synproxy = 0ULL; - unsigned long long sutime_proc_stat = 0ULL; - unsigned long long sutime_proc_meminfo = 0ULL; - unsigned long long sutime_proc_vmstat = 0ULL; - unsigned long long sutime_proc_net_rpc_nfsd = 0ULL; - unsigned long long sutime_proc_sys_kernel_random_entropy_avail = 0ULL; - unsigned long long sutime_proc_interrupts = 0ULL; - unsigned long long sutime_proc_softirqs = 0ULL; - unsigned long long sutime_proc_loadavg = 0ULL; - unsigned long long sutime_sys_kernel_mm_ksm = 0ULL; - - // the next time we will run - aligned properly - unsigned long long sunext = (time(NULL) - (time(NULL) % rrd_update_every) + rrd_update_every) * 1000000ULL; - unsigned long long sunow; - - RRDSET *stcpu = NULL, *stcpu_thread = NULL, *stclients = NULL, *streqs = NULL, *stbytes = NULL; - - for(;1;) { - if(unlikely(netdata_exit)) break; - - // delay until it is our time to run - while((sunow = timems()) < sunext) - usleep((useconds_t)(sunext - sunow)); - - // find the next time we need to run - while(timems() > sunext) - sunext += rrd_update_every * 1000000ULL; - - if(unlikely(netdata_exit)) break; - - // BEGIN -- the job to be done - - if(!vdo_sys_kernel_mm_ksm) { - debug(D_PROCNETDEV_LOOP, "PROCNETDEV: calling do_sys_kernel_mm_ksm()."); - - sunow = timems(); - vdo_sys_kernel_mm_ksm = do_sys_kernel_mm_ksm(rrd_update_every, (sutime_sys_kernel_mm_ksm > 0)?sunow - sutime_sys_kernel_mm_ksm:0ULL); - sutime_sys_kernel_mm_ksm = sunow; - } - if(unlikely(netdata_exit)) break; - - if(!vdo_proc_loadavg) { - debug(D_PROCNETDEV_LOOP, "PROCNETDEV: calling do_proc_loadavg()."); - sunow = timems(); - vdo_proc_loadavg = do_proc_loadavg(rrd_update_every, (sutime_proc_loadavg > 0)?sunow - sutime_proc_loadavg:0ULL); - sutime_proc_loadavg = sunow; - } - if(unlikely(netdata_exit)) break; - - if(!vdo_proc_interrupts) { - debug(D_PROCNETDEV_LOOP, "PROCNETDEV: calling do_proc_interrupts()."); - sunow = timems(); - vdo_proc_interrupts = do_proc_interrupts(rrd_update_every, (sutime_proc_interrupts > 0)?sunow - sutime_proc_interrupts:0ULL); - sutime_proc_interrupts = sunow; - } - if(unlikely(netdata_exit)) break; - - if(!vdo_proc_softirqs) { - debug(D_PROCNETDEV_LOOP, "PROCNETDEV: calling do_proc_softirqs()."); - sunow = timems(); - vdo_proc_softirqs = do_proc_softirqs(rrd_update_every, (sutime_proc_softirqs > 0)?sunow - sutime_proc_softirqs:0ULL); - sutime_proc_softirqs = sunow; - } - if(unlikely(netdata_exit)) break; - - if(!vdo_proc_sys_kernel_random_entropy_avail) { - debug(D_PROCNETDEV_LOOP, "PROCNETDEV: calling do_proc_sys_kernel_random_entropy_avail()."); - sunow = timems(); - vdo_proc_sys_kernel_random_entropy_avail = do_proc_sys_kernel_random_entropy_avail(rrd_update_every, (sutime_proc_sys_kernel_random_entropy_avail > 0)?sunow - sutime_proc_sys_kernel_random_entropy_avail:0ULL); - sutime_proc_sys_kernel_random_entropy_avail = sunow; - } - if(unlikely(netdata_exit)) break; - - if(!vdo_proc_net_dev) { - debug(D_PROCNETDEV_LOOP, "PROCNETDEV: calling do_proc_net_dev()."); - sunow = timems(); - vdo_proc_net_dev = do_proc_net_dev(rrd_update_every, (sutime_proc_net_dev > 0)?sunow - sutime_proc_net_dev:0ULL); - sutime_proc_net_dev = sunow; - } - if(unlikely(netdata_exit)) break; - - if(!vdo_proc_diskstats) { - debug(D_PROCNETDEV_LOOP, "PROCNETDEV: calling do_proc_diskstats()."); - sunow = timems(); - vdo_proc_diskstats = do_proc_diskstats(rrd_update_every, (sutime_proc_diskstats > 0)?sunow - sutime_proc_diskstats:0ULL); - sutime_proc_diskstats = sunow; - } - if(unlikely(netdata_exit)) break; - - if(!vdo_proc_net_snmp) { - debug(D_PROCNETDEV_LOOP, "PROCNETDEV: calling do_proc_net_snmp()."); - sunow = timems(); - vdo_proc_net_snmp = do_proc_net_snmp(rrd_update_every, (sutime_proc_net_snmp > 0)?sunow - sutime_proc_net_snmp:0ULL); - sutime_proc_net_snmp = sunow; - } - if(unlikely(netdata_exit)) break; - - if(!vdo_proc_net_snmp6) { - debug(D_PROCNETDEV_LOOP, "PROCNETDEV: calling do_proc_net_snmp6()."); - sunow = timems(); - vdo_proc_net_snmp6 = do_proc_net_snmp6(rrd_update_every, (sutime_proc_net_snmp6 > 0)?sunow - sutime_proc_net_snmp6:0ULL); - sutime_proc_net_snmp6 = sunow; - } - if(unlikely(netdata_exit)) break; - - if(!vdo_proc_net_netstat) { - debug(D_PROCNETDEV_LOOP, "PROCNETDEV: calling do_proc_net_netstat()."); - sunow = timems(); - vdo_proc_net_netstat = do_proc_net_netstat(rrd_update_every, (sutime_proc_net_netstat > 0)?sunow - sutime_proc_net_netstat:0ULL); - sutime_proc_net_netstat = sunow; - } - if(unlikely(netdata_exit)) break; - - if(!vdo_proc_net_stat_conntrack) { - debug(D_PROCNETDEV_LOOP, "PROCNETDEV: calling do_proc_net_stat_conntrack()."); - sunow = timems(); - vdo_proc_net_stat_conntrack = do_proc_net_stat_conntrack(rrd_update_every, (sutime_proc_net_stat_conntrack > 0)?sunow - sutime_proc_net_stat_conntrack:0ULL); - sutime_proc_net_stat_conntrack = sunow; - } - if(unlikely(netdata_exit)) break; - - if(!vdo_proc_net_ip_vs_stats) { - debug(D_PROCNETDEV_LOOP, "PROCNETDEV: calling vdo_proc_net_ip_vs_stats()."); - sunow = timems(); - vdo_proc_net_ip_vs_stats = do_proc_net_ip_vs_stats(rrd_update_every, (sutime_proc_net_ip_vs_stats > 0)?sunow - sutime_proc_net_ip_vs_stats:0ULL); - sutime_proc_net_ip_vs_stats = sunow; - } - if(unlikely(netdata_exit)) break; - - if(!vdo_proc_net_stat_synproxy) { - debug(D_PROCNETDEV_LOOP, "PROCNETDEV: calling vdo_proc_net_stat_synproxy()."); - sunow = timems(); - vdo_proc_net_stat_synproxy = do_proc_net_stat_synproxy(rrd_update_every, (sutime_proc_net_stat_synproxy > 0)?sunow - sutime_proc_net_stat_synproxy:0ULL); - sutime_proc_net_stat_synproxy = sunow; - } - if(unlikely(netdata_exit)) break; - - if(!vdo_proc_stat) { - debug(D_PROCNETDEV_LOOP, "PROCNETDEV: calling do_proc_stat()."); - sunow = timems(); - vdo_proc_stat = do_proc_stat(rrd_update_every, (sutime_proc_stat > 0)?sunow - sutime_proc_stat:0ULL); - sutime_proc_stat = sunow; - } - if(unlikely(netdata_exit)) break; - - if(!vdo_proc_meminfo) { - debug(D_PROCNETDEV_LOOP, "PROCNETDEV: calling vdo_proc_meminfo()."); - sunow = timems(); - vdo_proc_meminfo = do_proc_meminfo(rrd_update_every, (sutime_proc_meminfo > 0)?sunow - sutime_proc_meminfo:0ULL); - sutime_proc_meminfo = sunow; - } - if(unlikely(netdata_exit)) break; - - if(!vdo_proc_vmstat) { - debug(D_PROCNETDEV_LOOP, "PROCNETDEV: calling vdo_proc_vmstat()."); - sunow = timems(); - vdo_proc_vmstat = do_proc_vmstat(rrd_update_every, (sutime_proc_vmstat > 0)?sunow - sutime_proc_vmstat:0ULL); - sutime_proc_vmstat = sunow; - } - if(unlikely(netdata_exit)) break; - - if(!vdo_proc_net_rpc_nfsd) { - debug(D_PROCNETDEV_LOOP, "PROCNETDEV: calling do_proc_net_rpc_nfsd()."); - sunow = timems(); - vdo_proc_net_rpc_nfsd = do_proc_net_rpc_nfsd(rrd_update_every, (sutime_proc_net_rpc_nfsd > 0)?sunow - sutime_proc_net_rpc_nfsd:0ULL); - sutime_proc_net_rpc_nfsd = sunow; - } - if(unlikely(netdata_exit)) break; - - // END -- the job is done - - // -------------------------------------------------------------------- - - if(!vdo_cpu_netdata) { - getrusage(RUSAGE_THREAD, &thread); - getrusage(RUSAGE_SELF, &me); - - if(!stcpu_thread) stcpu_thread = rrdset_find("netdata.plugin_proc_cpu"); - if(!stcpu_thread) { - stcpu_thread = rrdset_create("netdata", "plugin_proc_cpu", NULL, "proc.internal", NULL, "NetData Proc Plugin CPU usage", "milliseconds/s", 132000, rrd_update_every, RRDSET_TYPE_STACKED); - - rrddim_add(stcpu_thread, "user", NULL, 1, 1000, RRDDIM_INCREMENTAL); - rrddim_add(stcpu_thread, "system", NULL, 1, 1000, RRDDIM_INCREMENTAL); - } - else rrdset_next(stcpu_thread); - - rrddim_set(stcpu_thread, "user" , thread.ru_utime.tv_sec * 1000000ULL + thread.ru_utime.tv_usec); - rrddim_set(stcpu_thread, "system", thread.ru_stime.tv_sec * 1000000ULL + thread.ru_stime.tv_usec); - rrdset_done(stcpu_thread); - - // ---------------------------------------------------------------- - - if(!stcpu) stcpu = rrdset_find("netdata.server_cpu"); - if(!stcpu) { - stcpu = rrdset_create("netdata", "server_cpu", NULL, "netdata", NULL, "NetData CPU usage", "milliseconds/s", 130000, rrd_update_every, RRDSET_TYPE_STACKED); - - rrddim_add(stcpu, "user", NULL, 1, 1000, RRDDIM_INCREMENTAL); - rrddim_add(stcpu, "system", NULL, 1, 1000, RRDDIM_INCREMENTAL); - } - else rrdset_next(stcpu); - - rrddim_set(stcpu, "user" , me.ru_utime.tv_sec * 1000000ULL + me.ru_utime.tv_usec); - rrddim_set(stcpu, "system", me.ru_stime.tv_sec * 1000000ULL + me.ru_stime.tv_usec); - rrdset_done(stcpu); - - // ---------------------------------------------------------------- - - if(!stclients) stclients = rrdset_find("netdata.clients"); - if(!stclients) { - stclients = rrdset_create("netdata", "clients", NULL, "netdata", NULL, "NetData Web Clients", "connected clients", 130100, rrd_update_every, RRDSET_TYPE_LINE); - - rrddim_add(stclients, "clients", NULL, 1, 1, RRDDIM_ABSOLUTE); - } - else rrdset_next(stclients); - - rrddim_set(stclients, "clients", global_statistics.connected_clients); - rrdset_done(stclients); - - // ---------------------------------------------------------------- - - if(!streqs) streqs = rrdset_find("netdata.requests"); - if(!streqs) { - streqs = rrdset_create("netdata", "requests", NULL, "netdata", NULL, "NetData Web Requests", "requests/s", 130200, rrd_update_every, RRDSET_TYPE_LINE); - - rrddim_add(streqs, "requests", NULL, 1, 1, RRDDIM_INCREMENTAL); - } - else rrdset_next(streqs); - - rrddim_set(streqs, "requests", global_statistics.web_requests); - rrdset_done(streqs); - - // ---------------------------------------------------------------- - - if(!stbytes) stbytes = rrdset_find("netdata.net"); - if(!stbytes) { - stbytes = rrdset_create("netdata", "net", NULL, "netdata", NULL, "NetData Network Traffic", "kilobits/s", 130300, rrd_update_every, RRDSET_TYPE_AREA); - - rrddim_add(stbytes, "in", NULL, 8, 1024, RRDDIM_INCREMENTAL); - rrddim_add(stbytes, "out", NULL, -8, 1024, RRDDIM_INCREMENTAL); - } - else rrdset_next(stbytes); - - rrddim_set(stbytes, "in", global_statistics.bytes_received); - rrddim_set(stbytes, "out", global_statistics.bytes_sent); - rrdset_done(stbytes); - - // ---------------------------------------------------------------- - - registry_statistics(); - } - } - - pthread_exit(NULL); - return NULL; + (void)ptr; + + info("PROC Plugin thread created with task id %d", gettid()); + + if(pthread_setcanceltype(PTHREAD_CANCEL_DEFERRED, NULL) != 0) + error("Cannot set pthread cancel type to DEFERRED."); + + if(pthread_setcancelstate(PTHREAD_CANCEL_ENABLE, NULL) != 0) + error("Cannot set pthread cancel state to ENABLE."); + + // disable (by default) various interface that are not needed + config_get_boolean("plugin:proc:/proc/net/dev:lo", "enabled", 0); + config_get_boolean("plugin:proc:/proc/net/dev:fireqos_monitor", "enabled", 0); + + // when ZERO, attempt to do it + int vdo_proc_net_dev = !config_get_boolean("plugin:proc", "/proc/net/dev", 1); + int vdo_proc_diskstats = !config_get_boolean("plugin:proc", "/proc/diskstats", 1); + int vdo_proc_net_snmp = !config_get_boolean("plugin:proc", "/proc/net/snmp", 1); + int vdo_proc_net_snmp6 = !config_get_boolean("plugin:proc", "/proc/net/snmp6", 1); + int vdo_proc_net_netstat = !config_get_boolean("plugin:proc", "/proc/net/netstat", 1); + int vdo_proc_net_stat_conntrack = !config_get_boolean("plugin:proc", "/proc/net/stat/conntrack", 1); + int vdo_proc_net_ip_vs_stats = !config_get_boolean("plugin:proc", "/proc/net/ip_vs/stats", 1); + int vdo_proc_net_stat_synproxy = !config_get_boolean("plugin:proc", "/proc/net/stat/synproxy", 1); + int vdo_proc_stat = !config_get_boolean("plugin:proc", "/proc/stat", 1); + int vdo_proc_meminfo = !config_get_boolean("plugin:proc", "/proc/meminfo", 1); + int vdo_proc_vmstat = !config_get_boolean("plugin:proc", "/proc/vmstat", 1); + int vdo_proc_net_rpc_nfsd = !config_get_boolean("plugin:proc", "/proc/net/rpc/nfsd", 1); + int vdo_proc_sys_kernel_random_entropy_avail = !config_get_boolean("plugin:proc", "/proc/sys/kernel/random/entropy_avail", 1); + int vdo_proc_interrupts = !config_get_boolean("plugin:proc", "/proc/interrupts", 1); + int vdo_proc_softirqs = !config_get_boolean("plugin:proc", "/proc/softirqs", 1); + int vdo_proc_loadavg = !config_get_boolean("plugin:proc", "/proc/loadavg", 1); + int vdo_sys_kernel_mm_ksm = !config_get_boolean("plugin:proc", "/sys/kernel/mm/ksm", 1); + int vdo_cpu_netdata = !config_get_boolean("plugin:proc", "netdata server resources", 1); + + // keep track of the time each module was called + unsigned long long sutime_proc_net_dev = 0ULL; + unsigned long long sutime_proc_diskstats = 0ULL; + unsigned long long sutime_proc_net_snmp = 0ULL; + unsigned long long sutime_proc_net_snmp6 = 0ULL; + unsigned long long sutime_proc_net_netstat = 0ULL; + unsigned long long sutime_proc_net_stat_conntrack = 0ULL; + unsigned long long sutime_proc_net_ip_vs_stats = 0ULL; + unsigned long long sutime_proc_net_stat_synproxy = 0ULL; + unsigned long long sutime_proc_stat = 0ULL; + unsigned long long sutime_proc_meminfo = 0ULL; + unsigned long long sutime_proc_vmstat = 0ULL; + unsigned long long sutime_proc_net_rpc_nfsd = 0ULL; + unsigned long long sutime_proc_sys_kernel_random_entropy_avail = 0ULL; + unsigned long long sutime_proc_interrupts = 0ULL; + unsigned long long sutime_proc_softirqs = 0ULL; + unsigned long long sutime_proc_loadavg = 0ULL; + unsigned long long sutime_sys_kernel_mm_ksm = 0ULL; + + // the next time we will run - aligned properly + unsigned long long sunext = (time(NULL) - (time(NULL) % rrd_update_every) + rrd_update_every) * 1000000ULL; + unsigned long long sunow; + + for(;1;) { + if(unlikely(netdata_exit)) break; + + // delay until it is our time to run + while((sunow = time_usec()) < sunext) + sleep_usec(sunext - sunow); + + // find the next time we need to run + while(time_usec() > sunext) + sunext += rrd_update_every * 1000000ULL; + + if(unlikely(netdata_exit)) break; + + // BEGIN -- the job to be done + + if(!vdo_sys_kernel_mm_ksm) { + debug(D_PROCNETDEV_LOOP, "PROCNETDEV: calling do_sys_kernel_mm_ksm()."); + + sunow = time_usec(); + vdo_sys_kernel_mm_ksm = do_sys_kernel_mm_ksm(rrd_update_every, (sutime_sys_kernel_mm_ksm > 0)?sunow - sutime_sys_kernel_mm_ksm:0ULL); + sutime_sys_kernel_mm_ksm = sunow; + } + if(unlikely(netdata_exit)) break; + + if(!vdo_proc_loadavg) { + debug(D_PROCNETDEV_LOOP, "PROCNETDEV: calling do_proc_loadavg()."); + sunow = time_usec(); + vdo_proc_loadavg = do_proc_loadavg(rrd_update_every, (sutime_proc_loadavg > 0)?sunow - sutime_proc_loadavg:0ULL); + sutime_proc_loadavg = sunow; + } + if(unlikely(netdata_exit)) break; + + if(!vdo_proc_interrupts) { + debug(D_PROCNETDEV_LOOP, "PROCNETDEV: calling do_proc_interrupts()."); + sunow = time_usec(); + vdo_proc_interrupts = do_proc_interrupts(rrd_update_every, (sutime_proc_interrupts > 0)?sunow - sutime_proc_interrupts:0ULL); + sutime_proc_interrupts = sunow; + } + if(unlikely(netdata_exit)) break; + + if(!vdo_proc_softirqs) { + debug(D_PROCNETDEV_LOOP, "PROCNETDEV: calling do_proc_softirqs()."); + sunow = time_usec(); + vdo_proc_softirqs = do_proc_softirqs(rrd_update_every, (sutime_proc_softirqs > 0)?sunow - sutime_proc_softirqs:0ULL); + sutime_proc_softirqs = sunow; + } + if(unlikely(netdata_exit)) break; + + if(!vdo_proc_sys_kernel_random_entropy_avail) { + debug(D_PROCNETDEV_LOOP, "PROCNETDEV: calling do_proc_sys_kernel_random_entropy_avail()."); + sunow = time_usec(); + vdo_proc_sys_kernel_random_entropy_avail = do_proc_sys_kernel_random_entropy_avail(rrd_update_every, (sutime_proc_sys_kernel_random_entropy_avail > 0)?sunow - sutime_proc_sys_kernel_random_entropy_avail:0ULL); + sutime_proc_sys_kernel_random_entropy_avail = sunow; + } + if(unlikely(netdata_exit)) break; + + if(!vdo_proc_net_dev) { + debug(D_PROCNETDEV_LOOP, "PROCNETDEV: calling do_proc_net_dev()."); + sunow = time_usec(); + vdo_proc_net_dev = do_proc_net_dev(rrd_update_every, (sutime_proc_net_dev > 0)?sunow - sutime_proc_net_dev:0ULL); + sutime_proc_net_dev = sunow; + } + if(unlikely(netdata_exit)) break; + + if(!vdo_proc_diskstats) { + debug(D_PROCNETDEV_LOOP, "PROCNETDEV: calling do_proc_diskstats()."); + sunow = time_usec(); + vdo_proc_diskstats = do_proc_diskstats(rrd_update_every, (sutime_proc_diskstats > 0)?sunow - sutime_proc_diskstats:0ULL); + sutime_proc_diskstats = sunow; + } + if(unlikely(netdata_exit)) break; + + if(!vdo_proc_net_snmp) { + debug(D_PROCNETDEV_LOOP, "PROCNETDEV: calling do_proc_net_snmp()."); + sunow = time_usec(); + vdo_proc_net_snmp = do_proc_net_snmp(rrd_update_every, (sutime_proc_net_snmp > 0)?sunow - sutime_proc_net_snmp:0ULL); + sutime_proc_net_snmp = sunow; + } + if(unlikely(netdata_exit)) break; + + if(!vdo_proc_net_snmp6) { + debug(D_PROCNETDEV_LOOP, "PROCNETDEV: calling do_proc_net_snmp6()."); + sunow = time_usec(); + vdo_proc_net_snmp6 = do_proc_net_snmp6(rrd_update_every, (sutime_proc_net_snmp6 > 0)?sunow - sutime_proc_net_snmp6:0ULL); + sutime_proc_net_snmp6 = sunow; + } + if(unlikely(netdata_exit)) break; + + if(!vdo_proc_net_netstat) { + debug(D_PROCNETDEV_LOOP, "PROCNETDEV: calling do_proc_net_netstat()."); + sunow = time_usec(); + vdo_proc_net_netstat = do_proc_net_netstat(rrd_update_every, (sutime_proc_net_netstat > 0)?sunow - sutime_proc_net_netstat:0ULL); + sutime_proc_net_netstat = sunow; + } + if(unlikely(netdata_exit)) break; + + if(!vdo_proc_net_stat_conntrack) { + debug(D_PROCNETDEV_LOOP, "PROCNETDEV: calling do_proc_net_stat_conntrack()."); + sunow = time_usec(); + vdo_proc_net_stat_conntrack = do_proc_net_stat_conntrack(rrd_update_every, (sutime_proc_net_stat_conntrack > 0)?sunow - sutime_proc_net_stat_conntrack:0ULL); + sutime_proc_net_stat_conntrack = sunow; + } + if(unlikely(netdata_exit)) break; + + if(!vdo_proc_net_ip_vs_stats) { + debug(D_PROCNETDEV_LOOP, "PROCNETDEV: calling vdo_proc_net_ip_vs_stats()."); + sunow = time_usec(); + vdo_proc_net_ip_vs_stats = do_proc_net_ip_vs_stats(rrd_update_every, (sutime_proc_net_ip_vs_stats > 0)?sunow - sutime_proc_net_ip_vs_stats:0ULL); + sutime_proc_net_ip_vs_stats = sunow; + } + if(unlikely(netdata_exit)) break; + + if(!vdo_proc_net_stat_synproxy) { + debug(D_PROCNETDEV_LOOP, "PROCNETDEV: calling vdo_proc_net_stat_synproxy()."); + sunow = time_usec(); + vdo_proc_net_stat_synproxy = do_proc_net_stat_synproxy(rrd_update_every, (sutime_proc_net_stat_synproxy > 0)?sunow - sutime_proc_net_stat_synproxy:0ULL); + sutime_proc_net_stat_synproxy = sunow; + } + if(unlikely(netdata_exit)) break; + + if(!vdo_proc_stat) { + debug(D_PROCNETDEV_LOOP, "PROCNETDEV: calling do_proc_stat()."); + sunow = time_usec(); + vdo_proc_stat = do_proc_stat(rrd_update_every, (sutime_proc_stat > 0)?sunow - sutime_proc_stat:0ULL); + sutime_proc_stat = sunow; + } + if(unlikely(netdata_exit)) break; + + if(!vdo_proc_meminfo) { + debug(D_PROCNETDEV_LOOP, "PROCNETDEV: calling vdo_proc_meminfo()."); + sunow = time_usec(); + vdo_proc_meminfo = do_proc_meminfo(rrd_update_every, (sutime_proc_meminfo > 0)?sunow - sutime_proc_meminfo:0ULL); + sutime_proc_meminfo = sunow; + } + if(unlikely(netdata_exit)) break; + + if(!vdo_proc_vmstat) { + debug(D_PROCNETDEV_LOOP, "PROCNETDEV: calling vdo_proc_vmstat()."); + sunow = time_usec(); + vdo_proc_vmstat = do_proc_vmstat(rrd_update_every, (sutime_proc_vmstat > 0)?sunow - sutime_proc_vmstat:0ULL); + sutime_proc_vmstat = sunow; + } + if(unlikely(netdata_exit)) break; + + if(!vdo_proc_net_rpc_nfsd) { + debug(D_PROCNETDEV_LOOP, "PROCNETDEV: calling do_proc_net_rpc_nfsd()."); + sunow = time_usec(); + vdo_proc_net_rpc_nfsd = do_proc_net_rpc_nfsd(rrd_update_every, (sutime_proc_net_rpc_nfsd > 0)?sunow - sutime_proc_net_rpc_nfsd:0ULL); + sutime_proc_net_rpc_nfsd = sunow; + } + if(unlikely(netdata_exit)) break; + + // END -- the job is done + + // -------------------------------------------------------------------- + + if(!vdo_cpu_netdata) { + global_statistics_charts(); + registry_statistics(); + } + } + + pthread_exit(NULL); + return NULL; } diff --git a/src/plugin_tc.c b/src/plugin_tc.c index 3d3e35217..408069dba 100644 --- a/src/plugin_tc.c +++ b/src/plugin_tc.c @@ -1,24 +1,7 @@ -#ifdef HAVE_CONFIG_H -#include <config.h> -#endif -#include <stdio.h> -#include <stdlib.h> -#include <string.h> -#include <sys/types.h> -#include <sys/resource.h> - -#include "avl.h" -#include "log.h" #include "common.h" -#include "appconfig.h" -#include "rrd.h" -#include "popen.h" -#include "plugin_tc.h" -#include "main.h" -#include "../config.h" -#define RRD_TYPE_TC "tc" -#define RRD_TYPE_TC_LEN strlen(RRD_TYPE_TC) +#define RRD_TYPE_TC "tc" +#define RRD_TYPE_TC_LEN strlen(RRD_TYPE_TC) // ---------------------------------------------------------------------------- // /sbin/tc processor @@ -27,54 +10,71 @@ #define TC_LINE_MAX 1024 struct tc_class { - avl avl; - - char *id; - uint32_t hash; - - char *name; - - char *leafid; - uint32_t leaf_hash; - - char *parentid; - uint32_t parent_hash; + avl avl; + + char *id; + uint32_t hash; + + char *name; + + char *leafid; + uint32_t leaf_hash; + + char *parentid; + uint32_t parent_hash; + + char hasparent; + char isleaf; + unsigned long long bytes; + unsigned long long packets; + unsigned long long dropped; + unsigned long long overlimits; + unsigned long long requeues; + unsigned long long lended; + unsigned long long borrowed; + unsigned long long giants; + unsigned long long tokens; + unsigned long long ctokens; + + RRDDIM *rd_bytes; + RRDDIM *rd_packets; + RRDDIM *rd_dropped; + + char name_updated; + char updated; // updated bytes + int seen; // seen in the tc list (even without bytes) + + struct tc_class *next; + struct tc_class *prev; +}; - char hasparent; - char isleaf; - unsigned long long bytes; - unsigned long long packets; - unsigned long long dropped; - unsigned long long overlimits; - unsigned long long requeues; - unsigned long long lended; - unsigned long long borrowed; - unsigned long long giants; - unsigned long long tokens; - unsigned long long ctokens; +struct tc_device { + avl avl; - char updated; // updated bytes - char seen; // seen in the tc list (even without bytes) + char *id; + uint32_t hash; - struct tc_class *next; - struct tc_class *prev; -}; + char *name; + char *family; -struct tc_device { - avl avl; + char name_updated; + char family_updated; - char *id; - uint32_t hash; + char enabled; + char enabled_bytes; + char enabled_packets; + char enabled_dropped; - char *name; - char *family; + RRDSET *st_bytes; + RRDSET *st_packets; + RRDSET *st_dropped; - avl_tree classes_index; + avl_tree classes_index; - struct tc_class *classes; + struct tc_class *classes; - struct tc_device *next; - struct tc_device *prev; + struct tc_device *next; + struct tc_device *prev; }; @@ -84,25 +84,25 @@ struct tc_device *tc_device_root = NULL; // tc_device index static int tc_device_compare(void* a, void* b) { - if(((struct tc_device *)a)->hash < ((struct tc_device *)b)->hash) return -1; - else if(((struct tc_device *)a)->hash > ((struct tc_device *)b)->hash) return 1; - else return strcmp(((struct tc_device *)a)->id, ((struct tc_device *)b)->id); + if(((struct tc_device *)a)->hash < ((struct tc_device *)b)->hash) return -1; + else if(((struct tc_device *)a)->hash > ((struct tc_device *)b)->hash) return 1; + else return strcmp(((struct tc_device *)a)->id, ((struct tc_device *)b)->id); } avl_tree tc_device_root_index = { - NULL, - tc_device_compare + NULL, + tc_device_compare }; #define tc_device_index_add(st) avl_insert(&tc_device_root_index, (avl *)(st)) #define tc_device_index_del(st) avl_remove(&tc_device_root_index, (avl *)(st)) static inline struct tc_device *tc_device_index_find(const char *id, uint32_t hash) { - struct tc_device tmp; - tmp.id = (char *)id; - tmp.hash = (hash)?hash:simple_hash(tmp.id); + struct tc_device tmp; + tmp.id = (char *)id; + tmp.hash = (hash)?hash:simple_hash(tmp.id); - return (struct tc_device *)avl_search(&(tc_device_root_index), (avl *)&tmp); + return (struct tc_device *)avl_search(&(tc_device_root_index), (avl *)&tmp); } @@ -110,621 +110,766 @@ static inline struct tc_device *tc_device_index_find(const char *id, uint32_t ha // tc_class index static int tc_class_compare(void* a, void* b) { - if(((struct tc_class *)a)->hash < ((struct tc_class *)b)->hash) return -1; - else if(((struct tc_class *)a)->hash > ((struct tc_class *)b)->hash) return 1; - else return strcmp(((struct tc_class *)a)->id, ((struct tc_class *)b)->id); + if(((struct tc_class *)a)->hash < ((struct tc_class *)b)->hash) return -1; + else if(((struct tc_class *)a)->hash > ((struct tc_class *)b)->hash) return 1; + else return strcmp(((struct tc_class *)a)->id, ((struct tc_class *)b)->id); } #define tc_class_index_add(st, rd) avl_insert(&((st)->classes_index), (avl *)(rd)) #define tc_class_index_del(st, rd) avl_remove(&((st)->classes_index), (avl *)(rd)) static inline struct tc_class *tc_class_index_find(struct tc_device *st, const char *id, uint32_t hash) { - struct tc_class tmp; - tmp.id = (char *)id; - tmp.hash = (hash)?hash:simple_hash(tmp.id); + struct tc_class tmp; + tmp.id = (char *)id; + tmp.hash = (hash)?hash:simple_hash(tmp.id); - return (struct tc_class *)avl_search(&(st->classes_index), (avl *) &tmp); + return (struct tc_class *)avl_search(&(st->classes_index), (avl *) &tmp); } // ---------------------------------------------------------------------------- static inline void tc_class_free(struct tc_device *n, struct tc_class *c) { - debug(D_TC_LOOP, "Removing from device '%s' class '%s', parentid '%s', leafid '%s', seen=%d", n->id, c->id, c->parentid?c->parentid:"", c->leafid?c->leafid:"", c->seen); - - if(c->next) c->next->prev = c->prev; - if(c->prev) c->prev->next = c->next; - if(n->classes == c) { - if(c->next) n->classes = c->next; - else n->classes = c->prev; - } - - tc_class_index_del(n, c); - - if(c->id) free(c->id); - if(c->name) free(c->name); - if(c->leafid) free(c->leafid); - if(c->parentid) free(c->parentid); - - free(c); + if(c == n->classes) { + if(c->next) + n->classes = c->next; + else + n->classes = c->prev; + } + if(c->next) c->next->prev = c->prev; + if(c->prev) c->prev->next = c->next; + + debug(D_TC_LOOP, "Removing from device '%s' class '%s', parentid '%s', leafid '%s', seen=%d", n->id, c->id, c->parentid?c->parentid:"", c->leafid?c->leafid:"", c->seen); + + tc_class_index_del(n, c); + + freez(c->id); + freez(c->name); + freez(c->leafid); + freez(c->parentid); + freez(c); } static inline void tc_device_classes_cleanup(struct tc_device *d) { - static int cleanup_every = 999; - - if(cleanup_every > 0) { - cleanup_every = (int) -config_get_number("plugin:tc", "cleanup unused classes every", 60); - if(cleanup_every > 0) cleanup_every = -cleanup_every; - if(cleanup_every == 0) cleanup_every = -1; - } - - struct tc_class *c = d->classes; - while(c) { - if(c->seen < cleanup_every) { - struct tc_class *nc = c->next; - tc_class_free(d, c); - c = nc; - } - else c = c->next; - - if(c) { - c->updated = 0; - c->seen--; - } - } + static int cleanup_every = 999; + + if(unlikely(cleanup_every > 0)) { + cleanup_every = (int) config_get_number("plugin:tc", "cleanup unused classes every", 60); + if(cleanup_every < 0) cleanup_every = -cleanup_every; + } + + d->name_updated = 0; + d->family_updated = 0; + + struct tc_class *c = d->classes; + while(c) { + if(unlikely(cleanup_every > 0 && c->seen >= cleanup_every)) { + struct tc_class *nc = c->next; + tc_class_free(d, c); + c = nc; + } + else { + c->updated = 0; + c->name_updated = 0; + + c = c->next; + } + } } -static inline void tc_device_commit(struct tc_device *d) -{ - static int enable_new_interfaces = -1; - - if(enable_new_interfaces == -1) enable_new_interfaces = config_get_boolean("plugin:tc", "enable new interfaces detected at runtime", 1); - - // we only need to add leaf classes - struct tc_class *c, *x; - - // set all classes - for(c = d->classes ; c ; c = c->next) { - c->isleaf = 1; - c->hasparent = 0; - } - - // mark the classes as leafs and parents - for(c = d->classes ; c ; c = c->next) { - if(!c->updated) continue; - - for(x = d->classes ; x ; x = x->next) { - if(!x->updated) continue; - - if(c == x) continue; - - if(x->parentid && ( - ( c->hash == x->parent_hash && strcmp(c->id, x->parentid) == 0) || - (c->leafid && c->leaf_hash == x->parent_hash && strcmp(c->leafid, x->parentid) == 0))) { - // debug(D_TC_LOOP, "TC: In device '%s', class '%s' (leafid: '%s') has as leaf class '%s' (parentid: '%s').", d->name?d->name:d->id, c->name?c->name:c->id, c->leafid?c->leafid:c->id, x->name?x->name:x->id, x->parentid?x->parentid:x->id); - c->isleaf = 0; - x->hasparent = 1; - } - } - } - - // debugging: - /* - for ( c = d->classes ; c ; c = c->next) { - if(c->isleaf && c->hasparent) debug(D_TC_LOOP, "TC: Device %s, class %s, OK", d->name, c->id); - else debug(D_TC_LOOP, "TC: Device %s, class %s, IGNORE (isleaf: %d, hasparent: %d, parent: %s)", d->name, c->id, c->isleaf, c->hasparent, c->parentid); - } - */ - - // we need at least a class - for(c = d->classes ; c ; c = c->next) { - // debug(D_TC_LOOP, "TC: Device '%s', class '%s', isLeaf=%d, HasParent=%d, Seen=%d", d->name?d->name:d->id, c->name?c->name:c->id, c->isleaf, c->hasparent, c->seen); - if(!c->updated) continue; - if(c->isleaf && c->hasparent) break; - } - if(!c) { - debug(D_TC_LOOP, "TC: Ignoring TC device '%s'. No leaf classes.", d->name?d->name:d->id); - tc_device_classes_cleanup(d); - return; - } - - char var_name[CONFIG_MAX_NAME + 1]; - snprintfz(var_name, CONFIG_MAX_NAME, "qos for %s", d->id); - if(config_get_boolean("plugin:tc", var_name, enable_new_interfaces)) { - RRDSET *st = rrdset_find_bytype(RRD_TYPE_TC, d->id); - if(!st) { - debug(D_TC_LOOP, "TC: Creating new chart for device '%s'", d->name?d->name:d->id); - - st = rrdset_create(RRD_TYPE_TC, d->id, d->name?d->name:d->id, d->family?d->family:d->id, RRD_TYPE_TC ".qos", "Class Usage", "kilobits/s", 7000, rrd_update_every, RRDSET_TYPE_STACKED); - - for(c = d->classes ; c ; c = c->next) { - if(!c->updated) continue; - - if(c->isleaf && c->hasparent) - rrddim_add(st, c->id, c->name?c->name:c->id, 8, 1024, RRDDIM_INCREMENTAL); - } - } - else { - debug(D_TC_LOOP, "TC: Updating chart for device '%s'", d->name?d->name:d->id); - rrdset_next_plugins(st); - - if(d->name && strcmp(d->id, d->name) != 0) rrdset_set_name(st, d->name); - } - - for(c = d->classes ; c ; c = c->next) { - if(!c->updated) continue; - - if(c->isleaf && c->hasparent) { - RRDDIM *rd = rrddim_find(st, c->id); - - if(!rd) { - debug(D_TC_LOOP, "TC: Adding to chart '%s', dimension '%s'", st->id, c->id, c->name); - - // new class, we have to add it - rd = rrddim_add(st, c->id, c->name?c->name:c->id, 8, 1024, RRDDIM_INCREMENTAL); - } - else debug(D_TC_LOOP, "TC: Updating chart '%s', dimension '%s'", st->id, c->id); - - rrddim_set_by_pointer(st, rd, c->bytes); - - // if it has a name, different to the id - if(c->name) { - // update the rrd dimension with the new name - debug(D_TC_LOOP, "TC: Setting chart '%s', dimension '%s' name to '%s'", st->id, rd->id, c->name); - rrddim_set_name(st, rd, c->name); - - free(c->name); - c->name = NULL; - } - } - } - rrdset_done(st); - } - - tc_device_classes_cleanup(d); +static inline void tc_device_commit(struct tc_device *d) { + static int enable_new_interfaces = -1, enable_bytes = -1, enable_packets = -1, enable_dropped = -1; + + if(unlikely(enable_new_interfaces == -1)) { + enable_new_interfaces = config_get_boolean_ondemand("plugin:tc", "enable new interfaces detected at runtime", CONFIG_ONDEMAND_YES); + enable_bytes = config_get_boolean_ondemand("plugin:tc", "enable traffic charts for all interfaces", CONFIG_ONDEMAND_ONDEMAND); + enable_packets = config_get_boolean_ondemand("plugin:tc", "enable packets charts for all interfaces", CONFIG_ONDEMAND_ONDEMAND); + enable_dropped = config_get_boolean_ondemand("plugin:tc", "enable dropped charts for all interfaces", CONFIG_ONDEMAND_ONDEMAND); + } + + // we only need to add leaf classes + struct tc_class *c, *x; + unsigned long long bytes_sum = 0, packets_sum = 0, dropped_sum = 0; + int active_classes = 0; + + // set all classes + for(c = d->classes ; c ; c = c->next) { + c->isleaf = 1; + c->hasparent = 0; + } + + // mark the classes as leafs and parents + for(c = d->classes ; c ; c = c->next) { + if(unlikely(!c->updated)) continue; + + for(x = d->classes ; x ; x = x->next) { + if(unlikely(!x->updated)) continue; + + if(unlikely(c == x)) continue; + + if(x->parentid && ( + ( c->hash == x->parent_hash && strcmp(c->id, x->parentid) == 0) || + (c->leafid && c->leaf_hash == x->parent_hash && strcmp(c->leafid, x->parentid) == 0))) { + // debug(D_TC_LOOP, "TC: In device '%s', class '%s' (leafid: '%s') has as leaf class '%s' (parentid: '%s').", d->name?d->name:d->id, c->name?c->name:c->id, c->leafid?c->leafid:c->id, x->name?x->name:x->id, x->parentid?x->parentid:x->id); + c->isleaf = 0; + x->hasparent = 1; + } + } + } + + // debugging: + /* + for ( c = d->classes ; c ; c = c->next) { + if(c->isleaf && c->hasparent) debug(D_TC_LOOP, "TC: Device %s, class %s, OK", d->name, c->id); + else debug(D_TC_LOOP, "TC: Device %s, class %s, IGNORE (isleaf: %d, hasparent: %d, parent: %s)", d->name, c->id, c->isleaf, c->hasparent, c->parentid); + } + */ + + // we need at least a class + for(c = d->classes ; c ; c = c->next) { + // debug(D_TC_LOOP, "TC: Device '%s', class '%s', isLeaf=%d, HasParent=%d, Seen=%d", d->name?d->name:d->id, c->name?c->name:c->id, c->isleaf, c->hasparent, c->seen); + if(!c->updated) continue; + if(c->isleaf && c->hasparent) { + active_classes++; + bytes_sum += c->bytes; + packets_sum += c->packets; + dropped_sum += c->dropped; + } + } + + if(unlikely(!active_classes)) { + debug(D_TC_LOOP, "TC: Ignoring TC device '%s'. No leaf classes.", d->name?d->name:d->id); + tc_device_classes_cleanup(d); + return; + } + + if(unlikely(d->enabled == -1)) { + char var_name[CONFIG_MAX_NAME + 1]; + snprintfz(var_name, CONFIG_MAX_NAME, "qos for %s", d->id); + d->enabled = config_get_boolean_ondemand("plugin:tc", var_name, enable_new_interfaces); + + snprintfz(var_name, CONFIG_MAX_NAME, "traffic chart for %s", d->id); + d->enabled_bytes = config_get_boolean_ondemand("plugin:tc", var_name, enable_bytes); + + snprintfz(var_name, CONFIG_MAX_NAME, "packets chart for %s", d->id); + d->enabled_packets = config_get_boolean_ondemand("plugin:tc", var_name, enable_packets); + + snprintfz(var_name, CONFIG_MAX_NAME, "dropped packets chart for %s", d->id); + d->enabled_dropped = config_get_boolean_ondemand("plugin:tc", var_name, enable_dropped); + } + + if(likely(d->enabled)) { + // -------------------------------------------------------------------- + // bytes + + if(d->enabled_bytes == CONFIG_ONDEMAND_YES || (d->enabled_bytes == CONFIG_ONDEMAND_ONDEMAND && bytes_sum)) { + d->enabled_bytes = CONFIG_ONDEMAND_YES; + + if(unlikely(!d->st_bytes)) { + d->st_bytes = rrdset_find_bytype(RRD_TYPE_TC, d->id); + if(unlikely(!d->st_bytes)) { + debug(D_TC_LOOP, "TC: Creating new chart for device '%s'", d->name?d->name:d->id); + d->st_bytes = rrdset_create(RRD_TYPE_TC, d->id, d->name?d->name:d->id, d->family?d->family:d->id, RRD_TYPE_TC ".qos", "Class Usage", "kilobits/s", 7000, rrd_update_every, RRDSET_TYPE_STACKED); + } + } + else { + debug(D_TC_LOOP, "TC: Updating chart for device '%s'", d->name?d->name:d->id); + rrdset_next_plugins(d->st_bytes); + + if(unlikely(d->name_updated && d->name && strcmp(d->id, d->name) != 0)) { + rrdset_set_name(d->st_bytes, d->name); + d->name_updated = 0; + } + + // FIXME + // update the family + } + + for(c = d->classes ; c ; c = c->next) { + if(unlikely(!c->updated)) continue; + + if(c->isleaf && c->hasparent) { + c->seen++; + + if(unlikely(!c->rd_bytes)) { + c->rd_bytes = rrddim_find(d->st_bytes, c->id); + if(unlikely(!c->rd_bytes)) { + debug(D_TC_LOOP, "TC: Adding to chart '%s', dimension '%s' (name: '%s')", d->st_bytes->id, c->id, c->name); + + // new class, we have to add it + c->rd_bytes = rrddim_add(d->st_bytes, c->id, c->name?c->name:c->id, 8, 1024, RRDDIM_INCREMENTAL); + } + else debug(D_TC_LOOP, "TC: Updating chart '%s', dimension '%s'", d->st_bytes->id, c->id); + } + + rrddim_set_by_pointer(d->st_bytes, c->rd_bytes, c->bytes); + + // if it has a name, different to the id + if(unlikely(c->name_updated && c->name && strcmp(c->id, c->name) != 0)) { + // update the rrd dimension with the new name + debug(D_TC_LOOP, "TC: Setting chart '%s', dimension '%s' name to '%s'", d->st_bytes->id, c->rd_bytes->id, c->name); + rrddim_set_name(d->st_bytes, c->rd_bytes, c->name); + } + } + } + rrdset_done(d->st_bytes); + } + + // -------------------------------------------------------------------- + // packets + + if(d->enabled_packets == CONFIG_ONDEMAND_YES || (d->enabled_packets == CONFIG_ONDEMAND_ONDEMAND && packets_sum)) { + d->enabled_packets = CONFIG_ONDEMAND_YES; + + if(unlikely(!d->st_packets)) { + char id[RRD_ID_LENGTH_MAX + 1]; + char name[RRD_ID_LENGTH_MAX + 1]; + snprintfz(id, RRD_ID_LENGTH_MAX, "%s_packets", d->id); + snprintfz(name, RRD_ID_LENGTH_MAX, "%s_packets", d->name?d->name:d->id); + + d->st_packets = rrdset_find_bytype(RRD_TYPE_TC, id); + if(unlikely(!d->st_packets)) { + debug(D_TC_LOOP, "TC: Creating new _packets chart for device '%s'", d->name?d->name:d->id); + d->st_packets = rrdset_create(RRD_TYPE_TC, id, name, d->family?d->family:d->id, RRD_TYPE_TC ".qos_packets", "Class Packets", "packets/s", 7010, rrd_update_every, RRDSET_TYPE_STACKED); + } + } + else { + debug(D_TC_LOOP, "TC: Updating _packets chart for device '%s'", d->name?d->name:d->id); + rrdset_next_plugins(d->st_packets); + + // FIXME + // update the family + } + + for(c = d->classes ; c ; c = c->next) { + if(unlikely(!c->updated)) continue; + + if(c->isleaf && c->hasparent) { + if(unlikely(!c->rd_packets)) { + c->rd_packets = rrddim_find(d->st_packets, c->id); + if(unlikely(!c->rd_packets)) { + debug(D_TC_LOOP, "TC: Adding to chart '%s', dimension '%s' (name: '%s')", d->st_packets->id, c->id, c->name); + + // new class, we have to add it + c->rd_packets = rrddim_add(d->st_packets, c->id, c->name?c->name:c->id, 1, 1, RRDDIM_INCREMENTAL); + } + else debug(D_TC_LOOP, "TC: Updating chart '%s', dimension '%s'", d->st_packets->id, c->id); + } + + rrddim_set_by_pointer(d->st_packets, c->rd_packets, c->packets); + + // if it has a name, different to the id + if(unlikely(c->name_updated && c->name && strcmp(c->id, c->name) != 0)) { + // update the rrd dimension with the new name + debug(D_TC_LOOP, "TC: Setting chart '%s', dimension '%s' name to '%s'", d->st_packets->id, c->rd_packets->id, c->name); + rrddim_set_name(d->st_packets, c->rd_packets, c->name); + } + } + } + rrdset_done(d->st_packets); + } + + // -------------------------------------------------------------------- + // dropped + + if(d->enabled_dropped == CONFIG_ONDEMAND_YES || (d->enabled_dropped == CONFIG_ONDEMAND_ONDEMAND && dropped_sum)) { + d->enabled_dropped = CONFIG_ONDEMAND_YES; + + if(unlikely(!d->st_dropped)) { + char id[RRD_ID_LENGTH_MAX + 1]; + char name[RRD_ID_LENGTH_MAX + 1]; + snprintfz(id, RRD_ID_LENGTH_MAX, "%s_dropped", d->id); + snprintfz(name, RRD_ID_LENGTH_MAX, "%s_dropped", d->name?d->name:d->id); + + d->st_dropped = rrdset_find_bytype(RRD_TYPE_TC, id); + if(unlikely(!d->st_dropped)) { + debug(D_TC_LOOP, "TC: Creating new _dropped chart for device '%s'", d->name?d->name:d->id); + d->st_dropped = rrdset_create(RRD_TYPE_TC, id, name, d->family?d->family:d->id, RRD_TYPE_TC ".qos_dropped", "Class Dropped Packets", "packets/s", 7020, rrd_update_every, RRDSET_TYPE_STACKED); + } + } + else { + debug(D_TC_LOOP, "TC: Updating _dropped chart for device '%s'", d->name?d->name:d->id); + rrdset_next_plugins(d->st_dropped); + + // FIXME + // update the family + } + + for(c = d->classes ; c ; c = c->next) { + if(unlikely(!c->updated)) continue; + + if(c->isleaf && c->hasparent) { + if(unlikely(!c->rd_dropped)) { + c->rd_dropped = rrddim_find(d->st_dropped, c->id); + if(unlikely(!c->rd_dropped)) { + debug(D_TC_LOOP, "TC: Adding to chart '%s', dimension '%s' (name: '%s')", d->st_dropped->id, c->id, c->name); + + // new class, we have to add it + c->rd_dropped = rrddim_add(d->st_dropped, c->id, c->name?c->name:c->id, 1, 1, RRDDIM_INCREMENTAL); + } + else debug(D_TC_LOOP, "TC: Updating chart '%s', dimension '%s'", d->st_dropped->id, c->id); + } + + rrddim_set_by_pointer(d->st_dropped, c->rd_dropped, c->dropped); + + // if it has a name, different to the id + if(unlikely(c->name_updated && c->name && strcmp(c->id, c->name) != 0)) { + // update the rrd dimension with the new name + debug(D_TC_LOOP, "TC: Setting chart '%s', dimension '%s' name to '%s'", d->st_dropped->id, c->rd_dropped->id, c->name); + rrddim_set_name(d->st_dropped, c->rd_dropped, c->name); + } + } + } + rrdset_done(d->st_dropped); + } + } + + tc_device_classes_cleanup(d); } static inline void tc_device_set_class_name(struct tc_device *d, char *id, char *name) { - struct tc_class *c = tc_class_index_find(d, id, 0); - if(c) { - if(c->name) free(c->name); - c->name = NULL; - - if(name && *name && strcmp(c->id, name) != 0) { - debug(D_TC_LOOP, "TC: Setting device '%s', class '%s' name to '%s'", d->id, id, name); - c->name = strdup(name); - } - } + struct tc_class *c = tc_class_index_find(d, id, 0); + if(likely(c)) { + freez(c->name); + c->name = NULL; + + if(likely(name && *name && strcmp(c->id, name) != 0)) { + debug(D_TC_LOOP, "TC: Setting device '%s', class '%s' name to '%s'", d->id, id, name); + c->name = strdupz(name); + c->name_updated = 1; + } + } } static inline void tc_device_set_device_name(struct tc_device *d, char *name) { - if(d->name) free(d->name); - d->name = NULL; - - if(name && *name && strcmp(d->id, name) != 0) { - debug(D_TC_LOOP, "TC: Setting device '%s' name to '%s'", d->id, name); - d->name = strdup(name); - } + freez(d->name); + d->name = NULL; + + if(likely(name && *name && strcmp(d->id, name) != 0)) { + debug(D_TC_LOOP, "TC: Setting device '%s' name to '%s'", d->id, name); + d->name = strdupz(name); + d->name_updated = 1; + } } static inline void tc_device_set_device_family(struct tc_device *d, char *family) { - if(d->family) free(d->family); - d->family = NULL; - - if(family && *family && strcmp(d->id, family) != 0) { - debug(D_TC_LOOP, "TC: Setting device '%s' family to '%s'", d->id, family); - d->family = strdup(family); - } - // no need for null termination - it is already null + freez(d->family); + d->family = NULL; + + if(likely(family && *family && strcmp(d->id, family) != 0)) { + debug(D_TC_LOOP, "TC: Setting device '%s' family to '%s'", d->id, family); + d->family = strdupz(family); + d->family_updated = 1; + } + // no need for null termination - it is already null } static inline struct tc_device *tc_device_create(char *id) { - struct tc_device *d = tc_device_index_find(id, 0); - - if(!d) { - debug(D_TC_LOOP, "TC: Creating device '%s'", id); - - d = calloc(1, sizeof(struct tc_device)); - if(!d) { - fatal("Cannot allocate memory for tc_device %s", id); - return NULL; - } - - d->id = strdup(id); - d->hash = simple_hash(d->id); - - avl_init(&d->classes_index, tc_class_compare); - tc_device_index_add(d); - - if(!tc_device_root) { - tc_device_root = d; - } - else { - d->next = tc_device_root; - tc_device_root->prev = d; - tc_device_root = d; - } - } - - return(d); + struct tc_device *d = tc_device_index_find(id, 0); + + if(!d) { + debug(D_TC_LOOP, "TC: Creating device '%s'", id); + + d = callocz(1, sizeof(struct tc_device)); + + d->id = strdupz(id); + d->hash = simple_hash(d->id); + d->enabled = -1; + + avl_init(&d->classes_index, tc_class_compare); + tc_device_index_add(d); + + if(!tc_device_root) { + tc_device_root = d; + } + else { + d->next = tc_device_root; + tc_device_root->prev = d; + tc_device_root = d; + } + } + + return(d); } static inline struct tc_class *tc_class_add(struct tc_device *n, char *id, char *parentid, char *leafid) { - struct tc_class *c = tc_class_index_find(n, id, 0); + struct tc_class *c = tc_class_index_find(n, id, 0); - if(!c) { - debug(D_TC_LOOP, "TC: Creating in device '%s', class id '%s', parentid '%s', leafid '%s'", n->id, id, parentid?parentid:"", leafid?leafid:""); + if(!c) { + debug(D_TC_LOOP, "TC: Creating in device '%s', class id '%s', parentid '%s', leafid '%s'", n->id, id, parentid?parentid:"", leafid?leafid:""); - c = calloc(1, sizeof(struct tc_class)); - if(!c) { - fatal("Cannot allocate memory for tc class"); - return NULL; - } + c = callocz(1, sizeof(struct tc_class)); - if(n->classes) n->classes->prev = c; - c->next = n->classes; - n->classes = c; + if(n->classes) n->classes->prev = c; + c->next = n->classes; + n->classes = c; - c->id = strdup(id); - if(!c->id) { - free(c); - return NULL; - } - c->hash = simple_hash(c->id); + c->id = strdupz(id); + c->hash = simple_hash(c->id); - if(parentid && *parentid) { - c->parentid = strdup(parentid); - c->parent_hash = simple_hash(c->parentid); - } + if(parentid && *parentid) { + c->parentid = strdupz(parentid); + c->parent_hash = simple_hash(c->parentid); + } - if(leafid && *leafid) { - c->leafid = strdup(leafid); - c->leaf_hash = simple_hash(c->leafid); - } + if(leafid && *leafid) { + c->leafid = strdupz(leafid); + c->leaf_hash = simple_hash(c->leafid); + } - tc_class_index_add(n, c); - } + tc_class_index_add(n, c); + } - c->seen = 1; + c->seen = 1; - return(c); + return(c); } static inline void tc_device_free(struct tc_device *n) { - if(n->next) n->next->prev = n->prev; - if(n->prev) n->prev->next = n->next; - if(tc_device_root == n) { - if(n->next) tc_device_root = n->next; - else tc_device_root = n->prev; - } + if(n->next) n->next->prev = n->prev; + if(n->prev) n->prev->next = n->next; + if(tc_device_root == n) { + if(n->next) tc_device_root = n->next; + else tc_device_root = n->prev; + } - tc_device_index_del(n); + tc_device_index_del(n); - while(n->classes) tc_class_free(n, n->classes); + while(n->classes) tc_class_free(n, n->classes); - if(n->id) free(n->id); - if(n->name) free(n->name); - if(n->family) free(n->family); - - free(n); + freez(n->id); + freez(n->name); + freez(n->family); + freez(n); } static inline void tc_device_free_all() { - while(tc_device_root) - tc_device_free(tc_device_root); + while(tc_device_root) + tc_device_free(tc_device_root); } #define MAX_WORDS 20 static inline int tc_space(char c) { - switch(c) { - case ' ': - case '\t': - case '\r': - case '\n': - return 1; - - default: - return 0; - } + switch(c) { + case ' ': + case '\t': + case '\r': + case '\n': + return 1; + + default: + return 0; + } } static inline void tc_split_words(char *str, char **words, int max_words) { - char *s = str; - int i = 0; + char *s = str; + int i = 0; - // skip all white space - while(tc_space(*s)) s++; + // skip all white space + while(tc_space(*s)) s++; - // store the first word - words[i++] = s; + // store the first word + words[i++] = s; - // while we have something - while(*s) { - // if it is a space - if(tc_space(*s)) { + // while we have something + while(*s) { + // if it is a space + if(unlikely(tc_space(*s))) { - // terminate the word - *s++ = '\0'; + // terminate the word + *s++ = '\0'; - // skip all white space - while(tc_space(*s)) s++; + // skip all white space + while(tc_space(*s)) s++; - // if we reached the end, stop - if(!*s) break; + // if we reached the end, stop + if(!*s) break; - // store the next word - if(i < max_words) words[i++] = s; - else break; - } - else s++; - } + // store the next word + if(i < max_words) words[i++] = s; + else break; + } + else s++; + } - // terminate the words - while(i < max_words) words[i++] = NULL; + // terminate the words + while(i < max_words) words[i++] = NULL; } pid_t tc_child_pid = 0; -void *tc_main(void *ptr) -{ - if(ptr) { ; } +void *tc_main(void *ptr) { + (void)ptr; - info("TC thread created with task id %d", gettid()); + info("TC thread created with task id %d", gettid()); - if(pthread_setcanceltype(PTHREAD_CANCEL_DEFERRED, NULL) != 0) - error("Cannot set pthread cancel type to DEFERRED."); + if(pthread_setcanceltype(PTHREAD_CANCEL_DEFERRED, NULL) != 0) + error("Cannot set pthread cancel type to DEFERRED."); - if(pthread_setcancelstate(PTHREAD_CANCEL_ENABLE, NULL) != 0) - error("Cannot set pthread cancel state to ENABLE."); + if(pthread_setcancelstate(PTHREAD_CANCEL_ENABLE, NULL) != 0) + error("Cannot set pthread cancel state to ENABLE."); - struct rusage thread; - RRDSET *stcpu = NULL, *sttime = NULL; + struct rusage thread; + RRDSET *stcpu = NULL, *sttime = NULL; - char buffer[TC_LINE_MAX+1] = ""; - char *words[MAX_WORDS] = { NULL }; + char buffer[TC_LINE_MAX+1] = ""; + char *words[MAX_WORDS] = { NULL }; - uint32_t BEGIN_HASH = simple_hash("BEGIN"); - uint32_t END_HASH = simple_hash("END"); - uint32_t CLASS_HASH = simple_hash("class"); - uint32_t SENT_HASH = simple_hash("Sent"); - uint32_t LENDED_HASH = simple_hash("lended:"); - uint32_t TOKENS_HASH = simple_hash("tokens:"); - uint32_t SETDEVICENAME_HASH = simple_hash("SETDEVICENAME"); - uint32_t SETDEVICEGROUP_HASH = simple_hash("SETDEVICEGROUP"); - uint32_t SETCLASSNAME_HASH = simple_hash("SETCLASSNAME"); - uint32_t WORKTIME_HASH = simple_hash("WORKTIME"); + uint32_t BEGIN_HASH = simple_hash("BEGIN"); + uint32_t END_HASH = simple_hash("END"); + uint32_t CLASS_HASH = simple_hash("class"); + uint32_t SENT_HASH = simple_hash("Sent"); + uint32_t LENDED_HASH = simple_hash("lended:"); + uint32_t TOKENS_HASH = simple_hash("tokens:"); + uint32_t SETDEVICENAME_HASH = simple_hash("SETDEVICENAME"); + uint32_t SETDEVICEGROUP_HASH = simple_hash("SETDEVICEGROUP"); + uint32_t SETCLASSNAME_HASH = simple_hash("SETCLASSNAME"); + uint32_t WORKTIME_HASH = simple_hash("WORKTIME"); #ifdef DETACH_PLUGINS_FROM_NETDATA - uint32_t MYPID_HASH = simple_hash("MYPID"); + uint32_t MYPID_HASH = simple_hash("MYPID"); #endif - uint32_t first_hash; - - for(;1;) { - if(unlikely(netdata_exit)) break; - - FILE *fp; - struct tc_device *device = NULL; - struct tc_class *class = NULL; - - snprintfz(buffer, TC_LINE_MAX, "exec %s %d", config_get("plugin:tc", "script to run to get tc values", PLUGINS_DIR "/tc-qos-helper.sh"), rrd_update_every); - debug(D_TC_LOOP, "executing '%s'", buffer); - // fp = popen(buffer, "r"); - fp = mypopen(buffer, &tc_child_pid); - if(!fp) { - error("TC: Cannot popen(\"%s\", \"r\").", buffer); - pthread_exit(NULL); - return NULL; - } - - while(fgets(buffer, TC_LINE_MAX, fp) != NULL) { - if(unlikely(netdata_exit)) break; - - buffer[TC_LINE_MAX] = '\0'; - // debug(D_TC_LOOP, "TC: read '%s'", buffer); - - tc_split_words(buffer, words, MAX_WORDS); - if(!words[0] || !*words[0]) { - // debug(D_TC_LOOP, "empty line"); - continue; - } - // else debug(D_TC_LOOP, "First word is '%s'", words[0]); - - first_hash = simple_hash(words[0]); - - if(device && first_hash == CLASS_HASH && strcmp(words[0], "class") == 0) { - // debug(D_TC_LOOP, "CLASS line on class id='%s', parent='%s', parentid='%s', leaf='%s', leafid='%s'", words[2], words[3], words[4], words[5], words[6]); - - // clear the last class - class = NULL; - - // words[1] : class type - // words[2] : N:XX - // words[3] : parent or root - if(words[1] && words[2] && words[3] && (strcmp(words[3], "parent") == 0 || strcmp(words[3], "root") == 0)) { - //char *type = words[1]; // the class: htb, fq_codel, etc - - // we are only interested for HTB classes - //if(strcmp(type, "htb") != 0) continue; - - char *id = words[2]; // the class major:minor - char *parent = words[3]; // 'parent' or 'root' - char *parentid = words[4]; // the parent's id - char *leaf = words[5]; // 'leaf' - char *leafid = words[6]; // leafid - - if(strcmp(parent, "root") == 0) { - parentid = NULL; - leafid = NULL; - } - else if(!leaf || strcmp(leaf, "leaf") != 0) - leafid = NULL; - - char leafbuf[20 + 1] = ""; - if(leafid && leafid[strlen(leafid) - 1] == ':') { - strncpyz(leafbuf, leafid, 20 - 1); - strcat(leafbuf, "1"); - leafid = leafbuf; - } - - class = tc_class_add(device, id, parentid, leafid); - } - } - else if(first_hash == END_HASH && strcmp(words[0], "END") == 0) { - // debug(D_TC_LOOP, "END line"); - - if(device) { - if(pthread_setcancelstate(PTHREAD_CANCEL_DISABLE, NULL) != 0) - error("Cannot set pthread cancel state to DISABLE."); - - tc_device_commit(device); - // tc_device_free(device); - device = NULL; - class = NULL; - - if(pthread_setcancelstate(PTHREAD_CANCEL_ENABLE, NULL) != 0) - error("Cannot set pthread cancel state to ENABLE."); - } - } - else if(first_hash == BEGIN_HASH && strcmp(words[0], "BEGIN") == 0) { - // debug(D_TC_LOOP, "BEGIN line on device '%s'", words[1]); - - if(device) { - // tc_device_free(device); - device = NULL; - class = NULL; - } - - if(words[1] && *words[1]) { - device = tc_device_create(words[1]); - class = NULL; - } - } - else if(device && class && first_hash == SENT_HASH && strcmp(words[0], "Sent") == 0) { - // debug(D_TC_LOOP, "SENT line '%s'", words[1]); - if(words[1] && *words[1]) { - class->bytes = strtoull(words[1], NULL, 10); - class->updated = 1; - } - - if(words[3] && *words[3]) - class->packets = strtoull(words[3], NULL, 10); - - if(words[6] && *words[6]) - class->dropped = strtoull(words[6], NULL, 10); - - if(words[8] && *words[8]) - class->overlimits = strtoull(words[8], NULL, 10); - - if(words[10] && *words[10]) - class->requeues = strtoull(words[8], NULL, 10); - } - else if(device && class && class->updated && first_hash == LENDED_HASH && strcmp(words[0], "lended:") == 0) { - // debug(D_TC_LOOP, "LENDED line '%s'", words[1]); - if(words[1] && *words[1]) - class->lended = strtoull(words[1], NULL, 10); - - if(words[3] && *words[3]) - class->borrowed = strtoull(words[3], NULL, 10); - - if(words[5] && *words[5]) - class->giants = strtoull(words[5], NULL, 10); - } - else if(device && class && class->updated && first_hash == TOKENS_HASH && strcmp(words[0], "tokens:") == 0) { - // debug(D_TC_LOOP, "TOKENS line '%s'", words[1]); - if(words[1] && *words[1]) - class->tokens = strtoull(words[1], NULL, 10); - - if(words[3] && *words[3]) - class->ctokens = strtoull(words[3], NULL, 10); - } - else if(device && first_hash == SETDEVICENAME_HASH && strcmp(words[0], "SETDEVICENAME") == 0) { - // debug(D_TC_LOOP, "SETDEVICENAME line '%s'", words[1]); - if(words[1] && *words[1]) tc_device_set_device_name(device, words[1]); - } - else if(device && first_hash == SETDEVICEGROUP_HASH && strcmp(words[0], "SETDEVICEGROUP") == 0) { - // debug(D_TC_LOOP, "SETDEVICEGROUP line '%s'", words[1]); - if(words[1] && *words[1]) tc_device_set_device_family(device, words[1]); - } - else if(device && first_hash == SETCLASSNAME_HASH && strcmp(words[0], "SETCLASSNAME") == 0) { - // debug(D_TC_LOOP, "SETCLASSNAME line '%s' '%s'", words[1], words[2]); - char *id = words[1]; - char *path = words[2]; - if(id && *id && path && *path) tc_device_set_class_name(device, id, path); - } - else if(first_hash == WORKTIME_HASH && strcmp(words[0], "WORKTIME") == 0) { - // debug(D_TC_LOOP, "WORKTIME line '%s' '%s'", words[1], words[2]); - getrusage(RUSAGE_THREAD, &thread); - - if(!stcpu) stcpu = rrdset_find("netdata.plugin_tc_cpu"); - if(!stcpu) { - stcpu = rrdset_create("netdata", "plugin_tc_cpu", NULL, "tc.helper", NULL, "NetData TC CPU usage", "milliseconds/s", 135000, rrd_update_every, RRDSET_TYPE_STACKED); - rrddim_add(stcpu, "user", NULL, 1, 1000, RRDDIM_INCREMENTAL); - rrddim_add(stcpu, "system", NULL, 1, 1000, RRDDIM_INCREMENTAL); - } - else rrdset_next(stcpu); - - rrddim_set(stcpu, "user" , thread.ru_utime.tv_sec * 1000000ULL + thread.ru_utime.tv_usec); - rrddim_set(stcpu, "system", thread.ru_stime.tv_sec * 1000000ULL + thread.ru_stime.tv_usec); - rrdset_done(stcpu); - - if(!sttime) stcpu = rrdset_find("netdata.plugin_tc_time"); - if(!sttime) { - sttime = rrdset_create("netdata", "plugin_tc_time", NULL, "tc.helper", NULL, "NetData TC script execution", "milliseconds/run", 135001, rrd_update_every, RRDSET_TYPE_AREA); - rrddim_add(sttime, "run_time", "run time", 1, 1, RRDDIM_ABSOLUTE); - } - else rrdset_next(sttime); - - rrddim_set(sttime, "run_time", atoll(words[1])); - rrdset_done(sttime); - - } + uint32_t first_hash; + + snprintfz(buffer, TC_LINE_MAX, "%s/tc-qos-helper.sh", config_get("plugins", "plugins directory", PLUGINS_DIR)); + char *tc_script = config_get("plugin:tc", "script to run to get tc values", buffer); + + for(;1;) { + if(unlikely(netdata_exit)) break; + + FILE *fp; + struct tc_device *device = NULL; + struct tc_class *class = NULL; + + snprintfz(buffer, TC_LINE_MAX, "exec %s %d", tc_script, rrd_update_every); + debug(D_TC_LOOP, "executing '%s'", buffer); + + fp = mypopen(buffer, &tc_child_pid); + if(unlikely(!fp)) { + error("TC: Cannot popen(\"%s\", \"r\").", buffer); + pthread_exit(NULL); + return NULL; + } + + while(fgets(buffer, TC_LINE_MAX, fp) != NULL) { + if(unlikely(netdata_exit)) break; + + buffer[TC_LINE_MAX] = '\0'; + // debug(D_TC_LOOP, "TC: read '%s'", buffer); + + tc_split_words(buffer, words, MAX_WORDS); + + if(unlikely(!words[0] || !*words[0])) { + // debug(D_TC_LOOP, "empty line"); + continue; + } + // else debug(D_TC_LOOP, "First word is '%s'", words[0]); + + first_hash = simple_hash(words[0]); + + if(unlikely(device && first_hash == CLASS_HASH && strcmp(words[0], "class") == 0)) { + // debug(D_TC_LOOP, "CLASS line on class id='%s', parent='%s', parentid='%s', leaf='%s', leafid='%s'", words[2], words[3], words[4], words[5], words[6]); + + // words[1] : class type + // words[2] : N:XX + // words[3] : parent or root + if(likely(words[1] && words[2] && words[3] && (strcmp(words[3], "parent") == 0 || strcmp(words[3], "root") == 0))) { + //char *type = words[1]; // the class: htb, fq_codel, etc + + // we are only interested for HTB classes + //if(strcmp(type, "htb") != 0) continue; + + char *id = words[2]; // the class major:minor + char *parent = words[3]; // 'parent' or 'root' + char *parentid = words[4]; // the parent's id + char *leaf = words[5]; // 'leaf' + char *leafid = words[6]; // leafid + + if(strcmp(parent, "root") == 0) { + parentid = NULL; + leafid = NULL; + } + else if(!leaf || strcmp(leaf, "leaf") != 0) + leafid = NULL; + + char leafbuf[20 + 1] = ""; + if(leafid && leafid[strlen(leafid) - 1] == ':') { + strncpyz(leafbuf, leafid, 20 - 1); + strcat(leafbuf, "1"); + leafid = leafbuf; + } + + class = tc_class_add(device, id, parentid, leafid); + } + else { + // clear the last class + class = NULL; + } + } + else if(unlikely(first_hash == END_HASH && strcmp(words[0], "END") == 0)) { + // debug(D_TC_LOOP, "END line"); + + if(likely(device)) { + if(pthread_setcancelstate(PTHREAD_CANCEL_DISABLE, NULL) != 0) + error("Cannot set pthread cancel state to DISABLE."); + + tc_device_commit(device); + // tc_device_free(device); + + if(pthread_setcancelstate(PTHREAD_CANCEL_ENABLE, NULL) != 0) + error("Cannot set pthread cancel state to ENABLE."); + } + + device = NULL; + class = NULL; + } + else if(unlikely(first_hash == BEGIN_HASH && strcmp(words[0], "BEGIN") == 0)) { + // debug(D_TC_LOOP, "BEGIN line on device '%s'", words[1]); + + if(likely(words[1] && *words[1])) { + device = tc_device_create(words[1]); + } + else { + // tc_device_free(device); + device = NULL; + } + + class = NULL; + } + else if(unlikely(device && class && first_hash == SENT_HASH && strcmp(words[0], "Sent") == 0)) { + // debug(D_TC_LOOP, "SENT line '%s'", words[1]); + if(likely(words[1] && *words[1])) { + class->bytes = strtoull(words[1], NULL, 10); + class->updated = 1; + } + else { + class->updated = 0; + } + + if(likely(words[3] && *words[3])) + class->packets = strtoull(words[3], NULL, 10); + + if(likely(words[6] && *words[6])) + class->dropped = strtoull(words[6], NULL, 10); + + if(likely(words[8] && *words[8])) + class->overlimits = strtoull(words[8], NULL, 10); + + if(likely(words[10] && *words[10])) + class->requeues = strtoull(words[8], NULL, 10); + } + else if(unlikely(device && class && class->updated && first_hash == LENDED_HASH && strcmp(words[0], "lended:") == 0)) { + // debug(D_TC_LOOP, "LENDED line '%s'", words[1]); + if(likely(words[1] && *words[1])) + class->lended = strtoull(words[1], NULL, 10); + + if(likely(words[3] && *words[3])) + class->borrowed = strtoull(words[3], NULL, 10); + + if(likely(words[5] && *words[5])) + class->giants = strtoull(words[5], NULL, 10); + } + else if(unlikely(device && class && class->updated && first_hash == TOKENS_HASH && strcmp(words[0], "tokens:") == 0)) { + // debug(D_TC_LOOP, "TOKENS line '%s'", words[1]); + if(likely(words[1] && *words[1])) + class->tokens = strtoull(words[1], NULL, 10); + + if(likely(words[3] && *words[3])) + class->ctokens = strtoull(words[3], NULL, 10); + } + else if(unlikely(device && first_hash == SETDEVICENAME_HASH && strcmp(words[0], "SETDEVICENAME") == 0)) { + // debug(D_TC_LOOP, "SETDEVICENAME line '%s'", words[1]); + if(likely(words[1] && *words[1])) + tc_device_set_device_name(device, words[1]); + } + else if(unlikely(device && first_hash == SETDEVICEGROUP_HASH && strcmp(words[0], "SETDEVICEGROUP") == 0)) { + // debug(D_TC_LOOP, "SETDEVICEGROUP line '%s'", words[1]); + if(likely(words[1] && *words[1])) + tc_device_set_device_family(device, words[1]); + } + else if(unlikely(device && first_hash == SETCLASSNAME_HASH && strcmp(words[0], "SETCLASSNAME") == 0)) { + // debug(D_TC_LOOP, "SETCLASSNAME line '%s' '%s'", words[1], words[2]); + char *id = words[1]; + char *path = words[2]; + if(likely(id && *id && path && *path)) + tc_device_set_class_name(device, id, path); + } + else if(unlikely(first_hash == WORKTIME_HASH && strcmp(words[0], "WORKTIME") == 0)) { + // debug(D_TC_LOOP, "WORKTIME line '%s' '%s'", words[1], words[2]); + getrusage(RUSAGE_THREAD, &thread); + + if(unlikely(!stcpu)) stcpu = rrdset_find("netdata.plugin_tc_cpu"); + if(unlikely(!stcpu)) { + stcpu = rrdset_create("netdata", "plugin_tc_cpu", NULL, "tc.helper", NULL, "NetData TC CPU usage", "milliseconds/s", 135000, rrd_update_every, RRDSET_TYPE_STACKED); + rrddim_add(stcpu, "user", NULL, 1, 1000, RRDDIM_INCREMENTAL); + rrddim_add(stcpu, "system", NULL, 1, 1000, RRDDIM_INCREMENTAL); + } + else rrdset_next(stcpu); + + rrddim_set(stcpu, "user" , thread.ru_utime.tv_sec * 1000000ULL + thread.ru_utime.tv_usec); + rrddim_set(stcpu, "system", thread.ru_stime.tv_sec * 1000000ULL + thread.ru_stime.tv_usec); + rrdset_done(stcpu); + + if(unlikely(!sttime)) stcpu = rrdset_find("netdata.plugin_tc_time"); + if(unlikely(!sttime)) { + sttime = rrdset_create("netdata", "plugin_tc_time", NULL, "tc.helper", NULL, "NetData TC script execution", "milliseconds/run", 135001, rrd_update_every, RRDSET_TYPE_AREA); + rrddim_add(sttime, "run_time", "run time", 1, 1, RRDDIM_ABSOLUTE); + } + else rrdset_next(sttime); + + rrddim_set(sttime, "run_time", atoll(words[1])); + rrdset_done(sttime); + + } #ifdef DETACH_PLUGINS_FROM_NETDATA - else if(first_hash == MYPID_HASH && (strcmp(words[0], "MYPID") == 0)) { - // debug(D_TC_LOOP, "MYPID line '%s'", words[1]); - char *id = words[1]; - pid_t pid = atol(id); + else if(unlikely(first_hash == MYPID_HASH && (strcmp(words[0], "MYPID") == 0))) { + // debug(D_TC_LOOP, "MYPID line '%s'", words[1]); + char *id = words[1]; + pid_t pid = atol(id); - if(pid) tc_child_pid = pid; + if(likely(pid)) tc_child_pid = pid; - debug(D_TC_LOOP, "TC: Child PID is %d.", tc_child_pid); - } + debug(D_TC_LOOP, "TC: Child PID is %d.", tc_child_pid); + } #endif - //else { - // debug(D_TC_LOOP, "IGNORED line"); - //} - } - // fgets() failed or loop broke - int code = mypclose(fp, tc_child_pid); - tc_child_pid = 0; - - if(device) { - // tc_device_free(device); - device = NULL; - class = NULL; - } - - if(netdata_exit) { - tc_device_free_all(); - pthread_exit(NULL); - return NULL; - } - - if(code == 1 || code == 127) { - // 1 = DISABLE - // 127 = cannot even run it - error("TC: tc-qos-helper.sh exited with code %d. Disabling it.", code); - - tc_device_free_all(); - pthread_exit(NULL); - return NULL; - } - - sleep((unsigned int) rrd_update_every); - } - - pthread_exit(NULL); - return NULL; + //else { + // debug(D_TC_LOOP, "IGNORED line"); + //} + } + + // fgets() failed or loop broke + int code = mypclose(fp, tc_child_pid); + tc_child_pid = 0; + + if(unlikely(device)) { + // tc_device_free(device); + device = NULL; + class = NULL; + } + + if(unlikely(netdata_exit)) { + tc_device_free_all(); + pthread_exit(NULL); + return NULL; + } + + if(code == 1 || code == 127) { + // 1 = DISABLE + // 127 = cannot even run it + error("TC: tc-qos-helper.sh exited with code %d. Disabling it.", code); + + tc_device_free_all(); + pthread_exit(NULL); + return NULL; + } + + sleep((unsigned int) rrd_update_every); + } + + pthread_exit(NULL); + return NULL; } diff --git a/src/plugins_d.c b/src/plugins_d.c index 0ccbd36e4..627cc90e5 100644 --- a/src/plugins_d.c +++ b/src/plugins_d.c @@ -1,531 +1,547 @@ -#ifdef HAVE_CONFIG_H -#include <config.h> -#endif -#include <sys/types.h> -#include <dirent.h> -#include <pthread.h> -#include <string.h> -#include <stdio.h> -#include <stdlib.h> -#include <signal.h> - -#include "main.h" #include "common.h" -#include "appconfig.h" -#include "log.h" -#include "rrd.h" -#include "popen.h" -#include "plugins_d.h" -#include "../config.h" struct plugind *pluginsd_root = NULL; #define MAX_WORDS 20 static inline int pluginsd_space(char c) { - switch(c) { - case ' ': - case '\t': - case '\r': - case '\n': - case '=': - return 1; - - default: - return 0; - } + switch(c) { + case ' ': + case '\t': + case '\r': + case '\n': + case '=': + return 1; + + default: + return 0; + } } static int pluginsd_split_words(char *str, char **words, int max_words) { - char *s = str, quote = 0; - int i = 0, j; - - // skip all white space - while(unlikely(pluginsd_space(*s))) s++; - - // check for quote - if(unlikely(*s == '\'' || *s == '"')) { - quote = *s; // remember the quote - s++; // skip the quote - } - - // store the first word - words[i++] = s; - - // while we have something - while(likely(*s)) { - // if it is escape - if(unlikely(*s == '\\' && s[1])) { - s += 2; - continue; - } - - // if it is quote - else if(unlikely(*s == quote)) { - quote = 0; - *s = ' '; - continue; - } - - // if it is a space - else if(unlikely(quote == 0 && pluginsd_space(*s))) { - - // terminate the word - *s++ = '\0'; - - // skip all white space - while(likely(pluginsd_space(*s))) s++; - - // check for quote - if(unlikely(*s == '\'' || *s == '"')) { - quote = *s; // remember the quote - s++; // skip the quote - } - - // if we reached the end, stop - if(unlikely(!*s)) break; - - // store the next word - if(likely(i < max_words)) words[i++] = s; - else break; - } - - // anything else - else s++; - } - - // terminate the words - j = i; - while(likely(j < max_words)) words[j++] = NULL; - - return i; + char *s = str, quote = 0; + int i = 0, j; + + // skip all white space + while(unlikely(pluginsd_space(*s))) s++; + + // check for quote + if(unlikely(*s == '\'' || *s == '"')) { + quote = *s; // remember the quote + s++; // skip the quote + } + + // store the first word + words[i++] = s; + + // while we have something + while(likely(*s)) { + // if it is escape + if(unlikely(*s == '\\' && s[1])) { + s += 2; + continue; + } + + // if it is quote + else if(unlikely(*s == quote)) { + quote = 0; + *s = ' '; + continue; + } + + // if it is a space + else if(unlikely(quote == 0 && pluginsd_space(*s))) { + + // terminate the word + *s++ = '\0'; + + // skip all white space + while(likely(pluginsd_space(*s))) s++; + + // check for quote + if(unlikely(*s == '\'' || *s == '"')) { + quote = *s; // remember the quote + s++; // skip the quote + } + + // if we reached the end, stop + if(unlikely(!*s)) break; + + // store the next word + if(likely(i < max_words)) words[i++] = s; + else break; + } + + // anything else + else s++; + } + + // terminate the words + j = i; + while(likely(j < max_words)) words[j++] = NULL; + + return i; } void *pluginsd_worker_thread(void *arg) { - struct plugind *cd = (struct plugind *)arg; - char line[PLUGINSD_LINE_MAX + 1]; + struct plugind *cd = (struct plugind *)arg; + char line[PLUGINSD_LINE_MAX + 1]; #ifdef DETACH_PLUGINS_FROM_NETDATA - unsigned long long usec = 0, susec = 0; - struct timeval last = {0, 0} , now = {0, 0}; + unsigned long long usec = 0, susec = 0; + struct timeval last = {0, 0} , now = {0, 0}; #endif - char *words[MAX_WORDS] = { NULL }; - uint32_t SET_HASH = simple_hash("SET"); - uint32_t BEGIN_HASH = simple_hash("BEGIN"); - uint32_t END_HASH = simple_hash("END"); - uint32_t FLUSH_HASH = simple_hash("FLUSH"); - uint32_t CHART_HASH = simple_hash("CHART"); - uint32_t DIMENSION_HASH = simple_hash("DIMENSION"); - uint32_t DISABLE_HASH = simple_hash("DISABLE"); + char *words[MAX_WORDS] = { NULL }; + uint32_t SET_HASH = simple_hash("SET"); + uint32_t BEGIN_HASH = simple_hash("BEGIN"); + uint32_t END_HASH = simple_hash("END"); + uint32_t FLUSH_HASH = simple_hash("FLUSH"); + uint32_t CHART_HASH = simple_hash("CHART"); + uint32_t DIMENSION_HASH = simple_hash("DIMENSION"); + uint32_t DISABLE_HASH = simple_hash("DISABLE"); #ifdef DETACH_PLUGINS_FROM_NETDATA - uint32_t MYPID_HASH = simple_hash("MYPID"); - uint32_t STOPPING_WAKE_ME_UP_PLEASE_HASH = simple_hash("STOPPING_WAKE_ME_UP_PLEASE"); + uint32_t MYPID_HASH = simple_hash("MYPID"); + uint32_t STOPPING_WAKE_ME_UP_PLEASE_HASH = simple_hash("STOPPING_WAKE_ME_UP_PLEASE"); #endif - while(likely(1)) { - if(unlikely(netdata_exit)) break; - - FILE *fp = mypopen(cd->cmd, &cd->pid); - if(unlikely(!fp)) { - error("Cannot popen(\"%s\", \"r\").", cd->cmd); - break; - } - - info("PLUGINSD: '%s' running on pid %d", cd->fullfilename, cd->pid); - - RRDSET *st = NULL; - unsigned long long count = 0; - char *s; - uint32_t hash; - - while(likely(fgets(line, PLUGINSD_LINE_MAX, fp) != NULL)) { - if(unlikely(netdata_exit)) break; - - line[PLUGINSD_LINE_MAX] = '\0'; - - // debug(D_PLUGINSD, "PLUGINSD: %s: %s", cd->filename, line); - - int w = pluginsd_split_words(line, words, MAX_WORDS); - s = words[0]; - if(unlikely(!s || !*s || !w)) { - // debug(D_PLUGINSD, "PLUGINSD: empty line"); - continue; - } - - // debug(D_PLUGINSD, "PLUGINSD: words 0='%s' 1='%s' 2='%s' 3='%s' 4='%s' 5='%s' 6='%s' 7='%s' 8='%s' 9='%s'", words[0], words[1], words[2], words[3], words[4], words[5], words[6], words[7], words[8], words[9]); - - hash = simple_hash(s); - - if(likely(hash == SET_HASH && !strcmp(s, "SET"))) { - char *dimension = words[1]; - char *value = words[2]; - - if(unlikely(!dimension || !*dimension)) { - error("PLUGINSD: '%s' is requesting a SET on chart '%s', without a dimension. Disabling it.", cd->fullfilename, st->id); - cd->enabled = 0; - killpid(cd->pid, SIGTERM); - break; - } - - if(unlikely(!value || !*value)) value = NULL; - - if(unlikely(!st)) { - error("PLUGINSD: '%s' is requesting a SET on dimension %s with value %s, without a BEGIN. Disabling it.", cd->fullfilename, dimension, value?value:"<nothing>"); - cd->enabled = 0; - killpid(cd->pid, SIGTERM); - break; - } - - if(unlikely(st->debug)) debug(D_PLUGINSD, "PLUGINSD: '%s' is setting dimension %s/%s to %s", cd->fullfilename, st->id, dimension, value?value:"<nothing>"); - - if(value) rrddim_set(st, dimension, strtoll(value, NULL, 0)); - - count++; - } - else if(likely(hash == BEGIN_HASH && !strcmp(s, "BEGIN"))) { - char *id = words[1]; - char *microseconds_txt = words[2]; - - if(unlikely(!id)) { - error("PLUGINSD: '%s' is requesting a BEGIN without a chart id. Disabling it.", cd->fullfilename); - cd->enabled = 0; - killpid(cd->pid, SIGTERM); - break; - } - - st = rrdset_find(id); - if(unlikely(!st)) { - error("PLUGINSD: '%s' is requesting a BEGIN on chart '%s', which does not exist. Disabling it.", cd->fullfilename, id); - cd->enabled = 0; - killpid(cd->pid, SIGTERM); - break; - } - - if(likely(st->counter_done)) { - unsigned long long microseconds = 0; - if(microseconds_txt && *microseconds_txt) microseconds = strtoull(microseconds_txt, NULL, 10); - if(microseconds) rrdset_next_usec(st, microseconds); - else rrdset_next_plugins(st); - } - } - else if(likely(hash == END_HASH && !strcmp(s, "END"))) { - if(unlikely(!st)) { - error("PLUGINSD: '%s' is requesting an END, without a BEGIN. Disabling it.", cd->fullfilename); - cd->enabled = 0; - killpid(cd->pid, SIGTERM); - break; - } - - if(unlikely(st->debug)) debug(D_PLUGINSD, "PLUGINSD: '%s' is requesting an END on chart %s", cd->fullfilename, st->id); - - rrdset_done(st); - st = NULL; - } - else if(likely(hash == FLUSH_HASH && !strcmp(s, "FLUSH"))) { - debug(D_PLUGINSD, "PLUGINSD: '%s' is requesting a FLUSH", cd->fullfilename); - st = NULL; - } - else if(likely(hash == CHART_HASH && !strcmp(s, "CHART"))) { - int noname = 0; - st = NULL; - - if((words[1]) != NULL && (words[2]) != NULL && strcmp(words[1], words[2]) == 0) - noname = 1; - - char *type = words[1]; - char *id = NULL; - if(likely(type)) { - id = strchr(type, '.'); - if(likely(id)) { *id = '\0'; id++; } - } - char *name = words[2]; - char *title = words[3]; - char *units = words[4]; - char *family = words[5]; - char *context = words[6]; - char *chart = words[7]; - char *priority_s = words[8]; - char *update_every_s = words[9]; - - if(unlikely(!type || !*type || !id || !*id)) { - error("PLUGINSD: '%s' is requesting a CHART, without a type.id. Disabling it.", cd->fullfilename); - cd->enabled = 0; - killpid(cd->pid, SIGTERM); - break; - } - - int priority = 1000; - if(likely(priority_s)) priority = atoi(priority_s); - - int update_every = cd->update_every; - if(likely(update_every_s)) update_every = atoi(update_every_s); - if(unlikely(!update_every)) update_every = cd->update_every; - - int chart_type = RRDSET_TYPE_LINE; - if(unlikely(chart)) chart_type = rrdset_type_id(chart); - - if(unlikely(noname || !name || !*name || strcasecmp(name, "NULL") == 0 || strcasecmp(name, "(NULL)") == 0)) name = NULL; - if(unlikely(!family || !*family)) family = NULL; - if(unlikely(!context || !*context)) context = NULL; - - st = rrdset_find_bytype(type, id); - if(unlikely(!st)) { - debug(D_PLUGINSD, "PLUGINSD: Creating chart type='%s', id='%s', name='%s', family='%s', context='%s', chart='%s', priority=%d, update_every=%d" - , type, id - , name?name:"" - , family?family:"" - , context?context:"" - , rrdset_type_name(chart_type) - , priority - , update_every - ); - - st = rrdset_create(type, id, name, family, context, title, units, priority, update_every, chart_type); - cd->update_every = update_every; - } - else debug(D_PLUGINSD, "PLUGINSD: Chart '%s' already exists. Not adding it again.", st->id); - } - else if(likely(hash == DIMENSION_HASH && !strcmp(s, "DIMENSION"))) { - char *id = words[1]; - char *name = words[2]; - char *algorithm = words[3]; - char *multiplier_s = words[4]; - char *divisor_s = words[5]; - char *options = words[6]; - - if(unlikely(!id || !*id)) { - error("PLUGINSD: '%s' is requesting a DIMENSION, without an id. Disabling it.", cd->fullfilename); - cd->enabled = 0; - killpid(cd->pid, SIGTERM); - break; - } - - if(unlikely(!st)) { - error("PLUGINSD: '%s' is requesting a DIMENSION, without a CHART. Disabling it.", cd->fullfilename); - cd->enabled = 0; - killpid(cd->pid, SIGTERM); - break; - } - - long multiplier = 1; - if(multiplier_s && *multiplier_s) multiplier = strtol(multiplier_s, NULL, 0); - if(unlikely(!multiplier)) multiplier = 1; - - long divisor = 1; - if(likely(divisor_s && *divisor_s)) divisor = strtol(divisor_s, NULL, 0); - if(unlikely(!divisor)) divisor = 1; - - if(unlikely(!algorithm || !*algorithm)) algorithm = "absolute"; - - if(unlikely(st->debug)) debug(D_PLUGINSD, "PLUGINSD: Creating dimension in chart %s, id='%s', name='%s', algorithm='%s', multiplier=%ld, divisor=%ld, hidden='%s'" - , st->id - , id - , name?name:"" - , rrddim_algorithm_name(rrddim_algorithm_id(algorithm)) - , multiplier - , divisor - , options?options:"" - ); - - RRDDIM *rd = rrddim_find(st, id); - if(unlikely(!rd)) { - rd = rrddim_add(st, id, name, multiplier, divisor, rrddim_algorithm_id(algorithm)); - rd->flags = 0x00000000; - if(options && *options) { - if(strstr(options, "hidden") != NULL) rd->flags |= RRDDIM_FLAG_HIDDEN; - if(strstr(options, "noreset") != NULL) rd->flags |= RRDDIM_FLAG_DONT_DETECT_RESETS_OR_OVERFLOWS; - if(strstr(options, "nooverflow") != NULL) rd->flags |= RRDDIM_FLAG_DONT_DETECT_RESETS_OR_OVERFLOWS; - } - } - else if(unlikely(st->debug)) debug(D_PLUGINSD, "PLUGINSD: dimension %s/%s already exists. Not adding it again.", st->id, id); - } - else if(unlikely(hash == DISABLE_HASH && !strcmp(s, "DISABLE"))) { - error("PLUGINSD: '%s' called DISABLE. Disabling it.", cd->fullfilename); - cd->enabled = 0; - killpid(cd->pid, SIGTERM); - break; - } + size_t count = 0; + + while(likely(1)) { + if(unlikely(netdata_exit)) break; + + FILE *fp = mypopen(cd->cmd, &cd->pid); + if(unlikely(!fp)) { + error("Cannot popen(\"%s\", \"r\").", cd->cmd); + break; + } + + info("PLUGINSD: '%s' running on pid %d", cd->fullfilename, cd->pid); + + RRDSET *st = NULL; + char *s; + uint32_t hash; + + while(likely(fgets(line, PLUGINSD_LINE_MAX, fp) != NULL)) { + if(unlikely(netdata_exit)) break; + + line[PLUGINSD_LINE_MAX] = '\0'; + + // debug(D_PLUGINSD, "PLUGINSD: %s: %s", cd->filename, line); + + int w = pluginsd_split_words(line, words, MAX_WORDS); + s = words[0]; + if(unlikely(!s || !*s || !w)) { + // debug(D_PLUGINSD, "PLUGINSD: empty line"); + continue; + } + + // debug(D_PLUGINSD, "PLUGINSD: words 0='%s' 1='%s' 2='%s' 3='%s' 4='%s' 5='%s' 6='%s' 7='%s' 8='%s' 9='%s'", words[0], words[1], words[2], words[3], words[4], words[5], words[6], words[7], words[8], words[9]); + + hash = simple_hash(s); + + if(likely(hash == SET_HASH && !strcmp(s, "SET"))) { + char *dimension = words[1]; + char *value = words[2]; + + if(unlikely(!dimension || !*dimension)) { + error("PLUGINSD: '%s' is requesting a SET on chart '%s', without a dimension. Disabling it.", cd->fullfilename, st->id); + cd->enabled = 0; + killpid(cd->pid, SIGTERM); + break; + } + + if(unlikely(!value || !*value)) value = NULL; + + if(unlikely(!st)) { + error("PLUGINSD: '%s' is requesting a SET on dimension %s with value %s, without a BEGIN. Disabling it.", cd->fullfilename, dimension, value?value:"<nothing>"); + cd->enabled = 0; + killpid(cd->pid, SIGTERM); + break; + } + + if(unlikely(st->debug)) debug(D_PLUGINSD, "PLUGINSD: '%s' is setting dimension %s/%s to %s", cd->fullfilename, st->id, dimension, value?value:"<nothing>"); + + if(value) rrddim_set(st, dimension, strtoll(value, NULL, 0)); + } + else if(likely(hash == BEGIN_HASH && !strcmp(s, "BEGIN"))) { + char *id = words[1]; + char *microseconds_txt = words[2]; + + if(unlikely(!id)) { + error("PLUGINSD: '%s' is requesting a BEGIN without a chart id. Disabling it.", cd->fullfilename); + cd->enabled = 0; + killpid(cd->pid, SIGTERM); + break; + } + + st = rrdset_find(id); + if(unlikely(!st)) { + error("PLUGINSD: '%s' is requesting a BEGIN on chart '%s', which does not exist. Disabling it.", cd->fullfilename, id); + cd->enabled = 0; + killpid(cd->pid, SIGTERM); + break; + } + + if(likely(st->counter_done)) { + unsigned long long microseconds = 0; + if(microseconds_txt && *microseconds_txt) microseconds = strtoull(microseconds_txt, NULL, 10); + if(microseconds) rrdset_next_usec(st, microseconds); + else rrdset_next_plugins(st); + } + } + else if(likely(hash == END_HASH && !strcmp(s, "END"))) { + if(unlikely(!st)) { + error("PLUGINSD: '%s' is requesting an END, without a BEGIN. Disabling it.", cd->fullfilename); + cd->enabled = 0; + killpid(cd->pid, SIGTERM); + break; + } + + if(unlikely(st->debug)) debug(D_PLUGINSD, "PLUGINSD: '%s' is requesting an END on chart %s", cd->fullfilename, st->id); + + rrdset_done(st); + st = NULL; + + count++; + } + else if(likely(hash == FLUSH_HASH && !strcmp(s, "FLUSH"))) { + debug(D_PLUGINSD, "PLUGINSD: '%s' is requesting a FLUSH", cd->fullfilename); + st = NULL; + } + else if(likely(hash == CHART_HASH && !strcmp(s, "CHART"))) { + int noname = 0; + st = NULL; + + if((words[1]) != NULL && (words[2]) != NULL && strcmp(words[1], words[2]) == 0) + noname = 1; + + char *type = words[1]; + char *id = NULL; + if(likely(type)) { + id = strchr(type, '.'); + if(likely(id)) { *id = '\0'; id++; } + } + char *name = words[2]; + char *title = words[3]; + char *units = words[4]; + char *family = words[5]; + char *context = words[6]; + char *chart = words[7]; + char *priority_s = words[8]; + char *update_every_s = words[9]; + + if(unlikely(!type || !*type || !id || !*id)) { + error("PLUGINSD: '%s' is requesting a CHART, without a type.id. Disabling it.", cd->fullfilename); + cd->enabled = 0; + killpid(cd->pid, SIGTERM); + break; + } + + int priority = 1000; + if(likely(priority_s)) priority = atoi(priority_s); + + int update_every = cd->update_every; + if(likely(update_every_s)) update_every = atoi(update_every_s); + if(unlikely(!update_every)) update_every = cd->update_every; + + int chart_type = RRDSET_TYPE_LINE; + if(unlikely(chart)) chart_type = rrdset_type_id(chart); + + if(unlikely(noname || !name || !*name || strcasecmp(name, "NULL") == 0 || strcasecmp(name, "(NULL)") == 0)) name = NULL; + if(unlikely(!family || !*family)) family = NULL; + if(unlikely(!context || !*context)) context = NULL; + + st = rrdset_find_bytype(type, id); + if(unlikely(!st)) { + debug(D_PLUGINSD, "PLUGINSD: Creating chart type='%s', id='%s', name='%s', family='%s', context='%s', chart='%s', priority=%d, update_every=%d" + , type, id + , name?name:"" + , family?family:"" + , context?context:"" + , rrdset_type_name(chart_type) + , priority + , update_every + ); + + st = rrdset_create(type, id, name, family, context, title, units, priority, update_every, chart_type); + cd->update_every = update_every; + } + else debug(D_PLUGINSD, "PLUGINSD: Chart '%s' already exists. Not adding it again.", st->id); + } + else if(likely(hash == DIMENSION_HASH && !strcmp(s, "DIMENSION"))) { + char *id = words[1]; + char *name = words[2]; + char *algorithm = words[3]; + char *multiplier_s = words[4]; + char *divisor_s = words[5]; + char *options = words[6]; + + if(unlikely(!id || !*id)) { + error("PLUGINSD: '%s' is requesting a DIMENSION, without an id. Disabling it.", cd->fullfilename); + cd->enabled = 0; + killpid(cd->pid, SIGTERM); + break; + } + + if(unlikely(!st)) { + error("PLUGINSD: '%s' is requesting a DIMENSION, without a CHART. Disabling it.", cd->fullfilename); + cd->enabled = 0; + killpid(cd->pid, SIGTERM); + break; + } + + long multiplier = 1; + if(multiplier_s && *multiplier_s) multiplier = strtol(multiplier_s, NULL, 0); + if(unlikely(!multiplier)) multiplier = 1; + + long divisor = 1; + if(likely(divisor_s && *divisor_s)) divisor = strtol(divisor_s, NULL, 0); + if(unlikely(!divisor)) divisor = 1; + + if(unlikely(!algorithm || !*algorithm)) algorithm = "absolute"; + + if(unlikely(st->debug)) debug(D_PLUGINSD, "PLUGINSD: Creating dimension in chart %s, id='%s', name='%s', algorithm='%s', multiplier=%ld, divisor=%ld, hidden='%s'" + , st->id + , id + , name?name:"" + , rrddim_algorithm_name(rrddim_algorithm_id(algorithm)) + , multiplier + , divisor + , options?options:"" + ); + + RRDDIM *rd = rrddim_find(st, id); + if(unlikely(!rd)) { + rd = rrddim_add(st, id, name, multiplier, divisor, rrddim_algorithm_id(algorithm)); + rd->flags = 0x00000000; + if(options && *options) { + if(strstr(options, "hidden") != NULL) rd->flags |= RRDDIM_FLAG_HIDDEN; + if(strstr(options, "noreset") != NULL) rd->flags |= RRDDIM_FLAG_DONT_DETECT_RESETS_OR_OVERFLOWS; + if(strstr(options, "nooverflow") != NULL) rd->flags |= RRDDIM_FLAG_DONT_DETECT_RESETS_OR_OVERFLOWS; + } + } + else if(unlikely(st->debug)) debug(D_PLUGINSD, "PLUGINSD: dimension %s/%s already exists. Not adding it again.", st->id, id); + } + else if(unlikely(hash == DISABLE_HASH && !strcmp(s, "DISABLE"))) { + error("PLUGINSD: '%s' called DISABLE. Disabling it.", cd->fullfilename); + cd->enabled = 0; + killpid(cd->pid, SIGTERM); + break; + } #ifdef DETACH_PLUGINS_FROM_NETDATA - else if(likely(hash == MYPID_HASH && !strcmp(s, "MYPID"))) { - char *pid_s = words[1]; - pid_t pid = strtod(pid_s, NULL, 0); - - if(likely(pid)) cd->pid = pid; - debug(D_PLUGINSD, "PLUGINSD: %s is on pid %d", cd->id, cd->pid); - } - else if(likely(hash == STOPPING_WAKE_ME_UP_PLEASE_HASH && !strcmp(s, "STOPPING_WAKE_ME_UP_PLEASE"))) { - error("PLUGINSD: '%s' (pid %d) called STOPPING_WAKE_ME_UP_PLEASE.", cd->fullfilename, cd->pid); - - gettimeofday(&now, NULL); - if(unlikely(!usec && !susec)) { - // our first run - susec = cd->rrd_update_every * 1000000ULL; - } - else { - // second+ run - usec = usecdiff(&now, &last) - susec; - error("PLUGINSD: %s last loop took %llu usec (worked for %llu, sleeped for %llu).\n", cd->fullfilename, usec + susec, usec, susec); - if(unlikely(usec < (rrd_update_every * 1000000ULL / 2ULL))) susec = (rrd_update_every * 1000000ULL) - usec; - else susec = rrd_update_every * 1000000ULL / 2ULL; - } - - error("PLUGINSD: %s sleeping for %llu. Will kill with SIGCONT pid %d to wake it up.\n", cd->fullfilename, susec, cd->pid); - usleep(susec); - killpid(cd->pid, SIGCONT); - bcopy(&now, &last, sizeof(struct timeval)); - break; - } + else if(likely(hash == MYPID_HASH && !strcmp(s, "MYPID"))) { + char *pid_s = words[1]; + pid_t pid = strtod(pid_s, NULL, 0); + + if(likely(pid)) cd->pid = pid; + debug(D_PLUGINSD, "PLUGINSD: %s is on pid %d", cd->id, cd->pid); + } + else if(likely(hash == STOPPING_WAKE_ME_UP_PLEASE_HASH && !strcmp(s, "STOPPING_WAKE_ME_UP_PLEASE"))) { + error("PLUGINSD: '%s' (pid %d) called STOPPING_WAKE_ME_UP_PLEASE.", cd->fullfilename, cd->pid); + + gettimeofday(&now, NULL); + if(unlikely(!usec && !susec)) { + // our first run + susec = cd->rrd_update_every * 1000000ULL; + } + else { + // second+ run + usec = usec_dt(&now, &last) - susec; + error("PLUGINSD: %s last loop took %llu usec (worked for %llu, sleeped for %llu).\n", cd->fullfilename, usec + susec, usec, susec); + if(unlikely(usec < (rrd_update_every * 1000000ULL / 2ULL))) susec = (rrd_update_every * 1000000ULL) - usec; + else susec = rrd_update_every * 1000000ULL / 2ULL; + } + + error("PLUGINSD: %s sleeping for %llu. Will kill with SIGCONT pid %d to wake it up.\n", cd->fullfilename, susec, cd->pid); + usleep(susec); + killpid(cd->pid, SIGCONT); + bcopy(&now, &last, sizeof(struct timeval)); + break; + } #endif - else { - error("PLUGINSD: '%s' is sending command '%s' which is not known by netdata. Disabling it.", cd->fullfilename, s); - cd->enabled = 0; - killpid(cd->pid, SIGTERM); - break; - } - } - - info("PLUGINSD: '%s' on pid %d stopped.", cd->fullfilename, cd->pid); - - // fgets() failed or loop broke - int code = mypclose(fp, cd->pid); - if(code == 1 || code == 127) { - // 1 = DISABLE - // 127 = cannot even run it - error("PLUGINSD: '%s' (pid %d) exited with code %d. Disabling it.", cd->fullfilename, cd->pid, code); - cd->enabled = 0; - } - - if(netdata_exit) { - cd->pid = 0; - cd->enabled = 0; - cd->obsolete = 1; - pthread_exit(NULL); - return NULL; - } - - if(unlikely(!count && cd->enabled)) { - error("PLUGINSD: '%s' (pid %d) does not generate useful output. Waiting a bit before starting it again.", cd->fullfilename, cd->pid); - sleep((unsigned int) (cd->update_every * 10)); - } - - cd->pid = 0; - if(likely(cd->enabled)) sleep((unsigned int) cd->update_every); - else break; - } - - cd->obsolete = 1; - pthread_exit(NULL); - return NULL; + else { + error("PLUGINSD: '%s' is sending command '%s' which is not known by netdata. Disabling it.", cd->fullfilename, s); + cd->enabled = 0; + killpid(cd->pid, SIGTERM); + break; + } + } + if(likely(count)) { + cd->successful_collections += count; + cd->serial_failures = 0; + } + else + cd->serial_failures++; + + info("PLUGINSD: '%s' on pid %d stopped after %zu successful data collections (ENDs).", cd->fullfilename, cd->pid, count); + + // get the return code + int code = mypclose(fp, cd->pid); + + if(netdata_exit) { + cd->pid = 0; + cd->enabled = 0; + cd->obsolete = 1; + pthread_exit(NULL); + return NULL; + } + + if(code != 0) { + // the plugin reports failure + + if(likely(!cd->successful_collections)) { + // nothing collected - disable it + error("PLUGINSD: '%s' exited with error code %d. Disabling it.", cd->fullfilename, code); + cd->enabled = 0; + } + else { + // we have collected something + + if(likely(cd->serial_failures <= 10)) { + error("PLUGINSD: '%s' exited with error code %d, but has given useful output in the past (%zu times). Waiting a bit before starting it again.", cd->fullfilename, code, cd->successful_collections); + sleep((unsigned int) (cd->update_every * 10)); + } + else { + error("PLUGINSD: '%s' exited with error code %d, but has given useful output in the past (%zu times). We tried %zu times to restart it, but it failed to generate data. Disabling it.", cd->fullfilename, code, cd->successful_collections, cd->serial_failures); + cd->enabled = 0; + } + } + } + else { + // the plugin reports success + + if(unlikely(!cd->successful_collections)) { + // we have collected nothing so far + + if(likely(cd->serial_failures <= 10)) { + error("PLUGINSD: '%s' (pid %d) does not generate useful output but it reports success (exits with 0). Waiting a bit before starting it again.", cd->fullfilename, cd->pid); + sleep((unsigned int) (cd->update_every * 10)); + } + else { + error("PLUGINSD: '%s' (pid %d) does not generate useful output, although it reports success (exits with 0), but we have tried %zu times to collect something. Disabling it.", cd->fullfilename, cd->pid, cd->serial_failures); + cd->enabled = 0; + } + } + else + sleep((unsigned int) cd->update_every); + } + cd->pid = 0; + + if(unlikely(!cd->enabled)) + break; + } + + cd->obsolete = 1; + pthread_exit(NULL); + return NULL; } -void *pluginsd_main(void *ptr) -{ - if(ptr) { ; } - - info("PLUGINS.D thread created with task id %d", gettid()); - - if(pthread_setcanceltype(PTHREAD_CANCEL_DEFERRED, NULL) != 0) - error("Cannot set pthread cancel type to DEFERRED."); - - if(pthread_setcancelstate(PTHREAD_CANCEL_ENABLE, NULL) != 0) - error("Cannot set pthread cancel state to ENABLE."); - - char *dir_name = config_get("plugins", "plugins directory", PLUGINS_DIR); - int automatic_run = config_get_boolean("plugins", "enable running new plugins", 1); - int scan_frequency = (int) config_get_number("plugins", "check for new plugins every", 60); - DIR *dir = NULL; - struct dirent *file = NULL; - struct plugind *cd; - - // enable the apps plugin by default - // config_get_boolean("plugins", "apps", 1); - - if(scan_frequency < 1) scan_frequency = 1; - - while(likely(1)) { - if(unlikely(netdata_exit)) break; - - dir = opendir(dir_name); - if(unlikely(!dir)) { - error("Cannot open directory '%s'.", dir_name); - pthread_exit(NULL); - return NULL; - } - - while(likely((file = readdir(dir)))) { - if(unlikely(netdata_exit)) break; - - debug(D_PLUGINSD, "PLUGINSD: Examining file '%s'", file->d_name); - - if(unlikely(strcmp(file->d_name, ".") == 0 || strcmp(file->d_name, "..") == 0)) continue; - - int len = (int) strlen(file->d_name); - if(unlikely(len <= (int)PLUGINSD_FILE_SUFFIX_LEN)) continue; - if(unlikely(strcmp(PLUGINSD_FILE_SUFFIX, &file->d_name[len - (int)PLUGINSD_FILE_SUFFIX_LEN]) != 0)) { - debug(D_PLUGINSD, "PLUGINSD: File '%s' does not end in '%s'.", file->d_name, PLUGINSD_FILE_SUFFIX); - continue; - } - - char pluginname[CONFIG_MAX_NAME + 1]; - snprintfz(pluginname, CONFIG_MAX_NAME, "%.*s", (int)(len - PLUGINSD_FILE_SUFFIX_LEN), file->d_name); - int enabled = config_get_boolean("plugins", pluginname, automatic_run); - - if(unlikely(!enabled)) { - debug(D_PLUGINSD, "PLUGINSD: plugin '%s' is not enabled", file->d_name); - continue; - } - - // check if it runs already - for(cd = pluginsd_root ; likely(cd) ; cd = cd->next) { - if(unlikely(strcmp(cd->filename, file->d_name) == 0)) break; - } - if(likely(cd && !cd->obsolete)) { - debug(D_PLUGINSD, "PLUGINSD: plugin '%s' is already running", cd->filename); - continue; - } - - // it is not running - // allocate a new one, or use the obsolete one - if(unlikely(!cd)) { - cd = calloc(sizeof(struct plugind), 1); - if(unlikely(!cd)) fatal("Cannot allocate memory for plugin."); - - snprintfz(cd->id, CONFIG_MAX_NAME, "plugin:%s", pluginname); - - strncpyz(cd->filename, file->d_name, FILENAME_MAX); - snprintfz(cd->fullfilename, FILENAME_MAX, "%s/%s", dir_name, cd->filename); - - cd->enabled = enabled; - cd->update_every = (int) config_get_number(cd->id, "update every", rrd_update_every); - cd->started_t = time(NULL); - - char *def = ""; - snprintfz(cd->cmd, PLUGINSD_CMD_MAX, "exec %s %d %s", cd->fullfilename, cd->update_every, config_get(cd->id, "command options", def)); - - // link it - if(likely(pluginsd_root)) cd->next = pluginsd_root; - pluginsd_root = cd; - } - cd->obsolete = 0; - - if(unlikely(!cd->enabled)) continue; - - // spawn a new thread for it - if(unlikely(pthread_create(&cd->thread, NULL, pluginsd_worker_thread, cd) != 0)) { - error("PLUGINSD: failed to create new thread for plugin '%s'.", cd->filename); - cd->obsolete = 1; - } - else if(unlikely(pthread_detach(cd->thread) != 0)) - error("PLUGINSD: Cannot request detach of newly created thread for plugin '%s'.", cd->filename); - } - - closedir(dir); - sleep((unsigned int) scan_frequency); - } - - pthread_exit(NULL); - return NULL; +void *pluginsd_main(void *ptr) { + (void)ptr; + + info("PLUGINS.D thread created with task id %d", gettid()); + + if(pthread_setcanceltype(PTHREAD_CANCEL_DEFERRED, NULL) != 0) + error("Cannot set pthread cancel type to DEFERRED."); + + if(pthread_setcancelstate(PTHREAD_CANCEL_ENABLE, NULL) != 0) + error("Cannot set pthread cancel state to ENABLE."); + + char *dir_name = config_get("plugins", "plugins directory", PLUGINS_DIR); + int automatic_run = config_get_boolean("plugins", "enable running new plugins", 1); + int scan_frequency = (int) config_get_number("plugins", "check for new plugins every", 60); + DIR *dir = NULL; + struct dirent *file = NULL; + struct plugind *cd; + + // enable the apps plugin by default + // config_get_boolean("plugins", "apps", 1); + + if(scan_frequency < 1) scan_frequency = 1; + + while(likely(1)) { + if(unlikely(netdata_exit)) break; + + dir = opendir(dir_name); + if(unlikely(!dir)) { + error("Cannot open directory '%s'.", dir_name); + pthread_exit(NULL); + return NULL; + } + + while(likely((file = readdir(dir)))) { + if(unlikely(netdata_exit)) break; + + debug(D_PLUGINSD, "PLUGINSD: Examining file '%s'", file->d_name); + + if(unlikely(strcmp(file->d_name, ".") == 0 || strcmp(file->d_name, "..") == 0)) continue; + + int len = (int) strlen(file->d_name); + if(unlikely(len <= (int)PLUGINSD_FILE_SUFFIX_LEN)) continue; + if(unlikely(strcmp(PLUGINSD_FILE_SUFFIX, &file->d_name[len - (int)PLUGINSD_FILE_SUFFIX_LEN]) != 0)) { + debug(D_PLUGINSD, "PLUGINSD: File '%s' does not end in '%s'.", file->d_name, PLUGINSD_FILE_SUFFIX); + continue; + } + + char pluginname[CONFIG_MAX_NAME + 1]; + snprintfz(pluginname, CONFIG_MAX_NAME, "%.*s", (int)(len - PLUGINSD_FILE_SUFFIX_LEN), file->d_name); + int enabled = config_get_boolean("plugins", pluginname, automatic_run); + + if(unlikely(!enabled)) { + debug(D_PLUGINSD, "PLUGINSD: plugin '%s' is not enabled", file->d_name); + continue; + } + + // check if it runs already + for(cd = pluginsd_root ; likely(cd) ; cd = cd->next) { + if(unlikely(strcmp(cd->filename, file->d_name) == 0)) break; + } + if(likely(cd && !cd->obsolete)) { + debug(D_PLUGINSD, "PLUGINSD: plugin '%s' is already running", cd->filename); + continue; + } + + // it is not running + // allocate a new one, or use the obsolete one + if(unlikely(!cd)) { + cd = callocz(sizeof(struct plugind), 1); + + snprintfz(cd->id, CONFIG_MAX_NAME, "plugin:%s", pluginname); + + strncpyz(cd->filename, file->d_name, FILENAME_MAX); + snprintfz(cd->fullfilename, FILENAME_MAX, "%s/%s", dir_name, cd->filename); + + cd->enabled = enabled; + cd->update_every = (int) config_get_number(cd->id, "update every", rrd_update_every); + cd->started_t = time(NULL); + + char *def = ""; + snprintfz(cd->cmd, PLUGINSD_CMD_MAX, "exec %s %d %s", cd->fullfilename, cd->update_every, config_get(cd->id, "command options", def)); + + // link it + if(likely(pluginsd_root)) cd->next = pluginsd_root; + pluginsd_root = cd; + } + cd->obsolete = 0; + + if(unlikely(!cd->enabled)) continue; + + // spawn a new thread for it + if(unlikely(pthread_create(&cd->thread, NULL, pluginsd_worker_thread, cd) != 0)) { + error("PLUGINSD: failed to create new thread for plugin '%s'.", cd->filename); + cd->obsolete = 1; + } + else if(unlikely(pthread_detach(cd->thread) != 0)) + error("PLUGINSD: Cannot request detach of newly created thread for plugin '%s'.", cd->filename); + } + + closedir(dir); + sleep((unsigned int) scan_frequency); + } + + pthread_exit(NULL); + return NULL; } diff --git a/src/plugins_d.h b/src/plugins_d.h index 11e89e0ac..6f1fbd6e1 100644 --- a/src/plugins_d.h +++ b/src/plugins_d.h @@ -1,7 +1,3 @@ -#include <sys/types.h> -#include <unistd.h> - - #ifndef NETDATA_PLUGINS_D_H #define NETDATA_PLUGINS_D_H 1 @@ -11,22 +7,28 @@ #define PLUGINSD_LINE_MAX 1024 struct plugind { - char id[CONFIG_MAX_NAME+1]; // config node id + char id[CONFIG_MAX_NAME+1]; // config node id + + char filename[FILENAME_MAX+1]; // just the filename + char fullfilename[FILENAME_MAX+1]; // with path + char cmd[PLUGINSD_CMD_MAX+1]; // the command that is executes + + pid_t pid; + pthread_t thread; - char filename[FILENAME_MAX+1]; // just the filename - char fullfilename[FILENAME_MAX+1]; // with path - char cmd[PLUGINSD_CMD_MAX+1]; // the command that is executes + size_t successful_collections; // the number of times we have seen + // values collected from this plugin - pid_t pid; - pthread_t thread; + size_t serial_failures; // the number of times the plugin started + // without collecting values - int update_every; - int obsolete; - int enabled; + int update_every; // the plugin default data collection frequency + int obsolete; // do not touch this structure after setting this to 1 + int enabled; // if this is enabled or not - time_t started_t; + time_t started_t; - struct plugind *next; + struct plugind *next; }; extern struct plugind *pluginsd_root; diff --git a/src/popen.c b/src/popen.c index 06f27c0b7..193efc0f3 100644 --- a/src/popen.c +++ b/src/popen.c @@ -1,54 +1,43 @@ -#ifdef HAVE_CONFIG_H -#include <config.h> -#endif -#include <signal.h> -#include <unistd.h> -#include <stdlib.h> -#include <sys/types.h> -#include <sys/wait.h> - -#include "log.h" -#include "popen.h" #include "common.h" /* struct mypopen { - pid_t pid; - FILE *fp; - struct mypopen *next; - struct mypopen *prev; + pid_t pid; + FILE *fp; + struct mypopen *next; + struct mypopen *prev; }; static struct mypopen *mypopen_root = NULL; static void mypopen_add(FILE *fp, pid_t *pid) { - struct mypopen *mp = malloc(sizeof(struct mypopen)); - if(!mp) { - fatal("Cannot allocate %zu bytes", sizeof(struct mypopen)) - return; - } - - mp->fp = fp; - mp->pid = pid; - mp->next = popen_root; - mp->prev = NULL; - if(mypopen_root) mypopen_root->prev = mp; - mypopen_root = mp; + struct mypopen *mp = malloc(sizeof(struct mypopen)); + if(!mp) { + fatal("Cannot allocate %zu bytes", sizeof(struct mypopen)) + return; + } + + mp->fp = fp; + mp->pid = pid; + mp->next = popen_root; + mp->prev = NULL; + if(mypopen_root) mypopen_root->prev = mp; + mypopen_root = mp; } static void mypopen_del(FILE *fp) { - struct mypopen *mp; - - for(mp = mypopen_root; mp; mp = mp->next) - if(mp->fd == fp) break; - - if(!mp) error("Cannot find mypopen() file pointer in open childs."); - else { - if(mp->next) mp->next->prev = mp->prev; - if(mp->prev) mp->prev->next = mp->next; - if(mypopen_root == mp) mypopen_root = mp->next; - free(mp); - } + struct mypopen *mp; + + for(mp = mypopen_root; mp; mp = mp->next) + if(mp->fd == fp) break; + + if(!mp) error("Cannot find mypopen() file pointer in open childs."); + else { + if(mp->next) mp->next->prev = mp->prev; + if(mp->prev) mp->prev->next = mp->next; + if(mypopen_root == mp) mypopen_root = mp->next; + free(mp); + } } */ #define PIPE_READ 0 @@ -56,133 +45,152 @@ static void mypopen_del(FILE *fp) { FILE *mypopen(const char *command, pid_t *pidptr) { - int pipefd[2]; - - if(pipe(pipefd) == -1) return NULL; - - int pid = fork(); - if(pid == -1) { - close(pipefd[PIPE_READ]); - close(pipefd[PIPE_WRITE]); - return NULL; - } - if(pid != 0) { - // the parent - *pidptr = pid; - close(pipefd[PIPE_WRITE]); - FILE *fp = fdopen(pipefd[PIPE_READ], "r"); - /*mypopen_add(fp, pid);*/ - return(fp); - } - // the child - - // close all files - int i; - for(i = (int) (sysconf(_SC_OPEN_MAX) - 1); i > 0; i--) - if(i != STDIN_FILENO && i != STDERR_FILENO && i != pipefd[PIPE_WRITE]) close(i); - - // move the pipe to stdout - if(pipefd[PIPE_WRITE] != STDOUT_FILENO) { - dup2(pipefd[PIPE_WRITE], STDOUT_FILENO); - close(pipefd[PIPE_WRITE]); - } + int pipefd[2]; + + if(pipe(pipefd) == -1) return NULL; + + int pid = fork(); + if(pid == -1) { + close(pipefd[PIPE_READ]); + close(pipefd[PIPE_WRITE]); + return NULL; + } + if(pid != 0) { + // the parent + *pidptr = pid; + close(pipefd[PIPE_WRITE]); + FILE *fp = fdopen(pipefd[PIPE_READ], "r"); + /*mypopen_add(fp, pid);*/ + return(fp); + } + // the child + + // close all files + int i; + for(i = (int) (sysconf(_SC_OPEN_MAX) - 1); i > 0; i--) + if(i != STDIN_FILENO && i != STDERR_FILENO && i != pipefd[PIPE_WRITE]) close(i); + + // move the pipe to stdout + if(pipefd[PIPE_WRITE] != STDOUT_FILENO) { + dup2(pipefd[PIPE_WRITE], STDOUT_FILENO); + close(pipefd[PIPE_WRITE]); + } #ifdef DETACH_PLUGINS_FROM_NETDATA - // this was an attempt to detach the child and use the suspend mode charts.d - // unfortunatelly it does not work as expected. + // this was an attempt to detach the child and use the suspend mode charts.d + // unfortunatelly it does not work as expected. + + // fork again to become session leader + pid = fork(); + if(pid == -1) + error("pre-execution of command '%s' on pid %d: Cannot fork 2nd time.", command, getpid()); - // fork again to become session leader - pid = fork(); - if(pid == -1) fprintf(stderr, "Cannot fork again on pid %d\n", getpid()); - if(pid != 0) { - // the parent - exit(0); - } + if(pid != 0) { + // the parent + exit(0); + } - // set a new process group id for just this child - if( setpgid(0, 0) != 0 ) - error("Cannot set a new process group for pid %d (%s)", getpid(), strerror(errno)); + // set a new process group id for just this child + if( setpgid(0, 0) != 0 ) + error("pre-execution of command '%s' on pid %d: Cannot set a new process group.", command, getpid()); - if( getpgid(0) != getpid() ) - error("Process group set is incorrect. Expected %d, found %d", getpid(), getpgid(0)); + if( getpgid(0) != getpid() ) + error("pre-execution of command '%s' on pid %d: Cannot set a new process group. Process group set is incorrect. Expected %d, found %d", command, getpid(), getpid(), getpgid(0)); - if( setsid() != 0 ) - error("Cannot set session id for pid %d (%s)", getpid(), strerror(errno)); + if( setsid() != 0 ) + error("pre-execution of command '%s' on pid %d: Cannot set session id.", command, getpid()); - fprintf(stdout, "MYPID %d\n", getpid()); - fflush(NULL); + fprintf(stdout, "MYPID %d\n", getpid()); + fflush(NULL); #endif - // reset all signals - { - sigset_t sigset; - sigfillset(&sigset); - - if(pthread_sigmask(SIG_UNBLOCK, &sigset, NULL) == -1) { - error("Could not block signals for threads"); - } - // We only need to reset ignored signals. - // Signals with signal handlers are reset by default. - struct sigaction sa; - sigemptyset(&sa.sa_mask); - sa.sa_handler = SIG_DFL; - sa.sa_flags = 0; - if(sigaction(SIGPIPE, &sa, NULL) == -1) { - error("Failed to change signal handler for SIGTERM"); - } - } - - - info("executing command: '%s' on pid %d.", command, getpid()); - execl("/bin/sh", "sh", "-c", command, NULL); - exit(1); + // reset all signals + { + sigset_t sigset; + sigfillset(&sigset); + + if(pthread_sigmask(SIG_UNBLOCK, &sigset, NULL) == -1) + error("pre-execution of command '%s' on pid %d: could not unblock signals for threads.", command, getpid()); + + // We only need to reset ignored signals. + // Signals with signal handlers are reset by default. + struct sigaction sa; + sigemptyset(&sa.sa_mask); + sa.sa_handler = SIG_DFL; + sa.sa_flags = 0; + + if(sigaction(SIGINT, &sa, NULL) == -1) + error("pre-execution of command '%s' on pid %d: failed to set default signal handler for SIGINT.", command, getpid()); + + if(sigaction(SIGTERM, &sa, NULL) == -1) + error("pre-execution of command '%s' on pid %d: failed to set default signal handler for SIGTERM.", command, getpid()); + + if(sigaction(SIGPIPE, &sa, NULL) == -1) + error("pre-execution of command '%s' on pid %d: failed to set default signal handler for SIGPIPE.", command, getpid()); + + if(sigaction(SIGHUP, &sa, NULL) == -1) + error("pre-execution of command '%s' on pid %d: failed to set default signal handler for SIGHUP.", command, getpid()); + + if(sigaction(SIGUSR1, &sa, NULL) == -1) + error("pre-execution of command '%s' on pid %d: failed to set default signal handler for SIGUSR1.", command, getpid()); + + if(sigaction(SIGUSR2, &sa, NULL) == -1) + error("pre-execution of command '%s' on pid %d: failed to set default signal handler for SIGUSR2.", command, getpid()); + } + + info("executing command: '%s' on pid %d.", command, getpid()); + execl("/bin/sh", "sh", "-c", command, NULL); + exit(1); } int mypclose(FILE *fp, pid_t pid) { - debug(D_EXIT, "Request to mypclose() on pid %d", pid); - - /*mypopen_del(fp);*/ - fclose(fp); - - siginfo_t info; - if(waitid(P_PID, (id_t) pid, &info, WEXITED) != -1) { - switch(info.si_code) { - case CLD_EXITED: - error("pid %d exited with code %d.", info.si_pid, info.si_status); - return(info.si_status); - break; - - case CLD_KILLED: - error("pid %d killed by signal %d.", info.si_pid, info.si_status); - return(-1); - break; - - case CLD_DUMPED: - error("pid %d core dumped by signal %d.", info.si_pid, info.si_status); - return(-2); - break; - - case CLD_STOPPED: - error("pid %d stopped by signal %d.", info.si_pid, info.si_status); - return(0); - break; - - case CLD_TRAPPED: - error("pid %d trapped by signal %d.", info.si_pid, info.si_status); - return(-4); - break; - - case CLD_CONTINUED: - error("pid %d continued by signal %d.", info.si_pid, info.si_status); - return(0); - break; - - default: - error("pid %d gave us a SIGCHLD with code %d and status %d.", info.si_pid, info.si_code, info.si_status); - return(-5); - break; - } - } - else error("Cannot waitid() for pid %d", pid); - return 0; + debug(D_EXIT, "Request to mypclose() on pid %d", pid); + + /*mypopen_del(fp);*/ + fclose(fp); + + siginfo_t info; + if(waitid(P_PID, (id_t) pid, &info, WEXITED) != -1) { + switch(info.si_code) { + case CLD_EXITED: + if(info.si_status) + error("child pid %d exited with code %d.", info.si_pid, info.si_status); + return(info.si_status); + break; + + case CLD_KILLED: + error("child pid %d killed by signal %d.", info.si_pid, info.si_status); + return(-1); + break; + + case CLD_DUMPED: + error("child pid %d core dumped by signal %d.", info.si_pid, info.si_status); + return(-2); + break; + + case CLD_STOPPED: + error("child pid %d stopped by signal %d.", info.si_pid, info.si_status); + return(0); + break; + + case CLD_TRAPPED: + error("child pid %d trapped by signal %d.", info.si_pid, info.si_status); + return(-4); + break; + + case CLD_CONTINUED: + error("child pid %d continued by signal %d.", info.si_pid, info.si_status); + return(0); + break; + + default: + error("child pid %d gave us a SIGCHLD with code %d and status %d.", info.si_pid, info.si_code, info.si_status); + return(-5); + break; + } + } + else + error("Cannot waitid() for pid %d", pid); + + return 0; } diff --git a/src/popen.h b/src/popen.h index 10680f0c8..90845e1fb 100644 --- a/src/popen.h +++ b/src/popen.h @@ -1,7 +1,3 @@ -#include <stdio.h> -#include <unistd.h> -#include <sys/types.h> - #ifndef NETDATA_POPEN_H #define NETDATA_POPEN_H 1 diff --git a/src/proc_diskstats.c b/src/proc_diskstats.c index c62a1351c..459d5a133 100644 --- a/src/proc_diskstats.c +++ b/src/proc_diskstats.c @@ -1,578 +1,700 @@ -#ifdef HAVE_CONFIG_H -#include <config.h> -#endif -#include <stdio.h> -#include <stdlib.h> -#include <string.h> -#include <sys/types.h> -#include <sys/stat.h> -#include <fcntl.h> - #include "common.h" -#include "log.h" -#include "appconfig.h" -#include "procfile.h" -#include "rrd.h" -#include "plugin_proc.h" - -#include "proc_self_mountinfo.h" #define RRD_TYPE_DISK "disk" -struct disk { - unsigned long major; - unsigned long minor; - int partition_id; // -1 = this is not a partition - char *family; - struct disk *next; +#define DISK_TYPE_PHYSICAL 1 +#define DISK_TYPE_PARTITION 2 +#define DISK_TYPE_CONTAINER 3 + +static struct disk { + char *disk; // the name of the disk (sda, sdb, etc) + unsigned long major; + unsigned long minor; + int sector_size; + int type; + char *mount_point; + + // disk options caching + int configured; + int do_io; + int do_ops; + int do_mops; + int do_iotime; + int do_qops; + int do_util; + int do_backlog; + int do_space; + int do_inodes; + + struct disk *next; } *disk_root = NULL; -struct disk *get_disk(unsigned long major, unsigned long minor) { - static char path_find_block_device_partition[FILENAME_MAX + 1] = ""; - static struct mountinfo *mountinfo_root = NULL; - struct disk *d; - - // search for it in our RAM list. - // this is sequential, but since we just walk through - // and the number of disks / partitions in a system - // should not be that many, it should be acceptable - for(d = disk_root; d ; d = d->next) - if(unlikely(d->major == major && d->minor == minor)) - break; - - // if we found it, return it - if(likely(d)) - return d; - - if(unlikely(!path_find_block_device_partition[0])) { - char filename[FILENAME_MAX + 1]; - snprintfz(filename, FILENAME_MAX, "%s%s", global_host_prefix, "/sys/dev/block/%lu:%lu/partition"); - snprintfz(path_find_block_device_partition, FILENAME_MAX, "%s", config_get("plugin:proc:/proc/diskstats", "path to get block device partition", filename)); - } - - // not found - // create a new disk structure - d = (struct disk *)malloc(sizeof(struct disk)); - if(!d) fatal("Cannot allocate memory for struct disk in proc_diskstats."); - - d->major = major; - d->minor = minor; - d->partition_id = -1; - d->next = NULL; - - // append it to the list - if(!disk_root) - disk_root = d; - else { - struct disk *last; - for(last = disk_root; last->next ;last = last->next); - last->next = d; - } - - // find if it is a partition - // by reading /sys/dev/block/MAJOR:MINOR/partition - char buffer[FILENAME_MAX + 1]; - snprintfz(buffer, FILENAME_MAX, path_find_block_device_partition, major, minor); - - int fd = open(buffer, O_RDONLY, 0666); - if(likely(fd != -1)) { - // we opened it - int bytes = read(fd, buffer, FILENAME_MAX); - close(fd); - - if(bytes > 0) - d->partition_id = strtoul(buffer, NULL, 10); - } - // if the /partition file does not exist, it is a disk, not a partition - - // ------------------------------------------------------------------------ - // check if we can find its mount point - - // mountinfo_find() can be called with NULL mountinfo_root - struct mountinfo *mi = mountinfo_find(mountinfo_root, d->major, d->minor); - if(unlikely(!mi)) { - // mountinfo_free() can be called with NULL mountinfo_root - mountinfo_free(mountinfo_root); - - // re-read mountinfo in case something changed - mountinfo_root = mountinfo_read(); - - // search again for this disk - mi = mountinfo_find(mountinfo_root, d->major, d->minor); - } - - if(mi) - d->family = strdup(mi->mount_point); - // no need to check for NULL - else - d->family = NULL; - - return d; +static struct mountinfo *disk_mountinfo_root = NULL; + +static struct disk *get_disk(unsigned long major, unsigned long minor, char *disk) { + static char path_to_get_hw_sector_size[FILENAME_MAX + 1] = ""; + static char path_to_get_hw_sector_size_partitions[FILENAME_MAX + 1] = ""; + static char path_find_block_device[FILENAME_MAX + 1] = ""; + struct disk *d; + + // search for it in our RAM list. + // this is sequential, but since we just walk through + // and the number of disks / partitions in a system + // should not be that many, it should be acceptable + for(d = disk_root; d ; d = d->next) + if(unlikely(d->major == major && d->minor == minor)) + break; + + // if we found it, return it + if(likely(d)) + return d; + + // not found + // create a new disk structure + d = (struct disk *)mallocz(sizeof(struct disk)); + + d->disk = strdupz(disk); + d->major = major; + d->minor = minor; + d->type = DISK_TYPE_PHYSICAL; // Default type. Changed later if not correct. + d->configured = 0; + d->sector_size = 512; // the default, will be changed below + d->next = NULL; + + // append it to the list + if(!disk_root) + disk_root = d; + else { + struct disk *last; + for(last = disk_root; last->next ;last = last->next); + last->next = d; + } + + // ------------------------------------------------------------------------ + // find the type of the device + + char buffer[FILENAME_MAX + 1]; + + // get the default path for finding info about the block device + if(unlikely(!path_find_block_device[0])) { + snprintfz(buffer, FILENAME_MAX, "%s%s", global_host_prefix, "/sys/dev/block/%lu:%lu/%s"); + snprintfz(path_find_block_device, FILENAME_MAX, "%s", config_get("plugin:proc:/proc/diskstats", "path to get block device infos", buffer)); + } + + // find if it is a partition + // by checking if /sys/dev/block/MAJOR:MINOR/partition is readable. + snprintfz(buffer, FILENAME_MAX, path_find_block_device, major, minor, "partition"); + if(access(buffer, R_OK) == 0) { + d->type = DISK_TYPE_PARTITION; + } else { + // find if it is a container + // by checking if /sys/dev/block/MAJOR:MINOR/slaves has entries + snprintfz(buffer, FILENAME_MAX, path_find_block_device, major, minor, "slaves/"); + DIR *dirp = opendir(buffer); + if (dirp != NULL) { + struct dirent *dp; + while( (dp = readdir(dirp)) ) { + // . and .. are also files in empty folders. + if(strcmp(dp->d_name, ".") == 0 || strcmp(dp->d_name, "..") == 0) { + continue; + } + + d->type = DISK_TYPE_CONTAINER; + + // Stop the loop after we found one file. + break; + } + if(closedir(dirp) == -1) + error("Unable to close dir %s", buffer); + } + } + + // ------------------------------------------------------------------------ + // check if we can find its mount point + + // mountinfo_find() can be called with NULL disk_mountinfo_root + struct mountinfo *mi = mountinfo_find(disk_mountinfo_root, d->major, d->minor); + if(unlikely(!mi)) { + // mountinfo_free() can be called with NULL disk_mountinfo_root + mountinfo_free(disk_mountinfo_root); + + // re-read mountinfo in case something changed + disk_mountinfo_root = mountinfo_read(); + + // search again for this disk + mi = mountinfo_find(disk_mountinfo_root, d->major, d->minor); + } + + if(mi) + d->mount_point = strdupz(mi->mount_point); + // no need to check for NULL + else + d->mount_point = NULL; + + // ------------------------------------------------------------------------ + // find the disk sector size + + if(!path_to_get_hw_sector_size[0]) { + snprintfz(buffer, FILENAME_MAX, "%s%s", global_host_prefix, "/sys/block/%s/queue/hw_sector_size"); + snprintfz(path_to_get_hw_sector_size, FILENAME_MAX, "%s", config_get("plugin:proc:/proc/diskstats", "path to get h/w sector size", buffer)); + } + if(!path_to_get_hw_sector_size_partitions[0]) { + snprintfz(buffer, FILENAME_MAX, "%s%s", global_host_prefix, "/sys/dev/block/%lu:%lu/subsystem/%s/../queue/hw_sector_size"); + snprintfz(path_to_get_hw_sector_size_partitions, FILENAME_MAX, "%s", config_get("plugin:proc:/proc/diskstats", "path to get h/w sector size for partitions", buffer)); + } + + { + char tf[FILENAME_MAX + 1], *t; + strncpyz(tf, d->disk, FILENAME_MAX); + + // replace all / with ! + for(t = tf; *t ;t++) + if(*t == '/') *t = '!'; + + if(d->type == DISK_TYPE_PARTITION) + snprintfz(buffer, FILENAME_MAX, path_to_get_hw_sector_size_partitions, d->major, d->minor, tf); + else + snprintfz(buffer, FILENAME_MAX, path_to_get_hw_sector_size, tf); + + FILE *fpss = fopen(buffer, "r"); + if(fpss) { + char buffer2[1024 + 1]; + char *tmp = fgets(buffer2, 1024, fpss); + + if(tmp) { + d->sector_size = atoi(tmp); + if(d->sector_size <= 0) { + error("Invalid sector size %d for device %s in %s. Assuming 512.", d->sector_size, d->disk, buffer); + d->sector_size = 512; + } + } + else error("Cannot read data for sector size for device %s from %s. Assuming 512.", d->disk, buffer); + + fclose(fpss); + } + else error("Cannot read sector size for device %s from %s. Assuming 512.", d->disk, buffer); + } + + return d; +} + +static inline int select_positive_option(int option1, int option2) { + if(option1 == CONFIG_ONDEMAND_YES || option2 == CONFIG_ONDEMAND_YES) + return CONFIG_ONDEMAND_YES; + else if(option1 == CONFIG_ONDEMAND_ONDEMAND || option2 == CONFIG_ONDEMAND_ONDEMAND) + return CONFIG_ONDEMAND_ONDEMAND; + + return CONFIG_ONDEMAND_NO; } int do_proc_diskstats(int update_every, unsigned long long dt) { - static procfile *ff = NULL; - static char path_to_get_hw_sector_size[FILENAME_MAX + 1] = ""; - static int enable_new_disks = -1; - static int do_io = -1, do_ops = -1, do_mops = -1, do_iotime = -1, do_qops = -1, do_util = -1, do_backlog = -1; - - if(enable_new_disks == -1) enable_new_disks = config_get_boolean_ondemand("plugin:proc:/proc/diskstats", "enable new disks detected at runtime", CONFIG_ONDEMAND_ONDEMAND); - - if(do_io == -1) do_io = config_get_boolean_ondemand("plugin:proc:/proc/diskstats", "bandwidth for all disks", CONFIG_ONDEMAND_ONDEMAND); - if(do_ops == -1) do_ops = config_get_boolean_ondemand("plugin:proc:/proc/diskstats", "operations for all disks", CONFIG_ONDEMAND_ONDEMAND); - if(do_mops == -1) do_mops = config_get_boolean_ondemand("plugin:proc:/proc/diskstats", "merged operations for all disks", CONFIG_ONDEMAND_ONDEMAND); - if(do_iotime == -1) do_iotime = config_get_boolean_ondemand("plugin:proc:/proc/diskstats", "i/o time for all disks", CONFIG_ONDEMAND_ONDEMAND); - if(do_qops == -1) do_qops = config_get_boolean_ondemand("plugin:proc:/proc/diskstats", "queued operations for all disks", CONFIG_ONDEMAND_ONDEMAND); - if(do_util == -1) do_util = config_get_boolean_ondemand("plugin:proc:/proc/diskstats", "utilization percentage for all disks", CONFIG_ONDEMAND_ONDEMAND); - if(do_backlog == -1)do_backlog = config_get_boolean_ondemand("plugin:proc:/proc/diskstats", "backlog for all disks", CONFIG_ONDEMAND_ONDEMAND); - - if(!ff) { - char filename[FILENAME_MAX + 1]; - snprintfz(filename, FILENAME_MAX, "%s%s", global_host_prefix, "/proc/diskstats"); - ff = procfile_open(config_get("plugin:proc:/proc/diskstats", "filename to monitor", filename), " \t", PROCFILE_FLAG_DEFAULT); - } - if(!ff) return 1; - - if(!path_to_get_hw_sector_size[0]) { - char filename[FILENAME_MAX + 1]; - snprintfz(filename, FILENAME_MAX, "%s%s", global_host_prefix, "/sys/block/%s/queue/hw_sector_size"); - snprintfz(path_to_get_hw_sector_size, FILENAME_MAX, "%s", config_get("plugin:proc:/proc/diskstats", "path to get h/w sector size", filename)); - } - - ff = procfile_readall(ff); - if(!ff) return 0; // we return 0, so that we will retry to open it next time - - uint32_t lines = procfile_lines(ff), l; - uint32_t words; - - for(l = 0; l < lines ;l++) { - char *disk; - unsigned long long major = 0, minor = 0, - reads = 0, mreads = 0, readsectors = 0, readms = 0, - writes = 0, mwrites = 0, writesectors = 0, writems = 0, - queued_ios = 0, busy_ms = 0, backlog_ms = 0; - - unsigned long long last_reads = 0, last_readsectors = 0, last_readms = 0, - last_writes = 0, last_writesectors = 0, last_writems = 0, - last_busy_ms = 0; - - words = procfile_linewords(ff, l); - if(words < 14) continue; - - major = strtoull(procfile_lineword(ff, l, 0), NULL, 10); - minor = strtoull(procfile_lineword(ff, l, 1), NULL, 10); - disk = procfile_lineword(ff, l, 2); - - // # of reads completed # of writes completed - // This is the total number of reads or writes completed successfully. - reads = strtoull(procfile_lineword(ff, l, 3), NULL, 10); // rd_ios - writes = strtoull(procfile_lineword(ff, l, 7), NULL, 10); // wr_ios - - // # of reads merged # of writes merged - // Reads and writes which are adjacent to each other may be merged for - // efficiency. Thus two 4K reads may become one 8K read before it is - // ultimately handed to the disk, and so it will be counted (and queued) - mreads = strtoull(procfile_lineword(ff, l, 4), NULL, 10); // rd_merges_or_rd_sec - mwrites = strtoull(procfile_lineword(ff, l, 8), NULL, 10); // wr_merges - - // # of sectors read # of sectors written - // This is the total number of sectors read or written successfully. - readsectors = strtoull(procfile_lineword(ff, l, 5), NULL, 10); // rd_sec_or_wr_ios - writesectors = strtoull(procfile_lineword(ff, l, 9), NULL, 10); // wr_sec - - // # of milliseconds spent reading # of milliseconds spent writing - // This is the total number of milliseconds spent by all reads or writes (as - // measured from __make_request() to end_that_request_last()). - readms = strtoull(procfile_lineword(ff, l, 6), NULL, 10); // rd_ticks_or_wr_sec - writems = strtoull(procfile_lineword(ff, l, 10), NULL, 10); // wr_ticks - - // # of I/Os currently in progress - // The only field that should go to zero. Incremented as requests are - // given to appropriate struct request_queue and decremented as they finish. - queued_ios = strtoull(procfile_lineword(ff, l, 11), NULL, 10); // ios_pgr - - // # of milliseconds spent doing I/Os - // This field increases so long as field queued_ios is nonzero. - busy_ms = strtoull(procfile_lineword(ff, l, 12), NULL, 10); // tot_ticks - - // weighted # of milliseconds spent doing I/Os - // This field is incremented at each I/O start, I/O completion, I/O - // merge, or read of these stats by the number of I/Os in progress - // (field queued_ios) times the number of milliseconds spent doing I/O since the - // last update of this field. This can provide an easy measure of both - // I/O completion time and the backlog that may be accumulating. - backlog_ms = strtoull(procfile_lineword(ff, l, 13), NULL, 10); // rq_ticks - - int def_enabled = 0; - - // remove slashes from disk names - char *s; - for(s = disk; *s ;s++) if(*s == '/') *s = '_'; - - struct disk *d = get_disk(major, minor); - if(d->partition_id == -1) - def_enabled = enable_new_disks; - else - def_enabled = 0; - - char *family = d->family; - if(!family) family = disk; - -/* - switch(major) { - case 9: // MDs - case 43: // network block - case 144: // nfs - case 145: // nfs - case 146: // nfs - case 199: // veritas - case 201: // veritas - case 251: // dm - case 253: // virtio - def_enabled = enable_new_disks; - break; - - case 48: // RAID - case 49: // RAID - case 50: // RAID - case 51: // RAID - case 52: // RAID - case 53: // RAID - case 54: // RAID - case 55: // RAID - case 112: // RAID - case 136: // RAID - case 137: // RAID - case 138: // RAID - case 139: // RAID - case 140: // RAID - case 141: // RAID - case 142: // RAID - case 143: // RAID - case 179: // MMC - case 180: // USB - if(minor % 8) def_enabled = 0; // partitions - else def_enabled = enable_new_disks; - break; - - case 8: // scsi disks - case 65: // scsi disks - case 66: // scsi disks - case 67: // scsi disks - case 68: // scsi disks - case 69: // scsi disks - case 70: // scsi disks - case 71: // scsi disks - case 72: // scsi disks - case 73: // scsi disks - case 74: // scsi disks - case 75: // scsi disks - case 76: // scsi disks - case 77: // scsi disks - case 78: // scsi disks - case 79: // scsi disks - case 80: // i2o - case 81: // i2o - case 82: // i2o - case 83: // i2o - case 84: // i2o - case 85: // i2o - case 86: // i2o - case 87: // i2o - case 101: // hyperdisk - case 102: // compressed - case 104: // scsi - case 105: // scsi - case 106: // scsi - case 107: // scsi - case 108: // scsi - case 109: // scsi - case 110: // scsi - case 111: // scsi - case 114: // bios raid - case 116: // ram board - case 128: // scsi - case 129: // scsi - case 130: // scsi - case 131: // scsi - case 132: // scsi - case 133: // scsi - case 134: // scsi - case 135: // scsi - case 153: // raid - case 202: // xen - case 254: // virtio3 - case 256: // flash - case 257: // flash - case 259: // nvme0n1 issue #119 - if(minor % 16) def_enabled = 0; // partitions - else def_enabled = enable_new_disks; - break; - - case 160: // raid - case 161: // raid - if(minor % 32) def_enabled = 0; // partitions - else def_enabled = enable_new_disks; - break; - - case 3: // ide - case 13: // 8bit ide - case 22: // ide - case 33: // ide - case 34: // ide - case 56: // ide - case 57: // ide - case 88: // ide - case 89: // ide - case 90: // ide - case 91: // ide - if(minor % 64) def_enabled = 0; // partitions - else def_enabled = enable_new_disks; - break; - - case 252: // zram - def_enabled = 0; - break; - - default: - def_enabled = 0; - break; - } -*/ - - int ddo_io = do_io, ddo_ops = do_ops, ddo_mops = do_mops, ddo_iotime = do_iotime, ddo_qops = do_qops, ddo_util = do_util, ddo_backlog = do_backlog; - - // check which charts are enabled for this disk - { - char var_name[4096 + 1]; - snprintfz(var_name, 4096, "plugin:proc:/proc/diskstats:%s", disk); - def_enabled = config_get_boolean_ondemand(var_name, "enabled", def_enabled); - if(def_enabled == CONFIG_ONDEMAND_NO) continue; - if(def_enabled == CONFIG_ONDEMAND_ONDEMAND && !reads && !writes) continue; - - - ddo_io = config_get_boolean_ondemand(var_name, "bandwidth", ddo_io); - ddo_ops = config_get_boolean_ondemand(var_name, "operations", ddo_ops); - ddo_mops = config_get_boolean_ondemand(var_name, "merged operations", ddo_mops); - ddo_iotime = config_get_boolean_ondemand(var_name, "i/o time", ddo_iotime); - ddo_qops = config_get_boolean_ondemand(var_name, "queued operations", ddo_qops); - ddo_util = config_get_boolean_ondemand(var_name, "utilization percentage", ddo_util); - ddo_backlog = config_get_boolean_ondemand(var_name, "backlog", ddo_backlog); - - // by default, do not add charts that do not have values - if(ddo_io == CONFIG_ONDEMAND_ONDEMAND && !reads && !writes) ddo_io = 0; - if(ddo_mops == CONFIG_ONDEMAND_ONDEMAND && mreads == 0 && mwrites == 0) ddo_mops = 0; - if(ddo_iotime == CONFIG_ONDEMAND_ONDEMAND && readms == 0 && writems == 0) ddo_iotime = 0; - if(ddo_util == CONFIG_ONDEMAND_ONDEMAND && busy_ms == 0) ddo_util = 0; - if(ddo_backlog == CONFIG_ONDEMAND_ONDEMAND && backlog_ms == 0) ddo_backlog = 0; - if(ddo_qops == CONFIG_ONDEMAND_ONDEMAND && backlog_ms == 0) ddo_qops = 0; - - // for absolute values, we need to switch the setting to 'yes' - // to allow it refresh from now on - if(ddo_qops == CONFIG_ONDEMAND_ONDEMAND) config_set(var_name, "queued operations", "yes"); - } - - RRDSET *st; - - // -------------------------------------------------------------------- - - int sector_size = 512; - if(ddo_io) { - st = rrdset_find_bytype(RRD_TYPE_DISK, disk); - if(!st) { - char tf[FILENAME_MAX + 1], *t; - char ssfilename[FILENAME_MAX + 1]; - - strncpyz(tf, disk, FILENAME_MAX); - - // replace all / with ! - while((t = strchr(tf, '/'))) *t = '!'; - - snprintfz(ssfilename, FILENAME_MAX, path_to_get_hw_sector_size, tf); - FILE *fpss = fopen(ssfilename, "r"); - if(fpss) { - char ssbuffer[1025]; - char *tmp = fgets(ssbuffer, 1024, fpss); - - if(tmp) { - sector_size = atoi(tmp); - if(sector_size <= 0) { - error("Invalid sector size %d for device %s in %s. Assuming 512.", sector_size, disk, ssfilename); - sector_size = 512; - } - } - else error("Cannot read data for sector size for device %s from %s. Assuming 512.", disk, ssfilename); - - fclose(fpss); - } - else error("Cannot read sector size for device %s from %s. Assuming 512.", disk, ssfilename); - - st = rrdset_create(RRD_TYPE_DISK, disk, NULL, family, "disk.io", "Disk I/O Bandwidth", "kilobytes/s", 2000, update_every, RRDSET_TYPE_AREA); - - rrddim_add(st, "reads", NULL, sector_size, 1024, RRDDIM_INCREMENTAL); - rrddim_add(st, "writes", NULL, sector_size * -1, 1024, RRDDIM_INCREMENTAL); - } - else rrdset_next_usec(st, dt); - - last_readsectors = rrddim_set(st, "reads", readsectors); - last_writesectors = rrddim_set(st, "writes", writesectors); - rrdset_done(st); - } - - // -------------------------------------------------------------------- - - if(ddo_ops) { - st = rrdset_find_bytype("disk_ops", disk); - if(!st) { - st = rrdset_create("disk_ops", disk, NULL, family, "disk.ops", "Disk Completed I/O Operations", "operations/s", 2001, update_every, RRDSET_TYPE_LINE); - st->isdetail = 1; - - rrddim_add(st, "reads", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "writes", NULL, -1, 1, RRDDIM_INCREMENTAL); - } - else rrdset_next_usec(st, dt); - - last_reads = rrddim_set(st, "reads", reads); - last_writes = rrddim_set(st, "writes", writes); - rrdset_done(st); - } - - // -------------------------------------------------------------------- - - if(ddo_qops) { - st = rrdset_find_bytype("disk_qops", disk); - if(!st) { - st = rrdset_create("disk_qops", disk, NULL, family, "disk.qops", "Disk Current I/O Operations", "operations", 2002, update_every, RRDSET_TYPE_LINE); - st->isdetail = 1; - - rrddim_add(st, "operations", NULL, 1, 1, RRDDIM_ABSOLUTE); - } - else rrdset_next_usec(st, dt); - - rrddim_set(st, "operations", queued_ios); - rrdset_done(st); - } - - // -------------------------------------------------------------------- - - if(ddo_backlog) { - st = rrdset_find_bytype("disk_backlog", disk); - if(!st) { - st = rrdset_create("disk_backlog", disk, NULL, family, "disk.backlog", "Disk Backlog", "backlog (ms)", 2003, update_every, RRDSET_TYPE_AREA); - st->isdetail = 1; - - rrddim_add(st, "backlog", NULL, 1, 10, RRDDIM_INCREMENTAL); - } - else rrdset_next_usec(st, dt); - - rrddim_set(st, "backlog", backlog_ms); - rrdset_done(st); - } - - // -------------------------------------------------------------------- - - if(ddo_util) { - st = rrdset_find_bytype("disk_util", disk); - if(!st) { - st = rrdset_create("disk_util", disk, NULL, family, "disk.util", "Disk Utilization Time", "% of time working", 2004, update_every, RRDSET_TYPE_AREA); - st->isdetail = 1; - - rrddim_add(st, "utilization", NULL, 1, 10, RRDDIM_INCREMENTAL); - } - else rrdset_next_usec(st, dt); - - last_busy_ms = rrddim_set(st, "utilization", busy_ms); - rrdset_done(st); - } - - // -------------------------------------------------------------------- - - if(ddo_mops) { - st = rrdset_find_bytype("disk_mops", disk); - if(!st) { - st = rrdset_create("disk_mops", disk, NULL, family, "disk.mops", "Disk Merged Operations", "merged operations/s", 2021, update_every, RRDSET_TYPE_LINE); - st->isdetail = 1; - - rrddim_add(st, "reads", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "writes", NULL, -1, 1, RRDDIM_INCREMENTAL); - } - else rrdset_next_usec(st, dt); - - rrddim_set(st, "reads", mreads); - rrddim_set(st, "writes", mwrites); - rrdset_done(st); - } - - // -------------------------------------------------------------------- - - if(ddo_iotime) { - st = rrdset_find_bytype("disk_iotime", disk); - if(!st) { - st = rrdset_create("disk_iotime", disk, NULL, family, "disk.iotime", "Disk Total I/O Time", "milliseconds/s", 2022, update_every, RRDSET_TYPE_LINE); - st->isdetail = 1; - - rrddim_add(st, "reads", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "writes", NULL, -1, 1, RRDDIM_INCREMENTAL); - } - else rrdset_next_usec(st, dt); - - last_readms = rrddim_set(st, "reads", readms); - last_writems = rrddim_set(st, "writes", writems); - rrdset_done(st); - } - - // -------------------------------------------------------------------- - // calculate differential charts - // only if this is not the first time we run - - if(dt) { - if(ddo_iotime && ddo_ops) { - st = rrdset_find_bytype("disk_await", disk); - if(!st) { - st = rrdset_create("disk_await", disk, NULL, family, "disk.await", "Average Completed I/O Operation Time", "ms per operation", 2005, update_every, RRDSET_TYPE_LINE); - st->isdetail = 1; - - rrddim_add(st, "reads", NULL, 1, 1, RRDDIM_ABSOLUTE); - rrddim_add(st, "writes", NULL, -1, 1, RRDDIM_ABSOLUTE); - } - else rrdset_next_usec(st, dt); - - rrddim_set(st, "reads", (reads - last_reads) ? (readms - last_readms) / (reads - last_reads) : 0); - rrddim_set(st, "writes", (writes - last_writes) ? (writems - last_writems) / (writes - last_writes) : 0); - rrdset_done(st); - } - - if(ddo_io && ddo_ops) { - st = rrdset_find_bytype("disk_avgsz", disk); - if(!st) { - st = rrdset_create("disk_avgsz", disk, NULL, family, "disk.avgsz", "Average Completed I/O Operation Bandwidth", "kilobytes per operation", 2006, update_every, RRDSET_TYPE_AREA); - st->isdetail = 1; - - rrddim_add(st, "reads", NULL, sector_size, 1024, RRDDIM_ABSOLUTE); - rrddim_add(st, "writes", NULL, -sector_size, 1024, RRDDIM_ABSOLUTE); - } - else rrdset_next_usec(st, dt); - - rrddim_set(st, "reads", (reads - last_reads) ? (readsectors - last_readsectors) / (reads - last_reads) : 0); - rrddim_set(st, "writes", (writes - last_writes) ? (writesectors - last_writesectors) / (writes - last_writes) : 0); - rrdset_done(st); - } - - if(ddo_util && ddo_ops) { - st = rrdset_find_bytype("disk_svctm", disk); - if(!st) { - st = rrdset_create("disk_svctm", disk, NULL, family, "disk.svctm", "Average Service Time", "ms per operation", 2007, update_every, RRDSET_TYPE_LINE); - st->isdetail = 1; - - rrddim_add(st, "svctm", NULL, 1, 1, RRDDIM_ABSOLUTE); - } - else rrdset_next_usec(st, dt); - - rrddim_set(st, "svctm", ((reads - last_reads) + (writes - last_writes)) ? (busy_ms - last_busy_ms) / ((reads - last_reads) + (writes - last_writes)) : 0); - rrdset_done(st); - } - } - } - - return 0; + static procfile *ff = NULL; + static struct statvfs buff_statvfs; + static struct stat buff_stat; + static int global_enable_new_disks_detected_at_runtime = CONFIG_ONDEMAND_YES, + global_enable_performance_for_physical_disks = CONFIG_ONDEMAND_ONDEMAND, + global_enable_performance_for_virtual_disks = CONFIG_ONDEMAND_NO, + global_enable_performance_for_partitions = CONFIG_ONDEMAND_NO, + global_enable_performance_for_mountpoints = CONFIG_ONDEMAND_NO, + global_enable_performance_for_virtual_mountpoints = CONFIG_ONDEMAND_ONDEMAND, + global_enable_space_for_mountpoints = CONFIG_ONDEMAND_ONDEMAND, + global_do_io = CONFIG_ONDEMAND_ONDEMAND, + global_do_ops = CONFIG_ONDEMAND_ONDEMAND, + global_do_mops = CONFIG_ONDEMAND_ONDEMAND, + global_do_iotime = CONFIG_ONDEMAND_ONDEMAND, + global_do_qops = CONFIG_ONDEMAND_ONDEMAND, + global_do_util = CONFIG_ONDEMAND_ONDEMAND, + global_do_backlog = CONFIG_ONDEMAND_ONDEMAND, + global_do_space = CONFIG_ONDEMAND_ONDEMAND, + global_do_inodes = CONFIG_ONDEMAND_ONDEMAND, + globals_initialized = 0; + + if(unlikely(!globals_initialized)) { + global_enable_new_disks_detected_at_runtime = config_get_boolean("plugin:proc:/proc/diskstats", "enable new disks detected at runtime", global_enable_new_disks_detected_at_runtime); + + global_enable_performance_for_physical_disks = config_get_boolean_ondemand("plugin:proc:/proc/diskstats", "performance metrics for physical disks", global_enable_performance_for_physical_disks); + global_enable_performance_for_virtual_disks = config_get_boolean_ondemand("plugin:proc:/proc/diskstats", "performance metrics for virtual disks", global_enable_performance_for_virtual_disks); + global_enable_performance_for_partitions = config_get_boolean_ondemand("plugin:proc:/proc/diskstats", "performance metrics for partitions", global_enable_performance_for_partitions); + global_enable_performance_for_mountpoints = config_get_boolean_ondemand("plugin:proc:/proc/diskstats", "performance metrics for mounted filesystems", global_enable_performance_for_mountpoints); + global_enable_performance_for_virtual_mountpoints = config_get_boolean_ondemand("plugin:proc:/proc/diskstats", "performance metrics for mounted virtual disks", global_enable_performance_for_virtual_mountpoints); + global_enable_space_for_mountpoints = config_get_boolean_ondemand("plugin:proc:/proc/diskstats", "space metrics for mounted filesystems", global_enable_space_for_mountpoints); + + global_do_io = config_get_boolean_ondemand("plugin:proc:/proc/diskstats", "bandwidth for all disks", global_do_io); + global_do_ops = config_get_boolean_ondemand("plugin:proc:/proc/diskstats", "operations for all disks", global_do_ops); + global_do_mops = config_get_boolean_ondemand("plugin:proc:/proc/diskstats", "merged operations for all disks", global_do_mops); + global_do_iotime = config_get_boolean_ondemand("plugin:proc:/proc/diskstats", "i/o time for all disks", global_do_iotime); + global_do_qops = config_get_boolean_ondemand("plugin:proc:/proc/diskstats", "queued operations for all disks", global_do_qops); + global_do_util = config_get_boolean_ondemand("plugin:proc:/proc/diskstats", "utilization percentage for all disks", global_do_util); + global_do_backlog = config_get_boolean_ondemand("plugin:proc:/proc/diskstats", "backlog for all disks", global_do_backlog); + global_do_space = config_get_boolean_ondemand("plugin:proc:/proc/diskstats", "space usage for all disks", global_do_space); + global_do_inodes = config_get_boolean_ondemand("plugin:proc:/proc/diskstats", "inodes usage for all disks", global_do_inodes); + + globals_initialized = 1; + } + + if(!ff) { + char filename[FILENAME_MAX + 1]; + snprintfz(filename, FILENAME_MAX, "%s%s", global_host_prefix, "/proc/diskstats"); + ff = procfile_open(config_get("plugin:proc:/proc/diskstats", "filename to monitor", filename), " \t", PROCFILE_FLAG_DEFAULT); + } + if(!ff) return 1; + + ff = procfile_readall(ff); + if(!ff) return 0; // we return 0, so that we will retry to open it next time + + uint32_t lines = procfile_lines(ff), l; + uint32_t words; + + for(l = 0; l < lines ;l++) { + // -------------------------------------------------------------------------- + // Read parameters + + char *disk; + unsigned long long major = 0, minor = 0, + reads = 0, mreads = 0, readsectors = 0, readms = 0, + writes = 0, mwrites = 0, writesectors = 0, writems = 0, + queued_ios = 0, busy_ms = 0, backlog_ms = 0, + space_avail = 0, space_avail_root = 0, space_used = 0, + inodes_avail = 0, inodes_avail_root = 0, inodes_used = 0; + + unsigned long long last_reads = 0, last_readsectors = 0, last_readms = 0, + last_writes = 0, last_writesectors = 0, last_writems = 0, + last_busy_ms = 0; + + words = procfile_linewords(ff, l); + if(words < 14) continue; + + major = strtoull(procfile_lineword(ff, l, 0), NULL, 10); + minor = strtoull(procfile_lineword(ff, l, 1), NULL, 10); + disk = procfile_lineword(ff, l, 2); + + // # of reads completed # of writes completed + // This is the total number of reads or writes completed successfully. + reads = strtoull(procfile_lineword(ff, l, 3), NULL, 10); // rd_ios + writes = strtoull(procfile_lineword(ff, l, 7), NULL, 10); // wr_ios + + // # of reads merged # of writes merged + // Reads and writes which are adjacent to each other may be merged for + // efficiency. Thus two 4K reads may become one 8K read before it is + // ultimately handed to the disk, and so it will be counted (and queued) + mreads = strtoull(procfile_lineword(ff, l, 4), NULL, 10); // rd_merges_or_rd_sec + mwrites = strtoull(procfile_lineword(ff, l, 8), NULL, 10); // wr_merges + + // # of sectors read # of sectors written + // This is the total number of sectors read or written successfully. + readsectors = strtoull(procfile_lineword(ff, l, 5), NULL, 10); // rd_sec_or_wr_ios + writesectors = strtoull(procfile_lineword(ff, l, 9), NULL, 10); // wr_sec + + // # of milliseconds spent reading # of milliseconds spent writing + // This is the total number of milliseconds spent by all reads or writes (as + // measured from __make_request() to end_that_request_last()). + readms = strtoull(procfile_lineword(ff, l, 6), NULL, 10); // rd_ticks_or_wr_sec + writems = strtoull(procfile_lineword(ff, l, 10), NULL, 10); // wr_ticks + + // # of I/Os currently in progress + // The only field that should go to zero. Incremented as requests are + // given to appropriate struct request_queue and decremented as they finish. + queued_ios = strtoull(procfile_lineword(ff, l, 11), NULL, 10); // ios_pgr + + // # of milliseconds spent doing I/Os + // This field increases so long as field queued_ios is nonzero. + busy_ms = strtoull(procfile_lineword(ff, l, 12), NULL, 10); // tot_ticks + + // weighted # of milliseconds spent doing I/Os + // This field is incremented at each I/O start, I/O completion, I/O + // merge, or read of these stats by the number of I/Os in progress + // (field queued_ios) times the number of milliseconds spent doing I/O since the + // last update of this field. This can provide an easy measure of both + // I/O completion time and the backlog that may be accumulating. + backlog_ms = strtoull(procfile_lineword(ff, l, 13), NULL, 10); // rq_ticks + + + // -------------------------------------------------------------------------- + // remove slashes from disk names + char *s; + for(s = disk; *s ;s++) + if(*s == '/') *s = '_'; + + // -------------------------------------------------------------------------- + // get a disk structure for the disk + + struct disk *d = get_disk(major, minor, disk); + + + // -------------------------------------------------------------------------- + // Set its family based on mount point + + char *family = d->mount_point; + if(!family) family = disk; + + + // -------------------------------------------------------------------------- + // Check the configuration for the device + + if(unlikely(!d->configured)) { + char var_name[4096 + 1]; + snprintfz(var_name, 4096, "plugin:proc:/proc/diskstats:%s", disk); + + int def_enable = config_get_boolean_ondemand(var_name, "enable", global_enable_new_disks_detected_at_runtime); + if(def_enable == CONFIG_ONDEMAND_NO) { + // the user does not want any metrics for this disk + d->do_io = CONFIG_ONDEMAND_NO; + d->do_ops = CONFIG_ONDEMAND_NO; + d->do_mops = CONFIG_ONDEMAND_NO; + d->do_iotime = CONFIG_ONDEMAND_NO; + d->do_qops = CONFIG_ONDEMAND_NO; + d->do_util = CONFIG_ONDEMAND_NO; + d->do_backlog = CONFIG_ONDEMAND_NO; + d->do_space = CONFIG_ONDEMAND_NO; + d->do_inodes = CONFIG_ONDEMAND_NO; + } + else { + // this disk is enabled + // check its direct settings + + int def_performance = CONFIG_ONDEMAND_ONDEMAND; + int def_space = (d->mount_point)?CONFIG_ONDEMAND_ONDEMAND:CONFIG_ONDEMAND_NO; + + // since this is 'on demand' we can figure the performance settings + // based on the type of disk + + switch(d->type) { + case DISK_TYPE_PHYSICAL: + def_performance = global_enable_performance_for_physical_disks; + break; + + case DISK_TYPE_PARTITION: + def_performance = global_enable_performance_for_partitions; + break; + + case DISK_TYPE_CONTAINER: + def_performance = global_enable_performance_for_virtual_disks; + + if(d->mount_point) + def_performance = select_positive_option(def_performance, global_enable_performance_for_virtual_mountpoints); + + break; + } + + if(d->mount_point) + def_performance = select_positive_option(def_performance, global_enable_performance_for_mountpoints); + + // ------------------------------------------------------------ + // now we have def_performance and def_space + // to work further + + // def_performance + // check the user configuration (this will also show our 'on demand' decision) + def_performance = config_get_boolean_ondemand(var_name, "enable performance metrics", def_performance); + + int ddo_io = CONFIG_ONDEMAND_NO, + ddo_ops = CONFIG_ONDEMAND_NO, + ddo_mops = CONFIG_ONDEMAND_NO, + ddo_iotime = CONFIG_ONDEMAND_NO, + ddo_qops = CONFIG_ONDEMAND_NO, + ddo_util = CONFIG_ONDEMAND_NO, + ddo_backlog = CONFIG_ONDEMAND_NO; + + // we enable individual performance charts only when def_performance is not disabled + if(def_performance != CONFIG_ONDEMAND_NO) { + ddo_io = global_do_io, + ddo_ops = global_do_ops, + ddo_mops = global_do_mops, + ddo_iotime = global_do_iotime, + ddo_qops = global_do_qops, + ddo_util = global_do_util, + ddo_backlog = global_do_backlog; + } + + d->do_io = config_get_boolean_ondemand(var_name, "bandwidth", ddo_io); + d->do_ops = config_get_boolean_ondemand(var_name, "operations", ddo_ops); + d->do_mops = config_get_boolean_ondemand(var_name, "merged operations", ddo_mops); + d->do_iotime = config_get_boolean_ondemand(var_name, "i/o time", ddo_iotime); + d->do_qops = config_get_boolean_ondemand(var_name, "queued operations", ddo_qops); + d->do_util = config_get_boolean_ondemand(var_name, "utilization percentage", ddo_util); + d->do_backlog = config_get_boolean_ondemand(var_name, "backlog", ddo_backlog); + + // def_space + if(d->mount_point) { + // check the user configuration (this will also show our 'on demand' decision) + def_space = config_get_boolean_ondemand(var_name, "enable space metrics", def_space); + + int ddo_space = def_space, + ddo_inodes = def_space; + + d->do_space = config_get_boolean_ondemand(var_name, "space usage", ddo_space); + d->do_inodes = config_get_boolean_ondemand(var_name, "inodes usage", ddo_inodes); + } + else { + // don't show settings for this disk + d->do_space = CONFIG_ONDEMAND_NO; + d->do_inodes = CONFIG_ONDEMAND_NO; + } + } + + d->configured = 1; + } + + RRDSET *st; + + // -------------------------------------------------------------------------- + // Do performance metrics + + if(d->do_io == CONFIG_ONDEMAND_YES || (d->do_io == CONFIG_ONDEMAND_ONDEMAND && (readsectors || writesectors))) { + d->do_io = CONFIG_ONDEMAND_YES; + + st = rrdset_find_bytype(RRD_TYPE_DISK, disk); + if(!st) { + st = rrdset_create(RRD_TYPE_DISK, disk, NULL, family, "disk.io", "Disk I/O Bandwidth", "kilobytes/s", 2000, update_every, RRDSET_TYPE_AREA); + + rrddim_add(st, "reads", NULL, d->sector_size, 1024, RRDDIM_INCREMENTAL); + rrddim_add(st, "writes", NULL, d->sector_size * -1, 1024, RRDDIM_INCREMENTAL); + } + else rrdset_next_usec(st, dt); + + last_readsectors = rrddim_set(st, "reads", readsectors); + last_writesectors = rrddim_set(st, "writes", writesectors); + rrdset_done(st); + } + + // -------------------------------------------------------------------- + + if(d->do_ops == CONFIG_ONDEMAND_YES || (d->do_ops == CONFIG_ONDEMAND_ONDEMAND && (reads || writes))) { + d->do_ops = CONFIG_ONDEMAND_YES; + + st = rrdset_find_bytype("disk_ops", disk); + if(!st) { + st = rrdset_create("disk_ops", disk, NULL, family, "disk.ops", "Disk Completed I/O Operations", "operations/s", 2001, update_every, RRDSET_TYPE_LINE); + st->isdetail = 1; + + rrddim_add(st, "reads", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "writes", NULL, -1, 1, RRDDIM_INCREMENTAL); + } + else rrdset_next_usec(st, dt); + + last_reads = rrddim_set(st, "reads", reads); + last_writes = rrddim_set(st, "writes", writes); + rrdset_done(st); + } + + // -------------------------------------------------------------------- + + if(d->do_qops == CONFIG_ONDEMAND_YES || (d->do_qops == CONFIG_ONDEMAND_ONDEMAND && queued_ios)) { + d->do_qops = CONFIG_ONDEMAND_YES; + + st = rrdset_find_bytype("disk_qops", disk); + if(!st) { + st = rrdset_create("disk_qops", disk, NULL, family, "disk.qops", "Disk Current I/O Operations", "operations", 2002, update_every, RRDSET_TYPE_LINE); + st->isdetail = 1; + + rrddim_add(st, "operations", NULL, 1, 1, RRDDIM_ABSOLUTE); + } + else rrdset_next_usec(st, dt); + + rrddim_set(st, "operations", queued_ios); + rrdset_done(st); + } + + // -------------------------------------------------------------------- + + if(d->do_backlog == CONFIG_ONDEMAND_YES || (d->do_backlog == CONFIG_ONDEMAND_ONDEMAND && backlog_ms)) { + d->do_backlog = CONFIG_ONDEMAND_YES; + + st = rrdset_find_bytype("disk_backlog", disk); + if(!st) { + st = rrdset_create("disk_backlog", disk, NULL, family, "disk.backlog", "Disk Backlog", "backlog (ms)", 2003, update_every, RRDSET_TYPE_AREA); + st->isdetail = 1; + + rrddim_add(st, "backlog", NULL, 1, 10, RRDDIM_INCREMENTAL); + } + else rrdset_next_usec(st, dt); + + rrddim_set(st, "backlog", backlog_ms); + rrdset_done(st); + } + + // -------------------------------------------------------------------- + + if(d->do_util == CONFIG_ONDEMAND_YES || (d->do_util == CONFIG_ONDEMAND_ONDEMAND && busy_ms)) { + d->do_util = CONFIG_ONDEMAND_YES; + + st = rrdset_find_bytype("disk_util", disk); + if(!st) { + st = rrdset_create("disk_util", disk, NULL, family, "disk.util", "Disk Utilization Time", "% of time working", 2004, update_every, RRDSET_TYPE_AREA); + st->isdetail = 1; + + rrddim_add(st, "utilization", NULL, 1, 10, RRDDIM_INCREMENTAL); + } + else rrdset_next_usec(st, dt); + + last_busy_ms = rrddim_set(st, "utilization", busy_ms); + rrdset_done(st); + } + + // -------------------------------------------------------------------- + + if(d->do_mops == CONFIG_ONDEMAND_YES || (d->do_mops == CONFIG_ONDEMAND_ONDEMAND && (mreads || mwrites))) { + d->do_mops = CONFIG_ONDEMAND_YES; + + st = rrdset_find_bytype("disk_mops", disk); + if(!st) { + st = rrdset_create("disk_mops", disk, NULL, family, "disk.mops", "Disk Merged Operations", "merged operations/s", 2021, update_every, RRDSET_TYPE_LINE); + st->isdetail = 1; + + rrddim_add(st, "reads", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "writes", NULL, -1, 1, RRDDIM_INCREMENTAL); + } + else rrdset_next_usec(st, dt); + + rrddim_set(st, "reads", mreads); + rrddim_set(st, "writes", mwrites); + rrdset_done(st); + } + + // -------------------------------------------------------------------- + + if(d->do_iotime == CONFIG_ONDEMAND_YES || (d->do_iotime == CONFIG_ONDEMAND_ONDEMAND && (readms || writems))) { + d->do_iotime = CONFIG_ONDEMAND_YES; + + st = rrdset_find_bytype("disk_iotime", disk); + if(!st) { + st = rrdset_create("disk_iotime", disk, NULL, family, "disk.iotime", "Disk Total I/O Time", "milliseconds/s", 2022, update_every, RRDSET_TYPE_LINE); + st->isdetail = 1; + + rrddim_add(st, "reads", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "writes", NULL, -1, 1, RRDDIM_INCREMENTAL); + } + else rrdset_next_usec(st, dt); + + last_readms = rrddim_set(st, "reads", readms); + last_writems = rrddim_set(st, "writes", writems); + rrdset_done(st); + } + + // -------------------------------------------------------------------- + // calculate differential charts + // only if this is not the first time we run + + if(dt) { + if( (d->do_iotime == CONFIG_ONDEMAND_YES || (d->do_iotime == CONFIG_ONDEMAND_ONDEMAND && (readms || writems))) && + (d->do_ops == CONFIG_ONDEMAND_YES || (d->do_ops == CONFIG_ONDEMAND_ONDEMAND && (reads || writes)))) { + st = rrdset_find_bytype("disk_await", disk); + if(!st) { + st = rrdset_create("disk_await", disk, NULL, family, "disk.await", "Average Completed I/O Operation Time", "ms per operation", 2005, update_every, RRDSET_TYPE_LINE); + st->isdetail = 1; + + rrddim_add(st, "reads", NULL, 1, 1, RRDDIM_ABSOLUTE); + rrddim_add(st, "writes", NULL, -1, 1, RRDDIM_ABSOLUTE); + } + else rrdset_next_usec(st, dt); + + rrddim_set(st, "reads", (reads - last_reads) ? (readms - last_readms) / (reads - last_reads) : 0); + rrddim_set(st, "writes", (writes - last_writes) ? (writems - last_writems) / (writes - last_writes) : 0); + rrdset_done(st); + } + + if( (d->do_io == CONFIG_ONDEMAND_YES || (d->do_io == CONFIG_ONDEMAND_ONDEMAND && (readsectors || writesectors))) && + (d->do_ops == CONFIG_ONDEMAND_YES || (d->do_ops == CONFIG_ONDEMAND_ONDEMAND && (reads || writes)))) { + st = rrdset_find_bytype("disk_avgsz", disk); + if(!st) { + st = rrdset_create("disk_avgsz", disk, NULL, family, "disk.avgsz", "Average Completed I/O Operation Bandwidth", "kilobytes per operation", 2006, update_every, RRDSET_TYPE_AREA); + st->isdetail = 1; + + rrddim_add(st, "reads", NULL, d->sector_size, 1024, RRDDIM_ABSOLUTE); + rrddim_add(st, "writes", NULL, d->sector_size * -1, 1024, RRDDIM_ABSOLUTE); + } + else rrdset_next_usec(st, dt); + + rrddim_set(st, "reads", (reads - last_reads) ? (readsectors - last_readsectors) / (reads - last_reads) : 0); + rrddim_set(st, "writes", (writes - last_writes) ? (writesectors - last_writesectors) / (writes - last_writes) : 0); + rrdset_done(st); + } + + if( (d->do_util == CONFIG_ONDEMAND_YES || (d->do_util == CONFIG_ONDEMAND_ONDEMAND && busy_ms)) && + (d->do_ops == CONFIG_ONDEMAND_YES || (d->do_ops == CONFIG_ONDEMAND_ONDEMAND && (reads || writes)))) { + st = rrdset_find_bytype("disk_svctm", disk); + if(!st) { + st = rrdset_create("disk_svctm", disk, NULL, family, "disk.svctm", "Average Service Time", "ms per operation", 2007, update_every, RRDSET_TYPE_LINE); + st->isdetail = 1; + + rrddim_add(st, "svctm", NULL, 1, 1, RRDDIM_ABSOLUTE); + } + else rrdset_next_usec(st, dt); + + rrddim_set(st, "svctm", ((reads - last_reads) + (writes - last_writes)) ? (busy_ms - last_busy_ms) / ((reads - last_reads) + (writes - last_writes)) : 0); + rrdset_done(st); + } + } + + // -------------------------------------------------------------------------- + // space metrics + + if(d->mount_point && (d->do_space || d->do_inodes) ) { + // collect space metrics using statvfs + + if (statvfs(d->mount_point, &buff_statvfs) < 0) + error("Failed statvfs() for '%s' (disk '%s')", d->mount_point, d->disk); + else { + space_avail = buff_statvfs.f_bavail * buff_statvfs.f_bsize; + space_avail_root = (buff_statvfs.f_bfree - buff_statvfs.f_bavail) * buff_statvfs.f_bsize; + space_used = (buff_statvfs.f_blocks - buff_statvfs.f_bfree) * buff_statvfs.f_bsize; + + inodes_avail = buff_statvfs.f_favail; + inodes_avail_root = buff_statvfs.f_ffree - buff_statvfs.f_favail; + inodes_used = buff_statvfs.f_files - buff_statvfs.f_ffree; + + // verify we collected the metrics for the right disk. + // if not the mountpoint has changed. + + if(stat(d->mount_point, &buff_stat) == -1) + error("Failed to stat() for '%s' (disk '%s')", d->mount_point, d->disk); + else { + if(major(buff_stat.st_dev) == major && minor(buff_stat.st_dev) == minor) { + + // -------------------------------------------------------------------------- + + if(d->do_space == CONFIG_ONDEMAND_YES || (d->do_space == CONFIG_ONDEMAND_ONDEMAND && (space_avail || space_avail_root || space_used))) { + st = rrdset_find_bytype("disk_space", disk); + if(!st) { + st = rrdset_create("disk_space", disk, NULL, family, "disk.space", "Disk Space Usage", "GB", 2023, update_every, RRDSET_TYPE_STACKED); + st->isdetail = 1; + + rrddim_add(st, "avail", NULL, 1, 1024*1024*1024, RRDDIM_ABSOLUTE); + rrddim_add(st, "used" , NULL, 1, 1024*1024*1024, RRDDIM_ABSOLUTE); + rrddim_add(st, "reserved_for_root", "reserved for root", 1, 1024*1024*1024, RRDDIM_ABSOLUTE); + } + else rrdset_next_usec(st, dt); + + rrddim_set(st, "avail", space_avail); + rrddim_set(st, "used", space_used); + rrddim_set(st, "reserved_for_root", space_avail_root); + rrdset_done(st); + } + + // -------------------------------------------------------------------------- + + if(d->do_inodes == CONFIG_ONDEMAND_YES || (d->do_inodes == CONFIG_ONDEMAND_ONDEMAND && (inodes_avail || inodes_avail_root || inodes_used))) { + st = rrdset_find_bytype("disk_inodes", disk); + if(!st) { + st = rrdset_create("disk_inodes", disk, NULL, family, "disk.inodes", "Disk Inodes Usage", "Inodes", 2024, update_every, RRDSET_TYPE_STACKED); + st->isdetail = 1; + + rrddim_add(st, "avail", NULL, 1, 1, RRDDIM_ABSOLUTE); + rrddim_add(st, "used" , NULL, 1, 1, RRDDIM_ABSOLUTE); + rrddim_add(st, "reserved_for_root", "reserved for root", 1, 1, RRDDIM_ABSOLUTE); + } + else rrdset_next_usec(st, dt); + + rrddim_set(st, "avail", inodes_avail); + rrddim_set(st, "used", inodes_used); + rrddim_set(st, "reserved_for_root", inodes_avail_root); + rrdset_done(st); + } + } + } + } + } + } + + return 0; } diff --git a/src/proc_interrupts.c b/src/proc_interrupts.c index ad00c2022..fe0e9b1a5 100644 --- a/src/proc_interrupts.c +++ b/src/proc_interrupts.c @@ -1,26 +1,13 @@ -#ifdef HAVE_CONFIG_H -#include <config.h> -#endif -#include <stdio.h> -#include <stdlib.h> -#include <string.h> -#include <ctype.h> - #include "common.h" -#include "appconfig.h" -#include "procfile.h" -#include "rrd.h" -#include "plugin_proc.h" -#include "log.h" #define MAX_INTERRUPT_NAME 50 struct interrupt { - int used; - char *id; - char name[MAX_INTERRUPT_NAME + 1]; - unsigned long long total; - unsigned long long value[]; + int used; + char *id; + char name[MAX_INTERRUPT_NAME + 1]; + unsigned long long total; + unsigned long long value[]; }; // since each interrupt is variable in size @@ -31,160 +18,157 @@ struct interrupt { #define irrindex(base, line, cpus) ((struct interrupt *)&((char *)(base))[line * recordsize(cpus)]) static inline struct interrupt *get_interrupts_array(int lines, int cpus) { - static struct interrupt *irrs = NULL; - static int allocated = 0; - - if(lines < allocated) return irrs; - else { - irrs = (struct interrupt *)realloc(irrs, lines * recordsize(cpus)); - if(!irrs) - fatal("Cannot allocate memory for %d interrupts", lines); + static struct interrupt *irrs = NULL; + static int allocated = 0; - allocated = lines; - } + if(lines < allocated) return irrs; + else { + irrs = (struct interrupt *)reallocz(irrs, lines * recordsize(cpus)); + allocated = lines; + } - return irrs; + return irrs; } int do_proc_interrupts(int update_every, unsigned long long dt) { - static procfile *ff = NULL; - static int cpus = -1, do_per_core = -1; - struct interrupt *irrs = NULL; - - if(dt) {}; - - if(do_per_core == -1) do_per_core = config_get_boolean("plugin:proc:/proc/interrupts", "interrupts per core", 1); - - if(!ff) { - char filename[FILENAME_MAX + 1]; - snprintfz(filename, FILENAME_MAX, "%s%s", global_host_prefix, "/proc/interrupts"); - ff = procfile_open(config_get("plugin:proc:/proc/interrupts", "filename to monitor", filename), " \t", PROCFILE_FLAG_DEFAULT); - } - if(!ff) return 1; - - ff = procfile_readall(ff); - if(!ff) return 0; // we return 0, so that we will retry to open it next time - - uint32_t lines = procfile_lines(ff), l; - uint32_t words = procfile_linewords(ff, 0), w; - - if(!lines) { - error("Cannot read /proc/interrupts, zero lines reported."); - return 1; - } - - // find how many CPUs are there - if(cpus == -1) { - cpus = 0; - for(w = 0; w < words ; w++) { - if(strncmp(procfile_lineword(ff, 0, w), "CPU", 3) == 0) - cpus++; - } - } - - if(!cpus) { - error("PLUGIN: PROC_INTERRUPTS: Cannot find the number of CPUs in /proc/interrupts"); - return 1; - } - - // allocate the size we need; - irrs = get_interrupts_array(lines, cpus); - irrs[0].used = 0; - - // loop through all lines - for(l = 1; l < lines ;l++) { - struct interrupt *irr = irrindex(irrs, l, cpus); - irr->used = 0; - irr->total = 0; - - words = procfile_linewords(ff, l); - if(!words) continue; - - irr->id = procfile_lineword(ff, l, 0); - if(!irr->id || !irr->id[0]) continue; - - int idlen = strlen(irr->id); - if(irr->id[idlen - 1] == ':') - irr->id[idlen - 1] = '\0'; - - int c; - for(c = 0; c < cpus ;c++) { - if((c + 1) < (int)words) - irr->value[c] = strtoull(procfile_lineword(ff, l, (uint32_t)(c + 1)), NULL, 10); - else - irr->value[c] = 0; - - irr->total += irr->value[c]; - } - - if(isdigit(irr->id[0]) && (uint32_t)(cpus + 2) < words) { - strncpyz(irr->name, procfile_lineword(ff, l, words - 1), MAX_INTERRUPT_NAME); - int nlen = strlen(irr->name); - if(nlen < (MAX_INTERRUPT_NAME-1)) { - irr->name[nlen] = '_'; - strncpyz(&irr->name[nlen + 1], irr->id, MAX_INTERRUPT_NAME - nlen); - } - } - else { - strncpyz(irr->name, irr->id, MAX_INTERRUPT_NAME); - } - - irr->used = 1; - } - - RRDSET *st; - - // -------------------------------------------------------------------- - - st = rrdset_find_bytype("system", "interrupts"); - if(!st) { - st = rrdset_create("system", "interrupts", NULL, "interrupts", NULL, "System interrupts", "interrupts/s", 1000, update_every, RRDSET_TYPE_STACKED); - - for(l = 0; l < lines ;l++) { - struct interrupt *irr = irrindex(irrs, l, cpus); - if(!irr->used) continue; - rrddim_add(st, irr->id, irr->name, 1, 1, RRDDIM_INCREMENTAL); - } - } - else rrdset_next(st); - - for(l = 0; l < lines ;l++) { - struct interrupt *irr = irrindex(irrs, l, cpus); - if(!irr->used) continue; - rrddim_set(st, irr->id, irr->total); - } - rrdset_done(st); - - if(do_per_core) { - int c; - - for(c = 0; c < cpus ; c++) { - char id[256+1]; - snprintfz(id, 256, "cpu%d_interrupts", c); - - st = rrdset_find_bytype("cpu", id); - if(!st) { - char name[256+1], title[256+1]; - snprintfz(name, 256, "cpu%d_interrupts", c); - snprintfz(title, 256, "CPU%d Interrupts", c); - st = rrdset_create("cpu", id, name, "interrupts", "cpu.interrupts", title, "interrupts/s", 2000 + c, update_every, RRDSET_TYPE_STACKED); - - for(l = 0; l < lines ;l++) { - struct interrupt *irr = irrindex(irrs, l, cpus); - if(!irr->used) continue; - rrddim_add(st, irr->id, irr->name, 1, 1, RRDDIM_INCREMENTAL); - } - } - else rrdset_next(st); - - for(l = 0; l < lines ;l++) { - struct interrupt *irr = irrindex(irrs, l, cpus); - if(!irr->used) continue; - rrddim_set(st, irr->id, irr->value[c]); - } - rrdset_done(st); - } - } - - return 0; + static procfile *ff = NULL; + static int cpus = -1, do_per_core = -1; + struct interrupt *irrs = NULL; + + if(dt) {}; + + if(do_per_core == -1) do_per_core = config_get_boolean("plugin:proc:/proc/interrupts", "interrupts per core", 1); + + if(!ff) { + char filename[FILENAME_MAX + 1]; + snprintfz(filename, FILENAME_MAX, "%s%s", global_host_prefix, "/proc/interrupts"); + ff = procfile_open(config_get("plugin:proc:/proc/interrupts", "filename to monitor", filename), " \t", PROCFILE_FLAG_DEFAULT); + } + if(!ff) return 1; + + ff = procfile_readall(ff); + if(!ff) return 0; // we return 0, so that we will retry to open it next time + + uint32_t lines = procfile_lines(ff), l; + uint32_t words = procfile_linewords(ff, 0), w; + + if(!lines) { + error("Cannot read /proc/interrupts, zero lines reported."); + return 1; + } + + // find how many CPUs are there + if(cpus == -1) { + cpus = 0; + for(w = 0; w < words ; w++) { + if(strncmp(procfile_lineword(ff, 0, w), "CPU", 3) == 0) + cpus++; + } + } + + if(!cpus) { + error("PLUGIN: PROC_INTERRUPTS: Cannot find the number of CPUs in /proc/interrupts"); + return 1; + } + + // allocate the size we need; + irrs = get_interrupts_array(lines, cpus); + irrs[0].used = 0; + + // loop through all lines + for(l = 1; l < lines ;l++) { + struct interrupt *irr = irrindex(irrs, l, cpus); + irr->used = 0; + irr->total = 0; + + words = procfile_linewords(ff, l); + if(!words) continue; + + irr->id = procfile_lineword(ff, l, 0); + if(!irr->id || !irr->id[0]) continue; + + int idlen = strlen(irr->id); + if(irr->id[idlen - 1] == ':') + irr->id[idlen - 1] = '\0'; + + int c; + for(c = 0; c < cpus ;c++) { + if((c + 1) < (int)words) + irr->value[c] = strtoull(procfile_lineword(ff, l, (uint32_t)(c + 1)), NULL, 10); + else + irr->value[c] = 0; + + irr->total += irr->value[c]; + } + + if(isdigit(irr->id[0]) && (uint32_t)(cpus + 2) < words) { + strncpyz(irr->name, procfile_lineword(ff, l, words - 1), MAX_INTERRUPT_NAME); + int nlen = strlen(irr->name); + if(nlen < (MAX_INTERRUPT_NAME-1)) { + irr->name[nlen] = '_'; + strncpyz(&irr->name[nlen + 1], irr->id, MAX_INTERRUPT_NAME - nlen); + } + } + else { + strncpyz(irr->name, irr->id, MAX_INTERRUPT_NAME); + } + + irr->used = 1; + } + + RRDSET *st; + + // -------------------------------------------------------------------- + + st = rrdset_find_bytype("system", "interrupts"); + if(!st) { + st = rrdset_create("system", "interrupts", NULL, "interrupts", NULL, "System interrupts", "interrupts/s", 1000, update_every, RRDSET_TYPE_STACKED); + + for(l = 0; l < lines ;l++) { + struct interrupt *irr = irrindex(irrs, l, cpus); + if(!irr->used) continue; + rrddim_add(st, irr->id, irr->name, 1, 1, RRDDIM_INCREMENTAL); + } + } + else rrdset_next(st); + + for(l = 0; l < lines ;l++) { + struct interrupt *irr = irrindex(irrs, l, cpus); + if(!irr->used) continue; + rrddim_set(st, irr->id, irr->total); + } + rrdset_done(st); + + if(do_per_core) { + int c; + + for(c = 0; c < cpus ; c++) { + char id[256+1]; + snprintfz(id, 256, "cpu%d_interrupts", c); + + st = rrdset_find_bytype("cpu", id); + if(!st) { + char name[256+1], title[256+1]; + snprintfz(name, 256, "cpu%d_interrupts", c); + snprintfz(title, 256, "CPU%d Interrupts", c); + st = rrdset_create("cpu", id, name, "interrupts", "cpu.interrupts", title, "interrupts/s", 2000 + c, update_every, RRDSET_TYPE_STACKED); + + for(l = 0; l < lines ;l++) { + struct interrupt *irr = irrindex(irrs, l, cpus); + if(!irr->used) continue; + rrddim_add(st, irr->id, irr->name, 1, 1, RRDDIM_INCREMENTAL); + } + } + else rrdset_next(st); + + for(l = 0; l < lines ;l++) { + struct interrupt *irr = irrindex(irrs, l, cpus); + if(!irr->used) continue; + rrddim_set(st, irr->id, irr->value[c]); + } + rrdset_done(st); + } + } + + return 0; } diff --git a/src/proc_loadavg.c b/src/proc_loadavg.c index c8e893b99..44ea70191 100644 --- a/src/proc_loadavg.c +++ b/src/proc_loadavg.c @@ -1,89 +1,80 @@ -#ifdef HAVE_CONFIG_H -#include <config.h> -#endif -#include <stdio.h> -#include <stdlib.h> -#include <string.h> - -#include "log.h" #include "common.h" -#include "appconfig.h" -#include "procfile.h" -#include "rrd.h" -#include "plugin_proc.h" + +// linux calculates this once every 5 seconds +#define MIN_LOADAVG_UPDATE_EVERY 5 int do_proc_loadavg(int update_every, unsigned long long dt) { - static procfile *ff = NULL; - static int do_loadavg = -1, do_all_processes = -1; + static procfile *ff = NULL; + static int do_loadavg = -1, do_all_processes = -1; - if(dt) {}; + if(dt) {}; - if(!ff) { - char filename[FILENAME_MAX + 1]; - snprintfz(filename, FILENAME_MAX, "%s%s", global_host_prefix, "/proc/loadavg"); - ff = procfile_open(config_get("plugin:proc:/proc/loadavg", "filename to monitor", filename), " \t,:|/", PROCFILE_FLAG_DEFAULT); - } - if(!ff) return 1; + if(!ff) { + char filename[FILENAME_MAX + 1]; + snprintfz(filename, FILENAME_MAX, "%s%s", global_host_prefix, "/proc/loadavg"); + ff = procfile_open(config_get("plugin:proc:/proc/loadavg", "filename to monitor", filename), " \t,:|/", PROCFILE_FLAG_DEFAULT); + } + if(!ff) return 1; - ff = procfile_readall(ff); - if(!ff) return 0; // we return 0, so that we will retry to open it next time + ff = procfile_readall(ff); + if(!ff) return 0; // we return 0, so that we will retry to open it next time - if(do_loadavg == -1) do_loadavg = config_get_boolean("plugin:proc:/proc/loadavg", "enable load average", 1); - if(do_all_processes == -1) do_all_processes = config_get_boolean("plugin:proc:/proc/loadavg", "enable total processes", 1); + if(do_loadavg == -1) do_loadavg = config_get_boolean("plugin:proc:/proc/loadavg", "enable load average", 1); + if(do_all_processes == -1) do_all_processes = config_get_boolean("plugin:proc:/proc/loadavg", "enable total processes", 1); - if(procfile_lines(ff) < 1) { - error("/proc/loadavg has no lines."); - return 1; - } - if(procfile_linewords(ff, 0) < 6) { - error("/proc/loadavg has less than 6 words in it."); - return 1; - } + if(procfile_lines(ff) < 1) { + error("/proc/loadavg has no lines."); + return 1; + } + if(procfile_linewords(ff, 0) < 6) { + error("/proc/loadavg has less than 6 words in it."); + return 1; + } - double load1 = strtod(procfile_lineword(ff, 0, 0), NULL); - double load5 = strtod(procfile_lineword(ff, 0, 1), NULL); - double load15 = strtod(procfile_lineword(ff, 0, 2), NULL); + double load1 = strtod(procfile_lineword(ff, 0, 0), NULL); + double load5 = strtod(procfile_lineword(ff, 0, 1), NULL); + double load15 = strtod(procfile_lineword(ff, 0, 2), NULL); - //unsigned long long running_processes = strtoull(procfile_lineword(ff, 0, 3), NULL, 10); - unsigned long long active_processes = strtoull(procfile_lineword(ff, 0, 4), NULL, 10); - //unsigned long long next_pid = strtoull(procfile_lineword(ff, 0, 5), NULL, 10); + //unsigned long long running_processes = strtoull(procfile_lineword(ff, 0, 3), NULL, 10); + unsigned long long active_processes = strtoull(procfile_lineword(ff, 0, 4), NULL, 10); + //unsigned long long next_pid = strtoull(procfile_lineword(ff, 0, 5), NULL, 10); - RRDSET *st; + RRDSET *st; - // -------------------------------------------------------------------- + // -------------------------------------------------------------------- - if(do_loadavg) { - st = rrdset_find_byname("system.load"); - if(!st) { - st = rrdset_create("system", "load", NULL, "load", NULL, "System Load Average", "load", 100, update_every, RRDSET_TYPE_LINE); + if(do_loadavg) { + st = rrdset_find_byname("system.load"); + if(!st) { + st = rrdset_create("system", "load", NULL, "load", NULL, "System Load Average", "load", 100, (update_every < MIN_LOADAVG_UPDATE_EVERY)?MIN_LOADAVG_UPDATE_EVERY:update_every, RRDSET_TYPE_LINE); - rrddim_add(st, "load1", NULL, 1, 1000, RRDDIM_ABSOLUTE); - rrddim_add(st, "load5", NULL, 1, 1000, RRDDIM_ABSOLUTE); - rrddim_add(st, "load15", NULL, 1, 1000, RRDDIM_ABSOLUTE); - } - else rrdset_next(st); + rrddim_add(st, "load1", NULL, 1, 1000, RRDDIM_ABSOLUTE); + rrddim_add(st, "load5", NULL, 1, 1000, RRDDIM_ABSOLUTE); + rrddim_add(st, "load15", NULL, 1, 1000, RRDDIM_ABSOLUTE); + } + else rrdset_next(st); - rrddim_set(st, "load1", load1 * 1000); - rrddim_set(st, "load5", load5 * 1000); - rrddim_set(st, "load15", load15 * 1000); - rrdset_done(st); - } + rrddim_set(st, "load1", load1 * 1000); + rrddim_set(st, "load5", load5 * 1000); + rrddim_set(st, "load15", load15 * 1000); + rrdset_done(st); + } - // -------------------------------------------------------------------- + // -------------------------------------------------------------------- - if(do_all_processes) { - st = rrdset_find_byname("system.active_processes"); - if(!st) { - st = rrdset_create("system", "active_processes", NULL, "processes", NULL, "System Active Processes", "processes", 750, update_every, RRDSET_TYPE_LINE); + if(do_all_processes) { + st = rrdset_find_byname("system.active_processes"); + if(!st) { + st = rrdset_create("system", "active_processes", NULL, "processes", NULL, "System Active Processes", "processes", 750, update_every, RRDSET_TYPE_LINE); - rrddim_add(st, "active", NULL, 1, 1, RRDDIM_ABSOLUTE); - } - else rrdset_next(st); + rrddim_add(st, "active", NULL, 1, 1, RRDDIM_ABSOLUTE); + } + else rrdset_next(st); - rrddim_set(st, "active", active_processes); - rrdset_done(st); - } + rrddim_set(st, "active", active_processes); + rrdset_done(st); + } - return 0; + return 0; } diff --git a/src/proc_meminfo.c b/src/proc_meminfo.c index 611b4ed21..4295cd6da 100644 --- a/src/proc_meminfo.c +++ b/src/proc_meminfo.c @@ -1,254 +1,242 @@ -#ifdef HAVE_CONFIG_H -#include <config.h> -#endif -#include <stdio.h> -#include <stdlib.h> -#include <string.h> - #include "common.h" -#include "log.h" -#include "appconfig.h" -#include "procfile.h" -#include "rrd.h" -#include "plugin_proc.h" #define MAX_PROC_MEMINFO_LINE 4096 #define MAX_PROC_MEMINFO_NAME 1024 int do_proc_meminfo(int update_every, unsigned long long dt) { - static procfile *ff = NULL; - - static int do_ram = -1, do_swap = -1, do_hwcorrupt = -1, do_committed = -1, do_writeback = -1, do_kernel = -1, do_slab = -1; - - if(do_ram == -1) do_ram = config_get_boolean("plugin:proc:/proc/meminfo", "system ram", 1); - if(do_swap == -1) do_swap = config_get_boolean("plugin:proc:/proc/meminfo", "system swap", 1); - if(do_hwcorrupt == -1) do_hwcorrupt = config_get_boolean_ondemand("plugin:proc:/proc/meminfo", "hardware corrupted ECC", CONFIG_ONDEMAND_ONDEMAND); - if(do_committed == -1) do_committed = config_get_boolean("plugin:proc:/proc/meminfo", "committed memory", 1); - if(do_writeback == -1) do_writeback = config_get_boolean("plugin:proc:/proc/meminfo", "writeback memory", 1); - if(do_kernel == -1) do_kernel = config_get_boolean("plugin:proc:/proc/meminfo", "kernel memory", 1); - if(do_slab == -1) do_slab = config_get_boolean("plugin:proc:/proc/meminfo", "slab memory", 1); - - if(dt) {}; - - if(!ff) { - char filename[FILENAME_MAX + 1]; - snprintfz(filename, FILENAME_MAX, "%s%s", global_host_prefix, "/proc/meminfo"); - ff = procfile_open(config_get("plugin:proc:/proc/meminfo", "filename to monitor", filename), " \t:", PROCFILE_FLAG_DEFAULT); - } - if(!ff) return 1; - - ff = procfile_readall(ff); - if(!ff) return 0; // we return 0, so that we will retry to open it next time - - uint32_t lines = procfile_lines(ff), l; - uint32_t words; - - int hwcorrupted = 0; - - unsigned long long MemTotal = 0, MemFree = 0, Buffers = 0, Cached = 0, SwapCached = 0, - Active = 0, Inactive = 0, ActiveAnon = 0, InactiveAnon = 0, ActiveFile = 0, InactiveFile = 0, - Unevictable = 0, Mlocked = 0, SwapTotal = 0, SwapFree = 0, Dirty = 0, Writeback = 0, AnonPages = 0, - Mapped = 0, Shmem = 0, Slab = 0, SReclaimable = 0, SUnreclaim = 0, KernelStack = 0, PageTables = 0, - NFS_Unstable = 0, Bounce = 0, WritebackTmp = 0, CommitLimit = 0, Committed_AS = 0, - VmallocTotal = 0, VmallocUsed = 0, VmallocChunk = 0, - AnonHugePages = 0, HugePages_Total = 0, HugePages_Free = 0, HugePages_Rsvd = 0, HugePages_Surp = 0, Hugepagesize = 0, - DirectMap4k = 0, DirectMap2M = 0, HardwareCorrupted = 0; - - for(l = 0; l < lines ;l++) { - words = procfile_linewords(ff, l); - if(words < 2) continue; - - char *name = procfile_lineword(ff, l, 0); - unsigned long long value = strtoull(procfile_lineword(ff, l, 1), NULL, 10); - - if(!MemTotal && strcmp(name, "MemTotal") == 0) MemTotal = value; - else if(!MemFree && strcmp(name, "MemFree") == 0) MemFree = value; - else if(!Buffers && strcmp(name, "Buffers") == 0) Buffers = value; - else if(!Cached && strcmp(name, "Cached") == 0) Cached = value; - else if(!SwapCached && strcmp(name, "SwapCached") == 0) SwapCached = value; - else if(!Active && strcmp(name, "Active") == 0) Active = value; - else if(!Inactive && strcmp(name, "Inactive") == 0) Inactive = value; - else if(!ActiveAnon && strcmp(name, "ActiveAnon") == 0) ActiveAnon = value; - else if(!InactiveAnon && strcmp(name, "InactiveAnon") == 0) InactiveAnon = value; - else if(!ActiveFile && strcmp(name, "ActiveFile") == 0) ActiveFile = value; - else if(!InactiveFile && strcmp(name, "InactiveFile") == 0) InactiveFile = value; - else if(!Unevictable && strcmp(name, "Unevictable") == 0) Unevictable = value; - else if(!Mlocked && strcmp(name, "Mlocked") == 0) Mlocked = value; - else if(!SwapTotal && strcmp(name, "SwapTotal") == 0) SwapTotal = value; - else if(!SwapFree && strcmp(name, "SwapFree") == 0) SwapFree = value; - else if(!Dirty && strcmp(name, "Dirty") == 0) Dirty = value; - else if(!Writeback && strcmp(name, "Writeback") == 0) Writeback = value; - else if(!AnonPages && strcmp(name, "AnonPages") == 0) AnonPages = value; - else if(!Mapped && strcmp(name, "Mapped") == 0) Mapped = value; - else if(!Shmem && strcmp(name, "Shmem") == 0) Shmem = value; - else if(!Slab && strcmp(name, "Slab") == 0) Slab = value; - else if(!SReclaimable && strcmp(name, "SReclaimable") == 0) SReclaimable = value; - else if(!SUnreclaim && strcmp(name, "SUnreclaim") == 0) SUnreclaim = value; - else if(!KernelStack && strcmp(name, "KernelStack") == 0) KernelStack = value; - else if(!PageTables && strcmp(name, "PageTables") == 0) PageTables = value; - else if(!NFS_Unstable && strcmp(name, "NFS_Unstable") == 0) NFS_Unstable = value; - else if(!Bounce && strcmp(name, "Bounce") == 0) Bounce = value; - else if(!WritebackTmp && strcmp(name, "WritebackTmp") == 0) WritebackTmp = value; - else if(!CommitLimit && strcmp(name, "CommitLimit") == 0) CommitLimit = value; - else if(!Committed_AS && strcmp(name, "Committed_AS") == 0) Committed_AS = value; - else if(!VmallocTotal && strcmp(name, "VmallocTotal") == 0) VmallocTotal = value; - else if(!VmallocUsed && strcmp(name, "VmallocUsed") == 0) VmallocUsed = value; - else if(!VmallocChunk && strcmp(name, "VmallocChunk") == 0) VmallocChunk = value; - else if(!HardwareCorrupted && strcmp(name, "HardwareCorrupted") == 0) { HardwareCorrupted = value; hwcorrupted = 1; } - else if(!AnonHugePages && strcmp(name, "AnonHugePages") == 0) AnonHugePages = value; - else if(!HugePages_Total && strcmp(name, "HugePages_Total") == 0) HugePages_Total = value; - else if(!HugePages_Free && strcmp(name, "HugePages_Free") == 0) HugePages_Free = value; - else if(!HugePages_Rsvd && strcmp(name, "HugePages_Rsvd") == 0) HugePages_Rsvd = value; - else if(!HugePages_Surp && strcmp(name, "HugePages_Surp") == 0) HugePages_Surp = value; - else if(!Hugepagesize && strcmp(name, "Hugepagesize") == 0) Hugepagesize = value; - else if(!DirectMap4k && strcmp(name, "DirectMap4k") == 0) DirectMap4k = value; - else if(!DirectMap2M && strcmp(name, "DirectMap2M") == 0) DirectMap2M = value; - } - - RRDSET *st; - - // -------------------------------------------------------------------- - - // http://stackoverflow.com/questions/3019748/how-to-reliably-measure-available-memory-in-linux - unsigned long long MemUsed = MemTotal - MemFree - Cached - Buffers; - - if(do_ram) { - st = rrdset_find("system.ram"); - if(!st) { - st = rrdset_create("system", "ram", NULL, "ram", NULL, "System RAM", "MB", 200, update_every, RRDSET_TYPE_STACKED); - - rrddim_add(st, "buffers", NULL, 1, 1024, RRDDIM_ABSOLUTE); - rrddim_add(st, "used", NULL, 1, 1024, RRDDIM_ABSOLUTE); - rrddim_add(st, "cached", NULL, 1, 1024, RRDDIM_ABSOLUTE); - rrddim_add(st, "free", NULL, 1, 1024, RRDDIM_ABSOLUTE); - } - else rrdset_next(st); - - rrddim_set(st, "used", MemUsed); - rrddim_set(st, "free", MemFree); - rrddim_set(st, "cached", Cached); - rrddim_set(st, "buffers", Buffers); - rrdset_done(st); - } - - // -------------------------------------------------------------------- - - unsigned long long SwapUsed = SwapTotal - SwapFree; - - if(do_swap) { - st = rrdset_find("system.swap"); - if(!st) { - st = rrdset_create("system", "swap", NULL, "swap", NULL, "System Swap", "MB", 201, update_every, RRDSET_TYPE_STACKED); - st->isdetail = 1; - - rrddim_add(st, "free", NULL, 1, 1024, RRDDIM_ABSOLUTE); - rrddim_add(st, "used", NULL, 1, 1024, RRDDIM_ABSOLUTE); - } - else rrdset_next(st); - - rrddim_set(st, "used", SwapUsed); - rrddim_set(st, "free", SwapFree); - rrdset_done(st); - } - - // -------------------------------------------------------------------- - - if(hwcorrupted && do_hwcorrupt && HardwareCorrupted > 0) { - do_hwcorrupt = CONFIG_ONDEMAND_YES; - - st = rrdset_find("mem.hwcorrupt"); - if(!st) { - st = rrdset_create("mem", "hwcorrupt", NULL, "errors", NULL, "Hardware Corrupted ECC", "MB", 9000, update_every, RRDSET_TYPE_LINE); - st->isdetail = 1; - - rrddim_add(st, "HardwareCorrupted", NULL, 1, 1024, RRDDIM_ABSOLUTE); - } - else rrdset_next(st); - - rrddim_set(st, "HardwareCorrupted", HardwareCorrupted); - rrdset_done(st); - } - - // -------------------------------------------------------------------- - - if(do_committed) { - st = rrdset_find("mem.committed"); - if(!st) { - st = rrdset_create("mem", "committed", NULL, "system", NULL, "Committed (Allocated) Memory", "MB", 5000, update_every, RRDSET_TYPE_AREA); - st->isdetail = 1; - - rrddim_add(st, "Committed_AS", NULL, 1, 1024, RRDDIM_ABSOLUTE); - } - else rrdset_next(st); - - rrddim_set(st, "Committed_AS", Committed_AS); - rrdset_done(st); - } - - // -------------------------------------------------------------------- - - if(do_writeback) { - st = rrdset_find("mem.writeback"); - if(!st) { - st = rrdset_create("mem", "writeback", NULL, "kernel", NULL, "Writeback Memory", "MB", 4000, update_every, RRDSET_TYPE_LINE); - st->isdetail = 1; - - rrddim_add(st, "Dirty", NULL, 1, 1024, RRDDIM_ABSOLUTE); - rrddim_add(st, "Writeback", NULL, 1, 1024, RRDDIM_ABSOLUTE); - rrddim_add(st, "FuseWriteback", NULL, 1, 1024, RRDDIM_ABSOLUTE); - rrddim_add(st, "NfsWriteback", NULL, 1, 1024, RRDDIM_ABSOLUTE); - rrddim_add(st, "Bounce", NULL, 1, 1024, RRDDIM_ABSOLUTE); - } - else rrdset_next(st); - - rrddim_set(st, "Dirty", Dirty); - rrddim_set(st, "Writeback", Writeback); - rrddim_set(st, "FuseWriteback", WritebackTmp); - rrddim_set(st, "NfsWriteback", NFS_Unstable); - rrddim_set(st, "Bounce", Bounce); - rrdset_done(st); - } - - // -------------------------------------------------------------------- - - if(do_kernel) { - st = rrdset_find("mem.kernel"); - if(!st) { - st = rrdset_create("mem", "kernel", NULL, "kernel", NULL, "Memory Used by Kernel", "MB", 6000, update_every, RRDSET_TYPE_STACKED); - st->isdetail = 1; - - rrddim_add(st, "Slab", NULL, 1, 1024, RRDDIM_ABSOLUTE); - rrddim_add(st, "KernelStack", NULL, 1, 1024, RRDDIM_ABSOLUTE); - rrddim_add(st, "PageTables", NULL, 1, 1024, RRDDIM_ABSOLUTE); - rrddim_add(st, "VmallocUsed", NULL, 1, 1024, RRDDIM_ABSOLUTE); - } - else rrdset_next(st); - - rrddim_set(st, "KernelStack", KernelStack); - rrddim_set(st, "Slab", Slab); - rrddim_set(st, "PageTables", PageTables); - rrddim_set(st, "VmallocUsed", VmallocUsed); - rrdset_done(st); - } - - // -------------------------------------------------------------------- - - if(do_slab) { - st = rrdset_find("mem.slab"); - if(!st) { - st = rrdset_create("mem", "slab", NULL, "slab", NULL, "Reclaimable Kernel Memory", "MB", 6500, update_every, RRDSET_TYPE_STACKED); - st->isdetail = 1; - - rrddim_add(st, "reclaimable", NULL, 1, 1024, RRDDIM_ABSOLUTE); - rrddim_add(st, "unreclaimable", NULL, 1, 1024, RRDDIM_ABSOLUTE); - } - else rrdset_next(st); - - rrddim_set(st, "reclaimable", SReclaimable); - rrddim_set(st, "unreclaimable", SUnreclaim); - rrdset_done(st); - } - - return 0; + static procfile *ff = NULL; + + static int do_ram = -1, do_swap = -1, do_hwcorrupt = -1, do_committed = -1, do_writeback = -1, do_kernel = -1, do_slab = -1; + + if(do_ram == -1) do_ram = config_get_boolean("plugin:proc:/proc/meminfo", "system ram", 1); + if(do_swap == -1) do_swap = config_get_boolean("plugin:proc:/proc/meminfo", "system swap", 1); + if(do_hwcorrupt == -1) do_hwcorrupt = config_get_boolean_ondemand("plugin:proc:/proc/meminfo", "hardware corrupted ECC", CONFIG_ONDEMAND_ONDEMAND); + if(do_committed == -1) do_committed = config_get_boolean("plugin:proc:/proc/meminfo", "committed memory", 1); + if(do_writeback == -1) do_writeback = config_get_boolean("plugin:proc:/proc/meminfo", "writeback memory", 1); + if(do_kernel == -1) do_kernel = config_get_boolean("plugin:proc:/proc/meminfo", "kernel memory", 1); + if(do_slab == -1) do_slab = config_get_boolean("plugin:proc:/proc/meminfo", "slab memory", 1); + + if(dt) {}; + + if(!ff) { + char filename[FILENAME_MAX + 1]; + snprintfz(filename, FILENAME_MAX, "%s%s", global_host_prefix, "/proc/meminfo"); + ff = procfile_open(config_get("plugin:proc:/proc/meminfo", "filename to monitor", filename), " \t:", PROCFILE_FLAG_DEFAULT); + } + if(!ff) return 1; + + ff = procfile_readall(ff); + if(!ff) return 0; // we return 0, so that we will retry to open it next time + + uint32_t lines = procfile_lines(ff), l; + uint32_t words; + + int hwcorrupted = 0; + + unsigned long long MemTotal = 0, MemFree = 0, Buffers = 0, Cached = 0, SwapCached = 0, + Active = 0, Inactive = 0, ActiveAnon = 0, InactiveAnon = 0, ActiveFile = 0, InactiveFile = 0, + Unevictable = 0, Mlocked = 0, SwapTotal = 0, SwapFree = 0, Dirty = 0, Writeback = 0, AnonPages = 0, + Mapped = 0, Shmem = 0, Slab = 0, SReclaimable = 0, SUnreclaim = 0, KernelStack = 0, PageTables = 0, + NFS_Unstable = 0, Bounce = 0, WritebackTmp = 0, CommitLimit = 0, Committed_AS = 0, + VmallocTotal = 0, VmallocUsed = 0, VmallocChunk = 0, + AnonHugePages = 0, HugePages_Total = 0, HugePages_Free = 0, HugePages_Rsvd = 0, HugePages_Surp = 0, Hugepagesize = 0, + DirectMap4k = 0, DirectMap2M = 0, HardwareCorrupted = 0; + + for(l = 0; l < lines ;l++) { + words = procfile_linewords(ff, l); + if(words < 2) continue; + + char *name = procfile_lineword(ff, l, 0); + unsigned long long value = strtoull(procfile_lineword(ff, l, 1), NULL, 10); + + if(!MemTotal && strcmp(name, "MemTotal") == 0) MemTotal = value; + else if(!MemFree && strcmp(name, "MemFree") == 0) MemFree = value; + else if(!Buffers && strcmp(name, "Buffers") == 0) Buffers = value; + else if(!Cached && strcmp(name, "Cached") == 0) Cached = value; + else if(!SwapCached && strcmp(name, "SwapCached") == 0) SwapCached = value; + else if(!Active && strcmp(name, "Active") == 0) Active = value; + else if(!Inactive && strcmp(name, "Inactive") == 0) Inactive = value; + else if(!ActiveAnon && strcmp(name, "ActiveAnon") == 0) ActiveAnon = value; + else if(!InactiveAnon && strcmp(name, "InactiveAnon") == 0) InactiveAnon = value; + else if(!ActiveFile && strcmp(name, "ActiveFile") == 0) ActiveFile = value; + else if(!InactiveFile && strcmp(name, "InactiveFile") == 0) InactiveFile = value; + else if(!Unevictable && strcmp(name, "Unevictable") == 0) Unevictable = value; + else if(!Mlocked && strcmp(name, "Mlocked") == 0) Mlocked = value; + else if(!SwapTotal && strcmp(name, "SwapTotal") == 0) SwapTotal = value; + else if(!SwapFree && strcmp(name, "SwapFree") == 0) SwapFree = value; + else if(!Dirty && strcmp(name, "Dirty") == 0) Dirty = value; + else if(!Writeback && strcmp(name, "Writeback") == 0) Writeback = value; + else if(!AnonPages && strcmp(name, "AnonPages") == 0) AnonPages = value; + else if(!Mapped && strcmp(name, "Mapped") == 0) Mapped = value; + else if(!Shmem && strcmp(name, "Shmem") == 0) Shmem = value; + else if(!Slab && strcmp(name, "Slab") == 0) Slab = value; + else if(!SReclaimable && strcmp(name, "SReclaimable") == 0) SReclaimable = value; + else if(!SUnreclaim && strcmp(name, "SUnreclaim") == 0) SUnreclaim = value; + else if(!KernelStack && strcmp(name, "KernelStack") == 0) KernelStack = value; + else if(!PageTables && strcmp(name, "PageTables") == 0) PageTables = value; + else if(!NFS_Unstable && strcmp(name, "NFS_Unstable") == 0) NFS_Unstable = value; + else if(!Bounce && strcmp(name, "Bounce") == 0) Bounce = value; + else if(!WritebackTmp && strcmp(name, "WritebackTmp") == 0) WritebackTmp = value; + else if(!CommitLimit && strcmp(name, "CommitLimit") == 0) CommitLimit = value; + else if(!Committed_AS && strcmp(name, "Committed_AS") == 0) Committed_AS = value; + else if(!VmallocTotal && strcmp(name, "VmallocTotal") == 0) VmallocTotal = value; + else if(!VmallocUsed && strcmp(name, "VmallocUsed") == 0) VmallocUsed = value; + else if(!VmallocChunk && strcmp(name, "VmallocChunk") == 0) VmallocChunk = value; + else if(!HardwareCorrupted && strcmp(name, "HardwareCorrupted") == 0) { HardwareCorrupted = value; hwcorrupted = 1; } + else if(!AnonHugePages && strcmp(name, "AnonHugePages") == 0) AnonHugePages = value; + else if(!HugePages_Total && strcmp(name, "HugePages_Total") == 0) HugePages_Total = value; + else if(!HugePages_Free && strcmp(name, "HugePages_Free") == 0) HugePages_Free = value; + else if(!HugePages_Rsvd && strcmp(name, "HugePages_Rsvd") == 0) HugePages_Rsvd = value; + else if(!HugePages_Surp && strcmp(name, "HugePages_Surp") == 0) HugePages_Surp = value; + else if(!Hugepagesize && strcmp(name, "Hugepagesize") == 0) Hugepagesize = value; + else if(!DirectMap4k && strcmp(name, "DirectMap4k") == 0) DirectMap4k = value; + else if(!DirectMap2M && strcmp(name, "DirectMap2M") == 0) DirectMap2M = value; + } + + RRDSET *st; + + // -------------------------------------------------------------------- + + // http://stackoverflow.com/questions/3019748/how-to-reliably-measure-available-memory-in-linux + unsigned long long MemUsed = MemTotal - MemFree - Cached - Buffers; + + if(do_ram) { + st = rrdset_find("system.ram"); + if(!st) { + st = rrdset_create("system", "ram", NULL, "ram", NULL, "System RAM", "MB", 200, update_every, RRDSET_TYPE_STACKED); + + rrddim_add(st, "buffers", NULL, 1, 1024, RRDDIM_ABSOLUTE); + rrddim_add(st, "used", NULL, 1, 1024, RRDDIM_ABSOLUTE); + rrddim_add(st, "cached", NULL, 1, 1024, RRDDIM_ABSOLUTE); + rrddim_add(st, "free", NULL, 1, 1024, RRDDIM_ABSOLUTE); + } + else rrdset_next(st); + + rrddim_set(st, "used", MemUsed); + rrddim_set(st, "free", MemFree); + rrddim_set(st, "cached", Cached); + rrddim_set(st, "buffers", Buffers); + rrdset_done(st); + } + + // -------------------------------------------------------------------- + + unsigned long long SwapUsed = SwapTotal - SwapFree; + + if(do_swap) { + st = rrdset_find("system.swap"); + if(!st) { + st = rrdset_create("system", "swap", NULL, "swap", NULL, "System Swap", "MB", 201, update_every, RRDSET_TYPE_STACKED); + st->isdetail = 1; + + rrddim_add(st, "free", NULL, 1, 1024, RRDDIM_ABSOLUTE); + rrddim_add(st, "used", NULL, 1, 1024, RRDDIM_ABSOLUTE); + } + else rrdset_next(st); + + rrddim_set(st, "used", SwapUsed); + rrddim_set(st, "free", SwapFree); + rrdset_done(st); + } + + // -------------------------------------------------------------------- + + if(hwcorrupted && do_hwcorrupt && HardwareCorrupted > 0) { + do_hwcorrupt = CONFIG_ONDEMAND_YES; + + st = rrdset_find("mem.hwcorrupt"); + if(!st) { + st = rrdset_create("mem", "hwcorrupt", NULL, "errors", NULL, "Hardware Corrupted ECC", "MB", 9000, update_every, RRDSET_TYPE_LINE); + st->isdetail = 1; + + rrddim_add(st, "HardwareCorrupted", NULL, 1, 1024, RRDDIM_ABSOLUTE); + } + else rrdset_next(st); + + rrddim_set(st, "HardwareCorrupted", HardwareCorrupted); + rrdset_done(st); + } + + // -------------------------------------------------------------------- + + if(do_committed) { + st = rrdset_find("mem.committed"); + if(!st) { + st = rrdset_create("mem", "committed", NULL, "system", NULL, "Committed (Allocated) Memory", "MB", 5000, update_every, RRDSET_TYPE_AREA); + st->isdetail = 1; + + rrddim_add(st, "Committed_AS", NULL, 1, 1024, RRDDIM_ABSOLUTE); + } + else rrdset_next(st); + + rrddim_set(st, "Committed_AS", Committed_AS); + rrdset_done(st); + } + + // -------------------------------------------------------------------- + + if(do_writeback) { + st = rrdset_find("mem.writeback"); + if(!st) { + st = rrdset_create("mem", "writeback", NULL, "kernel", NULL, "Writeback Memory", "MB", 4000, update_every, RRDSET_TYPE_LINE); + st->isdetail = 1; + + rrddim_add(st, "Dirty", NULL, 1, 1024, RRDDIM_ABSOLUTE); + rrddim_add(st, "Writeback", NULL, 1, 1024, RRDDIM_ABSOLUTE); + rrddim_add(st, "FuseWriteback", NULL, 1, 1024, RRDDIM_ABSOLUTE); + rrddim_add(st, "NfsWriteback", NULL, 1, 1024, RRDDIM_ABSOLUTE); + rrddim_add(st, "Bounce", NULL, 1, 1024, RRDDIM_ABSOLUTE); + } + else rrdset_next(st); + + rrddim_set(st, "Dirty", Dirty); + rrddim_set(st, "Writeback", Writeback); + rrddim_set(st, "FuseWriteback", WritebackTmp); + rrddim_set(st, "NfsWriteback", NFS_Unstable); + rrddim_set(st, "Bounce", Bounce); + rrdset_done(st); + } + + // -------------------------------------------------------------------- + + if(do_kernel) { + st = rrdset_find("mem.kernel"); + if(!st) { + st = rrdset_create("mem", "kernel", NULL, "kernel", NULL, "Memory Used by Kernel", "MB", 6000, update_every, RRDSET_TYPE_STACKED); + st->isdetail = 1; + + rrddim_add(st, "Slab", NULL, 1, 1024, RRDDIM_ABSOLUTE); + rrddim_add(st, "KernelStack", NULL, 1, 1024, RRDDIM_ABSOLUTE); + rrddim_add(st, "PageTables", NULL, 1, 1024, RRDDIM_ABSOLUTE); + rrddim_add(st, "VmallocUsed", NULL, 1, 1024, RRDDIM_ABSOLUTE); + } + else rrdset_next(st); + + rrddim_set(st, "KernelStack", KernelStack); + rrddim_set(st, "Slab", Slab); + rrddim_set(st, "PageTables", PageTables); + rrddim_set(st, "VmallocUsed", VmallocUsed); + rrdset_done(st); + } + + // -------------------------------------------------------------------- + + if(do_slab) { + st = rrdset_find("mem.slab"); + if(!st) { + st = rrdset_create("mem", "slab", NULL, "slab", NULL, "Reclaimable Kernel Memory", "MB", 6500, update_every, RRDSET_TYPE_STACKED); + st->isdetail = 1; + + rrddim_add(st, "reclaimable", NULL, 1, 1024, RRDDIM_ABSOLUTE); + rrddim_add(st, "unreclaimable", NULL, 1, 1024, RRDDIM_ABSOLUTE); + } + else rrdset_next(st); + + rrddim_set(st, "reclaimable", SReclaimable); + rrddim_set(st, "unreclaimable", SUnreclaim); + rrdset_done(st); + } + + return 0; } diff --git a/src/proc_net_dev.c b/src/proc_net_dev.c index 12d8078c7..53981182a 100644 --- a/src/proc_net_dev.c +++ b/src/proc_net_dev.c @@ -1,247 +1,236 @@ -#ifdef HAVE_CONFIG_H -#include <config.h> -#endif -#include <stdio.h> -#include <stdlib.h> -#include <string.h> - #include "common.h" -#include "appconfig.h" -#include "procfile.h" -#include "rrd.h" -#include "plugin_proc.h" int do_proc_net_dev(int update_every, unsigned long long dt) { - static procfile *ff = NULL; - static int enable_new_interfaces = -1, enable_ifb_interfaces = -1; - static int do_bandwidth = -1, do_packets = -1, do_errors = -1, do_drops = -1, do_fifo = -1, do_compressed = -1, do_events = -1; - - if(dt) {}; - - if(!ff) { - char filename[FILENAME_MAX + 1]; - snprintfz(filename, FILENAME_MAX, "%s%s", global_host_prefix, "/proc/net/dev"); - ff = procfile_open(config_get("plugin:proc:/proc/net/dev", "filename to monitor", filename), " \t,:|", PROCFILE_FLAG_DEFAULT); - } - if(!ff) return 1; - - ff = procfile_readall(ff); - if(!ff) return 0; // we return 0, so that we will retry to open it next time - - if(enable_new_interfaces == -1) enable_new_interfaces = config_get_boolean_ondemand("plugin:proc:/proc/net/dev", "enable new interfaces detected at runtime", CONFIG_ONDEMAND_ONDEMAND); - if(enable_ifb_interfaces == -1) enable_ifb_interfaces = config_get_boolean_ondemand("plugin:proc:/proc/net/dev", "enable ifb interfaces", CONFIG_ONDEMAND_NO); - - if(do_bandwidth == -1) do_bandwidth = config_get_boolean_ondemand("plugin:proc:/proc/net/dev", "bandwidth for all interfaces", CONFIG_ONDEMAND_ONDEMAND); - if(do_packets == -1) do_packets = config_get_boolean_ondemand("plugin:proc:/proc/net/dev", "packets for all interfaces", CONFIG_ONDEMAND_ONDEMAND); - if(do_errors == -1) do_errors = config_get_boolean_ondemand("plugin:proc:/proc/net/dev", "errors for all interfaces", CONFIG_ONDEMAND_ONDEMAND); - if(do_drops == -1) do_drops = config_get_boolean_ondemand("plugin:proc:/proc/net/dev", "drops for all interfaces", CONFIG_ONDEMAND_ONDEMAND); - if(do_fifo == -1) do_fifo = config_get_boolean_ondemand("plugin:proc:/proc/net/dev", "fifo for all interfaces", CONFIG_ONDEMAND_ONDEMAND); - if(do_compressed == -1) do_compressed = config_get_boolean_ondemand("plugin:proc:/proc/net/dev", "compressed packets for all interfaces", CONFIG_ONDEMAND_ONDEMAND); - if(do_events == -1) do_events = config_get_boolean_ondemand("plugin:proc:/proc/net/dev", "frames, collisions, carrier counters for all interfaces", CONFIG_ONDEMAND_ONDEMAND); - - uint32_t lines = procfile_lines(ff), l; - uint32_t words; - - char *iface; - unsigned long long rbytes, rpackets, rerrors, rdrops, rfifo, rframe, rcompressed, rmulticast; - unsigned long long tbytes, tpackets, terrors, tdrops, tfifo, tcollisions, tcarrier, tcompressed; - - for(l = 2; l < lines ;l++) { - words = procfile_linewords(ff, l); - if(words < 17) continue; - - iface = procfile_lineword(ff, l, 0); - - rbytes = strtoull(procfile_lineword(ff, l, 1), NULL, 10); - rpackets = strtoull(procfile_lineword(ff, l, 2), NULL, 10); - rerrors = strtoull(procfile_lineword(ff, l, 3), NULL, 10); - rdrops = strtoull(procfile_lineword(ff, l, 4), NULL, 10); - rfifo = strtoull(procfile_lineword(ff, l, 5), NULL, 10); - rframe = strtoull(procfile_lineword(ff, l, 6), NULL, 10); - rcompressed = strtoull(procfile_lineword(ff, l, 7), NULL, 10); - rmulticast = strtoull(procfile_lineword(ff, l, 8), NULL, 10); - - tbytes = strtoull(procfile_lineword(ff, l, 9), NULL, 10); - tpackets = strtoull(procfile_lineword(ff, l, 10), NULL, 10); - terrors = strtoull(procfile_lineword(ff, l, 11), NULL, 10); - tdrops = strtoull(procfile_lineword(ff, l, 12), NULL, 10); - tfifo = strtoull(procfile_lineword(ff, l, 13), NULL, 10); - tcollisions = strtoull(procfile_lineword(ff, l, 14), NULL, 10); - tcarrier = strtoull(procfile_lineword(ff, l, 15), NULL, 10); - tcompressed = strtoull(procfile_lineword(ff, l, 16), NULL, 10); - - int ddo_bandwidth = do_bandwidth, ddo_packets = do_packets, ddo_errors = do_errors, ddo_drops = do_drops, ddo_fifo = do_fifo, ddo_compressed = do_compressed, ddo_events = do_events; - - int default_enable = enable_new_interfaces; - - // prevent unused interfaces from creating charts - if(strcmp(iface, "lo") == 0) - default_enable = 0; - else { - int len = strlen(iface); - if(len >= 4 && strcmp(&iface[len-4], "-ifb") == 0) - default_enable = enable_ifb_interfaces; - } - - // check if the user wants it - { - char var_name[512 + 1]; - snprintfz(var_name, 512, "plugin:proc:/proc/net/dev:%s", iface); - default_enable = config_get_boolean_ondemand(var_name, "enabled", default_enable); - if(default_enable == CONFIG_ONDEMAND_NO) continue; - if(default_enable == CONFIG_ONDEMAND_ONDEMAND && !rbytes && !tbytes) continue; - - ddo_bandwidth = config_get_boolean_ondemand(var_name, "bandwidth", ddo_bandwidth); - ddo_packets = config_get_boolean_ondemand(var_name, "packets", ddo_packets); - ddo_errors = config_get_boolean_ondemand(var_name, "errors", ddo_errors); - ddo_drops = config_get_boolean_ondemand(var_name, "drops", ddo_drops); - ddo_fifo = config_get_boolean_ondemand(var_name, "fifo", ddo_fifo); - ddo_compressed = config_get_boolean_ondemand(var_name, "compressed", ddo_compressed); - ddo_events = config_get_boolean_ondemand(var_name, "events", ddo_events); - - if(ddo_bandwidth == CONFIG_ONDEMAND_ONDEMAND && rbytes == 0 && tbytes == 0) ddo_bandwidth = 0; - if(ddo_errors == CONFIG_ONDEMAND_ONDEMAND && rerrors == 0 && terrors == 0) ddo_errors = 0; - if(ddo_drops == CONFIG_ONDEMAND_ONDEMAND && rdrops == 0 && tdrops == 0) ddo_drops = 0; - if(ddo_fifo == CONFIG_ONDEMAND_ONDEMAND && rfifo == 0 && tfifo == 0) ddo_fifo = 0; - if(ddo_compressed == CONFIG_ONDEMAND_ONDEMAND && rcompressed == 0 && tcompressed == 0) ddo_compressed = 0; - if(ddo_events == CONFIG_ONDEMAND_ONDEMAND && rframe == 0 && tcollisions == 0 && tcarrier == 0) ddo_events = 0; - - // for absolute values, we need to switch the setting to 'yes' - // to allow it refresh from now on - // if(ddo_fifo == CONFIG_ONDEMAND_ONDEMAND) config_set(var_name, "fifo", "yes"); - } - - RRDSET *st; - - // -------------------------------------------------------------------- - - if(ddo_bandwidth) { - st = rrdset_find_bytype("net", iface); - if(!st) { - st = rrdset_create("net", iface, NULL, iface, "net.net", "Bandwidth", "kilobits/s", 7000, update_every, RRDSET_TYPE_AREA); - - rrddim_add(st, "received", NULL, 8, 1024, RRDDIM_INCREMENTAL); - rrddim_add(st, "sent", NULL, -8, 1024, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); - - rrddim_set(st, "received", rbytes); - rrddim_set(st, "sent", tbytes); - rrdset_done(st); - } - - // -------------------------------------------------------------------- - - if(ddo_packets) { - st = rrdset_find_bytype("net_packets", iface); - if(!st) { - st = rrdset_create("net_packets", iface, NULL, iface, "net.packets", "Packets", "packets/s", 7001, update_every, RRDSET_TYPE_LINE); - st->isdetail = 1; - - rrddim_add(st, "received", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "sent", NULL, -1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "multicast", NULL, 1, 1, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); - - rrddim_set(st, "received", rpackets); - rrddim_set(st, "sent", tpackets); - rrddim_set(st, "multicast", rmulticast); - rrdset_done(st); - } - - // -------------------------------------------------------------------- - - if(ddo_errors) { - st = rrdset_find_bytype("net_errors", iface); - if(!st) { - st = rrdset_create("net_errors", iface, NULL, iface, "net.errors", "Interface Errors", "errors/s", 7002, update_every, RRDSET_TYPE_LINE); - st->isdetail = 1; - - rrddim_add(st, "inbound", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "outbound", NULL, -1, 1, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); - - rrddim_set(st, "inbound", rerrors); - rrddim_set(st, "outbound", terrors); - rrdset_done(st); - } - - // -------------------------------------------------------------------- - - if(ddo_drops) { - st = rrdset_find_bytype("net_drops", iface); - if(!st) { - st = rrdset_create("net_drops", iface, NULL, iface, "net.drops", "Interface Drops", "drops/s", 7003, update_every, RRDSET_TYPE_LINE); - st->isdetail = 1; - - rrddim_add(st, "inbound", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "outbound", NULL, -1, 1, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); - - rrddim_set(st, "inbound", rdrops); - rrddim_set(st, "outbound", tdrops); - rrdset_done(st); - } - - // -------------------------------------------------------------------- - - if(ddo_fifo) { - st = rrdset_find_bytype("net_fifo", iface); - if(!st) { - st = rrdset_create("net_fifo", iface, NULL, iface, "net.fifo", "Interface FIFO Buffer Errors", "errors", 7004, update_every, RRDSET_TYPE_LINE); - st->isdetail = 1; - - rrddim_add(st, "receive", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "transmit", NULL, -1, 1, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); - - rrddim_set(st, "receive", rfifo); - rrddim_set(st, "transmit", tfifo); - rrdset_done(st); - } - - // -------------------------------------------------------------------- - - if(ddo_compressed) { - st = rrdset_find_bytype("net_compressed", iface); - if(!st) { - st = rrdset_create("net_compressed", iface, NULL, iface, "net.compressed", "Compressed Packets", "packets/s", 7005, update_every, RRDSET_TYPE_LINE); - st->isdetail = 1; - - rrddim_add(st, "received", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "sent", NULL, -1, 1, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); - - rrddim_set(st, "received", rcompressed); - rrddim_set(st, "sent", tcompressed); - rrdset_done(st); - } - - // -------------------------------------------------------------------- - - if(ddo_events) { - st = rrdset_find_bytype("net_events", iface); - if(!st) { - st = rrdset_create("net_events", iface, NULL, iface, "net.events", "Network Interface Events", "events/s", 7006, update_every, RRDSET_TYPE_LINE); - st->isdetail = 1; - - rrddim_add(st, "frames", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "collisions", NULL, -1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "carrier", NULL, -1, 1, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); - - rrddim_set(st, "frames", rframe); - rrddim_set(st, "collisions", tcollisions); - rrddim_set(st, "carrier", tcarrier); - rrdset_done(st); - } - } - - return 0; + static procfile *ff = NULL; + static int enable_new_interfaces = -1, enable_ifb_interfaces = -1; + static int do_bandwidth = -1, do_packets = -1, do_errors = -1, do_drops = -1, do_fifo = -1, do_compressed = -1, do_events = -1; + + if(dt) {}; + + if(!ff) { + char filename[FILENAME_MAX + 1]; + snprintfz(filename, FILENAME_MAX, "%s%s", global_host_prefix, "/proc/net/dev"); + ff = procfile_open(config_get("plugin:proc:/proc/net/dev", "filename to monitor", filename), " \t,:|", PROCFILE_FLAG_DEFAULT); + } + if(!ff) return 1; + + ff = procfile_readall(ff); + if(!ff) return 0; // we return 0, so that we will retry to open it next time + + if(enable_new_interfaces == -1) enable_new_interfaces = config_get_boolean_ondemand("plugin:proc:/proc/net/dev", "enable new interfaces detected at runtime", CONFIG_ONDEMAND_ONDEMAND); + if(enable_ifb_interfaces == -1) enable_ifb_interfaces = config_get_boolean_ondemand("plugin:proc:/proc/net/dev", "enable ifb interfaces", CONFIG_ONDEMAND_NO); + + if(do_bandwidth == -1) do_bandwidth = config_get_boolean_ondemand("plugin:proc:/proc/net/dev", "bandwidth for all interfaces", CONFIG_ONDEMAND_ONDEMAND); + if(do_packets == -1) do_packets = config_get_boolean_ondemand("plugin:proc:/proc/net/dev", "packets for all interfaces", CONFIG_ONDEMAND_ONDEMAND); + if(do_errors == -1) do_errors = config_get_boolean_ondemand("plugin:proc:/proc/net/dev", "errors for all interfaces", CONFIG_ONDEMAND_ONDEMAND); + if(do_drops == -1) do_drops = config_get_boolean_ondemand("plugin:proc:/proc/net/dev", "drops for all interfaces", CONFIG_ONDEMAND_ONDEMAND); + if(do_fifo == -1) do_fifo = config_get_boolean_ondemand("plugin:proc:/proc/net/dev", "fifo for all interfaces", CONFIG_ONDEMAND_ONDEMAND); + if(do_compressed == -1) do_compressed = config_get_boolean_ondemand("plugin:proc:/proc/net/dev", "compressed packets for all interfaces", CONFIG_ONDEMAND_ONDEMAND); + if(do_events == -1) do_events = config_get_boolean_ondemand("plugin:proc:/proc/net/dev", "frames, collisions, carrier counters for all interfaces", CONFIG_ONDEMAND_ONDEMAND); + + uint32_t lines = procfile_lines(ff), l; + uint32_t words; + + char *iface; + unsigned long long rbytes, rpackets, rerrors, rdrops, rfifo, rframe, rcompressed, rmulticast; + unsigned long long tbytes, tpackets, terrors, tdrops, tfifo, tcollisions, tcarrier, tcompressed; + + for(l = 2; l < lines ;l++) { + words = procfile_linewords(ff, l); + if(words < 17) continue; + + iface = procfile_lineword(ff, l, 0); + + rbytes = strtoull(procfile_lineword(ff, l, 1), NULL, 10); + rpackets = strtoull(procfile_lineword(ff, l, 2), NULL, 10); + rerrors = strtoull(procfile_lineword(ff, l, 3), NULL, 10); + rdrops = strtoull(procfile_lineword(ff, l, 4), NULL, 10); + rfifo = strtoull(procfile_lineword(ff, l, 5), NULL, 10); + rframe = strtoull(procfile_lineword(ff, l, 6), NULL, 10); + rcompressed = strtoull(procfile_lineword(ff, l, 7), NULL, 10); + rmulticast = strtoull(procfile_lineword(ff, l, 8), NULL, 10); + + tbytes = strtoull(procfile_lineword(ff, l, 9), NULL, 10); + tpackets = strtoull(procfile_lineword(ff, l, 10), NULL, 10); + terrors = strtoull(procfile_lineword(ff, l, 11), NULL, 10); + tdrops = strtoull(procfile_lineword(ff, l, 12), NULL, 10); + tfifo = strtoull(procfile_lineword(ff, l, 13), NULL, 10); + tcollisions = strtoull(procfile_lineword(ff, l, 14), NULL, 10); + tcarrier = strtoull(procfile_lineword(ff, l, 15), NULL, 10); + tcompressed = strtoull(procfile_lineword(ff, l, 16), NULL, 10); + + int ddo_bandwidth = do_bandwidth, ddo_packets = do_packets, ddo_errors = do_errors, ddo_drops = do_drops, ddo_fifo = do_fifo, ddo_compressed = do_compressed, ddo_events = do_events; + + int default_enable = enable_new_interfaces; + + // prevent unused interfaces from creating charts + if(strcmp(iface, "lo") == 0) + default_enable = 0; + else { + int len = strlen(iface); + if(len >= 4 && strcmp(&iface[len-4], "-ifb") == 0) + default_enable = enable_ifb_interfaces; + } + + // check if the user wants it + { + char var_name[512 + 1]; + snprintfz(var_name, 512, "plugin:proc:/proc/net/dev:%s", iface); + default_enable = config_get_boolean_ondemand(var_name, "enabled", default_enable); + if(default_enable == CONFIG_ONDEMAND_NO) continue; + if(default_enable == CONFIG_ONDEMAND_ONDEMAND && !rbytes && !tbytes) continue; + + ddo_bandwidth = config_get_boolean_ondemand(var_name, "bandwidth", ddo_bandwidth); + ddo_packets = config_get_boolean_ondemand(var_name, "packets", ddo_packets); + ddo_errors = config_get_boolean_ondemand(var_name, "errors", ddo_errors); + ddo_drops = config_get_boolean_ondemand(var_name, "drops", ddo_drops); + ddo_fifo = config_get_boolean_ondemand(var_name, "fifo", ddo_fifo); + ddo_compressed = config_get_boolean_ondemand(var_name, "compressed", ddo_compressed); + ddo_events = config_get_boolean_ondemand(var_name, "events", ddo_events); + + if(ddo_bandwidth == CONFIG_ONDEMAND_ONDEMAND && rbytes == 0 && tbytes == 0) ddo_bandwidth = 0; + if(ddo_errors == CONFIG_ONDEMAND_ONDEMAND && rerrors == 0 && terrors == 0) ddo_errors = 0; + if(ddo_drops == CONFIG_ONDEMAND_ONDEMAND && rdrops == 0 && tdrops == 0) ddo_drops = 0; + if(ddo_fifo == CONFIG_ONDEMAND_ONDEMAND && rfifo == 0 && tfifo == 0) ddo_fifo = 0; + if(ddo_compressed == CONFIG_ONDEMAND_ONDEMAND && rcompressed == 0 && tcompressed == 0) ddo_compressed = 0; + if(ddo_events == CONFIG_ONDEMAND_ONDEMAND && rframe == 0 && tcollisions == 0 && tcarrier == 0) ddo_events = 0; + + // for absolute values, we need to switch the setting to 'yes' + // to allow it refresh from now on + // if(ddo_fifo == CONFIG_ONDEMAND_ONDEMAND) config_set(var_name, "fifo", "yes"); + } + + RRDSET *st; + + // -------------------------------------------------------------------- + + if(ddo_bandwidth) { + st = rrdset_find_bytype("net", iface); + if(!st) { + st = rrdset_create("net", iface, NULL, iface, "net.net", "Bandwidth", "kilobits/s", 7000, update_every, RRDSET_TYPE_AREA); + + rrddim_add(st, "received", NULL, 8, 1024, RRDDIM_INCREMENTAL); + rrddim_add(st, "sent", NULL, -8, 1024, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); + + rrddim_set(st, "received", rbytes); + rrddim_set(st, "sent", tbytes); + rrdset_done(st); + } + + // -------------------------------------------------------------------- + + if(ddo_packets) { + st = rrdset_find_bytype("net_packets", iface); + if(!st) { + st = rrdset_create("net_packets", iface, NULL, iface, "net.packets", "Packets", "packets/s", 7001, update_every, RRDSET_TYPE_LINE); + st->isdetail = 1; + + rrddim_add(st, "received", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "sent", NULL, -1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "multicast", NULL, 1, 1, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); + + rrddim_set(st, "received", rpackets); + rrddim_set(st, "sent", tpackets); + rrddim_set(st, "multicast", rmulticast); + rrdset_done(st); + } + + // -------------------------------------------------------------------- + + if(ddo_errors) { + st = rrdset_find_bytype("net_errors", iface); + if(!st) { + st = rrdset_create("net_errors", iface, NULL, iface, "net.errors", "Interface Errors", "errors/s", 7002, update_every, RRDSET_TYPE_LINE); + st->isdetail = 1; + + rrddim_add(st, "inbound", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "outbound", NULL, -1, 1, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); + + rrddim_set(st, "inbound", rerrors); + rrddim_set(st, "outbound", terrors); + rrdset_done(st); + } + + // -------------------------------------------------------------------- + + if(ddo_drops) { + st = rrdset_find_bytype("net_drops", iface); + if(!st) { + st = rrdset_create("net_drops", iface, NULL, iface, "net.drops", "Interface Drops", "drops/s", 7003, update_every, RRDSET_TYPE_LINE); + st->isdetail = 1; + + rrddim_add(st, "inbound", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "outbound", NULL, -1, 1, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); + + rrddim_set(st, "inbound", rdrops); + rrddim_set(st, "outbound", tdrops); + rrdset_done(st); + } + + // -------------------------------------------------------------------- + + if(ddo_fifo) { + st = rrdset_find_bytype("net_fifo", iface); + if(!st) { + st = rrdset_create("net_fifo", iface, NULL, iface, "net.fifo", "Interface FIFO Buffer Errors", "errors", 7004, update_every, RRDSET_TYPE_LINE); + st->isdetail = 1; + + rrddim_add(st, "receive", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "transmit", NULL, -1, 1, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); + + rrddim_set(st, "receive", rfifo); + rrddim_set(st, "transmit", tfifo); + rrdset_done(st); + } + + // -------------------------------------------------------------------- + + if(ddo_compressed) { + st = rrdset_find_bytype("net_compressed", iface); + if(!st) { + st = rrdset_create("net_compressed", iface, NULL, iface, "net.compressed", "Compressed Packets", "packets/s", 7005, update_every, RRDSET_TYPE_LINE); + st->isdetail = 1; + + rrddim_add(st, "received", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "sent", NULL, -1, 1, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); + + rrddim_set(st, "received", rcompressed); + rrddim_set(st, "sent", tcompressed); + rrdset_done(st); + } + + // -------------------------------------------------------------------- + + if(ddo_events) { + st = rrdset_find_bytype("net_events", iface); + if(!st) { + st = rrdset_create("net_events", iface, NULL, iface, "net.events", "Network Interface Events", "events/s", 7006, update_every, RRDSET_TYPE_LINE); + st->isdetail = 1; + + rrddim_add(st, "frames", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "collisions", NULL, -1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "carrier", NULL, -1, 1, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); + + rrddim_set(st, "frames", rframe); + rrddim_set(st, "collisions", tcollisions); + rrddim_set(st, "carrier", tcarrier); + rrdset_done(st); + } + } + + return 0; } diff --git a/src/proc_net_ip_vs_stats.c b/src/proc_net_ip_vs_stats.c index ffb5da7b5..96efd7925 100644 --- a/src/proc_net_ip_vs_stats.c +++ b/src/proc_net_ip_vs_stats.c @@ -1,102 +1,92 @@ -#ifdef HAVE_CONFIG_H -#include <config.h> -#endif -#include <stdio.h> -#include <stdlib.h> - #include "common.h" -#include "appconfig.h" -#include "procfile.h" -#include "rrd.h" -#include "plugin_proc.h" -#define RRD_TYPE_NET_IPVS "ipvs" -#define RRD_TYPE_NET_IPVS_LEN strlen(RRD_TYPE_NET_IPVS) +#define RRD_TYPE_NET_IPVS "ipvs" +#define RRD_TYPE_NET_IPVS_LEN strlen(RRD_TYPE_NET_IPVS) int do_proc_net_ip_vs_stats(int update_every, unsigned long long dt) { - static int do_bandwidth = -1, do_sockets = -1, do_packets = -1; - static procfile *ff = NULL; + static int do_bandwidth = -1, do_sockets = -1, do_packets = -1; + static procfile *ff = NULL; - if(do_bandwidth == -1) do_bandwidth = config_get_boolean("plugin:proc:/proc/net/ip_vs_stats", "IPVS bandwidth", 1); - if(do_sockets == -1) do_sockets = config_get_boolean("plugin:proc:/proc/net/ip_vs_stats", "IPVS connections", 1); - if(do_packets == -1) do_packets = config_get_boolean("plugin:proc:/proc/net/ip_vs_stats", "IPVS packets", 1); + if(do_bandwidth == -1) do_bandwidth = config_get_boolean("plugin:proc:/proc/net/ip_vs_stats", "IPVS bandwidth", 1); + if(do_sockets == -1) do_sockets = config_get_boolean("plugin:proc:/proc/net/ip_vs_stats", "IPVS connections", 1); + if(do_packets == -1) do_packets = config_get_boolean("plugin:proc:/proc/net/ip_vs_stats", "IPVS packets", 1); - if(dt) {}; + if(dt) {}; - if(!ff) { - char filename[FILENAME_MAX + 1]; - snprintfz(filename, FILENAME_MAX, "%s%s", global_host_prefix, "/proc/net/ip_vs_stats"); - ff = procfile_open(config_get("plugin:proc:/proc/net/ip_vs_stats", "filename to monitor", filename), " \t,:|", PROCFILE_FLAG_DEFAULT); - } - if(!ff) return 1; + if(!ff) { + char filename[FILENAME_MAX + 1]; + snprintfz(filename, FILENAME_MAX, "%s%s", global_host_prefix, "/proc/net/ip_vs_stats"); + ff = procfile_open(config_get("plugin:proc:/proc/net/ip_vs_stats", "filename to monitor", filename), " \t,:|", PROCFILE_FLAG_DEFAULT); + } + if(!ff) return 1; - ff = procfile_readall(ff); - if(!ff) return 0; // we return 0, so that we will retry to open it next time + ff = procfile_readall(ff); + if(!ff) return 0; // we return 0, so that we will retry to open it next time - // make sure we have 3 lines - if(procfile_lines(ff) < 3) return 1; + // make sure we have 3 lines + if(procfile_lines(ff) < 3) return 1; - // make sure we have 5 words on the 3rd line - if(procfile_linewords(ff, 2) < 5) return 1; + // make sure we have 5 words on the 3rd line + if(procfile_linewords(ff, 2) < 5) return 1; - unsigned long long entries, InPackets, OutPackets, InBytes, OutBytes; + unsigned long long entries, InPackets, OutPackets, InBytes, OutBytes; - entries = strtoull(procfile_lineword(ff, 2, 0), NULL, 16); - InPackets = strtoull(procfile_lineword(ff, 2, 1), NULL, 16); - OutPackets = strtoull(procfile_lineword(ff, 2, 2), NULL, 16); - InBytes = strtoull(procfile_lineword(ff, 2, 3), NULL, 16); - OutBytes = strtoull(procfile_lineword(ff, 2, 4), NULL, 16); + entries = strtoull(procfile_lineword(ff, 2, 0), NULL, 16); + InPackets = strtoull(procfile_lineword(ff, 2, 1), NULL, 16); + OutPackets = strtoull(procfile_lineword(ff, 2, 2), NULL, 16); + InBytes = strtoull(procfile_lineword(ff, 2, 3), NULL, 16); + OutBytes = strtoull(procfile_lineword(ff, 2, 4), NULL, 16); - RRDSET *st; + RRDSET *st; - // -------------------------------------------------------------------- + // -------------------------------------------------------------------- - if(do_sockets) { - st = rrdset_find(RRD_TYPE_NET_IPVS ".sockets"); - if(!st) { - st = rrdset_create(RRD_TYPE_NET_IPVS, "sockets", NULL, RRD_TYPE_NET_IPVS, NULL, "IPVS New Connections", "connections/s", 1001, update_every, RRDSET_TYPE_LINE); + if(do_sockets) { + st = rrdset_find(RRD_TYPE_NET_IPVS ".sockets"); + if(!st) { + st = rrdset_create(RRD_TYPE_NET_IPVS, "sockets", NULL, RRD_TYPE_NET_IPVS, NULL, "IPVS New Connections", "connections/s", 1001, update_every, RRDSET_TYPE_LINE); - rrddim_add(st, "connections", NULL, 1, 1, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); + rrddim_add(st, "connections", NULL, 1, 1, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); - rrddim_set(st, "connections", entries); - rrdset_done(st); - } + rrddim_set(st, "connections", entries); + rrdset_done(st); + } - // -------------------------------------------------------------------- + // -------------------------------------------------------------------- - if(do_packets) { - st = rrdset_find(RRD_TYPE_NET_IPVS ".packets"); - if(!st) { - st = rrdset_create(RRD_TYPE_NET_IPVS, "packets", NULL, RRD_TYPE_NET_IPVS, NULL, "IPVS Packets", "packets/s", 1002, update_every, RRDSET_TYPE_LINE); + if(do_packets) { + st = rrdset_find(RRD_TYPE_NET_IPVS ".packets"); + if(!st) { + st = rrdset_create(RRD_TYPE_NET_IPVS, "packets", NULL, RRD_TYPE_NET_IPVS, NULL, "IPVS Packets", "packets/s", 1002, update_every, RRDSET_TYPE_LINE); - rrddim_add(st, "received", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "sent", NULL, -1, 1, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); + rrddim_add(st, "received", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "sent", NULL, -1, 1, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); - rrddim_set(st, "received", InPackets); - rrddim_set(st, "sent", OutPackets); - rrdset_done(st); - } + rrddim_set(st, "received", InPackets); + rrddim_set(st, "sent", OutPackets); + rrdset_done(st); + } - // -------------------------------------------------------------------- + // -------------------------------------------------------------------- - if(do_bandwidth) { - st = rrdset_find(RRD_TYPE_NET_IPVS ".net"); - if(!st) { - st = rrdset_create(RRD_TYPE_NET_IPVS, "net", NULL, RRD_TYPE_NET_IPVS, NULL, "IPVS Bandwidth", "kilobits/s", 1000, update_every, RRDSET_TYPE_AREA); + if(do_bandwidth) { + st = rrdset_find(RRD_TYPE_NET_IPVS ".net"); + if(!st) { + st = rrdset_create(RRD_TYPE_NET_IPVS, "net", NULL, RRD_TYPE_NET_IPVS, NULL, "IPVS Bandwidth", "kilobits/s", 1000, update_every, RRDSET_TYPE_AREA); - rrddim_add(st, "received", NULL, 8, 1024, RRDDIM_INCREMENTAL); - rrddim_add(st, "sent", NULL, -8, 1024, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); + rrddim_add(st, "received", NULL, 8, 1024, RRDDIM_INCREMENTAL); + rrddim_add(st, "sent", NULL, -8, 1024, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); - rrddim_set(st, "received", InBytes); - rrddim_set(st, "sent", OutBytes); - rrdset_done(st); - } + rrddim_set(st, "received", InBytes); + rrddim_set(st, "sent", OutBytes); + rrdset_done(st); + } - return 0; + return 0; } diff --git a/src/proc_net_netstat.c b/src/proc_net_netstat.c index c8c12c1db..fe1c4d979 100644 --- a/src/proc_net_netstat.c +++ b/src/proc_net_netstat.c @@ -1,191 +1,179 @@ -#ifdef HAVE_CONFIG_H -#include <config.h> -#endif -#include <stdio.h> -#include <stdlib.h> -#include <string.h> - #include "common.h" -#include "log.h" -#include "appconfig.h" -#include "procfile.h" -#include "rrd.h" -#include "plugin_proc.h" int do_proc_net_netstat(int update_every, unsigned long long dt) { - static int do_bandwidth = -1, do_inerrors = -1, do_mcast = -1, do_bcast = -1, do_mcast_p = -1, do_bcast_p = -1; - static procfile *ff = NULL; - - if(do_bandwidth == -1) do_bandwidth = config_get_boolean_ondemand("plugin:proc:/proc/net/netstat", "bandwidth", CONFIG_ONDEMAND_ONDEMAND); - if(do_inerrors == -1) do_inerrors = config_get_boolean_ondemand("plugin:proc:/proc/net/netstat", "input errors", CONFIG_ONDEMAND_ONDEMAND); - if(do_mcast == -1) do_mcast = config_get_boolean_ondemand("plugin:proc:/proc/net/netstat", "multicast bandwidth", CONFIG_ONDEMAND_ONDEMAND); - if(do_bcast == -1) do_bcast = config_get_boolean_ondemand("plugin:proc:/proc/net/netstat", "broadcast bandwidth", CONFIG_ONDEMAND_ONDEMAND); - if(do_mcast_p == -1) do_mcast_p = config_get_boolean_ondemand("plugin:proc:/proc/net/netstat", "multicast packets", CONFIG_ONDEMAND_ONDEMAND); - if(do_bcast_p == -1) do_bcast_p = config_get_boolean_ondemand("plugin:proc:/proc/net/netstat", "broadcast packets", CONFIG_ONDEMAND_ONDEMAND); - - if(dt) {}; - - if(!ff) { - char filename[FILENAME_MAX + 1]; - snprintfz(filename, FILENAME_MAX, "%s%s", global_host_prefix, "/proc/net/netstat"); - ff = procfile_open(config_get("plugin:proc:/proc/net/netstat", "filename to monitor", filename), " \t:", PROCFILE_FLAG_DEFAULT); - } - if(!ff) return 1; - - ff = procfile_readall(ff); - if(!ff) return 0; // we return 0, so that we will retry to open it next time - - uint32_t lines = procfile_lines(ff), l; - uint32_t words; - - for(l = 0; l < lines ;l++) { - if(strcmp(procfile_lineword(ff, l, 0), "IpExt") == 0) { - l++; // we need the next line - - if(strcmp(procfile_lineword(ff, l, 0), "IpExt") != 0) { - error("Cannot read IpExt line from /proc/net/netstat."); - break; - } - words = procfile_linewords(ff, l); - if(words < 12) { - error("Cannot read /proc/net/netstat IpExt line. Expected 12 params, read %d.", words); - continue; - } - - unsigned long long - InNoRoutes = 0, InTruncatedPkts = 0, - InOctets = 0, InMcastPkts = 0, InBcastPkts = 0, InMcastOctets = 0, InBcastOctets = 0, - OutOctets = 0, OutMcastPkts = 0, OutBcastPkts = 0, OutMcastOctets = 0, OutBcastOctets = 0; - - InNoRoutes = strtoull(procfile_lineword(ff, l, 1), NULL, 10); - InTruncatedPkts = strtoull(procfile_lineword(ff, l, 2), NULL, 10); - InMcastPkts = strtoull(procfile_lineword(ff, l, 3), NULL, 10); - OutMcastPkts = strtoull(procfile_lineword(ff, l, 4), NULL, 10); - InBcastPkts = strtoull(procfile_lineword(ff, l, 5), NULL, 10); - OutBcastPkts = strtoull(procfile_lineword(ff, l, 6), NULL, 10); - InOctets = strtoull(procfile_lineword(ff, l, 7), NULL, 10); - OutOctets = strtoull(procfile_lineword(ff, l, 8), NULL, 10); - InMcastOctets = strtoull(procfile_lineword(ff, l, 9), NULL, 10); - OutMcastOctets = strtoull(procfile_lineword(ff, l, 10), NULL, 10); - InBcastOctets = strtoull(procfile_lineword(ff, l, 11), NULL, 10); - OutBcastOctets = strtoull(procfile_lineword(ff, l, 12), NULL, 10); - - RRDSET *st; - - // -------------------------------------------------------------------- - - if(do_bandwidth == CONFIG_ONDEMAND_YES || (do_bandwidth == CONFIG_ONDEMAND_ONDEMAND && (InOctets || OutOctets))) { - do_bandwidth = CONFIG_ONDEMAND_YES; - st = rrdset_find("system.ipv4"); - if(!st) { - st = rrdset_create("system", "ipv4", NULL, "network", NULL, "IPv4 Bandwidth", "kilobits/s", 500, update_every, RRDSET_TYPE_AREA); - - rrddim_add(st, "received", NULL, 8, 1024, RRDDIM_INCREMENTAL); - rrddim_add(st, "sent", NULL, -8, 1024, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); - - rrddim_set(st, "sent", OutOctets); - rrddim_set(st, "received", InOctets); - rrdset_done(st); - } - - // -------------------------------------------------------------------- - - if(do_inerrors == CONFIG_ONDEMAND_YES || (do_inerrors == CONFIG_ONDEMAND_ONDEMAND && (InNoRoutes || InTruncatedPkts))) { - do_inerrors = CONFIG_ONDEMAND_YES; - st = rrdset_find("ipv4.inerrors"); - if(!st) { - st = rrdset_create("ipv4", "inerrors", NULL, "errors", NULL, "IPv4 Input Errors", "packets/s", 4000, update_every, RRDSET_TYPE_LINE); - st->isdetail = 1; - - rrddim_add(st, "noroutes", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "truncated", NULL, 1, 1, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); - - rrddim_set(st, "noroutes", InNoRoutes); - rrddim_set(st, "truncated", InTruncatedPkts); - rrdset_done(st); - } - - // -------------------------------------------------------------------- - - if(do_mcast == CONFIG_ONDEMAND_YES || (do_mcast == CONFIG_ONDEMAND_ONDEMAND && (InMcastOctets || OutMcastOctets))) { - do_mcast = CONFIG_ONDEMAND_YES; - st = rrdset_find("ipv4.mcast"); - if(!st) { - st = rrdset_create("ipv4", "mcast", NULL, "multicast", NULL, "IPv4 Multicast Bandwidth", "kilobits/s", 9000, update_every, RRDSET_TYPE_AREA); - st->isdetail = 1; - - rrddim_add(st, "received", NULL, 8, 1024, RRDDIM_INCREMENTAL); - rrddim_add(st, "sent", NULL, -8, 1024, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); - - rrddim_set(st, "sent", OutMcastOctets); - rrddim_set(st, "received", InMcastOctets); - rrdset_done(st); - } - - // -------------------------------------------------------------------- - - if(do_bcast == CONFIG_ONDEMAND_YES || (do_bcast == CONFIG_ONDEMAND_ONDEMAND && (InBcastOctets || OutBcastOctets))) { - do_bcast = CONFIG_ONDEMAND_YES; - st = rrdset_find("ipv4.bcast"); - if(!st) { - st = rrdset_create("ipv4", "bcast", NULL, "broadcast", NULL, "IPv4 Broadcast Bandwidth", "kilobits/s", 8000, update_every, RRDSET_TYPE_AREA); - st->isdetail = 1; - - rrddim_add(st, "received", NULL, 8, 1024, RRDDIM_INCREMENTAL); - rrddim_add(st, "sent", NULL, -8, 1024, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); - - rrddim_set(st, "sent", OutBcastOctets); - rrddim_set(st, "received", InBcastOctets); - rrdset_done(st); - } - - // -------------------------------------------------------------------- - - if(do_mcast_p == CONFIG_ONDEMAND_YES || (do_mcast_p == CONFIG_ONDEMAND_ONDEMAND && (InMcastPkts || OutMcastPkts))) { - do_mcast_p = CONFIG_ONDEMAND_YES; - st = rrdset_find("ipv4.mcastpkts"); - if(!st) { - st = rrdset_create("ipv4", "mcastpkts", NULL, "multicast", NULL, "IPv4 Multicast Packets", "packets/s", 9500, update_every, RRDSET_TYPE_LINE); - st->isdetail = 1; - - rrddim_add(st, "received", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "sent", NULL, -1, 1, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); - - rrddim_set(st, "sent", OutMcastPkts); - rrddim_set(st, "received", InMcastPkts); - rrdset_done(st); - } - - // -------------------------------------------------------------------- - - if(do_bcast_p == CONFIG_ONDEMAND_YES || (do_bcast_p == CONFIG_ONDEMAND_ONDEMAND && (InBcastPkts || OutBcastPkts))) { - do_bcast_p = CONFIG_ONDEMAND_YES; - st = rrdset_find("ipv4.bcastpkts"); - if(!st) { - st = rrdset_create("ipv4", "bcastpkts", NULL, "broadcast", NULL, "IPv4 Broadcast Packets", "packets/s", 8500, update_every, RRDSET_TYPE_LINE); - st->isdetail = 1; - - rrddim_add(st, "received", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "sent", NULL, -1, 1, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); - - rrddim_set(st, "sent", OutBcastPkts); - rrddim_set(st, "received", InBcastPkts); - rrdset_done(st); - } - } - } - - return 0; + static int do_bandwidth = -1, do_inerrors = -1, do_mcast = -1, do_bcast = -1, do_mcast_p = -1, do_bcast_p = -1; + static procfile *ff = NULL; + + if(do_bandwidth == -1) do_bandwidth = config_get_boolean_ondemand("plugin:proc:/proc/net/netstat", "bandwidth", CONFIG_ONDEMAND_ONDEMAND); + if(do_inerrors == -1) do_inerrors = config_get_boolean_ondemand("plugin:proc:/proc/net/netstat", "input errors", CONFIG_ONDEMAND_ONDEMAND); + if(do_mcast == -1) do_mcast = config_get_boolean_ondemand("plugin:proc:/proc/net/netstat", "multicast bandwidth", CONFIG_ONDEMAND_ONDEMAND); + if(do_bcast == -1) do_bcast = config_get_boolean_ondemand("plugin:proc:/proc/net/netstat", "broadcast bandwidth", CONFIG_ONDEMAND_ONDEMAND); + if(do_mcast_p == -1) do_mcast_p = config_get_boolean_ondemand("plugin:proc:/proc/net/netstat", "multicast packets", CONFIG_ONDEMAND_ONDEMAND); + if(do_bcast_p == -1) do_bcast_p = config_get_boolean_ondemand("plugin:proc:/proc/net/netstat", "broadcast packets", CONFIG_ONDEMAND_ONDEMAND); + + if(dt) {}; + + if(!ff) { + char filename[FILENAME_MAX + 1]; + snprintfz(filename, FILENAME_MAX, "%s%s", global_host_prefix, "/proc/net/netstat"); + ff = procfile_open(config_get("plugin:proc:/proc/net/netstat", "filename to monitor", filename), " \t:", PROCFILE_FLAG_DEFAULT); + } + if(!ff) return 1; + + ff = procfile_readall(ff); + if(!ff) return 0; // we return 0, so that we will retry to open it next time + + uint32_t lines = procfile_lines(ff), l; + uint32_t words; + + for(l = 0; l < lines ;l++) { + if(strcmp(procfile_lineword(ff, l, 0), "IpExt") == 0) { + l++; // we need the next line + + if(strcmp(procfile_lineword(ff, l, 0), "IpExt") != 0) { + error("Cannot read IpExt line from /proc/net/netstat."); + break; + } + words = procfile_linewords(ff, l); + if(words < 12) { + error("Cannot read /proc/net/netstat IpExt line. Expected 12 params, read %u.", words); + continue; + } + + unsigned long long + InNoRoutes = 0, InTruncatedPkts = 0, + InOctets = 0, InMcastPkts = 0, InBcastPkts = 0, InMcastOctets = 0, InBcastOctets = 0, + OutOctets = 0, OutMcastPkts = 0, OutBcastPkts = 0, OutMcastOctets = 0, OutBcastOctets = 0; + + InNoRoutes = strtoull(procfile_lineword(ff, l, 1), NULL, 10); + InTruncatedPkts = strtoull(procfile_lineword(ff, l, 2), NULL, 10); + InMcastPkts = strtoull(procfile_lineword(ff, l, 3), NULL, 10); + OutMcastPkts = strtoull(procfile_lineword(ff, l, 4), NULL, 10); + InBcastPkts = strtoull(procfile_lineword(ff, l, 5), NULL, 10); + OutBcastPkts = strtoull(procfile_lineword(ff, l, 6), NULL, 10); + InOctets = strtoull(procfile_lineword(ff, l, 7), NULL, 10); + OutOctets = strtoull(procfile_lineword(ff, l, 8), NULL, 10); + InMcastOctets = strtoull(procfile_lineword(ff, l, 9), NULL, 10); + OutMcastOctets = strtoull(procfile_lineword(ff, l, 10), NULL, 10); + InBcastOctets = strtoull(procfile_lineword(ff, l, 11), NULL, 10); + OutBcastOctets = strtoull(procfile_lineword(ff, l, 12), NULL, 10); + + RRDSET *st; + + // -------------------------------------------------------------------- + + if(do_bandwidth == CONFIG_ONDEMAND_YES || (do_bandwidth == CONFIG_ONDEMAND_ONDEMAND && (InOctets || OutOctets))) { + do_bandwidth = CONFIG_ONDEMAND_YES; + st = rrdset_find("system.ipv4"); + if(!st) { + st = rrdset_create("system", "ipv4", NULL, "network", NULL, "IPv4 Bandwidth", "kilobits/s", 500, update_every, RRDSET_TYPE_AREA); + + rrddim_add(st, "received", NULL, 8, 1024, RRDDIM_INCREMENTAL); + rrddim_add(st, "sent", NULL, -8, 1024, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); + + rrddim_set(st, "sent", OutOctets); + rrddim_set(st, "received", InOctets); + rrdset_done(st); + } + + // -------------------------------------------------------------------- + + if(do_inerrors == CONFIG_ONDEMAND_YES || (do_inerrors == CONFIG_ONDEMAND_ONDEMAND && (InNoRoutes || InTruncatedPkts))) { + do_inerrors = CONFIG_ONDEMAND_YES; + st = rrdset_find("ipv4.inerrors"); + if(!st) { + st = rrdset_create("ipv4", "inerrors", NULL, "errors", NULL, "IPv4 Input Errors", "packets/s", 4000, update_every, RRDSET_TYPE_LINE); + st->isdetail = 1; + + rrddim_add(st, "noroutes", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "truncated", NULL, 1, 1, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); + + rrddim_set(st, "noroutes", InNoRoutes); + rrddim_set(st, "truncated", InTruncatedPkts); + rrdset_done(st); + } + + // -------------------------------------------------------------------- + + if(do_mcast == CONFIG_ONDEMAND_YES || (do_mcast == CONFIG_ONDEMAND_ONDEMAND && (InMcastOctets || OutMcastOctets))) { + do_mcast = CONFIG_ONDEMAND_YES; + st = rrdset_find("ipv4.mcast"); + if(!st) { + st = rrdset_create("ipv4", "mcast", NULL, "multicast", NULL, "IPv4 Multicast Bandwidth", "kilobits/s", 9000, update_every, RRDSET_TYPE_AREA); + st->isdetail = 1; + + rrddim_add(st, "received", NULL, 8, 1024, RRDDIM_INCREMENTAL); + rrddim_add(st, "sent", NULL, -8, 1024, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); + + rrddim_set(st, "sent", OutMcastOctets); + rrddim_set(st, "received", InMcastOctets); + rrdset_done(st); + } + + // -------------------------------------------------------------------- + + if(do_bcast == CONFIG_ONDEMAND_YES || (do_bcast == CONFIG_ONDEMAND_ONDEMAND && (InBcastOctets || OutBcastOctets))) { + do_bcast = CONFIG_ONDEMAND_YES; + st = rrdset_find("ipv4.bcast"); + if(!st) { + st = rrdset_create("ipv4", "bcast", NULL, "broadcast", NULL, "IPv4 Broadcast Bandwidth", "kilobits/s", 8000, update_every, RRDSET_TYPE_AREA); + st->isdetail = 1; + + rrddim_add(st, "received", NULL, 8, 1024, RRDDIM_INCREMENTAL); + rrddim_add(st, "sent", NULL, -8, 1024, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); + + rrddim_set(st, "sent", OutBcastOctets); + rrddim_set(st, "received", InBcastOctets); + rrdset_done(st); + } + + // -------------------------------------------------------------------- + + if(do_mcast_p == CONFIG_ONDEMAND_YES || (do_mcast_p == CONFIG_ONDEMAND_ONDEMAND && (InMcastPkts || OutMcastPkts))) { + do_mcast_p = CONFIG_ONDEMAND_YES; + st = rrdset_find("ipv4.mcastpkts"); + if(!st) { + st = rrdset_create("ipv4", "mcastpkts", NULL, "multicast", NULL, "IPv4 Multicast Packets", "packets/s", 9500, update_every, RRDSET_TYPE_LINE); + st->isdetail = 1; + + rrddim_add(st, "received", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "sent", NULL, -1, 1, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); + + rrddim_set(st, "sent", OutMcastPkts); + rrddim_set(st, "received", InMcastPkts); + rrdset_done(st); + } + + // -------------------------------------------------------------------- + + if(do_bcast_p == CONFIG_ONDEMAND_YES || (do_bcast_p == CONFIG_ONDEMAND_ONDEMAND && (InBcastPkts || OutBcastPkts))) { + do_bcast_p = CONFIG_ONDEMAND_YES; + st = rrdset_find("ipv4.bcastpkts"); + if(!st) { + st = rrdset_create("ipv4", "bcastpkts", NULL, "broadcast", NULL, "IPv4 Broadcast Packets", "packets/s", 8500, update_every, RRDSET_TYPE_LINE); + st->isdetail = 1; + + rrddim_add(st, "received", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "sent", NULL, -1, 1, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); + + rrddim_set(st, "sent", OutBcastPkts); + rrddim_set(st, "received", InBcastPkts); + rrdset_done(st); + } + } + } + + return 0; } diff --git a/src/proc_net_rpc_nfsd.c b/src/proc_net_rpc_nfsd.c index 6c6dd7066..0323b4dfb 100644 --- a/src/proc_net_rpc_nfsd.c +++ b/src/proc_net_rpc_nfsd.c @@ -1,693 +1,681 @@ -#ifdef HAVE_CONFIG_H -#include <config.h> -#endif -#include <stdio.h> -#include <stdlib.h> -#include <string.h> - #include "common.h" -#include "log.h" -#include "appconfig.h" -#include "procfile.h" -#include "rrd.h" -#include "plugin_proc.h" struct nfsd_procs { - char name[30]; - unsigned long long proc2; - unsigned long long proc3; - unsigned long long proc4; - int present2; - int present3; - int present4; + char name[30]; + unsigned long long proc2; + unsigned long long proc3; + unsigned long long proc4; + int present2; + int present3; + int present4; }; struct nfsd_procs nfsd_proc_values[] = { - { "null", 0ULL, 0ULL, 0ULL, 0, 0, 0 }, - { "getattr", 0ULL, 0ULL, 0ULL, 0, 0, 0 }, - { "setattr", 0ULL, 0ULL, 0ULL, 0, 0, 0 }, - { "lookup", 0ULL, 0ULL, 0ULL, 0, 0, 0 }, - { "access", 0ULL, 0ULL, 0ULL, 0, 0, 0 }, - { "readlink", 0ULL, 0ULL, 0ULL, 0, 0, 0 }, - { "read", 0ULL, 0ULL, 0ULL, 0, 0, 0 }, - { "write", 0ULL, 0ULL, 0ULL, 0, 0, 0 }, - { "create", 0ULL, 0ULL, 0ULL, 0, 0, 0 }, - { "mkdir", 0ULL, 0ULL, 0ULL, 0, 0, 0 }, - { "symlink", 0ULL, 0ULL, 0ULL, 0, 0, 0 }, - { "mknod", 0ULL, 0ULL, 0ULL, 0, 0, 0 }, - { "remove", 0ULL, 0ULL, 0ULL, 0, 0, 0 }, - { "rmdir", 0ULL, 0ULL, 0ULL, 0, 0, 0 }, - { "rename", 0ULL, 0ULL, 0ULL, 0, 0, 0 }, - { "link", 0ULL, 0ULL, 0ULL, 0, 0, 0 }, - { "readdir", 0ULL, 0ULL, 0ULL, 0, 0, 0 }, - { "readdirplus", 0ULL, 0ULL, 0ULL, 0, 0, 0 }, - { "fsstat", 0ULL, 0ULL, 0ULL, 0, 0, 0 }, - { "fsinfo", 0ULL, 0ULL, 0ULL, 0, 0, 0 }, - { "pathconf", 0ULL, 0ULL, 0ULL, 0, 0, 0 }, - { "commit", 0ULL, 0ULL, 0ULL, 0, 0, 0 }, - { "", 0ULL, 0ULL, 0ULL, 0, 0, 0 }, + { "null", 0ULL, 0ULL, 0ULL, 0, 0, 0 }, + { "getattr", 0ULL, 0ULL, 0ULL, 0, 0, 0 }, + { "setattr", 0ULL, 0ULL, 0ULL, 0, 0, 0 }, + { "lookup", 0ULL, 0ULL, 0ULL, 0, 0, 0 }, + { "access", 0ULL, 0ULL, 0ULL, 0, 0, 0 }, + { "readlink", 0ULL, 0ULL, 0ULL, 0, 0, 0 }, + { "read", 0ULL, 0ULL, 0ULL, 0, 0, 0 }, + { "write", 0ULL, 0ULL, 0ULL, 0, 0, 0 }, + { "create", 0ULL, 0ULL, 0ULL, 0, 0, 0 }, + { "mkdir", 0ULL, 0ULL, 0ULL, 0, 0, 0 }, + { "symlink", 0ULL, 0ULL, 0ULL, 0, 0, 0 }, + { "mknod", 0ULL, 0ULL, 0ULL, 0, 0, 0 }, + { "remove", 0ULL, 0ULL, 0ULL, 0, 0, 0 }, + { "rmdir", 0ULL, 0ULL, 0ULL, 0, 0, 0 }, + { "rename", 0ULL, 0ULL, 0ULL, 0, 0, 0 }, + { "link", 0ULL, 0ULL, 0ULL, 0, 0, 0 }, + { "readdir", 0ULL, 0ULL, 0ULL, 0, 0, 0 }, + { "readdirplus", 0ULL, 0ULL, 0ULL, 0, 0, 0 }, + { "fsstat", 0ULL, 0ULL, 0ULL, 0, 0, 0 }, + { "fsinfo", 0ULL, 0ULL, 0ULL, 0, 0, 0 }, + { "pathconf", 0ULL, 0ULL, 0ULL, 0, 0, 0 }, + { "commit", 0ULL, 0ULL, 0ULL, 0, 0, 0 }, + { "", 0ULL, 0ULL, 0ULL, 0, 0, 0 }, }; struct nfsd4_ops { - char name[30]; - unsigned long long value; - int present; + char name[30]; + unsigned long long value; + int present; }; struct nfsd4_ops nfsd4_ops_values[] = { - { "access", 0ULL, 0}, - { "close", 0ULL, 0}, - { "commit", 0ULL, 0}, - { "create", 0ULL, 0}, - { "delegpurge", 0ULL, 0}, - { "delegreturn", 0ULL, 0}, - { "getattr", 0ULL, 0}, - { "getfh", 0ULL, 0}, - { "link", 0ULL, 0}, - { "lock", 0ULL, 0}, - { "lockt", 0ULL, 0}, - { "locku", 0ULL, 0}, - { "lookup", 0ULL, 0}, - { "lookupp", 0ULL, 0}, - { "nverify", 0ULL, 0}, - { "open", 0ULL, 0}, - { "openattr", 0ULL, 0}, - { "open_confirm", 0ULL, 0}, - { "open_downgrade", 0ULL, 0}, - { "putfh", 0ULL, 0}, - { "putpubfh", 0ULL, 0}, - { "putrootfh", 0ULL, 0}, - { "read", 0ULL, 0}, - { "readdir", 0ULL, 0}, - { "readlink", 0ULL, 0}, - { "remove", 0ULL, 0}, - { "rename", 0ULL, 0}, - { "renew", 0ULL, 0}, - { "restorefh", 0ULL, 0}, - { "savefh", 0ULL, 0}, - { "secinfo", 0ULL, 0}, - { "setattr", 0ULL, 0}, - { "setclientid", 0ULL, 0}, - { "setclientid_confirm", 0ULL, 0}, - { "verify", 0ULL, 0}, - { "write", 0ULL, 0}, - { "release_lockowner", 0ULL, 0}, - - /* nfs41 */ - { "backchannel_ctl", 0ULL, 0}, - { "bind_conn_to_session", 0ULL, 0}, - { "exchange_id", 0ULL, 0}, - { "create_session", 0ULL, 0}, - { "destroy_session", 0ULL, 0}, - { "free_stateid", 0ULL, 0}, - { "get_dir_delegation", 0ULL, 0}, - { "getdeviceinfo", 0ULL, 0}, - { "getdevicelist", 0ULL, 0}, - { "layoutcommit", 0ULL, 0}, - { "layoutget", 0ULL, 0}, - { "layoutreturn", 0ULL, 0}, - { "secinfo_no_name", 0ULL, 0}, - { "sequence", 0ULL, 0}, - { "set_ssv", 0ULL, 0}, - { "test_stateid", 0ULL, 0}, - { "want_delegation", 0ULL, 0}, - { "destroy_clientid", 0ULL, 0}, - { "reclaim_complete", 0ULL, 0}, - - /* nfs42 */ - { "allocate", 0ULL, 0}, - { "copy", 0ULL, 0}, - { "copy_notify", 0ULL, 0}, - { "deallocate", 0ULL, 0}, - { "io_advise", 0ULL, 0}, - { "layouterror", 0ULL, 0}, - { "layoutstats", 0ULL, 0}, - { "offload_cancel", 0ULL, 0}, - { "offload_status", 0ULL, 0}, - { "read_plus", 0ULL, 0}, - { "seek", 0ULL, 0}, - { "write_same", 0ULL, 0}, - - /* termination */ - { "", 0ULL, 0 } + { "access", 0ULL, 0}, + { "close", 0ULL, 0}, + { "commit", 0ULL, 0}, + { "create", 0ULL, 0}, + { "delegpurge", 0ULL, 0}, + { "delegreturn", 0ULL, 0}, + { "getattr", 0ULL, 0}, + { "getfh", 0ULL, 0}, + { "link", 0ULL, 0}, + { "lock", 0ULL, 0}, + { "lockt", 0ULL, 0}, + { "locku", 0ULL, 0}, + { "lookup", 0ULL, 0}, + { "lookupp", 0ULL, 0}, + { "nverify", 0ULL, 0}, + { "open", 0ULL, 0}, + { "openattr", 0ULL, 0}, + { "open_confirm", 0ULL, 0}, + { "open_downgrade", 0ULL, 0}, + { "putfh", 0ULL, 0}, + { "putpubfh", 0ULL, 0}, + { "putrootfh", 0ULL, 0}, + { "read", 0ULL, 0}, + { "readdir", 0ULL, 0}, + { "readlink", 0ULL, 0}, + { "remove", 0ULL, 0}, + { "rename", 0ULL, 0}, + { "renew", 0ULL, 0}, + { "restorefh", 0ULL, 0}, + { "savefh", 0ULL, 0}, + { "secinfo", 0ULL, 0}, + { "setattr", 0ULL, 0}, + { "setclientid", 0ULL, 0}, + { "setclientid_confirm", 0ULL, 0}, + { "verify", 0ULL, 0}, + { "write", 0ULL, 0}, + { "release_lockowner", 0ULL, 0}, + + /* nfs41 */ + { "backchannel_ctl", 0ULL, 0}, + { "bind_conn_to_session", 0ULL, 0}, + { "exchange_id", 0ULL, 0}, + { "create_session", 0ULL, 0}, + { "destroy_session", 0ULL, 0}, + { "free_stateid", 0ULL, 0}, + { "get_dir_delegation", 0ULL, 0}, + { "getdeviceinfo", 0ULL, 0}, + { "getdevicelist", 0ULL, 0}, + { "layoutcommit", 0ULL, 0}, + { "layoutget", 0ULL, 0}, + { "layoutreturn", 0ULL, 0}, + { "secinfo_no_name", 0ULL, 0}, + { "sequence", 0ULL, 0}, + { "set_ssv", 0ULL, 0}, + { "test_stateid", 0ULL, 0}, + { "want_delegation", 0ULL, 0}, + { "destroy_clientid", 0ULL, 0}, + { "reclaim_complete", 0ULL, 0}, + + /* nfs42 */ + { "allocate", 0ULL, 0}, + { "copy", 0ULL, 0}, + { "copy_notify", 0ULL, 0}, + { "deallocate", 0ULL, 0}, + { "io_advise", 0ULL, 0}, + { "layouterror", 0ULL, 0}, + { "layoutstats", 0ULL, 0}, + { "offload_cancel", 0ULL, 0}, + { "offload_status", 0ULL, 0}, + { "read_plus", 0ULL, 0}, + { "seek", 0ULL, 0}, + { "write_same", 0ULL, 0}, + + /* termination */ + { "", 0ULL, 0 } }; int do_proc_net_rpc_nfsd(int update_every, unsigned long long dt) { - static procfile *ff = NULL; - static int do_rc = -1, do_fh = -1, do_io = -1, do_th = -1, do_ra = -1, do_net = -1, do_rpc = -1, do_proc2 = -1, do_proc3 = -1, do_proc4 = -1, do_proc4ops = -1; - static int ra_warning = 0, th_warning = 0, proc2_warning = 0, proc3_warning = 0, proc4_warning = 0, proc4ops_warning = 0; - - if(dt) {}; - - if(!ff) { - char filename[FILENAME_MAX + 1]; - snprintfz(filename, FILENAME_MAX, "%s%s", global_host_prefix, "/proc/net/rpc/nfsd"); - ff = procfile_open(config_get("plugin:proc:/proc/net/rpc/nfsd", "filename to monitor", filename), " \t", PROCFILE_FLAG_DEFAULT); - } - if(!ff) return 1; - - ff = procfile_readall(ff); - if(!ff) return 0; // we return 0, so that we will retry to open it next time - - if(do_rc == -1) do_rc = config_get_boolean("plugin:proc:/proc/net/rpc/nfsd", "read cache", 1); - if(do_fh == -1) do_fh = config_get_boolean("plugin:proc:/proc/net/rpc/nfsd", "file handles", 1); - if(do_io == -1) do_io = config_get_boolean("plugin:proc:/proc/net/rpc/nfsd", "I/O", 1); - if(do_th == -1) do_th = config_get_boolean("plugin:proc:/proc/net/rpc/nfsd", "threads", 1); - if(do_ra == -1) do_ra = config_get_boolean("plugin:proc:/proc/net/rpc/nfsd", "read ahead", 1); - if(do_net == -1) do_net = config_get_boolean("plugin:proc:/proc/net/rpc/nfsd", "network", 1); - if(do_rpc == -1) do_rpc = config_get_boolean("plugin:proc:/proc/net/rpc/nfsd", "rpc", 1); - if(do_proc2 == -1) do_proc2 = config_get_boolean("plugin:proc:/proc/net/rpc/nfsd", "NFS v2 procedures", 1); - if(do_proc3 == -1) do_proc3 = config_get_boolean("plugin:proc:/proc/net/rpc/nfsd", "NFS v3 procedures", 1); - if(do_proc4 == -1) do_proc4 = config_get_boolean("plugin:proc:/proc/net/rpc/nfsd", "NFS v4 procedures", 1); - if(do_proc4ops == -1) do_proc4ops = config_get_boolean("plugin:proc:/proc/net/rpc/nfsd", "NFS v4 operations", 1); - - // if they are enabled, reset them to 1 - // later we do them =2 to avoid doing strcmp for all lines - if(do_rc) do_rc = 1; - if(do_fh) do_fh = 1; - if(do_io) do_io = 1; - if(do_th) do_th = 1; - if(do_ra) do_ra = 1; - if(do_net) do_net = 1; - if(do_rpc) do_rpc = 1; - if(do_proc2) do_proc2 = 1; - if(do_proc3) do_proc3 = 1; - if(do_proc4) do_proc4 = 1; - if(do_proc4ops) do_proc4ops = 1; - - uint32_t lines = procfile_lines(ff), l; - uint32_t words; - - char *type; - unsigned long long rc_hits = 0, rc_misses = 0, rc_nocache = 0; - unsigned long long fh_stale = 0, fh_total_lookups = 0, fh_anonymous_lookups = 0, fh_dir_not_in_dcache = 0, fh_non_dir_not_in_dcache = 0; - unsigned long long io_read = 0, io_write = 0; - unsigned long long th_threads = 0, th_fullcnt = 0, th_hist10 = 0, th_hist20 = 0, th_hist30 = 0, th_hist40 = 0, th_hist50 = 0, th_hist60 = 0, th_hist70 = 0, th_hist80 = 0, th_hist90 = 0, th_hist100 = 0; - unsigned long long ra_size = 0, ra_hist10 = 0, ra_hist20 = 0, ra_hist30 = 0, ra_hist40 = 0, ra_hist50 = 0, ra_hist60 = 0, ra_hist70 = 0, ra_hist80 = 0, ra_hist90 = 0, ra_hist100 = 0, ra_none = 0; - unsigned long long net_count = 0, net_udp_count = 0, net_tcp_count = 0, net_tcp_connections = 0; - unsigned long long rpc_count = 0, rpc_bad_format = 0, rpc_bad_auth = 0, rpc_bad_client = 0; - - for(l = 0; l < lines ;l++) { - words = procfile_linewords(ff, l); - if(!words) continue; - - type = procfile_lineword(ff, l, 0); - - if(do_rc == 1 && strcmp(type, "rc") == 0) { - if(words < 4) { - error("%s line of /proc/net/rpc/nfsd has %d words, expected %d", type, words, 4); - continue; - } - - rc_hits = strtoull(procfile_lineword(ff, l, 1), NULL, 10); - rc_misses = strtoull(procfile_lineword(ff, l, 2), NULL, 10); - rc_nocache = strtoull(procfile_lineword(ff, l, 3), NULL, 10); - - unsigned long long sum = rc_hits + rc_misses + rc_nocache; - if(sum == 0ULL) do_rc = -1; - else do_rc = 2; - } - else if(do_fh == 1 && strcmp(type, "fh") == 0) { - if(words < 6) { - error("%s line of /proc/net/rpc/nfsd has %d words, expected %d", type, words, 6); - continue; - } - - fh_stale = strtoull(procfile_lineword(ff, l, 1), NULL, 10); - fh_total_lookups = strtoull(procfile_lineword(ff, l, 2), NULL, 10); - fh_anonymous_lookups = strtoull(procfile_lineword(ff, l, 3), NULL, 10); - fh_dir_not_in_dcache = strtoull(procfile_lineword(ff, l, 4), NULL, 10); - fh_non_dir_not_in_dcache = strtoull(procfile_lineword(ff, l, 5), NULL, 10); - - unsigned long long sum = fh_stale + fh_total_lookups + fh_anonymous_lookups + fh_dir_not_in_dcache + fh_non_dir_not_in_dcache; - if(sum == 0ULL) do_fh = -1; - else do_fh = 2; - } - else if(do_io == 1 && strcmp(type, "io") == 0) { - if(words < 3) { - error("%s line of /proc/net/rpc/nfsd has %d words, expected %d", type, words, 3); - continue; - } - - io_read = strtoull(procfile_lineword(ff, l, 1), NULL, 10); - io_write = strtoull(procfile_lineword(ff, l, 2), NULL, 10); - - unsigned long long sum = io_read + io_write; - if(sum == 0ULL) do_io = -1; - else do_io = 2; - } - else if(do_th == 1 && strcmp(type, "th") == 0) { - if(words < 13) { - error("%s line of /proc/net/rpc/nfsd has %d words, expected %d", type, words, 13); - continue; - } - - th_threads = strtoull(procfile_lineword(ff, l, 1), NULL, 10); - th_fullcnt = strtoull(procfile_lineword(ff, l, 2), NULL, 10); - th_hist10 = (unsigned long long)(atof(procfile_lineword(ff, l, 3)) * 1000.0); - th_hist20 = (unsigned long long)(atof(procfile_lineword(ff, l, 4)) * 1000.0); - th_hist30 = (unsigned long long)(atof(procfile_lineword(ff, l, 5)) * 1000.0); - th_hist40 = (unsigned long long)(atof(procfile_lineword(ff, l, 6)) * 1000.0); - th_hist50 = (unsigned long long)(atof(procfile_lineword(ff, l, 7)) * 1000.0); - th_hist60 = (unsigned long long)(atof(procfile_lineword(ff, l, 8)) * 1000.0); - th_hist70 = (unsigned long long)(atof(procfile_lineword(ff, l, 9)) * 1000.0); - th_hist80 = (unsigned long long)(atof(procfile_lineword(ff, l, 10)) * 1000.0); - th_hist90 = (unsigned long long)(atof(procfile_lineword(ff, l, 11)) * 1000.0); - th_hist100 = (unsigned long long)(atof(procfile_lineword(ff, l, 12)) * 1000.0); - - // threads histogram has been disabled on recent kernels - // http://permalink.gmane.org/gmane.linux.nfs/24528 - unsigned long long sum = th_hist10 + th_hist20 + th_hist30 + th_hist40 + th_hist50 + th_hist60 + th_hist70 + th_hist80 + th_hist90 + th_hist100; - if(sum == 0ULL) { - if(!th_warning) { - info("Disabling /proc/net/rpc/nfsd threads histogram. It seems unused on this machine. It will be enabled automatically when found with data in it."); - th_warning = 1; - } - do_th = -1; - } - else do_th = 2; - } - else if(do_ra == 1 && strcmp(type, "ra") == 0) { - if(words < 13) { - error("%s line of /proc/net/rpc/nfsd has %d words, expected %d", type, words, 13); - continue; - } - - ra_size = strtoull(procfile_lineword(ff, l, 1), NULL, 10); - ra_hist10 = strtoull(procfile_lineword(ff, l, 2), NULL, 10); - ra_hist20 = strtoull(procfile_lineword(ff, l, 3), NULL, 10); - ra_hist30 = strtoull(procfile_lineword(ff, l, 4), NULL, 10); - ra_hist40 = strtoull(procfile_lineword(ff, l, 5), NULL, 10); - ra_hist50 = strtoull(procfile_lineword(ff, l, 6), NULL, 10); - ra_hist60 = strtoull(procfile_lineword(ff, l, 7), NULL, 10); - ra_hist70 = strtoull(procfile_lineword(ff, l, 8), NULL, 10); - ra_hist80 = strtoull(procfile_lineword(ff, l, 9), NULL, 10); - ra_hist90 = strtoull(procfile_lineword(ff, l, 10), NULL, 10); - ra_hist100 = strtoull(procfile_lineword(ff, l, 11), NULL, 10); - ra_none = strtoull(procfile_lineword(ff, l, 12), NULL, 10); - - unsigned long long sum = ra_hist10 + ra_hist20 + ra_hist30 + ra_hist40 + ra_hist50 + ra_hist60 + ra_hist70 + ra_hist80 + ra_hist90 + ra_hist100 + ra_none; - if(sum == 0ULL) { - if(!ra_warning) { - info("Disabling /proc/net/rpc/nfsd read ahead histogram. It seems unused on this machine. It will be enabled automatically when found with data in it."); - ra_warning = 1; - } - do_ra = -1; - } - else do_ra = 2; - } - else if(do_net == 1 && strcmp(type, "net") == 0) { - if(words < 5) { - error("%s line of /proc/net/rpc/nfsd has %d words, expected %d", type, words, 5); - continue; - } - - net_count = strtoull(procfile_lineword(ff, l, 1), NULL, 10); - net_udp_count = strtoull(procfile_lineword(ff, l, 2), NULL, 10); - net_tcp_count = strtoull(procfile_lineword(ff, l, 3), NULL, 10); - net_tcp_connections = strtoull(procfile_lineword(ff, l, 4), NULL, 10); - - unsigned long long sum = net_count + net_udp_count + net_tcp_count + net_tcp_connections; - if(sum == 0ULL) do_net = -1; - else do_net = 2; - } - else if(do_rpc == 1 && strcmp(type, "rpc") == 0) { - if(words < 6) { - error("%s line of /proc/net/rpc/nfsd has %d words, expected %d", type, words, 6); - continue; - } - - rpc_count = strtoull(procfile_lineword(ff, l, 1), NULL, 10); - rpc_bad_format = strtoull(procfile_lineword(ff, l, 2), NULL, 10); - rpc_bad_auth = strtoull(procfile_lineword(ff, l, 3), NULL, 10); - rpc_bad_client = strtoull(procfile_lineword(ff, l, 4), NULL, 10); - - unsigned long long sum = rpc_count + rpc_bad_format + rpc_bad_auth + rpc_bad_client; - if(sum == 0ULL) do_rpc = -1; - else do_rpc = 2; - } - else if(do_proc2 == 1 && strcmp(type, "proc2") == 0) { - // the first number is the count of numbers present - // so we start for word 2 - - unsigned long long sum = 0; - unsigned int i, j; - for(i = 0, j = 2; j < words && nfsd_proc_values[i].name[0] ; i++, j++) { - nfsd_proc_values[i].proc2 = strtoull(procfile_lineword(ff, l, j), NULL, 10); - nfsd_proc_values[i].present2 = 1; - sum += nfsd_proc_values[i].proc2; - } - - if(sum == 0ULL) { - if(!proc2_warning) { - error("Disabling /proc/net/rpc/nfsd v2 procedure calls chart. It seems unused on this machine. It will be enabled automatically when found with data in it."); - proc2_warning = 1; - } - do_proc2 = 0; - } - else do_proc2 = 2; - } - else if(do_proc3 == 1 && strcmp(type, "proc3") == 0) { - // the first number is the count of numbers present - // so we start for word 2 - - unsigned long long sum = 0; - unsigned int i, j; - for(i = 0, j = 2; j < words && nfsd_proc_values[i].name[0] ; i++, j++) { - nfsd_proc_values[i].proc3 = strtoull(procfile_lineword(ff, l, j), NULL, 10); - nfsd_proc_values[i].present3 = 1; - sum += nfsd_proc_values[i].proc3; - } - - if(sum == 0ULL) { - if(!proc3_warning) { - info("Disabling /proc/net/rpc/nfsd v3 procedure calls chart. It seems unused on this machine. It will be enabled automatically when found with data in it."); - proc3_warning = 1; - } - do_proc3 = 0; - } - else do_proc3 = 2; - } - else if(do_proc4 == 1 && strcmp(type, "proc4") == 0) { - // the first number is the count of numbers present - // so we start for word 2 - - unsigned long long sum = 0; - unsigned int i, j; - for(i = 0, j = 2; j < words && nfsd_proc_values[i].name[0] ; i++, j++) { - nfsd_proc_values[i].proc4 = strtoull(procfile_lineword(ff, l, j), NULL, 10); - nfsd_proc_values[i].present4 = 1; - sum += nfsd_proc_values[i].proc4; - } - - if(sum == 0ULL) { - if(!proc4_warning) { - info("Disabling /proc/net/rpc/nfsd v4 procedure calls chart. It seems unused on this machine. It will be enabled automatically when found with data in it."); - proc4_warning = 1; - } - do_proc4 = 0; - } - else do_proc4 = 2; - } - else if(do_proc4ops == 1 && strcmp(type, "proc4ops") == 0) { - // the first number is the count of numbers present - // so we start for word 2 - - unsigned long long sum = 0; - unsigned int i, j; - for(i = 0, j = 2; j < words && nfsd4_ops_values[i].name[0] ; i++, j++) { - nfsd4_ops_values[i].value = strtoull(procfile_lineword(ff, l, j), NULL, 10); - nfsd4_ops_values[i].present = 1; - sum += nfsd4_ops_values[i].value; - } - - if(sum == 0ULL) { - if(!proc4ops_warning) { - info("Disabling /proc/net/rpc/nfsd v4 operations chart. It seems unused on this machine. It will be enabled automatically when found with data in it."); - proc4ops_warning = 1; - } - do_proc4ops = 0; - } - else do_proc4ops = 2; - } - } - - RRDSET *st; - - // -------------------------------------------------------------------- - - if(do_rc == 2) { - st = rrdset_find_bytype("nfsd", "readcache"); - if(!st) { - st = rrdset_create("nfsd", "readcache", NULL, "nfsd", NULL, "Read Cache", "reads/s", 5000, update_every, RRDSET_TYPE_STACKED); - - rrddim_add(st, "hits", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "misses", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "nocache", NULL, 1, 1, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); - - rrddim_set(st, "hits", rc_hits); - rrddim_set(st, "misses", rc_misses); - rrddim_set(st, "nocache", rc_nocache); - rrdset_done(st); - } - - // -------------------------------------------------------------------- - - if(do_fh == 2) { - st = rrdset_find_bytype("nfsd", "filehandles"); - if(!st) { - st = rrdset_create("nfsd", "filehandles", NULL, "nfsd", NULL, "File Handles", "handles/s", 5001, update_every, RRDSET_TYPE_LINE); - st->isdetail = 1; - - rrddim_add(st, "stale", NULL, 1, 1, RRDDIM_ABSOLUTE); - rrddim_add(st, "total_lookups", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "anonymous_lookups", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "dir_not_in_dcache", NULL, -1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "non_dir_not_in_dcache", NULL, -1, 1, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); - - rrddim_set(st, "stale", fh_stale); - rrddim_set(st, "total_lookups", fh_total_lookups); - rrddim_set(st, "anonymous_lookups", fh_anonymous_lookups); - rrddim_set(st, "dir_not_in_dcache", fh_dir_not_in_dcache); - rrddim_set(st, "non_dir_not_in_dcache", fh_non_dir_not_in_dcache); - rrdset_done(st); - } - - // -------------------------------------------------------------------- - - if(do_io == 2) { - st = rrdset_find_bytype("nfsd", "io"); - if(!st) { - st = rrdset_create("nfsd", "io", NULL, "nfsd", NULL, "I/O", "kilobytes/s", 5002, update_every, RRDSET_TYPE_AREA); - - rrddim_add(st, "read", NULL, 1, 1000, RRDDIM_INCREMENTAL); - rrddim_add(st, "write", NULL, -1, 1000, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); - - rrddim_set(st, "read", io_read); - rrddim_set(st, "write", io_write); - rrdset_done(st); - } - - // -------------------------------------------------------------------- - - if(do_th == 2) { - st = rrdset_find_bytype("nfsd", "threads"); - if(!st) { - st = rrdset_create("nfsd", "threads", NULL, "nfsd", NULL, "Threads", "threads", 5003, update_every, RRDSET_TYPE_LINE); - - rrddim_add(st, "threads", NULL, 1, 1, RRDDIM_ABSOLUTE); - } - else rrdset_next(st); - - rrddim_set(st, "threads", th_threads); - rrdset_done(st); - - st = rrdset_find_bytype("nfsd", "threads_fullcnt"); - if(!st) { - st = rrdset_create("nfsd", "threads_fullcnt", NULL, "nfsd", NULL, "Threads Full Count", "ops/s", 5004, update_every, RRDSET_TYPE_LINE); - - rrddim_add(st, "full_count", NULL, 1, 1, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); - - rrddim_set(st, "full_count", th_fullcnt); - rrdset_done(st); - - st = rrdset_find_bytype("nfsd", "threads_histogram"); - if(!st) { - st = rrdset_create("nfsd", "threads_histogram", NULL, "nfsd", NULL, "Threads Usage Histogram", "percentage", 5005, update_every, RRDSET_TYPE_LINE); - - rrddim_add(st, "0%-10%", NULL, 1, 1000, RRDDIM_ABSOLUTE); - rrddim_add(st, "10%-20%", NULL, 1, 1000, RRDDIM_ABSOLUTE); - rrddim_add(st, "20%-30%", NULL, 1, 1000, RRDDIM_ABSOLUTE); - rrddim_add(st, "30%-40%", NULL, 1, 1000, RRDDIM_ABSOLUTE); - rrddim_add(st, "40%-50%", NULL, 1, 1000, RRDDIM_ABSOLUTE); - rrddim_add(st, "50%-60%", NULL, 1, 1000, RRDDIM_ABSOLUTE); - rrddim_add(st, "60%-70%", NULL, 1, 1000, RRDDIM_ABSOLUTE); - rrddim_add(st, "70%-80%", NULL, 1, 1000, RRDDIM_ABSOLUTE); - rrddim_add(st, "80%-90%", NULL, 1, 1000, RRDDIM_ABSOLUTE); - rrddim_add(st, "90%-100%", NULL, 1, 1000, RRDDIM_ABSOLUTE); - } - else rrdset_next(st); - - rrddim_set(st, "0%-10%", th_hist10); - rrddim_set(st, "10%-20%", th_hist20); - rrddim_set(st, "20%-30%", th_hist30); - rrddim_set(st, "30%-40%", th_hist40); - rrddim_set(st, "40%-50%", th_hist50); - rrddim_set(st, "50%-60%", th_hist60); - rrddim_set(st, "60%-70%", th_hist70); - rrddim_set(st, "70%-80%", th_hist80); - rrddim_set(st, "80%-90%", th_hist90); - rrddim_set(st, "90%-100%", th_hist100); - rrdset_done(st); - } - - // -------------------------------------------------------------------- - - if(do_ra == 2) { - st = rrdset_find_bytype("nfsd", "readahead"); - if(!st) { - st = rrdset_create("nfsd", "readahead", NULL, "nfsd", NULL, "Read Ahead Depth", "percentage", 5005, update_every, RRDSET_TYPE_STACKED); - - rrddim_add(st, "10%", NULL, 1, 1, RRDDIM_PCENT_OVER_DIFF_TOTAL); - rrddim_add(st, "20%", NULL, 1, 1, RRDDIM_PCENT_OVER_DIFF_TOTAL); - rrddim_add(st, "30%", NULL, 1, 1, RRDDIM_PCENT_OVER_DIFF_TOTAL); - rrddim_add(st, "40%", NULL, 1, 1, RRDDIM_PCENT_OVER_DIFF_TOTAL); - rrddim_add(st, "50%", NULL, 1, 1, RRDDIM_PCENT_OVER_DIFF_TOTAL); - rrddim_add(st, "60%", NULL, 1, 1, RRDDIM_PCENT_OVER_DIFF_TOTAL); - rrddim_add(st, "70%", NULL, 1, 1, RRDDIM_PCENT_OVER_DIFF_TOTAL); - rrddim_add(st, "80%", NULL, 1, 1, RRDDIM_PCENT_OVER_DIFF_TOTAL); - rrddim_add(st, "90%", NULL, 1, 1, RRDDIM_PCENT_OVER_DIFF_TOTAL); - rrddim_add(st, "100%", NULL, 1, 1, RRDDIM_PCENT_OVER_DIFF_TOTAL); - rrddim_add(st, "misses", NULL, 1, 1, RRDDIM_PCENT_OVER_DIFF_TOTAL); - } - else rrdset_next(st); - - // ignore ra_size - if(ra_size) {}; - - rrddim_set(st, "10%", ra_hist10); - rrddim_set(st, "20%", ra_hist20); - rrddim_set(st, "30%", ra_hist30); - rrddim_set(st, "40%", ra_hist40); - rrddim_set(st, "50%", ra_hist50); - rrddim_set(st, "60%", ra_hist60); - rrddim_set(st, "70%", ra_hist70); - rrddim_set(st, "80%", ra_hist80); - rrddim_set(st, "90%", ra_hist90); - rrddim_set(st, "100%", ra_hist100); - rrddim_set(st, "misses", ra_none); - rrdset_done(st); - } - - // -------------------------------------------------------------------- - - if(do_net == 2) { - st = rrdset_find_bytype("nfsd", "net"); - if(!st) { - st = rrdset_create("nfsd", "net", NULL, "nfsd", NULL, "Network Reads", "reads/s", 5007, update_every, RRDSET_TYPE_STACKED); - st->isdetail = 1; - - rrddim_add(st, "udp", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "tcp", NULL, 1, 1, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); - - // ignore net_count, net_tcp_connections - if(net_count) {}; - if(net_tcp_connections) {}; - - rrddim_set(st, "udp", net_udp_count); - rrddim_set(st, "tcp", net_tcp_count); - rrdset_done(st); - } - - // -------------------------------------------------------------------- - - if(do_rpc == 2) { - st = rrdset_find_bytype("nfsd", "rpc"); - if(!st) { - st = rrdset_create("nfsd", "rpc", NULL, "nfsd", NULL, "Remote Procedure Calls", "calls/s", 5008, update_every, RRDSET_TYPE_LINE); - st->isdetail = 1; - - rrddim_add(st, "all", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "bad_format", NULL, -1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "bad_auth", NULL, -1, 1, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); - - // ignore rpc_bad_client - if(rpc_bad_client) {}; - - rrddim_set(st, "all", rpc_count); - rrddim_set(st, "bad_format", rpc_bad_format); - rrddim_set(st, "bad_auth", rpc_bad_auth); - rrdset_done(st); - } - - // -------------------------------------------------------------------- - - if(do_proc2 == 2) { - unsigned int i; - st = rrdset_find_bytype("nfsd", "proc2"); - if(!st) { - st = rrdset_create("nfsd", "proc2", NULL, "nfsd", NULL, "NFS v2 Calls", "calls/s", 5009, update_every, RRDSET_TYPE_STACKED); - - for(i = 0; nfsd_proc_values[i].present2 ; i++) - rrddim_add(st, nfsd_proc_values[i].name, NULL, 1, 1, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); - - for(i = 0; nfsd_proc_values[i].present2 ; i++) - rrddim_set(st, nfsd_proc_values[i].name, nfsd_proc_values[i].proc2); - - rrdset_done(st); - } - - // -------------------------------------------------------------------- - - if(do_proc3 == 2) { - unsigned int i; - st = rrdset_find_bytype("nfsd", "proc3"); - if(!st) { - st = rrdset_create("nfsd", "proc3", NULL, "nfsd", NULL, "NFS v3 Calls", "calls/s", 5010, update_every, RRDSET_TYPE_STACKED); - - for(i = 0; nfsd_proc_values[i].present3 ; i++) - rrddim_add(st, nfsd_proc_values[i].name, NULL, 1, 1, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); - - for(i = 0; nfsd_proc_values[i].present3 ; i++) - rrddim_set(st, nfsd_proc_values[i].name, nfsd_proc_values[i].proc3); - - rrdset_done(st); - } - - // -------------------------------------------------------------------- - - if(do_proc4 == 2) { - unsigned int i; - st = rrdset_find_bytype("nfsd", "proc4"); - if(!st) { - st = rrdset_create("nfsd", "proc4", NULL, "nfsd", NULL, "NFS v4 Calls", "calls/s", 5011, update_every, RRDSET_TYPE_STACKED); - - for(i = 0; nfsd_proc_values[i].present4 ; i++) - rrddim_add(st, nfsd_proc_values[i].name, NULL, 1, 1, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); + static procfile *ff = NULL; + static int do_rc = -1, do_fh = -1, do_io = -1, do_th = -1, do_ra = -1, do_net = -1, do_rpc = -1, do_proc2 = -1, do_proc3 = -1, do_proc4 = -1, do_proc4ops = -1; + static int ra_warning = 0, th_warning = 0, proc2_warning = 0, proc3_warning = 0, proc4_warning = 0, proc4ops_warning = 0; + + if(dt) {}; + + if(!ff) { + char filename[FILENAME_MAX + 1]; + snprintfz(filename, FILENAME_MAX, "%s%s", global_host_prefix, "/proc/net/rpc/nfsd"); + ff = procfile_open(config_get("plugin:proc:/proc/net/rpc/nfsd", "filename to monitor", filename), " \t", PROCFILE_FLAG_DEFAULT); + } + if(!ff) return 1; + + ff = procfile_readall(ff); + if(!ff) return 0; // we return 0, so that we will retry to open it next time + + if(do_rc == -1) do_rc = config_get_boolean("plugin:proc:/proc/net/rpc/nfsd", "read cache", 1); + if(do_fh == -1) do_fh = config_get_boolean("plugin:proc:/proc/net/rpc/nfsd", "file handles", 1); + if(do_io == -1) do_io = config_get_boolean("plugin:proc:/proc/net/rpc/nfsd", "I/O", 1); + if(do_th == -1) do_th = config_get_boolean("plugin:proc:/proc/net/rpc/nfsd", "threads", 1); + if(do_ra == -1) do_ra = config_get_boolean("plugin:proc:/proc/net/rpc/nfsd", "read ahead", 1); + if(do_net == -1) do_net = config_get_boolean("plugin:proc:/proc/net/rpc/nfsd", "network", 1); + if(do_rpc == -1) do_rpc = config_get_boolean("plugin:proc:/proc/net/rpc/nfsd", "rpc", 1); + if(do_proc2 == -1) do_proc2 = config_get_boolean("plugin:proc:/proc/net/rpc/nfsd", "NFS v2 procedures", 1); + if(do_proc3 == -1) do_proc3 = config_get_boolean("plugin:proc:/proc/net/rpc/nfsd", "NFS v3 procedures", 1); + if(do_proc4 == -1) do_proc4 = config_get_boolean("plugin:proc:/proc/net/rpc/nfsd", "NFS v4 procedures", 1); + if(do_proc4ops == -1) do_proc4ops = config_get_boolean("plugin:proc:/proc/net/rpc/nfsd", "NFS v4 operations", 1); + + // if they are enabled, reset them to 1 + // later we do them =2 to avoid doing strcmp for all lines + if(do_rc) do_rc = 1; + if(do_fh) do_fh = 1; + if(do_io) do_io = 1; + if(do_th) do_th = 1; + if(do_ra) do_ra = 1; + if(do_net) do_net = 1; + if(do_rpc) do_rpc = 1; + if(do_proc2) do_proc2 = 1; + if(do_proc3) do_proc3 = 1; + if(do_proc4) do_proc4 = 1; + if(do_proc4ops) do_proc4ops = 1; + + uint32_t lines = procfile_lines(ff), l; + uint32_t words; + + char *type; + unsigned long long rc_hits = 0, rc_misses = 0, rc_nocache = 0; + unsigned long long fh_stale = 0, fh_total_lookups = 0, fh_anonymous_lookups = 0, fh_dir_not_in_dcache = 0, fh_non_dir_not_in_dcache = 0; + unsigned long long io_read = 0, io_write = 0; + unsigned long long th_threads = 0, th_fullcnt = 0, th_hist10 = 0, th_hist20 = 0, th_hist30 = 0, th_hist40 = 0, th_hist50 = 0, th_hist60 = 0, th_hist70 = 0, th_hist80 = 0, th_hist90 = 0, th_hist100 = 0; + unsigned long long ra_size = 0, ra_hist10 = 0, ra_hist20 = 0, ra_hist30 = 0, ra_hist40 = 0, ra_hist50 = 0, ra_hist60 = 0, ra_hist70 = 0, ra_hist80 = 0, ra_hist90 = 0, ra_hist100 = 0, ra_none = 0; + unsigned long long net_count = 0, net_udp_count = 0, net_tcp_count = 0, net_tcp_connections = 0; + unsigned long long rpc_count = 0, rpc_bad_format = 0, rpc_bad_auth = 0, rpc_bad_client = 0; + + for(l = 0; l < lines ;l++) { + words = procfile_linewords(ff, l); + if(!words) continue; + + type = procfile_lineword(ff, l, 0); + + if(do_rc == 1 && strcmp(type, "rc") == 0) { + if(words < 4) { + error("%s line of /proc/net/rpc/nfsd has %u words, expected %d", type, words, 4); + continue; + } + + rc_hits = strtoull(procfile_lineword(ff, l, 1), NULL, 10); + rc_misses = strtoull(procfile_lineword(ff, l, 2), NULL, 10); + rc_nocache = strtoull(procfile_lineword(ff, l, 3), NULL, 10); + + unsigned long long sum = rc_hits + rc_misses + rc_nocache; + if(sum == 0ULL) do_rc = -1; + else do_rc = 2; + } + else if(do_fh == 1 && strcmp(type, "fh") == 0) { + if(words < 6) { + error("%s line of /proc/net/rpc/nfsd has %u words, expected %d", type, words, 6); + continue; + } + + fh_stale = strtoull(procfile_lineword(ff, l, 1), NULL, 10); + fh_total_lookups = strtoull(procfile_lineword(ff, l, 2), NULL, 10); + fh_anonymous_lookups = strtoull(procfile_lineword(ff, l, 3), NULL, 10); + fh_dir_not_in_dcache = strtoull(procfile_lineword(ff, l, 4), NULL, 10); + fh_non_dir_not_in_dcache = strtoull(procfile_lineword(ff, l, 5), NULL, 10); + + unsigned long long sum = fh_stale + fh_total_lookups + fh_anonymous_lookups + fh_dir_not_in_dcache + fh_non_dir_not_in_dcache; + if(sum == 0ULL) do_fh = -1; + else do_fh = 2; + } + else if(do_io == 1 && strcmp(type, "io") == 0) { + if(words < 3) { + error("%s line of /proc/net/rpc/nfsd has %u words, expected %d", type, words, 3); + continue; + } + + io_read = strtoull(procfile_lineword(ff, l, 1), NULL, 10); + io_write = strtoull(procfile_lineword(ff, l, 2), NULL, 10); + + unsigned long long sum = io_read + io_write; + if(sum == 0ULL) do_io = -1; + else do_io = 2; + } + else if(do_th == 1 && strcmp(type, "th") == 0) { + if(words < 13) { + error("%s line of /proc/net/rpc/nfsd has %u words, expected %d", type, words, 13); + continue; + } + + th_threads = strtoull(procfile_lineword(ff, l, 1), NULL, 10); + th_fullcnt = strtoull(procfile_lineword(ff, l, 2), NULL, 10); + th_hist10 = (unsigned long long)(atof(procfile_lineword(ff, l, 3)) * 1000.0); + th_hist20 = (unsigned long long)(atof(procfile_lineword(ff, l, 4)) * 1000.0); + th_hist30 = (unsigned long long)(atof(procfile_lineword(ff, l, 5)) * 1000.0); + th_hist40 = (unsigned long long)(atof(procfile_lineword(ff, l, 6)) * 1000.0); + th_hist50 = (unsigned long long)(atof(procfile_lineword(ff, l, 7)) * 1000.0); + th_hist60 = (unsigned long long)(atof(procfile_lineword(ff, l, 8)) * 1000.0); + th_hist70 = (unsigned long long)(atof(procfile_lineword(ff, l, 9)) * 1000.0); + th_hist80 = (unsigned long long)(atof(procfile_lineword(ff, l, 10)) * 1000.0); + th_hist90 = (unsigned long long)(atof(procfile_lineword(ff, l, 11)) * 1000.0); + th_hist100 = (unsigned long long)(atof(procfile_lineword(ff, l, 12)) * 1000.0); + + // threads histogram has been disabled on recent kernels + // http://permalink.gmane.org/gmane.linux.nfs/24528 + unsigned long long sum = th_hist10 + th_hist20 + th_hist30 + th_hist40 + th_hist50 + th_hist60 + th_hist70 + th_hist80 + th_hist90 + th_hist100; + if(sum == 0ULL) { + if(!th_warning) { + info("Disabling /proc/net/rpc/nfsd threads histogram. It seems unused on this machine. It will be enabled automatically when found with data in it."); + th_warning = 1; + } + do_th = -1; + } + else do_th = 2; + } + else if(do_ra == 1 && strcmp(type, "ra") == 0) { + if(words < 13) { + error("%s line of /proc/net/rpc/nfsd has %u words, expected %d", type, words, 13); + continue; + } + + ra_size = strtoull(procfile_lineword(ff, l, 1), NULL, 10); + ra_hist10 = strtoull(procfile_lineword(ff, l, 2), NULL, 10); + ra_hist20 = strtoull(procfile_lineword(ff, l, 3), NULL, 10); + ra_hist30 = strtoull(procfile_lineword(ff, l, 4), NULL, 10); + ra_hist40 = strtoull(procfile_lineword(ff, l, 5), NULL, 10); + ra_hist50 = strtoull(procfile_lineword(ff, l, 6), NULL, 10); + ra_hist60 = strtoull(procfile_lineword(ff, l, 7), NULL, 10); + ra_hist70 = strtoull(procfile_lineword(ff, l, 8), NULL, 10); + ra_hist80 = strtoull(procfile_lineword(ff, l, 9), NULL, 10); + ra_hist90 = strtoull(procfile_lineword(ff, l, 10), NULL, 10); + ra_hist100 = strtoull(procfile_lineword(ff, l, 11), NULL, 10); + ra_none = strtoull(procfile_lineword(ff, l, 12), NULL, 10); + + unsigned long long sum = ra_hist10 + ra_hist20 + ra_hist30 + ra_hist40 + ra_hist50 + ra_hist60 + ra_hist70 + ra_hist80 + ra_hist90 + ra_hist100 + ra_none; + if(sum == 0ULL) { + if(!ra_warning) { + info("Disabling /proc/net/rpc/nfsd read ahead histogram. It seems unused on this machine. It will be enabled automatically when found with data in it."); + ra_warning = 1; + } + do_ra = -1; + } + else do_ra = 2; + } + else if(do_net == 1 && strcmp(type, "net") == 0) { + if(words < 5) { + error("%s line of /proc/net/rpc/nfsd has %u words, expected %d", type, words, 5); + continue; + } + + net_count = strtoull(procfile_lineword(ff, l, 1), NULL, 10); + net_udp_count = strtoull(procfile_lineword(ff, l, 2), NULL, 10); + net_tcp_count = strtoull(procfile_lineword(ff, l, 3), NULL, 10); + net_tcp_connections = strtoull(procfile_lineword(ff, l, 4), NULL, 10); + + unsigned long long sum = net_count + net_udp_count + net_tcp_count + net_tcp_connections; + if(sum == 0ULL) do_net = -1; + else do_net = 2; + } + else if(do_rpc == 1 && strcmp(type, "rpc") == 0) { + if(words < 6) { + error("%s line of /proc/net/rpc/nfsd has %u words, expected %d", type, words, 6); + continue; + } + + rpc_count = strtoull(procfile_lineword(ff, l, 1), NULL, 10); + rpc_bad_format = strtoull(procfile_lineword(ff, l, 2), NULL, 10); + rpc_bad_auth = strtoull(procfile_lineword(ff, l, 3), NULL, 10); + rpc_bad_client = strtoull(procfile_lineword(ff, l, 4), NULL, 10); + + unsigned long long sum = rpc_count + rpc_bad_format + rpc_bad_auth + rpc_bad_client; + if(sum == 0ULL) do_rpc = -1; + else do_rpc = 2; + } + else if(do_proc2 == 1 && strcmp(type, "proc2") == 0) { + // the first number is the count of numbers present + // so we start for word 2 + + unsigned long long sum = 0; + unsigned int i, j; + for(i = 0, j = 2; j < words && nfsd_proc_values[i].name[0] ; i++, j++) { + nfsd_proc_values[i].proc2 = strtoull(procfile_lineword(ff, l, j), NULL, 10); + nfsd_proc_values[i].present2 = 1; + sum += nfsd_proc_values[i].proc2; + } + + if(sum == 0ULL) { + if(!proc2_warning) { + error("Disabling /proc/net/rpc/nfsd v2 procedure calls chart. It seems unused on this machine. It will be enabled automatically when found with data in it."); + proc2_warning = 1; + } + do_proc2 = 0; + } + else do_proc2 = 2; + } + else if(do_proc3 == 1 && strcmp(type, "proc3") == 0) { + // the first number is the count of numbers present + // so we start for word 2 + + unsigned long long sum = 0; + unsigned int i, j; + for(i = 0, j = 2; j < words && nfsd_proc_values[i].name[0] ; i++, j++) { + nfsd_proc_values[i].proc3 = strtoull(procfile_lineword(ff, l, j), NULL, 10); + nfsd_proc_values[i].present3 = 1; + sum += nfsd_proc_values[i].proc3; + } + + if(sum == 0ULL) { + if(!proc3_warning) { + info("Disabling /proc/net/rpc/nfsd v3 procedure calls chart. It seems unused on this machine. It will be enabled automatically when found with data in it."); + proc3_warning = 1; + } + do_proc3 = 0; + } + else do_proc3 = 2; + } + else if(do_proc4 == 1 && strcmp(type, "proc4") == 0) { + // the first number is the count of numbers present + // so we start for word 2 + + unsigned long long sum = 0; + unsigned int i, j; + for(i = 0, j = 2; j < words && nfsd_proc_values[i].name[0] ; i++, j++) { + nfsd_proc_values[i].proc4 = strtoull(procfile_lineword(ff, l, j), NULL, 10); + nfsd_proc_values[i].present4 = 1; + sum += nfsd_proc_values[i].proc4; + } + + if(sum == 0ULL) { + if(!proc4_warning) { + info("Disabling /proc/net/rpc/nfsd v4 procedure calls chart. It seems unused on this machine. It will be enabled automatically when found with data in it."); + proc4_warning = 1; + } + do_proc4 = 0; + } + else do_proc4 = 2; + } + else if(do_proc4ops == 1 && strcmp(type, "proc4ops") == 0) { + // the first number is the count of numbers present + // so we start for word 2 + + unsigned long long sum = 0; + unsigned int i, j; + for(i = 0, j = 2; j < words && nfsd4_ops_values[i].name[0] ; i++, j++) { + nfsd4_ops_values[i].value = strtoull(procfile_lineword(ff, l, j), NULL, 10); + nfsd4_ops_values[i].present = 1; + sum += nfsd4_ops_values[i].value; + } + + if(sum == 0ULL) { + if(!proc4ops_warning) { + info("Disabling /proc/net/rpc/nfsd v4 operations chart. It seems unused on this machine. It will be enabled automatically when found with data in it."); + proc4ops_warning = 1; + } + do_proc4ops = 0; + } + else do_proc4ops = 2; + } + } + + RRDSET *st; + + // -------------------------------------------------------------------- + + if(do_rc == 2) { + st = rrdset_find_bytype("nfsd", "readcache"); + if(!st) { + st = rrdset_create("nfsd", "readcache", NULL, "nfsd", NULL, "Read Cache", "reads/s", 5000, update_every, RRDSET_TYPE_STACKED); + + rrddim_add(st, "hits", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "misses", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "nocache", NULL, 1, 1, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); + + rrddim_set(st, "hits", rc_hits); + rrddim_set(st, "misses", rc_misses); + rrddim_set(st, "nocache", rc_nocache); + rrdset_done(st); + } + + // -------------------------------------------------------------------- + + if(do_fh == 2) { + st = rrdset_find_bytype("nfsd", "filehandles"); + if(!st) { + st = rrdset_create("nfsd", "filehandles", NULL, "nfsd", NULL, "File Handles", "handles/s", 5001, update_every, RRDSET_TYPE_LINE); + st->isdetail = 1; + + rrddim_add(st, "stale", NULL, 1, 1, RRDDIM_ABSOLUTE); + rrddim_add(st, "total_lookups", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "anonymous_lookups", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "dir_not_in_dcache", NULL, -1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "non_dir_not_in_dcache", NULL, -1, 1, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); + + rrddim_set(st, "stale", fh_stale); + rrddim_set(st, "total_lookups", fh_total_lookups); + rrddim_set(st, "anonymous_lookups", fh_anonymous_lookups); + rrddim_set(st, "dir_not_in_dcache", fh_dir_not_in_dcache); + rrddim_set(st, "non_dir_not_in_dcache", fh_non_dir_not_in_dcache); + rrdset_done(st); + } + + // -------------------------------------------------------------------- + + if(do_io == 2) { + st = rrdset_find_bytype("nfsd", "io"); + if(!st) { + st = rrdset_create("nfsd", "io", NULL, "nfsd", NULL, "I/O", "kilobytes/s", 5002, update_every, RRDSET_TYPE_AREA); + + rrddim_add(st, "read", NULL, 1, 1000, RRDDIM_INCREMENTAL); + rrddim_add(st, "write", NULL, -1, 1000, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); + + rrddim_set(st, "read", io_read); + rrddim_set(st, "write", io_write); + rrdset_done(st); + } + + // -------------------------------------------------------------------- + + if(do_th == 2) { + st = rrdset_find_bytype("nfsd", "threads"); + if(!st) { + st = rrdset_create("nfsd", "threads", NULL, "nfsd", NULL, "Threads", "threads", 5003, update_every, RRDSET_TYPE_LINE); + + rrddim_add(st, "threads", NULL, 1, 1, RRDDIM_ABSOLUTE); + } + else rrdset_next(st); + + rrddim_set(st, "threads", th_threads); + rrdset_done(st); + + st = rrdset_find_bytype("nfsd", "threads_fullcnt"); + if(!st) { + st = rrdset_create("nfsd", "threads_fullcnt", NULL, "nfsd", NULL, "Threads Full Count", "ops/s", 5004, update_every, RRDSET_TYPE_LINE); + + rrddim_add(st, "full_count", NULL, 1, 1, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); + + rrddim_set(st, "full_count", th_fullcnt); + rrdset_done(st); + + st = rrdset_find_bytype("nfsd", "threads_histogram"); + if(!st) { + st = rrdset_create("nfsd", "threads_histogram", NULL, "nfsd", NULL, "Threads Usage Histogram", "percentage", 5005, update_every, RRDSET_TYPE_LINE); + + rrddim_add(st, "0%-10%", NULL, 1, 1000, RRDDIM_ABSOLUTE); + rrddim_add(st, "10%-20%", NULL, 1, 1000, RRDDIM_ABSOLUTE); + rrddim_add(st, "20%-30%", NULL, 1, 1000, RRDDIM_ABSOLUTE); + rrddim_add(st, "30%-40%", NULL, 1, 1000, RRDDIM_ABSOLUTE); + rrddim_add(st, "40%-50%", NULL, 1, 1000, RRDDIM_ABSOLUTE); + rrddim_add(st, "50%-60%", NULL, 1, 1000, RRDDIM_ABSOLUTE); + rrddim_add(st, "60%-70%", NULL, 1, 1000, RRDDIM_ABSOLUTE); + rrddim_add(st, "70%-80%", NULL, 1, 1000, RRDDIM_ABSOLUTE); + rrddim_add(st, "80%-90%", NULL, 1, 1000, RRDDIM_ABSOLUTE); + rrddim_add(st, "90%-100%", NULL, 1, 1000, RRDDIM_ABSOLUTE); + } + else rrdset_next(st); + + rrddim_set(st, "0%-10%", th_hist10); + rrddim_set(st, "10%-20%", th_hist20); + rrddim_set(st, "20%-30%", th_hist30); + rrddim_set(st, "30%-40%", th_hist40); + rrddim_set(st, "40%-50%", th_hist50); + rrddim_set(st, "50%-60%", th_hist60); + rrddim_set(st, "60%-70%", th_hist70); + rrddim_set(st, "70%-80%", th_hist80); + rrddim_set(st, "80%-90%", th_hist90); + rrddim_set(st, "90%-100%", th_hist100); + rrdset_done(st); + } + + // -------------------------------------------------------------------- + + if(do_ra == 2) { + st = rrdset_find_bytype("nfsd", "readahead"); + if(!st) { + st = rrdset_create("nfsd", "readahead", NULL, "nfsd", NULL, "Read Ahead Depth", "percentage", 5005, update_every, RRDSET_TYPE_STACKED); + + rrddim_add(st, "10%", NULL, 1, 1, RRDDIM_PCENT_OVER_DIFF_TOTAL); + rrddim_add(st, "20%", NULL, 1, 1, RRDDIM_PCENT_OVER_DIFF_TOTAL); + rrddim_add(st, "30%", NULL, 1, 1, RRDDIM_PCENT_OVER_DIFF_TOTAL); + rrddim_add(st, "40%", NULL, 1, 1, RRDDIM_PCENT_OVER_DIFF_TOTAL); + rrddim_add(st, "50%", NULL, 1, 1, RRDDIM_PCENT_OVER_DIFF_TOTAL); + rrddim_add(st, "60%", NULL, 1, 1, RRDDIM_PCENT_OVER_DIFF_TOTAL); + rrddim_add(st, "70%", NULL, 1, 1, RRDDIM_PCENT_OVER_DIFF_TOTAL); + rrddim_add(st, "80%", NULL, 1, 1, RRDDIM_PCENT_OVER_DIFF_TOTAL); + rrddim_add(st, "90%", NULL, 1, 1, RRDDIM_PCENT_OVER_DIFF_TOTAL); + rrddim_add(st, "100%", NULL, 1, 1, RRDDIM_PCENT_OVER_DIFF_TOTAL); + rrddim_add(st, "misses", NULL, 1, 1, RRDDIM_PCENT_OVER_DIFF_TOTAL); + } + else rrdset_next(st); + + // ignore ra_size + if(ra_size) {}; + + rrddim_set(st, "10%", ra_hist10); + rrddim_set(st, "20%", ra_hist20); + rrddim_set(st, "30%", ra_hist30); + rrddim_set(st, "40%", ra_hist40); + rrddim_set(st, "50%", ra_hist50); + rrddim_set(st, "60%", ra_hist60); + rrddim_set(st, "70%", ra_hist70); + rrddim_set(st, "80%", ra_hist80); + rrddim_set(st, "90%", ra_hist90); + rrddim_set(st, "100%", ra_hist100); + rrddim_set(st, "misses", ra_none); + rrdset_done(st); + } + + // -------------------------------------------------------------------- + + if(do_net == 2) { + st = rrdset_find_bytype("nfsd", "net"); + if(!st) { + st = rrdset_create("nfsd", "net", NULL, "nfsd", NULL, "Network Reads", "reads/s", 5007, update_every, RRDSET_TYPE_STACKED); + st->isdetail = 1; + + rrddim_add(st, "udp", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "tcp", NULL, 1, 1, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); + + // ignore net_count, net_tcp_connections + if(net_count) {}; + if(net_tcp_connections) {}; + + rrddim_set(st, "udp", net_udp_count); + rrddim_set(st, "tcp", net_tcp_count); + rrdset_done(st); + } + + // -------------------------------------------------------------------- + + if(do_rpc == 2) { + st = rrdset_find_bytype("nfsd", "rpc"); + if(!st) { + st = rrdset_create("nfsd", "rpc", NULL, "nfsd", NULL, "Remote Procedure Calls", "calls/s", 5008, update_every, RRDSET_TYPE_LINE); + st->isdetail = 1; + + rrddim_add(st, "all", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "bad_format", NULL, -1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "bad_auth", NULL, -1, 1, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); + + // ignore rpc_bad_client + if(rpc_bad_client) {}; + + rrddim_set(st, "all", rpc_count); + rrddim_set(st, "bad_format", rpc_bad_format); + rrddim_set(st, "bad_auth", rpc_bad_auth); + rrdset_done(st); + } + + // -------------------------------------------------------------------- + + if(do_proc2 == 2) { + unsigned int i; + st = rrdset_find_bytype("nfsd", "proc2"); + if(!st) { + st = rrdset_create("nfsd", "proc2", NULL, "nfsd", NULL, "NFS v2 Calls", "calls/s", 5009, update_every, RRDSET_TYPE_STACKED); + + for(i = 0; nfsd_proc_values[i].present2 ; i++) + rrddim_add(st, nfsd_proc_values[i].name, NULL, 1, 1, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); + + for(i = 0; nfsd_proc_values[i].present2 ; i++) + rrddim_set(st, nfsd_proc_values[i].name, nfsd_proc_values[i].proc2); + + rrdset_done(st); + } + + // -------------------------------------------------------------------- + + if(do_proc3 == 2) { + unsigned int i; + st = rrdset_find_bytype("nfsd", "proc3"); + if(!st) { + st = rrdset_create("nfsd", "proc3", NULL, "nfsd", NULL, "NFS v3 Calls", "calls/s", 5010, update_every, RRDSET_TYPE_STACKED); + + for(i = 0; nfsd_proc_values[i].present3 ; i++) + rrddim_add(st, nfsd_proc_values[i].name, NULL, 1, 1, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); + + for(i = 0; nfsd_proc_values[i].present3 ; i++) + rrddim_set(st, nfsd_proc_values[i].name, nfsd_proc_values[i].proc3); + + rrdset_done(st); + } + + // -------------------------------------------------------------------- + + if(do_proc4 == 2) { + unsigned int i; + st = rrdset_find_bytype("nfsd", "proc4"); + if(!st) { + st = rrdset_create("nfsd", "proc4", NULL, "nfsd", NULL, "NFS v4 Calls", "calls/s", 5011, update_every, RRDSET_TYPE_STACKED); + + for(i = 0; nfsd_proc_values[i].present4 ; i++) + rrddim_add(st, nfsd_proc_values[i].name, NULL, 1, 1, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); - for(i = 0; nfsd_proc_values[i].present4 ; i++) - rrddim_set(st, nfsd_proc_values[i].name, nfsd_proc_values[i].proc4); + for(i = 0; nfsd_proc_values[i].present4 ; i++) + rrddim_set(st, nfsd_proc_values[i].name, nfsd_proc_values[i].proc4); - rrdset_done(st); - } + rrdset_done(st); + } - // -------------------------------------------------------------------- + // -------------------------------------------------------------------- - if(do_proc4ops == 2) { - unsigned int i; - st = rrdset_find_bytype("nfsd", "proc4ops"); - if(!st) { - st = rrdset_create("nfsd", "proc4ops", NULL, "nfsd", NULL, "NFS v4 Operations", "operations/s", 5012, update_every, RRDSET_TYPE_STACKED); + if(do_proc4ops == 2) { + unsigned int i; + st = rrdset_find_bytype("nfsd", "proc4ops"); + if(!st) { + st = rrdset_create("nfsd", "proc4ops", NULL, "nfsd", NULL, "NFS v4 Operations", "operations/s", 5012, update_every, RRDSET_TYPE_STACKED); - for(i = 0; nfsd4_ops_values[i].present ; i++) - rrddim_add(st, nfsd4_ops_values[i].name, NULL, 1, 1, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); + for(i = 0; nfsd4_ops_values[i].present ; i++) + rrddim_add(st, nfsd4_ops_values[i].name, NULL, 1, 1, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); - for(i = 0; nfsd4_ops_values[i].present ; i++) - rrddim_set(st, nfsd4_ops_values[i].name, nfsd4_ops_values[i].value); + for(i = 0; nfsd4_ops_values[i].present ; i++) + rrddim_set(st, nfsd4_ops_values[i].name, nfsd4_ops_values[i].value); - rrdset_done(st); - } + rrdset_done(st); + } - return 0; + return 0; } diff --git a/src/proc_net_snmp.c b/src/proc_net_snmp.c index e0ac6a263..a773f55f6 100644 --- a/src/proc_net_snmp.c +++ b/src/proc_net_snmp.c @@ -1,366 +1,354 @@ -#ifdef HAVE_CONFIG_H -#include <config.h> -#endif -#include <stdio.h> -#include <stdlib.h> -#include <string.h> - #include "common.h" -#include "log.h" -#include "appconfig.h" -#include "procfile.h" -#include "rrd.h" -#include "plugin_proc.h" -#define RRD_TYPE_NET_SNMP "ipv4" -#define RRD_TYPE_NET_SNMP_LEN strlen(RRD_TYPE_NET_SNMP) +#define RRD_TYPE_NET_SNMP "ipv4" +#define RRD_TYPE_NET_SNMP_LEN strlen(RRD_TYPE_NET_SNMP) int do_proc_net_snmp(int update_every, unsigned long long dt) { - static procfile *ff = NULL; - static int do_ip_packets = -1, do_ip_fragsout = -1, do_ip_fragsin = -1, do_ip_errors = -1, - do_tcp_sockets = -1, do_tcp_packets = -1, do_tcp_errors = -1, do_tcp_handshake = -1, - do_udp_packets = -1, do_udp_errors = -1; - - if(do_ip_packets == -1) do_ip_packets = config_get_boolean("plugin:proc:/proc/net/snmp", "ipv4 packets", 1); - if(do_ip_fragsout == -1) do_ip_fragsout = config_get_boolean("plugin:proc:/proc/net/snmp", "ipv4 fragments sent", 1); - if(do_ip_fragsin == -1) do_ip_fragsin = config_get_boolean("plugin:proc:/proc/net/snmp", "ipv4 fragments assembly", 1); - if(do_ip_errors == -1) do_ip_errors = config_get_boolean("plugin:proc:/proc/net/snmp", "ipv4 errors", 1); - if(do_tcp_sockets == -1) do_tcp_sockets = config_get_boolean("plugin:proc:/proc/net/snmp", "ipv4 TCP connections", 1); - if(do_tcp_packets == -1) do_tcp_packets = config_get_boolean("plugin:proc:/proc/net/snmp", "ipv4 TCP packets", 1); - if(do_tcp_errors == -1) do_tcp_errors = config_get_boolean("plugin:proc:/proc/net/snmp", "ipv4 TCP errors", 1); - if(do_tcp_handshake == -1) do_tcp_handshake = config_get_boolean("plugin:proc:/proc/net/snmp", "ipv4 TCP handshake issues", 1); - if(do_udp_packets == -1) do_udp_packets = config_get_boolean("plugin:proc:/proc/net/snmp", "ipv4 UDP packets", 1); - if(do_udp_errors == -1) do_udp_errors = config_get_boolean("plugin:proc:/proc/net/snmp", "ipv4 UDP errors", 1); - - if(dt) {}; - - if(!ff) { - char filename[FILENAME_MAX + 1]; - snprintfz(filename, FILENAME_MAX, "%s%s", global_host_prefix, "/proc/net/snmp"); - ff = procfile_open(config_get("plugin:proc:/proc/net/snmp", "filename to monitor", filename), " \t:", PROCFILE_FLAG_DEFAULT); - } - if(!ff) return 1; - - ff = procfile_readall(ff); - if(!ff) return 0; // we return 0, so that we will retry to open it next time - - uint32_t lines = procfile_lines(ff), l; - uint32_t words; - - RRDSET *st; - - for(l = 0; l < lines ;l++) { - if(strcmp(procfile_lineword(ff, l, 0), "Ip") == 0) { - l++; - - if(strcmp(procfile_lineword(ff, l, 0), "Ip") != 0) { - error("Cannot read Ip line from /proc/net/snmp."); - break; - } - - words = procfile_linewords(ff, l); - if(words < 20) { - error("Cannot read /proc/net/snmp Ip line. Expected 20 params, read %d.", words); - continue; - } - - // see also http://net-snmp.sourceforge.net/docs/mibs/ip.html - unsigned long long Forwarding, DefaultTTL, InReceives, InHdrErrors, InAddrErrors, ForwDatagrams, InUnknownProtos, InDiscards, InDelivers, - OutRequests, OutDiscards, OutNoRoutes, ReasmTimeout, ReasmReqds, ReasmOKs, ReasmFails, FragOKs, FragFails, FragCreates; - - Forwarding = strtoull(procfile_lineword(ff, l, 1), NULL, 10); - DefaultTTL = strtoull(procfile_lineword(ff, l, 2), NULL, 10); - InReceives = strtoull(procfile_lineword(ff, l, 3), NULL, 10); - InHdrErrors = strtoull(procfile_lineword(ff, l, 4), NULL, 10); - InAddrErrors = strtoull(procfile_lineword(ff, l, 5), NULL, 10); - ForwDatagrams = strtoull(procfile_lineword(ff, l, 6), NULL, 10); - InUnknownProtos = strtoull(procfile_lineword(ff, l, 7), NULL, 10); - InDiscards = strtoull(procfile_lineword(ff, l, 8), NULL, 10); - InDelivers = strtoull(procfile_lineword(ff, l, 9), NULL, 10); - OutRequests = strtoull(procfile_lineword(ff, l, 10), NULL, 10); - OutDiscards = strtoull(procfile_lineword(ff, l, 11), NULL, 10); - OutNoRoutes = strtoull(procfile_lineword(ff, l, 12), NULL, 10); - ReasmTimeout = strtoull(procfile_lineword(ff, l, 13), NULL, 10); - ReasmReqds = strtoull(procfile_lineword(ff, l, 14), NULL, 10); - ReasmOKs = strtoull(procfile_lineword(ff, l, 15), NULL, 10); - ReasmFails = strtoull(procfile_lineword(ff, l, 16), NULL, 10); - FragOKs = strtoull(procfile_lineword(ff, l, 17), NULL, 10); - FragFails = strtoull(procfile_lineword(ff, l, 18), NULL, 10); - FragCreates = strtoull(procfile_lineword(ff, l, 19), NULL, 10); - - // these are not counters - if(Forwarding) {}; // is forwarding enabled? - if(DefaultTTL) {}; // the default ttl on packets - if(ReasmTimeout) {}; // Reassembly timeout - - // this counter is not used - if(InDelivers) {}; // total number of packets delivered to IP user-protocols - - // -------------------------------------------------------------------- - - if(do_ip_packets) { - st = rrdset_find(RRD_TYPE_NET_SNMP ".packets"); - if(!st) { - st = rrdset_create(RRD_TYPE_NET_SNMP, "packets", NULL, "packets", NULL, "IPv4 Packets", "packets/s", 3000, update_every, RRDSET_TYPE_LINE); - - rrddim_add(st, "received", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "sent", NULL, -1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "forwarded", NULL, 1, 1, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); - - rrddim_set(st, "sent", OutRequests); - rrddim_set(st, "received", InReceives); - rrddim_set(st, "forwarded", ForwDatagrams); - rrdset_done(st); - } - - // -------------------------------------------------------------------- - - if(do_ip_fragsout) { - st = rrdset_find(RRD_TYPE_NET_SNMP ".fragsout"); - if(!st) { - st = rrdset_create(RRD_TYPE_NET_SNMP, "fragsout", NULL, "fragments", NULL, "IPv4 Fragments Sent", "packets/s", 3010, update_every, RRDSET_TYPE_LINE); - st->isdetail = 1; - - rrddim_add(st, "ok", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "failed", NULL, -1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "all", NULL, 1, 1, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); - - rrddim_set(st, "ok", FragOKs); - rrddim_set(st, "failed", FragFails); - rrddim_set(st, "all", FragCreates); - rrdset_done(st); - } - - // -------------------------------------------------------------------- - - if(do_ip_fragsin) { - st = rrdset_find(RRD_TYPE_NET_SNMP ".fragsin"); - if(!st) { - st = rrdset_create(RRD_TYPE_NET_SNMP, "fragsin", NULL, "fragments", NULL, "IPv4 Fragments Reassembly", "packets/s", 3011, update_every, RRDSET_TYPE_LINE); - st->isdetail = 1; - - rrddim_add(st, "ok", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "failed", NULL, -1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "all", NULL, 1, 1, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); - - rrddim_set(st, "ok", ReasmOKs); - rrddim_set(st, "failed", ReasmFails); - rrddim_set(st, "all", ReasmReqds); - rrdset_done(st); - } - - // -------------------------------------------------------------------- - - if(do_ip_errors) { - st = rrdset_find(RRD_TYPE_NET_SNMP ".errors"); - if(!st) { - st = rrdset_create(RRD_TYPE_NET_SNMP, "errors", NULL, "errors", NULL, "IPv4 Errors", "packets/s", 3002, update_every, RRDSET_TYPE_LINE); - st->isdetail = 1; - - rrddim_add(st, "InDiscards", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "OutDiscards", NULL, -1, 1, RRDDIM_INCREMENTAL); - - rrddim_add(st, "InHdrErrors", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "InAddrErrors", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "InUnknownProtos", NULL, 1, 1, RRDDIM_INCREMENTAL); - - rrddim_add(st, "OutNoRoutes", NULL, -1, 1, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); - - rrddim_set(st, "InDiscards", InDiscards); - rrddim_set(st, "OutDiscards", OutDiscards); - rrddim_set(st, "InHdrErrors", InHdrErrors); - rrddim_set(st, "InAddrErrors", InAddrErrors); - rrddim_set(st, "InUnknownProtos", InUnknownProtos); - rrddim_set(st, "OutNoRoutes", OutNoRoutes); - rrdset_done(st); - } - } - else if(strcmp(procfile_lineword(ff, l, 0), "Tcp") == 0) { - l++; - - if(strcmp(procfile_lineword(ff, l, 0), "Tcp") != 0) { - error("Cannot read Tcp line from /proc/net/snmp."); - break; - } - - words = procfile_linewords(ff, l); - if(words < 15) { - error("Cannot read /proc/net/snmp Tcp line. Expected 15 params, read %d.", words); - continue; - } - - unsigned long long RtoAlgorithm, RtoMin, RtoMax, MaxConn, ActiveOpens, PassiveOpens, AttemptFails, EstabResets, - CurrEstab, InSegs, OutSegs, RetransSegs, InErrs, OutRsts; - - RtoAlgorithm = strtoull(procfile_lineword(ff, l, 1), NULL, 10); - RtoMin = strtoull(procfile_lineword(ff, l, 2), NULL, 10); - RtoMax = strtoull(procfile_lineword(ff, l, 3), NULL, 10); - MaxConn = strtoull(procfile_lineword(ff, l, 4), NULL, 10); - ActiveOpens = strtoull(procfile_lineword(ff, l, 5), NULL, 10); - PassiveOpens = strtoull(procfile_lineword(ff, l, 6), NULL, 10); - AttemptFails = strtoull(procfile_lineword(ff, l, 7), NULL, 10); - EstabResets = strtoull(procfile_lineword(ff, l, 8), NULL, 10); - CurrEstab = strtoull(procfile_lineword(ff, l, 9), NULL, 10); - InSegs = strtoull(procfile_lineword(ff, l, 10), NULL, 10); - OutSegs = strtoull(procfile_lineword(ff, l, 11), NULL, 10); - RetransSegs = strtoull(procfile_lineword(ff, l, 12), NULL, 10); - InErrs = strtoull(procfile_lineword(ff, l, 13), NULL, 10); - OutRsts = strtoull(procfile_lineword(ff, l, 14), NULL, 10); - - // these are not counters - if(RtoAlgorithm) {}; - if(RtoMin) {}; - if(RtoMax) {}; - if(MaxConn) {}; - - // -------------------------------------------------------------------- - - // see http://net-snmp.sourceforge.net/docs/mibs/tcp.html - if(do_tcp_sockets) { - st = rrdset_find(RRD_TYPE_NET_SNMP ".tcpsock"); - if(!st) { - st = rrdset_create(RRD_TYPE_NET_SNMP, "tcpsock", NULL, "tcp", NULL, "IPv4 TCP Connections", "active connections", 2500, update_every, RRDSET_TYPE_LINE); - - rrddim_add(st, "connections", NULL, 1, 1, RRDDIM_ABSOLUTE); - } - else rrdset_next(st); - - rrddim_set(st, "connections", CurrEstab); - rrdset_done(st); - } - - // -------------------------------------------------------------------- - - if(do_tcp_packets) { - st = rrdset_find(RRD_TYPE_NET_SNMP ".tcppackets"); - if(!st) { - st = rrdset_create(RRD_TYPE_NET_SNMP, "tcppackets", NULL, "tcp", NULL, "IPv4 TCP Packets", "packets/s", 2600, update_every, RRDSET_TYPE_LINE); - - rrddim_add(st, "received", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "sent", NULL, -1, 1, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); - - rrddim_set(st, "received", InSegs); - rrddim_set(st, "sent", OutSegs); - rrdset_done(st); - } - - // -------------------------------------------------------------------- - - if(do_tcp_errors) { - st = rrdset_find(RRD_TYPE_NET_SNMP ".tcperrors"); - if(!st) { - st = rrdset_create(RRD_TYPE_NET_SNMP, "tcperrors", NULL, "tcp", NULL, "IPv4 TCP Errors", "packets/s", 2700, update_every, RRDSET_TYPE_LINE); - st->isdetail = 1; - - rrddim_add(st, "InErrs", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "RetransSegs", NULL, -1, 1, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); - - rrddim_set(st, "InErrs", InErrs); - rrddim_set(st, "RetransSegs", RetransSegs); - rrdset_done(st); - } - - // -------------------------------------------------------------------- - - if(do_tcp_handshake) { - st = rrdset_find(RRD_TYPE_NET_SNMP ".tcphandshake"); - if(!st) { - st = rrdset_create(RRD_TYPE_NET_SNMP, "tcphandshake", NULL, "tcp", NULL, "IPv4 TCP Handshake Issues", "events/s", 2900, update_every, RRDSET_TYPE_LINE); - st->isdetail = 1; - - rrddim_add(st, "EstabResets", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "OutRsts", NULL, -1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "ActiveOpens", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "PassiveOpens", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "AttemptFails", NULL, 1, 1, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); - - rrddim_set(st, "EstabResets", EstabResets); - rrddim_set(st, "OutRsts", OutRsts); - rrddim_set(st, "ActiveOpens", ActiveOpens); - rrddim_set(st, "PassiveOpens", PassiveOpens); - rrddim_set(st, "AttemptFails", AttemptFails); - rrdset_done(st); - } - } - else if(strcmp(procfile_lineword(ff, l, 0), "Udp") == 0) { - l++; - - if(strcmp(procfile_lineword(ff, l, 0), "Udp") != 0) { - error("Cannot read Udp line from /proc/net/snmp."); - break; - } - - words = procfile_linewords(ff, l); - if(words < 7) { - error("Cannot read /proc/net/snmp Udp line. Expected 7 params, read %d.", words); - continue; - } - - unsigned long long InDatagrams, NoPorts, InErrors, OutDatagrams, RcvbufErrors, SndbufErrors; - - InDatagrams = strtoull(procfile_lineword(ff, l, 1), NULL, 10); - NoPorts = strtoull(procfile_lineword(ff, l, 2), NULL, 10); - InErrors = strtoull(procfile_lineword(ff, l, 3), NULL, 10); - OutDatagrams = strtoull(procfile_lineword(ff, l, 4), NULL, 10); - RcvbufErrors = strtoull(procfile_lineword(ff, l, 5), NULL, 10); - SndbufErrors = strtoull(procfile_lineword(ff, l, 6), NULL, 10); - - // -------------------------------------------------------------------- - - // see http://net-snmp.sourceforge.net/docs/mibs/udp.html - if(do_udp_packets) { - st = rrdset_find(RRD_TYPE_NET_SNMP ".udppackets"); - if(!st) { - st = rrdset_create(RRD_TYPE_NET_SNMP, "udppackets", NULL, "udp", NULL, "IPv4 UDP Packets", "packets/s", 2601, update_every, RRDSET_TYPE_LINE); - - rrddim_add(st, "received", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "sent", NULL, -1, 1, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); - - rrddim_set(st, "received", InDatagrams); - rrddim_set(st, "sent", OutDatagrams); - rrdset_done(st); - } - - // -------------------------------------------------------------------- - - if(do_udp_errors) { - st = rrdset_find(RRD_TYPE_NET_SNMP ".udperrors"); - if(!st) { - st = rrdset_create(RRD_TYPE_NET_SNMP, "udperrors", NULL, "udp", NULL, "IPv4 UDP Errors", "events/s", 2701, update_every, RRDSET_TYPE_LINE); - st->isdetail = 1; - - rrddim_add(st, "RcvbufErrors", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "SndbufErrors", NULL, -1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "InErrors", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "NoPorts", NULL, 1, 1, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); - - rrddim_set(st, "InErrors", InErrors); - rrddim_set(st, "NoPorts", NoPorts); - rrddim_set(st, "RcvbufErrors", RcvbufErrors); - rrddim_set(st, "SndbufErrors", SndbufErrors); - rrdset_done(st); - } - } - } - - return 0; + static procfile *ff = NULL; + static int do_ip_packets = -1, do_ip_fragsout = -1, do_ip_fragsin = -1, do_ip_errors = -1, + do_tcp_sockets = -1, do_tcp_packets = -1, do_tcp_errors = -1, do_tcp_handshake = -1, + do_udp_packets = -1, do_udp_errors = -1; + + if(do_ip_packets == -1) do_ip_packets = config_get_boolean("plugin:proc:/proc/net/snmp", "ipv4 packets", 1); + if(do_ip_fragsout == -1) do_ip_fragsout = config_get_boolean("plugin:proc:/proc/net/snmp", "ipv4 fragments sent", 1); + if(do_ip_fragsin == -1) do_ip_fragsin = config_get_boolean("plugin:proc:/proc/net/snmp", "ipv4 fragments assembly", 1); + if(do_ip_errors == -1) do_ip_errors = config_get_boolean("plugin:proc:/proc/net/snmp", "ipv4 errors", 1); + if(do_tcp_sockets == -1) do_tcp_sockets = config_get_boolean("plugin:proc:/proc/net/snmp", "ipv4 TCP connections", 1); + if(do_tcp_packets == -1) do_tcp_packets = config_get_boolean("plugin:proc:/proc/net/snmp", "ipv4 TCP packets", 1); + if(do_tcp_errors == -1) do_tcp_errors = config_get_boolean("plugin:proc:/proc/net/snmp", "ipv4 TCP errors", 1); + if(do_tcp_handshake == -1) do_tcp_handshake = config_get_boolean("plugin:proc:/proc/net/snmp", "ipv4 TCP handshake issues", 1); + if(do_udp_packets == -1) do_udp_packets = config_get_boolean("plugin:proc:/proc/net/snmp", "ipv4 UDP packets", 1); + if(do_udp_errors == -1) do_udp_errors = config_get_boolean("plugin:proc:/proc/net/snmp", "ipv4 UDP errors", 1); + + if(dt) {}; + + if(!ff) { + char filename[FILENAME_MAX + 1]; + snprintfz(filename, FILENAME_MAX, "%s%s", global_host_prefix, "/proc/net/snmp"); + ff = procfile_open(config_get("plugin:proc:/proc/net/snmp", "filename to monitor", filename), " \t:", PROCFILE_FLAG_DEFAULT); + } + if(!ff) return 1; + + ff = procfile_readall(ff); + if(!ff) return 0; // we return 0, so that we will retry to open it next time + + uint32_t lines = procfile_lines(ff), l; + uint32_t words; + + RRDSET *st; + + for(l = 0; l < lines ;l++) { + if(strcmp(procfile_lineword(ff, l, 0), "Ip") == 0) { + l++; + + if(strcmp(procfile_lineword(ff, l, 0), "Ip") != 0) { + error("Cannot read Ip line from /proc/net/snmp."); + break; + } + + words = procfile_linewords(ff, l); + if(words < 20) { + error("Cannot read /proc/net/snmp Ip line. Expected 20 params, read %u.", words); + continue; + } + + // see also http://net-snmp.sourceforge.net/docs/mibs/ip.html + unsigned long long Forwarding, DefaultTTL, InReceives, InHdrErrors, InAddrErrors, ForwDatagrams, InUnknownProtos, InDiscards, InDelivers, + OutRequests, OutDiscards, OutNoRoutes, ReasmTimeout, ReasmReqds, ReasmOKs, ReasmFails, FragOKs, FragFails, FragCreates; + + Forwarding = strtoull(procfile_lineword(ff, l, 1), NULL, 10); + DefaultTTL = strtoull(procfile_lineword(ff, l, 2), NULL, 10); + InReceives = strtoull(procfile_lineword(ff, l, 3), NULL, 10); + InHdrErrors = strtoull(procfile_lineword(ff, l, 4), NULL, 10); + InAddrErrors = strtoull(procfile_lineword(ff, l, 5), NULL, 10); + ForwDatagrams = strtoull(procfile_lineword(ff, l, 6), NULL, 10); + InUnknownProtos = strtoull(procfile_lineword(ff, l, 7), NULL, 10); + InDiscards = strtoull(procfile_lineword(ff, l, 8), NULL, 10); + InDelivers = strtoull(procfile_lineword(ff, l, 9), NULL, 10); + OutRequests = strtoull(procfile_lineword(ff, l, 10), NULL, 10); + OutDiscards = strtoull(procfile_lineword(ff, l, 11), NULL, 10); + OutNoRoutes = strtoull(procfile_lineword(ff, l, 12), NULL, 10); + ReasmTimeout = strtoull(procfile_lineword(ff, l, 13), NULL, 10); + ReasmReqds = strtoull(procfile_lineword(ff, l, 14), NULL, 10); + ReasmOKs = strtoull(procfile_lineword(ff, l, 15), NULL, 10); + ReasmFails = strtoull(procfile_lineword(ff, l, 16), NULL, 10); + FragOKs = strtoull(procfile_lineword(ff, l, 17), NULL, 10); + FragFails = strtoull(procfile_lineword(ff, l, 18), NULL, 10); + FragCreates = strtoull(procfile_lineword(ff, l, 19), NULL, 10); + + // these are not counters + if(Forwarding) {}; // is forwarding enabled? + if(DefaultTTL) {}; // the default ttl on packets + if(ReasmTimeout) {}; // Reassembly timeout + + // this counter is not used + if(InDelivers) {}; // total number of packets delivered to IP user-protocols + + // -------------------------------------------------------------------- + + if(do_ip_packets) { + st = rrdset_find(RRD_TYPE_NET_SNMP ".packets"); + if(!st) { + st = rrdset_create(RRD_TYPE_NET_SNMP, "packets", NULL, "packets", NULL, "IPv4 Packets", "packets/s", 3000, update_every, RRDSET_TYPE_LINE); + + rrddim_add(st, "received", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "sent", NULL, -1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "forwarded", NULL, 1, 1, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); + + rrddim_set(st, "sent", OutRequests); + rrddim_set(st, "received", InReceives); + rrddim_set(st, "forwarded", ForwDatagrams); + rrdset_done(st); + } + + // -------------------------------------------------------------------- + + if(do_ip_fragsout) { + st = rrdset_find(RRD_TYPE_NET_SNMP ".fragsout"); + if(!st) { + st = rrdset_create(RRD_TYPE_NET_SNMP, "fragsout", NULL, "fragments", NULL, "IPv4 Fragments Sent", "packets/s", 3010, update_every, RRDSET_TYPE_LINE); + st->isdetail = 1; + + rrddim_add(st, "ok", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "failed", NULL, -1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "all", NULL, 1, 1, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); + + rrddim_set(st, "ok", FragOKs); + rrddim_set(st, "failed", FragFails); + rrddim_set(st, "all", FragCreates); + rrdset_done(st); + } + + // -------------------------------------------------------------------- + + if(do_ip_fragsin) { + st = rrdset_find(RRD_TYPE_NET_SNMP ".fragsin"); + if(!st) { + st = rrdset_create(RRD_TYPE_NET_SNMP, "fragsin", NULL, "fragments", NULL, "IPv4 Fragments Reassembly", "packets/s", 3011, update_every, RRDSET_TYPE_LINE); + st->isdetail = 1; + + rrddim_add(st, "ok", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "failed", NULL, -1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "all", NULL, 1, 1, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); + + rrddim_set(st, "ok", ReasmOKs); + rrddim_set(st, "failed", ReasmFails); + rrddim_set(st, "all", ReasmReqds); + rrdset_done(st); + } + + // -------------------------------------------------------------------- + + if(do_ip_errors) { + st = rrdset_find(RRD_TYPE_NET_SNMP ".errors"); + if(!st) { + st = rrdset_create(RRD_TYPE_NET_SNMP, "errors", NULL, "errors", NULL, "IPv4 Errors", "packets/s", 3002, update_every, RRDSET_TYPE_LINE); + st->isdetail = 1; + + rrddim_add(st, "InDiscards", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "OutDiscards", NULL, -1, 1, RRDDIM_INCREMENTAL); + + rrddim_add(st, "InHdrErrors", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "InAddrErrors", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "InUnknownProtos", NULL, 1, 1, RRDDIM_INCREMENTAL); + + rrddim_add(st, "OutNoRoutes", NULL, -1, 1, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); + + rrddim_set(st, "InDiscards", InDiscards); + rrddim_set(st, "OutDiscards", OutDiscards); + rrddim_set(st, "InHdrErrors", InHdrErrors); + rrddim_set(st, "InAddrErrors", InAddrErrors); + rrddim_set(st, "InUnknownProtos", InUnknownProtos); + rrddim_set(st, "OutNoRoutes", OutNoRoutes); + rrdset_done(st); + } + } + else if(strcmp(procfile_lineword(ff, l, 0), "Tcp") == 0) { + l++; + + if(strcmp(procfile_lineword(ff, l, 0), "Tcp") != 0) { + error("Cannot read Tcp line from /proc/net/snmp."); + break; + } + + words = procfile_linewords(ff, l); + if(words < 15) { + error("Cannot read /proc/net/snmp Tcp line. Expected 15 params, read %u.", words); + continue; + } + + unsigned long long RtoAlgorithm, RtoMin, RtoMax, MaxConn, ActiveOpens, PassiveOpens, AttemptFails, EstabResets, + CurrEstab, InSegs, OutSegs, RetransSegs, InErrs, OutRsts; + + RtoAlgorithm = strtoull(procfile_lineword(ff, l, 1), NULL, 10); + RtoMin = strtoull(procfile_lineword(ff, l, 2), NULL, 10); + RtoMax = strtoull(procfile_lineword(ff, l, 3), NULL, 10); + MaxConn = strtoull(procfile_lineword(ff, l, 4), NULL, 10); + ActiveOpens = strtoull(procfile_lineword(ff, l, 5), NULL, 10); + PassiveOpens = strtoull(procfile_lineword(ff, l, 6), NULL, 10); + AttemptFails = strtoull(procfile_lineword(ff, l, 7), NULL, 10); + EstabResets = strtoull(procfile_lineword(ff, l, 8), NULL, 10); + CurrEstab = strtoull(procfile_lineword(ff, l, 9), NULL, 10); + InSegs = strtoull(procfile_lineword(ff, l, 10), NULL, 10); + OutSegs = strtoull(procfile_lineword(ff, l, 11), NULL, 10); + RetransSegs = strtoull(procfile_lineword(ff, l, 12), NULL, 10); + InErrs = strtoull(procfile_lineword(ff, l, 13), NULL, 10); + OutRsts = strtoull(procfile_lineword(ff, l, 14), NULL, 10); + + // these are not counters + if(RtoAlgorithm) {}; + if(RtoMin) {}; + if(RtoMax) {}; + if(MaxConn) {}; + + // -------------------------------------------------------------------- + + // see http://net-snmp.sourceforge.net/docs/mibs/tcp.html + if(do_tcp_sockets) { + st = rrdset_find(RRD_TYPE_NET_SNMP ".tcpsock"); + if(!st) { + st = rrdset_create(RRD_TYPE_NET_SNMP, "tcpsock", NULL, "tcp", NULL, "IPv4 TCP Connections", "active connections", 2500, update_every, RRDSET_TYPE_LINE); + + rrddim_add(st, "connections", NULL, 1, 1, RRDDIM_ABSOLUTE); + } + else rrdset_next(st); + + rrddim_set(st, "connections", CurrEstab); + rrdset_done(st); + } + + // -------------------------------------------------------------------- + + if(do_tcp_packets) { + st = rrdset_find(RRD_TYPE_NET_SNMP ".tcppackets"); + if(!st) { + st = rrdset_create(RRD_TYPE_NET_SNMP, "tcppackets", NULL, "tcp", NULL, "IPv4 TCP Packets", "packets/s", 2600, update_every, RRDSET_TYPE_LINE); + + rrddim_add(st, "received", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "sent", NULL, -1, 1, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); + + rrddim_set(st, "received", InSegs); + rrddim_set(st, "sent", OutSegs); + rrdset_done(st); + } + + // -------------------------------------------------------------------- + + if(do_tcp_errors) { + st = rrdset_find(RRD_TYPE_NET_SNMP ".tcperrors"); + if(!st) { + st = rrdset_create(RRD_TYPE_NET_SNMP, "tcperrors", NULL, "tcp", NULL, "IPv4 TCP Errors", "packets/s", 2700, update_every, RRDSET_TYPE_LINE); + st->isdetail = 1; + + rrddim_add(st, "InErrs", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "RetransSegs", NULL, -1, 1, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); + + rrddim_set(st, "InErrs", InErrs); + rrddim_set(st, "RetransSegs", RetransSegs); + rrdset_done(st); + } + + // -------------------------------------------------------------------- + + if(do_tcp_handshake) { + st = rrdset_find(RRD_TYPE_NET_SNMP ".tcphandshake"); + if(!st) { + st = rrdset_create(RRD_TYPE_NET_SNMP, "tcphandshake", NULL, "tcp", NULL, "IPv4 TCP Handshake Issues", "events/s", 2900, update_every, RRDSET_TYPE_LINE); + st->isdetail = 1; + + rrddim_add(st, "EstabResets", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "OutRsts", NULL, -1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "ActiveOpens", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "PassiveOpens", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "AttemptFails", NULL, 1, 1, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); + + rrddim_set(st, "EstabResets", EstabResets); + rrddim_set(st, "OutRsts", OutRsts); + rrddim_set(st, "ActiveOpens", ActiveOpens); + rrddim_set(st, "PassiveOpens", PassiveOpens); + rrddim_set(st, "AttemptFails", AttemptFails); + rrdset_done(st); + } + } + else if(strcmp(procfile_lineword(ff, l, 0), "Udp") == 0) { + l++; + + if(strcmp(procfile_lineword(ff, l, 0), "Udp") != 0) { + error("Cannot read Udp line from /proc/net/snmp."); + break; + } + + words = procfile_linewords(ff, l); + if(words < 7) { + error("Cannot read /proc/net/snmp Udp line. Expected 7 params, read %u.", words); + continue; + } + + unsigned long long InDatagrams, NoPorts, InErrors, OutDatagrams, RcvbufErrors, SndbufErrors; + + InDatagrams = strtoull(procfile_lineword(ff, l, 1), NULL, 10); + NoPorts = strtoull(procfile_lineword(ff, l, 2), NULL, 10); + InErrors = strtoull(procfile_lineword(ff, l, 3), NULL, 10); + OutDatagrams = strtoull(procfile_lineword(ff, l, 4), NULL, 10); + RcvbufErrors = strtoull(procfile_lineword(ff, l, 5), NULL, 10); + SndbufErrors = strtoull(procfile_lineword(ff, l, 6), NULL, 10); + + // -------------------------------------------------------------------- + + // see http://net-snmp.sourceforge.net/docs/mibs/udp.html + if(do_udp_packets) { + st = rrdset_find(RRD_TYPE_NET_SNMP ".udppackets"); + if(!st) { + st = rrdset_create(RRD_TYPE_NET_SNMP, "udppackets", NULL, "udp", NULL, "IPv4 UDP Packets", "packets/s", 2601, update_every, RRDSET_TYPE_LINE); + + rrddim_add(st, "received", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "sent", NULL, -1, 1, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); + + rrddim_set(st, "received", InDatagrams); + rrddim_set(st, "sent", OutDatagrams); + rrdset_done(st); + } + + // -------------------------------------------------------------------- + + if(do_udp_errors) { + st = rrdset_find(RRD_TYPE_NET_SNMP ".udperrors"); + if(!st) { + st = rrdset_create(RRD_TYPE_NET_SNMP, "udperrors", NULL, "udp", NULL, "IPv4 UDP Errors", "events/s", 2701, update_every, RRDSET_TYPE_LINE); + st->isdetail = 1; + + rrddim_add(st, "RcvbufErrors", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "SndbufErrors", NULL, -1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "InErrors", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "NoPorts", NULL, 1, 1, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); + + rrddim_set(st, "InErrors", InErrors); + rrddim_set(st, "NoPorts", NoPorts); + rrddim_set(st, "RcvbufErrors", RcvbufErrors); + rrddim_set(st, "SndbufErrors", SndbufErrors); + rrdset_done(st); + } + } + } + + return 0; } diff --git a/src/proc_net_snmp6.c b/src/proc_net_snmp6.c index 885835a8c..97dc20edd 100644 --- a/src/proc_net_snmp6.c +++ b/src/proc_net_snmp6.c @@ -1,1067 +1,1054 @@ -#ifdef HAVE_CONFIG_H -#include <config.h> -#endif -#include <stdio.h> -#include <stdlib.h> -#include <string.h> -#include <unistd.h> - #include "common.h" -#include "log.h" -#include "appconfig.h" -#include "procfile.h" -#include "rrd.h" -#include "plugin_proc.h" -#define RRD_TYPE_NET_SNMP6 "ipv6" -#define RRD_TYPE_NET_SNMP6_LEN strlen(RRD_TYPE_NET_SNMP6) +#define RRD_TYPE_NET_SNMP6 "ipv6" +#define RRD_TYPE_NET_SNMP6_LEN strlen(RRD_TYPE_NET_SNMP6) int do_proc_net_snmp6(int update_every, unsigned long long dt) { - static procfile *ff = NULL; - static int gen_hashes = -1; - - static int do_ip_packets = -1, do_ip_fragsout = -1, do_ip_fragsin = -1, do_ip_errors = -1, - do_udplite_packets = -1, do_udplite_errors = -1, - do_udp_packets = -1, do_udp_errors = -1, - do_bandwidth = -1, do_mcast = -1, do_bcast = -1, do_mcast_p = -1, - do_icmp = -1, do_icmp_redir = -1, do_icmp_errors = -1, do_icmp_echos = -1, do_icmp_groupmemb = -1, - do_icmp_router = -1, do_icmp_neighbor = -1, do_icmp_mldv2 = -1, do_icmp_types = -1, do_ect = -1; - - static uint32_t hash_Ip6InReceives = -1; - - static uint32_t hash_Ip6InHdrErrors = -1; - static uint32_t hash_Ip6InTooBigErrors = -1; - static uint32_t hash_Ip6InNoRoutes = -1; - static uint32_t hash_Ip6InAddrErrors = -1; - static uint32_t hash_Ip6InUnknownProtos = -1; - static uint32_t hash_Ip6InTruncatedPkts = -1; - static uint32_t hash_Ip6InDiscards = -1; - static uint32_t hash_Ip6InDelivers = -1; - - static uint32_t hash_Ip6OutForwDatagrams = -1; - static uint32_t hash_Ip6OutRequests = -1; - static uint32_t hash_Ip6OutDiscards = -1; - static uint32_t hash_Ip6OutNoRoutes = -1; - - static uint32_t hash_Ip6ReasmTimeout = -1; - static uint32_t hash_Ip6ReasmReqds = -1; - static uint32_t hash_Ip6ReasmOKs = -1; - static uint32_t hash_Ip6ReasmFails = -1; - - static uint32_t hash_Ip6FragOKs = -1; - static uint32_t hash_Ip6FragFails = -1; - static uint32_t hash_Ip6FragCreates = -1; - - static uint32_t hash_Ip6InMcastPkts = -1; - static uint32_t hash_Ip6OutMcastPkts = -1; - - static uint32_t hash_Ip6InOctets = -1; - static uint32_t hash_Ip6OutOctets = -1; - - static uint32_t hash_Ip6InMcastOctets = -1; - static uint32_t hash_Ip6OutMcastOctets = -1; - static uint32_t hash_Ip6InBcastOctets = -1; - static uint32_t hash_Ip6OutBcastOctets = -1; - - static uint32_t hash_Ip6InNoECTPkts = -1; - static uint32_t hash_Ip6InECT1Pkts = -1; - static uint32_t hash_Ip6InECT0Pkts = -1; - static uint32_t hash_Ip6InCEPkts = -1; - - static uint32_t hash_Icmp6InMsgs = -1; - static uint32_t hash_Icmp6InErrors = -1; - static uint32_t hash_Icmp6OutMsgs = -1; - static uint32_t hash_Icmp6OutErrors = -1; - static uint32_t hash_Icmp6InCsumErrors = -1; - static uint32_t hash_Icmp6InDestUnreachs = -1; - static uint32_t hash_Icmp6InPktTooBigs = -1; - static uint32_t hash_Icmp6InTimeExcds = -1; - static uint32_t hash_Icmp6InParmProblems = -1; - static uint32_t hash_Icmp6InEchos = -1; - static uint32_t hash_Icmp6InEchoReplies = -1; - static uint32_t hash_Icmp6InGroupMembQueries = -1; - static uint32_t hash_Icmp6InGroupMembResponses = -1; - static uint32_t hash_Icmp6InGroupMembReductions = -1; - static uint32_t hash_Icmp6InRouterSolicits = -1; - static uint32_t hash_Icmp6InRouterAdvertisements = -1; - static uint32_t hash_Icmp6InNeighborSolicits = -1; - static uint32_t hash_Icmp6InNeighborAdvertisements = -1; - static uint32_t hash_Icmp6InRedirects = -1; - static uint32_t hash_Icmp6InMLDv2Reports = -1; - static uint32_t hash_Icmp6OutDestUnreachs = -1; - static uint32_t hash_Icmp6OutPktTooBigs = -1; - static uint32_t hash_Icmp6OutTimeExcds = -1; - static uint32_t hash_Icmp6OutParmProblems = -1; - static uint32_t hash_Icmp6OutEchos = -1; - static uint32_t hash_Icmp6OutEchoReplies = -1; - static uint32_t hash_Icmp6OutGroupMembQueries = -1; - static uint32_t hash_Icmp6OutGroupMembResponses = -1; - static uint32_t hash_Icmp6OutGroupMembReductions = -1; - static uint32_t hash_Icmp6OutRouterSolicits = -1; - static uint32_t hash_Icmp6OutRouterAdvertisements = -1; - static uint32_t hash_Icmp6OutNeighborSolicits = -1; - static uint32_t hash_Icmp6OutNeighborAdvertisements = -1; - static uint32_t hash_Icmp6OutRedirects = -1; - static uint32_t hash_Icmp6OutMLDv2Reports = -1; - static uint32_t hash_Icmp6InType1 = -1; - static uint32_t hash_Icmp6InType128 = -1; - static uint32_t hash_Icmp6InType129 = -1; - static uint32_t hash_Icmp6InType136 = -1; - static uint32_t hash_Icmp6OutType1 = -1; - static uint32_t hash_Icmp6OutType128 = -1; - static uint32_t hash_Icmp6OutType129 = -1; - static uint32_t hash_Icmp6OutType133 = -1; - static uint32_t hash_Icmp6OutType135 = -1; - static uint32_t hash_Icmp6OutType143 = -1; - - static uint32_t hash_Udp6InDatagrams = -1; - static uint32_t hash_Udp6NoPorts = -1; - static uint32_t hash_Udp6InErrors = -1; - static uint32_t hash_Udp6OutDatagrams = -1; - static uint32_t hash_Udp6RcvbufErrors = -1; - static uint32_t hash_Udp6SndbufErrors = -1; - static uint32_t hash_Udp6InCsumErrors = -1; - static uint32_t hash_Udp6IgnoredMulti = -1; - - static uint32_t hash_UdpLite6InDatagrams = -1; - static uint32_t hash_UdpLite6NoPorts = -1; - static uint32_t hash_UdpLite6InErrors = -1; - static uint32_t hash_UdpLite6OutDatagrams = -1; - static uint32_t hash_UdpLite6RcvbufErrors = -1; - static uint32_t hash_UdpLite6SndbufErrors = -1; - static uint32_t hash_UdpLite6InCsumErrors = -1; - - if(gen_hashes != 1) { - gen_hashes = 1; - hash_Ip6InReceives = simple_hash("Ip6InReceives"); - hash_Ip6InHdrErrors = simple_hash("Ip6InHdrErrors"); - hash_Ip6InTooBigErrors = simple_hash("Ip6InTooBigErrors"); - hash_Ip6InNoRoutes = simple_hash("Ip6InNoRoutes"); - hash_Ip6InAddrErrors = simple_hash("Ip6InAddrErrors"); - hash_Ip6InUnknownProtos = simple_hash("Ip6InUnknownProtos"); - hash_Ip6InTruncatedPkts = simple_hash("Ip6InTruncatedPkts"); - hash_Ip6InDiscards = simple_hash("Ip6InDiscards"); - hash_Ip6InDelivers = simple_hash("Ip6InDelivers"); - hash_Ip6OutForwDatagrams = simple_hash("Ip6OutForwDatagrams"); - hash_Ip6OutRequests = simple_hash("Ip6OutRequests"); - hash_Ip6OutDiscards = simple_hash("Ip6OutDiscards"); - hash_Ip6OutNoRoutes = simple_hash("Ip6OutNoRoutes"); - hash_Ip6ReasmTimeout = simple_hash("Ip6ReasmTimeout"); - hash_Ip6ReasmReqds = simple_hash("Ip6ReasmReqds"); - hash_Ip6ReasmOKs = simple_hash("Ip6ReasmOKs"); - hash_Ip6ReasmFails = simple_hash("Ip6ReasmFails"); - hash_Ip6FragOKs = simple_hash("Ip6FragOKs"); - hash_Ip6FragFails = simple_hash("Ip6FragFails"); - hash_Ip6FragCreates = simple_hash("Ip6FragCreates"); - hash_Ip6InMcastPkts = simple_hash("Ip6InMcastPkts"); - hash_Ip6OutMcastPkts = simple_hash("Ip6OutMcastPkts"); - hash_Ip6InOctets = simple_hash("Ip6InOctets"); - hash_Ip6OutOctets = simple_hash("Ip6OutOctets"); - hash_Ip6InMcastOctets = simple_hash("Ip6InMcastOctets"); - hash_Ip6OutMcastOctets = simple_hash("Ip6OutMcastOctets"); - hash_Ip6InBcastOctets = simple_hash("Ip6InBcastOctets"); - hash_Ip6OutBcastOctets = simple_hash("Ip6OutBcastOctets"); - hash_Ip6InNoECTPkts = simple_hash("Ip6InNoECTPkts"); - hash_Ip6InECT1Pkts = simple_hash("Ip6InECT1Pkts"); - hash_Ip6InECT0Pkts = simple_hash("Ip6InECT0Pkts"); - hash_Ip6InCEPkts = simple_hash("Ip6InCEPkts"); - hash_Icmp6InMsgs = simple_hash("Icmp6InMsgs"); - hash_Icmp6InErrors = simple_hash("Icmp6InErrors"); - hash_Icmp6OutMsgs = simple_hash("Icmp6OutMsgs"); - hash_Icmp6OutErrors = simple_hash("Icmp6OutErrors"); - hash_Icmp6InCsumErrors = simple_hash("Icmp6InCsumErrors"); - hash_Icmp6InDestUnreachs = simple_hash("Icmp6InDestUnreachs"); - hash_Icmp6InPktTooBigs = simple_hash("Icmp6InPktTooBigs"); - hash_Icmp6InTimeExcds = simple_hash("Icmp6InTimeExcds"); - hash_Icmp6InParmProblems = simple_hash("Icmp6InParmProblems"); - hash_Icmp6InEchos = simple_hash("Icmp6InEchos"); - hash_Icmp6InEchoReplies = simple_hash("Icmp6InEchoReplies"); - hash_Icmp6InGroupMembQueries = simple_hash("Icmp6InGroupMembQueries"); - hash_Icmp6InGroupMembResponses = simple_hash("Icmp6InGroupMembResponses"); - hash_Icmp6InGroupMembReductions = simple_hash("Icmp6InGroupMembReductions"); - hash_Icmp6InRouterSolicits = simple_hash("Icmp6InRouterSolicits"); - hash_Icmp6InRouterAdvertisements = simple_hash("Icmp6InRouterAdvertisements"); - hash_Icmp6InNeighborSolicits = simple_hash("Icmp6InNeighborSolicits"); - hash_Icmp6InNeighborAdvertisements = simple_hash("Icmp6InNeighborAdvertisements"); - hash_Icmp6InRedirects = simple_hash("Icmp6InRedirects"); - hash_Icmp6InMLDv2Reports = simple_hash("Icmp6InMLDv2Reports"); - hash_Icmp6OutDestUnreachs = simple_hash("Icmp6OutDestUnreachs"); - hash_Icmp6OutPktTooBigs = simple_hash("Icmp6OutPktTooBigs"); - hash_Icmp6OutTimeExcds = simple_hash("Icmp6OutTimeExcds"); - hash_Icmp6OutParmProblems = simple_hash("Icmp6OutParmProblems"); - hash_Icmp6OutEchos = simple_hash("Icmp6OutEchos"); - hash_Icmp6OutEchoReplies = simple_hash("Icmp6OutEchoReplies"); - hash_Icmp6OutGroupMembQueries = simple_hash("Icmp6OutGroupMembQueries"); - hash_Icmp6OutGroupMembResponses = simple_hash("Icmp6OutGroupMembResponses"); - hash_Icmp6OutGroupMembReductions = simple_hash("Icmp6OutGroupMembReductions"); - hash_Icmp6OutRouterSolicits = simple_hash("Icmp6OutRouterSolicits"); - hash_Icmp6OutRouterAdvertisements = simple_hash("Icmp6OutRouterAdvertisements"); - hash_Icmp6OutNeighborSolicits = simple_hash("Icmp6OutNeighborSolicits"); - hash_Icmp6OutNeighborAdvertisements = simple_hash("Icmp6OutNeighborAdvertisements"); - hash_Icmp6OutRedirects = simple_hash("Icmp6OutRedirects"); - hash_Icmp6OutMLDv2Reports = simple_hash("Icmp6OutMLDv2Reports"); - hash_Icmp6InType1 = simple_hash("Icmp6InType1"); - hash_Icmp6InType128 = simple_hash("Icmp6InType128"); - hash_Icmp6InType129 = simple_hash("Icmp6InType129"); - hash_Icmp6InType136 = simple_hash("Icmp6InType136"); - hash_Icmp6OutType1 = simple_hash("Icmp6OutType1"); - hash_Icmp6OutType128 = simple_hash("Icmp6OutType128"); - hash_Icmp6OutType129 = simple_hash("Icmp6OutType129"); - hash_Icmp6OutType133 = simple_hash("Icmp6OutType133"); - hash_Icmp6OutType135 = simple_hash("Icmp6OutType135"); - hash_Icmp6OutType143 = simple_hash("Icmp6OutType143"); - hash_Udp6InDatagrams = simple_hash("Udp6InDatagrams"); - hash_Udp6NoPorts = simple_hash("Udp6NoPorts"); - hash_Udp6InErrors = simple_hash("Udp6InErrors"); - hash_Udp6OutDatagrams = simple_hash("Udp6OutDatagrams"); - hash_Udp6RcvbufErrors = simple_hash("Udp6RcvbufErrors"); - hash_Udp6SndbufErrors = simple_hash("Udp6SndbufErrors"); - hash_Udp6InCsumErrors = simple_hash("Udp6InCsumErrors"); - hash_Udp6IgnoredMulti = simple_hash("Udp6IgnoredMulti"); - hash_UdpLite6InDatagrams = simple_hash("UdpLite6InDatagrams"); - hash_UdpLite6NoPorts = simple_hash("UdpLite6NoPorts"); - hash_UdpLite6InErrors = simple_hash("UdpLite6InErrors"); - hash_UdpLite6OutDatagrams = simple_hash("UdpLite6OutDatagrams"); - hash_UdpLite6RcvbufErrors = simple_hash("UdpLite6RcvbufErrors"); - hash_UdpLite6SndbufErrors = simple_hash("UdpLite6SndbufErrors"); - hash_UdpLite6InCsumErrors = simple_hash("UdpLite6InCsumErrors"); - } - - if(do_ip_packets == -1) do_ip_packets = config_get_boolean_ondemand("plugin:proc:/proc/net/snmp6", "ipv6 packets", CONFIG_ONDEMAND_ONDEMAND); - if(do_ip_fragsout == -1) do_ip_fragsout = config_get_boolean_ondemand("plugin:proc:/proc/net/snmp6", "ipv6 fragments sent", CONFIG_ONDEMAND_ONDEMAND); - if(do_ip_fragsin == -1) do_ip_fragsin = config_get_boolean_ondemand("plugin:proc:/proc/net/snmp6", "ipv6 fragments assembly", CONFIG_ONDEMAND_ONDEMAND); - if(do_ip_errors == -1) do_ip_errors = config_get_boolean_ondemand("plugin:proc:/proc/net/snmp6", "ipv6 errors", CONFIG_ONDEMAND_ONDEMAND); - if(do_udp_packets == -1) do_udp_packets = config_get_boolean_ondemand("plugin:proc:/proc/net/snmp6", "ipv6 UDP packets", CONFIG_ONDEMAND_ONDEMAND); - if(do_udp_errors == -1) do_udp_errors = config_get_boolean_ondemand("plugin:proc:/proc/net/snmp6", "ipv6 UDP errors", CONFIG_ONDEMAND_ONDEMAND); - if(do_udplite_packets == -1) do_udplite_packets = config_get_boolean_ondemand("plugin:proc:/proc/net/snmp6", "ipv6 UDPlite packets", CONFIG_ONDEMAND_ONDEMAND); - if(do_udplite_errors == -1) do_udplite_errors = config_get_boolean_ondemand("plugin:proc:/proc/net/snmp6", "ipv6 UDPlite errors", CONFIG_ONDEMAND_ONDEMAND); - if(do_bandwidth == -1) do_bandwidth = config_get_boolean_ondemand("plugin:proc:/proc/net/snmp6", "bandwidth", CONFIG_ONDEMAND_ONDEMAND); - if(do_mcast == -1) do_mcast = config_get_boolean_ondemand("plugin:proc:/proc/net/snmp6", "multicast bandwidth", CONFIG_ONDEMAND_ONDEMAND); - if(do_bcast == -1) do_bcast = config_get_boolean_ondemand("plugin:proc:/proc/net/snmp6", "broadcast bandwidth", CONFIG_ONDEMAND_ONDEMAND); - if(do_mcast_p == -1) do_mcast_p = config_get_boolean_ondemand("plugin:proc:/proc/net/snmp6", "multicast packets", CONFIG_ONDEMAND_ONDEMAND); - if(do_icmp == -1) do_icmp = config_get_boolean_ondemand("plugin:proc:/proc/net/snmp6", "icmp", CONFIG_ONDEMAND_ONDEMAND); - if(do_icmp_redir == -1) do_icmp_redir = config_get_boolean_ondemand("plugin:proc:/proc/net/snmp6", "icmp redirects", CONFIG_ONDEMAND_ONDEMAND); - if(do_icmp_errors == -1) do_icmp_errors = config_get_boolean_ondemand("plugin:proc:/proc/net/snmp6", "icmp errors", CONFIG_ONDEMAND_ONDEMAND); - if(do_icmp_echos == -1) do_icmp_echos = config_get_boolean_ondemand("plugin:proc:/proc/net/snmp6", "icmp echos", CONFIG_ONDEMAND_ONDEMAND); - if(do_icmp_groupmemb == -1) do_icmp_groupmemb = config_get_boolean_ondemand("plugin:proc:/proc/net/snmp6", "icmp group membership", CONFIG_ONDEMAND_ONDEMAND); - if(do_icmp_router == -1) do_icmp_router = config_get_boolean_ondemand("plugin:proc:/proc/net/snmp6", "icmp router", CONFIG_ONDEMAND_ONDEMAND); - if(do_icmp_neighbor == -1) do_icmp_neighbor = config_get_boolean_ondemand("plugin:proc:/proc/net/snmp6", "icmp neighbor", CONFIG_ONDEMAND_ONDEMAND); - if(do_icmp_mldv2 == -1) do_icmp_mldv2 = config_get_boolean_ondemand("plugin:proc:/proc/net/snmp6", "icmp mldv2", CONFIG_ONDEMAND_ONDEMAND); - if(do_icmp_types == -1) do_icmp_types = config_get_boolean_ondemand("plugin:proc:/proc/net/snmp6", "icmp types", CONFIG_ONDEMAND_ONDEMAND); - if(do_ect == -1) do_ect = config_get_boolean_ondemand("plugin:proc:/proc/net/snmp6", "ect", CONFIG_ONDEMAND_ONDEMAND); - - if(dt) {}; - - if(!ff) { - char filename[FILENAME_MAX + 1]; - snprintfz(filename, FILENAME_MAX, "%s%s", global_host_prefix, "/proc/net/snmp6"); - ff = procfile_open(config_get("plugin:proc:/proc/net/snmp6", "filename to monitor", filename), " \t:", PROCFILE_FLAG_DEFAULT); - } - if(!ff) return 1; - - ff = procfile_readall(ff); - if(!ff) return 0; // we return 0, so that we will retry to open it next time - - uint32_t lines = procfile_lines(ff), l; - uint32_t words; - - unsigned long long Ip6InReceives = 0ULL; - unsigned long long Ip6InHdrErrors = 0ULL; - unsigned long long Ip6InTooBigErrors = 0ULL; - unsigned long long Ip6InNoRoutes = 0ULL; - unsigned long long Ip6InAddrErrors = 0ULL; - unsigned long long Ip6InUnknownProtos = 0ULL; - unsigned long long Ip6InTruncatedPkts = 0ULL; - unsigned long long Ip6InDiscards = 0ULL; - unsigned long long Ip6InDelivers = 0ULL; - unsigned long long Ip6OutForwDatagrams = 0ULL; - unsigned long long Ip6OutRequests = 0ULL; - unsigned long long Ip6OutDiscards = 0ULL; - unsigned long long Ip6OutNoRoutes = 0ULL; - unsigned long long Ip6ReasmTimeout = 0ULL; - unsigned long long Ip6ReasmReqds = 0ULL; - unsigned long long Ip6ReasmOKs = 0ULL; - unsigned long long Ip6ReasmFails = 0ULL; - unsigned long long Ip6FragOKs = 0ULL; - unsigned long long Ip6FragFails = 0ULL; - unsigned long long Ip6FragCreates = 0ULL; - unsigned long long Ip6InMcastPkts = 0ULL; - unsigned long long Ip6OutMcastPkts = 0ULL; - unsigned long long Ip6InOctets = 0ULL; - unsigned long long Ip6OutOctets = 0ULL; - unsigned long long Ip6InMcastOctets = 0ULL; - unsigned long long Ip6OutMcastOctets = 0ULL; - unsigned long long Ip6InBcastOctets = 0ULL; - unsigned long long Ip6OutBcastOctets = 0ULL; - unsigned long long Ip6InNoECTPkts = 0ULL; - unsigned long long Ip6InECT1Pkts = 0ULL; - unsigned long long Ip6InECT0Pkts = 0ULL; - unsigned long long Ip6InCEPkts = 0ULL; - unsigned long long Icmp6InMsgs = 0ULL; - unsigned long long Icmp6InErrors = 0ULL; - unsigned long long Icmp6OutMsgs = 0ULL; - unsigned long long Icmp6OutErrors = 0ULL; - unsigned long long Icmp6InCsumErrors = 0ULL; - unsigned long long Icmp6InDestUnreachs = 0ULL; - unsigned long long Icmp6InPktTooBigs = 0ULL; - unsigned long long Icmp6InTimeExcds = 0ULL; - unsigned long long Icmp6InParmProblems = 0ULL; - unsigned long long Icmp6InEchos = 0ULL; - unsigned long long Icmp6InEchoReplies = 0ULL; - unsigned long long Icmp6InGroupMembQueries = 0ULL; - unsigned long long Icmp6InGroupMembResponses = 0ULL; - unsigned long long Icmp6InGroupMembReductions = 0ULL; - unsigned long long Icmp6InRouterSolicits = 0ULL; - unsigned long long Icmp6InRouterAdvertisements = 0ULL; - unsigned long long Icmp6InNeighborSolicits = 0ULL; - unsigned long long Icmp6InNeighborAdvertisements = 0ULL; - unsigned long long Icmp6InRedirects = 0ULL; - unsigned long long Icmp6InMLDv2Reports = 0ULL; - unsigned long long Icmp6OutDestUnreachs = 0ULL; - unsigned long long Icmp6OutPktTooBigs = 0ULL; - unsigned long long Icmp6OutTimeExcds = 0ULL; - unsigned long long Icmp6OutParmProblems = 0ULL; - unsigned long long Icmp6OutEchos = 0ULL; - unsigned long long Icmp6OutEchoReplies = 0ULL; - unsigned long long Icmp6OutGroupMembQueries = 0ULL; - unsigned long long Icmp6OutGroupMembResponses = 0ULL; - unsigned long long Icmp6OutGroupMembReductions = 0ULL; - unsigned long long Icmp6OutRouterSolicits = 0ULL; - unsigned long long Icmp6OutRouterAdvertisements = 0ULL; - unsigned long long Icmp6OutNeighborSolicits = 0ULL; - unsigned long long Icmp6OutNeighborAdvertisements = 0ULL; - unsigned long long Icmp6OutRedirects = 0ULL; - unsigned long long Icmp6OutMLDv2Reports = 0ULL; - unsigned long long Icmp6InType1 = 0ULL; - unsigned long long Icmp6InType128 = 0ULL; - unsigned long long Icmp6InType129 = 0ULL; - unsigned long long Icmp6InType136 = 0ULL; - unsigned long long Icmp6OutType1 = 0ULL; - unsigned long long Icmp6OutType128 = 0ULL; - unsigned long long Icmp6OutType129 = 0ULL; - unsigned long long Icmp6OutType133 = 0ULL; - unsigned long long Icmp6OutType135 = 0ULL; - unsigned long long Icmp6OutType143 = 0ULL; - unsigned long long Udp6InDatagrams = 0ULL; - unsigned long long Udp6NoPorts = 0ULL; - unsigned long long Udp6InErrors = 0ULL; - unsigned long long Udp6OutDatagrams = 0ULL; - unsigned long long Udp6RcvbufErrors = 0ULL; - unsigned long long Udp6SndbufErrors = 0ULL; - unsigned long long Udp6InCsumErrors = 0ULL; - unsigned long long Udp6IgnoredMulti = 0ULL; - unsigned long long UdpLite6InDatagrams = 0ULL; - unsigned long long UdpLite6NoPorts = 0ULL; - unsigned long long UdpLite6InErrors = 0ULL; - unsigned long long UdpLite6OutDatagrams = 0ULL; - unsigned long long UdpLite6RcvbufErrors = 0ULL; - unsigned long long UdpLite6SndbufErrors = 0ULL; - unsigned long long UdpLite6InCsumErrors = 0ULL; - - for(l = 0; l < lines ;l++) { - words = procfile_linewords(ff, l); - if(words < 2) { - if(words) error("Cannot read /proc/net/snmp6 line %d. Expected 2 params, read %d.", l, words); - continue; - } - - char *name = procfile_lineword(ff, l, 0); - char * value = procfile_lineword(ff, l, 1); - if(!name || !*name || !value || !*value) continue; - - uint32_t hash = simple_hash(name); - - if(0) ; - else if(hash == hash_Ip6InReceives && strcmp(name, "Ip6InReceives") == 0) Ip6InReceives = strtoull(value, NULL, 10); - else if(hash == hash_Ip6InHdrErrors && strcmp(name, "Ip6InHdrErrors") == 0) Ip6InHdrErrors = strtoull(value, NULL, 10); - else if(hash == hash_Ip6InTooBigErrors && strcmp(name, "Ip6InTooBigErrors") == 0) Ip6InTooBigErrors = strtoull(value, NULL, 10); - else if(hash == hash_Ip6InNoRoutes && strcmp(name, "Ip6InNoRoutes") == 0) Ip6InNoRoutes = strtoull(value, NULL, 10); - else if(hash == hash_Ip6InAddrErrors && strcmp(name, "Ip6InAddrErrors") == 0) Ip6InAddrErrors = strtoull(value, NULL, 10); - else if(hash == hash_Ip6InUnknownProtos && strcmp(name, "Ip6InUnknownProtos") == 0) Ip6InUnknownProtos = strtoull(value, NULL, 10); - else if(hash == hash_Ip6InTruncatedPkts && strcmp(name, "Ip6InTruncatedPkts") == 0) Ip6InTruncatedPkts = strtoull(value, NULL, 10); - else if(hash == hash_Ip6InDiscards && strcmp(name, "Ip6InDiscards") == 0) Ip6InDiscards = strtoull(value, NULL, 10); - else if(hash == hash_Ip6InDelivers && strcmp(name, "Ip6InDelivers") == 0) Ip6InDelivers = strtoull(value, NULL, 10); - else if(hash == hash_Ip6OutForwDatagrams && strcmp(name, "Ip6OutForwDatagrams") == 0) Ip6OutForwDatagrams = strtoull(value, NULL, 10); - else if(hash == hash_Ip6OutRequests && strcmp(name, "Ip6OutRequests") == 0) Ip6OutRequests = strtoull(value, NULL, 10); - else if(hash == hash_Ip6OutDiscards && strcmp(name, "Ip6OutDiscards") == 0) Ip6OutDiscards = strtoull(value, NULL, 10); - else if(hash == hash_Ip6OutNoRoutes && strcmp(name, "Ip6OutNoRoutes") == 0) Ip6OutNoRoutes = strtoull(value, NULL, 10); - else if(hash == hash_Ip6ReasmTimeout && strcmp(name, "Ip6ReasmTimeout") == 0) Ip6ReasmTimeout = strtoull(value, NULL, 10); - else if(hash == hash_Ip6ReasmReqds && strcmp(name, "Ip6ReasmReqds") == 0) Ip6ReasmReqds = strtoull(value, NULL, 10); - else if(hash == hash_Ip6ReasmOKs && strcmp(name, "Ip6ReasmOKs") == 0) Ip6ReasmOKs = strtoull(value, NULL, 10); - else if(hash == hash_Ip6ReasmFails && strcmp(name, "Ip6ReasmFails") == 0) Ip6ReasmFails = strtoull(value, NULL, 10); - else if(hash == hash_Ip6FragOKs && strcmp(name, "Ip6FragOKs") == 0) Ip6FragOKs = strtoull(value, NULL, 10); - else if(hash == hash_Ip6FragFails && strcmp(name, "Ip6FragFails") == 0) Ip6FragFails = strtoull(value, NULL, 10); - else if(hash == hash_Ip6FragCreates && strcmp(name, "Ip6FragCreates") == 0) Ip6FragCreates = strtoull(value, NULL, 10); - else if(hash == hash_Ip6InMcastPkts && strcmp(name, "Ip6InMcastPkts") == 0) Ip6InMcastPkts = strtoull(value, NULL, 10); - else if(hash == hash_Ip6OutMcastPkts && strcmp(name, "Ip6OutMcastPkts") == 0) Ip6OutMcastPkts = strtoull(value, NULL, 10); - else if(hash == hash_Ip6InOctets && strcmp(name, "Ip6InOctets") == 0) Ip6InOctets = strtoull(value, NULL, 10); - else if(hash == hash_Ip6OutOctets && strcmp(name, "Ip6OutOctets") == 0) Ip6OutOctets = strtoull(value, NULL, 10); - else if(hash == hash_Ip6InMcastOctets && strcmp(name, "Ip6InMcastOctets") == 0) Ip6InMcastOctets = strtoull(value, NULL, 10); - else if(hash == hash_Ip6OutMcastOctets && strcmp(name, "Ip6OutMcastOctets") == 0) Ip6OutMcastOctets = strtoull(value, NULL, 10); - else if(hash == hash_Ip6InBcastOctets && strcmp(name, "Ip6InBcastOctets") == 0) Ip6InBcastOctets = strtoull(value, NULL, 10); - else if(hash == hash_Ip6OutBcastOctets && strcmp(name, "Ip6OutBcastOctets") == 0) Ip6OutBcastOctets = strtoull(value, NULL, 10); - else if(hash == hash_Ip6InNoECTPkts && strcmp(name, "Ip6InNoECTPkts") == 0) Ip6InNoECTPkts = strtoull(value, NULL, 10); - else if(hash == hash_Ip6InECT1Pkts && strcmp(name, "Ip6InECT1Pkts") == 0) Ip6InECT1Pkts = strtoull(value, NULL, 10); - else if(hash == hash_Ip6InECT0Pkts && strcmp(name, "Ip6InECT0Pkts") == 0) Ip6InECT0Pkts = strtoull(value, NULL, 10); - else if(hash == hash_Ip6InCEPkts && strcmp(name, "Ip6InCEPkts") == 0) Ip6InCEPkts = strtoull(value, NULL, 10); - else if(hash == hash_Icmp6InMsgs && strcmp(name, "Icmp6InMsgs") == 0) Icmp6InMsgs = strtoull(value, NULL, 10); - else if(hash == hash_Icmp6InErrors && strcmp(name, "Icmp6InErrors") == 0) Icmp6InErrors = strtoull(value, NULL, 10); - else if(hash == hash_Icmp6OutMsgs && strcmp(name, "Icmp6OutMsgs") == 0) Icmp6OutMsgs = strtoull(value, NULL, 10); - else if(hash == hash_Icmp6OutErrors && strcmp(name, "Icmp6OutErrors") == 0) Icmp6OutErrors = strtoull(value, NULL, 10); - else if(hash == hash_Icmp6InCsumErrors && strcmp(name, "Icmp6InCsumErrors") == 0) Icmp6InCsumErrors = strtoull(value, NULL, 10); - else if(hash == hash_Icmp6InDestUnreachs && strcmp(name, "Icmp6InDestUnreachs") == 0) Icmp6InDestUnreachs = strtoull(value, NULL, 10); - else if(hash == hash_Icmp6InPktTooBigs && strcmp(name, "Icmp6InPktTooBigs") == 0) Icmp6InPktTooBigs = strtoull(value, NULL, 10); - else if(hash == hash_Icmp6InTimeExcds && strcmp(name, "Icmp6InTimeExcds") == 0) Icmp6InTimeExcds = strtoull(value, NULL, 10); - else if(hash == hash_Icmp6InParmProblems && strcmp(name, "Icmp6InParmProblems") == 0) Icmp6InParmProblems = strtoull(value, NULL, 10); - else if(hash == hash_Icmp6InEchos && strcmp(name, "Icmp6InEchos") == 0) Icmp6InEchos = strtoull(value, NULL, 10); - else if(hash == hash_Icmp6InEchoReplies && strcmp(name, "Icmp6InEchoReplies") == 0) Icmp6InEchoReplies = strtoull(value, NULL, 10); - else if(hash == hash_Icmp6InGroupMembQueries && strcmp(name, "Icmp6InGroupMembQueries") == 0) Icmp6InGroupMembQueries = strtoull(value, NULL, 10); - else if(hash == hash_Icmp6InGroupMembResponses && strcmp(name, "Icmp6InGroupMembResponses") == 0) Icmp6InGroupMembResponses = strtoull(value, NULL, 10); - else if(hash == hash_Icmp6InGroupMembReductions && strcmp(name, "Icmp6InGroupMembReductions") == 0) Icmp6InGroupMembReductions = strtoull(value, NULL, 10); - else if(hash == hash_Icmp6InRouterSolicits && strcmp(name, "Icmp6InRouterSolicits") == 0) Icmp6InRouterSolicits = strtoull(value, NULL, 10); - else if(hash == hash_Icmp6InRouterAdvertisements && strcmp(name, "Icmp6InRouterAdvertisements") == 0) Icmp6InRouterAdvertisements = strtoull(value, NULL, 10); - else if(hash == hash_Icmp6InNeighborSolicits && strcmp(name, "Icmp6InNeighborSolicits") == 0) Icmp6InNeighborSolicits = strtoull(value, NULL, 10); - else if(hash == hash_Icmp6InNeighborAdvertisements && strcmp(name, "Icmp6InNeighborAdvertisements") == 0) Icmp6InNeighborAdvertisements = strtoull(value, NULL, 10); - else if(hash == hash_Icmp6InRedirects && strcmp(name, "Icmp6InRedirects") == 0) Icmp6InRedirects = strtoull(value, NULL, 10); - else if(hash == hash_Icmp6InMLDv2Reports && strcmp(name, "Icmp6InMLDv2Reports") == 0) Icmp6InMLDv2Reports = strtoull(value, NULL, 10); - else if(hash == hash_Icmp6OutDestUnreachs && strcmp(name, "Icmp6OutDestUnreachs") == 0) Icmp6OutDestUnreachs = strtoull(value, NULL, 10); - else if(hash == hash_Icmp6OutPktTooBigs && strcmp(name, "Icmp6OutPktTooBigs") == 0) Icmp6OutPktTooBigs = strtoull(value, NULL, 10); - else if(hash == hash_Icmp6OutTimeExcds && strcmp(name, "Icmp6OutTimeExcds") == 0) Icmp6OutTimeExcds = strtoull(value, NULL, 10); - else if(hash == hash_Icmp6OutParmProblems && strcmp(name, "Icmp6OutParmProblems") == 0) Icmp6OutParmProblems = strtoull(value, NULL, 10); - else if(hash == hash_Icmp6OutEchos && strcmp(name, "Icmp6OutEchos") == 0) Icmp6OutEchos = strtoull(value, NULL, 10); - else if(hash == hash_Icmp6OutEchoReplies && strcmp(name, "Icmp6OutEchoReplies") == 0) Icmp6OutEchoReplies = strtoull(value, NULL, 10); - else if(hash == hash_Icmp6OutGroupMembQueries && strcmp(name, "Icmp6OutGroupMembQueries") == 0) Icmp6OutGroupMembQueries = strtoull(value, NULL, 10); - else if(hash == hash_Icmp6OutGroupMembResponses && strcmp(name, "Icmp6OutGroupMembResponses") == 0) Icmp6OutGroupMembResponses = strtoull(value, NULL, 10); - else if(hash == hash_Icmp6OutGroupMembReductions && strcmp(name, "Icmp6OutGroupMembReductions") == 0) Icmp6OutGroupMembReductions = strtoull(value, NULL, 10); - else if(hash == hash_Icmp6OutRouterSolicits && strcmp(name, "Icmp6OutRouterSolicits") == 0) Icmp6OutRouterSolicits = strtoull(value, NULL, 10); - else if(hash == hash_Icmp6OutRouterAdvertisements && strcmp(name, "Icmp6OutRouterAdvertisements") == 0) Icmp6OutRouterAdvertisements = strtoull(value, NULL, 10); - else if(hash == hash_Icmp6OutNeighborSolicits && strcmp(name, "Icmp6OutNeighborSolicits") == 0) Icmp6OutNeighborSolicits = strtoull(value, NULL, 10); - else if(hash == hash_Icmp6OutNeighborAdvertisements && strcmp(name, "Icmp6OutNeighborAdvertisements") == 0) Icmp6OutNeighborAdvertisements = strtoull(value, NULL, 10); - else if(hash == hash_Icmp6OutRedirects && strcmp(name, "Icmp6OutRedirects") == 0) Icmp6OutRedirects = strtoull(value, NULL, 10); - else if(hash == hash_Icmp6OutMLDv2Reports && strcmp(name, "Icmp6OutMLDv2Reports") == 0) Icmp6OutMLDv2Reports = strtoull(value, NULL, 10); - else if(hash == hash_Icmp6InType1 && strcmp(name, "Icmp6InType1") == 0) Icmp6InType1 = strtoull(value, NULL, 10); - else if(hash == hash_Icmp6InType128 && strcmp(name, "Icmp6InType128") == 0) Icmp6InType128 = strtoull(value, NULL, 10); - else if(hash == hash_Icmp6InType129 && strcmp(name, "Icmp6InType129") == 0) Icmp6InType129 = strtoull(value, NULL, 10); - else if(hash == hash_Icmp6InType136 && strcmp(name, "Icmp6InType136") == 0) Icmp6InType136 = strtoull(value, NULL, 10); - else if(hash == hash_Icmp6OutType1 && strcmp(name, "Icmp6OutType1") == 0) Icmp6OutType1 = strtoull(value, NULL, 10); - else if(hash == hash_Icmp6OutType128 && strcmp(name, "Icmp6OutType128") == 0) Icmp6OutType128 = strtoull(value, NULL, 10); - else if(hash == hash_Icmp6OutType129 && strcmp(name, "Icmp6OutType129") == 0) Icmp6OutType129 = strtoull(value, NULL, 10); - else if(hash == hash_Icmp6OutType133 && strcmp(name, "Icmp6OutType133") == 0) Icmp6OutType133 = strtoull(value, NULL, 10); - else if(hash == hash_Icmp6OutType135 && strcmp(name, "Icmp6OutType135") == 0) Icmp6OutType135 = strtoull(value, NULL, 10); - else if(hash == hash_Icmp6OutType143 && strcmp(name, "Icmp6OutType143") == 0) Icmp6OutType143 = strtoull(value, NULL, 10); - else if(hash == hash_Udp6InDatagrams && strcmp(name, "Udp6InDatagrams") == 0) Udp6InDatagrams = strtoull(value, NULL, 10); - else if(hash == hash_Udp6NoPorts && strcmp(name, "Udp6NoPorts") == 0) Udp6NoPorts = strtoull(value, NULL, 10); - else if(hash == hash_Udp6InErrors && strcmp(name, "Udp6InErrors") == 0) Udp6InErrors = strtoull(value, NULL, 10); - else if(hash == hash_Udp6OutDatagrams && strcmp(name, "Udp6OutDatagrams") == 0) Udp6OutDatagrams = strtoull(value, NULL, 10); - else if(hash == hash_Udp6RcvbufErrors && strcmp(name, "Udp6RcvbufErrors") == 0) Udp6RcvbufErrors = strtoull(value, NULL, 10); - else if(hash == hash_Udp6SndbufErrors && strcmp(name, "Udp6SndbufErrors") == 0) Udp6SndbufErrors = strtoull(value, NULL, 10); - else if(hash == hash_Udp6InCsumErrors && strcmp(name, "Udp6InCsumErrors") == 0) Udp6InCsumErrors = strtoull(value, NULL, 10); - else if(hash == hash_Udp6IgnoredMulti && strcmp(name, "Udp6IgnoredMulti") == 0) Udp6IgnoredMulti = strtoull(value, NULL, 10); - else if(hash == hash_UdpLite6InDatagrams && strcmp(name, "UdpLite6InDatagrams") == 0) UdpLite6InDatagrams = strtoull(value, NULL, 10); - else if(hash == hash_UdpLite6NoPorts && strcmp(name, "UdpLite6NoPorts") == 0) UdpLite6NoPorts = strtoull(value, NULL, 10); - else if(hash == hash_UdpLite6InErrors && strcmp(name, "UdpLite6InErrors") == 0) UdpLite6InErrors = strtoull(value, NULL, 10); - else if(hash == hash_UdpLite6OutDatagrams && strcmp(name, "UdpLite6OutDatagrams") == 0) UdpLite6OutDatagrams = strtoull(value, NULL, 10); - else if(hash == hash_UdpLite6RcvbufErrors && strcmp(name, "UdpLite6RcvbufErrors") == 0) UdpLite6RcvbufErrors = strtoull(value, NULL, 10); - else if(hash == hash_UdpLite6SndbufErrors && strcmp(name, "UdpLite6SndbufErrors") == 0) UdpLite6SndbufErrors = strtoull(value, NULL, 10); - else if(hash == hash_UdpLite6InCsumErrors && strcmp(name, "UdpLite6InCsumErrors") == 0) UdpLite6InCsumErrors = strtoull(value, NULL, 10); - } - - RRDSET *st; - - // -------------------------------------------------------------------- - - if(do_bandwidth == CONFIG_ONDEMAND_YES || (do_bandwidth == CONFIG_ONDEMAND_ONDEMAND && (Ip6InOctets || Ip6OutOctets))) { - do_bandwidth = CONFIG_ONDEMAND_YES; - st = rrdset_find("system.ipv6"); - if(!st) { - st = rrdset_create("system", "ipv6", NULL, "network", NULL, "IPv6 Bandwidth", "kilobits/s", 500, update_every, RRDSET_TYPE_AREA); - - rrddim_add(st, "received", NULL, 8, 1024, RRDDIM_INCREMENTAL); - rrddim_add(st, "sent", NULL, -8, 1024, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); - - rrddim_set(st, "sent", Ip6OutOctets); - rrddim_set(st, "received", Ip6InOctets); - rrdset_done(st); - } - - // -------------------------------------------------------------------- - - if(do_ip_packets == CONFIG_ONDEMAND_YES || (do_ip_packets == CONFIG_ONDEMAND_ONDEMAND && (Ip6InReceives || Ip6OutRequests || Ip6InDelivers || Ip6OutForwDatagrams))) { - do_ip_packets = CONFIG_ONDEMAND_YES; - st = rrdset_find(RRD_TYPE_NET_SNMP6 ".packets"); - if(!st) { - st = rrdset_create(RRD_TYPE_NET_SNMP6, "packets", NULL, "packets", NULL, "IPv6 Packets", "packets/s", 3000, update_every, RRDSET_TYPE_LINE); - - rrddim_add(st, "received", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "sent", NULL, -1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "forwarded", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "delivers", NULL, -1, 1, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); - - rrddim_set(st, "sent", Ip6OutRequests); - rrddim_set(st, "received", Ip6InReceives); - rrddim_set(st, "forwarded", Ip6InDelivers); - rrddim_set(st, "delivers", Ip6OutForwDatagrams); - rrdset_done(st); - } - - // -------------------------------------------------------------------- - - if(do_ip_fragsout == CONFIG_ONDEMAND_YES || (do_ip_fragsout == CONFIG_ONDEMAND_ONDEMAND && (Ip6FragOKs || Ip6FragFails || Ip6FragCreates))) { - do_ip_fragsout = CONFIG_ONDEMAND_YES; - st = rrdset_find(RRD_TYPE_NET_SNMP6 ".fragsout"); - if(!st) { - st = rrdset_create(RRD_TYPE_NET_SNMP6, "fragsout", NULL, "fragments", NULL, "IPv6 Fragments Sent", "packets/s", 3010, update_every, RRDSET_TYPE_LINE); - st->isdetail = 1; - - rrddim_add(st, "ok", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "failed", NULL, -1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "all", NULL, 1, 1, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); - - rrddim_set(st, "ok", Ip6FragOKs); - rrddim_set(st, "failed", Ip6FragFails); - rrddim_set(st, "all", Ip6FragCreates); - rrdset_done(st); - } - - // -------------------------------------------------------------------- - - if(do_ip_fragsin == CONFIG_ONDEMAND_YES || (do_ip_fragsin == CONFIG_ONDEMAND_ONDEMAND - && ( - Ip6ReasmOKs - || Ip6ReasmFails - || Ip6ReasmTimeout - || Ip6ReasmReqds - ))) { - do_ip_fragsin = CONFIG_ONDEMAND_YES; - st = rrdset_find(RRD_TYPE_NET_SNMP6 ".fragsin"); - if(!st) { - st = rrdset_create(RRD_TYPE_NET_SNMP6, "fragsin", NULL, "fragments", NULL, "IPv6 Fragments Reassembly", "packets/s", 3011, update_every, RRDSET_TYPE_LINE); - st->isdetail = 1; - - rrddim_add(st, "ok", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "failed", NULL, -1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "timeout", NULL, -1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "all", NULL, 1, 1, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); - - rrddim_set(st, "ok", Ip6ReasmOKs); - rrddim_set(st, "failed", Ip6ReasmFails); - rrddim_set(st, "timeout", Ip6ReasmTimeout); - rrddim_set(st, "all", Ip6ReasmReqds); - rrdset_done(st); - } - - // -------------------------------------------------------------------- - - if(do_ip_errors == CONFIG_ONDEMAND_YES || (do_ip_errors == CONFIG_ONDEMAND_ONDEMAND - && ( - Ip6InDiscards - || Ip6OutDiscards - || Ip6InHdrErrors - || Ip6InAddrErrors - || Ip6InUnknownProtos - || Ip6InTooBigErrors - || Ip6InTruncatedPkts - || Ip6InNoRoutes - ))) { - do_ip_errors = CONFIG_ONDEMAND_YES; - st = rrdset_find(RRD_TYPE_NET_SNMP6 ".errors"); - if(!st) { - st = rrdset_create(RRD_TYPE_NET_SNMP6, "errors", NULL, "errors", NULL, "IPv6 Errors", "packets/s", 3002, update_every, RRDSET_TYPE_LINE); - st->isdetail = 1; - - rrddim_add(st, "InDiscards", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "OutDiscards", NULL, -1, 1, RRDDIM_INCREMENTAL); - - rrddim_add(st, "InHdrErrors", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "InAddrErrors", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "InUnknownProtos", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "InTooBigErrors", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "InTruncatedPkts", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "InNoRoutes", NULL, 1, 1, RRDDIM_INCREMENTAL); - - rrddim_add(st, "OutNoRoutes", NULL, -1, 1, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); - - rrddim_set(st, "InDiscards", Ip6InDiscards); - rrddim_set(st, "OutDiscards", Ip6OutDiscards); - - rrddim_set(st, "InHdrErrors", Ip6InHdrErrors); - rrddim_set(st, "InAddrErrors", Ip6InAddrErrors); - rrddim_set(st, "InUnknownProtos", Ip6InUnknownProtos); - rrddim_set(st, "InTooBigErrors", Ip6InTooBigErrors); - rrddim_set(st, "InTruncatedPkts", Ip6InTruncatedPkts); - rrddim_set(st, "InNoRoutes", Ip6InNoRoutes); - - rrddim_set(st, "OutNoRoutes", Ip6OutNoRoutes); - rrdset_done(st); - } - - // -------------------------------------------------------------------- - - if(do_udp_packets == CONFIG_ONDEMAND_YES || (do_udp_packets == CONFIG_ONDEMAND_ONDEMAND && (Udp6InDatagrams || Udp6OutDatagrams))) { - do_udp_packets = CONFIG_ONDEMAND_YES; - st = rrdset_find(RRD_TYPE_NET_SNMP6 ".udppackets"); - if(!st) { - st = rrdset_create(RRD_TYPE_NET_SNMP6, "udppackets", NULL, "udp", NULL, "IPv6 UDP Packets", "packets/s", 3601, update_every, RRDSET_TYPE_LINE); - - rrddim_add(st, "received", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "sent", NULL, -1, 1, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); - - rrddim_set(st, "received", Udp6InDatagrams); - rrddim_set(st, "sent", Udp6OutDatagrams); - rrdset_done(st); - } - - // -------------------------------------------------------------------- - - if(do_udp_errors == CONFIG_ONDEMAND_YES || (do_udp_errors == CONFIG_ONDEMAND_ONDEMAND - && ( - Udp6InErrors - || Udp6NoPorts - || Udp6RcvbufErrors - || Udp6SndbufErrors - || Udp6InCsumErrors - || Udp6IgnoredMulti - ))) { - do_udp_errors = CONFIG_ONDEMAND_YES; - st = rrdset_find(RRD_TYPE_NET_SNMP6 ".udperrors"); - if(!st) { - st = rrdset_create(RRD_TYPE_NET_SNMP6, "udperrors", NULL, "udp", NULL, "IPv6 UDP Errors", "events/s", 3701, update_every, RRDSET_TYPE_LINE); - st->isdetail = 1; - - rrddim_add(st, "RcvbufErrors", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "SndbufErrors", NULL, -1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "InErrors", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "NoPorts", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "InCsumErrors", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "IgnoredMulti", NULL, 1, 1, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); - - rrddim_set(st, "InErrors", Udp6InErrors); - rrddim_set(st, "NoPorts", Udp6NoPorts); - rrddim_set(st, "RcvbufErrors", Udp6RcvbufErrors); - rrddim_set(st, "SndbufErrors", Udp6SndbufErrors); - rrddim_set(st, "InCsumErrors", Udp6InCsumErrors); - rrddim_set(st, "IgnoredMulti", Udp6IgnoredMulti); - rrdset_done(st); - } - - // -------------------------------------------------------------------- - - if(do_udplite_packets == CONFIG_ONDEMAND_YES || (do_udplite_packets == CONFIG_ONDEMAND_ONDEMAND && (UdpLite6InDatagrams || UdpLite6OutDatagrams))) { - do_udplite_packets = CONFIG_ONDEMAND_YES; - st = rrdset_find(RRD_TYPE_NET_SNMP6 ".udplitepackets"); - if(!st) { - st = rrdset_create(RRD_TYPE_NET_SNMP6, "udplitepackets", NULL, "udplite", NULL, "IPv6 UDPlite Packets", "packets/s", 3601, update_every, RRDSET_TYPE_LINE); - - rrddim_add(st, "received", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "sent", NULL, -1, 1, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); - - rrddim_set(st, "received", UdpLite6InDatagrams); - rrddim_set(st, "sent", UdpLite6OutDatagrams); - rrdset_done(st); - } - - // -------------------------------------------------------------------- - - if(do_udplite_errors == CONFIG_ONDEMAND_YES || (do_udplite_errors == CONFIG_ONDEMAND_ONDEMAND - && ( - UdpLite6InErrors - || UdpLite6NoPorts - || UdpLite6RcvbufErrors - || UdpLite6SndbufErrors - || Udp6InCsumErrors - || UdpLite6InCsumErrors - ))) { - do_udplite_errors = CONFIG_ONDEMAND_YES; - st = rrdset_find(RRD_TYPE_NET_SNMP6 ".udpliteerrors"); - if(!st) { - st = rrdset_create(RRD_TYPE_NET_SNMP6, "udpliteerrors", NULL, "udplite", NULL, "IPv6 UDP Lite Errors", "events/s", 3701, update_every, RRDSET_TYPE_LINE); - st->isdetail = 1; - - rrddim_add(st, "RcvbufErrors", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "SndbufErrors", NULL, -1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "InErrors", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "NoPorts", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "InCsumErrors", NULL, 1, 1, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); - - rrddim_set(st, "InErrors", UdpLite6InErrors); - rrddim_set(st, "NoPorts", UdpLite6NoPorts); - rrddim_set(st, "RcvbufErrors", UdpLite6RcvbufErrors); - rrddim_set(st, "SndbufErrors", UdpLite6SndbufErrors); - rrddim_set(st, "InCsumErrors", UdpLite6InCsumErrors); - rrdset_done(st); - } - - // -------------------------------------------------------------------- - - if(do_mcast == CONFIG_ONDEMAND_YES || (do_mcast == CONFIG_ONDEMAND_ONDEMAND && (Ip6OutMcastOctets || Ip6InMcastOctets))) { - do_mcast = CONFIG_ONDEMAND_YES; - st = rrdset_find(RRD_TYPE_NET_SNMP6 ".mcast"); - if(!st) { - st = rrdset_create(RRD_TYPE_NET_SNMP6, "mcast", NULL, "multicast", NULL, "IPv6 Multicast Bandwidth", "kilobits/s", 9000, update_every, RRDSET_TYPE_AREA); - st->isdetail = 1; - - rrddim_add(st, "received", NULL, 8, 1024, RRDDIM_INCREMENTAL); - rrddim_add(st, "sent", NULL, -8, 1024, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); - - rrddim_set(st, "sent", Ip6OutMcastOctets); - rrddim_set(st, "received", Ip6InMcastOctets); - rrdset_done(st); - } - - // -------------------------------------------------------------------- - - if(do_bcast == CONFIG_ONDEMAND_YES || (do_bcast == CONFIG_ONDEMAND_ONDEMAND && (Ip6OutBcastOctets || Ip6InBcastOctets))) { - do_bcast = CONFIG_ONDEMAND_YES; - st = rrdset_find(RRD_TYPE_NET_SNMP6 ".bcast"); - if(!st) { - st = rrdset_create(RRD_TYPE_NET_SNMP6, "bcast", NULL, "broadcast", NULL, "IPv6 Broadcast Bandwidth", "kilobits/s", 8000, update_every, RRDSET_TYPE_AREA); - st->isdetail = 1; - - rrddim_add(st, "received", NULL, 8, 1024, RRDDIM_INCREMENTAL); - rrddim_add(st, "sent", NULL, -8, 1024, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); - - rrddim_set(st, "sent", Ip6OutBcastOctets); - rrddim_set(st, "received", Ip6InBcastOctets); - rrdset_done(st); - } - - // -------------------------------------------------------------------- - - if(do_mcast_p == CONFIG_ONDEMAND_YES || (do_mcast_p == CONFIG_ONDEMAND_ONDEMAND && (Ip6OutMcastPkts || Ip6InMcastPkts))) { - do_mcast_p = CONFIG_ONDEMAND_YES; - st = rrdset_find(RRD_TYPE_NET_SNMP6 ".mcastpkts"); - if(!st) { - st = rrdset_create(RRD_TYPE_NET_SNMP6, "mcastpkts", NULL, "multicast", NULL, "IPv6 Multicast Packets", "packets/s", 9500, update_every, RRDSET_TYPE_LINE); - st->isdetail = 1; - - rrddim_add(st, "received", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "sent", NULL, -1, 1, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); - - rrddim_set(st, "sent", Ip6OutMcastPkts); - rrddim_set(st, "received", Ip6InMcastPkts); - rrdset_done(st); - } - - // -------------------------------------------------------------------- - - if(do_icmp == CONFIG_ONDEMAND_YES || (do_icmp == CONFIG_ONDEMAND_ONDEMAND && (Icmp6InMsgs || Icmp6OutMsgs))) { - do_icmp = CONFIG_ONDEMAND_YES; - st = rrdset_find(RRD_TYPE_NET_SNMP6 ".icmp"); - if(!st) { - st = rrdset_create(RRD_TYPE_NET_SNMP6, "icmp", NULL, "icmp", NULL, "IPv6 ICMP Messages", "messages/s", 10000, update_every, RRDSET_TYPE_LINE); - - rrddim_add(st, "received", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "sent", NULL, -1, 1, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); - - rrddim_set(st, "sent", Icmp6InMsgs); - rrddim_set(st, "received", Icmp6OutMsgs); - rrdset_done(st); - } - - // -------------------------------------------------------------------- - - if(do_icmp_redir == CONFIG_ONDEMAND_YES || (do_icmp_redir == CONFIG_ONDEMAND_ONDEMAND && (Icmp6InRedirects || Icmp6OutRedirects))) { - do_icmp_redir = CONFIG_ONDEMAND_YES; - st = rrdset_find(RRD_TYPE_NET_SNMP6 ".icmpredir"); - if(!st) { - st = rrdset_create(RRD_TYPE_NET_SNMP6, "icmpredir", NULL, "icmp", NULL, "IPv6 ICMP Redirects", "redirects/s", 10050, update_every, RRDSET_TYPE_LINE); - - rrddim_add(st, "received", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "sent", NULL, -1, 1, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); - - rrddim_set(st, "sent", Icmp6InRedirects); - rrddim_set(st, "received", Icmp6OutRedirects); - rrdset_done(st); - } - - // -------------------------------------------------------------------- - - if(do_icmp_errors == CONFIG_ONDEMAND_YES || (do_icmp_errors == CONFIG_ONDEMAND_ONDEMAND - && ( - Icmp6InErrors - || Icmp6OutErrors - || Icmp6InCsumErrors - || Icmp6InDestUnreachs - || Icmp6InPktTooBigs - || Icmp6InTimeExcds - || Icmp6InParmProblems - || Icmp6OutDestUnreachs - || Icmp6OutPktTooBigs - || Icmp6OutTimeExcds - || Icmp6OutParmProblems - ))) { - do_icmp_errors = CONFIG_ONDEMAND_YES; - st = rrdset_find(RRD_TYPE_NET_SNMP6 ".icmperrors"); - if(!st) { - st = rrdset_create(RRD_TYPE_NET_SNMP6, "icmperrors", NULL, "icmp", NULL, "IPv6 ICMP Errors", "errors/s", 10100, update_every, RRDSET_TYPE_LINE); - - rrddim_add(st, "InErrors", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "OutErrors", NULL, -1, 1, RRDDIM_INCREMENTAL); - - rrddim_add(st, "InCsumErrors", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "InDestUnreachs", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "InPktTooBigs", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "InTimeExcds", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "InParmProblems", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "OutDestUnreachs", NULL, -1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "OutPktTooBigs", NULL, -1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "OutTimeExcds", NULL, -1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "OutParmProblems", NULL, -1, 1, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); - - rrddim_set(st, "InErrors", Icmp6InErrors); - rrddim_set(st, "OutErrors", Icmp6OutErrors); - rrddim_set(st, "InCsumErrors", Icmp6InCsumErrors); - rrddim_set(st, "InDestUnreachs", Icmp6InDestUnreachs); - rrddim_set(st, "InPktTooBigs", Icmp6InPktTooBigs); - rrddim_set(st, "InTimeExcds", Icmp6InTimeExcds); - rrddim_set(st, "InParmProblems", Icmp6InParmProblems); - rrddim_set(st, "OutDestUnreachs", Icmp6OutDestUnreachs); - rrddim_set(st, "OutPktTooBigs", Icmp6OutPktTooBigs); - rrddim_set(st, "OutTimeExcds", Icmp6OutTimeExcds); - rrddim_set(st, "OutParmProblems", Icmp6OutParmProblems); - rrdset_done(st); - } - - // -------------------------------------------------------------------- - - if(do_icmp_echos == CONFIG_ONDEMAND_YES || (do_icmp_echos == CONFIG_ONDEMAND_ONDEMAND - && ( - Icmp6InEchos - || Icmp6OutEchos - || Icmp6InEchoReplies - || Icmp6OutEchoReplies - ))) { - do_icmp_echos = CONFIG_ONDEMAND_YES; - st = rrdset_find(RRD_TYPE_NET_SNMP6 ".icmpechos"); - if(!st) { - st = rrdset_create(RRD_TYPE_NET_SNMP6, "icmpechos", NULL, "icmp", NULL, "IPv6 ICMP Echo", "messages/s", 10200, update_every, RRDSET_TYPE_LINE); - - rrddim_add(st, "InEchos", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "OutEchos", NULL, -1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "InEchoReplies", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "OutEchoReplies", NULL, -1, 1, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); - - rrddim_set(st, "InEchos", Icmp6InEchos); - rrddim_set(st, "OutEchos", Icmp6OutEchos); - rrddim_set(st, "InEchoReplies", Icmp6InEchoReplies); - rrddim_set(st, "OutEchoReplies", Icmp6OutEchoReplies); - rrdset_done(st); - } - - // -------------------------------------------------------------------- - - if(do_icmp_groupmemb == CONFIG_ONDEMAND_YES || (do_icmp_groupmemb == CONFIG_ONDEMAND_ONDEMAND - && ( - Icmp6InGroupMembQueries - || Icmp6OutGroupMembQueries - || Icmp6InGroupMembResponses - || Icmp6OutGroupMembResponses - || Icmp6InGroupMembReductions - || Icmp6OutGroupMembReductions - ))) { - do_icmp_groupmemb = CONFIG_ONDEMAND_YES; - st = rrdset_find(RRD_TYPE_NET_SNMP6 ".groupmemb"); - if(!st) { - st = rrdset_create(RRD_TYPE_NET_SNMP6, "groupmemb", NULL, "icmp", NULL, "IPv6 ICMP Group Membership", "messages/s", 10300, update_every, RRDSET_TYPE_LINE); - - rrddim_add(st, "InQueries", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "OutQueries", NULL, -1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "InResponses", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "OutResponses", NULL, -1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "InReductions", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "OutReductions", NULL, -1, 1, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); - - rrddim_set(st, "InQueries", Icmp6InGroupMembQueries); - rrddim_set(st, "OutQueries", Icmp6OutGroupMembQueries); - rrddim_set(st, "InResponses", Icmp6InGroupMembResponses); - rrddim_set(st, "OutResponses", Icmp6OutGroupMembResponses); - rrddim_set(st, "InReductions", Icmp6InGroupMembReductions); - rrddim_set(st, "OutReductions", Icmp6OutGroupMembReductions); - rrdset_done(st); - } - - // -------------------------------------------------------------------- - - if(do_icmp_router == CONFIG_ONDEMAND_YES || (do_icmp_router == CONFIG_ONDEMAND_ONDEMAND - && ( - Icmp6InRouterSolicits - || Icmp6OutRouterSolicits - || Icmp6InRouterAdvertisements - || Icmp6OutRouterAdvertisements - ))) { - do_icmp_router = CONFIG_ONDEMAND_YES; - st = rrdset_find(RRD_TYPE_NET_SNMP6 ".icmprouter"); - if(!st) { - st = rrdset_create(RRD_TYPE_NET_SNMP6, "icmprouter", NULL, "icmp", NULL, "IPv6 Router Messages", "messages/s", 10400, update_every, RRDSET_TYPE_LINE); - - rrddim_add(st, "InSolicits", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "OutSolicits", NULL, -1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "InAdvertisements", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "OutAdvertisements", NULL, -1, 1, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); - - rrddim_set(st, "InSolicits", Icmp6InRouterSolicits); - rrddim_set(st, "OutSolicits", Icmp6OutRouterSolicits); - rrddim_set(st, "InAdvertisements", Icmp6InRouterAdvertisements); - rrddim_set(st, "OutAdvertisements", Icmp6OutRouterAdvertisements); - rrdset_done(st); - } - - // -------------------------------------------------------------------- - - if(do_icmp_neighbor == CONFIG_ONDEMAND_YES || (do_icmp_neighbor == CONFIG_ONDEMAND_ONDEMAND - && ( - Icmp6InNeighborSolicits - || Icmp6OutNeighborSolicits - || Icmp6InNeighborAdvertisements - || Icmp6OutNeighborAdvertisements - ))) { - do_icmp_neighbor = CONFIG_ONDEMAND_YES; - st = rrdset_find(RRD_TYPE_NET_SNMP6 ".icmpneighbor"); - if(!st) { - st = rrdset_create(RRD_TYPE_NET_SNMP6, "icmpneighbor", NULL, "icmp", NULL, "IPv6 Neighbor Messages", "messages/s", 10500, update_every, RRDSET_TYPE_LINE); - - rrddim_add(st, "InSolicits", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "OutSolicits", NULL, -1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "InAdvertisements", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "OutAdvertisements", NULL, -1, 1, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); - - rrddim_set(st, "InSolicits", Icmp6InNeighborSolicits); - rrddim_set(st, "OutSolicits", Icmp6OutNeighborSolicits); - rrddim_set(st, "InAdvertisements", Icmp6InNeighborAdvertisements); - rrddim_set(st, "OutAdvertisements", Icmp6OutNeighborAdvertisements); - rrdset_done(st); - } - - // -------------------------------------------------------------------- - - if(do_icmp_mldv2 == CONFIG_ONDEMAND_YES || (do_icmp_mldv2 == CONFIG_ONDEMAND_ONDEMAND && (Icmp6InMLDv2Reports || Icmp6OutMLDv2Reports))) { - do_icmp_mldv2 = CONFIG_ONDEMAND_YES; - st = rrdset_find(RRD_TYPE_NET_SNMP6 ".icmpmldv2"); - if(!st) { - st = rrdset_create(RRD_TYPE_NET_SNMP6, "icmpmldv2", NULL, "icmp", NULL, "IPv6 ICMP MLDv2 Reports", "reports/s", 10600, update_every, RRDSET_TYPE_LINE); - - rrddim_add(st, "received", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "sent", NULL, -1, 1, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); - - rrddim_set(st, "sent", Icmp6InMLDv2Reports); - rrddim_set(st, "received", Icmp6OutMLDv2Reports); - rrdset_done(st); - } - - // -------------------------------------------------------------------- - - if(do_icmp_types == CONFIG_ONDEMAND_YES || (do_icmp_types == CONFIG_ONDEMAND_ONDEMAND - && ( - Icmp6InType1 - || Icmp6InType128 - || Icmp6InType129 - || Icmp6InType136 - || Icmp6OutType1 - || Icmp6OutType128 - || Icmp6OutType129 - || Icmp6OutType133 - || Icmp6OutType135 - || Icmp6OutType143 - ))) { - do_icmp_types = CONFIG_ONDEMAND_YES; - st = rrdset_find(RRD_TYPE_NET_SNMP6 ".icmptypes"); - if(!st) { - st = rrdset_create(RRD_TYPE_NET_SNMP6, "icmptypes", NULL, "icmp", NULL, "IPv6 ICMP Types", "messages/s", 10700, update_every, RRDSET_TYPE_LINE); - - rrddim_add(st, "InType1", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "InType128", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "InType129", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "InType136", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "OutType1", NULL, -1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "OutType128", NULL, -1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "OutType129", NULL, -1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "OutType133", NULL, -1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "OutType135", NULL, -1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "OutType143", NULL, -1, 1, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); - - rrddim_set(st, "InType1", Icmp6InType1); - rrddim_set(st, "InType128", Icmp6InType128); - rrddim_set(st, "InType129", Icmp6InType129); - rrddim_set(st, "InType136", Icmp6InType136); - rrddim_set(st, "OutType1", Icmp6OutType1); - rrddim_set(st, "OutType128", Icmp6OutType128); - rrddim_set(st, "OutType129", Icmp6OutType129); - rrddim_set(st, "OutType133", Icmp6OutType133); - rrddim_set(st, "OutType135", Icmp6OutType135); - rrddim_set(st, "OutType143", Icmp6OutType143); - rrdset_done(st); - } - - // -------------------------------------------------------------------- - - if(do_ect == CONFIG_ONDEMAND_YES || (do_ect == CONFIG_ONDEMAND_ONDEMAND - && ( - Ip6InNoECTPkts - || Ip6InECT1Pkts - || Ip6InECT0Pkts - || Ip6InCEPkts - ))) { - do_ect = CONFIG_ONDEMAND_YES; - st = rrdset_find(RRD_TYPE_NET_SNMP6 ".ect"); - if(!st) { - st = rrdset_create(RRD_TYPE_NET_SNMP6, "ect", NULL, "packets", NULL, "IPv6 ECT Packets", "packets/s", 10800, update_every, RRDSET_TYPE_LINE); - - rrddim_add(st, "InNoECTPkts", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "InECT1Pkts", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "InECT0Pkts", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "InCEPkts", NULL, 1, 1, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); - - rrddim_set(st, "InNoECTPkts", Ip6InNoECTPkts); - rrddim_set(st, "InECT1Pkts", Ip6InECT1Pkts); - rrddim_set(st, "InECT0Pkts", Ip6InECT0Pkts); - rrddim_set(st, "InCEPkts", Ip6InCEPkts); - rrdset_done(st); - } - - return 0; + static procfile *ff = NULL; + static int gen_hashes = -1; + + static int do_ip_packets = -1, do_ip_fragsout = -1, do_ip_fragsin = -1, do_ip_errors = -1, + do_udplite_packets = -1, do_udplite_errors = -1, + do_udp_packets = -1, do_udp_errors = -1, + do_bandwidth = -1, do_mcast = -1, do_bcast = -1, do_mcast_p = -1, + do_icmp = -1, do_icmp_redir = -1, do_icmp_errors = -1, do_icmp_echos = -1, do_icmp_groupmemb = -1, + do_icmp_router = -1, do_icmp_neighbor = -1, do_icmp_mldv2 = -1, do_icmp_types = -1, do_ect = -1; + + static uint32_t hash_Ip6InReceives = -1; + + static uint32_t hash_Ip6InHdrErrors = -1; + static uint32_t hash_Ip6InTooBigErrors = -1; + static uint32_t hash_Ip6InNoRoutes = -1; + static uint32_t hash_Ip6InAddrErrors = -1; + static uint32_t hash_Ip6InUnknownProtos = -1; + static uint32_t hash_Ip6InTruncatedPkts = -1; + static uint32_t hash_Ip6InDiscards = -1; + static uint32_t hash_Ip6InDelivers = -1; + + static uint32_t hash_Ip6OutForwDatagrams = -1; + static uint32_t hash_Ip6OutRequests = -1; + static uint32_t hash_Ip6OutDiscards = -1; + static uint32_t hash_Ip6OutNoRoutes = -1; + + static uint32_t hash_Ip6ReasmTimeout = -1; + static uint32_t hash_Ip6ReasmReqds = -1; + static uint32_t hash_Ip6ReasmOKs = -1; + static uint32_t hash_Ip6ReasmFails = -1; + + static uint32_t hash_Ip6FragOKs = -1; + static uint32_t hash_Ip6FragFails = -1; + static uint32_t hash_Ip6FragCreates = -1; + + static uint32_t hash_Ip6InMcastPkts = -1; + static uint32_t hash_Ip6OutMcastPkts = -1; + + static uint32_t hash_Ip6InOctets = -1; + static uint32_t hash_Ip6OutOctets = -1; + + static uint32_t hash_Ip6InMcastOctets = -1; + static uint32_t hash_Ip6OutMcastOctets = -1; + static uint32_t hash_Ip6InBcastOctets = -1; + static uint32_t hash_Ip6OutBcastOctets = -1; + + static uint32_t hash_Ip6InNoECTPkts = -1; + static uint32_t hash_Ip6InECT1Pkts = -1; + static uint32_t hash_Ip6InECT0Pkts = -1; + static uint32_t hash_Ip6InCEPkts = -1; + + static uint32_t hash_Icmp6InMsgs = -1; + static uint32_t hash_Icmp6InErrors = -1; + static uint32_t hash_Icmp6OutMsgs = -1; + static uint32_t hash_Icmp6OutErrors = -1; + static uint32_t hash_Icmp6InCsumErrors = -1; + static uint32_t hash_Icmp6InDestUnreachs = -1; + static uint32_t hash_Icmp6InPktTooBigs = -1; + static uint32_t hash_Icmp6InTimeExcds = -1; + static uint32_t hash_Icmp6InParmProblems = -1; + static uint32_t hash_Icmp6InEchos = -1; + static uint32_t hash_Icmp6InEchoReplies = -1; + static uint32_t hash_Icmp6InGroupMembQueries = -1; + static uint32_t hash_Icmp6InGroupMembResponses = -1; + static uint32_t hash_Icmp6InGroupMembReductions = -1; + static uint32_t hash_Icmp6InRouterSolicits = -1; + static uint32_t hash_Icmp6InRouterAdvertisements = -1; + static uint32_t hash_Icmp6InNeighborSolicits = -1; + static uint32_t hash_Icmp6InNeighborAdvertisements = -1; + static uint32_t hash_Icmp6InRedirects = -1; + static uint32_t hash_Icmp6InMLDv2Reports = -1; + static uint32_t hash_Icmp6OutDestUnreachs = -1; + static uint32_t hash_Icmp6OutPktTooBigs = -1; + static uint32_t hash_Icmp6OutTimeExcds = -1; + static uint32_t hash_Icmp6OutParmProblems = -1; + static uint32_t hash_Icmp6OutEchos = -1; + static uint32_t hash_Icmp6OutEchoReplies = -1; + static uint32_t hash_Icmp6OutGroupMembQueries = -1; + static uint32_t hash_Icmp6OutGroupMembResponses = -1; + static uint32_t hash_Icmp6OutGroupMembReductions = -1; + static uint32_t hash_Icmp6OutRouterSolicits = -1; + static uint32_t hash_Icmp6OutRouterAdvertisements = -1; + static uint32_t hash_Icmp6OutNeighborSolicits = -1; + static uint32_t hash_Icmp6OutNeighborAdvertisements = -1; + static uint32_t hash_Icmp6OutRedirects = -1; + static uint32_t hash_Icmp6OutMLDv2Reports = -1; + static uint32_t hash_Icmp6InType1 = -1; + static uint32_t hash_Icmp6InType128 = -1; + static uint32_t hash_Icmp6InType129 = -1; + static uint32_t hash_Icmp6InType136 = -1; + static uint32_t hash_Icmp6OutType1 = -1; + static uint32_t hash_Icmp6OutType128 = -1; + static uint32_t hash_Icmp6OutType129 = -1; + static uint32_t hash_Icmp6OutType133 = -1; + static uint32_t hash_Icmp6OutType135 = -1; + static uint32_t hash_Icmp6OutType143 = -1; + + static uint32_t hash_Udp6InDatagrams = -1; + static uint32_t hash_Udp6NoPorts = -1; + static uint32_t hash_Udp6InErrors = -1; + static uint32_t hash_Udp6OutDatagrams = -1; + static uint32_t hash_Udp6RcvbufErrors = -1; + static uint32_t hash_Udp6SndbufErrors = -1; + static uint32_t hash_Udp6InCsumErrors = -1; + static uint32_t hash_Udp6IgnoredMulti = -1; + + static uint32_t hash_UdpLite6InDatagrams = -1; + static uint32_t hash_UdpLite6NoPorts = -1; + static uint32_t hash_UdpLite6InErrors = -1; + static uint32_t hash_UdpLite6OutDatagrams = -1; + static uint32_t hash_UdpLite6RcvbufErrors = -1; + static uint32_t hash_UdpLite6SndbufErrors = -1; + static uint32_t hash_UdpLite6InCsumErrors = -1; + + if(gen_hashes != 1) { + gen_hashes = 1; + hash_Ip6InReceives = simple_hash("Ip6InReceives"); + hash_Ip6InHdrErrors = simple_hash("Ip6InHdrErrors"); + hash_Ip6InTooBigErrors = simple_hash("Ip6InTooBigErrors"); + hash_Ip6InNoRoutes = simple_hash("Ip6InNoRoutes"); + hash_Ip6InAddrErrors = simple_hash("Ip6InAddrErrors"); + hash_Ip6InUnknownProtos = simple_hash("Ip6InUnknownProtos"); + hash_Ip6InTruncatedPkts = simple_hash("Ip6InTruncatedPkts"); + hash_Ip6InDiscards = simple_hash("Ip6InDiscards"); + hash_Ip6InDelivers = simple_hash("Ip6InDelivers"); + hash_Ip6OutForwDatagrams = simple_hash("Ip6OutForwDatagrams"); + hash_Ip6OutRequests = simple_hash("Ip6OutRequests"); + hash_Ip6OutDiscards = simple_hash("Ip6OutDiscards"); + hash_Ip6OutNoRoutes = simple_hash("Ip6OutNoRoutes"); + hash_Ip6ReasmTimeout = simple_hash("Ip6ReasmTimeout"); + hash_Ip6ReasmReqds = simple_hash("Ip6ReasmReqds"); + hash_Ip6ReasmOKs = simple_hash("Ip6ReasmOKs"); + hash_Ip6ReasmFails = simple_hash("Ip6ReasmFails"); + hash_Ip6FragOKs = simple_hash("Ip6FragOKs"); + hash_Ip6FragFails = simple_hash("Ip6FragFails"); + hash_Ip6FragCreates = simple_hash("Ip6FragCreates"); + hash_Ip6InMcastPkts = simple_hash("Ip6InMcastPkts"); + hash_Ip6OutMcastPkts = simple_hash("Ip6OutMcastPkts"); + hash_Ip6InOctets = simple_hash("Ip6InOctets"); + hash_Ip6OutOctets = simple_hash("Ip6OutOctets"); + hash_Ip6InMcastOctets = simple_hash("Ip6InMcastOctets"); + hash_Ip6OutMcastOctets = simple_hash("Ip6OutMcastOctets"); + hash_Ip6InBcastOctets = simple_hash("Ip6InBcastOctets"); + hash_Ip6OutBcastOctets = simple_hash("Ip6OutBcastOctets"); + hash_Ip6InNoECTPkts = simple_hash("Ip6InNoECTPkts"); + hash_Ip6InECT1Pkts = simple_hash("Ip6InECT1Pkts"); + hash_Ip6InECT0Pkts = simple_hash("Ip6InECT0Pkts"); + hash_Ip6InCEPkts = simple_hash("Ip6InCEPkts"); + hash_Icmp6InMsgs = simple_hash("Icmp6InMsgs"); + hash_Icmp6InErrors = simple_hash("Icmp6InErrors"); + hash_Icmp6OutMsgs = simple_hash("Icmp6OutMsgs"); + hash_Icmp6OutErrors = simple_hash("Icmp6OutErrors"); + hash_Icmp6InCsumErrors = simple_hash("Icmp6InCsumErrors"); + hash_Icmp6InDestUnreachs = simple_hash("Icmp6InDestUnreachs"); + hash_Icmp6InPktTooBigs = simple_hash("Icmp6InPktTooBigs"); + hash_Icmp6InTimeExcds = simple_hash("Icmp6InTimeExcds"); + hash_Icmp6InParmProblems = simple_hash("Icmp6InParmProblems"); + hash_Icmp6InEchos = simple_hash("Icmp6InEchos"); + hash_Icmp6InEchoReplies = simple_hash("Icmp6InEchoReplies"); + hash_Icmp6InGroupMembQueries = simple_hash("Icmp6InGroupMembQueries"); + hash_Icmp6InGroupMembResponses = simple_hash("Icmp6InGroupMembResponses"); + hash_Icmp6InGroupMembReductions = simple_hash("Icmp6InGroupMembReductions"); + hash_Icmp6InRouterSolicits = simple_hash("Icmp6InRouterSolicits"); + hash_Icmp6InRouterAdvertisements = simple_hash("Icmp6InRouterAdvertisements"); + hash_Icmp6InNeighborSolicits = simple_hash("Icmp6InNeighborSolicits"); + hash_Icmp6InNeighborAdvertisements = simple_hash("Icmp6InNeighborAdvertisements"); + hash_Icmp6InRedirects = simple_hash("Icmp6InRedirects"); + hash_Icmp6InMLDv2Reports = simple_hash("Icmp6InMLDv2Reports"); + hash_Icmp6OutDestUnreachs = simple_hash("Icmp6OutDestUnreachs"); + hash_Icmp6OutPktTooBigs = simple_hash("Icmp6OutPktTooBigs"); + hash_Icmp6OutTimeExcds = simple_hash("Icmp6OutTimeExcds"); + hash_Icmp6OutParmProblems = simple_hash("Icmp6OutParmProblems"); + hash_Icmp6OutEchos = simple_hash("Icmp6OutEchos"); + hash_Icmp6OutEchoReplies = simple_hash("Icmp6OutEchoReplies"); + hash_Icmp6OutGroupMembQueries = simple_hash("Icmp6OutGroupMembQueries"); + hash_Icmp6OutGroupMembResponses = simple_hash("Icmp6OutGroupMembResponses"); + hash_Icmp6OutGroupMembReductions = simple_hash("Icmp6OutGroupMembReductions"); + hash_Icmp6OutRouterSolicits = simple_hash("Icmp6OutRouterSolicits"); + hash_Icmp6OutRouterAdvertisements = simple_hash("Icmp6OutRouterAdvertisements"); + hash_Icmp6OutNeighborSolicits = simple_hash("Icmp6OutNeighborSolicits"); + hash_Icmp6OutNeighborAdvertisements = simple_hash("Icmp6OutNeighborAdvertisements"); + hash_Icmp6OutRedirects = simple_hash("Icmp6OutRedirects"); + hash_Icmp6OutMLDv2Reports = simple_hash("Icmp6OutMLDv2Reports"); + hash_Icmp6InType1 = simple_hash("Icmp6InType1"); + hash_Icmp6InType128 = simple_hash("Icmp6InType128"); + hash_Icmp6InType129 = simple_hash("Icmp6InType129"); + hash_Icmp6InType136 = simple_hash("Icmp6InType136"); + hash_Icmp6OutType1 = simple_hash("Icmp6OutType1"); + hash_Icmp6OutType128 = simple_hash("Icmp6OutType128"); + hash_Icmp6OutType129 = simple_hash("Icmp6OutType129"); + hash_Icmp6OutType133 = simple_hash("Icmp6OutType133"); + hash_Icmp6OutType135 = simple_hash("Icmp6OutType135"); + hash_Icmp6OutType143 = simple_hash("Icmp6OutType143"); + hash_Udp6InDatagrams = simple_hash("Udp6InDatagrams"); + hash_Udp6NoPorts = simple_hash("Udp6NoPorts"); + hash_Udp6InErrors = simple_hash("Udp6InErrors"); + hash_Udp6OutDatagrams = simple_hash("Udp6OutDatagrams"); + hash_Udp6RcvbufErrors = simple_hash("Udp6RcvbufErrors"); + hash_Udp6SndbufErrors = simple_hash("Udp6SndbufErrors"); + hash_Udp6InCsumErrors = simple_hash("Udp6InCsumErrors"); + hash_Udp6IgnoredMulti = simple_hash("Udp6IgnoredMulti"); + hash_UdpLite6InDatagrams = simple_hash("UdpLite6InDatagrams"); + hash_UdpLite6NoPorts = simple_hash("UdpLite6NoPorts"); + hash_UdpLite6InErrors = simple_hash("UdpLite6InErrors"); + hash_UdpLite6OutDatagrams = simple_hash("UdpLite6OutDatagrams"); + hash_UdpLite6RcvbufErrors = simple_hash("UdpLite6RcvbufErrors"); + hash_UdpLite6SndbufErrors = simple_hash("UdpLite6SndbufErrors"); + hash_UdpLite6InCsumErrors = simple_hash("UdpLite6InCsumErrors"); + } + + if(do_ip_packets == -1) do_ip_packets = config_get_boolean_ondemand("plugin:proc:/proc/net/snmp6", "ipv6 packets", CONFIG_ONDEMAND_ONDEMAND); + if(do_ip_fragsout == -1) do_ip_fragsout = config_get_boolean_ondemand("plugin:proc:/proc/net/snmp6", "ipv6 fragments sent", CONFIG_ONDEMAND_ONDEMAND); + if(do_ip_fragsin == -1) do_ip_fragsin = config_get_boolean_ondemand("plugin:proc:/proc/net/snmp6", "ipv6 fragments assembly", CONFIG_ONDEMAND_ONDEMAND); + if(do_ip_errors == -1) do_ip_errors = config_get_boolean_ondemand("plugin:proc:/proc/net/snmp6", "ipv6 errors", CONFIG_ONDEMAND_ONDEMAND); + if(do_udp_packets == -1) do_udp_packets = config_get_boolean_ondemand("plugin:proc:/proc/net/snmp6", "ipv6 UDP packets", CONFIG_ONDEMAND_ONDEMAND); + if(do_udp_errors == -1) do_udp_errors = config_get_boolean_ondemand("plugin:proc:/proc/net/snmp6", "ipv6 UDP errors", CONFIG_ONDEMAND_ONDEMAND); + if(do_udplite_packets == -1) do_udplite_packets = config_get_boolean_ondemand("plugin:proc:/proc/net/snmp6", "ipv6 UDPlite packets", CONFIG_ONDEMAND_ONDEMAND); + if(do_udplite_errors == -1) do_udplite_errors = config_get_boolean_ondemand("plugin:proc:/proc/net/snmp6", "ipv6 UDPlite errors", CONFIG_ONDEMAND_ONDEMAND); + if(do_bandwidth == -1) do_bandwidth = config_get_boolean_ondemand("plugin:proc:/proc/net/snmp6", "bandwidth", CONFIG_ONDEMAND_ONDEMAND); + if(do_mcast == -1) do_mcast = config_get_boolean_ondemand("plugin:proc:/proc/net/snmp6", "multicast bandwidth", CONFIG_ONDEMAND_ONDEMAND); + if(do_bcast == -1) do_bcast = config_get_boolean_ondemand("plugin:proc:/proc/net/snmp6", "broadcast bandwidth", CONFIG_ONDEMAND_ONDEMAND); + if(do_mcast_p == -1) do_mcast_p = config_get_boolean_ondemand("plugin:proc:/proc/net/snmp6", "multicast packets", CONFIG_ONDEMAND_ONDEMAND); + if(do_icmp == -1) do_icmp = config_get_boolean_ondemand("plugin:proc:/proc/net/snmp6", "icmp", CONFIG_ONDEMAND_ONDEMAND); + if(do_icmp_redir == -1) do_icmp_redir = config_get_boolean_ondemand("plugin:proc:/proc/net/snmp6", "icmp redirects", CONFIG_ONDEMAND_ONDEMAND); + if(do_icmp_errors == -1) do_icmp_errors = config_get_boolean_ondemand("plugin:proc:/proc/net/snmp6", "icmp errors", CONFIG_ONDEMAND_ONDEMAND); + if(do_icmp_echos == -1) do_icmp_echos = config_get_boolean_ondemand("plugin:proc:/proc/net/snmp6", "icmp echos", CONFIG_ONDEMAND_ONDEMAND); + if(do_icmp_groupmemb == -1) do_icmp_groupmemb = config_get_boolean_ondemand("plugin:proc:/proc/net/snmp6", "icmp group membership", CONFIG_ONDEMAND_ONDEMAND); + if(do_icmp_router == -1) do_icmp_router = config_get_boolean_ondemand("plugin:proc:/proc/net/snmp6", "icmp router", CONFIG_ONDEMAND_ONDEMAND); + if(do_icmp_neighbor == -1) do_icmp_neighbor = config_get_boolean_ondemand("plugin:proc:/proc/net/snmp6", "icmp neighbor", CONFIG_ONDEMAND_ONDEMAND); + if(do_icmp_mldv2 == -1) do_icmp_mldv2 = config_get_boolean_ondemand("plugin:proc:/proc/net/snmp6", "icmp mldv2", CONFIG_ONDEMAND_ONDEMAND); + if(do_icmp_types == -1) do_icmp_types = config_get_boolean_ondemand("plugin:proc:/proc/net/snmp6", "icmp types", CONFIG_ONDEMAND_ONDEMAND); + if(do_ect == -1) do_ect = config_get_boolean_ondemand("plugin:proc:/proc/net/snmp6", "ect", CONFIG_ONDEMAND_ONDEMAND); + + if(dt) {}; + + if(!ff) { + char filename[FILENAME_MAX + 1]; + snprintfz(filename, FILENAME_MAX, "%s%s", global_host_prefix, "/proc/net/snmp6"); + ff = procfile_open(config_get("plugin:proc:/proc/net/snmp6", "filename to monitor", filename), " \t:", PROCFILE_FLAG_DEFAULT); + } + if(!ff) return 1; + + ff = procfile_readall(ff); + if(!ff) return 0; // we return 0, so that we will retry to open it next time + + uint32_t lines = procfile_lines(ff), l; + uint32_t words; + + unsigned long long Ip6InReceives = 0ULL; + unsigned long long Ip6InHdrErrors = 0ULL; + unsigned long long Ip6InTooBigErrors = 0ULL; + unsigned long long Ip6InNoRoutes = 0ULL; + unsigned long long Ip6InAddrErrors = 0ULL; + unsigned long long Ip6InUnknownProtos = 0ULL; + unsigned long long Ip6InTruncatedPkts = 0ULL; + unsigned long long Ip6InDiscards = 0ULL; + unsigned long long Ip6InDelivers = 0ULL; + unsigned long long Ip6OutForwDatagrams = 0ULL; + unsigned long long Ip6OutRequests = 0ULL; + unsigned long long Ip6OutDiscards = 0ULL; + unsigned long long Ip6OutNoRoutes = 0ULL; + unsigned long long Ip6ReasmTimeout = 0ULL; + unsigned long long Ip6ReasmReqds = 0ULL; + unsigned long long Ip6ReasmOKs = 0ULL; + unsigned long long Ip6ReasmFails = 0ULL; + unsigned long long Ip6FragOKs = 0ULL; + unsigned long long Ip6FragFails = 0ULL; + unsigned long long Ip6FragCreates = 0ULL; + unsigned long long Ip6InMcastPkts = 0ULL; + unsigned long long Ip6OutMcastPkts = 0ULL; + unsigned long long Ip6InOctets = 0ULL; + unsigned long long Ip6OutOctets = 0ULL; + unsigned long long Ip6InMcastOctets = 0ULL; + unsigned long long Ip6OutMcastOctets = 0ULL; + unsigned long long Ip6InBcastOctets = 0ULL; + unsigned long long Ip6OutBcastOctets = 0ULL; + unsigned long long Ip6InNoECTPkts = 0ULL; + unsigned long long Ip6InECT1Pkts = 0ULL; + unsigned long long Ip6InECT0Pkts = 0ULL; + unsigned long long Ip6InCEPkts = 0ULL; + unsigned long long Icmp6InMsgs = 0ULL; + unsigned long long Icmp6InErrors = 0ULL; + unsigned long long Icmp6OutMsgs = 0ULL; + unsigned long long Icmp6OutErrors = 0ULL; + unsigned long long Icmp6InCsumErrors = 0ULL; + unsigned long long Icmp6InDestUnreachs = 0ULL; + unsigned long long Icmp6InPktTooBigs = 0ULL; + unsigned long long Icmp6InTimeExcds = 0ULL; + unsigned long long Icmp6InParmProblems = 0ULL; + unsigned long long Icmp6InEchos = 0ULL; + unsigned long long Icmp6InEchoReplies = 0ULL; + unsigned long long Icmp6InGroupMembQueries = 0ULL; + unsigned long long Icmp6InGroupMembResponses = 0ULL; + unsigned long long Icmp6InGroupMembReductions = 0ULL; + unsigned long long Icmp6InRouterSolicits = 0ULL; + unsigned long long Icmp6InRouterAdvertisements = 0ULL; + unsigned long long Icmp6InNeighborSolicits = 0ULL; + unsigned long long Icmp6InNeighborAdvertisements = 0ULL; + unsigned long long Icmp6InRedirects = 0ULL; + unsigned long long Icmp6InMLDv2Reports = 0ULL; + unsigned long long Icmp6OutDestUnreachs = 0ULL; + unsigned long long Icmp6OutPktTooBigs = 0ULL; + unsigned long long Icmp6OutTimeExcds = 0ULL; + unsigned long long Icmp6OutParmProblems = 0ULL; + unsigned long long Icmp6OutEchos = 0ULL; + unsigned long long Icmp6OutEchoReplies = 0ULL; + unsigned long long Icmp6OutGroupMembQueries = 0ULL; + unsigned long long Icmp6OutGroupMembResponses = 0ULL; + unsigned long long Icmp6OutGroupMembReductions = 0ULL; + unsigned long long Icmp6OutRouterSolicits = 0ULL; + unsigned long long Icmp6OutRouterAdvertisements = 0ULL; + unsigned long long Icmp6OutNeighborSolicits = 0ULL; + unsigned long long Icmp6OutNeighborAdvertisements = 0ULL; + unsigned long long Icmp6OutRedirects = 0ULL; + unsigned long long Icmp6OutMLDv2Reports = 0ULL; + unsigned long long Icmp6InType1 = 0ULL; + unsigned long long Icmp6InType128 = 0ULL; + unsigned long long Icmp6InType129 = 0ULL; + unsigned long long Icmp6InType136 = 0ULL; + unsigned long long Icmp6OutType1 = 0ULL; + unsigned long long Icmp6OutType128 = 0ULL; + unsigned long long Icmp6OutType129 = 0ULL; + unsigned long long Icmp6OutType133 = 0ULL; + unsigned long long Icmp6OutType135 = 0ULL; + unsigned long long Icmp6OutType143 = 0ULL; + unsigned long long Udp6InDatagrams = 0ULL; + unsigned long long Udp6NoPorts = 0ULL; + unsigned long long Udp6InErrors = 0ULL; + unsigned long long Udp6OutDatagrams = 0ULL; + unsigned long long Udp6RcvbufErrors = 0ULL; + unsigned long long Udp6SndbufErrors = 0ULL; + unsigned long long Udp6InCsumErrors = 0ULL; + unsigned long long Udp6IgnoredMulti = 0ULL; + unsigned long long UdpLite6InDatagrams = 0ULL; + unsigned long long UdpLite6NoPorts = 0ULL; + unsigned long long UdpLite6InErrors = 0ULL; + unsigned long long UdpLite6OutDatagrams = 0ULL; + unsigned long long UdpLite6RcvbufErrors = 0ULL; + unsigned long long UdpLite6SndbufErrors = 0ULL; + unsigned long long UdpLite6InCsumErrors = 0ULL; + + for(l = 0; l < lines ;l++) { + words = procfile_linewords(ff, l); + if(words < 2) { + if(words) error("Cannot read /proc/net/snmp6 line %u. Expected 2 params, read %u.", l, words); + continue; + } + + char *name = procfile_lineword(ff, l, 0); + char * value = procfile_lineword(ff, l, 1); + if(!name || !*name || !value || !*value) continue; + + uint32_t hash = simple_hash(name); + + if(0) ; + else if(hash == hash_Ip6InReceives && strcmp(name, "Ip6InReceives") == 0) Ip6InReceives = strtoull(value, NULL, 10); + else if(hash == hash_Ip6InHdrErrors && strcmp(name, "Ip6InHdrErrors") == 0) Ip6InHdrErrors = strtoull(value, NULL, 10); + else if(hash == hash_Ip6InTooBigErrors && strcmp(name, "Ip6InTooBigErrors") == 0) Ip6InTooBigErrors = strtoull(value, NULL, 10); + else if(hash == hash_Ip6InNoRoutes && strcmp(name, "Ip6InNoRoutes") == 0) Ip6InNoRoutes = strtoull(value, NULL, 10); + else if(hash == hash_Ip6InAddrErrors && strcmp(name, "Ip6InAddrErrors") == 0) Ip6InAddrErrors = strtoull(value, NULL, 10); + else if(hash == hash_Ip6InUnknownProtos && strcmp(name, "Ip6InUnknownProtos") == 0) Ip6InUnknownProtos = strtoull(value, NULL, 10); + else if(hash == hash_Ip6InTruncatedPkts && strcmp(name, "Ip6InTruncatedPkts") == 0) Ip6InTruncatedPkts = strtoull(value, NULL, 10); + else if(hash == hash_Ip6InDiscards && strcmp(name, "Ip6InDiscards") == 0) Ip6InDiscards = strtoull(value, NULL, 10); + else if(hash == hash_Ip6InDelivers && strcmp(name, "Ip6InDelivers") == 0) Ip6InDelivers = strtoull(value, NULL, 10); + else if(hash == hash_Ip6OutForwDatagrams && strcmp(name, "Ip6OutForwDatagrams") == 0) Ip6OutForwDatagrams = strtoull(value, NULL, 10); + else if(hash == hash_Ip6OutRequests && strcmp(name, "Ip6OutRequests") == 0) Ip6OutRequests = strtoull(value, NULL, 10); + else if(hash == hash_Ip6OutDiscards && strcmp(name, "Ip6OutDiscards") == 0) Ip6OutDiscards = strtoull(value, NULL, 10); + else if(hash == hash_Ip6OutNoRoutes && strcmp(name, "Ip6OutNoRoutes") == 0) Ip6OutNoRoutes = strtoull(value, NULL, 10); + else if(hash == hash_Ip6ReasmTimeout && strcmp(name, "Ip6ReasmTimeout") == 0) Ip6ReasmTimeout = strtoull(value, NULL, 10); + else if(hash == hash_Ip6ReasmReqds && strcmp(name, "Ip6ReasmReqds") == 0) Ip6ReasmReqds = strtoull(value, NULL, 10); + else if(hash == hash_Ip6ReasmOKs && strcmp(name, "Ip6ReasmOKs") == 0) Ip6ReasmOKs = strtoull(value, NULL, 10); + else if(hash == hash_Ip6ReasmFails && strcmp(name, "Ip6ReasmFails") == 0) Ip6ReasmFails = strtoull(value, NULL, 10); + else if(hash == hash_Ip6FragOKs && strcmp(name, "Ip6FragOKs") == 0) Ip6FragOKs = strtoull(value, NULL, 10); + else if(hash == hash_Ip6FragFails && strcmp(name, "Ip6FragFails") == 0) Ip6FragFails = strtoull(value, NULL, 10); + else if(hash == hash_Ip6FragCreates && strcmp(name, "Ip6FragCreates") == 0) Ip6FragCreates = strtoull(value, NULL, 10); + else if(hash == hash_Ip6InMcastPkts && strcmp(name, "Ip6InMcastPkts") == 0) Ip6InMcastPkts = strtoull(value, NULL, 10); + else if(hash == hash_Ip6OutMcastPkts && strcmp(name, "Ip6OutMcastPkts") == 0) Ip6OutMcastPkts = strtoull(value, NULL, 10); + else if(hash == hash_Ip6InOctets && strcmp(name, "Ip6InOctets") == 0) Ip6InOctets = strtoull(value, NULL, 10); + else if(hash == hash_Ip6OutOctets && strcmp(name, "Ip6OutOctets") == 0) Ip6OutOctets = strtoull(value, NULL, 10); + else if(hash == hash_Ip6InMcastOctets && strcmp(name, "Ip6InMcastOctets") == 0) Ip6InMcastOctets = strtoull(value, NULL, 10); + else if(hash == hash_Ip6OutMcastOctets && strcmp(name, "Ip6OutMcastOctets") == 0) Ip6OutMcastOctets = strtoull(value, NULL, 10); + else if(hash == hash_Ip6InBcastOctets && strcmp(name, "Ip6InBcastOctets") == 0) Ip6InBcastOctets = strtoull(value, NULL, 10); + else if(hash == hash_Ip6OutBcastOctets && strcmp(name, "Ip6OutBcastOctets") == 0) Ip6OutBcastOctets = strtoull(value, NULL, 10); + else if(hash == hash_Ip6InNoECTPkts && strcmp(name, "Ip6InNoECTPkts") == 0) Ip6InNoECTPkts = strtoull(value, NULL, 10); + else if(hash == hash_Ip6InECT1Pkts && strcmp(name, "Ip6InECT1Pkts") == 0) Ip6InECT1Pkts = strtoull(value, NULL, 10); + else if(hash == hash_Ip6InECT0Pkts && strcmp(name, "Ip6InECT0Pkts") == 0) Ip6InECT0Pkts = strtoull(value, NULL, 10); + else if(hash == hash_Ip6InCEPkts && strcmp(name, "Ip6InCEPkts") == 0) Ip6InCEPkts = strtoull(value, NULL, 10); + else if(hash == hash_Icmp6InMsgs && strcmp(name, "Icmp6InMsgs") == 0) Icmp6InMsgs = strtoull(value, NULL, 10); + else if(hash == hash_Icmp6InErrors && strcmp(name, "Icmp6InErrors") == 0) Icmp6InErrors = strtoull(value, NULL, 10); + else if(hash == hash_Icmp6OutMsgs && strcmp(name, "Icmp6OutMsgs") == 0) Icmp6OutMsgs = strtoull(value, NULL, 10); + else if(hash == hash_Icmp6OutErrors && strcmp(name, "Icmp6OutErrors") == 0) Icmp6OutErrors = strtoull(value, NULL, 10); + else if(hash == hash_Icmp6InCsumErrors && strcmp(name, "Icmp6InCsumErrors") == 0) Icmp6InCsumErrors = strtoull(value, NULL, 10); + else if(hash == hash_Icmp6InDestUnreachs && strcmp(name, "Icmp6InDestUnreachs") == 0) Icmp6InDestUnreachs = strtoull(value, NULL, 10); + else if(hash == hash_Icmp6InPktTooBigs && strcmp(name, "Icmp6InPktTooBigs") == 0) Icmp6InPktTooBigs = strtoull(value, NULL, 10); + else if(hash == hash_Icmp6InTimeExcds && strcmp(name, "Icmp6InTimeExcds") == 0) Icmp6InTimeExcds = strtoull(value, NULL, 10); + else if(hash == hash_Icmp6InParmProblems && strcmp(name, "Icmp6InParmProblems") == 0) Icmp6InParmProblems = strtoull(value, NULL, 10); + else if(hash == hash_Icmp6InEchos && strcmp(name, "Icmp6InEchos") == 0) Icmp6InEchos = strtoull(value, NULL, 10); + else if(hash == hash_Icmp6InEchoReplies && strcmp(name, "Icmp6InEchoReplies") == 0) Icmp6InEchoReplies = strtoull(value, NULL, 10); + else if(hash == hash_Icmp6InGroupMembQueries && strcmp(name, "Icmp6InGroupMembQueries") == 0) Icmp6InGroupMembQueries = strtoull(value, NULL, 10); + else if(hash == hash_Icmp6InGroupMembResponses && strcmp(name, "Icmp6InGroupMembResponses") == 0) Icmp6InGroupMembResponses = strtoull(value, NULL, 10); + else if(hash == hash_Icmp6InGroupMembReductions && strcmp(name, "Icmp6InGroupMembReductions") == 0) Icmp6InGroupMembReductions = strtoull(value, NULL, 10); + else if(hash == hash_Icmp6InRouterSolicits && strcmp(name, "Icmp6InRouterSolicits") == 0) Icmp6InRouterSolicits = strtoull(value, NULL, 10); + else if(hash == hash_Icmp6InRouterAdvertisements && strcmp(name, "Icmp6InRouterAdvertisements") == 0) Icmp6InRouterAdvertisements = strtoull(value, NULL, 10); + else if(hash == hash_Icmp6InNeighborSolicits && strcmp(name, "Icmp6InNeighborSolicits") == 0) Icmp6InNeighborSolicits = strtoull(value, NULL, 10); + else if(hash == hash_Icmp6InNeighborAdvertisements && strcmp(name, "Icmp6InNeighborAdvertisements") == 0) Icmp6InNeighborAdvertisements = strtoull(value, NULL, 10); + else if(hash == hash_Icmp6InRedirects && strcmp(name, "Icmp6InRedirects") == 0) Icmp6InRedirects = strtoull(value, NULL, 10); + else if(hash == hash_Icmp6InMLDv2Reports && strcmp(name, "Icmp6InMLDv2Reports") == 0) Icmp6InMLDv2Reports = strtoull(value, NULL, 10); + else if(hash == hash_Icmp6OutDestUnreachs && strcmp(name, "Icmp6OutDestUnreachs") == 0) Icmp6OutDestUnreachs = strtoull(value, NULL, 10); + else if(hash == hash_Icmp6OutPktTooBigs && strcmp(name, "Icmp6OutPktTooBigs") == 0) Icmp6OutPktTooBigs = strtoull(value, NULL, 10); + else if(hash == hash_Icmp6OutTimeExcds && strcmp(name, "Icmp6OutTimeExcds") == 0) Icmp6OutTimeExcds = strtoull(value, NULL, 10); + else if(hash == hash_Icmp6OutParmProblems && strcmp(name, "Icmp6OutParmProblems") == 0) Icmp6OutParmProblems = strtoull(value, NULL, 10); + else if(hash == hash_Icmp6OutEchos && strcmp(name, "Icmp6OutEchos") == 0) Icmp6OutEchos = strtoull(value, NULL, 10); + else if(hash == hash_Icmp6OutEchoReplies && strcmp(name, "Icmp6OutEchoReplies") == 0) Icmp6OutEchoReplies = strtoull(value, NULL, 10); + else if(hash == hash_Icmp6OutGroupMembQueries && strcmp(name, "Icmp6OutGroupMembQueries") == 0) Icmp6OutGroupMembQueries = strtoull(value, NULL, 10); + else if(hash == hash_Icmp6OutGroupMembResponses && strcmp(name, "Icmp6OutGroupMembResponses") == 0) Icmp6OutGroupMembResponses = strtoull(value, NULL, 10); + else if(hash == hash_Icmp6OutGroupMembReductions && strcmp(name, "Icmp6OutGroupMembReductions") == 0) Icmp6OutGroupMembReductions = strtoull(value, NULL, 10); + else if(hash == hash_Icmp6OutRouterSolicits && strcmp(name, "Icmp6OutRouterSolicits") == 0) Icmp6OutRouterSolicits = strtoull(value, NULL, 10); + else if(hash == hash_Icmp6OutRouterAdvertisements && strcmp(name, "Icmp6OutRouterAdvertisements") == 0) Icmp6OutRouterAdvertisements = strtoull(value, NULL, 10); + else if(hash == hash_Icmp6OutNeighborSolicits && strcmp(name, "Icmp6OutNeighborSolicits") == 0) Icmp6OutNeighborSolicits = strtoull(value, NULL, 10); + else if(hash == hash_Icmp6OutNeighborAdvertisements && strcmp(name, "Icmp6OutNeighborAdvertisements") == 0) Icmp6OutNeighborAdvertisements = strtoull(value, NULL, 10); + else if(hash == hash_Icmp6OutRedirects && strcmp(name, "Icmp6OutRedirects") == 0) Icmp6OutRedirects = strtoull(value, NULL, 10); + else if(hash == hash_Icmp6OutMLDv2Reports && strcmp(name, "Icmp6OutMLDv2Reports") == 0) Icmp6OutMLDv2Reports = strtoull(value, NULL, 10); + else if(hash == hash_Icmp6InType1 && strcmp(name, "Icmp6InType1") == 0) Icmp6InType1 = strtoull(value, NULL, 10); + else if(hash == hash_Icmp6InType128 && strcmp(name, "Icmp6InType128") == 0) Icmp6InType128 = strtoull(value, NULL, 10); + else if(hash == hash_Icmp6InType129 && strcmp(name, "Icmp6InType129") == 0) Icmp6InType129 = strtoull(value, NULL, 10); + else if(hash == hash_Icmp6InType136 && strcmp(name, "Icmp6InType136") == 0) Icmp6InType136 = strtoull(value, NULL, 10); + else if(hash == hash_Icmp6OutType1 && strcmp(name, "Icmp6OutType1") == 0) Icmp6OutType1 = strtoull(value, NULL, 10); + else if(hash == hash_Icmp6OutType128 && strcmp(name, "Icmp6OutType128") == 0) Icmp6OutType128 = strtoull(value, NULL, 10); + else if(hash == hash_Icmp6OutType129 && strcmp(name, "Icmp6OutType129") == 0) Icmp6OutType129 = strtoull(value, NULL, 10); + else if(hash == hash_Icmp6OutType133 && strcmp(name, "Icmp6OutType133") == 0) Icmp6OutType133 = strtoull(value, NULL, 10); + else if(hash == hash_Icmp6OutType135 && strcmp(name, "Icmp6OutType135") == 0) Icmp6OutType135 = strtoull(value, NULL, 10); + else if(hash == hash_Icmp6OutType143 && strcmp(name, "Icmp6OutType143") == 0) Icmp6OutType143 = strtoull(value, NULL, 10); + else if(hash == hash_Udp6InDatagrams && strcmp(name, "Udp6InDatagrams") == 0) Udp6InDatagrams = strtoull(value, NULL, 10); + else if(hash == hash_Udp6NoPorts && strcmp(name, "Udp6NoPorts") == 0) Udp6NoPorts = strtoull(value, NULL, 10); + else if(hash == hash_Udp6InErrors && strcmp(name, "Udp6InErrors") == 0) Udp6InErrors = strtoull(value, NULL, 10); + else if(hash == hash_Udp6OutDatagrams && strcmp(name, "Udp6OutDatagrams") == 0) Udp6OutDatagrams = strtoull(value, NULL, 10); + else if(hash == hash_Udp6RcvbufErrors && strcmp(name, "Udp6RcvbufErrors") == 0) Udp6RcvbufErrors = strtoull(value, NULL, 10); + else if(hash == hash_Udp6SndbufErrors && strcmp(name, "Udp6SndbufErrors") == 0) Udp6SndbufErrors = strtoull(value, NULL, 10); + else if(hash == hash_Udp6InCsumErrors && strcmp(name, "Udp6InCsumErrors") == 0) Udp6InCsumErrors = strtoull(value, NULL, 10); + else if(hash == hash_Udp6IgnoredMulti && strcmp(name, "Udp6IgnoredMulti") == 0) Udp6IgnoredMulti = strtoull(value, NULL, 10); + else if(hash == hash_UdpLite6InDatagrams && strcmp(name, "UdpLite6InDatagrams") == 0) UdpLite6InDatagrams = strtoull(value, NULL, 10); + else if(hash == hash_UdpLite6NoPorts && strcmp(name, "UdpLite6NoPorts") == 0) UdpLite6NoPorts = strtoull(value, NULL, 10); + else if(hash == hash_UdpLite6InErrors && strcmp(name, "UdpLite6InErrors") == 0) UdpLite6InErrors = strtoull(value, NULL, 10); + else if(hash == hash_UdpLite6OutDatagrams && strcmp(name, "UdpLite6OutDatagrams") == 0) UdpLite6OutDatagrams = strtoull(value, NULL, 10); + else if(hash == hash_UdpLite6RcvbufErrors && strcmp(name, "UdpLite6RcvbufErrors") == 0) UdpLite6RcvbufErrors = strtoull(value, NULL, 10); + else if(hash == hash_UdpLite6SndbufErrors && strcmp(name, "UdpLite6SndbufErrors") == 0) UdpLite6SndbufErrors = strtoull(value, NULL, 10); + else if(hash == hash_UdpLite6InCsumErrors && strcmp(name, "UdpLite6InCsumErrors") == 0) UdpLite6InCsumErrors = strtoull(value, NULL, 10); + } + + RRDSET *st; + + // -------------------------------------------------------------------- + + if(do_bandwidth == CONFIG_ONDEMAND_YES || (do_bandwidth == CONFIG_ONDEMAND_ONDEMAND && (Ip6InOctets || Ip6OutOctets))) { + do_bandwidth = CONFIG_ONDEMAND_YES; + st = rrdset_find("system.ipv6"); + if(!st) { + st = rrdset_create("system", "ipv6", NULL, "network", NULL, "IPv6 Bandwidth", "kilobits/s", 500, update_every, RRDSET_TYPE_AREA); + + rrddim_add(st, "received", NULL, 8, 1024, RRDDIM_INCREMENTAL); + rrddim_add(st, "sent", NULL, -8, 1024, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); + + rrddim_set(st, "sent", Ip6OutOctets); + rrddim_set(st, "received", Ip6InOctets); + rrdset_done(st); + } + + // -------------------------------------------------------------------- + + if(do_ip_packets == CONFIG_ONDEMAND_YES || (do_ip_packets == CONFIG_ONDEMAND_ONDEMAND && (Ip6InReceives || Ip6OutRequests || Ip6InDelivers || Ip6OutForwDatagrams))) { + do_ip_packets = CONFIG_ONDEMAND_YES; + st = rrdset_find(RRD_TYPE_NET_SNMP6 ".packets"); + if(!st) { + st = rrdset_create(RRD_TYPE_NET_SNMP6, "packets", NULL, "packets", NULL, "IPv6 Packets", "packets/s", 3000, update_every, RRDSET_TYPE_LINE); + + rrddim_add(st, "received", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "sent", NULL, -1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "forwarded", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "delivers", NULL, -1, 1, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); + + rrddim_set(st, "sent", Ip6OutRequests); + rrddim_set(st, "received", Ip6InReceives); + rrddim_set(st, "forwarded", Ip6InDelivers); + rrddim_set(st, "delivers", Ip6OutForwDatagrams); + rrdset_done(st); + } + + // -------------------------------------------------------------------- + + if(do_ip_fragsout == CONFIG_ONDEMAND_YES || (do_ip_fragsout == CONFIG_ONDEMAND_ONDEMAND && (Ip6FragOKs || Ip6FragFails || Ip6FragCreates))) { + do_ip_fragsout = CONFIG_ONDEMAND_YES; + st = rrdset_find(RRD_TYPE_NET_SNMP6 ".fragsout"); + if(!st) { + st = rrdset_create(RRD_TYPE_NET_SNMP6, "fragsout", NULL, "fragments", NULL, "IPv6 Fragments Sent", "packets/s", 3010, update_every, RRDSET_TYPE_LINE); + st->isdetail = 1; + + rrddim_add(st, "ok", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "failed", NULL, -1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "all", NULL, 1, 1, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); + + rrddim_set(st, "ok", Ip6FragOKs); + rrddim_set(st, "failed", Ip6FragFails); + rrddim_set(st, "all", Ip6FragCreates); + rrdset_done(st); + } + + // -------------------------------------------------------------------- + + if(do_ip_fragsin == CONFIG_ONDEMAND_YES || (do_ip_fragsin == CONFIG_ONDEMAND_ONDEMAND + && ( + Ip6ReasmOKs + || Ip6ReasmFails + || Ip6ReasmTimeout + || Ip6ReasmReqds + ))) { + do_ip_fragsin = CONFIG_ONDEMAND_YES; + st = rrdset_find(RRD_TYPE_NET_SNMP6 ".fragsin"); + if(!st) { + st = rrdset_create(RRD_TYPE_NET_SNMP6, "fragsin", NULL, "fragments", NULL, "IPv6 Fragments Reassembly", "packets/s", 3011, update_every, RRDSET_TYPE_LINE); + st->isdetail = 1; + + rrddim_add(st, "ok", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "failed", NULL, -1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "timeout", NULL, -1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "all", NULL, 1, 1, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); + + rrddim_set(st, "ok", Ip6ReasmOKs); + rrddim_set(st, "failed", Ip6ReasmFails); + rrddim_set(st, "timeout", Ip6ReasmTimeout); + rrddim_set(st, "all", Ip6ReasmReqds); + rrdset_done(st); + } + + // -------------------------------------------------------------------- + + if(do_ip_errors == CONFIG_ONDEMAND_YES || (do_ip_errors == CONFIG_ONDEMAND_ONDEMAND + && ( + Ip6InDiscards + || Ip6OutDiscards + || Ip6InHdrErrors + || Ip6InAddrErrors + || Ip6InUnknownProtos + || Ip6InTooBigErrors + || Ip6InTruncatedPkts + || Ip6InNoRoutes + ))) { + do_ip_errors = CONFIG_ONDEMAND_YES; + st = rrdset_find(RRD_TYPE_NET_SNMP6 ".errors"); + if(!st) { + st = rrdset_create(RRD_TYPE_NET_SNMP6, "errors", NULL, "errors", NULL, "IPv6 Errors", "packets/s", 3002, update_every, RRDSET_TYPE_LINE); + st->isdetail = 1; + + rrddim_add(st, "InDiscards", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "OutDiscards", NULL, -1, 1, RRDDIM_INCREMENTAL); + + rrddim_add(st, "InHdrErrors", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "InAddrErrors", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "InUnknownProtos", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "InTooBigErrors", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "InTruncatedPkts", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "InNoRoutes", NULL, 1, 1, RRDDIM_INCREMENTAL); + + rrddim_add(st, "OutNoRoutes", NULL, -1, 1, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); + + rrddim_set(st, "InDiscards", Ip6InDiscards); + rrddim_set(st, "OutDiscards", Ip6OutDiscards); + + rrddim_set(st, "InHdrErrors", Ip6InHdrErrors); + rrddim_set(st, "InAddrErrors", Ip6InAddrErrors); + rrddim_set(st, "InUnknownProtos", Ip6InUnknownProtos); + rrddim_set(st, "InTooBigErrors", Ip6InTooBigErrors); + rrddim_set(st, "InTruncatedPkts", Ip6InTruncatedPkts); + rrddim_set(st, "InNoRoutes", Ip6InNoRoutes); + + rrddim_set(st, "OutNoRoutes", Ip6OutNoRoutes); + rrdset_done(st); + } + + // -------------------------------------------------------------------- + + if(do_udp_packets == CONFIG_ONDEMAND_YES || (do_udp_packets == CONFIG_ONDEMAND_ONDEMAND && (Udp6InDatagrams || Udp6OutDatagrams))) { + do_udp_packets = CONFIG_ONDEMAND_YES; + st = rrdset_find(RRD_TYPE_NET_SNMP6 ".udppackets"); + if(!st) { + st = rrdset_create(RRD_TYPE_NET_SNMP6, "udppackets", NULL, "udp", NULL, "IPv6 UDP Packets", "packets/s", 3601, update_every, RRDSET_TYPE_LINE); + + rrddim_add(st, "received", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "sent", NULL, -1, 1, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); + + rrddim_set(st, "received", Udp6InDatagrams); + rrddim_set(st, "sent", Udp6OutDatagrams); + rrdset_done(st); + } + + // -------------------------------------------------------------------- + + if(do_udp_errors == CONFIG_ONDEMAND_YES || (do_udp_errors == CONFIG_ONDEMAND_ONDEMAND + && ( + Udp6InErrors + || Udp6NoPorts + || Udp6RcvbufErrors + || Udp6SndbufErrors + || Udp6InCsumErrors + || Udp6IgnoredMulti + ))) { + do_udp_errors = CONFIG_ONDEMAND_YES; + st = rrdset_find(RRD_TYPE_NET_SNMP6 ".udperrors"); + if(!st) { + st = rrdset_create(RRD_TYPE_NET_SNMP6, "udperrors", NULL, "udp", NULL, "IPv6 UDP Errors", "events/s", 3701, update_every, RRDSET_TYPE_LINE); + st->isdetail = 1; + + rrddim_add(st, "RcvbufErrors", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "SndbufErrors", NULL, -1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "InErrors", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "NoPorts", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "InCsumErrors", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "IgnoredMulti", NULL, 1, 1, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); + + rrddim_set(st, "InErrors", Udp6InErrors); + rrddim_set(st, "NoPorts", Udp6NoPorts); + rrddim_set(st, "RcvbufErrors", Udp6RcvbufErrors); + rrddim_set(st, "SndbufErrors", Udp6SndbufErrors); + rrddim_set(st, "InCsumErrors", Udp6InCsumErrors); + rrddim_set(st, "IgnoredMulti", Udp6IgnoredMulti); + rrdset_done(st); + } + + // -------------------------------------------------------------------- + + if(do_udplite_packets == CONFIG_ONDEMAND_YES || (do_udplite_packets == CONFIG_ONDEMAND_ONDEMAND && (UdpLite6InDatagrams || UdpLite6OutDatagrams))) { + do_udplite_packets = CONFIG_ONDEMAND_YES; + st = rrdset_find(RRD_TYPE_NET_SNMP6 ".udplitepackets"); + if(!st) { + st = rrdset_create(RRD_TYPE_NET_SNMP6, "udplitepackets", NULL, "udplite", NULL, "IPv6 UDPlite Packets", "packets/s", 3601, update_every, RRDSET_TYPE_LINE); + + rrddim_add(st, "received", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "sent", NULL, -1, 1, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); + + rrddim_set(st, "received", UdpLite6InDatagrams); + rrddim_set(st, "sent", UdpLite6OutDatagrams); + rrdset_done(st); + } + + // -------------------------------------------------------------------- + + if(do_udplite_errors == CONFIG_ONDEMAND_YES || (do_udplite_errors == CONFIG_ONDEMAND_ONDEMAND + && ( + UdpLite6InErrors + || UdpLite6NoPorts + || UdpLite6RcvbufErrors + || UdpLite6SndbufErrors + || Udp6InCsumErrors + || UdpLite6InCsumErrors + ))) { + do_udplite_errors = CONFIG_ONDEMAND_YES; + st = rrdset_find(RRD_TYPE_NET_SNMP6 ".udpliteerrors"); + if(!st) { + st = rrdset_create(RRD_TYPE_NET_SNMP6, "udpliteerrors", NULL, "udplite", NULL, "IPv6 UDP Lite Errors", "events/s", 3701, update_every, RRDSET_TYPE_LINE); + st->isdetail = 1; + + rrddim_add(st, "RcvbufErrors", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "SndbufErrors", NULL, -1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "InErrors", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "NoPorts", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "InCsumErrors", NULL, 1, 1, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); + + rrddim_set(st, "InErrors", UdpLite6InErrors); + rrddim_set(st, "NoPorts", UdpLite6NoPorts); + rrddim_set(st, "RcvbufErrors", UdpLite6RcvbufErrors); + rrddim_set(st, "SndbufErrors", UdpLite6SndbufErrors); + rrddim_set(st, "InCsumErrors", UdpLite6InCsumErrors); + rrdset_done(st); + } + + // -------------------------------------------------------------------- + + if(do_mcast == CONFIG_ONDEMAND_YES || (do_mcast == CONFIG_ONDEMAND_ONDEMAND && (Ip6OutMcastOctets || Ip6InMcastOctets))) { + do_mcast = CONFIG_ONDEMAND_YES; + st = rrdset_find(RRD_TYPE_NET_SNMP6 ".mcast"); + if(!st) { + st = rrdset_create(RRD_TYPE_NET_SNMP6, "mcast", NULL, "multicast", NULL, "IPv6 Multicast Bandwidth", "kilobits/s", 9000, update_every, RRDSET_TYPE_AREA); + st->isdetail = 1; + + rrddim_add(st, "received", NULL, 8, 1024, RRDDIM_INCREMENTAL); + rrddim_add(st, "sent", NULL, -8, 1024, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); + + rrddim_set(st, "sent", Ip6OutMcastOctets); + rrddim_set(st, "received", Ip6InMcastOctets); + rrdset_done(st); + } + + // -------------------------------------------------------------------- + + if(do_bcast == CONFIG_ONDEMAND_YES || (do_bcast == CONFIG_ONDEMAND_ONDEMAND && (Ip6OutBcastOctets || Ip6InBcastOctets))) { + do_bcast = CONFIG_ONDEMAND_YES; + st = rrdset_find(RRD_TYPE_NET_SNMP6 ".bcast"); + if(!st) { + st = rrdset_create(RRD_TYPE_NET_SNMP6, "bcast", NULL, "broadcast", NULL, "IPv6 Broadcast Bandwidth", "kilobits/s", 8000, update_every, RRDSET_TYPE_AREA); + st->isdetail = 1; + + rrddim_add(st, "received", NULL, 8, 1024, RRDDIM_INCREMENTAL); + rrddim_add(st, "sent", NULL, -8, 1024, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); + + rrddim_set(st, "sent", Ip6OutBcastOctets); + rrddim_set(st, "received", Ip6InBcastOctets); + rrdset_done(st); + } + + // -------------------------------------------------------------------- + + if(do_mcast_p == CONFIG_ONDEMAND_YES || (do_mcast_p == CONFIG_ONDEMAND_ONDEMAND && (Ip6OutMcastPkts || Ip6InMcastPkts))) { + do_mcast_p = CONFIG_ONDEMAND_YES; + st = rrdset_find(RRD_TYPE_NET_SNMP6 ".mcastpkts"); + if(!st) { + st = rrdset_create(RRD_TYPE_NET_SNMP6, "mcastpkts", NULL, "multicast", NULL, "IPv6 Multicast Packets", "packets/s", 9500, update_every, RRDSET_TYPE_LINE); + st->isdetail = 1; + + rrddim_add(st, "received", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "sent", NULL, -1, 1, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); + + rrddim_set(st, "sent", Ip6OutMcastPkts); + rrddim_set(st, "received", Ip6InMcastPkts); + rrdset_done(st); + } + + // -------------------------------------------------------------------- + + if(do_icmp == CONFIG_ONDEMAND_YES || (do_icmp == CONFIG_ONDEMAND_ONDEMAND && (Icmp6InMsgs || Icmp6OutMsgs))) { + do_icmp = CONFIG_ONDEMAND_YES; + st = rrdset_find(RRD_TYPE_NET_SNMP6 ".icmp"); + if(!st) { + st = rrdset_create(RRD_TYPE_NET_SNMP6, "icmp", NULL, "icmp", NULL, "IPv6 ICMP Messages", "messages/s", 10000, update_every, RRDSET_TYPE_LINE); + + rrddim_add(st, "received", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "sent", NULL, -1, 1, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); + + rrddim_set(st, "sent", Icmp6InMsgs); + rrddim_set(st, "received", Icmp6OutMsgs); + rrdset_done(st); + } + + // -------------------------------------------------------------------- + + if(do_icmp_redir == CONFIG_ONDEMAND_YES || (do_icmp_redir == CONFIG_ONDEMAND_ONDEMAND && (Icmp6InRedirects || Icmp6OutRedirects))) { + do_icmp_redir = CONFIG_ONDEMAND_YES; + st = rrdset_find(RRD_TYPE_NET_SNMP6 ".icmpredir"); + if(!st) { + st = rrdset_create(RRD_TYPE_NET_SNMP6, "icmpredir", NULL, "icmp", NULL, "IPv6 ICMP Redirects", "redirects/s", 10050, update_every, RRDSET_TYPE_LINE); + + rrddim_add(st, "received", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "sent", NULL, -1, 1, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); + + rrddim_set(st, "sent", Icmp6InRedirects); + rrddim_set(st, "received", Icmp6OutRedirects); + rrdset_done(st); + } + + // -------------------------------------------------------------------- + + if(do_icmp_errors == CONFIG_ONDEMAND_YES || (do_icmp_errors == CONFIG_ONDEMAND_ONDEMAND + && ( + Icmp6InErrors + || Icmp6OutErrors + || Icmp6InCsumErrors + || Icmp6InDestUnreachs + || Icmp6InPktTooBigs + || Icmp6InTimeExcds + || Icmp6InParmProblems + || Icmp6OutDestUnreachs + || Icmp6OutPktTooBigs + || Icmp6OutTimeExcds + || Icmp6OutParmProblems + ))) { + do_icmp_errors = CONFIG_ONDEMAND_YES; + st = rrdset_find(RRD_TYPE_NET_SNMP6 ".icmperrors"); + if(!st) { + st = rrdset_create(RRD_TYPE_NET_SNMP6, "icmperrors", NULL, "icmp", NULL, "IPv6 ICMP Errors", "errors/s", 10100, update_every, RRDSET_TYPE_LINE); + + rrddim_add(st, "InErrors", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "OutErrors", NULL, -1, 1, RRDDIM_INCREMENTAL); + + rrddim_add(st, "InCsumErrors", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "InDestUnreachs", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "InPktTooBigs", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "InTimeExcds", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "InParmProblems", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "OutDestUnreachs", NULL, -1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "OutPktTooBigs", NULL, -1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "OutTimeExcds", NULL, -1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "OutParmProblems", NULL, -1, 1, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); + + rrddim_set(st, "InErrors", Icmp6InErrors); + rrddim_set(st, "OutErrors", Icmp6OutErrors); + rrddim_set(st, "InCsumErrors", Icmp6InCsumErrors); + rrddim_set(st, "InDestUnreachs", Icmp6InDestUnreachs); + rrddim_set(st, "InPktTooBigs", Icmp6InPktTooBigs); + rrddim_set(st, "InTimeExcds", Icmp6InTimeExcds); + rrddim_set(st, "InParmProblems", Icmp6InParmProblems); + rrddim_set(st, "OutDestUnreachs", Icmp6OutDestUnreachs); + rrddim_set(st, "OutPktTooBigs", Icmp6OutPktTooBigs); + rrddim_set(st, "OutTimeExcds", Icmp6OutTimeExcds); + rrddim_set(st, "OutParmProblems", Icmp6OutParmProblems); + rrdset_done(st); + } + + // -------------------------------------------------------------------- + + if(do_icmp_echos == CONFIG_ONDEMAND_YES || (do_icmp_echos == CONFIG_ONDEMAND_ONDEMAND + && ( + Icmp6InEchos + || Icmp6OutEchos + || Icmp6InEchoReplies + || Icmp6OutEchoReplies + ))) { + do_icmp_echos = CONFIG_ONDEMAND_YES; + st = rrdset_find(RRD_TYPE_NET_SNMP6 ".icmpechos"); + if(!st) { + st = rrdset_create(RRD_TYPE_NET_SNMP6, "icmpechos", NULL, "icmp", NULL, "IPv6 ICMP Echo", "messages/s", 10200, update_every, RRDSET_TYPE_LINE); + + rrddim_add(st, "InEchos", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "OutEchos", NULL, -1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "InEchoReplies", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "OutEchoReplies", NULL, -1, 1, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); + + rrddim_set(st, "InEchos", Icmp6InEchos); + rrddim_set(st, "OutEchos", Icmp6OutEchos); + rrddim_set(st, "InEchoReplies", Icmp6InEchoReplies); + rrddim_set(st, "OutEchoReplies", Icmp6OutEchoReplies); + rrdset_done(st); + } + + // -------------------------------------------------------------------- + + if(do_icmp_groupmemb == CONFIG_ONDEMAND_YES || (do_icmp_groupmemb == CONFIG_ONDEMAND_ONDEMAND + && ( + Icmp6InGroupMembQueries + || Icmp6OutGroupMembQueries + || Icmp6InGroupMembResponses + || Icmp6OutGroupMembResponses + || Icmp6InGroupMembReductions + || Icmp6OutGroupMembReductions + ))) { + do_icmp_groupmemb = CONFIG_ONDEMAND_YES; + st = rrdset_find(RRD_TYPE_NET_SNMP6 ".groupmemb"); + if(!st) { + st = rrdset_create(RRD_TYPE_NET_SNMP6, "groupmemb", NULL, "icmp", NULL, "IPv6 ICMP Group Membership", "messages/s", 10300, update_every, RRDSET_TYPE_LINE); + + rrddim_add(st, "InQueries", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "OutQueries", NULL, -1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "InResponses", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "OutResponses", NULL, -1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "InReductions", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "OutReductions", NULL, -1, 1, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); + + rrddim_set(st, "InQueries", Icmp6InGroupMembQueries); + rrddim_set(st, "OutQueries", Icmp6OutGroupMembQueries); + rrddim_set(st, "InResponses", Icmp6InGroupMembResponses); + rrddim_set(st, "OutResponses", Icmp6OutGroupMembResponses); + rrddim_set(st, "InReductions", Icmp6InGroupMembReductions); + rrddim_set(st, "OutReductions", Icmp6OutGroupMembReductions); + rrdset_done(st); + } + + // -------------------------------------------------------------------- + + if(do_icmp_router == CONFIG_ONDEMAND_YES || (do_icmp_router == CONFIG_ONDEMAND_ONDEMAND + && ( + Icmp6InRouterSolicits + || Icmp6OutRouterSolicits + || Icmp6InRouterAdvertisements + || Icmp6OutRouterAdvertisements + ))) { + do_icmp_router = CONFIG_ONDEMAND_YES; + st = rrdset_find(RRD_TYPE_NET_SNMP6 ".icmprouter"); + if(!st) { + st = rrdset_create(RRD_TYPE_NET_SNMP6, "icmprouter", NULL, "icmp", NULL, "IPv6 Router Messages", "messages/s", 10400, update_every, RRDSET_TYPE_LINE); + + rrddim_add(st, "InSolicits", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "OutSolicits", NULL, -1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "InAdvertisements", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "OutAdvertisements", NULL, -1, 1, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); + + rrddim_set(st, "InSolicits", Icmp6InRouterSolicits); + rrddim_set(st, "OutSolicits", Icmp6OutRouterSolicits); + rrddim_set(st, "InAdvertisements", Icmp6InRouterAdvertisements); + rrddim_set(st, "OutAdvertisements", Icmp6OutRouterAdvertisements); + rrdset_done(st); + } + + // -------------------------------------------------------------------- + + if(do_icmp_neighbor == CONFIG_ONDEMAND_YES || (do_icmp_neighbor == CONFIG_ONDEMAND_ONDEMAND + && ( + Icmp6InNeighborSolicits + || Icmp6OutNeighborSolicits + || Icmp6InNeighborAdvertisements + || Icmp6OutNeighborAdvertisements + ))) { + do_icmp_neighbor = CONFIG_ONDEMAND_YES; + st = rrdset_find(RRD_TYPE_NET_SNMP6 ".icmpneighbor"); + if(!st) { + st = rrdset_create(RRD_TYPE_NET_SNMP6, "icmpneighbor", NULL, "icmp", NULL, "IPv6 Neighbor Messages", "messages/s", 10500, update_every, RRDSET_TYPE_LINE); + + rrddim_add(st, "InSolicits", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "OutSolicits", NULL, -1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "InAdvertisements", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "OutAdvertisements", NULL, -1, 1, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); + + rrddim_set(st, "InSolicits", Icmp6InNeighborSolicits); + rrddim_set(st, "OutSolicits", Icmp6OutNeighborSolicits); + rrddim_set(st, "InAdvertisements", Icmp6InNeighborAdvertisements); + rrddim_set(st, "OutAdvertisements", Icmp6OutNeighborAdvertisements); + rrdset_done(st); + } + + // -------------------------------------------------------------------- + + if(do_icmp_mldv2 == CONFIG_ONDEMAND_YES || (do_icmp_mldv2 == CONFIG_ONDEMAND_ONDEMAND && (Icmp6InMLDv2Reports || Icmp6OutMLDv2Reports))) { + do_icmp_mldv2 = CONFIG_ONDEMAND_YES; + st = rrdset_find(RRD_TYPE_NET_SNMP6 ".icmpmldv2"); + if(!st) { + st = rrdset_create(RRD_TYPE_NET_SNMP6, "icmpmldv2", NULL, "icmp", NULL, "IPv6 ICMP MLDv2 Reports", "reports/s", 10600, update_every, RRDSET_TYPE_LINE); + + rrddim_add(st, "received", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "sent", NULL, -1, 1, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); + + rrddim_set(st, "sent", Icmp6InMLDv2Reports); + rrddim_set(st, "received", Icmp6OutMLDv2Reports); + rrdset_done(st); + } + + // -------------------------------------------------------------------- + + if(do_icmp_types == CONFIG_ONDEMAND_YES || (do_icmp_types == CONFIG_ONDEMAND_ONDEMAND + && ( + Icmp6InType1 + || Icmp6InType128 + || Icmp6InType129 + || Icmp6InType136 + || Icmp6OutType1 + || Icmp6OutType128 + || Icmp6OutType129 + || Icmp6OutType133 + || Icmp6OutType135 + || Icmp6OutType143 + ))) { + do_icmp_types = CONFIG_ONDEMAND_YES; + st = rrdset_find(RRD_TYPE_NET_SNMP6 ".icmptypes"); + if(!st) { + st = rrdset_create(RRD_TYPE_NET_SNMP6, "icmptypes", NULL, "icmp", NULL, "IPv6 ICMP Types", "messages/s", 10700, update_every, RRDSET_TYPE_LINE); + + rrddim_add(st, "InType1", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "InType128", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "InType129", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "InType136", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "OutType1", NULL, -1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "OutType128", NULL, -1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "OutType129", NULL, -1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "OutType133", NULL, -1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "OutType135", NULL, -1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "OutType143", NULL, -1, 1, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); + + rrddim_set(st, "InType1", Icmp6InType1); + rrddim_set(st, "InType128", Icmp6InType128); + rrddim_set(st, "InType129", Icmp6InType129); + rrddim_set(st, "InType136", Icmp6InType136); + rrddim_set(st, "OutType1", Icmp6OutType1); + rrddim_set(st, "OutType128", Icmp6OutType128); + rrddim_set(st, "OutType129", Icmp6OutType129); + rrddim_set(st, "OutType133", Icmp6OutType133); + rrddim_set(st, "OutType135", Icmp6OutType135); + rrddim_set(st, "OutType143", Icmp6OutType143); + rrdset_done(st); + } + + // -------------------------------------------------------------------- + + if(do_ect == CONFIG_ONDEMAND_YES || (do_ect == CONFIG_ONDEMAND_ONDEMAND + && ( + Ip6InNoECTPkts + || Ip6InECT1Pkts + || Ip6InECT0Pkts + || Ip6InCEPkts + ))) { + do_ect = CONFIG_ONDEMAND_YES; + st = rrdset_find(RRD_TYPE_NET_SNMP6 ".ect"); + if(!st) { + st = rrdset_create(RRD_TYPE_NET_SNMP6, "ect", NULL, "packets", NULL, "IPv6 ECT Packets", "packets/s", 10800, update_every, RRDSET_TYPE_LINE); + + rrddim_add(st, "InNoECTPkts", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "InECT1Pkts", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "InECT0Pkts", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "InCEPkts", NULL, 1, 1, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); + + rrddim_set(st, "InNoECTPkts", Ip6InNoECTPkts); + rrddim_set(st, "InECT1Pkts", Ip6InECT1Pkts); + rrddim_set(st, "InECT0Pkts", Ip6InECT0Pkts); + rrddim_set(st, "InCEPkts", Ip6InCEPkts); + rrdset_done(st); + } + + return 0; } diff --git a/src/proc_net_stat_conntrack.c b/src/proc_net_stat_conntrack.c index 7d754a1d9..8234b20d2 100644 --- a/src/proc_net_stat_conntrack.c +++ b/src/proc_net_stat_conntrack.c @@ -1,215 +1,203 @@ -#ifdef HAVE_CONFIG_H -#include <config.h> -#endif -#include <stdio.h> -#include <stdlib.h> -#include <string.h> - #include "common.h" -#include "log.h" -#include "appconfig.h" -#include "procfile.h" -#include "rrd.h" -#include "plugin_proc.h" -#define RRD_TYPE_NET_STAT_NETFILTER "netfilter" -#define RRD_TYPE_NET_STAT_CONNTRACK "conntrack" -#define RRD_TYPE_NET_STAT_CONNTRACK_LEN strlen(RRD_TYPE_NET_STAT_CONNTRACK) +#define RRD_TYPE_NET_STAT_NETFILTER "netfilter" +#define RRD_TYPE_NET_STAT_CONNTRACK "conntrack" +#define RRD_TYPE_NET_STAT_CONNTRACK_LEN strlen(RRD_TYPE_NET_STAT_CONNTRACK) int do_proc_net_stat_conntrack(int update_every, unsigned long long dt) { - static procfile *ff = NULL; - static int do_sockets = -1, do_new = -1, do_changes = -1, do_expect = -1, do_search = -1, do_errors = -1; - - if(do_sockets == -1) do_sockets = config_get_boolean("plugin:proc:/proc/net/stat/nf_conntrack", "netfilter connections", 1); - if(do_new == -1) do_new = config_get_boolean("plugin:proc:/proc/net/stat/nf_conntrack", "netfilter new connections", 1); - if(do_changes == -1) do_changes = config_get_boolean("plugin:proc:/proc/net/stat/nf_conntrack", "netfilter connection changes", 1); - if(do_expect == -1) do_expect = config_get_boolean("plugin:proc:/proc/net/stat/nf_conntrack", "netfilter connection expectations", 1); - if(do_search == -1) do_search = config_get_boolean("plugin:proc:/proc/net/stat/nf_conntrack", "netfilter connection searches", 1); - if(do_errors == -1) do_errors = config_get_boolean("plugin:proc:/proc/net/stat/nf_conntrack", "netfilter errors", 1); - - if(dt) {}; - - if(!ff) { - char filename[FILENAME_MAX + 1]; - snprintfz(filename, FILENAME_MAX, "%s%s", global_host_prefix, "/proc/net/stat/nf_conntrack"); - ff = procfile_open(config_get("plugin:proc:/proc/net/stat/nf_conntrack", "filename to monitor", filename), " \t:", PROCFILE_FLAG_DEFAULT); - } - if(!ff) return 1; - - ff = procfile_readall(ff); - if(!ff) return 0; // we return 0, so that we will retry to open it next time - - uint32_t lines = procfile_lines(ff), l; - uint32_t words; - - unsigned long long aentries = 0, asearched = 0, afound = 0, anew = 0, ainvalid = 0, aignore = 0, adelete = 0, adelete_list = 0, - ainsert = 0, ainsert_failed = 0, adrop = 0, aearly_drop = 0, aicmp_error = 0, aexpect_new = 0, aexpect_create = 0, aexpect_delete = 0, asearch_restart = 0; - - for(l = 1; l < lines ;l++) { - words = procfile_linewords(ff, l); - if(words < 17) { - if(words) error("Cannot read /proc/net/stat/nf_conntrack line. Expected 17 params, read %d.", words); - continue; - } - - unsigned long long tentries = 0, tsearched = 0, tfound = 0, tnew = 0, tinvalid = 0, tignore = 0, tdelete = 0, tdelete_list = 0, tinsert = 0, tinsert_failed = 0, tdrop = 0, tearly_drop = 0, ticmp_error = 0, texpect_new = 0, texpect_create = 0, texpect_delete = 0, tsearch_restart = 0; - - tentries = strtoull(procfile_lineword(ff, l, 0), NULL, 16); - tsearched = strtoull(procfile_lineword(ff, l, 1), NULL, 16); - tfound = strtoull(procfile_lineword(ff, l, 2), NULL, 16); - tnew = strtoull(procfile_lineword(ff, l, 3), NULL, 16); - tinvalid = strtoull(procfile_lineword(ff, l, 4), NULL, 16); - tignore = strtoull(procfile_lineword(ff, l, 5), NULL, 16); - tdelete = strtoull(procfile_lineword(ff, l, 6), NULL, 16); - tdelete_list = strtoull(procfile_lineword(ff, l, 7), NULL, 16); - tinsert = strtoull(procfile_lineword(ff, l, 8), NULL, 16); - tinsert_failed = strtoull(procfile_lineword(ff, l, 9), NULL, 16); - tdrop = strtoull(procfile_lineword(ff, l, 10), NULL, 16); - tearly_drop = strtoull(procfile_lineword(ff, l, 11), NULL, 16); - ticmp_error = strtoull(procfile_lineword(ff, l, 12), NULL, 16); - texpect_new = strtoull(procfile_lineword(ff, l, 13), NULL, 16); - texpect_create = strtoull(procfile_lineword(ff, l, 14), NULL, 16); - texpect_delete = strtoull(procfile_lineword(ff, l, 15), NULL, 16); - tsearch_restart = strtoull(procfile_lineword(ff, l, 16), NULL, 16); - - if(!aentries) aentries = tentries; - - // sum all the cpus together - asearched += tsearched; // conntrack.search - afound += tfound; // conntrack.search - anew += tnew; // conntrack.new - ainvalid += tinvalid; // conntrack.new - aignore += tignore; // conntrack.new - adelete += tdelete; // conntrack.changes - adelete_list += tdelete_list; // conntrack.changes - ainsert += tinsert; // conntrack.changes - ainsert_failed += tinsert_failed; // conntrack.errors - adrop += tdrop; // conntrack.errors - aearly_drop += tearly_drop; // conntrack.errors - aicmp_error += ticmp_error; // conntrack.errors - aexpect_new += texpect_new; // conntrack.expect - aexpect_create += texpect_create; // conntrack.expect - aexpect_delete += texpect_delete; // conntrack.expect - asearch_restart += tsearch_restart; // conntrack.search - } - - RRDSET *st; - - // -------------------------------------------------------------------- - - if(do_sockets) { - st = rrdset_find(RRD_TYPE_NET_STAT_NETFILTER "." RRD_TYPE_NET_STAT_CONNTRACK "_sockets"); - if(!st) { - st = rrdset_create(RRD_TYPE_NET_STAT_NETFILTER, RRD_TYPE_NET_STAT_CONNTRACK "_sockets", NULL, RRD_TYPE_NET_STAT_CONNTRACK, NULL, "Connection Tracker Connections", "active connections", 1000, update_every, RRDSET_TYPE_LINE); - - rrddim_add(st, "connections", NULL, 1, 1, RRDDIM_ABSOLUTE); - } - else rrdset_next(st); - - rrddim_set(st, "connections", aentries); - rrdset_done(st); - } - - // -------------------------------------------------------------------- - - if(do_new) { - st = rrdset_find(RRD_TYPE_NET_STAT_NETFILTER "." RRD_TYPE_NET_STAT_CONNTRACK "_new"); - if(!st) { - st = rrdset_create(RRD_TYPE_NET_STAT_NETFILTER, RRD_TYPE_NET_STAT_CONNTRACK "_new", NULL, RRD_TYPE_NET_STAT_CONNTRACK, NULL, "Connection Tracker New Connections", "connections/s", 1001, update_every, RRDSET_TYPE_LINE); - - rrddim_add(st, "new", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "ignore", NULL, -1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "invalid", NULL, -1, 1, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); - - rrddim_set(st, "new", anew); - rrddim_set(st, "ignore", aignore); - rrddim_set(st, "invalid", ainvalid); - rrdset_done(st); - } - - // -------------------------------------------------------------------- - - if(do_changes) { - st = rrdset_find(RRD_TYPE_NET_STAT_NETFILTER "." RRD_TYPE_NET_STAT_CONNTRACK "_changes"); - if(!st) { - st = rrdset_create(RRD_TYPE_NET_STAT_NETFILTER, RRD_TYPE_NET_STAT_CONNTRACK "_changes", NULL, RRD_TYPE_NET_STAT_CONNTRACK, NULL, "Connection Tracker Changes", "changes/s", 1002, update_every, RRDSET_TYPE_LINE); - st->isdetail = 1; - - rrddim_add(st, "inserted", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "deleted", NULL, -1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "delete_list", NULL, -1, 1, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); - - rrddim_set(st, "inserted", ainsert); - rrddim_set(st, "deleted", adelete); - rrddim_set(st, "delete_list", adelete_list); - rrdset_done(st); - } - - // -------------------------------------------------------------------- - - if(do_expect) { - st = rrdset_find(RRD_TYPE_NET_STAT_NETFILTER "." RRD_TYPE_NET_STAT_CONNTRACK "_expect"); - if(!st) { - st = rrdset_create(RRD_TYPE_NET_STAT_NETFILTER, RRD_TYPE_NET_STAT_CONNTRACK "_expect", NULL, RRD_TYPE_NET_STAT_CONNTRACK, NULL, "Connection Tracker Expectations", "expectations/s", 1003, update_every, RRDSET_TYPE_LINE); - st->isdetail = 1; - - rrddim_add(st, "created", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "deleted", NULL, -1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "new", NULL, 1, 1, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); - - rrddim_set(st, "created", aexpect_create); - rrddim_set(st, "deleted", aexpect_delete); - rrddim_set(st, "new", aexpect_new); - rrdset_done(st); - } - - // -------------------------------------------------------------------- - - if(do_search) { - st = rrdset_find(RRD_TYPE_NET_STAT_NETFILTER "." RRD_TYPE_NET_STAT_CONNTRACK "_search"); - if(!st) { - st = rrdset_create(RRD_TYPE_NET_STAT_NETFILTER, RRD_TYPE_NET_STAT_CONNTRACK "_search", NULL, RRD_TYPE_NET_STAT_CONNTRACK, NULL, "Connection Tracker Searches", "searches/s", 1010, update_every, RRDSET_TYPE_LINE); - st->isdetail = 1; - - rrddim_add(st, "searched", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "restarted", NULL, -1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "found", NULL, 1, 1, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); - - rrddim_set(st, "searched", asearched); - rrddim_set(st, "restarted", asearch_restart); - rrddim_set(st, "found", afound); - rrdset_done(st); - } - - // -------------------------------------------------------------------- - - if(do_errors) { - st = rrdset_find(RRD_TYPE_NET_STAT_NETFILTER "." RRD_TYPE_NET_STAT_CONNTRACK "_errors"); - if(!st) { - st = rrdset_create(RRD_TYPE_NET_STAT_NETFILTER, RRD_TYPE_NET_STAT_CONNTRACK "_errors", NULL, RRD_TYPE_NET_STAT_CONNTRACK, NULL, "Connection Tracker Errors", "events/s", 1005, update_every, RRDSET_TYPE_LINE); - st->isdetail = 1; - - rrddim_add(st, "icmp_error", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "insert_failed", NULL, -1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "drop", NULL, -1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "early_drop", NULL, -1, 1, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); - - rrddim_set(st, "icmp_error", aicmp_error); - rrddim_set(st, "insert_failed", ainsert_failed); - rrddim_set(st, "drop", adrop); - rrddim_set(st, "early_drop", aearly_drop); - rrdset_done(st); - } - - return 0; + static procfile *ff = NULL; + static int do_sockets = -1, do_new = -1, do_changes = -1, do_expect = -1, do_search = -1, do_errors = -1; + + if(do_sockets == -1) do_sockets = config_get_boolean("plugin:proc:/proc/net/stat/nf_conntrack", "netfilter connections", 1); + if(do_new == -1) do_new = config_get_boolean("plugin:proc:/proc/net/stat/nf_conntrack", "netfilter new connections", 1); + if(do_changes == -1) do_changes = config_get_boolean("plugin:proc:/proc/net/stat/nf_conntrack", "netfilter connection changes", 1); + if(do_expect == -1) do_expect = config_get_boolean("plugin:proc:/proc/net/stat/nf_conntrack", "netfilter connection expectations", 1); + if(do_search == -1) do_search = config_get_boolean("plugin:proc:/proc/net/stat/nf_conntrack", "netfilter connection searches", 1); + if(do_errors == -1) do_errors = config_get_boolean("plugin:proc:/proc/net/stat/nf_conntrack", "netfilter errors", 1); + + if(dt) {}; + + if(!ff) { + char filename[FILENAME_MAX + 1]; + snprintfz(filename, FILENAME_MAX, "%s%s", global_host_prefix, "/proc/net/stat/nf_conntrack"); + ff = procfile_open(config_get("plugin:proc:/proc/net/stat/nf_conntrack", "filename to monitor", filename), " \t:", PROCFILE_FLAG_DEFAULT); + } + if(!ff) return 1; + + ff = procfile_readall(ff); + if(!ff) return 0; // we return 0, so that we will retry to open it next time + + uint32_t lines = procfile_lines(ff), l; + uint32_t words; + + unsigned long long aentries = 0, asearched = 0, afound = 0, anew = 0, ainvalid = 0, aignore = 0, adelete = 0, adelete_list = 0, + ainsert = 0, ainsert_failed = 0, adrop = 0, aearly_drop = 0, aicmp_error = 0, aexpect_new = 0, aexpect_create = 0, aexpect_delete = 0, asearch_restart = 0; + + for(l = 1; l < lines ;l++) { + words = procfile_linewords(ff, l); + if(words < 17) { + if(words) error("Cannot read /proc/net/stat/nf_conntrack line. Expected 17 params, read %u.", words); + continue; + } + + unsigned long long tentries = 0, tsearched = 0, tfound = 0, tnew = 0, tinvalid = 0, tignore = 0, tdelete = 0, tdelete_list = 0, tinsert = 0, tinsert_failed = 0, tdrop = 0, tearly_drop = 0, ticmp_error = 0, texpect_new = 0, texpect_create = 0, texpect_delete = 0, tsearch_restart = 0; + + tentries = strtoull(procfile_lineword(ff, l, 0), NULL, 16); + tsearched = strtoull(procfile_lineword(ff, l, 1), NULL, 16); + tfound = strtoull(procfile_lineword(ff, l, 2), NULL, 16); + tnew = strtoull(procfile_lineword(ff, l, 3), NULL, 16); + tinvalid = strtoull(procfile_lineword(ff, l, 4), NULL, 16); + tignore = strtoull(procfile_lineword(ff, l, 5), NULL, 16); + tdelete = strtoull(procfile_lineword(ff, l, 6), NULL, 16); + tdelete_list = strtoull(procfile_lineword(ff, l, 7), NULL, 16); + tinsert = strtoull(procfile_lineword(ff, l, 8), NULL, 16); + tinsert_failed = strtoull(procfile_lineword(ff, l, 9), NULL, 16); + tdrop = strtoull(procfile_lineword(ff, l, 10), NULL, 16); + tearly_drop = strtoull(procfile_lineword(ff, l, 11), NULL, 16); + ticmp_error = strtoull(procfile_lineword(ff, l, 12), NULL, 16); + texpect_new = strtoull(procfile_lineword(ff, l, 13), NULL, 16); + texpect_create = strtoull(procfile_lineword(ff, l, 14), NULL, 16); + texpect_delete = strtoull(procfile_lineword(ff, l, 15), NULL, 16); + tsearch_restart = strtoull(procfile_lineword(ff, l, 16), NULL, 16); + + if(!aentries) aentries = tentries; + + // sum all the cpus together + asearched += tsearched; // conntrack.search + afound += tfound; // conntrack.search + anew += tnew; // conntrack.new + ainvalid += tinvalid; // conntrack.new + aignore += tignore; // conntrack.new + adelete += tdelete; // conntrack.changes + adelete_list += tdelete_list; // conntrack.changes + ainsert += tinsert; // conntrack.changes + ainsert_failed += tinsert_failed; // conntrack.errors + adrop += tdrop; // conntrack.errors + aearly_drop += tearly_drop; // conntrack.errors + aicmp_error += ticmp_error; // conntrack.errors + aexpect_new += texpect_new; // conntrack.expect + aexpect_create += texpect_create; // conntrack.expect + aexpect_delete += texpect_delete; // conntrack.expect + asearch_restart += tsearch_restart; // conntrack.search + } + + RRDSET *st; + + // -------------------------------------------------------------------- + + if(do_sockets) { + st = rrdset_find(RRD_TYPE_NET_STAT_NETFILTER "." RRD_TYPE_NET_STAT_CONNTRACK "_sockets"); + if(!st) { + st = rrdset_create(RRD_TYPE_NET_STAT_NETFILTER, RRD_TYPE_NET_STAT_CONNTRACK "_sockets", NULL, RRD_TYPE_NET_STAT_CONNTRACK, NULL, "Connection Tracker Connections", "active connections", 1000, update_every, RRDSET_TYPE_LINE); + + rrddim_add(st, "connections", NULL, 1, 1, RRDDIM_ABSOLUTE); + } + else rrdset_next(st); + + rrddim_set(st, "connections", aentries); + rrdset_done(st); + } + + // -------------------------------------------------------------------- + + if(do_new) { + st = rrdset_find(RRD_TYPE_NET_STAT_NETFILTER "." RRD_TYPE_NET_STAT_CONNTRACK "_new"); + if(!st) { + st = rrdset_create(RRD_TYPE_NET_STAT_NETFILTER, RRD_TYPE_NET_STAT_CONNTRACK "_new", NULL, RRD_TYPE_NET_STAT_CONNTRACK, NULL, "Connection Tracker New Connections", "connections/s", 1001, update_every, RRDSET_TYPE_LINE); + + rrddim_add(st, "new", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "ignore", NULL, -1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "invalid", NULL, -1, 1, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); + + rrddim_set(st, "new", anew); + rrddim_set(st, "ignore", aignore); + rrddim_set(st, "invalid", ainvalid); + rrdset_done(st); + } + + // -------------------------------------------------------------------- + + if(do_changes) { + st = rrdset_find(RRD_TYPE_NET_STAT_NETFILTER "." RRD_TYPE_NET_STAT_CONNTRACK "_changes"); + if(!st) { + st = rrdset_create(RRD_TYPE_NET_STAT_NETFILTER, RRD_TYPE_NET_STAT_CONNTRACK "_changes", NULL, RRD_TYPE_NET_STAT_CONNTRACK, NULL, "Connection Tracker Changes", "changes/s", 1002, update_every, RRDSET_TYPE_LINE); + st->isdetail = 1; + + rrddim_add(st, "inserted", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "deleted", NULL, -1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "delete_list", NULL, -1, 1, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); + + rrddim_set(st, "inserted", ainsert); + rrddim_set(st, "deleted", adelete); + rrddim_set(st, "delete_list", adelete_list); + rrdset_done(st); + } + + // -------------------------------------------------------------------- + + if(do_expect) { + st = rrdset_find(RRD_TYPE_NET_STAT_NETFILTER "." RRD_TYPE_NET_STAT_CONNTRACK "_expect"); + if(!st) { + st = rrdset_create(RRD_TYPE_NET_STAT_NETFILTER, RRD_TYPE_NET_STAT_CONNTRACK "_expect", NULL, RRD_TYPE_NET_STAT_CONNTRACK, NULL, "Connection Tracker Expectations", "expectations/s", 1003, update_every, RRDSET_TYPE_LINE); + st->isdetail = 1; + + rrddim_add(st, "created", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "deleted", NULL, -1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "new", NULL, 1, 1, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); + + rrddim_set(st, "created", aexpect_create); + rrddim_set(st, "deleted", aexpect_delete); + rrddim_set(st, "new", aexpect_new); + rrdset_done(st); + } + + // -------------------------------------------------------------------- + + if(do_search) { + st = rrdset_find(RRD_TYPE_NET_STAT_NETFILTER "." RRD_TYPE_NET_STAT_CONNTRACK "_search"); + if(!st) { + st = rrdset_create(RRD_TYPE_NET_STAT_NETFILTER, RRD_TYPE_NET_STAT_CONNTRACK "_search", NULL, RRD_TYPE_NET_STAT_CONNTRACK, NULL, "Connection Tracker Searches", "searches/s", 1010, update_every, RRDSET_TYPE_LINE); + st->isdetail = 1; + + rrddim_add(st, "searched", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "restarted", NULL, -1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "found", NULL, 1, 1, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); + + rrddim_set(st, "searched", asearched); + rrddim_set(st, "restarted", asearch_restart); + rrddim_set(st, "found", afound); + rrdset_done(st); + } + + // -------------------------------------------------------------------- + + if(do_errors) { + st = rrdset_find(RRD_TYPE_NET_STAT_NETFILTER "." RRD_TYPE_NET_STAT_CONNTRACK "_errors"); + if(!st) { + st = rrdset_create(RRD_TYPE_NET_STAT_NETFILTER, RRD_TYPE_NET_STAT_CONNTRACK "_errors", NULL, RRD_TYPE_NET_STAT_CONNTRACK, NULL, "Connection Tracker Errors", "events/s", 1005, update_every, RRDSET_TYPE_LINE); + st->isdetail = 1; + + rrddim_add(st, "icmp_error", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "insert_failed", NULL, -1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "drop", NULL, -1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "early_drop", NULL, -1, 1, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); + + rrddim_set(st, "icmp_error", aicmp_error); + rrddim_set(st, "insert_failed", ainsert_failed); + rrddim_set(st, "drop", adrop); + rrddim_set(st, "early_drop", aearly_drop); + rrdset_done(st); + } + + return 0; } diff --git a/src/proc_net_stat_synproxy.c b/src/proc_net_stat_synproxy.c index 508b7d3b4..758c35dee 100644 --- a/src/proc_net_stat_synproxy.c +++ b/src/proc_net_stat_synproxy.c @@ -1,138 +1,127 @@ -#ifdef HAVE_CONFIG_H -#include <config.h> -#endif -#include <stdio.h> -#include <stdlib.h> - #include "common.h" -#include "appconfig.h" -#include "procfile.h" -#include "rrd.h" -#include "plugin_proc.h" -#include "log.h" -#define RRD_TYPE_NET_STAT_NETFILTER "netfilter" -#define RRD_TYPE_NET_STAT_SYNPROXY "synproxy" -#define RRD_TYPE_NET_STAT_SYNPROXY_LEN strlen(RRD_TYPE_NET_STAT_SYNPROXY) +#define RRD_TYPE_NET_STAT_NETFILTER "netfilter" +#define RRD_TYPE_NET_STAT_SYNPROXY "synproxy" +#define RRD_TYPE_NET_STAT_SYNPROXY_LEN strlen(RRD_TYPE_NET_STAT_SYNPROXY) int do_proc_net_stat_synproxy(int update_every, unsigned long long dt) { - static int do_entries = -1, do_cookies = -1, do_syns = -1, do_reopened = -1; - static procfile *ff = NULL; + static int do_entries = -1, do_cookies = -1, do_syns = -1, do_reopened = -1; + static procfile *ff = NULL; - if(do_entries == -1) do_entries = config_get_boolean_ondemand("plugin:proc:/proc/net/stat/synproxy", "SYNPROXY entries", CONFIG_ONDEMAND_ONDEMAND); - if(do_cookies == -1) do_cookies = config_get_boolean_ondemand("plugin:proc:/proc/net/stat/synproxy", "SYNPROXY cookies", CONFIG_ONDEMAND_ONDEMAND); - if(do_syns == -1) do_syns = config_get_boolean_ondemand("plugin:proc:/proc/net/stat/synproxy", "SYNPROXY SYN received", CONFIG_ONDEMAND_ONDEMAND); - if(do_reopened == -1) do_reopened = config_get_boolean_ondemand("plugin:proc:/proc/net/stat/synproxy", "SYNPROXY connections reopened", CONFIG_ONDEMAND_ONDEMAND); + if(do_entries == -1) do_entries = config_get_boolean_ondemand("plugin:proc:/proc/net/stat/synproxy", "SYNPROXY entries", CONFIG_ONDEMAND_ONDEMAND); + if(do_cookies == -1) do_cookies = config_get_boolean_ondemand("plugin:proc:/proc/net/stat/synproxy", "SYNPROXY cookies", CONFIG_ONDEMAND_ONDEMAND); + if(do_syns == -1) do_syns = config_get_boolean_ondemand("plugin:proc:/proc/net/stat/synproxy", "SYNPROXY SYN received", CONFIG_ONDEMAND_ONDEMAND); + if(do_reopened == -1) do_reopened = config_get_boolean_ondemand("plugin:proc:/proc/net/stat/synproxy", "SYNPROXY connections reopened", CONFIG_ONDEMAND_ONDEMAND); - if(dt) {}; + if(dt) {}; - if(!ff) { - char filename[FILENAME_MAX + 1]; - snprintfz(filename, FILENAME_MAX, "%s%s", global_host_prefix, "/proc/net/stat/synproxy"); - ff = procfile_open(config_get("plugin:proc:/proc/net/stat/synproxy", "filename to monitor", filename), " \t,:|", PROCFILE_FLAG_DEFAULT); - } - if(!ff) return 1; + if(!ff) { + char filename[FILENAME_MAX + 1]; + snprintfz(filename, FILENAME_MAX, "%s%s", global_host_prefix, "/proc/net/stat/synproxy"); + ff = procfile_open(config_get("plugin:proc:/proc/net/stat/synproxy", "filename to monitor", filename), " \t,:|", PROCFILE_FLAG_DEFAULT); + } + if(!ff) return 1; - ff = procfile_readall(ff); - if(!ff) return 0; // we return 0, so that we will retry to open it next time + ff = procfile_readall(ff); + if(!ff) return 0; // we return 0, so that we will retry to open it next time - // make sure we have 3 lines - unsigned long lines = procfile_lines(ff), l; - if(lines < 2) { - error("/proc/net/stat/synproxy has %d lines, expected no less than 2. Disabling it.", lines); - return 1; - } + // make sure we have 3 lines + size_t lines = procfile_lines(ff), l; + if(lines < 2) { + error("/proc/net/stat/synproxy has %zu lines, expected no less than 2. Disabling it.", lines); + return 1; + } - unsigned long long entries = 0, syn_received = 0, cookie_invalid = 0, cookie_valid = 0, cookie_retrans = 0, conn_reopened = 0; + unsigned long long entries = 0, syn_received = 0, cookie_invalid = 0, cookie_valid = 0, cookie_retrans = 0, conn_reopened = 0; - // synproxy gives its values per CPU - for(l = 1; l < lines ;l++) { - int words = procfile_linewords(ff, l); - if(words < 6) continue; + // synproxy gives its values per CPU + for(l = 1; l < lines ;l++) { + int words = procfile_linewords(ff, l); + if(words < 6) continue; - entries += strtoull(procfile_lineword(ff, l, 0), NULL, 16); - syn_received += strtoull(procfile_lineword(ff, l, 1), NULL, 16); - cookie_invalid += strtoull(procfile_lineword(ff, l, 2), NULL, 16); - cookie_valid += strtoull(procfile_lineword(ff, l, 3), NULL, 16); - cookie_retrans += strtoull(procfile_lineword(ff, l, 4), NULL, 16); - conn_reopened += strtoull(procfile_lineword(ff, l, 5), NULL, 16); - } + entries += strtoull(procfile_lineword(ff, l, 0), NULL, 16); + syn_received += strtoull(procfile_lineword(ff, l, 1), NULL, 16); + cookie_invalid += strtoull(procfile_lineword(ff, l, 2), NULL, 16); + cookie_valid += strtoull(procfile_lineword(ff, l, 3), NULL, 16); + cookie_retrans += strtoull(procfile_lineword(ff, l, 4), NULL, 16); + conn_reopened += strtoull(procfile_lineword(ff, l, 5), NULL, 16); + } - unsigned long long events = entries + syn_received + cookie_invalid + cookie_valid + cookie_retrans + conn_reopened; + unsigned long long events = entries + syn_received + cookie_invalid + cookie_valid + cookie_retrans + conn_reopened; - RRDSET *st; + RRDSET *st; - // -------------------------------------------------------------------- + // -------------------------------------------------------------------- - if((do_entries == CONFIG_ONDEMAND_ONDEMAND && events) || do_entries == CONFIG_ONDEMAND_YES) { - do_entries = CONFIG_ONDEMAND_YES; + if((do_entries == CONFIG_ONDEMAND_ONDEMAND && events) || do_entries == CONFIG_ONDEMAND_YES) { + do_entries = CONFIG_ONDEMAND_YES; - st = rrdset_find(RRD_TYPE_NET_STAT_NETFILTER "." RRD_TYPE_NET_STAT_SYNPROXY "_entries"); - if(!st) { - st = rrdset_create(RRD_TYPE_NET_STAT_NETFILTER, RRD_TYPE_NET_STAT_SYNPROXY "_entries", NULL, RRD_TYPE_NET_STAT_SYNPROXY, NULL, "SYNPROXY Entries Used", "entries", 1004, update_every, RRDSET_TYPE_LINE); + st = rrdset_find(RRD_TYPE_NET_STAT_NETFILTER "." RRD_TYPE_NET_STAT_SYNPROXY "_entries"); + if(!st) { + st = rrdset_create(RRD_TYPE_NET_STAT_NETFILTER, RRD_TYPE_NET_STAT_SYNPROXY "_entries", NULL, RRD_TYPE_NET_STAT_SYNPROXY, NULL, "SYNPROXY Entries Used", "entries", 1004, update_every, RRDSET_TYPE_LINE); - rrddim_add(st, "entries", NULL, 1, 1, RRDDIM_ABSOLUTE); - } - else rrdset_next(st); + rrddim_add(st, "entries", NULL, 1, 1, RRDDIM_ABSOLUTE); + } + else rrdset_next(st); - rrddim_set(st, "entries", entries); - rrdset_done(st); - } + rrddim_set(st, "entries", entries); + rrdset_done(st); + } - // -------------------------------------------------------------------- + // -------------------------------------------------------------------- - if((do_syns == CONFIG_ONDEMAND_ONDEMAND && events) || do_syns == CONFIG_ONDEMAND_YES) { - do_syns = CONFIG_ONDEMAND_YES; + if((do_syns == CONFIG_ONDEMAND_ONDEMAND && events) || do_syns == CONFIG_ONDEMAND_YES) { + do_syns = CONFIG_ONDEMAND_YES; - st = rrdset_find(RRD_TYPE_NET_STAT_NETFILTER "." RRD_TYPE_NET_STAT_SYNPROXY "_syn_received"); - if(!st) { - st = rrdset_create(RRD_TYPE_NET_STAT_NETFILTER, RRD_TYPE_NET_STAT_SYNPROXY "_syn_received", NULL, RRD_TYPE_NET_STAT_SYNPROXY, NULL, "SYNPROXY SYN Packets received", "SYN/s", 1001, update_every, RRDSET_TYPE_LINE); + st = rrdset_find(RRD_TYPE_NET_STAT_NETFILTER "." RRD_TYPE_NET_STAT_SYNPROXY "_syn_received"); + if(!st) { + st = rrdset_create(RRD_TYPE_NET_STAT_NETFILTER, RRD_TYPE_NET_STAT_SYNPROXY "_syn_received", NULL, RRD_TYPE_NET_STAT_SYNPROXY, NULL, "SYNPROXY SYN Packets received", "SYN/s", 1001, update_every, RRDSET_TYPE_LINE); - rrddim_add(st, "received", NULL, 1, 1, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); + rrddim_add(st, "received", NULL, 1, 1, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); - rrddim_set(st, "received", syn_received); - rrdset_done(st); - } + rrddim_set(st, "received", syn_received); + rrdset_done(st); + } - // -------------------------------------------------------------------- + // -------------------------------------------------------------------- - if((do_reopened == CONFIG_ONDEMAND_ONDEMAND && events) || do_reopened == CONFIG_ONDEMAND_YES) { - do_reopened = CONFIG_ONDEMAND_YES; + if((do_reopened == CONFIG_ONDEMAND_ONDEMAND && events) || do_reopened == CONFIG_ONDEMAND_YES) { + do_reopened = CONFIG_ONDEMAND_YES; - st = rrdset_find(RRD_TYPE_NET_STAT_NETFILTER "." RRD_TYPE_NET_STAT_SYNPROXY "_conn_reopened"); - if(!st) { - st = rrdset_create(RRD_TYPE_NET_STAT_NETFILTER, RRD_TYPE_NET_STAT_SYNPROXY "_conn_reopened", NULL, RRD_TYPE_NET_STAT_SYNPROXY, NULL, "SYNPROXY Connections Reopened", "connections/s", 1003, update_every, RRDSET_TYPE_LINE); + st = rrdset_find(RRD_TYPE_NET_STAT_NETFILTER "." RRD_TYPE_NET_STAT_SYNPROXY "_conn_reopened"); + if(!st) { + st = rrdset_create(RRD_TYPE_NET_STAT_NETFILTER, RRD_TYPE_NET_STAT_SYNPROXY "_conn_reopened", NULL, RRD_TYPE_NET_STAT_SYNPROXY, NULL, "SYNPROXY Connections Reopened", "connections/s", 1003, update_every, RRDSET_TYPE_LINE); - rrddim_add(st, "reopened", NULL, 1, 1, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); + rrddim_add(st, "reopened", NULL, 1, 1, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); - rrddim_set(st, "reopened", conn_reopened); - rrdset_done(st); - } + rrddim_set(st, "reopened", conn_reopened); + rrdset_done(st); + } - // -------------------------------------------------------------------- + // -------------------------------------------------------------------- - if((do_cookies == CONFIG_ONDEMAND_ONDEMAND && events) || do_cookies == CONFIG_ONDEMAND_YES) { - do_cookies = CONFIG_ONDEMAND_YES; + if((do_cookies == CONFIG_ONDEMAND_ONDEMAND && events) || do_cookies == CONFIG_ONDEMAND_YES) { + do_cookies = CONFIG_ONDEMAND_YES; - st = rrdset_find(RRD_TYPE_NET_STAT_NETFILTER "." RRD_TYPE_NET_STAT_SYNPROXY "_cookies"); - if(!st) { - st = rrdset_create(RRD_TYPE_NET_STAT_NETFILTER, RRD_TYPE_NET_STAT_SYNPROXY "_cookies", NULL, RRD_TYPE_NET_STAT_SYNPROXY, NULL, "SYNPROXY TCP Cookies", "cookies/s", 1002, update_every, RRDSET_TYPE_LINE); + st = rrdset_find(RRD_TYPE_NET_STAT_NETFILTER "." RRD_TYPE_NET_STAT_SYNPROXY "_cookies"); + if(!st) { + st = rrdset_create(RRD_TYPE_NET_STAT_NETFILTER, RRD_TYPE_NET_STAT_SYNPROXY "_cookies", NULL, RRD_TYPE_NET_STAT_SYNPROXY, NULL, "SYNPROXY TCP Cookies", "cookies/s", 1002, update_every, RRDSET_TYPE_LINE); - rrddim_add(st, "valid", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "invalid", NULL, -1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "retransmits", NULL, 1, 1, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); + rrddim_add(st, "valid", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "invalid", NULL, -1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "retransmits", NULL, 1, 1, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); - rrddim_set(st, "valid", cookie_valid); - rrddim_set(st, "invalid", cookie_invalid); - rrddim_set(st, "retransmits", cookie_retrans); - rrdset_done(st); - } + rrddim_set(st, "valid", cookie_valid); + rrddim_set(st, "invalid", cookie_invalid); + rrddim_set(st, "retransmits", cookie_retrans); + rrdset_done(st); + } - return 0; + return 0; } diff --git a/src/proc_self_mountinfo.c b/src/proc_self_mountinfo.c index 45630b4c0..51aea7aee 100644 --- a/src/proc_self_mountinfo.c +++ b/src/proc_self_mountinfo.c @@ -1,231 +1,242 @@ -#ifdef HAVE_CONFIG_H -#include <config.h> -#endif -#include <stdio.h> -#include <stdlib.h> -#include <string.h> -#include <sys/types.h> -#include <sys/stat.h> -#include <fcntl.h> - #include "common.h" -#include "log.h" -#include "appconfig.h" - -#include "proc_self_mountinfo.h" // find the mount info with the given major:minor // in the supplied linked list of mountinfo structures struct mountinfo *mountinfo_find(struct mountinfo *root, unsigned long major, unsigned long minor) { - struct mountinfo *mi; + struct mountinfo *mi; - for(mi = root; mi ; mi = mi->next) - if(mi->major == major && mi->minor == minor) - return mi; + for(mi = root; mi ; mi = mi->next) + if(mi->major == major && mi->minor == minor) + return mi; - return NULL; + return NULL; } // find the mount info with the given filesystem and mount_source // in the supplied linked list of mountinfo structures struct mountinfo *mountinfo_find_by_filesystem_mount_source(struct mountinfo *root, const char *filesystem, const char *mount_source) { - struct mountinfo *mi; - uint32_t filesystem_hash = simple_hash(filesystem), mount_source_hash = simple_hash(mount_source); - - for(mi = root; mi ; mi = mi->next) - if(mi->filesystem - && mi->mount_source - && mi->filesystem_hash == filesystem_hash - && mi->mount_source_hash == mount_source_hash - && !strcmp(mi->filesystem, filesystem) - && !strcmp(mi->mount_source, mount_source)) - return mi; - - return NULL; + struct mountinfo *mi; + uint32_t filesystem_hash = simple_hash(filesystem), mount_source_hash = simple_hash(mount_source); + + for(mi = root; mi ; mi = mi->next) + if(mi->filesystem + && mi->mount_source + && mi->filesystem_hash == filesystem_hash + && mi->mount_source_hash == mount_source_hash + && !strcmp(mi->filesystem, filesystem) + && !strcmp(mi->mount_source, mount_source)) + return mi; + + return NULL; } struct mountinfo *mountinfo_find_by_filesystem_super_option(struct mountinfo *root, const char *filesystem, const char *super_options) { - struct mountinfo *mi; - uint32_t filesystem_hash = simple_hash(filesystem); + struct mountinfo *mi; + uint32_t filesystem_hash = simple_hash(filesystem); - size_t solen = strlen(super_options); + size_t solen = strlen(super_options); - for(mi = root; mi ; mi = mi->next) - if(mi->filesystem - && mi->super_options - && mi->filesystem_hash == filesystem_hash - && !strcmp(mi->filesystem, filesystem)) { + for(mi = root; mi ; mi = mi->next) + if(mi->filesystem + && mi->super_options + && mi->filesystem_hash == filesystem_hash + && !strcmp(mi->filesystem, filesystem)) { - // super_options is a comma separated list - char *s = mi->super_options, *e; - while(*s) { - e = ++s; - while(*e && *e != ',') e++; + // super_options is a comma separated list + char *s = mi->super_options, *e; + while(*s) { + e = s + 1; + while(*e && *e != ',') e++; - size_t len = e - s; - if(len == solen && !strncmp(s, super_options, len)) - return mi; + size_t len = e - s; + if(len == solen && !strncmp(s, super_options, len)) + return mi; - if(*e == ',') s = ++e; - else s = e; - } - } + if(*e == ',') s = ++e; + else s = e; + } + } - return NULL; + return NULL; } // free a linked list of mountinfo structures void mountinfo_free(struct mountinfo *mi) { - if(unlikely(!mi)) - return; + if(unlikely(!mi)) + return; - if(likely(mi->next)) - mountinfo_free(mi->next); + if(likely(mi->next)) + mountinfo_free(mi->next); - if(mi->root) free(mi->root); - if(mi->mount_point) free(mi->mount_point); - if(mi->mount_options) free(mi->mount_options); + freez(mi->root); + freez(mi->mount_point); + freez(mi->mount_options); /* - if(mi->optional_fields_count) { - int i; - for(i = 0; i < mi->optional_fields_count ; i++) - free(*mi->optional_fields[i]); - } - free(mi->optional_fields); + if(mi->optional_fields_count) { + int i; + for(i = 0; i < mi->optional_fields_count ; i++) + free(*mi->optional_fields[i]); + } + free(mi->optional_fields); */ - free(mi->filesystem); - free(mi->mount_source); - free(mi->super_options); - free(mi); + freez(mi->filesystem); + freez(mi->mount_source); + freez(mi->super_options); + freez(mi); +} + +static char *strdupz_decoding_octal(const char *string) { + char *buffer = strdupz(string); + + char *d = buffer; + const char *s = string; + + while(*s) { + if(unlikely(*s == '\\')) { + s++; + if(likely(isdigit(*s) && isdigit(s[1]) && isdigit(s[2]))) { + char c = *s++ - '0'; + c <<= 3; + c |= *s++ - '0'; + c <<= 3; + c |= *s++ - '0'; + *d++ = c; + } + else *d++ = '_'; + } + else *d++ = *s++; + } + *d = '\0'; + + return buffer; } // read the whole mountinfo into a linked list struct mountinfo *mountinfo_read() { - procfile *ff = NULL; + procfile *ff = NULL; + + char filename[FILENAME_MAX + 1]; + snprintfz(filename, FILENAME_MAX, "%s/proc/self/mountinfo", global_host_prefix); + ff = procfile_open(filename, " \t", PROCFILE_FLAG_DEFAULT); + if(!ff) { + snprintfz(filename, FILENAME_MAX, "%s/proc/1/mountinfo", global_host_prefix); + ff = procfile_open(filename, " \t", PROCFILE_FLAG_DEFAULT); + if(!ff) return NULL; + } + + ff = procfile_readall(ff); + if(!ff) return NULL; - char filename[FILENAME_MAX + 1]; - snprintfz(filename, FILENAME_MAX, "%s/proc/self/mountinfo", global_host_prefix); - ff = procfile_open(filename, " \t", PROCFILE_FLAG_DEFAULT); - if(!ff) { - snprintfz(filename, FILENAME_MAX, "%s/proc/1/mountinfo", global_host_prefix); - ff = procfile_open(filename, " \t", PROCFILE_FLAG_DEFAULT); - if(!ff) return NULL; - } + struct mountinfo *root = NULL, *last = NULL, *mi = NULL; - ff = procfile_readall(ff); - if(!ff) return NULL; + unsigned long l, lines = procfile_lines(ff); + for(l = 0; l < lines ;l++) { + if(procfile_linewords(ff, l) < 5) + continue; - struct mountinfo *root = NULL, *last = NULL, *mi = NULL; + mi = mallocz(sizeof(struct mountinfo)); - unsigned long l, lines = procfile_lines(ff); - for(l = 0; l < lines ;l++) { - if(procfile_linewords(ff, l) < 5) - continue; + if(unlikely(!root)) + root = last = mi; + else + last->next = mi; - mi = malloc(sizeof(struct mountinfo)); - if(unlikely(!mi)) fatal("Cannot allocate memory for mountinfo"); + last = mi; + mi->next = NULL; - if(unlikely(!root)) - root = last = mi; - else - last->next = mi; + unsigned long w = 0; + mi->id = strtoul(procfile_lineword(ff, l, w), NULL, 10); w++; + mi->parentid = strtoul(procfile_lineword(ff, l, w), NULL, 10); w++; - last = mi; - mi->next = NULL; + char *major = procfile_lineword(ff, l, w), *minor; w++; + for(minor = major; *minor && *minor != ':' ;minor++) ; - unsigned long w = 0; - mi->id = strtoul(procfile_lineword(ff, l, w), NULL, 10); w++; - mi->parentid = strtoul(procfile_lineword(ff, l, w), NULL, 10); w++; + if(!*minor) { + error("Cannot parse major:minor on '%s' at line %lu of '%s'", major, l + 1, filename); + continue; + } - char *major = procfile_lineword(ff, l, w), *minor; w++; - for(minor = major; *minor && *minor != ':' ;minor++) ; - *minor = '\0'; - minor++; + *minor = '\0'; + minor++; - mi->major = strtoul(major, NULL, 10); - mi->minor = strtoul(minor, NULL, 10); + mi->major = strtoul(major, NULL, 10); + mi->minor = strtoul(minor, NULL, 10); - mi->root = strdup(procfile_lineword(ff, l, w)); w++; - if(unlikely(!mi->root)) fatal("Cannot allocate memory"); - mi->root_hash = simple_hash(mi->root); + mi->root = strdupz(procfile_lineword(ff, l, w)); w++; + mi->root_hash = simple_hash(mi->root); - mi->mount_point = strdup(procfile_lineword(ff, l, w)); w++; - if(unlikely(!mi->mount_point)) fatal("Cannot allocate memory"); - mi->mount_point_hash = simple_hash(mi->mount_point); + mi->mount_point = strdupz_decoding_octal(procfile_lineword(ff, l, w)); w++; + mi->mount_point_hash = simple_hash(mi->mount_point); - mi->mount_options = strdup(procfile_lineword(ff, l, w)); w++; - if(unlikely(!mi->mount_options)) fatal("Cannot allocate memory"); + mi->mount_options = strdupz(procfile_lineword(ff, l, w)); w++; - // count the optional fields + // count the optional fields /* - unsigned long wo = w; + unsigned long wo = w; */ - mi->optional_fields_count = 0; - char *s = procfile_lineword(ff, l, w); - while(*s && *s != '-') { - w++; - s = procfile_lineword(ff, l, w); - mi->optional_fields_count++; - } + mi->optional_fields_count = 0; + char *s = procfile_lineword(ff, l, w); + while(*s && *s != '-') { + w++; + s = procfile_lineword(ff, l, w); + mi->optional_fields_count++; + } /* - if(unlikely(mi->optional_fields_count)) { - // we have some optional fields - // read them into a new array of pointers; - - mi->optional_fields = malloc(mi->optional_fields_count * sizeof(char *)); - if(unlikely(!mi->optional_fields)) - fatal("Cannot allocate memory for %d mountinfo optional fields", mi->optional_fields_count); - - int i; - for(i = 0; i < mi->optional_fields_count ; i++) { - *mi->optional_fields[wo] = strdup(procfile_lineword(ff, l, w)); - if(!mi->optional_fields[wo]) fatal("Cannot allocate memory"); - wo++; - } - } - else - mi->optional_fields = NULL; + if(unlikely(mi->optional_fields_count)) { + // we have some optional fields + // read them into a new array of pointers; + + mi->optional_fields = malloc(mi->optional_fields_count * sizeof(char *)); + if(unlikely(!mi->optional_fields)) + fatal("Cannot allocate memory for %d mountinfo optional fields", mi->optional_fields_count); + + int i; + for(i = 0; i < mi->optional_fields_count ; i++) { + *mi->optional_fields[wo] = strdup(procfile_lineword(ff, l, w)); + if(!mi->optional_fields[wo]) fatal("Cannot allocate memory"); + wo++; + } + } + else + mi->optional_fields = NULL; */ - if(likely(*s == '-')) { - w++; + if(likely(*s == '-')) { + w++; - mi->filesystem = strdup(procfile_lineword(ff, l, w)); w++; - if(!mi->filesystem) fatal("Cannot allocate memory"); - mi->filesystem_hash = simple_hash(mi->filesystem); + mi->filesystem = strdupz(procfile_lineword(ff, l, w)); w++; + mi->filesystem_hash = simple_hash(mi->filesystem); - mi->mount_source = strdup(procfile_lineword(ff, l, w)); w++; - if(!mi->mount_source) fatal("Cannot allocate memory"); - mi->mount_source_hash = simple_hash(mi->mount_source); + mi->mount_source = strdupz(procfile_lineword(ff, l, w)); w++; + mi->mount_source_hash = simple_hash(mi->mount_source); - mi->super_options = strdup(procfile_lineword(ff, l, w)); w++; - if(!mi->super_options) fatal("Cannot allocate memory"); - } - else { - mi->filesystem = NULL; - mi->mount_source = NULL; - mi->super_options = NULL; - } + mi->super_options = strdupz(procfile_lineword(ff, l, w)); w++; + } + else { + mi->filesystem = NULL; + mi->mount_source = NULL; + mi->super_options = NULL; + } /* - info("MOUNTINFO: %u %u %u:%u root '%s', mount point '%s', mount options '%s', filesystem '%s', mount source '%s', super options '%s'", - mi->id, - mi->parentid, - mi->major, - mi->minor, - mi->root, - mi->mount_point, - mi->mount_options, - mi->filesystem, - mi->mount_source, - mi->super_options - ); + info("MOUNTINFO: %u %u %u:%u root '%s', mount point '%s', mount options '%s', filesystem '%s', mount source '%s', super options '%s'", + mi->id, + mi->parentid, + mi->major, + mi->minor, + mi->root, + mi->mount_point, + mi->mount_options, + mi->filesystem, + mi->mount_source, + mi->super_options + ); */ - } + } - procfile_close(ff); - return root; + procfile_close(ff); + return root; } diff --git a/src/proc_self_mountinfo.h b/src/proc_self_mountinfo.h index 51712a58a..c2d9688c1 100644 --- a/src/proc_self_mountinfo.h +++ b/src/proc_self_mountinfo.h @@ -1,35 +1,33 @@ -#include "procfile.h" - #ifndef NETDATA_PROC_SELF_MOUNTINFO_H #define NETDATA_PROC_SELF_MOUNTINFO_H 1 struct mountinfo { - long id; // mount ID: unique identifier of the mount (may be reused after umount(2)). - long parentid; // parent ID: ID of parent mount (or of self for the top of the mount tree). - unsigned long major; // major:minor: value of st_dev for files on filesystem (see stat(2)). - unsigned long minor; + long id; // mount ID: unique identifier of the mount (may be reused after umount(2)). + long parentid; // parent ID: ID of parent mount (or of self for the top of the mount tree). + unsigned long major; // major:minor: value of st_dev for files on filesystem (see stat(2)). + unsigned long minor; - char *root; // root: root of the mount within the filesystem. - uint32_t root_hash; + char *root; // root: root of the mount within the filesystem. + uint32_t root_hash; - char *mount_point; // mount point: mount point relative to the process's root. - uint32_t mount_point_hash; + char *mount_point; // mount point: mount point relative to the process's root. + uint32_t mount_point_hash; - char *mount_options; // mount options: per-mount options. + char *mount_options; // mount options: per-mount options. - int optional_fields_count; + int optional_fields_count; /* - char ***optional_fields; // optional fields: zero or more fields of the form "tag[:value]". + char ***optional_fields; // optional fields: zero or more fields of the form "tag[:value]". */ - char *filesystem; // filesystem type: name of filesystem in the form "type[.subtype]". - uint32_t filesystem_hash; + char *filesystem; // filesystem type: name of filesystem in the form "type[.subtype]". + uint32_t filesystem_hash; - char *mount_source; // mount source: filesystem-specific information or "none". - uint32_t mount_source_hash; + char *mount_source; // mount source: filesystem-specific information or "none". + uint32_t mount_source_hash; - char *super_options; // super options: per-superblock options. + char *super_options; // super options: per-superblock options. - struct mountinfo *next; + struct mountinfo *next; }; extern struct mountinfo *mountinfo_find(struct mountinfo *root, unsigned long major, unsigned long minor); diff --git a/src/proc_softirqs.c b/src/proc_softirqs.c index 96b5d3d30..a5165040a 100644 --- a/src/proc_softirqs.c +++ b/src/proc_softirqs.c @@ -1,26 +1,13 @@ -#ifdef HAVE_CONFIG_H -#include <config.h> -#endif -#include <stdio.h> -#include <stdlib.h> -#include <string.h> -#include <ctype.h> - #include "common.h" -#include "appconfig.h" -#include "procfile.h" -#include "rrd.h" -#include "plugin_proc.h" -#include "log.h" #define MAX_INTERRUPT_NAME 50 struct interrupt { - int used; - char *id; - char name[MAX_INTERRUPT_NAME + 1]; - unsigned long long total; - unsigned long long value[]; + int used; + char *id; + char name[MAX_INTERRUPT_NAME + 1]; + unsigned long long total; + unsigned long long value[]; }; // since each interrupt is variable in size @@ -31,160 +18,157 @@ struct interrupt { #define irrindex(base, line, cpus) ((struct interrupt *)&((char *)(base))[line * recordsize(cpus)]) static inline struct interrupt *get_interrupts_array(int lines, int cpus) { - static struct interrupt *irrs = NULL; - static int allocated = 0; - - if(lines < allocated) return irrs; - else { - irrs = (struct interrupt *)realloc(irrs, lines * recordsize(cpus)); - if(!irrs) - fatal("Cannot allocate memory for %d interrupts", lines); + static struct interrupt *irrs = NULL; + static int allocated = 0; - allocated = lines; - } + if(lines < allocated) return irrs; + else { + irrs = (struct interrupt *)reallocz(irrs, lines * recordsize(cpus)); + allocated = lines; + } - return irrs; + return irrs; } int do_proc_softirqs(int update_every, unsigned long long dt) { - static procfile *ff = NULL; - static int cpus = -1, do_per_core = -1; + static procfile *ff = NULL; + static int cpus = -1, do_per_core = -1; - struct interrupt *irrs = NULL; - - if(dt) {}; + struct interrupt *irrs = NULL; + + if(dt) {}; - if(do_per_core == -1) do_per_core = config_get_boolean("plugin:proc:/proc/softirqs", "interrupts per core", 1); - - if(!ff) { - char filename[FILENAME_MAX + 1]; - snprintfz(filename, FILENAME_MAX, "%s%s", global_host_prefix, "/proc/softirqs"); - ff = procfile_open(config_get("plugin:proc:/proc/softirqs", "filename to monitor", filename), " \t", PROCFILE_FLAG_DEFAULT); - } - if(!ff) return 1; - - ff = procfile_readall(ff); - if(!ff) return 0; // we return 0, so that we will retry to open it next time - - uint32_t lines = procfile_lines(ff), l; - uint32_t words = procfile_linewords(ff, 0), w; - - if(!lines) { - error("Cannot read /proc/softirqs, zero lines reported."); - return 1; - } - - // find how many CPUs are there - if(cpus == -1) { - cpus = 0; - for(w = 0; w < words ; w++) { - if(strncmp(procfile_lineword(ff, 0, w), "CPU", 3) == 0) - cpus++; - } - } - - if(!cpus) { - error("PLUGIN: PROC_SOFTIRQS: Cannot find the number of CPUs in /proc/softirqs"); - return 1; - } - - // allocate the size we need; - irrs = get_interrupts_array(lines, cpus); - irrs[0].used = 0; - - // loop through all lines - for(l = 1; l < lines ;l++) { - struct interrupt *irr = irrindex(irrs, l, cpus); - irr->used = 0; - irr->total = 0; - - words = procfile_linewords(ff, l); - if(!words) continue; - - irr->id = procfile_lineword(ff, l, 0); - if(!irr->id || !irr->id[0]) continue; - - int idlen = strlen(irr->id); - if(irr->id[idlen - 1] == ':') - irr->id[idlen - 1] = '\0'; - - int c; - for(c = 0; c < cpus ;c++) { - if((c + 1) < (int)words) - irr->value[c] = strtoull(procfile_lineword(ff, l, (uint32_t)(c + 1)), NULL, 10); - else - irr->value[c] = 0; - - irr->total += irr->value[c]; - } - - strncpyz(irr->name, irr->id, MAX_INTERRUPT_NAME); - - irr->used = 1; - } - - RRDSET *st; - - // -------------------------------------------------------------------- - - st = rrdset_find_bytype("system", "softirqs"); - if(!st) { - st = rrdset_create("system", "softirqs", NULL, "softirqs", NULL, "System softirqs", "softirqs/s", 950, update_every, RRDSET_TYPE_STACKED); - - for(l = 0; l < lines ;l++) { - struct interrupt *irr = irrindex(irrs, l, cpus); - if(!irr->used) continue; - rrddim_add(st, irr->id, irr->name, 1, 1, RRDDIM_INCREMENTAL); - } - } - else rrdset_next(st); - - for(l = 0; l < lines ;l++) { - struct interrupt *irr = irrindex(irrs, l, cpus); - if(!irr->used) continue; - rrddim_set(st, irr->id, irr->total); - } - rrdset_done(st); - - if(do_per_core) { - int c; - - for(c = 0; c < cpus ; c++) { - char id[256+1]; - snprintfz(id, 256, "cpu%d_softirqs", c); - - st = rrdset_find_bytype("cpu", id); - if(!st) { - // find if everything is zero - unsigned long long core_sum = 0 ; - for(l = 0; l < lines ;l++) { - struct interrupt *irr = irrindex(irrs, l, cpus); - if(!irr->used) continue; - core_sum += irr->value[c]; - } - if(core_sum == 0) continue; // try next core - - char name[256+1], title[256+1]; - snprintfz(name, 256, "cpu%d_softirqs", c); - snprintfz(title, 256, "CPU%d softirqs", c); - st = rrdset_create("cpu", id, name, "softirqs", "cpu.softirqs", title, "softirqs/s", 3000 + c, update_every, RRDSET_TYPE_STACKED); - - for(l = 0; l < lines ;l++) { - struct interrupt *irr = irrindex(irrs, l, cpus); - if(!irr->used) continue; - rrddim_add(st, irr->id, irr->name, 1, 1, RRDDIM_INCREMENTAL); - } - } - else rrdset_next(st); - - for(l = 0; l < lines ;l++) { - struct interrupt *irr = irrindex(irrs, l, cpus); - if(!irr->used) continue; - rrddim_set(st, irr->id, irr->value[c]); - } - rrdset_done(st); - } - } - - return 0; + if(do_per_core == -1) do_per_core = config_get_boolean("plugin:proc:/proc/softirqs", "interrupts per core", 1); + + if(!ff) { + char filename[FILENAME_MAX + 1]; + snprintfz(filename, FILENAME_MAX, "%s%s", global_host_prefix, "/proc/softirqs"); + ff = procfile_open(config_get("plugin:proc:/proc/softirqs", "filename to monitor", filename), " \t", PROCFILE_FLAG_DEFAULT); + } + if(!ff) return 1; + + ff = procfile_readall(ff); + if(!ff) return 0; // we return 0, so that we will retry to open it next time + + uint32_t lines = procfile_lines(ff), l; + uint32_t words = procfile_linewords(ff, 0), w; + + if(!lines) { + error("Cannot read /proc/softirqs, zero lines reported."); + return 1; + } + + // find how many CPUs are there + if(cpus == -1) { + cpus = 0; + for(w = 0; w < words ; w++) { + if(strncmp(procfile_lineword(ff, 0, w), "CPU", 3) == 0) + cpus++; + } + } + + if(!cpus) { + error("PLUGIN: PROC_SOFTIRQS: Cannot find the number of CPUs in /proc/softirqs"); + return 1; + } + + // allocate the size we need; + irrs = get_interrupts_array(lines, cpus); + irrs[0].used = 0; + + // loop through all lines + for(l = 1; l < lines ;l++) { + struct interrupt *irr = irrindex(irrs, l, cpus); + irr->used = 0; + irr->total = 0; + + words = procfile_linewords(ff, l); + if(!words) continue; + + irr->id = procfile_lineword(ff, l, 0); + if(!irr->id || !irr->id[0]) continue; + + int idlen = strlen(irr->id); + if(irr->id[idlen - 1] == ':') + irr->id[idlen - 1] = '\0'; + + int c; + for(c = 0; c < cpus ;c++) { + if((c + 1) < (int)words) + irr->value[c] = strtoull(procfile_lineword(ff, l, (uint32_t)(c + 1)), NULL, 10); + else + irr->value[c] = 0; + + irr->total += irr->value[c]; + } + + strncpyz(irr->name, irr->id, MAX_INTERRUPT_NAME); + + irr->used = 1; + } + + RRDSET *st; + + // -------------------------------------------------------------------- + + st = rrdset_find_bytype("system", "softirqs"); + if(!st) { + st = rrdset_create("system", "softirqs", NULL, "softirqs", NULL, "System softirqs", "softirqs/s", 950, update_every, RRDSET_TYPE_STACKED); + + for(l = 0; l < lines ;l++) { + struct interrupt *irr = irrindex(irrs, l, cpus); + if(!irr->used) continue; + rrddim_add(st, irr->id, irr->name, 1, 1, RRDDIM_INCREMENTAL); + } + } + else rrdset_next(st); + + for(l = 0; l < lines ;l++) { + struct interrupt *irr = irrindex(irrs, l, cpus); + if(!irr->used) continue; + rrddim_set(st, irr->id, irr->total); + } + rrdset_done(st); + + if(do_per_core) { + int c; + + for(c = 0; c < cpus ; c++) { + char id[256+1]; + snprintfz(id, 256, "cpu%d_softirqs", c); + + st = rrdset_find_bytype("cpu", id); + if(!st) { + // find if everything is zero + unsigned long long core_sum = 0 ; + for(l = 0; l < lines ;l++) { + struct interrupt *irr = irrindex(irrs, l, cpus); + if(!irr->used) continue; + core_sum += irr->value[c]; + } + if(core_sum == 0) continue; // try next core + + char name[256+1], title[256+1]; + snprintfz(name, 256, "cpu%d_softirqs", c); + snprintfz(title, 256, "CPU%d softirqs", c); + st = rrdset_create("cpu", id, name, "softirqs", "cpu.softirqs", title, "softirqs/s", 3000 + c, update_every, RRDSET_TYPE_STACKED); + + for(l = 0; l < lines ;l++) { + struct interrupt *irr = irrindex(irrs, l, cpus); + if(!irr->used) continue; + rrddim_add(st, irr->id, irr->name, 1, 1, RRDDIM_INCREMENTAL); + } + } + else rrdset_next(st); + + for(l = 0; l < lines ;l++) { + struct interrupt *irr = irrindex(irrs, l, cpus); + if(!irr->used) continue; + rrddim_set(st, irr->id, irr->value[c]); + } + rrdset_done(st); + } + } + + return 0; } diff --git a/src/proc_stat.c b/src/proc_stat.c index 154ba167d..88cb820b3 100644 --- a/src/proc_stat.c +++ b/src/proc_stat.c @@ -1,204 +1,211 @@ -#ifdef HAVE_CONFIG_H -#include <config.h> -#endif -#include <stdio.h> -#include <stdlib.h> -#include <string.h> - #include "common.h" -#include "log.h" -#include "appconfig.h" -#include "procfile.h" -#include "rrd.h" -#include "plugin_proc.h" - -#define RRD_TYPE_STAT "cpu" -#define RRD_TYPE_STAT_LEN strlen(RRD_TYPE_STAT) int do_proc_stat(int update_every, unsigned long long dt) { - static procfile *ff = NULL; - static int do_cpu = -1, do_cpu_cores = -1, do_interrupts = -1, do_context = -1, do_forks = -1, do_processes = -1; - - if(do_cpu == -1) do_cpu = config_get_boolean("plugin:proc:/proc/stat", "cpu utilization", 1); - if(do_cpu_cores == -1) do_cpu_cores = config_get_boolean("plugin:proc:/proc/stat", "per cpu core utilization", 1); - if(do_interrupts == -1) do_interrupts = config_get_boolean("plugin:proc:/proc/stat", "cpu interrupts", 1); - if(do_context == -1) do_context = config_get_boolean("plugin:proc:/proc/stat", "context switches", 1); - if(do_forks == -1) do_forks = config_get_boolean("plugin:proc:/proc/stat", "processes started", 1); - if(do_processes == -1) do_processes = config_get_boolean("plugin:proc:/proc/stat", "processes running", 1); - - if(dt) {}; - - if(!ff) { - char filename[FILENAME_MAX + 1]; - snprintfz(filename, FILENAME_MAX, "%s%s", global_host_prefix, "/proc/stat"); - ff = procfile_open(config_get("plugin:proc:/proc/stat", "filename to monitor", filename), " \t:", PROCFILE_FLAG_DEFAULT); - } - if(!ff) return 1; - - ff = procfile_readall(ff); - if(!ff) return 0; // we return 0, so that we will retry to open it next time - - uint32_t lines = procfile_lines(ff), l; - uint32_t words; - - unsigned long long processes = 0, running = 0 , blocked = 0; - RRDSET *st; - - for(l = 0; l < lines ;l++) { - if(strncmp(procfile_lineword(ff, l, 0), "cpu", 3) == 0) { - words = procfile_linewords(ff, l); - if(words < 9) { - error("Cannot read /proc/stat cpu line. Expected 9 params, read %d.", words); - continue; - } - - char *id; - unsigned long long user = 0, nice = 0, system = 0, idle = 0, iowait = 0, irq = 0, softirq = 0, steal = 0, guest = 0, guest_nice = 0; - - id = procfile_lineword(ff, l, 0); - user = strtoull(procfile_lineword(ff, l, 1), NULL, 10); - nice = strtoull(procfile_lineword(ff, l, 2), NULL, 10); - system = strtoull(procfile_lineword(ff, l, 3), NULL, 10); - idle = strtoull(procfile_lineword(ff, l, 4), NULL, 10); - iowait = strtoull(procfile_lineword(ff, l, 5), NULL, 10); - irq = strtoull(procfile_lineword(ff, l, 6), NULL, 10); - softirq = strtoull(procfile_lineword(ff, l, 7), NULL, 10); - steal = strtoull(procfile_lineword(ff, l, 8), NULL, 10); - if(words >= 10) guest = strtoull(procfile_lineword(ff, l, 9), NULL, 10); - if(words >= 11) guest_nice = strtoull(procfile_lineword(ff, l, 10), NULL, 10); - - char *title = "Core utilization"; - char *type = RRD_TYPE_STAT; - char *context = "cpu.cpu"; - char *family = "utilization"; - long priority = 1000; - int isthistotal = 0; - - if(strcmp(id, "cpu") == 0) { - isthistotal = 1; - type = "system"; - title = "Total CPU utilization"; - context = "system.cpu"; - family = id; - priority = 100; - } - - if((isthistotal && do_cpu) || (!isthistotal && do_cpu_cores)) { - st = rrdset_find_bytype(type, id); - if(!st) { - st = rrdset_create(type, id, NULL, family, context, title, "percentage", priority, update_every, RRDSET_TYPE_STACKED); - - long multiplier = 1; - long divisor = 1; // sysconf(_SC_CLK_TCK); - - rrddim_add(st, "guest_nice", NULL, multiplier, divisor, RRDDIM_PCENT_OVER_DIFF_TOTAL); - rrddim_add(st, "guest", NULL, multiplier, divisor, RRDDIM_PCENT_OVER_DIFF_TOTAL); - rrddim_add(st, "steal", NULL, multiplier, divisor, RRDDIM_PCENT_OVER_DIFF_TOTAL); - rrddim_add(st, "softirq", NULL, multiplier, divisor, RRDDIM_PCENT_OVER_DIFF_TOTAL); - rrddim_add(st, "irq", NULL, multiplier, divisor, RRDDIM_PCENT_OVER_DIFF_TOTAL); - rrddim_add(st, "user", NULL, multiplier, divisor, RRDDIM_PCENT_OVER_DIFF_TOTAL); - rrddim_add(st, "system", NULL, multiplier, divisor, RRDDIM_PCENT_OVER_DIFF_TOTAL); - rrddim_add(st, "nice", NULL, multiplier, divisor, RRDDIM_PCENT_OVER_DIFF_TOTAL); - rrddim_add(st, "iowait", NULL, multiplier, divisor, RRDDIM_PCENT_OVER_DIFF_TOTAL); - - rrddim_add(st, "idle", NULL, multiplier, divisor, RRDDIM_PCENT_OVER_DIFF_TOTAL); - rrddim_hide(st, "idle"); - } - else rrdset_next(st); - - rrddim_set(st, "user", user); - rrddim_set(st, "nice", nice); - rrddim_set(st, "system", system); - rrddim_set(st, "idle", idle); - rrddim_set(st, "iowait", iowait); - rrddim_set(st, "irq", irq); - rrddim_set(st, "softirq", softirq); - rrddim_set(st, "steal", steal); - rrddim_set(st, "guest", guest); - rrddim_set(st, "guest_nice", guest_nice); - rrdset_done(st); - } - } - else if(strcmp(procfile_lineword(ff, l, 0), "intr") == 0) { - unsigned long long value = strtoull(procfile_lineword(ff, l, 1), NULL, 10); - - // -------------------------------------------------------------------- - - if(do_interrupts) { - st = rrdset_find_bytype("system", "intr"); - if(!st) { - st = rrdset_create("system", "intr", NULL, "interrupts", NULL, "CPU Interrupts", "interrupts/s", 900, update_every, RRDSET_TYPE_LINE); - st->isdetail = 1; - - rrddim_add(st, "interrupts", NULL, 1, 1, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); - - rrddim_set(st, "interrupts", value); - rrdset_done(st); - } - } - else if(strcmp(procfile_lineword(ff, l, 0), "ctxt") == 0) { - unsigned long long value = strtoull(procfile_lineword(ff, l, 1), NULL, 10); - - // -------------------------------------------------------------------- - - if(do_context) { - st = rrdset_find_bytype("system", "ctxt"); - if(!st) { - st = rrdset_create("system", "ctxt", NULL, "processes", NULL, "CPU Context Switches", "context switches/s", 800, update_every, RRDSET_TYPE_LINE); - - rrddim_add(st, "switches", NULL, 1, 1, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); - - rrddim_set(st, "switches", value); - rrdset_done(st); - } - } - else if(!processes && strcmp(procfile_lineword(ff, l, 0), "processes") == 0) { - processes = strtoull(procfile_lineword(ff, l, 1), NULL, 10); - } - else if(!running && strcmp(procfile_lineword(ff, l, 0), "procs_running") == 0) { - running = strtoull(procfile_lineword(ff, l, 1), NULL, 10); - } - else if(!blocked && strcmp(procfile_lineword(ff, l, 0), "procs_blocked") == 0) { - blocked = strtoull(procfile_lineword(ff, l, 1), NULL, 10); - } - } - - // -------------------------------------------------------------------- - - if(do_forks) { - st = rrdset_find_bytype("system", "forks"); - if(!st) { - st = rrdset_create("system", "forks", NULL, "processes", NULL, "Started Processes", "processes/s", 700, update_every, RRDSET_TYPE_LINE); - st->isdetail = 1; - - rrddim_add(st, "started", NULL, 1, 1, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); - - rrddim_set(st, "started", processes); - rrdset_done(st); - } - - // -------------------------------------------------------------------- - - if(do_processes) { - st = rrdset_find_bytype("system", "processes"); - if(!st) { - st = rrdset_create("system", "processes", NULL, "processes", NULL, "System Processes", "processes", 600, update_every, RRDSET_TYPE_LINE); - - rrddim_add(st, "running", NULL, 1, 1, RRDDIM_ABSOLUTE); - rrddim_add(st, "blocked", NULL, -1, 1, RRDDIM_ABSOLUTE); - } - else rrdset_next(st); - - rrddim_set(st, "running", running); - rrddim_set(st, "blocked", blocked); - rrdset_done(st); - } - - return 0; + (void)dt; + + static procfile *ff = NULL; + static int do_cpu = -1, do_cpu_cores = -1, do_interrupts = -1, do_context = -1, do_forks = -1, do_processes = -1; + static uint32_t hash_intr, hash_ctxt, hash_processes, hash_procs_running, hash_procs_blocked; + + if(unlikely(do_cpu == -1)) { + do_cpu = config_get_boolean("plugin:proc:/proc/stat", "cpu utilization", 1); + do_cpu_cores = config_get_boolean("plugin:proc:/proc/stat", "per cpu core utilization", 1); + do_interrupts = config_get_boolean("plugin:proc:/proc/stat", "cpu interrupts", 1); + do_context = config_get_boolean("plugin:proc:/proc/stat", "context switches", 1); + do_forks = config_get_boolean("plugin:proc:/proc/stat", "processes started", 1); + do_processes = config_get_boolean("plugin:proc:/proc/stat", "processes running", 1); + + hash_intr = simple_hash("intr"); + hash_ctxt = simple_hash("ctxt"); + hash_processes = simple_hash("processes"); + hash_procs_running = simple_hash("procs_running"); + hash_procs_blocked = simple_hash("procs_blocked"); + } + + if(unlikely(!ff)) { + char filename[FILENAME_MAX + 1]; + snprintfz(filename, FILENAME_MAX, "%s%s", global_host_prefix, "/proc/stat"); + ff = procfile_open(config_get("plugin:proc:/proc/stat", "filename to monitor", filename), " \t:", PROCFILE_FLAG_DEFAULT); + if(unlikely(!ff)) return 1; + } + + ff = procfile_readall(ff); + if(unlikely(!ff)) return 0; // we return 0, so that we will retry to open it next time + + uint32_t lines = procfile_lines(ff), l; + uint32_t words; + + unsigned long long processes = 0, running = 0 , blocked = 0; + RRDSET *st; + + for(l = 0; l < lines ;l++) { + char *row_key = procfile_lineword(ff, l, 0); + uint32_t hash = simple_hash(row_key); + + // faster strncmp(row_key, "cpu", 3) == 0 + if(likely(row_key[0] == 'c' && row_key[1] == 'p' && row_key[2] == 'u')) { + words = procfile_linewords(ff, l); + if(unlikely(words < 9)) { + error("Cannot read /proc/stat cpu line. Expected 9 params, read %u.", words); + continue; + } + + char *id; + unsigned long long user = 0, nice = 0, system = 0, idle = 0, iowait = 0, irq = 0, softirq = 0, steal = 0, guest = 0, guest_nice = 0; + + id = row_key; + user = strtoull(procfile_lineword(ff, l, 1), NULL, 10); + nice = strtoull(procfile_lineword(ff, l, 2), NULL, 10); + system = strtoull(procfile_lineword(ff, l, 3), NULL, 10); + idle = strtoull(procfile_lineword(ff, l, 4), NULL, 10); + iowait = strtoull(procfile_lineword(ff, l, 5), NULL, 10); + irq = strtoull(procfile_lineword(ff, l, 6), NULL, 10); + softirq = strtoull(procfile_lineword(ff, l, 7), NULL, 10); + steal = strtoull(procfile_lineword(ff, l, 8), NULL, 10); + + guest = strtoull(procfile_lineword(ff, l, 9), NULL, 10); + user -= guest; + + guest_nice = strtoull(procfile_lineword(ff, l, 10), NULL, 10); + nice -= guest_nice; + + char *title, *type, *context, *family; + long priority; + int isthistotal; + + if(unlikely(strcmp(id, "cpu")) == 0) { + title = "Total CPU utilization"; + type = "system"; + context = "system.cpu"; + family = id; + priority = 100; + isthistotal = 1; + } + else { + title = "Core utilization"; + type = "cpu"; + context = "cpu.cpu"; + family = "utilization"; + priority = 1000; + isthistotal = 0; + } + + if(likely((isthistotal && do_cpu) || (!isthistotal && do_cpu_cores))) { + st = rrdset_find_bytype(type, id); + if(unlikely(!st)) { + st = rrdset_create(type, id, NULL, family, context, title, "percentage", priority, update_every, RRDSET_TYPE_STACKED); + + long multiplier = 1; + long divisor = 1; // sysconf(_SC_CLK_TCK); + + rrddim_add(st, "guest_nice", NULL, multiplier, divisor, RRDDIM_PCENT_OVER_DIFF_TOTAL); + rrddim_add(st, "guest", NULL, multiplier, divisor, RRDDIM_PCENT_OVER_DIFF_TOTAL); + rrddim_add(st, "steal", NULL, multiplier, divisor, RRDDIM_PCENT_OVER_DIFF_TOTAL); + rrddim_add(st, "softirq", NULL, multiplier, divisor, RRDDIM_PCENT_OVER_DIFF_TOTAL); + rrddim_add(st, "irq", NULL, multiplier, divisor, RRDDIM_PCENT_OVER_DIFF_TOTAL); + rrddim_add(st, "user", NULL, multiplier, divisor, RRDDIM_PCENT_OVER_DIFF_TOTAL); + rrddim_add(st, "system", NULL, multiplier, divisor, RRDDIM_PCENT_OVER_DIFF_TOTAL); + rrddim_add(st, "nice", NULL, multiplier, divisor, RRDDIM_PCENT_OVER_DIFF_TOTAL); + rrddim_add(st, "iowait", NULL, multiplier, divisor, RRDDIM_PCENT_OVER_DIFF_TOTAL); + + rrddim_add(st, "idle", NULL, multiplier, divisor, RRDDIM_PCENT_OVER_DIFF_TOTAL); + rrddim_hide(st, "idle"); + } + else rrdset_next(st); + + rrddim_set(st, "user", user); + rrddim_set(st, "nice", nice); + rrddim_set(st, "system", system); + rrddim_set(st, "idle", idle); + rrddim_set(st, "iowait", iowait); + rrddim_set(st, "irq", irq); + rrddim_set(st, "softirq", softirq); + rrddim_set(st, "steal", steal); + rrddim_set(st, "guest", guest); + rrddim_set(st, "guest_nice", guest_nice); + rrdset_done(st); + } + } + else if(hash == hash_intr && strcmp(row_key, "intr") == 0) { + unsigned long long value = strtoull(procfile_lineword(ff, l, 1), NULL, 10); + + // -------------------------------------------------------------------- + + if(likely(do_interrupts)) { + st = rrdset_find_bytype("system", "intr"); + if(unlikely(!st)) { + st = rrdset_create("system", "intr", NULL, "interrupts", NULL, "CPU Interrupts", "interrupts/s", 900, update_every, RRDSET_TYPE_LINE); + st->isdetail = 1; + + rrddim_add(st, "interrupts", NULL, 1, 1, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); + + rrddim_set(st, "interrupts", value); + rrdset_done(st); + } + } + else if(hash == hash_ctxt && strcmp(row_key, "ctxt") == 0) { + unsigned long long value = strtoull(procfile_lineword(ff, l, 1), NULL, 10); + + // -------------------------------------------------------------------- + + if(likely(do_context)) { + st = rrdset_find_bytype("system", "ctxt"); + if(unlikely(!st)) { + st = rrdset_create("system", "ctxt", NULL, "processes", NULL, "CPU Context Switches", "context switches/s", 800, update_every, RRDSET_TYPE_LINE); + + rrddim_add(st, "switches", NULL, 1, 1, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); + + rrddim_set(st, "switches", value); + rrdset_done(st); + } + } + else if(hash == hash_processes && !processes && strcmp(row_key, "processes") == 0) { + processes = strtoull(procfile_lineword(ff, l, 1), NULL, 10); + } + else if(hash == hash_procs_running && !running && strcmp(row_key, "procs_running") == 0) { + running = strtoull(procfile_lineword(ff, l, 1), NULL, 10); + } + else if(hash == hash_procs_blocked && !blocked && strcmp(row_key, "procs_blocked") == 0) { + blocked = strtoull(procfile_lineword(ff, l, 1), NULL, 10); + } + } + + // -------------------------------------------------------------------- + + if(likely(do_forks)) { + st = rrdset_find_bytype("system", "forks"); + if(unlikely(!st)) { + st = rrdset_create("system", "forks", NULL, "processes", NULL, "Started Processes", "processes/s", 700, update_every, RRDSET_TYPE_LINE); + st->isdetail = 1; + + rrddim_add(st, "started", NULL, 1, 1, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); + + rrddim_set(st, "started", processes); + rrdset_done(st); + } + + // -------------------------------------------------------------------- + + if(likely(do_processes)) { + st = rrdset_find_bytype("system", "processes"); + if(unlikely(!st)) { + st = rrdset_create("system", "processes", NULL, "processes", NULL, "System Processes", "processes", 600, update_every, RRDSET_TYPE_LINE); + + rrddim_add(st, "running", NULL, 1, 1, RRDDIM_ABSOLUTE); + rrddim_add(st, "blocked", NULL, -1, 1, RRDDIM_ABSOLUTE); + } + else rrdset_next(st); + + rrddim_set(st, "running", running); + rrddim_set(st, "blocked", blocked); + rrdset_done(st); + } + + return 0; } diff --git a/src/proc_sys_kernel_random_entropy_avail.c b/src/proc_sys_kernel_random_entropy_avail.c index d7d1e8261..9515dad61 100644 --- a/src/proc_sys_kernel_random_entropy_avail.c +++ b/src/proc_sys_kernel_random_entropy_avail.c @@ -1,41 +1,31 @@ -#ifdef HAVE_CONFIG_H -#include <config.h> -#endif -#include <stdio.h> -#include <stdlib.h> - #include "common.h" -#include "appconfig.h" -#include "procfile.h" -#include "rrd.h" -#include "plugin_proc.h" int do_proc_sys_kernel_random_entropy_avail(int update_every, unsigned long long dt) { - static procfile *ff = NULL; + static procfile *ff = NULL; - if(dt) {} ; + if(dt) {} ; - if(!ff) { - char filename[FILENAME_MAX + 1]; - snprintfz(filename, FILENAME_MAX, "%s%s", global_host_prefix, "/proc/sys/kernel/random/entropy_avail"); - ff = procfile_open(config_get("plugin:proc:/proc/sys/kernel/random/entropy_avail", "filename to monitor", filename), "", PROCFILE_FLAG_DEFAULT); - } - if(!ff) return 1; + if(!ff) { + char filename[FILENAME_MAX + 1]; + snprintfz(filename, FILENAME_MAX, "%s%s", global_host_prefix, "/proc/sys/kernel/random/entropy_avail"); + ff = procfile_open(config_get("plugin:proc:/proc/sys/kernel/random/entropy_avail", "filename to monitor", filename), "", PROCFILE_FLAG_DEFAULT); + } + if(!ff) return 1; - ff = procfile_readall(ff); - if(!ff) return 0; // we return 0, so that we will retry to open it next time + ff = procfile_readall(ff); + if(!ff) return 0; // we return 0, so that we will retry to open it next time - unsigned long long entropy = strtoull(procfile_lineword(ff, 0, 0), NULL, 10); + unsigned long long entropy = strtoull(procfile_lineword(ff, 0, 0), NULL, 10); - RRDSET *st = rrdset_find_bytype("system", "entropy"); - if(!st) { - st = rrdset_create("system", "entropy", NULL, "entropy", NULL, "Available Entropy", "entropy", 1000, update_every, RRDSET_TYPE_LINE); - rrddim_add(st, "entropy", NULL, 1, 1, RRDDIM_ABSOLUTE); - } - else rrdset_next(st); + RRDSET *st = rrdset_find_bytype("system", "entropy"); + if(!st) { + st = rrdset_create("system", "entropy", NULL, "entropy", NULL, "Available Entropy", "entropy", 1000, update_every, RRDSET_TYPE_LINE); + rrddim_add(st, "entropy", NULL, 1, 1, RRDDIM_ABSOLUTE); + } + else rrdset_next(st); - rrddim_set(st, "entropy", entropy); - rrdset_done(st); + rrddim_set(st, "entropy", entropy); + rrdset_done(st); - return 0; + return 0; } diff --git a/src/proc_vmstat.c b/src/proc_vmstat.c index 7b20ed8cf..f25d50c50 100644 --- a/src/proc_vmstat.c +++ b/src/proc_vmstat.c @@ -1,483 +1,470 @@ -#ifdef HAVE_CONFIG_H -#include <config.h> -#endif -#include <stdio.h> -#include <stdlib.h> -#include <string.h> -#include <unistd.h> - #include "common.h" -#include "log.h" -#include "appconfig.h" -#include "procfile.h" -#include "rrd.h" -#include "plugin_proc.h" int do_proc_vmstat(int update_every, unsigned long long dt) { - static procfile *ff = NULL; - static int do_swapio = -1, do_io = -1, do_pgfaults = -1, gen_hashes = -1; + static procfile *ff = NULL; + static int do_swapio = -1, do_io = -1, do_pgfaults = -1, gen_hashes = -1; - // static uint32_t hash_allocstall = -1; - // static uint32_t hash_compact_blocks_moved = -1; - // static uint32_t hash_compact_fail = -1; - // static uint32_t hash_compact_pagemigrate_failed = -1; - // static uint32_t hash_compact_pages_moved = -1; - // static uint32_t hash_compact_stall = -1; - // static uint32_t hash_compact_success = -1; - // static uint32_t hash_htlb_buddy_alloc_fail = -1; - // static uint32_t hash_htlb_buddy_alloc_success = -1; - // static uint32_t hash_kswapd_high_wmark_hit_quickly = -1; - // static uint32_t hash_kswapd_inodesteal = -1; - // static uint32_t hash_kswapd_low_wmark_hit_quickly = -1; - // static uint32_t hash_kswapd_skip_congestion_wait = -1; - // static uint32_t hash_nr_active_anon = -1; - // static uint32_t hash_nr_active_file = -1; - // static uint32_t hash_nr_anon_pages = -1; - // static uint32_t hash_nr_anon_transparent_hugepages = -1; - // static uint32_t hash_nr_bounce = -1; - // static uint32_t hash_nr_dirtied = -1; - // static uint32_t hash_nr_dirty = -1; - // static uint32_t hash_nr_dirty_background_threshold = -1; - // static uint32_t hash_nr_dirty_threshold = -1; - // static uint32_t hash_nr_file_pages = -1; - // static uint32_t hash_nr_free_pages = -1; - // static uint32_t hash_nr_inactive_anon = -1; - // static uint32_t hash_nr_inactive_file = -1; - // static uint32_t hash_nr_isolated_anon = -1; - // static uint32_t hash_nr_isolated_file = -1; - // static uint32_t hash_nr_kernel_stack = -1; - // static uint32_t hash_nr_mapped = -1; - // static uint32_t hash_nr_mlock = -1; - // static uint32_t hash_nr_page_table_pages = -1; - // static uint32_t hash_nr_shmem = -1; - // static uint32_t hash_nr_slab_reclaimable = -1; - // static uint32_t hash_nr_slab_unreclaimable = -1; - // static uint32_t hash_nr_unevictable = -1; - // static uint32_t hash_nr_unstable = -1; - // static uint32_t hash_nr_vmscan_immediate_reclaim = -1; - // static uint32_t hash_nr_vmscan_write = -1; - // static uint32_t hash_nr_writeback = -1; - // static uint32_t hash_nr_writeback_temp = -1; - // static uint32_t hash_nr_written = -1; - // static uint32_t hash_pageoutrun = -1; - // static uint32_t hash_pgactivate = -1; - // static uint32_t hash_pgalloc_dma = -1; - // static uint32_t hash_pgalloc_dma32 = -1; - // static uint32_t hash_pgalloc_movable = -1; - // static uint32_t hash_pgalloc_normal = -1; - // static uint32_t hash_pgdeactivate = -1; - static uint32_t hash_pgfault = -1; - // static uint32_t hash_pgfree = -1; - // static uint32_t hash_pginodesteal = -1; - static uint32_t hash_pgmajfault = -1; - static uint32_t hash_pgpgin = -1; - static uint32_t hash_pgpgout = -1; - // static uint32_t hash_pgrefill_dma = -1; - // static uint32_t hash_pgrefill_dma32 = -1; - // static uint32_t hash_pgrefill_movable = -1; - // static uint32_t hash_pgrefill_normal = -1; - // static uint32_t hash_pgrotated = -1; - // static uint32_t hash_pgscan_direct_dma = -1; - // static uint32_t hash_pgscan_direct_dma32 = -1; - // static uint32_t hash_pgscan_direct_movable = -1; - // static uint32_t hash_pgscan_direct_normal = -1; - // static uint32_t hash_pgscan_kswapd_dma = -1; - // static uint32_t hash_pgscan_kswapd_dma32 = -1; - // static uint32_t hash_pgscan_kswapd_movable = -1; - // static uint32_t hash_pgscan_kswapd_normal = -1; - // static uint32_t hash_pgsteal_direct_dma = -1; - // static uint32_t hash_pgsteal_direct_dma32 = -1; - // static uint32_t hash_pgsteal_direct_movable = -1; - // static uint32_t hash_pgsteal_direct_normal = -1; - // static uint32_t hash_pgsteal_kswapd_dma = -1; - // static uint32_t hash_pgsteal_kswapd_dma32 = -1; - // static uint32_t hash_pgsteal_kswapd_movable = -1; - // static uint32_t hash_pgsteal_kswapd_normal = -1; - static uint32_t hash_pswpin = -1; - static uint32_t hash_pswpout = -1; - // static uint32_t hash_slabs_scanned = -1; - // static uint32_t hash_thp_collapse_alloc = -1; - // static uint32_t hash_thp_collapse_alloc_failed = -1; - // static uint32_t hash_thp_fault_alloc = -1; - // static uint32_t hash_thp_fault_fallback = -1; - // static uint32_t hash_thp_split = -1; - // static uint32_t hash_unevictable_pgs_cleared = -1; - // static uint32_t hash_unevictable_pgs_culled = -1; - // static uint32_t hash_unevictable_pgs_mlocked = -1; - // static uint32_t hash_unevictable_pgs_mlockfreed = -1; - // static uint32_t hash_unevictable_pgs_munlocked = -1; - // static uint32_t hash_unevictable_pgs_rescued = -1; - // static uint32_t hash_unevictable_pgs_scanned = -1; - // static uint32_t hash_unevictable_pgs_stranded = -1; + // static uint32_t hash_allocstall = -1; + // static uint32_t hash_compact_blocks_moved = -1; + // static uint32_t hash_compact_fail = -1; + // static uint32_t hash_compact_pagemigrate_failed = -1; + // static uint32_t hash_compact_pages_moved = -1; + // static uint32_t hash_compact_stall = -1; + // static uint32_t hash_compact_success = -1; + // static uint32_t hash_htlb_buddy_alloc_fail = -1; + // static uint32_t hash_htlb_buddy_alloc_success = -1; + // static uint32_t hash_kswapd_high_wmark_hit_quickly = -1; + // static uint32_t hash_kswapd_inodesteal = -1; + // static uint32_t hash_kswapd_low_wmark_hit_quickly = -1; + // static uint32_t hash_kswapd_skip_congestion_wait = -1; + // static uint32_t hash_nr_active_anon = -1; + // static uint32_t hash_nr_active_file = -1; + // static uint32_t hash_nr_anon_pages = -1; + // static uint32_t hash_nr_anon_transparent_hugepages = -1; + // static uint32_t hash_nr_bounce = -1; + // static uint32_t hash_nr_dirtied = -1; + // static uint32_t hash_nr_dirty = -1; + // static uint32_t hash_nr_dirty_background_threshold = -1; + // static uint32_t hash_nr_dirty_threshold = -1; + // static uint32_t hash_nr_file_pages = -1; + // static uint32_t hash_nr_free_pages = -1; + // static uint32_t hash_nr_inactive_anon = -1; + // static uint32_t hash_nr_inactive_file = -1; + // static uint32_t hash_nr_isolated_anon = -1; + // static uint32_t hash_nr_isolated_file = -1; + // static uint32_t hash_nr_kernel_stack = -1; + // static uint32_t hash_nr_mapped = -1; + // static uint32_t hash_nr_mlock = -1; + // static uint32_t hash_nr_page_table_pages = -1; + // static uint32_t hash_nr_shmem = -1; + // static uint32_t hash_nr_slab_reclaimable = -1; + // static uint32_t hash_nr_slab_unreclaimable = -1; + // static uint32_t hash_nr_unevictable = -1; + // static uint32_t hash_nr_unstable = -1; + // static uint32_t hash_nr_vmscan_immediate_reclaim = -1; + // static uint32_t hash_nr_vmscan_write = -1; + // static uint32_t hash_nr_writeback = -1; + // static uint32_t hash_nr_writeback_temp = -1; + // static uint32_t hash_nr_written = -1; + // static uint32_t hash_pageoutrun = -1; + // static uint32_t hash_pgactivate = -1; + // static uint32_t hash_pgalloc_dma = -1; + // static uint32_t hash_pgalloc_dma32 = -1; + // static uint32_t hash_pgalloc_movable = -1; + // static uint32_t hash_pgalloc_normal = -1; + // static uint32_t hash_pgdeactivate = -1; + static uint32_t hash_pgfault = -1; + // static uint32_t hash_pgfree = -1; + // static uint32_t hash_pginodesteal = -1; + static uint32_t hash_pgmajfault = -1; + static uint32_t hash_pgpgin = -1; + static uint32_t hash_pgpgout = -1; + // static uint32_t hash_pgrefill_dma = -1; + // static uint32_t hash_pgrefill_dma32 = -1; + // static uint32_t hash_pgrefill_movable = -1; + // static uint32_t hash_pgrefill_normal = -1; + // static uint32_t hash_pgrotated = -1; + // static uint32_t hash_pgscan_direct_dma = -1; + // static uint32_t hash_pgscan_direct_dma32 = -1; + // static uint32_t hash_pgscan_direct_movable = -1; + // static uint32_t hash_pgscan_direct_normal = -1; + // static uint32_t hash_pgscan_kswapd_dma = -1; + // static uint32_t hash_pgscan_kswapd_dma32 = -1; + // static uint32_t hash_pgscan_kswapd_movable = -1; + // static uint32_t hash_pgscan_kswapd_normal = -1; + // static uint32_t hash_pgsteal_direct_dma = -1; + // static uint32_t hash_pgsteal_direct_dma32 = -1; + // static uint32_t hash_pgsteal_direct_movable = -1; + // static uint32_t hash_pgsteal_direct_normal = -1; + // static uint32_t hash_pgsteal_kswapd_dma = -1; + // static uint32_t hash_pgsteal_kswapd_dma32 = -1; + // static uint32_t hash_pgsteal_kswapd_movable = -1; + // static uint32_t hash_pgsteal_kswapd_normal = -1; + static uint32_t hash_pswpin = -1; + static uint32_t hash_pswpout = -1; + // static uint32_t hash_slabs_scanned = -1; + // static uint32_t hash_thp_collapse_alloc = -1; + // static uint32_t hash_thp_collapse_alloc_failed = -1; + // static uint32_t hash_thp_fault_alloc = -1; + // static uint32_t hash_thp_fault_fallback = -1; + // static uint32_t hash_thp_split = -1; + // static uint32_t hash_unevictable_pgs_cleared = -1; + // static uint32_t hash_unevictable_pgs_culled = -1; + // static uint32_t hash_unevictable_pgs_mlocked = -1; + // static uint32_t hash_unevictable_pgs_mlockfreed = -1; + // static uint32_t hash_unevictable_pgs_munlocked = -1; + // static uint32_t hash_unevictable_pgs_rescued = -1; + // static uint32_t hash_unevictable_pgs_scanned = -1; + // static uint32_t hash_unevictable_pgs_stranded = -1; - if(gen_hashes != 1) { - gen_hashes = 1; - // hash_allocstall = simple_hash("allocstall"); - // hash_compact_blocks_moved = simple_hash("compact_blocks_moved"); - // hash_compact_fail = simple_hash("compact_fail"); - // hash_compact_pagemigrate_failed = simple_hash("compact_pagemigrate_failed"); - // hash_compact_pages_moved = simple_hash("compact_pages_moved"); - // hash_compact_stall = simple_hash("compact_stall"); - // hash_compact_success = simple_hash("compact_success"); - // hash_htlb_buddy_alloc_fail = simple_hash("htlb_buddy_alloc_fail"); - // hash_htlb_buddy_alloc_success = simple_hash("htlb_buddy_alloc_success"); - // hash_kswapd_high_wmark_hit_quickly = simple_hash("kswapd_high_wmark_hit_quickly"); - // hash_kswapd_inodesteal = simple_hash("kswapd_inodesteal"); - // hash_kswapd_low_wmark_hit_quickly = simple_hash("kswapd_low_wmark_hit_quickly"); - // hash_kswapd_skip_congestion_wait = simple_hash("kswapd_skip_congestion_wait"); - // hash_nr_active_anon = simple_hash("nr_active_anon"); - // hash_nr_active_file = simple_hash("nr_active_file"); - // hash_nr_anon_pages = simple_hash("nr_anon_pages"); - // hash_nr_anon_transparent_hugepages = simple_hash("nr_anon_transparent_hugepages"); - // hash_nr_bounce = simple_hash("nr_bounce"); - // hash_nr_dirtied = simple_hash("nr_dirtied"); - // hash_nr_dirty = simple_hash("nr_dirty"); - // hash_nr_dirty_background_threshold = simple_hash("nr_dirty_background_threshold"); - // hash_nr_dirty_threshold = simple_hash("nr_dirty_threshold"); - // hash_nr_file_pages = simple_hash("nr_file_pages"); - // hash_nr_free_pages = simple_hash("nr_free_pages"); - // hash_nr_inactive_anon = simple_hash("nr_inactive_anon"); - // hash_nr_inactive_file = simple_hash("nr_inactive_file"); - // hash_nr_isolated_anon = simple_hash("nr_isolated_anon"); - // hash_nr_isolated_file = simple_hash("nr_isolated_file"); - // hash_nr_kernel_stack = simple_hash("nr_kernel_stack"); - // hash_nr_mapped = simple_hash("nr_mapped"); - // hash_nr_mlock = simple_hash("nr_mlock"); - // hash_nr_page_table_pages = simple_hash("nr_page_table_pages"); - // hash_nr_shmem = simple_hash("nr_shmem"); - // hash_nr_slab_reclaimable = simple_hash("nr_slab_reclaimable"); - // hash_nr_slab_unreclaimable = simple_hash("nr_slab_unreclaimable"); - // hash_nr_unevictable = simple_hash("nr_unevictable"); - // hash_nr_unstable = simple_hash("nr_unstable"); - // hash_nr_vmscan_immediate_reclaim = simple_hash("nr_vmscan_immediate_reclaim"); - // hash_nr_vmscan_write = simple_hash("nr_vmscan_write"); - // hash_nr_writeback = simple_hash("nr_writeback"); - // hash_nr_writeback_temp = simple_hash("nr_writeback_temp"); - // hash_nr_written = simple_hash("nr_written"); - // hash_pageoutrun = simple_hash("pageoutrun"); - // hash_pgactivate = simple_hash("pgactivate"); - // hash_pgalloc_dma = simple_hash("pgalloc_dma"); - // hash_pgalloc_dma32 = simple_hash("pgalloc_dma32"); - // hash_pgalloc_movable = simple_hash("pgalloc_movable"); - // hash_pgalloc_normal = simple_hash("pgalloc_normal"); - // hash_pgdeactivate = simple_hash("pgdeactivate"); - hash_pgfault = simple_hash("pgfault"); - // hash_pgfree = simple_hash("pgfree"); - // hash_pginodesteal = simple_hash("pginodesteal"); - hash_pgmajfault = simple_hash("pgmajfault"); - hash_pgpgin = simple_hash("pgpgin"); - hash_pgpgout = simple_hash("pgpgout"); - // hash_pgrefill_dma = simple_hash("pgrefill_dma"); - // hash_pgrefill_dma32 = simple_hash("pgrefill_dma32"); - // hash_pgrefill_movable = simple_hash("pgrefill_movable"); - // hash_pgrefill_normal = simple_hash("pgrefill_normal"); - // hash_pgrotated = simple_hash("pgrotated"); - // hash_pgscan_direct_dma = simple_hash("pgscan_direct_dma"); - // hash_pgscan_direct_dma32 = simple_hash("pgscan_direct_dma32"); - // hash_pgscan_direct_movable = simple_hash("pgscan_direct_movable"); - // hash_pgscan_direct_normal = simple_hash("pgscan_direct_normal"); - // hash_pgscan_kswapd_dma = simple_hash("pgscan_kswapd_dma"); - // hash_pgscan_kswapd_dma32 = simple_hash("pgscan_kswapd_dma32"); - // hash_pgscan_kswapd_movable = simple_hash("pgscan_kswapd_movable"); - // hash_pgscan_kswapd_normal = simple_hash("pgscan_kswapd_normal"); - // hash_pgsteal_direct_dma = simple_hash("pgsteal_direct_dma"); - // hash_pgsteal_direct_dma32 = simple_hash("pgsteal_direct_dma32"); - // hash_pgsteal_direct_movable = simple_hash("pgsteal_direct_movable"); - // hash_pgsteal_direct_normal = simple_hash("pgsteal_direct_normal"); - // hash_pgsteal_kswapd_dma = simple_hash("pgsteal_kswapd_dma"); - // hash_pgsteal_kswapd_dma32 = simple_hash("pgsteal_kswapd_dma32"); - // hash_pgsteal_kswapd_movable = simple_hash("pgsteal_kswapd_movable"); - // hash_pgsteal_kswapd_normal = simple_hash("pgsteal_kswapd_normal"); - hash_pswpin = simple_hash("pswpin"); - hash_pswpout = simple_hash("pswpout"); - // hash_slabs_scanned = simple_hash("slabs_scanned"); - // hash_thp_collapse_alloc = simple_hash("thp_collapse_alloc"); - // hash_thp_collapse_alloc_failed = simple_hash("thp_collapse_alloc_failed"); - // hash_thp_fault_alloc = simple_hash("thp_fault_alloc"); - // hash_thp_fault_fallback = simple_hash("thp_fault_fallback"); - // hash_thp_split = simple_hash("thp_split"); - // hash_unevictable_pgs_cleared = simple_hash("unevictable_pgs_cleared"); - // hash_unevictable_pgs_culled = simple_hash("unevictable_pgs_culled"); - // hash_unevictable_pgs_mlocked = simple_hash("unevictable_pgs_mlocked"); - // hash_unevictable_pgs_mlockfreed = simple_hash("unevictable_pgs_mlockfreed"); - // hash_unevictable_pgs_munlocked = simple_hash("unevictable_pgs_munlocked"); - // hash_unevictable_pgs_rescued = simple_hash("unevictable_pgs_rescued"); - // hash_unevictable_pgs_scanned = simple_hash("unevictable_pgs_scanned"); - // hash_unevictable_pgs_stranded = simple_hash("unevictable_pgs_stranded"); - } + if(gen_hashes != 1) { + gen_hashes = 1; + // hash_allocstall = simple_hash("allocstall"); + // hash_compact_blocks_moved = simple_hash("compact_blocks_moved"); + // hash_compact_fail = simple_hash("compact_fail"); + // hash_compact_pagemigrate_failed = simple_hash("compact_pagemigrate_failed"); + // hash_compact_pages_moved = simple_hash("compact_pages_moved"); + // hash_compact_stall = simple_hash("compact_stall"); + // hash_compact_success = simple_hash("compact_success"); + // hash_htlb_buddy_alloc_fail = simple_hash("htlb_buddy_alloc_fail"); + // hash_htlb_buddy_alloc_success = simple_hash("htlb_buddy_alloc_success"); + // hash_kswapd_high_wmark_hit_quickly = simple_hash("kswapd_high_wmark_hit_quickly"); + // hash_kswapd_inodesteal = simple_hash("kswapd_inodesteal"); + // hash_kswapd_low_wmark_hit_quickly = simple_hash("kswapd_low_wmark_hit_quickly"); + // hash_kswapd_skip_congestion_wait = simple_hash("kswapd_skip_congestion_wait"); + // hash_nr_active_anon = simple_hash("nr_active_anon"); + // hash_nr_active_file = simple_hash("nr_active_file"); + // hash_nr_anon_pages = simple_hash("nr_anon_pages"); + // hash_nr_anon_transparent_hugepages = simple_hash("nr_anon_transparent_hugepages"); + // hash_nr_bounce = simple_hash("nr_bounce"); + // hash_nr_dirtied = simple_hash("nr_dirtied"); + // hash_nr_dirty = simple_hash("nr_dirty"); + // hash_nr_dirty_background_threshold = simple_hash("nr_dirty_background_threshold"); + // hash_nr_dirty_threshold = simple_hash("nr_dirty_threshold"); + // hash_nr_file_pages = simple_hash("nr_file_pages"); + // hash_nr_free_pages = simple_hash("nr_free_pages"); + // hash_nr_inactive_anon = simple_hash("nr_inactive_anon"); + // hash_nr_inactive_file = simple_hash("nr_inactive_file"); + // hash_nr_isolated_anon = simple_hash("nr_isolated_anon"); + // hash_nr_isolated_file = simple_hash("nr_isolated_file"); + // hash_nr_kernel_stack = simple_hash("nr_kernel_stack"); + // hash_nr_mapped = simple_hash("nr_mapped"); + // hash_nr_mlock = simple_hash("nr_mlock"); + // hash_nr_page_table_pages = simple_hash("nr_page_table_pages"); + // hash_nr_shmem = simple_hash("nr_shmem"); + // hash_nr_slab_reclaimable = simple_hash("nr_slab_reclaimable"); + // hash_nr_slab_unreclaimable = simple_hash("nr_slab_unreclaimable"); + // hash_nr_unevictable = simple_hash("nr_unevictable"); + // hash_nr_unstable = simple_hash("nr_unstable"); + // hash_nr_vmscan_immediate_reclaim = simple_hash("nr_vmscan_immediate_reclaim"); + // hash_nr_vmscan_write = simple_hash("nr_vmscan_write"); + // hash_nr_writeback = simple_hash("nr_writeback"); + // hash_nr_writeback_temp = simple_hash("nr_writeback_temp"); + // hash_nr_written = simple_hash("nr_written"); + // hash_pageoutrun = simple_hash("pageoutrun"); + // hash_pgactivate = simple_hash("pgactivate"); + // hash_pgalloc_dma = simple_hash("pgalloc_dma"); + // hash_pgalloc_dma32 = simple_hash("pgalloc_dma32"); + // hash_pgalloc_movable = simple_hash("pgalloc_movable"); + // hash_pgalloc_normal = simple_hash("pgalloc_normal"); + // hash_pgdeactivate = simple_hash("pgdeactivate"); + hash_pgfault = simple_hash("pgfault"); + // hash_pgfree = simple_hash("pgfree"); + // hash_pginodesteal = simple_hash("pginodesteal"); + hash_pgmajfault = simple_hash("pgmajfault"); + hash_pgpgin = simple_hash("pgpgin"); + hash_pgpgout = simple_hash("pgpgout"); + // hash_pgrefill_dma = simple_hash("pgrefill_dma"); + // hash_pgrefill_dma32 = simple_hash("pgrefill_dma32"); + // hash_pgrefill_movable = simple_hash("pgrefill_movable"); + // hash_pgrefill_normal = simple_hash("pgrefill_normal"); + // hash_pgrotated = simple_hash("pgrotated"); + // hash_pgscan_direct_dma = simple_hash("pgscan_direct_dma"); + // hash_pgscan_direct_dma32 = simple_hash("pgscan_direct_dma32"); + // hash_pgscan_direct_movable = simple_hash("pgscan_direct_movable"); + // hash_pgscan_direct_normal = simple_hash("pgscan_direct_normal"); + // hash_pgscan_kswapd_dma = simple_hash("pgscan_kswapd_dma"); + // hash_pgscan_kswapd_dma32 = simple_hash("pgscan_kswapd_dma32"); + // hash_pgscan_kswapd_movable = simple_hash("pgscan_kswapd_movable"); + // hash_pgscan_kswapd_normal = simple_hash("pgscan_kswapd_normal"); + // hash_pgsteal_direct_dma = simple_hash("pgsteal_direct_dma"); + // hash_pgsteal_direct_dma32 = simple_hash("pgsteal_direct_dma32"); + // hash_pgsteal_direct_movable = simple_hash("pgsteal_direct_movable"); + // hash_pgsteal_direct_normal = simple_hash("pgsteal_direct_normal"); + // hash_pgsteal_kswapd_dma = simple_hash("pgsteal_kswapd_dma"); + // hash_pgsteal_kswapd_dma32 = simple_hash("pgsteal_kswapd_dma32"); + // hash_pgsteal_kswapd_movable = simple_hash("pgsteal_kswapd_movable"); + // hash_pgsteal_kswapd_normal = simple_hash("pgsteal_kswapd_normal"); + hash_pswpin = simple_hash("pswpin"); + hash_pswpout = simple_hash("pswpout"); + // hash_slabs_scanned = simple_hash("slabs_scanned"); + // hash_thp_collapse_alloc = simple_hash("thp_collapse_alloc"); + // hash_thp_collapse_alloc_failed = simple_hash("thp_collapse_alloc_failed"); + // hash_thp_fault_alloc = simple_hash("thp_fault_alloc"); + // hash_thp_fault_fallback = simple_hash("thp_fault_fallback"); + // hash_thp_split = simple_hash("thp_split"); + // hash_unevictable_pgs_cleared = simple_hash("unevictable_pgs_cleared"); + // hash_unevictable_pgs_culled = simple_hash("unevictable_pgs_culled"); + // hash_unevictable_pgs_mlocked = simple_hash("unevictable_pgs_mlocked"); + // hash_unevictable_pgs_mlockfreed = simple_hash("unevictable_pgs_mlockfreed"); + // hash_unevictable_pgs_munlocked = simple_hash("unevictable_pgs_munlocked"); + // hash_unevictable_pgs_rescued = simple_hash("unevictable_pgs_rescued"); + // hash_unevictable_pgs_scanned = simple_hash("unevictable_pgs_scanned"); + // hash_unevictable_pgs_stranded = simple_hash("unevictable_pgs_stranded"); + } - if(do_swapio == -1) do_swapio = config_get_boolean("plugin:proc:/proc/vmstat", "swap i/o", 1); - if(do_io == -1) do_io = config_get_boolean("plugin:proc:/proc/vmstat", "disk i/o", 1); - if(do_pgfaults == -1) do_pgfaults = config_get_boolean("plugin:proc:/proc/vmstat", "memory page faults", 1); + if(do_swapio == -1) do_swapio = config_get_boolean("plugin:proc:/proc/vmstat", "swap i/o", 1); + if(do_io == -1) do_io = config_get_boolean("plugin:proc:/proc/vmstat", "disk i/o", 1); + if(do_pgfaults == -1) do_pgfaults = config_get_boolean("plugin:proc:/proc/vmstat", "memory page faults", 1); - if(dt) {}; + if(dt) {}; - if(!ff) { - char filename[FILENAME_MAX + 1]; - snprintfz(filename, FILENAME_MAX, "%s%s", global_host_prefix, "/proc/vmstat"); - ff = procfile_open(config_get("plugin:proc:/proc/vmstat", "filename to monitor", filename), " \t:", PROCFILE_FLAG_DEFAULT); - } - if(!ff) return 1; + if(!ff) { + char filename[FILENAME_MAX + 1]; + snprintfz(filename, FILENAME_MAX, "%s%s", global_host_prefix, "/proc/vmstat"); + ff = procfile_open(config_get("plugin:proc:/proc/vmstat", "filename to monitor", filename), " \t:", PROCFILE_FLAG_DEFAULT); + } + if(!ff) return 1; - ff = procfile_readall(ff); - if(!ff) return 0; // we return 0, so that we will retry to open it next time + ff = procfile_readall(ff); + if(!ff) return 0; // we return 0, so that we will retry to open it next time - uint32_t lines = procfile_lines(ff), l; - uint32_t words; + uint32_t lines = procfile_lines(ff), l; + uint32_t words; - // unsigned long long allocstall = 0ULL; - // unsigned long long compact_blocks_moved = 0ULL; - // unsigned long long compact_fail = 0ULL; - // unsigned long long compact_pagemigrate_failed = 0ULL; - // unsigned long long compact_pages_moved = 0ULL; - // unsigned long long compact_stall = 0ULL; - // unsigned long long compact_success = 0ULL; - // unsigned long long htlb_buddy_alloc_fail = 0ULL; - // unsigned long long htlb_buddy_alloc_success = 0ULL; - // unsigned long long kswapd_high_wmark_hit_quickly = 0ULL; - // unsigned long long kswapd_inodesteal = 0ULL; - // unsigned long long kswapd_low_wmark_hit_quickly = 0ULL; - // unsigned long long kswapd_skip_congestion_wait = 0ULL; - // unsigned long long nr_active_anon = 0ULL; - // unsigned long long nr_active_file = 0ULL; - // unsigned long long nr_anon_pages = 0ULL; - // unsigned long long nr_anon_transparent_hugepages = 0ULL; - // unsigned long long nr_bounce = 0ULL; - // unsigned long long nr_dirtied = 0ULL; - // unsigned long long nr_dirty = 0ULL; - // unsigned long long nr_dirty_background_threshold = 0ULL; - // unsigned long long nr_dirty_threshold = 0ULL; - // unsigned long long nr_file_pages = 0ULL; - // unsigned long long nr_free_pages = 0ULL; - // unsigned long long nr_inactive_anon = 0ULL; - // unsigned long long nr_inactive_file = 0ULL; - // unsigned long long nr_isolated_anon = 0ULL; - // unsigned long long nr_isolated_file = 0ULL; - // unsigned long long nr_kernel_stack = 0ULL; - // unsigned long long nr_mapped = 0ULL; - // unsigned long long nr_mlock = 0ULL; - // unsigned long long nr_page_table_pages = 0ULL; - // unsigned long long nr_shmem = 0ULL; - // unsigned long long nr_slab_reclaimable = 0ULL; - // unsigned long long nr_slab_unreclaimable = 0ULL; - // unsigned long long nr_unevictable = 0ULL; - // unsigned long long nr_unstable = 0ULL; - // unsigned long long nr_vmscan_immediate_reclaim = 0ULL; - // unsigned long long nr_vmscan_write = 0ULL; - // unsigned long long nr_writeback = 0ULL; - // unsigned long long nr_writeback_temp = 0ULL; - // unsigned long long nr_written = 0ULL; - // unsigned long long pageoutrun = 0ULL; - // unsigned long long pgactivate = 0ULL; - // unsigned long long pgalloc_dma = 0ULL; - // unsigned long long pgalloc_dma32 = 0ULL; - // unsigned long long pgalloc_movable = 0ULL; - // unsigned long long pgalloc_normal = 0ULL; - // unsigned long long pgdeactivate = 0ULL; - unsigned long long pgfault = 0ULL; - // unsigned long long pgfree = 0ULL; - // unsigned long long pginodesteal = 0ULL; - unsigned long long pgmajfault = 0ULL; - unsigned long long pgpgin = 0ULL; - unsigned long long pgpgout = 0ULL; - // unsigned long long pgrefill_dma = 0ULL; - // unsigned long long pgrefill_dma32 = 0ULL; - // unsigned long long pgrefill_movable = 0ULL; - // unsigned long long pgrefill_normal = 0ULL; - // unsigned long long pgrotated = 0ULL; - // unsigned long long pgscan_direct_dma = 0ULL; - // unsigned long long pgscan_direct_dma32 = 0ULL; - // unsigned long long pgscan_direct_movable = 0ULL; - // unsigned long long pgscan_direct_normal = 0ULL; - // unsigned long long pgscan_kswapd_dma = 0ULL; - // unsigned long long pgscan_kswapd_dma32 = 0ULL; - // unsigned long long pgscan_kswapd_movable = 0ULL; - // unsigned long long pgscan_kswapd_normal = 0ULL; - // unsigned long long pgsteal_direct_dma = 0ULL; - // unsigned long long pgsteal_direct_dma32 = 0ULL; - // unsigned long long pgsteal_direct_movable = 0ULL; - // unsigned long long pgsteal_direct_normal = 0ULL; - // unsigned long long pgsteal_kswapd_dma = 0ULL; - // unsigned long long pgsteal_kswapd_dma32 = 0ULL; - // unsigned long long pgsteal_kswapd_movable = 0ULL; - // unsigned long long pgsteal_kswapd_normal = 0ULL; - unsigned long long pswpin = 0ULL; - unsigned long long pswpout = 0ULL; - // unsigned long long slabs_scanned = 0ULL; - // unsigned long long thp_collapse_alloc = 0ULL; - // unsigned long long thp_collapse_alloc_failed = 0ULL; - // unsigned long long thp_fault_alloc = 0ULL; - // unsigned long long thp_fault_fallback = 0ULL; - // unsigned long long thp_split = 0ULL; - // unsigned long long unevictable_pgs_cleared = 0ULL; - // unsigned long long unevictable_pgs_culled = 0ULL; - // unsigned long long unevictable_pgs_mlocked = 0ULL; - // unsigned long long unevictable_pgs_mlockfreed = 0ULL; - // unsigned long long unevictable_pgs_munlocked = 0ULL; - // unsigned long long unevictable_pgs_rescued = 0ULL; - // unsigned long long unevictable_pgs_scanned = 0ULL; - // unsigned long long unevictable_pgs_stranded = 0ULL; + // unsigned long long allocstall = 0ULL; + // unsigned long long compact_blocks_moved = 0ULL; + // unsigned long long compact_fail = 0ULL; + // unsigned long long compact_pagemigrate_failed = 0ULL; + // unsigned long long compact_pages_moved = 0ULL; + // unsigned long long compact_stall = 0ULL; + // unsigned long long compact_success = 0ULL; + // unsigned long long htlb_buddy_alloc_fail = 0ULL; + // unsigned long long htlb_buddy_alloc_success = 0ULL; + // unsigned long long kswapd_high_wmark_hit_quickly = 0ULL; + // unsigned long long kswapd_inodesteal = 0ULL; + // unsigned long long kswapd_low_wmark_hit_quickly = 0ULL; + // unsigned long long kswapd_skip_congestion_wait = 0ULL; + // unsigned long long nr_active_anon = 0ULL; + // unsigned long long nr_active_file = 0ULL; + // unsigned long long nr_anon_pages = 0ULL; + // unsigned long long nr_anon_transparent_hugepages = 0ULL; + // unsigned long long nr_bounce = 0ULL; + // unsigned long long nr_dirtied = 0ULL; + // unsigned long long nr_dirty = 0ULL; + // unsigned long long nr_dirty_background_threshold = 0ULL; + // unsigned long long nr_dirty_threshold = 0ULL; + // unsigned long long nr_file_pages = 0ULL; + // unsigned long long nr_free_pages = 0ULL; + // unsigned long long nr_inactive_anon = 0ULL; + // unsigned long long nr_inactive_file = 0ULL; + // unsigned long long nr_isolated_anon = 0ULL; + // unsigned long long nr_isolated_file = 0ULL; + // unsigned long long nr_kernel_stack = 0ULL; + // unsigned long long nr_mapped = 0ULL; + // unsigned long long nr_mlock = 0ULL; + // unsigned long long nr_page_table_pages = 0ULL; + // unsigned long long nr_shmem = 0ULL; + // unsigned long long nr_slab_reclaimable = 0ULL; + // unsigned long long nr_slab_unreclaimable = 0ULL; + // unsigned long long nr_unevictable = 0ULL; + // unsigned long long nr_unstable = 0ULL; + // unsigned long long nr_vmscan_immediate_reclaim = 0ULL; + // unsigned long long nr_vmscan_write = 0ULL; + // unsigned long long nr_writeback = 0ULL; + // unsigned long long nr_writeback_temp = 0ULL; + // unsigned long long nr_written = 0ULL; + // unsigned long long pageoutrun = 0ULL; + // unsigned long long pgactivate = 0ULL; + // unsigned long long pgalloc_dma = 0ULL; + // unsigned long long pgalloc_dma32 = 0ULL; + // unsigned long long pgalloc_movable = 0ULL; + // unsigned long long pgalloc_normal = 0ULL; + // unsigned long long pgdeactivate = 0ULL; + unsigned long long pgfault = 0ULL; + // unsigned long long pgfree = 0ULL; + // unsigned long long pginodesteal = 0ULL; + unsigned long long pgmajfault = 0ULL; + unsigned long long pgpgin = 0ULL; + unsigned long long pgpgout = 0ULL; + // unsigned long long pgrefill_dma = 0ULL; + // unsigned long long pgrefill_dma32 = 0ULL; + // unsigned long long pgrefill_movable = 0ULL; + // unsigned long long pgrefill_normal = 0ULL; + // unsigned long long pgrotated = 0ULL; + // unsigned long long pgscan_direct_dma = 0ULL; + // unsigned long long pgscan_direct_dma32 = 0ULL; + // unsigned long long pgscan_direct_movable = 0ULL; + // unsigned long long pgscan_direct_normal = 0ULL; + // unsigned long long pgscan_kswapd_dma = 0ULL; + // unsigned long long pgscan_kswapd_dma32 = 0ULL; + // unsigned long long pgscan_kswapd_movable = 0ULL; + // unsigned long long pgscan_kswapd_normal = 0ULL; + // unsigned long long pgsteal_direct_dma = 0ULL; + // unsigned long long pgsteal_direct_dma32 = 0ULL; + // unsigned long long pgsteal_direct_movable = 0ULL; + // unsigned long long pgsteal_direct_normal = 0ULL; + // unsigned long long pgsteal_kswapd_dma = 0ULL; + // unsigned long long pgsteal_kswapd_dma32 = 0ULL; + // unsigned long long pgsteal_kswapd_movable = 0ULL; + // unsigned long long pgsteal_kswapd_normal = 0ULL; + unsigned long long pswpin = 0ULL; + unsigned long long pswpout = 0ULL; + // unsigned long long slabs_scanned = 0ULL; + // unsigned long long thp_collapse_alloc = 0ULL; + // unsigned long long thp_collapse_alloc_failed = 0ULL; + // unsigned long long thp_fault_alloc = 0ULL; + // unsigned long long thp_fault_fallback = 0ULL; + // unsigned long long thp_split = 0ULL; + // unsigned long long unevictable_pgs_cleared = 0ULL; + // unsigned long long unevictable_pgs_culled = 0ULL; + // unsigned long long unevictable_pgs_mlocked = 0ULL; + // unsigned long long unevictable_pgs_mlockfreed = 0ULL; + // unsigned long long unevictable_pgs_munlocked = 0ULL; + // unsigned long long unevictable_pgs_rescued = 0ULL; + // unsigned long long unevictable_pgs_scanned = 0ULL; + // unsigned long long unevictable_pgs_stranded = 0ULL; - for(l = 0; l < lines ;l++) { - words = procfile_linewords(ff, l); - if(words < 2) { - if(words) error("Cannot read /proc/vmstat line %d. Expected 2 params, read %d.", l, words); - continue; - } + for(l = 0; l < lines ;l++) { + words = procfile_linewords(ff, l); + if(words < 2) { + if(words) error("Cannot read /proc/vmstat line %u. Expected 2 params, read %u.", l, words); + continue; + } - char *name = procfile_lineword(ff, l, 0); - char * value = procfile_lineword(ff, l, 1); - if(!name || !*name || !value || !*value) continue; + char *name = procfile_lineword(ff, l, 0); + char * value = procfile_lineword(ff, l, 1); + if(!name || !*name || !value || !*value) continue; - uint32_t hash = simple_hash(name); + uint32_t hash = simple_hash(name); - if(0) ; - // else if(hash == hash_allocstall && strcmp(name, "allocstall") == 0) allocstall = strtoull(value, NULL, 10); - // else if(hash == hash_compact_blocks_moved && strcmp(name, "compact_blocks_moved") == 0) compact_blocks_moved = strtoull(value, NULL, 10); - // else if(hash == hash_compact_fail && strcmp(name, "compact_fail") == 0) compact_fail = strtoull(value, NULL, 10); - // else if(hash == hash_compact_pagemigrate_failed && strcmp(name, "compact_pagemigrate_failed") == 0) compact_pagemigrate_failed = strtoull(value, NULL, 10); - // else if(hash == hash_compact_pages_moved && strcmp(name, "compact_pages_moved") == 0) compact_pages_moved = strtoull(value, NULL, 10); - // else if(hash == hash_compact_stall && strcmp(name, "compact_stall") == 0) compact_stall = strtoull(value, NULL, 10); - // else if(hash == hash_compact_success && strcmp(name, "compact_success") == 0) compact_success = strtoull(value, NULL, 10); - // else if(hash == hash_htlb_buddy_alloc_fail && strcmp(name, "htlb_buddy_alloc_fail") == 0) htlb_buddy_alloc_fail = strtoull(value, NULL, 10); - // else if(hash == hash_htlb_buddy_alloc_success && strcmp(name, "htlb_buddy_alloc_success") == 0) htlb_buddy_alloc_success = strtoull(value, NULL, 10); - // else if(hash == hash_kswapd_high_wmark_hit_quickly && strcmp(name, "kswapd_high_wmark_hit_quickly") == 0) kswapd_high_wmark_hit_quickly = strtoull(value, NULL, 10); - // else if(hash == hash_kswapd_inodesteal && strcmp(name, "kswapd_inodesteal") == 0) kswapd_inodesteal = strtoull(value, NULL, 10); - // else if(hash == hash_kswapd_low_wmark_hit_quickly && strcmp(name, "kswapd_low_wmark_hit_quickly") == 0) kswapd_low_wmark_hit_quickly = strtoull(value, NULL, 10); - // else if(hash == hash_kswapd_skip_congestion_wait && strcmp(name, "kswapd_skip_congestion_wait") == 0) kswapd_skip_congestion_wait = strtoull(value, NULL, 10); - // else if(hash == hash_nr_active_anon && strcmp(name, "nr_active_anon") == 0) nr_active_anon = strtoull(value, NULL, 10); - // else if(hash == hash_nr_active_file && strcmp(name, "nr_active_file") == 0) nr_active_file = strtoull(value, NULL, 10); - // else if(hash == hash_nr_anon_pages && strcmp(name, "nr_anon_pages") == 0) nr_anon_pages = strtoull(value, NULL, 10); - // else if(hash == hash_nr_anon_transparent_hugepages && strcmp(name, "nr_anon_transparent_hugepages") == 0) nr_anon_transparent_hugepages = strtoull(value, NULL, 10); - // else if(hash == hash_nr_bounce && strcmp(name, "nr_bounce") == 0) nr_bounce = strtoull(value, NULL, 10); - // else if(hash == hash_nr_dirtied && strcmp(name, "nr_dirtied") == 0) nr_dirtied = strtoull(value, NULL, 10); - // else if(hash == hash_nr_dirty && strcmp(name, "nr_dirty") == 0) nr_dirty = strtoull(value, NULL, 10); - // else if(hash == hash_nr_dirty_background_threshold && strcmp(name, "nr_dirty_background_threshold") == 0) nr_dirty_background_threshold = strtoull(value, NULL, 10); - // else if(hash == hash_nr_dirty_threshold && strcmp(name, "nr_dirty_threshold") == 0) nr_dirty_threshold = strtoull(value, NULL, 10); - // else if(hash == hash_nr_file_pages && strcmp(name, "nr_file_pages") == 0) nr_file_pages = strtoull(value, NULL, 10); - // else if(hash == hash_nr_free_pages && strcmp(name, "nr_free_pages") == 0) nr_free_pages = strtoull(value, NULL, 10); - // else if(hash == hash_nr_inactive_anon && strcmp(name, "nr_inactive_anon") == 0) nr_inactive_anon = strtoull(value, NULL, 10); - // else if(hash == hash_nr_inactive_file && strcmp(name, "nr_inactive_file") == 0) nr_inactive_file = strtoull(value, NULL, 10); - // else if(hash == hash_nr_isolated_anon && strcmp(name, "nr_isolated_anon") == 0) nr_isolated_anon = strtoull(value, NULL, 10); - // else if(hash == hash_nr_isolated_file && strcmp(name, "nr_isolated_file") == 0) nr_isolated_file = strtoull(value, NULL, 10); - // else if(hash == hash_nr_kernel_stack && strcmp(name, "nr_kernel_stack") == 0) nr_kernel_stack = strtoull(value, NULL, 10); - // else if(hash == hash_nr_mapped && strcmp(name, "nr_mapped") == 0) nr_mapped = strtoull(value, NULL, 10); - // else if(hash == hash_nr_mlock && strcmp(name, "nr_mlock") == 0) nr_mlock = strtoull(value, NULL, 10); - // else if(hash == hash_nr_page_table_pages && strcmp(name, "nr_page_table_pages") == 0) nr_page_table_pages = strtoull(value, NULL, 10); - // else if(hash == hash_nr_shmem && strcmp(name, "nr_shmem") == 0) nr_shmem = strtoull(value, NULL, 10); - // else if(hash == hash_nr_slab_reclaimable && strcmp(name, "nr_slab_reclaimable") == 0) nr_slab_reclaimable = strtoull(value, NULL, 10); - // else if(hash == hash_nr_slab_unreclaimable && strcmp(name, "nr_slab_unreclaimable") == 0) nr_slab_unreclaimable = strtoull(value, NULL, 10); - // else if(hash == hash_nr_unevictable && strcmp(name, "nr_unevictable") == 0) nr_unevictable = strtoull(value, NULL, 10); - // else if(hash == hash_nr_unstable && strcmp(name, "nr_unstable") == 0) nr_unstable = strtoull(value, NULL, 10); - // else if(hash == hash_nr_vmscan_immediate_reclaim && strcmp(name, "nr_vmscan_immediate_reclaim") == 0) nr_vmscan_immediate_reclaim = strtoull(value, NULL, 10); - // else if(hash == hash_nr_vmscan_write && strcmp(name, "nr_vmscan_write") == 0) nr_vmscan_write = strtoull(value, NULL, 10); - // else if(hash == hash_nr_writeback && strcmp(name, "nr_writeback") == 0) nr_writeback = strtoull(value, NULL, 10); - // else if(hash == hash_nr_writeback_temp && strcmp(name, "nr_writeback_temp") == 0) nr_writeback_temp = strtoull(value, NULL, 10); - // else if(hash == hash_nr_written && strcmp(name, "nr_written") == 0) nr_written = strtoull(value, NULL, 10); - // else if(hash == hash_pageoutrun && strcmp(name, "pageoutrun") == 0) pageoutrun = strtoull(value, NULL, 10); - // else if(hash == hash_pgactivate && strcmp(name, "pgactivate") == 0) pgactivate = strtoull(value, NULL, 10); - // else if(hash == hash_pgalloc_dma && strcmp(name, "pgalloc_dma") == 0) pgalloc_dma = strtoull(value, NULL, 10); - // else if(hash == hash_pgalloc_dma32 && strcmp(name, "pgalloc_dma32") == 0) pgalloc_dma32 = strtoull(value, NULL, 10); - // else if(hash == hash_pgalloc_movable && strcmp(name, "pgalloc_movable") == 0) pgalloc_movable = strtoull(value, NULL, 10); - // else if(hash == hash_pgalloc_normal && strcmp(name, "pgalloc_normal") == 0) pgalloc_normal = strtoull(value, NULL, 10); - // else if(hash == hash_pgdeactivate && strcmp(name, "pgdeactivate") == 0) pgdeactivate = strtoull(value, NULL, 10); - else if(hash == hash_pgfault && strcmp(name, "pgfault") == 0) pgfault = strtoull(value, NULL, 10); - // else if(hash == hash_pgfree && strcmp(name, "pgfree") == 0) pgfree = strtoull(value, NULL, 10); - // else if(hash == hash_pginodesteal && strcmp(name, "pginodesteal") == 0) pginodesteal = strtoull(value, NULL, 10); - else if(hash == hash_pgmajfault && strcmp(name, "pgmajfault") == 0) pgmajfault = strtoull(value, NULL, 10); - else if(hash == hash_pgpgin && strcmp(name, "pgpgin") == 0) pgpgin = strtoull(value, NULL, 10); - else if(hash == hash_pgpgout && strcmp(name, "pgpgout") == 0) pgpgout = strtoull(value, NULL, 10); - // else if(hash == hash_pgrefill_dma && strcmp(name, "pgrefill_dma") == 0) pgrefill_dma = strtoull(value, NULL, 10); - // else if(hash == hash_pgrefill_dma32 && strcmp(name, "pgrefill_dma32") == 0) pgrefill_dma32 = strtoull(value, NULL, 10); - // else if(hash == hash_pgrefill_movable && strcmp(name, "pgrefill_movable") == 0) pgrefill_movable = strtoull(value, NULL, 10); - // else if(hash == hash_pgrefill_normal && strcmp(name, "pgrefill_normal") == 0) pgrefill_normal = strtoull(value, NULL, 10); - // else if(hash == hash_pgrotated && strcmp(name, "pgrotated") == 0) pgrotated = strtoull(value, NULL, 10); - // else if(hash == hash_pgscan_direct_dma && strcmp(name, "pgscan_direct_dma") == 0) pgscan_direct_dma = strtoull(value, NULL, 10); - // else if(hash == hash_pgscan_direct_dma32 && strcmp(name, "pgscan_direct_dma32") == 0) pgscan_direct_dma32 = strtoull(value, NULL, 10); - // else if(hash == hash_pgscan_direct_movable && strcmp(name, "pgscan_direct_movable") == 0) pgscan_direct_movable = strtoull(value, NULL, 10); - // else if(hash == hash_pgscan_direct_normal && strcmp(name, "pgscan_direct_normal") == 0) pgscan_direct_normal = strtoull(value, NULL, 10); - // else if(hash == hash_pgscan_kswapd_dma && strcmp(name, "pgscan_kswapd_dma") == 0) pgscan_kswapd_dma = strtoull(value, NULL, 10); - // else if(hash == hash_pgscan_kswapd_dma32 && strcmp(name, "pgscan_kswapd_dma32") == 0) pgscan_kswapd_dma32 = strtoull(value, NULL, 10); - // else if(hash == hash_pgscan_kswapd_movable && strcmp(name, "pgscan_kswapd_movable") == 0) pgscan_kswapd_movable = strtoull(value, NULL, 10); - // else if(hash == hash_pgscan_kswapd_normal && strcmp(name, "pgscan_kswapd_normal") == 0) pgscan_kswapd_normal = strtoull(value, NULL, 10); - // else if(hash == hash_pgsteal_direct_dma && strcmp(name, "pgsteal_direct_dma") == 0) pgsteal_direct_dma = strtoull(value, NULL, 10); - // else if(hash == hash_pgsteal_direct_dma32 && strcmp(name, "pgsteal_direct_dma32") == 0) pgsteal_direct_dma32 = strtoull(value, NULL, 10); - // else if(hash == hash_pgsteal_direct_movable && strcmp(name, "pgsteal_direct_movable") == 0) pgsteal_direct_movable = strtoull(value, NULL, 10); - // else if(hash == hash_pgsteal_direct_normal && strcmp(name, "pgsteal_direct_normal") == 0) pgsteal_direct_normal = strtoull(value, NULL, 10); - // else if(hash == hash_pgsteal_kswapd_dma && strcmp(name, "pgsteal_kswapd_dma") == 0) pgsteal_kswapd_dma = strtoull(value, NULL, 10); - // else if(hash == hash_pgsteal_kswapd_dma32 && strcmp(name, "pgsteal_kswapd_dma32") == 0) pgsteal_kswapd_dma32 = strtoull(value, NULL, 10); - // else if(hash == hash_pgsteal_kswapd_movable && strcmp(name, "pgsteal_kswapd_movable") == 0) pgsteal_kswapd_movable = strtoull(value, NULL, 10); - // else if(hash == hash_pgsteal_kswapd_normal && strcmp(name, "pgsteal_kswapd_normal") == 0) pgsteal_kswapd_normal = strtoull(value, NULL, 10); - else if(hash == hash_pswpin && strcmp(name, "pswpin") == 0) pswpin = strtoull(value, NULL, 10); - else if(hash == hash_pswpout && strcmp(name, "pswpout") == 0) pswpout = strtoull(value, NULL, 10); - // else if(hash == hash_slabs_scanned && strcmp(name, "slabs_scanned") == 0) slabs_scanned = strtoull(value, NULL, 10); - // else if(hash == hash_thp_collapse_alloc && strcmp(name, "thp_collapse_alloc") == 0) thp_collapse_alloc = strtoull(value, NULL, 10); - // else if(hash == hash_thp_collapse_alloc_failed && strcmp(name, "thp_collapse_alloc_failed") == 0) thp_collapse_alloc_failed = strtoull(value, NULL, 10); - // else if(hash == hash_thp_fault_alloc && strcmp(name, "thp_fault_alloc") == 0) thp_fault_alloc = strtoull(value, NULL, 10); - // else if(hash == hash_thp_fault_fallback && strcmp(name, "thp_fault_fallback") == 0) thp_fault_fallback = strtoull(value, NULL, 10); - // else if(hash == hash_thp_split && strcmp(name, "thp_split") == 0) thp_split = strtoull(value, NULL, 10); - // else if(hash == hash_unevictable_pgs_cleared && strcmp(name, "unevictable_pgs_cleared") == 0) unevictable_pgs_cleared = strtoull(value, NULL, 10); - // else if(hash == hash_unevictable_pgs_culled && strcmp(name, "unevictable_pgs_culled") == 0) unevictable_pgs_culled = strtoull(value, NULL, 10); - // else if(hash == hash_unevictable_pgs_mlocked && strcmp(name, "unevictable_pgs_mlocked") == 0) unevictable_pgs_mlocked = strtoull(value, NULL, 10); - // else if(hash == hash_unevictable_pgs_mlockfreed && strcmp(name, "unevictable_pgs_mlockfreed") == 0) unevictable_pgs_mlockfreed = strtoull(value, NULL, 10); - // else if(hash == hash_unevictable_pgs_munlocked && strcmp(name, "unevictable_pgs_munlocked") == 0) unevictable_pgs_munlocked = strtoull(value, NULL, 10); - // else if(hash == hash_unevictable_pgs_rescued && strcmp(name, "unevictable_pgs_rescued") == 0) unevictable_pgs_rescued = strtoull(value, NULL, 10); - // else if(hash == hash_unevictable_pgs_scanned && strcmp(name, "unevictable_pgs_scanned") == 0) unevictable_pgs_scanned = strtoull(value, NULL, 10); - // else if(hash == hash_unevictable_pgs_stranded && strcmp(name, "unevictable_pgs_stranded") == 0) unevictable_pgs_stranded = strtoull(value, NULL, 10); - } + if(0) ; + // else if(hash == hash_allocstall && strcmp(name, "allocstall") == 0) allocstall = strtoull(value, NULL, 10); + // else if(hash == hash_compact_blocks_moved && strcmp(name, "compact_blocks_moved") == 0) compact_blocks_moved = strtoull(value, NULL, 10); + // else if(hash == hash_compact_fail && strcmp(name, "compact_fail") == 0) compact_fail = strtoull(value, NULL, 10); + // else if(hash == hash_compact_pagemigrate_failed && strcmp(name, "compact_pagemigrate_failed") == 0) compact_pagemigrate_failed = strtoull(value, NULL, 10); + // else if(hash == hash_compact_pages_moved && strcmp(name, "compact_pages_moved") == 0) compact_pages_moved = strtoull(value, NULL, 10); + // else if(hash == hash_compact_stall && strcmp(name, "compact_stall") == 0) compact_stall = strtoull(value, NULL, 10); + // else if(hash == hash_compact_success && strcmp(name, "compact_success") == 0) compact_success = strtoull(value, NULL, 10); + // else if(hash == hash_htlb_buddy_alloc_fail && strcmp(name, "htlb_buddy_alloc_fail") == 0) htlb_buddy_alloc_fail = strtoull(value, NULL, 10); + // else if(hash == hash_htlb_buddy_alloc_success && strcmp(name, "htlb_buddy_alloc_success") == 0) htlb_buddy_alloc_success = strtoull(value, NULL, 10); + // else if(hash == hash_kswapd_high_wmark_hit_quickly && strcmp(name, "kswapd_high_wmark_hit_quickly") == 0) kswapd_high_wmark_hit_quickly = strtoull(value, NULL, 10); + // else if(hash == hash_kswapd_inodesteal && strcmp(name, "kswapd_inodesteal") == 0) kswapd_inodesteal = strtoull(value, NULL, 10); + // else if(hash == hash_kswapd_low_wmark_hit_quickly && strcmp(name, "kswapd_low_wmark_hit_quickly") == 0) kswapd_low_wmark_hit_quickly = strtoull(value, NULL, 10); + // else if(hash == hash_kswapd_skip_congestion_wait && strcmp(name, "kswapd_skip_congestion_wait") == 0) kswapd_skip_congestion_wait = strtoull(value, NULL, 10); + // else if(hash == hash_nr_active_anon && strcmp(name, "nr_active_anon") == 0) nr_active_anon = strtoull(value, NULL, 10); + // else if(hash == hash_nr_active_file && strcmp(name, "nr_active_file") == 0) nr_active_file = strtoull(value, NULL, 10); + // else if(hash == hash_nr_anon_pages && strcmp(name, "nr_anon_pages") == 0) nr_anon_pages = strtoull(value, NULL, 10); + // else if(hash == hash_nr_anon_transparent_hugepages && strcmp(name, "nr_anon_transparent_hugepages") == 0) nr_anon_transparent_hugepages = strtoull(value, NULL, 10); + // else if(hash == hash_nr_bounce && strcmp(name, "nr_bounce") == 0) nr_bounce = strtoull(value, NULL, 10); + // else if(hash == hash_nr_dirtied && strcmp(name, "nr_dirtied") == 0) nr_dirtied = strtoull(value, NULL, 10); + // else if(hash == hash_nr_dirty && strcmp(name, "nr_dirty") == 0) nr_dirty = strtoull(value, NULL, 10); + // else if(hash == hash_nr_dirty_background_threshold && strcmp(name, "nr_dirty_background_threshold") == 0) nr_dirty_background_threshold = strtoull(value, NULL, 10); + // else if(hash == hash_nr_dirty_threshold && strcmp(name, "nr_dirty_threshold") == 0) nr_dirty_threshold = strtoull(value, NULL, 10); + // else if(hash == hash_nr_file_pages && strcmp(name, "nr_file_pages") == 0) nr_file_pages = strtoull(value, NULL, 10); + // else if(hash == hash_nr_free_pages && strcmp(name, "nr_free_pages") == 0) nr_free_pages = strtoull(value, NULL, 10); + // else if(hash == hash_nr_inactive_anon && strcmp(name, "nr_inactive_anon") == 0) nr_inactive_anon = strtoull(value, NULL, 10); + // else if(hash == hash_nr_inactive_file && strcmp(name, "nr_inactive_file") == 0) nr_inactive_file = strtoull(value, NULL, 10); + // else if(hash == hash_nr_isolated_anon && strcmp(name, "nr_isolated_anon") == 0) nr_isolated_anon = strtoull(value, NULL, 10); + // else if(hash == hash_nr_isolated_file && strcmp(name, "nr_isolated_file") == 0) nr_isolated_file = strtoull(value, NULL, 10); + // else if(hash == hash_nr_kernel_stack && strcmp(name, "nr_kernel_stack") == 0) nr_kernel_stack = strtoull(value, NULL, 10); + // else if(hash == hash_nr_mapped && strcmp(name, "nr_mapped") == 0) nr_mapped = strtoull(value, NULL, 10); + // else if(hash == hash_nr_mlock && strcmp(name, "nr_mlock") == 0) nr_mlock = strtoull(value, NULL, 10); + // else if(hash == hash_nr_page_table_pages && strcmp(name, "nr_page_table_pages") == 0) nr_page_table_pages = strtoull(value, NULL, 10); + // else if(hash == hash_nr_shmem && strcmp(name, "nr_shmem") == 0) nr_shmem = strtoull(value, NULL, 10); + // else if(hash == hash_nr_slab_reclaimable && strcmp(name, "nr_slab_reclaimable") == 0) nr_slab_reclaimable = strtoull(value, NULL, 10); + // else if(hash == hash_nr_slab_unreclaimable && strcmp(name, "nr_slab_unreclaimable") == 0) nr_slab_unreclaimable = strtoull(value, NULL, 10); + // else if(hash == hash_nr_unevictable && strcmp(name, "nr_unevictable") == 0) nr_unevictable = strtoull(value, NULL, 10); + // else if(hash == hash_nr_unstable && strcmp(name, "nr_unstable") == 0) nr_unstable = strtoull(value, NULL, 10); + // else if(hash == hash_nr_vmscan_immediate_reclaim && strcmp(name, "nr_vmscan_immediate_reclaim") == 0) nr_vmscan_immediate_reclaim = strtoull(value, NULL, 10); + // else if(hash == hash_nr_vmscan_write && strcmp(name, "nr_vmscan_write") == 0) nr_vmscan_write = strtoull(value, NULL, 10); + // else if(hash == hash_nr_writeback && strcmp(name, "nr_writeback") == 0) nr_writeback = strtoull(value, NULL, 10); + // else if(hash == hash_nr_writeback_temp && strcmp(name, "nr_writeback_temp") == 0) nr_writeback_temp = strtoull(value, NULL, 10); + // else if(hash == hash_nr_written && strcmp(name, "nr_written") == 0) nr_written = strtoull(value, NULL, 10); + // else if(hash == hash_pageoutrun && strcmp(name, "pageoutrun") == 0) pageoutrun = strtoull(value, NULL, 10); + // else if(hash == hash_pgactivate && strcmp(name, "pgactivate") == 0) pgactivate = strtoull(value, NULL, 10); + // else if(hash == hash_pgalloc_dma && strcmp(name, "pgalloc_dma") == 0) pgalloc_dma = strtoull(value, NULL, 10); + // else if(hash == hash_pgalloc_dma32 && strcmp(name, "pgalloc_dma32") == 0) pgalloc_dma32 = strtoull(value, NULL, 10); + // else if(hash == hash_pgalloc_movable && strcmp(name, "pgalloc_movable") == 0) pgalloc_movable = strtoull(value, NULL, 10); + // else if(hash == hash_pgalloc_normal && strcmp(name, "pgalloc_normal") == 0) pgalloc_normal = strtoull(value, NULL, 10); + // else if(hash == hash_pgdeactivate && strcmp(name, "pgdeactivate") == 0) pgdeactivate = strtoull(value, NULL, 10); + else if(hash == hash_pgfault && strcmp(name, "pgfault") == 0) pgfault = strtoull(value, NULL, 10); + // else if(hash == hash_pgfree && strcmp(name, "pgfree") == 0) pgfree = strtoull(value, NULL, 10); + // else if(hash == hash_pginodesteal && strcmp(name, "pginodesteal") == 0) pginodesteal = strtoull(value, NULL, 10); + else if(hash == hash_pgmajfault && strcmp(name, "pgmajfault") == 0) pgmajfault = strtoull(value, NULL, 10); + else if(hash == hash_pgpgin && strcmp(name, "pgpgin") == 0) pgpgin = strtoull(value, NULL, 10); + else if(hash == hash_pgpgout && strcmp(name, "pgpgout") == 0) pgpgout = strtoull(value, NULL, 10); + // else if(hash == hash_pgrefill_dma && strcmp(name, "pgrefill_dma") == 0) pgrefill_dma = strtoull(value, NULL, 10); + // else if(hash == hash_pgrefill_dma32 && strcmp(name, "pgrefill_dma32") == 0) pgrefill_dma32 = strtoull(value, NULL, 10); + // else if(hash == hash_pgrefill_movable && strcmp(name, "pgrefill_movable") == 0) pgrefill_movable = strtoull(value, NULL, 10); + // else if(hash == hash_pgrefill_normal && strcmp(name, "pgrefill_normal") == 0) pgrefill_normal = strtoull(value, NULL, 10); + // else if(hash == hash_pgrotated && strcmp(name, "pgrotated") == 0) pgrotated = strtoull(value, NULL, 10); + // else if(hash == hash_pgscan_direct_dma && strcmp(name, "pgscan_direct_dma") == 0) pgscan_direct_dma = strtoull(value, NULL, 10); + // else if(hash == hash_pgscan_direct_dma32 && strcmp(name, "pgscan_direct_dma32") == 0) pgscan_direct_dma32 = strtoull(value, NULL, 10); + // else if(hash == hash_pgscan_direct_movable && strcmp(name, "pgscan_direct_movable") == 0) pgscan_direct_movable = strtoull(value, NULL, 10); + // else if(hash == hash_pgscan_direct_normal && strcmp(name, "pgscan_direct_normal") == 0) pgscan_direct_normal = strtoull(value, NULL, 10); + // else if(hash == hash_pgscan_kswapd_dma && strcmp(name, "pgscan_kswapd_dma") == 0) pgscan_kswapd_dma = strtoull(value, NULL, 10); + // else if(hash == hash_pgscan_kswapd_dma32 && strcmp(name, "pgscan_kswapd_dma32") == 0) pgscan_kswapd_dma32 = strtoull(value, NULL, 10); + // else if(hash == hash_pgscan_kswapd_movable && strcmp(name, "pgscan_kswapd_movable") == 0) pgscan_kswapd_movable = strtoull(value, NULL, 10); + // else if(hash == hash_pgscan_kswapd_normal && strcmp(name, "pgscan_kswapd_normal") == 0) pgscan_kswapd_normal = strtoull(value, NULL, 10); + // else if(hash == hash_pgsteal_direct_dma && strcmp(name, "pgsteal_direct_dma") == 0) pgsteal_direct_dma = strtoull(value, NULL, 10); + // else if(hash == hash_pgsteal_direct_dma32 && strcmp(name, "pgsteal_direct_dma32") == 0) pgsteal_direct_dma32 = strtoull(value, NULL, 10); + // else if(hash == hash_pgsteal_direct_movable && strcmp(name, "pgsteal_direct_movable") == 0) pgsteal_direct_movable = strtoull(value, NULL, 10); + // else if(hash == hash_pgsteal_direct_normal && strcmp(name, "pgsteal_direct_normal") == 0) pgsteal_direct_normal = strtoull(value, NULL, 10); + // else if(hash == hash_pgsteal_kswapd_dma && strcmp(name, "pgsteal_kswapd_dma") == 0) pgsteal_kswapd_dma = strtoull(value, NULL, 10); + // else if(hash == hash_pgsteal_kswapd_dma32 && strcmp(name, "pgsteal_kswapd_dma32") == 0) pgsteal_kswapd_dma32 = strtoull(value, NULL, 10); + // else if(hash == hash_pgsteal_kswapd_movable && strcmp(name, "pgsteal_kswapd_movable") == 0) pgsteal_kswapd_movable = strtoull(value, NULL, 10); + // else if(hash == hash_pgsteal_kswapd_normal && strcmp(name, "pgsteal_kswapd_normal") == 0) pgsteal_kswapd_normal = strtoull(value, NULL, 10); + else if(hash == hash_pswpin && strcmp(name, "pswpin") == 0) pswpin = strtoull(value, NULL, 10); + else if(hash == hash_pswpout && strcmp(name, "pswpout") == 0) pswpout = strtoull(value, NULL, 10); + // else if(hash == hash_slabs_scanned && strcmp(name, "slabs_scanned") == 0) slabs_scanned = strtoull(value, NULL, 10); + // else if(hash == hash_thp_collapse_alloc && strcmp(name, "thp_collapse_alloc") == 0) thp_collapse_alloc = strtoull(value, NULL, 10); + // else if(hash == hash_thp_collapse_alloc_failed && strcmp(name, "thp_collapse_alloc_failed") == 0) thp_collapse_alloc_failed = strtoull(value, NULL, 10); + // else if(hash == hash_thp_fault_alloc && strcmp(name, "thp_fault_alloc") == 0) thp_fault_alloc = strtoull(value, NULL, 10); + // else if(hash == hash_thp_fault_fallback && strcmp(name, "thp_fault_fallback") == 0) thp_fault_fallback = strtoull(value, NULL, 10); + // else if(hash == hash_thp_split && strcmp(name, "thp_split") == 0) thp_split = strtoull(value, NULL, 10); + // else if(hash == hash_unevictable_pgs_cleared && strcmp(name, "unevictable_pgs_cleared") == 0) unevictable_pgs_cleared = strtoull(value, NULL, 10); + // else if(hash == hash_unevictable_pgs_culled && strcmp(name, "unevictable_pgs_culled") == 0) unevictable_pgs_culled = strtoull(value, NULL, 10); + // else if(hash == hash_unevictable_pgs_mlocked && strcmp(name, "unevictable_pgs_mlocked") == 0) unevictable_pgs_mlocked = strtoull(value, NULL, 10); + // else if(hash == hash_unevictable_pgs_mlockfreed && strcmp(name, "unevictable_pgs_mlockfreed") == 0) unevictable_pgs_mlockfreed = strtoull(value, NULL, 10); + // else if(hash == hash_unevictable_pgs_munlocked && strcmp(name, "unevictable_pgs_munlocked") == 0) unevictable_pgs_munlocked = strtoull(value, NULL, 10); + // else if(hash == hash_unevictable_pgs_rescued && strcmp(name, "unevictable_pgs_rescued") == 0) unevictable_pgs_rescued = strtoull(value, NULL, 10); + // else if(hash == hash_unevictable_pgs_scanned && strcmp(name, "unevictable_pgs_scanned") == 0) unevictable_pgs_scanned = strtoull(value, NULL, 10); + // else if(hash == hash_unevictable_pgs_stranded && strcmp(name, "unevictable_pgs_stranded") == 0) unevictable_pgs_stranded = strtoull(value, NULL, 10); + } - // -------------------------------------------------------------------- + // -------------------------------------------------------------------- - if(do_swapio) { - static RRDSET *st_swapio = NULL; - if(!st_swapio) { - st_swapio = rrdset_create("system", "swapio", NULL, "swap", NULL, "Swap I/O", "kilobytes/s", 250, update_every, RRDSET_TYPE_AREA); + if(do_swapio) { + static RRDSET *st_swapio = NULL; + if(!st_swapio) { + st_swapio = rrdset_create("system", "swapio", NULL, "swap", NULL, "Swap I/O", "kilobytes/s", 250, update_every, RRDSET_TYPE_AREA); - rrddim_add(st_swapio, "in", NULL, sysconf(_SC_PAGESIZE), 1024, RRDDIM_INCREMENTAL); - rrddim_add(st_swapio, "out", NULL, -sysconf(_SC_PAGESIZE), 1024, RRDDIM_INCREMENTAL); - } - else rrdset_next(st_swapio); + rrddim_add(st_swapio, "in", NULL, sysconf(_SC_PAGESIZE), 1024, RRDDIM_INCREMENTAL); + rrddim_add(st_swapio, "out", NULL, -sysconf(_SC_PAGESIZE), 1024, RRDDIM_INCREMENTAL); + } + else rrdset_next(st_swapio); - rrddim_set(st_swapio, "in", pswpin); - rrddim_set(st_swapio, "out", pswpout); - rrdset_done(st_swapio); - } + rrddim_set(st_swapio, "in", pswpin); + rrddim_set(st_swapio, "out", pswpout); + rrdset_done(st_swapio); + } - // -------------------------------------------------------------------- + // -------------------------------------------------------------------- - if(do_io) { - static RRDSET *st_io = NULL; - if(!st_io) { - st_io = rrdset_create("system", "io", NULL, "disk", NULL, "Disk I/O", "kilobytes/s", 150, update_every, RRDSET_TYPE_AREA); + if(do_io) { + static RRDSET *st_io = NULL; + if(!st_io) { + st_io = rrdset_create("system", "io", NULL, "disk", NULL, "Disk I/O", "kilobytes/s", 150, update_every, RRDSET_TYPE_AREA); - rrddim_add(st_io, "in", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st_io, "out", NULL, -1, 1, RRDDIM_INCREMENTAL); - } - else rrdset_next(st_io); + rrddim_add(st_io, "in", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st_io, "out", NULL, -1, 1, RRDDIM_INCREMENTAL); + } + else rrdset_next(st_io); - rrddim_set(st_io, "in", pgpgin); - rrddim_set(st_io, "out", pgpgout); - rrdset_done(st_io); - } + rrddim_set(st_io, "in", pgpgin); + rrddim_set(st_io, "out", pgpgout); + rrdset_done(st_io); + } - // -------------------------------------------------------------------- + // -------------------------------------------------------------------- - if(do_pgfaults) { - static RRDSET *st_pgfaults = NULL; - if(!st_pgfaults) { - st_pgfaults = rrdset_create("mem", "pgfaults", NULL, "system", NULL, "Memory Page Faults", "page faults/s", 500, update_every, RRDSET_TYPE_LINE); - st_pgfaults->isdetail = 1; + if(do_pgfaults) { + static RRDSET *st_pgfaults = NULL; + if(!st_pgfaults) { + st_pgfaults = rrdset_create("mem", "pgfaults", NULL, "system", NULL, "Memory Page Faults", "page faults/s", 500, update_every, RRDSET_TYPE_LINE); + st_pgfaults->isdetail = 1; - rrddim_add(st_pgfaults, "minor", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st_pgfaults, "major", NULL, -1, 1, RRDDIM_INCREMENTAL); - } - else rrdset_next(st_pgfaults); + rrddim_add(st_pgfaults, "minor", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st_pgfaults, "major", NULL, -1, 1, RRDDIM_INCREMENTAL); + } + else rrdset_next(st_pgfaults); - rrddim_set(st_pgfaults, "minor", pgfault); - rrddim_set(st_pgfaults, "major", pgmajfault); - rrdset_done(st_pgfaults); - } + rrddim_set(st_pgfaults, "minor", pgfault); + rrddim_set(st_pgfaults, "major", pgmajfault); + rrdset_done(st_pgfaults); + } - return 0; + return 0; } diff --git a/src/procfile.c b/src/procfile.c index 291f14519..e2aa60582 100644 --- a/src/procfile.c +++ b/src/procfile.c @@ -1,26 +1,4 @@ -#ifdef HAVE_CONFIG_H -#include <config.h> -#endif -#include <stdio.h> -#include <stdlib.h> -#include <unistd.h> -#include <errno.h> -#include <fcntl.h> -#include <string.h> -#include <malloc.h> -#include <ctype.h> -#include <time.h> -#include <sys/time.h> -#include <sys/wait.h> - -#include <sys/types.h> -#include <sys/stat.h> -#include <sys/mman.h> - #include "common.h" -#include "log.h" -#include "procfile.h" -#include "../config.h" #define PF_PREFIX "PROCFILE" @@ -41,48 +19,40 @@ size_t procfile_max_allocation = PROCFILE_INCREMENT_BUFFER; pfwords *pfwords_add(pfwords *fw, char *str) { - // debug(D_PROCFILE, PF_PREFIX ": adding word No %d: '%s'", fw->len, str); + // debug(D_PROCFILE, PF_PREFIX ": adding word No %d: '%s'", fw->len, str); - if(unlikely(fw->len == fw->size)) { - // debug(D_PROCFILE, PF_PREFIX ": expanding words"); + if(unlikely(fw->len == fw->size)) { + // debug(D_PROCFILE, PF_PREFIX ": expanding words"); - pfwords *new = realloc(fw, sizeof(pfwords) + (fw->size + PFWORDS_INCREASE_STEP) * sizeof(char *)); - if(unlikely(!new)) { - error(PF_PREFIX ": failed to expand words"); - free(fw); - return NULL; - } - fw = new; - fw->size += PFWORDS_INCREASE_STEP; - } + fw = reallocz(fw, sizeof(pfwords) + (fw->size + PFWORDS_INCREASE_STEP) * sizeof(char *)); + fw->size += PFWORDS_INCREASE_STEP; + } - fw->words[fw->len++] = str; + fw->words[fw->len++] = str; - return fw; + return fw; } pfwords *pfwords_new(void) { - // debug(D_PROCFILE, PF_PREFIX ": initializing words"); - - uint32_t size = (procfile_adaptive_initial_allocation) ? procfile_max_words : PFWORDS_INCREASE_STEP; + // debug(D_PROCFILE, PF_PREFIX ": initializing words"); - pfwords *new = malloc(sizeof(pfwords) + size * sizeof(char *)); - if(unlikely(!new)) return NULL; + uint32_t size = (procfile_adaptive_initial_allocation) ? procfile_max_words : PFWORDS_INCREASE_STEP; - new->len = 0; - new->size = size; - return new; + pfwords *new = mallocz(sizeof(pfwords) + size * sizeof(char *)); + new->len = 0; + new->size = size; + return new; } void pfwords_reset(pfwords *fw) { - // debug(D_PROCFILE, PF_PREFIX ": reseting words"); - fw->len = 0; + // debug(D_PROCFILE, PF_PREFIX ": reseting words"); + fw->len = 0; } void pfwords_free(pfwords *fw) { - // debug(D_PROCFILE, PF_PREFIX ": freeing words"); + // debug(D_PROCFILE, PF_PREFIX ": freeing words"); - free(fw); + freez(fw); } @@ -90,422 +60,402 @@ void pfwords_free(pfwords *fw) { // An array of lines pflines *pflines_add(pflines *fl, uint32_t first_word) { - // debug(D_PROCFILE, PF_PREFIX ": adding line %d at word %d", fl->len, first_word); + // debug(D_PROCFILE, PF_PREFIX ": adding line %d at word %d", fl->len, first_word); - if(unlikely(fl->len == fl->size)) { - // debug(D_PROCFILE, PF_PREFIX ": expanding lines"); + if(unlikely(fl->len == fl->size)) { + // debug(D_PROCFILE, PF_PREFIX ": expanding lines"); - pflines *new = realloc(fl, sizeof(pflines) + (fl->size + PFLINES_INCREASE_STEP) * sizeof(ffline)); - if(unlikely(!new)) { - error(PF_PREFIX ": failed to expand lines"); - free(fl); - return NULL; - } - fl = new; - fl->size += PFLINES_INCREASE_STEP; - } + fl = reallocz(fl, sizeof(pflines) + (fl->size + PFLINES_INCREASE_STEP) * sizeof(ffline)); + fl->size += PFLINES_INCREASE_STEP; + } - fl->lines[fl->len].words = 0; - fl->lines[fl->len++].first = first_word; + fl->lines[fl->len].words = 0; + fl->lines[fl->len++].first = first_word; - return fl; + return fl; } pflines *pflines_new(void) { - // debug(D_PROCFILE, PF_PREFIX ": initializing lines"); - - uint32_t size = (unlikely(procfile_adaptive_initial_allocation)) ? procfile_max_words : PFLINES_INCREASE_STEP; + // debug(D_PROCFILE, PF_PREFIX ": initializing lines"); - pflines *new = malloc(sizeof(pflines) + size * sizeof(ffline)); - if(unlikely(!new)) return NULL; + uint32_t size = (unlikely(procfile_adaptive_initial_allocation)) ? procfile_max_words : PFLINES_INCREASE_STEP; - new->len = 0; - new->size = size; - return new; + pflines *new = mallocz(sizeof(pflines) + size * sizeof(ffline)); + new->len = 0; + new->size = size; + return new; } void pflines_reset(pflines *fl) { - // debug(D_PROCFILE, PF_PREFIX ": reseting lines"); + // debug(D_PROCFILE, PF_PREFIX ": reseting lines"); - fl->len = 0; + fl->len = 0; } void pflines_free(pflines *fl) { - // debug(D_PROCFILE, PF_PREFIX ": freeing lines"); + // debug(D_PROCFILE, PF_PREFIX ": freeing lines"); - free(fl); + freez(fl); } // ---------------------------------------------------------------------------- // The procfile -#define PF_CHAR_IS_SEPARATOR ' ' -#define PF_CHAR_IS_NEWLINE 'N' -#define PF_CHAR_IS_WORD 'W' +#define PF_CHAR_IS_SEPARATOR ' ' +#define PF_CHAR_IS_NEWLINE 'N' +#define PF_CHAR_IS_WORD 'W' #define PF_CHAR_IS_QUOTE 'Q' #define PF_CHAR_IS_OPEN 'O' #define PF_CHAR_IS_CLOSE 'C' void procfile_close(procfile *ff) { - debug(D_PROCFILE, PF_PREFIX ": Closing file '%s'", ff->filename); + debug(D_PROCFILE, PF_PREFIX ": Closing file '%s'", ff->filename); - if(likely(ff->lines)) pflines_free(ff->lines); - if(likely(ff->words)) pfwords_free(ff->words); + if(likely(ff->lines)) pflines_free(ff->lines); + if(likely(ff->words)) pfwords_free(ff->words); - if(likely(ff->fd != -1)) close(ff->fd); - free(ff); + if(likely(ff->fd != -1)) close(ff->fd); + freez(ff); } procfile *procfile_parser(procfile *ff) { - debug(D_PROCFILE, PF_PREFIX ": Parsing file '%s'", ff->filename); - - char *s = ff->data, *e = &ff->data[ff->len], *t = ff->data, quote = 0; - uint32_t l = 0, w = 0; - int opened = 0; - - ff->lines = pflines_add(ff->lines, w); - if(unlikely(!ff->lines)) goto cleanup; - - while(likely(s < e)) { - // we are not at the end - - switch(ff->separators[(int)(*s)]) { - case PF_CHAR_IS_OPEN: - if(s == t) { - opened++; - t = ++s; - } - else if(opened) { - opened++; - s++; - } - else - s++; - continue; - - case PF_CHAR_IS_CLOSE: - if(opened) { - opened--; - - if(!opened) { - *s = '\0'; - ff->words = pfwords_add(ff->words, t); - if(unlikely(!ff->words)) goto cleanup; - - ff->lines->lines[l].words++; - w++; - - t = ++s; - } - else - s++; - } - else - s++; - continue; - - case PF_CHAR_IS_QUOTE: - if(unlikely(!quote && s == t)) { - // quote opened at the beginning - quote = *s; - t = ++s; - } - else if(unlikely(quote && quote == *s)) { - // quote closed - quote = 0; - - *s = '\0'; - ff->words = pfwords_add(ff->words, t); - if(unlikely(!ff->words)) goto cleanup; - - ff->lines->lines[l].words++; - w++; - - t = ++s; - } - else - s++; - continue; - - case PF_CHAR_IS_SEPARATOR: - if(unlikely(quote || opened)) { - // we are inside a quote - s++; - continue; - } - - if(unlikely(s == t)) { - // skip all leading white spaces - t = ++s; - continue; - } - - // end of word - *s = '\0'; - - ff->words = pfwords_add(ff->words, t); - if(unlikely(!ff->words)) goto cleanup; - - ff->lines->lines[l].words++; - w++; - - t = ++s; - continue; - - case PF_CHAR_IS_NEWLINE: - // end of line - *s = '\0'; - - ff->words = pfwords_add(ff->words, t); - if(unlikely(!ff->words)) goto cleanup; - - ff->lines->lines[l].words++; - w++; - - // debug(D_PROCFILE, PF_PREFIX ": ended line %d with %d words", l, ff->lines->lines[l].words); - - ff->lines = pflines_add(ff->lines, w); - if(unlikely(!ff->lines)) goto cleanup; - l++; - - t = ++s; - continue; - - default: - s++; - continue; - } - } - - if(likely(s > t && t < e)) { - // the last word - if(likely(ff->len < ff->size)) - *s = '\0'; - else { - // we are going to loose the last byte - ff->data[ff->size - 1] = '\0'; - } - - ff->words = pfwords_add(ff->words, t); - if(unlikely(!ff->words)) goto cleanup; - - ff->lines->lines[l].words++; - w++; - } - - return ff; + debug(D_PROCFILE, PF_PREFIX ": Parsing file '%s'", ff->filename); + + char *s = ff->data, *e = &ff->data[ff->len], *t = ff->data, quote = 0; + uint32_t l = 0, w = 0; + int opened = 0; + + ff->lines = pflines_add(ff->lines, w); + if(unlikely(!ff->lines)) goto cleanup; + + while(likely(s < e)) { + // we are not at the end + + switch(ff->separators[(uint8_t)(*s)]) { + case PF_CHAR_IS_OPEN: + if(s == t) { + opened++; + t = ++s; + } + else if(opened) { + opened++; + s++; + } + else + s++; + continue; + + case PF_CHAR_IS_CLOSE: + if(opened) { + opened--; + + if(!opened) { + *s = '\0'; + ff->words = pfwords_add(ff->words, t); + if(unlikely(!ff->words)) goto cleanup; + + ff->lines->lines[l].words++; + w++; + + t = ++s; + } + else + s++; + } + else + s++; + continue; + + case PF_CHAR_IS_QUOTE: + if(unlikely(!quote && s == t)) { + // quote opened at the beginning + quote = *s; + t = ++s; + } + else if(unlikely(quote && quote == *s)) { + // quote closed + quote = 0; + + *s = '\0'; + ff->words = pfwords_add(ff->words, t); + if(unlikely(!ff->words)) goto cleanup; + + ff->lines->lines[l].words++; + w++; + + t = ++s; + } + else + s++; + continue; + + case PF_CHAR_IS_SEPARATOR: + if(unlikely(quote || opened)) { + // we are inside a quote + s++; + continue; + } + + if(unlikely(s == t)) { + // skip all leading white spaces + t = ++s; + continue; + } + + // end of word + *s = '\0'; + + ff->words = pfwords_add(ff->words, t); + if(unlikely(!ff->words)) goto cleanup; + + ff->lines->lines[l].words++; + w++; + + t = ++s; + continue; + + case PF_CHAR_IS_NEWLINE: + // end of line + *s = '\0'; + + ff->words = pfwords_add(ff->words, t); + if(unlikely(!ff->words)) goto cleanup; + + ff->lines->lines[l].words++; + w++; + + // debug(D_PROCFILE, PF_PREFIX ": ended line %d with %d words", l, ff->lines->lines[l].words); + + ff->lines = pflines_add(ff->lines, w); + if(unlikely(!ff->lines)) goto cleanup; + l++; + + t = ++s; + continue; + + default: + s++; + continue; + } + } + + if(likely(s > t && t < e)) { + // the last word + if(likely(ff->len < ff->size)) + *s = '\0'; + else { + // we are going to loose the last byte + ff->data[ff->size - 1] = '\0'; + } + + ff->words = pfwords_add(ff->words, t); + if(unlikely(!ff->words)) goto cleanup; + + ff->lines->lines[l].words++; + w++; + } + + return ff; cleanup: - error(PF_PREFIX ": Failed to parse file '%s'", ff->filename); - procfile_close(ff); - return NULL; + error(PF_PREFIX ": Failed to parse file '%s'", ff->filename); + procfile_close(ff); + return NULL; } procfile *procfile_readall(procfile *ff) { - debug(D_PROCFILE, PF_PREFIX ": Reading file '%s'.", ff->filename); - - ssize_t s, r = 1, x; - ff->len = 0; - - while(likely(r > 0)) { - s = ff->len; - x = ff->size - s; - - if(!x) { - debug(D_PROCFILE, PF_PREFIX ": Expanding data buffer for file '%s'.", ff->filename); - - procfile *new = realloc(ff, sizeof(procfile) + ff->size + PROCFILE_INCREMENT_BUFFER); - if(unlikely(!new)) { - error(PF_PREFIX ": Cannot allocate memory for file '%s'", ff->filename); - procfile_close(ff); - return NULL; - } - ff = new; - ff->size += PROCFILE_INCREMENT_BUFFER; - } - - debug(D_PROCFILE, "Reading file '%s', from position %ld with length %ld", ff->filename, s, ff->size - s); - r = read(ff->fd, &ff->data[s], ff->size - s); - if(unlikely(r == -1)) { - if(unlikely(!(ff->flags & PROCFILE_FLAG_NO_ERROR_ON_FILE_IO))) error(PF_PREFIX ": Cannot read from file '%s'", ff->filename); - procfile_close(ff); - return NULL; - } - - ff->len += r; - } - - debug(D_PROCFILE, "Rewinding file '%s'", ff->filename); - if(unlikely(lseek(ff->fd, 0, SEEK_SET) == -1)) { - if(unlikely(!(ff->flags & PROCFILE_FLAG_NO_ERROR_ON_FILE_IO))) error(PF_PREFIX ": Cannot rewind on file '%s'.", ff->filename); - procfile_close(ff); - return NULL; - } - - pflines_reset(ff->lines); - pfwords_reset(ff->words); - - ff = procfile_parser(ff); - - if(unlikely(procfile_adaptive_initial_allocation)) { - if(unlikely(ff->len > procfile_max_allocation)) procfile_max_allocation = ff->len; - if(unlikely(ff->lines->len > procfile_max_lines)) procfile_max_lines = ff->lines->len; - if(unlikely(ff->words->len > procfile_max_words)) procfile_max_words = ff->words->len; - } - - debug(D_PROCFILE, "File '%s' updated.", ff->filename); - return ff; + debug(D_PROCFILE, PF_PREFIX ": Reading file '%s'.", ff->filename); + + ssize_t s, r = 1, x; + ff->len = 0; + + while(likely(r > 0)) { + s = ff->len; + x = ff->size - s; + + if(!x) { + debug(D_PROCFILE, PF_PREFIX ": Expanding data buffer for file '%s'.", ff->filename); + + ff = reallocz(ff, sizeof(procfile) + ff->size + PROCFILE_INCREMENT_BUFFER); + ff->size += PROCFILE_INCREMENT_BUFFER; + } + + debug(D_PROCFILE, "Reading file '%s', from position %ld with length %lu", ff->filename, s, ff->size - s); + r = read(ff->fd, &ff->data[s], ff->size - s); + if(unlikely(r == -1)) { + if(unlikely(!(ff->flags & PROCFILE_FLAG_NO_ERROR_ON_FILE_IO))) error(PF_PREFIX ": Cannot read from file '%s'", ff->filename); + procfile_close(ff); + return NULL; + } + + ff->len += r; + } + + debug(D_PROCFILE, "Rewinding file '%s'", ff->filename); + if(unlikely(lseek(ff->fd, 0, SEEK_SET) == -1)) { + if(unlikely(!(ff->flags & PROCFILE_FLAG_NO_ERROR_ON_FILE_IO))) error(PF_PREFIX ": Cannot rewind on file '%s'.", ff->filename); + procfile_close(ff); + return NULL; + } + + pflines_reset(ff->lines); + pfwords_reset(ff->words); + + ff = procfile_parser(ff); + + if(unlikely(procfile_adaptive_initial_allocation)) { + if(unlikely(ff->len > procfile_max_allocation)) procfile_max_allocation = ff->len; + if(unlikely(ff->lines->len > procfile_max_lines)) procfile_max_lines = ff->lines->len; + if(unlikely(ff->words->len > procfile_max_words)) procfile_max_words = ff->words->len; + } + + debug(D_PROCFILE, "File '%s' updated.", ff->filename); + return ff; } static void procfile_set_separators(procfile *ff, const char *separators) { - static char def[256] = { [0 ... 255] = 0 }; - int i; - - if(unlikely(!def[255])) { - // this is thread safe - // we check that the last byte is non-zero - // if it is zero, multiple threads may be executing this at the same time - // setting in def[] the exact same values - for(i = 0; likely(i < 256) ;i++) { - if(unlikely(i == '\n' || i == '\r')) def[i] = PF_CHAR_IS_NEWLINE; - else if(unlikely(isspace(i) || !isprint(i))) def[i] = PF_CHAR_IS_SEPARATOR; - else def[i] = PF_CHAR_IS_WORD; - } - } - - // copy the default - char *ffs = ff->separators, *ffd = def, *ffe = &def[256]; - while(likely(ffd != ffe)) *ffs++ = *ffd++; - - // set the separators - if(unlikely(!separators)) - separators = " \t=|"; - - ffs = ff->separators; - const char *s = separators; - while(likely(*s)) - ffs[(int)*s++] = PF_CHAR_IS_SEPARATOR; + static char def[256] = { [0 ... 255] = 0 }; + int i; + + if(unlikely(!def[255])) { + // this is thread safe + // we check that the last byte is non-zero + // if it is zero, multiple threads may be executing this at the same time + // setting in def[] the exact same values + for(i = 0; likely(i < 256) ;i++) { + if(unlikely(i == '\n' || i == '\r')) def[i] = PF_CHAR_IS_NEWLINE; + else if(unlikely(isspace(i) || !isprint(i))) def[i] = PF_CHAR_IS_SEPARATOR; + else def[i] = PF_CHAR_IS_WORD; + } + } + + // copy the default + char *ffs = ff->separators, *ffd = def, *ffe = &def[256]; + while(likely(ffd != ffe)) *ffs++ = *ffd++; + + // set the separators + if(unlikely(!separators)) + separators = " \t=|"; + + ffs = ff->separators; + const char *s = separators; + while(likely(*s)) + ffs[(int)*s++] = PF_CHAR_IS_SEPARATOR; } void procfile_set_quotes(procfile *ff, const char *quotes) { - // remove all quotes - int i; - for(i = 0; i < 256 ; i++) - if(ff->separators[i] == PF_CHAR_IS_QUOTE) - ff->separators[i] = PF_CHAR_IS_WORD; - - // if nothing given, return - if(unlikely(!quotes || !*quotes)) - return; - - // set the quotes - char *ffs = ff->separators; - const char *s = quotes; - while(likely(*s)) - ffs[(int)*s++] = PF_CHAR_IS_QUOTE; + // remove all quotes + int i; + for(i = 0; i < 256 ; i++) + if(ff->separators[i] == PF_CHAR_IS_QUOTE) + ff->separators[i] = PF_CHAR_IS_WORD; + + // if nothing given, return + if(unlikely(!quotes || !*quotes)) + return; + + // set the quotes + char *ffs = ff->separators; + const char *s = quotes; + while(likely(*s)) + ffs[(int)*s++] = PF_CHAR_IS_QUOTE; } void procfile_set_open_close(procfile *ff, const char *open, const char *close) { - // remove all open/close - int i; - for(i = 0; i < 256 ; i++) - if(ff->separators[i] == PF_CHAR_IS_OPEN || ff->separators[i] == PF_CHAR_IS_CLOSE) - ff->separators[i] = PF_CHAR_IS_WORD; - - // if nothing given, return - if(unlikely(!open || !*open || !close || !*close)) - return; - - // set the openings - char *ffs = ff->separators; - const char *s = open; - while(likely(*s)) - ffs[(int)*s++] = PF_CHAR_IS_OPEN; - - s = close; - while(likely(*s)) - ffs[(int)*s++] = PF_CHAR_IS_CLOSE; + // remove all open/close + int i; + for(i = 0; i < 256 ; i++) + if(ff->separators[i] == PF_CHAR_IS_OPEN || ff->separators[i] == PF_CHAR_IS_CLOSE) + ff->separators[i] = PF_CHAR_IS_WORD; + + // if nothing given, return + if(unlikely(!open || !*open || !close || !*close)) + return; + + // set the openings + char *ffs = ff->separators; + const char *s = open; + while(likely(*s)) + ffs[(int)*s++] = PF_CHAR_IS_OPEN; + + s = close; + while(likely(*s)) + ffs[(int)*s++] = PF_CHAR_IS_CLOSE; } procfile *procfile_open(const char *filename, const char *separators, uint32_t flags) { - debug(D_PROCFILE, PF_PREFIX ": Opening file '%s'", filename); - - int fd = open(filename, O_RDONLY, 0666); - if(unlikely(fd == -1)) { - if(unlikely(!(flags & PROCFILE_FLAG_NO_ERROR_ON_FILE_IO))) error(PF_PREFIX ": Cannot open file '%s'", filename); - return NULL; - } - - size_t size = (unlikely(procfile_adaptive_initial_allocation)) ? procfile_max_allocation : PROCFILE_INCREMENT_BUFFER; - procfile *ff = malloc(sizeof(procfile) + size); - if(unlikely(!ff)) { - error(PF_PREFIX ": Cannot allocate memory for file '%s'", filename); - close(fd); - return NULL; - } - - strncpyz(ff->filename, filename, FILENAME_MAX); - - ff->fd = fd; - ff->size = size; - ff->len = 0; - ff->flags = flags; - - ff->lines = pflines_new(); - ff->words = pfwords_new(); - - if(unlikely(!ff->lines || !ff->words)) { - error(PF_PREFIX ": Cannot initialize parser for file '%s'", filename); - procfile_close(ff); - return NULL; - } - - procfile_set_separators(ff, separators); - - debug(D_PROCFILE, "File '%s' opened.", filename); - return ff; + debug(D_PROCFILE, PF_PREFIX ": Opening file '%s'", filename); + + int fd = open(filename, O_RDONLY, 0666); + if(unlikely(fd == -1)) { + if(unlikely(!(flags & PROCFILE_FLAG_NO_ERROR_ON_FILE_IO))) error(PF_PREFIX ": Cannot open file '%s'", filename); + return NULL; + } + + size_t size = (unlikely(procfile_adaptive_initial_allocation)) ? procfile_max_allocation : PROCFILE_INCREMENT_BUFFER; + procfile *ff = mallocz(sizeof(procfile) + size); + strncpyz(ff->filename, filename, FILENAME_MAX); + + ff->fd = fd; + ff->size = size; + ff->len = 0; + ff->flags = flags; + + ff->lines = pflines_new(); + ff->words = pfwords_new(); + + if(unlikely(!ff->lines || !ff->words)) { + error(PF_PREFIX ": Cannot initialize parser for file '%s'", filename); + procfile_close(ff); + return NULL; + } + + procfile_set_separators(ff, separators); + + debug(D_PROCFILE, "File '%s' opened.", filename); + return ff; } procfile *procfile_reopen(procfile *ff, const char *filename, const char *separators, uint32_t flags) { - if(unlikely(!ff)) return procfile_open(filename, separators, flags); + if(unlikely(!ff)) return procfile_open(filename, separators, flags); - if(likely(ff->fd != -1)) close(ff->fd); + if(likely(ff->fd != -1)) close(ff->fd); - ff->fd = open(filename, O_RDONLY, 0666); - if(unlikely(ff->fd == -1)) { - procfile_close(ff); - return NULL; - } + ff->fd = open(filename, O_RDONLY, 0666); + if(unlikely(ff->fd == -1)) { + procfile_close(ff); + return NULL; + } - strncpyz(ff->filename, filename, FILENAME_MAX); + strncpyz(ff->filename, filename, FILENAME_MAX); - ff->flags = flags; + ff->flags = flags; - // do not do the separators again if NULL is given - if(likely(separators)) procfile_set_separators(ff, separators); + // do not do the separators again if NULL is given + if(likely(separators)) procfile_set_separators(ff, separators); - return ff; + return ff; } // ---------------------------------------------------------------------------- // example parsing of procfile data void procfile_print(procfile *ff) { - uint32_t lines = procfile_lines(ff), l; - uint32_t words, w; - char *s; + uint32_t lines = procfile_lines(ff), l; + uint32_t words, w; + char *s; - debug(D_PROCFILE, "File '%s' with %d lines and %d words", ff->filename, ff->lines->len, ff->words->len); + debug(D_PROCFILE, "File '%s' with %u lines and %u words", ff->filename, ff->lines->len, ff->words->len); - for(l = 0; likely(l < lines) ;l++) { - words = procfile_linewords(ff, l); + for(l = 0; likely(l < lines) ;l++) { + words = procfile_linewords(ff, l); - debug(D_PROCFILE, " line %d starts at word %d and has %d words", l, ff->lines->lines[l].first, ff->lines->lines[l].words); + debug(D_PROCFILE, " line %u starts at word %u and has %u words", l, ff->lines->lines[l].first, ff->lines->lines[l].words); - for(w = 0; likely(w < words) ;w++) { - s = procfile_lineword(ff, l, w); - debug(D_PROCFILE, " [%d.%d] '%s'", l, w, s); - } - } + for(w = 0; likely(w < words) ;w++) { + s = procfile_lineword(ff, l, w); + debug(D_PROCFILE, " [%u.%u] '%s'", l, w, s); + } + } } diff --git a/src/procfile.h b/src/procfile.h index 122e153f1..5e00b2584 100644 --- a/src/procfile.h +++ b/src/procfile.h @@ -30,9 +30,9 @@ // An array of words typedef struct { - uint32_t len; // used entries - uint32_t size; // capacity - char *words[]; // array of pointers + uint32_t len; // used entries + uint32_t size; // capacity + char *words[]; // array of pointers } pfwords; @@ -40,15 +40,15 @@ typedef struct { // An array of lines typedef struct { - uint32_t words; // how many words this line has - uint32_t first; // the id of the first word of this line - // in the words array + uint32_t words; // how many words this line has + uint32_t first; // the id of the first word of this line + // in the words array } ffline; typedef struct { - uint32_t len; // used entries - uint32_t size; // capacity - ffline lines[]; // array of lines + uint32_t len; // used entries + uint32_t size; // capacity + ffline lines[]; // array of lines } pflines; @@ -59,15 +59,15 @@ typedef struct { #define PROCFILE_FLAG_NO_ERROR_ON_FILE_IO 0x00000001 typedef struct { - char filename[FILENAME_MAX + 1]; - uint32_t flags; - int fd; // the file desriptor - size_t len; // the bytes we have placed into data - size_t size; // the bytes we have allocated for data - pflines *lines; - pfwords *words; - char separators[256]; - char data[]; // allocated buffer to keep file contents + char filename[FILENAME_MAX + 1]; + uint32_t flags; + int fd; // the file desriptor + size_t len; // the bytes we have placed into data + size_t size; // the bytes we have allocated for data + pflines *lines; + pfwords *words; + char separators[256]; + char data[]; // allocated buffer to keep file contents } procfile; // close the proc file and free all related memory diff --git a/src/registry.c b/src/registry.c index f39ce3e2e..f2319c478 100644 --- a/src/registry.c +++ b/src/registry.c @@ -1,28 +1,4 @@ -#ifdef HAVE_CONFIG_H -#include <config.h> -#endif - -#include <uuid/uuid.h> -#include <inttypes.h> -#include <stdlib.h> -#include <string.h> -#include <ctype.h> -#include <unistd.h> -#include <sys/stat.h> -#include <sys/types.h> -#include <errno.h> -#include <fcntl.h> - -#include "log.h" #include "common.h" -#include "dictionary.h" -#include "appconfig.h" - -#include "web_client.h" -#include "rrd.h" -#include "rrd2json.h" -#include "registry.h" - // ---------------------------------------------------------------------------- // TODO @@ -63,59 +39,60 @@ // COMMON structures struct registry { - int enabled; - - char machine_guid[36 + 1]; - - // entries counters / statistics - unsigned long long persons_count; - unsigned long long machines_count; - unsigned long long usages_count; - unsigned long long urls_count; - unsigned long long persons_urls_count; - unsigned long long machines_urls_count; - unsigned long long log_count; - - // memory counters / statistics - unsigned long long persons_memory; - unsigned long long machines_memory; - unsigned long long urls_memory; - unsigned long long persons_urls_memory; - unsigned long long machines_urls_memory; - - // configuration - unsigned long long save_registry_every_entries; - char *registry_domain; - char *hostname; - char *registry_to_announce; - time_t persons_expiration; // seconds to expire idle persons - - size_t max_url_length; - size_t max_name_length; - - // file/path names - char *pathname; - char *db_filename; - char *log_filename; - char *machine_guid_filename; - - // open files - FILE *log_fp; - - // the database - DICTIONARY *persons; // dictionary of PERSON *, with key the PERSON.guid - DICTIONARY *machines; // dictionary of MACHINE *, with key the MACHINE.guid - DICTIONARY *urls; // dictionary of URL *, with key the URL.url - - // concurrency locking - // we keep different locks for different things - // so that many tasks can be completed in parallel - pthread_mutex_t persons_lock; - pthread_mutex_t machines_lock; - pthread_mutex_t urls_lock; - pthread_mutex_t person_urls_lock; - pthread_mutex_t machine_urls_lock; - pthread_mutex_t log_lock; + int enabled; + + char machine_guid[36 + 1]; + + // entries counters / statistics + unsigned long long persons_count; + unsigned long long machines_count; + unsigned long long usages_count; + unsigned long long urls_count; + unsigned long long persons_urls_count; + unsigned long long machines_urls_count; + unsigned long long log_count; + + // memory counters / statistics + unsigned long long persons_memory; + unsigned long long machines_memory; + unsigned long long urls_memory; + unsigned long long persons_urls_memory; + unsigned long long machines_urls_memory; + + // configuration + unsigned long long save_registry_every_entries; + char *registry_domain; + char *hostname; + char *registry_to_announce; + time_t persons_expiration; // seconds to expire idle persons + int verify_cookies_redirects; + + size_t max_url_length; + size_t max_name_length; + + // file/path names + char *pathname; + char *db_filename; + char *log_filename; + char *machine_guid_filename; + + // open files + FILE *log_fp; + + // the database + DICTIONARY *persons; // dictionary of PERSON *, with key the PERSON.guid + DICTIONARY *machines; // dictionary of MACHINE *, with key the MACHINE.guid + DICTIONARY *urls; // dictionary of URL *, with key the URL.url + + // concurrency locking + // we keep different locks for different things + // so that many tasks can be completed in parallel + pthread_mutex_t persons_lock; + pthread_mutex_t machines_lock; + pthread_mutex_t urls_lock; + pthread_mutex_t person_urls_lock; + pthread_mutex_t machine_urls_lock; + pthread_mutex_t log_lock; } registry; @@ -126,9 +103,9 @@ struct registry { // we store them here and we keep pointers elsewhere struct url { - uint32_t links; // the number of links to this URL - when none is left, we free it - uint16_t len; // the length of the URL in bytes - char url[1]; // the URL - dynamically allocated to more size + uint32_t links; // the number of links to this URL - when none is left, we free it + uint16_t len; // the length of the URL in bytes + char url[1]; // the URL - dynamically allocated to more size }; typedef struct url URL; @@ -138,27 +115,27 @@ typedef struct url URL; // For each MACHINE-URL pair we keep this struct machine_url { - URL *url; // de-duplicated URL -// DICTIONARY *persons; // dictionary of PERSON * + URL *url; // de-duplicated URL +// DICTIONARY *persons; // dictionary of PERSON * - uint8_t flags; - uint32_t first_t; // the first time we saw this - uint32_t last_t; // the last time we saw this - uint32_t usages; // how many times this has been accessed + uint8_t flags; + uint32_t first_t; // the first time we saw this + uint32_t last_t; // the last time we saw this + uint32_t usages; // how many times this has been accessed }; typedef struct machine_url MACHINE_URL; // A machine struct machine { - char guid[36 + 1]; // the GUID + char guid[36 + 1]; // the GUID - uint32_t links; // the number of PERSON_URLs linked to this machine + uint32_t links; // the number of PERSON_URLs linked to this machine - DICTIONARY *urls; // MACHINE_URL * + DICTIONARY *urls; // MACHINE_URL * - uint32_t first_t; // the first time we saw this - uint32_t last_t; // the last time we saw this - uint32_t usages; // how many times this has been accessed + uint32_t first_t; // the first time we saw this + uint32_t last_t; // the last time we saw this + uint32_t usages; // how many times this has been accessed }; typedef struct machine MACHINE; @@ -168,28 +145,28 @@ typedef struct machine MACHINE; // for each PERSON-URL pair we keep this struct person_url { - URL *url; // de-duplicated URL - MACHINE *machine; // link the MACHINE of this URL + URL *url; // de-duplicated URL + MACHINE *machine; // link the MACHINE of this URL - uint8_t flags; - uint32_t first_t; // the first time we saw this - uint32_t last_t; // the last time we saw this - uint32_t usages; // how many times this has been accessed + uint8_t flags; + uint32_t first_t; // the first time we saw this + uint32_t last_t; // the last time we saw this + uint32_t usages; // how many times this has been accessed - char name[1]; // the name of the URL, as known by the user - // dynamically allocated to fit properly + char name[1]; // the name of the URL, as known by the user + // dynamically allocated to fit properly }; typedef struct person_url PERSON_URL; // A person struct person { - char guid[36 + 1]; // the person GUID + char guid[36 + 1]; // the person GUID - DICTIONARY *urls; // dictionary of PERSON_URL * + DICTIONARY *urls; // dictionary of PERSON_URL * - uint32_t first_t; // the first time we saw this - uint32_t last_t; // the last time we saw this - uint32_t usages; // how many times this has been accessed + uint32_t first_t; // the first time we saw this + uint32_t last_t; // the last time we saw this + uint32_t usages; // how many times this has been accessed }; typedef struct person PERSON; @@ -198,27 +175,27 @@ typedef struct person PERSON; // REGISTRY concurrency locking static inline void registry_persons_lock(void) { - pthread_mutex_lock(®istry.persons_lock); + pthread_mutex_lock(®istry.persons_lock); } static inline void registry_persons_unlock(void) { - pthread_mutex_unlock(®istry.persons_lock); + pthread_mutex_unlock(®istry.persons_lock); } static inline void registry_machines_lock(void) { - pthread_mutex_lock(®istry.machines_lock); + pthread_mutex_lock(®istry.machines_lock); } static inline void registry_machines_unlock(void) { - pthread_mutex_unlock(®istry.machines_lock); + pthread_mutex_unlock(®istry.machines_lock); } static inline void registry_urls_lock(void) { - pthread_mutex_lock(®istry.urls_lock); + pthread_mutex_lock(®istry.urls_lock); } static inline void registry_urls_unlock(void) { - pthread_mutex_unlock(®istry.urls_lock); + pthread_mutex_unlock(®istry.urls_lock); } // ideally, we should not lock the whole registry for @@ -226,13 +203,13 @@ static inline void registry_urls_unlock(void) { // however, to save the memory required for keeping a // mutex (40 bytes) per person, we do... static inline void registry_person_urls_lock(PERSON *p) { - (void)p; - pthread_mutex_lock(®istry.person_urls_lock); + (void)p; + pthread_mutex_lock(®istry.person_urls_lock); } static inline void registry_person_urls_unlock(PERSON *p) { - (void)p; - pthread_mutex_unlock(®istry.person_urls_lock); + (void)p; + pthread_mutex_unlock(®istry.person_urls_lock); } // ideally, we should not lock the whole registry for @@ -240,21 +217,21 @@ static inline void registry_person_urls_unlock(PERSON *p) { // however, to save the memory required for keeping a // mutex (40 bytes) per machine, we do... static inline void registry_machine_urls_lock(MACHINE *m) { - (void)m; - pthread_mutex_lock(®istry.machine_urls_lock); + (void)m; + pthread_mutex_lock(®istry.machine_urls_lock); } static inline void registry_machine_urls_unlock(MACHINE *m) { - (void)m; - pthread_mutex_unlock(®istry.machine_urls_lock); + (void)m; + pthread_mutex_unlock(®istry.machine_urls_lock); } static inline void registry_log_lock(void) { - pthread_mutex_lock(®istry.log_lock); + pthread_mutex_lock(®istry.log_lock); } static inline void registry_log_unlock(void) { - pthread_mutex_unlock(®istry.log_lock); + pthread_mutex_unlock(®istry.log_lock); } @@ -264,58 +241,58 @@ static inline void registry_log_unlock(void) { // parse a GUID and re-generated to be always lower case // this is used as a protection against the variations of GUIDs static inline int registry_regenerate_guid(const char *guid, char *result) { - uuid_t uuid; - if(unlikely(uuid_parse(guid, uuid) == -1)) { - info("Registry: GUID '%s' is not a valid GUID.", guid); - return -1; - } - else { - uuid_unparse_lower(uuid, result); + uuid_t uuid; + if(unlikely(uuid_parse(guid, uuid) == -1)) { + info("Registry: GUID '%s' is not a valid GUID.", guid); + return -1; + } + else { + uuid_unparse_lower(uuid, result); #ifdef NETDATA_INTERNAL_CHECKS - if(strcmp(guid, result)) - info("Registry: source GUID '%s' and re-generated GUID '%s' differ!", guid, result); + if(strcmp(guid, result)) + info("Registry: source GUID '%s' and re-generated GUID '%s' differ!", guid, result); #endif /* NETDATA_INTERNAL_CHECKS */ - } + } - return 0; + return 0; } // make sure the names of the machines / URLs do not contain any tabs // (which are used as our separator in the database files) // and are properly trimmed (before and after) static inline char *registry_fix_machine_name(char *name, size_t *len) { - char *s = name?name:""; + char *s = name?name:""; - // skip leading spaces - while(*s && isspace(*s)) s++; + // skip leading spaces + while(*s && isspace(*s)) s++; - // make sure all spaces are a SPACE - char *t = s; - while(*t) { - if(unlikely(isspace(*t))) - *t = ' '; + // make sure all spaces are a SPACE + char *t = s; + while(*t) { + if(unlikely(isspace(*t))) + *t = ' '; - t++; - } + t++; + } - // remove trailing spaces - while(--t >= s) { - if(*t == ' ') - *t = '\0'; - else - break; - } - t++; + // remove trailing spaces + while(--t >= s) { + if(*t == ' ') + *t = '\0'; + else + break; + } + t++; - if(likely(len)) - *len = (t - s); + if(likely(len)) + *len = (t - s); - return s; + return s; } static inline char *registry_fix_url(char *url, size_t *len) { - return registry_fix_machine_name(url, len); + return registry_fix_machine_name(url, len); } @@ -330,59 +307,57 @@ extern PERSON *registry_request_delete(char *person_guid, char *machine_guid, ch // URL static inline URL *registry_url_allocate_nolock(const char *url, size_t urllen) { - // protection from too big URLs - if(urllen > registry.max_url_length) - urllen = registry.max_url_length; - - debug(D_REGISTRY, "Registry: registry_url_allocate_nolock('%s'): allocating %zu bytes", url, sizeof(URL) + urllen); - URL *u = malloc(sizeof(URL) + urllen); - if(!u) fatal("Cannot allocate %zu bytes for URL '%s'", sizeof(URL) + urllen); + // protection from too big URLs + if(urllen > registry.max_url_length) + urllen = registry.max_url_length; - // a simple strcpy() should do the job - // but I prefer to be safe, since the caller specified urllen - strncpyz(u->url, url, urllen); + debug(D_REGISTRY, "Registry: registry_url_allocate_nolock('%s'): allocating %zu bytes", url, sizeof(URL) + urllen); + URL *u = mallocz(sizeof(URL) + urllen); - u->len = urllen; - u->links = 0; + // a simple strcpy() should do the job + // but I prefer to be safe, since the caller specified urllen + u->len = (uint16_t)urllen; + strncpyz(u->url, url, u->len); + u->links = 0; - registry.urls_memory += sizeof(URL) + urllen; + registry.urls_memory += sizeof(URL) + urllen; - debug(D_REGISTRY, "Registry: registry_url_allocate_nolock('%s'): indexing it", url); - dictionary_set(registry.urls, u->url, u, sizeof(URL)); + debug(D_REGISTRY, "Registry: registry_url_allocate_nolock('%s'): indexing it", url); + dictionary_set(registry.urls, u->url, u, sizeof(URL)); - return u; + return u; } static inline URL *registry_url_get(const char *url, size_t urllen) { - debug(D_REGISTRY, "Registry: registry_url_get('%s')", url); + debug(D_REGISTRY, "Registry: registry_url_get('%s')", url); - registry_urls_lock(); + registry_urls_lock(); - URL *u = dictionary_get(registry.urls, url); - if(!u) { - u = registry_url_allocate_nolock(url, urllen); - registry.urls_count++; - } + URL *u = dictionary_get(registry.urls, url); + if(!u) { + u = registry_url_allocate_nolock(url, urllen); + registry.urls_count++; + } - registry_urls_unlock(); + registry_urls_unlock(); - return u; + return u; } static inline void registry_url_link_nolock(URL *u) { - u->links++; - debug(D_REGISTRY, "Registry: registry_url_link_nolock('%s'): URL has now %u links", u->url, u->links); + u->links++; + debug(D_REGISTRY, "Registry: registry_url_link_nolock('%s'): URL has now %u links", u->url, u->links); } static inline void registry_url_unlink_nolock(URL *u) { - u->links--; - if(!u->links) { - debug(D_REGISTRY, "Registry: registry_url_unlink_nolock('%s'): No more links for this URL", u->url); - dictionary_del(registry.urls, u->url); - free(u); - } - else - debug(D_REGISTRY, "Registry: registry_url_unlink_nolock('%s'): URL has %u links left", u->url, u->links); + u->links--; + if(!u->links) { + debug(D_REGISTRY, "Registry: registry_url_unlink_nolock('%s'): No more links for this URL", u->url); + dictionary_del(registry.urls, u->url); + freez(u); + } + else + debug(D_REGISTRY, "Registry: registry_url_unlink_nolock('%s'): URL has %u links left", u->url, u->links); } @@ -390,78 +365,76 @@ static inline void registry_url_unlink_nolock(URL *u) { // MACHINE static inline MACHINE *registry_machine_find(const char *machine_guid) { - debug(D_REGISTRY, "Registry: registry_machine_find('%s')", machine_guid); - return dictionary_get(registry.machines, machine_guid); + debug(D_REGISTRY, "Registry: registry_machine_find('%s')", machine_guid); + return dictionary_get(registry.machines, machine_guid); } static inline MACHINE_URL *registry_machine_url_allocate(MACHINE *m, URL *u, time_t when) { - debug(D_REGISTRY, "registry_machine_link_to_url('%s', '%s'): allocating %zu bytes", m->guid, u->url, sizeof(MACHINE_URL)); + debug(D_REGISTRY, "registry_machine_link_to_url('%s', '%s'): allocating %zu bytes", m->guid, u->url, sizeof(MACHINE_URL)); - MACHINE_URL *mu = malloc(sizeof(MACHINE_URL)); - if(!mu) fatal("registry_machine_link_to_url('%s', '%s'): cannot allocate %zu bytes.", m->guid, u->url, sizeof(MACHINE_URL)); + MACHINE_URL *mu = mallocz(sizeof(MACHINE_URL)); - // mu->persons = dictionary_create(DICTIONARY_FLAGS); - // dictionary_set(mu->persons, p->guid, p, sizeof(PERSON)); + // mu->persons = dictionary_create(DICTIONARY_FLAGS); + // dictionary_set(mu->persons, p->guid, p, sizeof(PERSON)); - mu->first_t = mu->last_t = when; - mu->usages = 1; - mu->url = u; - mu->flags = REGISTRY_URL_FLAGS_DEFAULT; + mu->first_t = mu->last_t = (uint32_t)when; + mu->usages = 1; + mu->url = u; + mu->flags = REGISTRY_URL_FLAGS_DEFAULT; - registry.machines_urls_memory += sizeof(MACHINE_URL); + registry.machines_urls_memory += sizeof(MACHINE_URL); - debug(D_REGISTRY, "registry_machine_link_to_url('%s', '%s'): indexing URL in machine", m->guid, u->url); - dictionary_set(m->urls, u->url, mu, sizeof(MACHINE_URL)); - registry_url_link_nolock(u); + debug(D_REGISTRY, "registry_machine_link_to_url('%s', '%s'): indexing URL in machine", m->guid, u->url); + dictionary_set(m->urls, u->url, mu, sizeof(MACHINE_URL)); + registry_url_link_nolock(u); - return mu; + return mu; } static inline MACHINE *registry_machine_allocate(const char *machine_guid, time_t when) { - debug(D_REGISTRY, "Registry: registry_machine_allocate('%s'): creating new machine, sizeof(MACHINE)=%zu", machine_guid, sizeof(MACHINE)); + debug(D_REGISTRY, "Registry: registry_machine_allocate('%s'): creating new machine, sizeof(MACHINE)=%zu", machine_guid, sizeof(MACHINE)); - MACHINE *m = malloc(sizeof(MACHINE)); - if(!m) fatal("Registry: cannot allocate memory for new machine '%s'", machine_guid); + MACHINE *m = mallocz(sizeof(MACHINE)); - strncpyz(m->guid, machine_guid, 36); + strncpyz(m->guid, machine_guid, 36); - debug(D_REGISTRY, "Registry: registry_machine_allocate('%s'): creating dictionary of urls", machine_guid); - m->urls = dictionary_create(DICTIONARY_FLAGS); + debug(D_REGISTRY, "Registry: registry_machine_allocate('%s'): creating dictionary of urls", machine_guid); + m->urls = dictionary_create(DICTIONARY_FLAGS); - m->first_t = m->last_t = when; - m->usages = 0; + m->first_t = m->last_t = (uint32_t)when; + m->usages = 0; - registry.machines_memory += sizeof(MACHINE); + registry.machines_memory += sizeof(MACHINE); - registry.machines_count++; - dictionary_set(registry.machines, m->guid, m, sizeof(MACHINE)); + registry.machines_count++; + dictionary_set(registry.machines, m->guid, m, sizeof(MACHINE)); - return m; + return m; } // 1. validate machine GUID // 2. if it is valid, find it or create it and return it // 3. if it is not valid, return NULL static inline MACHINE *registry_machine_get(const char *machine_guid, time_t when) { - MACHINE *m = NULL; + MACHINE *m = NULL; - registry_machines_lock(); + registry_machines_lock(); - if(likely(machine_guid && *machine_guid)) { - // validate it is a GUID - char buf[36 + 1]; - if(unlikely(registry_regenerate_guid(machine_guid, buf) == -1)) - info("Registry: machine guid '%s' is not a valid guid. Ignoring it.", machine_guid); - else { - machine_guid = buf; - m = registry_machine_find(machine_guid); - if(!m) m = registry_machine_allocate(machine_guid, when); - } - } + if(likely(machine_guid && *machine_guid)) { + // validate it is a GUID + char buf[36 + 1]; + if(unlikely(registry_regenerate_guid(machine_guid, buf) == -1)) + info("Registry: machine guid '%s' is not a valid guid. Ignoring it.", machine_guid); + else { + machine_guid = buf; + m = registry_machine_find(machine_guid); + if(!m) m = registry_machine_allocate(machine_guid, when); + } + } - registry_machines_unlock(); + registry_machines_unlock(); - return m; + return m; } @@ -469,101 +442,99 @@ static inline MACHINE *registry_machine_get(const char *machine_guid, time_t whe // PERSON static inline PERSON *registry_person_find(const char *person_guid) { - debug(D_REGISTRY, "Registry: registry_person_find('%s')", person_guid); - return dictionary_get(registry.persons, person_guid); + debug(D_REGISTRY, "Registry: registry_person_find('%s')", person_guid); + return dictionary_get(registry.persons, person_guid); } static inline PERSON_URL *registry_person_url_allocate(PERSON *p, MACHINE *m, URL *u, char *name, size_t namelen, time_t when) { - // protection from too big names - if(namelen > registry.max_name_length) - namelen = registry.max_name_length; + // protection from too big names + if(namelen > registry.max_name_length) + namelen = registry.max_name_length; - debug(D_REGISTRY, "registry_person_url_allocate('%s', '%s', '%s'): allocating %zu bytes", p->guid, m->guid, u->url, - sizeof(PERSON_URL) + namelen); + debug(D_REGISTRY, "registry_person_url_allocate('%s', '%s', '%s'): allocating %zu bytes", p->guid, m->guid, u->url, + sizeof(PERSON_URL) + namelen); - PERSON_URL *pu = malloc(sizeof(PERSON_URL) + namelen); - if(!pu) fatal("registry_person_url_allocate('%s', '%s', '%s'): cannot allocate %zu bytes.", p->guid, m->guid, u->url, sizeof(PERSON_URL) + namelen); + PERSON_URL *pu = mallocz(sizeof(PERSON_URL) + namelen); - // a simple strcpy() should do the job - // but I prefer to be safe, since the caller specified urllen - strncpyz(pu->name, name, namelen); + // a simple strcpy() should do the job + // but I prefer to be safe, since the caller specified urllen + strncpyz(pu->name, name, namelen); - pu->machine = m; - pu->first_t = pu->last_t = when; - pu->usages = 1; - pu->url = u; - pu->flags = REGISTRY_URL_FLAGS_DEFAULT; - m->links++; + pu->machine = m; + pu->first_t = pu->last_t = when; + pu->usages = 1; + pu->url = u; + pu->flags = REGISTRY_URL_FLAGS_DEFAULT; + m->links++; - registry.persons_urls_memory += sizeof(PERSON_URL) + namelen; + registry.persons_urls_memory += sizeof(PERSON_URL) + namelen; - debug(D_REGISTRY, "registry_person_url_allocate('%s', '%s', '%s'): indexing URL in person", p->guid, m->guid, u->url); - dictionary_set(p->urls, u->url, pu, sizeof(PERSON_URL)); - registry_url_link_nolock(u); + debug(D_REGISTRY, "registry_person_url_allocate('%s', '%s', '%s'): indexing URL in person", p->guid, m->guid, u->url); + dictionary_set(p->urls, u->url, pu, sizeof(PERSON_URL)); + registry_url_link_nolock(u); - return pu; + return pu; } static inline PERSON_URL *registry_person_url_reallocate(PERSON *p, MACHINE *m, URL *u, char *name, size_t namelen, time_t when, PERSON_URL *pu) { - // this function is needed to change the name of a PERSON_URL + // this function is needed to change the name of a PERSON_URL - debug(D_REGISTRY, "registry_person_url_reallocate('%s', '%s', '%s'): allocating %zu bytes", p->guid, m->guid, u->url, - sizeof(PERSON_URL) + namelen); + debug(D_REGISTRY, "registry_person_url_reallocate('%s', '%s', '%s'): allocating %zu bytes", p->guid, m->guid, u->url, + sizeof(PERSON_URL) + namelen); - PERSON_URL *tpu = registry_person_url_allocate(p, m, u, name, namelen, when); - tpu->first_t = pu->first_t; - tpu->last_t = pu->last_t; - tpu->usages = pu->usages; + PERSON_URL *tpu = registry_person_url_allocate(p, m, u, name, namelen, when); + tpu->first_t = pu->first_t; + tpu->last_t = pu->last_t; + tpu->usages = pu->usages; - // ok, these are a hack - since the registry_person_url_allocate() is - // adding these, we have to subtract them - tpu->machine->links--; - registry.persons_urls_memory -= sizeof(PERSON_URL) + strlen(pu->name); - registry_url_unlink_nolock(u); + // ok, these are a hack - since the registry_person_url_allocate() is + // adding these, we have to subtract them + tpu->machine->links--; + registry.persons_urls_memory -= sizeof(PERSON_URL) + strlen(pu->name); + registry_url_unlink_nolock(u); - free(pu); + freez(pu); - return tpu; + return tpu; } static inline PERSON *registry_person_allocate(const char *person_guid, time_t when) { - PERSON *p = NULL; + PERSON *p = NULL; - debug(D_REGISTRY, "Registry: registry_person_allocate('%s'): allocating new person, sizeof(PERSON)=%zu", (person_guid)?person_guid:"", sizeof(PERSON)); + debug(D_REGISTRY, "Registry: registry_person_allocate('%s'): allocating new person, sizeof(PERSON)=%zu", (person_guid)?person_guid:"", sizeof(PERSON)); - p = malloc(sizeof(PERSON)); - if(!p) fatal("Registry: cannot allocate memory for new person."); + p = mallocz(sizeof(PERSON)); - if(!person_guid) { - for (; ;) { - uuid_t uuid; - uuid_generate(uuid); - uuid_unparse_lower(uuid, p->guid); + if(!person_guid) { + for (; ;) { + uuid_t uuid; + uuid_generate(uuid); + uuid_unparse_lower(uuid, p->guid); - debug(D_REGISTRY, "Registry: Checking if the generated person guid '%s' is unique", p->guid); - if (!dictionary_get(registry.persons, p->guid)) { - debug(D_REGISTRY, "Registry: generated person guid '%s' is unique", p->guid); - break; - } - else - info("Registry: generated person guid '%s' found in the registry. Retrying...", p->guid); - } - } - else - strncpyz(p->guid, person_guid, 36); + debug(D_REGISTRY, "Registry: Checking if the generated person guid '%s' is unique", p->guid); + if (!dictionary_get(registry.persons, p->guid)) { + debug(D_REGISTRY, "Registry: generated person guid '%s' is unique", p->guid); + break; + } + else + info("Registry: generated person guid '%s' found in the registry. Retrying...", p->guid); + } + } + else + strncpyz(p->guid, person_guid, 36); - debug(D_REGISTRY, "Registry: registry_person_allocate('%s'): creating dictionary of urls", p->guid); - p->urls = dictionary_create(DICTIONARY_FLAGS); + debug(D_REGISTRY, "Registry: registry_person_allocate('%s'): creating dictionary of urls", p->guid); + p->urls = dictionary_create(DICTIONARY_FLAGS); - p->first_t = p->last_t = when; - p->usages = 0; + p->first_t = p->last_t = when; + p->usages = 0; - registry.persons_memory += sizeof(PERSON); + registry.persons_memory += sizeof(PERSON); - registry.persons_count++; - dictionary_set(registry.persons, p->guid, p, sizeof(PERSON)); + registry.persons_count++; + dictionary_set(registry.persons, p->guid, p, sizeof(PERSON)); - return p; + return p; } @@ -572,258 +543,260 @@ static inline PERSON *registry_person_allocate(const char *person_guid, time_t w // 3. if it is not valid, create a new one // 4. return it static inline PERSON *registry_person_get(const char *person_guid, time_t when) { - PERSON *p = NULL; + PERSON *p = NULL; - registry_persons_lock(); + registry_persons_lock(); - if(person_guid && *person_guid) { - char buf[36 + 1]; - // validate it is a GUID - if(unlikely(registry_regenerate_guid(person_guid, buf) == -1)) - info("Registry: person guid '%s' is not a valid guid. Ignoring it.", person_guid); - else { - person_guid = buf; - p = registry_person_find(person_guid); - if(!p) person_guid = NULL; - } - } + if(person_guid && *person_guid) { + char buf[36 + 1]; + // validate it is a GUID + if(unlikely(registry_regenerate_guid(person_guid, buf) == -1)) + info("Registry: person guid '%s' is not a valid guid. Ignoring it.", person_guid); + else { + person_guid = buf; + p = registry_person_find(person_guid); + if(!p) person_guid = NULL; + } + } - if(!p) p = registry_person_allocate(NULL, when); + if(!p) p = registry_person_allocate(NULL, when); - registry_persons_unlock(); + registry_persons_unlock(); - return p; + return p; } // ---------------------------------------------------------------------------- // LINKING OF OBJECTS static inline PERSON_URL *registry_person_link_to_url(PERSON *p, MACHINE *m, URL *u, char *name, size_t namelen, time_t when) { - debug(D_REGISTRY, "registry_person_link_to_url('%s', '%s', '%s'): searching for URL in person", p->guid, m->guid, u->url); - - registry_person_urls_lock(p); - - PERSON_URL *pu = dictionary_get(p->urls, u->url); - if(!pu) { - debug(D_REGISTRY, "registry_person_link_to_url('%s', '%s', '%s'): not found", p->guid, m->guid, u->url); - pu = registry_person_url_allocate(p, m, u, name, namelen, when); - registry.persons_urls_count++; - } - else { - debug(D_REGISTRY, "registry_person_link_to_url('%s', '%s', '%s'): found", p->guid, m->guid, u->url); - pu->usages++; - if(likely(pu->last_t < when)) pu->last_t = when; - - if(pu->machine != m) { - MACHINE_URL *mu = dictionary_get(pu->machine->urls, u->url); - if(mu) { - info("registry_person_link_to_url('%s', '%s', '%s'): URL switched machines (old was '%s') - expiring it from previous machine.", - p->guid, m->guid, u->url, pu->machine->guid); - mu->flags |= REGISTRY_URL_FLAGS_EXPIRED; - } - else { - info("registry_person_link_to_url('%s', '%s', '%s'): URL switched machines (old was '%s') - but the URL is not linked to the old machine.", - p->guid, m->guid, u->url, pu->machine->guid); - } - - pu->machine->links--; - pu->machine = m; - } - - if(strcmp(pu->name, name)) { - // the name of the PERSON_URL has changed ! - pu = registry_person_url_reallocate(p, m, u, name, namelen, when, pu); - } - } - - p->usages++; - if(likely(p->last_t < when)) p->last_t = when; - - if(pu->flags & REGISTRY_URL_FLAGS_EXPIRED) { - info("registry_person_link_to_url('%s', '%s', '%s'): accessing an expired URL. Re-enabling URL.", p->guid, m->guid, u->url); - pu->flags &= ~REGISTRY_URL_FLAGS_EXPIRED; - } - - registry_person_urls_unlock(p); - - return pu; + debug(D_REGISTRY, "registry_person_link_to_url('%s', '%s', '%s'): searching for URL in person", p->guid, m->guid, u->url); + + registry_person_urls_lock(p); + + PERSON_URL *pu = dictionary_get(p->urls, u->url); + if(!pu) { + debug(D_REGISTRY, "registry_person_link_to_url('%s', '%s', '%s'): not found", p->guid, m->guid, u->url); + pu = registry_person_url_allocate(p, m, u, name, namelen, when); + registry.persons_urls_count++; + } + else { + debug(D_REGISTRY, "registry_person_link_to_url('%s', '%s', '%s'): found", p->guid, m->guid, u->url); + pu->usages++; + if(likely(pu->last_t < (uint32_t)when)) pu->last_t = when; + + if(pu->machine != m) { + MACHINE_URL *mu = dictionary_get(pu->machine->urls, u->url); + if(mu) { + info("registry_person_link_to_url('%s', '%s', '%s'): URL switched machines (old was '%s') - expiring it from previous machine.", + p->guid, m->guid, u->url, pu->machine->guid); + mu->flags |= REGISTRY_URL_FLAGS_EXPIRED; + } + else { + info("registry_person_link_to_url('%s', '%s', '%s'): URL switched machines (old was '%s') - but the URL is not linked to the old machine.", + p->guid, m->guid, u->url, pu->machine->guid); + } + + pu->machine->links--; + pu->machine = m; + } + + if(strcmp(pu->name, name)) { + // the name of the PERSON_URL has changed ! + pu = registry_person_url_reallocate(p, m, u, name, namelen, when, pu); + } + } + + p->usages++; + if(likely(p->last_t < (uint32_t)when)) p->last_t = when; + + if(pu->flags & REGISTRY_URL_FLAGS_EXPIRED) { + info("registry_person_link_to_url('%s', '%s', '%s'): accessing an expired URL. Re-enabling URL.", p->guid, m->guid, u->url); + pu->flags &= ~REGISTRY_URL_FLAGS_EXPIRED; + } + + registry_person_urls_unlock(p); + + return pu; } static inline MACHINE_URL *registry_machine_link_to_url(PERSON *p, MACHINE *m, URL *u, time_t when) { - debug(D_REGISTRY, "registry_machine_link_to_url('%s', '%s', '%s'): searching for URL in machine", p->guid, m->guid, u->url); + debug(D_REGISTRY, "registry_machine_link_to_url('%s', '%s', '%s'): searching for URL in machine", p->guid, m->guid, u->url); - registry_machine_urls_lock(m); + registry_machine_urls_lock(m); - MACHINE_URL *mu = dictionary_get(m->urls, u->url); - if(!mu) { - debug(D_REGISTRY, "registry_machine_link_to_url('%s', '%s', '%s'): not found", p->guid, m->guid, u->url); - mu = registry_machine_url_allocate(m, u, when); - registry.machines_urls_count++; - } - else { - debug(D_REGISTRY, "registry_machine_link_to_url('%s', '%s', '%s'): found", p->guid, m->guid, u->url); - mu->usages++; - if(likely(mu->last_t < when)) mu->last_t = when; - } + MACHINE_URL *mu = dictionary_get(m->urls, u->url); + if(!mu) { + debug(D_REGISTRY, "registry_machine_link_to_url('%s', '%s', '%s'): not found", p->guid, m->guid, u->url); + mu = registry_machine_url_allocate(m, u, when); + registry.machines_urls_count++; + } + else { + debug(D_REGISTRY, "registry_machine_link_to_url('%s', '%s', '%s'): found", p->guid, m->guid, u->url); + mu->usages++; + if(likely(mu->last_t < (uint32_t)when)) mu->last_t = when; + } - //debug(D_REGISTRY, "registry_machine_link_to_url('%s', '%s', '%s'): indexing person in machine", p->guid, m->guid, u->url); - //dictionary_set(mu->persons, p->guid, p, sizeof(PERSON)); + //debug(D_REGISTRY, "registry_machine_link_to_url('%s', '%s', '%s'): indexing person in machine", p->guid, m->guid, u->url); + //dictionary_set(mu->persons, p->guid, p, sizeof(PERSON)); - m->usages++; - if(likely(m->last_t < when)) m->last_t = when; + m->usages++; + if(likely(m->last_t < (uint32_t)when)) m->last_t = when; - if(mu->flags & REGISTRY_URL_FLAGS_EXPIRED) { - info("registry_machine_link_to_url('%s', '%s', '%s'): accessing an expired URL.", p->guid, m->guid, u->url); - mu->flags &= ~REGISTRY_URL_FLAGS_EXPIRED; - } + if(mu->flags & REGISTRY_URL_FLAGS_EXPIRED) { + info("registry_machine_link_to_url('%s', '%s', '%s'): accessing an expired URL.", p->guid, m->guid, u->url); + mu->flags &= ~REGISTRY_URL_FLAGS_EXPIRED; + } - registry_machine_urls_unlock(m); + registry_machine_urls_unlock(m); - return mu; + return mu; } // ---------------------------------------------------------------------------- // REGISTRY LOG LOAD/SAVE static inline int registry_should_save_db(void) { - debug(D_REGISTRY, "log entries %llu, max %llu", registry.log_count, registry.save_registry_every_entries); - return registry.log_count > registry.save_registry_every_entries; + debug(D_REGISTRY, "log entries %llu, max %llu", registry.log_count, registry.save_registry_every_entries); + return registry.log_count > registry.save_registry_every_entries; } static inline void registry_log(const char action, PERSON *p, MACHINE *m, URL *u, char *name) { - if(likely(registry.log_fp)) { - // we lock only if the file is open - // to allow replaying the log at registry_log_load() - registry_log_lock(); - - if(unlikely(fprintf(registry.log_fp, "%c\t%08x\t%s\t%s\t%s\t%s\n", - action, - p->last_t, - p->guid, - m->guid, - name, - u->url) < 0)) - error("Registry: failed to save log. Registry data may be lost in case of abnormal restart."); - - // we increase the counter even on failures - // so that the registry will be saved periodically - registry.log_count++; - - registry_log_unlock(); - - // this must be outside the log_lock(), or a deadlock will happen. - // registry_save() checks the same inside the log_lock, so only - // one thread will save the db - if(unlikely(registry_should_save_db())) - registry_save(); - } + if(likely(registry.log_fp)) { + // we lock only if the file is open + // to allow replaying the log at registry_log_load() + registry_log_lock(); + + if(unlikely(fprintf(registry.log_fp, "%c\t%08x\t%s\t%s\t%s\t%s\n", + action, + p->last_t, + p->guid, + m->guid, + name, + u->url) < 0)) + error("Registry: failed to save log. Registry data may be lost in case of abnormal restart."); + + // we increase the counter even on failures + // so that the registry will be saved periodically + registry.log_count++; + + registry_log_unlock(); + + // this must be outside the log_lock(), or a deadlock will happen. + // registry_save() checks the same inside the log_lock, so only + // one thread will save the db + if(unlikely(registry_should_save_db())) + registry_save(); + } } static inline int registry_log_open_nolock(void) { - if(registry.log_fp) - fclose(registry.log_fp); + if(registry.log_fp) + fclose(registry.log_fp); - registry.log_fp = fopen(registry.log_filename, "a"); + registry.log_fp = fopen(registry.log_filename, "a"); - if(registry.log_fp) { - if (setvbuf(registry.log_fp, NULL, _IOLBF, 0) != 0) - error("Cannot set line buffering on registry log file."); - return 0; - } + if(registry.log_fp) { + if (setvbuf(registry.log_fp, NULL, _IOLBF, 0) != 0) + error("Cannot set line buffering on registry log file."); + return 0; + } - error("Cannot open registry log file '%s'. Registry data will be lost in case of netdata or server crash.", registry.log_filename); - return -1; + error("Cannot open registry log file '%s'. Registry data will be lost in case of netdata or server crash.", registry.log_filename); + return -1; } static inline void registry_log_close_nolock(void) { - if(registry.log_fp) { - fclose(registry.log_fp); - registry.log_fp = NULL; - } + if(registry.log_fp) { + fclose(registry.log_fp); + registry.log_fp = NULL; + } } static inline void registry_log_recreate_nolock(void) { - if(registry.log_fp != NULL) { - registry_log_close_nolock(); + if(registry.log_fp != NULL) { + registry_log_close_nolock(); - // open it with truncate - registry.log_fp = fopen(registry.log_filename, "w"); - if(registry.log_fp) fclose(registry.log_fp); - else error("Cannot truncate registry log '%s'", registry.log_filename); + // open it with truncate + registry.log_fp = fopen(registry.log_filename, "w"); + if(registry.log_fp) fclose(registry.log_fp); + else error("Cannot truncate registry log '%s'", registry.log_filename); - registry.log_fp = NULL; + registry.log_fp = NULL; - registry_log_open_nolock(); - } + registry_log_open_nolock(); + } } int registry_log_load(void) { - char *s, buf[4096 + 1]; - size_t line = -1; - - // closing the log is required here - // otherwise we will append to it the values we read - registry_log_close_nolock(); - - debug(D_REGISTRY, "Registry: loading active db from: %s", registry.log_filename); - FILE *fp = fopen(registry.log_filename, "r"); - if(!fp) - error("Registry: cannot open registry file: %s", registry.log_filename); - else { - line = 0; - size_t len = 0; - while ((s = fgets_trim_len(buf, 4096, fp, &len))) { - line++; - - switch (s[0]) { - case 'A': // accesses - case 'D': // deletes - - // verify it is valid - if (unlikely(len < 85 || s[1] != '\t' || s[10] != '\t' || s[47] != '\t' || s[84] != '\t')) { - error("Registry: log line %u is wrong (len = %zu).", line, len); - continue; - } - s[1] = s[10] = s[47] = s[84] = '\0'; - - // get the variables - time_t when = strtoul(&s[2], NULL, 16); - char *person_guid = &s[11]; - char *machine_guid = &s[48]; - char *name = &s[85]; - - // skip the name to find the url - char *url = name; - while(*url && *url != '\t') url++; - if(!*url) { - error("Registry: log line %u does not have a url.", line); - continue; - } - *url++ = '\0'; - - // make sure the person exists - // without this, a new person guid will be created - PERSON *p = registry_person_find(person_guid); - if(!p) p = registry_person_allocate(person_guid, when); - - if(s[0] == 'A') - registry_request_access(p->guid, machine_guid, url, name, when); - else - registry_request_delete(p->guid, machine_guid, url, name, when); - - break; - - default: - error("Registry: ignoring line %zu of filename '%s': %s.", line, registry.log_filename, s); - break; - } - } - } - - // open the log again - registry_log_open_nolock(); - - return line; + char *s, buf[4096 + 1]; + size_t line = -1; + + // closing the log is required here + // otherwise we will append to it the values we read + registry_log_close_nolock(); + + debug(D_REGISTRY, "Registry: loading active db from: %s", registry.log_filename); + FILE *fp = fopen(registry.log_filename, "r"); + if(!fp) + error("Registry: cannot open registry file: %s", registry.log_filename); + else { + line = 0; + size_t len = 0; + while ((s = fgets_trim_len(buf, 4096, fp, &len))) { + line++; + + switch (s[0]) { + case 'A': // accesses + case 'D': // deletes + + // verify it is valid + if (unlikely(len < 85 || s[1] != '\t' || s[10] != '\t' || s[47] != '\t' || s[84] != '\t')) { + error("Registry: log line %zu is wrong (len = %zu).", line, len); + continue; + } + s[1] = s[10] = s[47] = s[84] = '\0'; + + // get the variables + time_t when = strtoul(&s[2], NULL, 16); + char *person_guid = &s[11]; + char *machine_guid = &s[48]; + char *name = &s[85]; + + // skip the name to find the url + char *url = name; + while(*url && *url != '\t') url++; + if(!*url) { + error("Registry: log line %zu does not have a url.", line); + continue; + } + *url++ = '\0'; + + // make sure the person exists + // without this, a new person guid will be created + PERSON *p = registry_person_find(person_guid); + if(!p) p = registry_person_allocate(person_guid, when); + + if(s[0] == 'A') + registry_request_access(p->guid, machine_guid, url, name, when); + else + registry_request_delete(p->guid, machine_guid, url, name, when); + + break; + + default: + error("Registry: ignoring line %zu of filename '%s': %s.", line, registry.log_filename, s); + break; + } + } + + fclose(fp); + } + + // open the log again + registry_log_open_nolock(); + + return line; } @@ -831,176 +804,176 @@ int registry_log_load(void) { // REGISTRY REQUESTS PERSON *registry_request_access(char *person_guid, char *machine_guid, char *url, char *name, time_t when) { - debug(D_REGISTRY, "registry_request_access('%s', '%s', '%s'): NEW REQUEST", (person_guid)?person_guid:"", machine_guid, url); + debug(D_REGISTRY, "registry_request_access('%s', '%s', '%s'): NEW REQUEST", (person_guid)?person_guid:"", machine_guid, url); - MACHINE *m = registry_machine_get(machine_guid, when); - if(!m) return NULL; + MACHINE *m = registry_machine_get(machine_guid, when); + if(!m) return NULL; - // make sure the name is valid - size_t namelen; - name = registry_fix_machine_name(name, &namelen); + // make sure the name is valid + size_t namelen; + name = registry_fix_machine_name(name, &namelen); - size_t urllen; - url = registry_fix_url(url, &urllen); + size_t urllen; + url = registry_fix_url(url, &urllen); - URL *u = registry_url_get(url, urllen); - PERSON *p = registry_person_get(person_guid, when); + URL *u = registry_url_get(url, urllen); + PERSON *p = registry_person_get(person_guid, when); - registry_person_link_to_url(p, m, u, name, namelen, when); - registry_machine_link_to_url(p, m, u, when); + registry_person_link_to_url(p, m, u, name, namelen, when); + registry_machine_link_to_url(p, m, u, when); - registry_log('A', p, m, u, name); + registry_log('A', p, m, u, name); - registry.usages_count++; - return p; + registry.usages_count++; + return p; } // verify the person, the machine and the URL exist in our DB PERSON_URL *registry_verify_request(char *person_guid, char *machine_guid, char *url, PERSON **pp, MACHINE **mm) { - char pbuf[36 + 1], mbuf[36 + 1]; - - if(!person_guid || !*person_guid || !machine_guid || !*machine_guid || !url || !*url) { - info("Registry Request Verification: invalid request! person: '%s', machine '%s', url '%s'", person_guid?person_guid:"UNSET", machine_guid?machine_guid:"UNSET", url?url:"UNSET"); - return NULL; - } - - // normalize the url - url = registry_fix_url(url, NULL); - - // make sure the person GUID is valid - if(registry_regenerate_guid(person_guid, pbuf) == -1) { - info("Registry Request Verification: invalid person GUID, person: '%s', machine '%s', url '%s'", person_guid, machine_guid, url); - return NULL; - } - person_guid = pbuf; - - // make sure the machine GUID is valid - if(registry_regenerate_guid(machine_guid, mbuf) == -1) { - info("Registry Request Verification: invalid machine GUID, person: '%s', machine '%s', url '%s'", person_guid, machine_guid, url); - return NULL; - } - machine_guid = mbuf; - - // make sure the machine exists - MACHINE *m = registry_machine_find(machine_guid); - if(!m) { - info("Registry Request Verification: machine not found, person: '%s', machine '%s', url '%s'", person_guid, machine_guid, url); - return NULL; - } - if(mm) *mm = m; - - // make sure the person exist - PERSON *p = registry_person_find(person_guid); - if(!p) { - info("Registry Request Verification: person not found, person: '%s', machine '%s', url '%s'", person_guid, machine_guid, url); - return NULL; - } - if(pp) *pp = p; - - PERSON_URL *pu = dictionary_get(p->urls, url); - if(!pu) { - info("Registry Request Verification: URL not found for person, person: '%s', machine '%s', url '%s'", person_guid, machine_guid, url); - return NULL; - } - return pu; + char pbuf[36 + 1], mbuf[36 + 1]; + + if(!person_guid || !*person_guid || !machine_guid || !*machine_guid || !url || !*url) { + info("Registry Request Verification: invalid request! person: '%s', machine '%s', url '%s'", person_guid?person_guid:"UNSET", machine_guid?machine_guid:"UNSET", url?url:"UNSET"); + return NULL; + } + + // normalize the url + url = registry_fix_url(url, NULL); + + // make sure the person GUID is valid + if(registry_regenerate_guid(person_guid, pbuf) == -1) { + info("Registry Request Verification: invalid person GUID, person: '%s', machine '%s', url '%s'", person_guid, machine_guid, url); + return NULL; + } + person_guid = pbuf; + + // make sure the machine GUID is valid + if(registry_regenerate_guid(machine_guid, mbuf) == -1) { + info("Registry Request Verification: invalid machine GUID, person: '%s', machine '%s', url '%s'", person_guid, machine_guid, url); + return NULL; + } + machine_guid = mbuf; + + // make sure the machine exists + MACHINE *m = registry_machine_find(machine_guid); + if(!m) { + info("Registry Request Verification: machine not found, person: '%s', machine '%s', url '%s'", person_guid, machine_guid, url); + return NULL; + } + if(mm) *mm = m; + + // make sure the person exist + PERSON *p = registry_person_find(person_guid); + if(!p) { + info("Registry Request Verification: person not found, person: '%s', machine '%s', url '%s'", person_guid, machine_guid, url); + return NULL; + } + if(pp) *pp = p; + + PERSON_URL *pu = dictionary_get(p->urls, url); + if(!pu) { + info("Registry Request Verification: URL not found for person, person: '%s', machine '%s', url '%s'", person_guid, machine_guid, url); + return NULL; + } + return pu; } PERSON *registry_request_delete(char *person_guid, char *machine_guid, char *url, char *delete_url, time_t when) { - (void)when; + (void)when; - PERSON *p = NULL; - MACHINE *m = NULL; - PERSON_URL *pu = registry_verify_request(person_guid, machine_guid, url, &p, &m); - if(!pu || !p || !m) return NULL; + PERSON *p = NULL; + MACHINE *m = NULL; + PERSON_URL *pu = registry_verify_request(person_guid, machine_guid, url, &p, &m); + if(!pu || !p || !m) return NULL; - // normalize the url - delete_url = registry_fix_url(delete_url, NULL); + // normalize the url + delete_url = registry_fix_url(delete_url, NULL); - // make sure the user is not deleting the url it uses - if(!strcmp(delete_url, pu->url->url)) { - info("Registry Delete Request: delete URL is the one currently accessed, person: '%s', machine '%s', url '%s', delete url '%s'", p->guid, m->guid, pu->url->url, delete_url); - return NULL; - } + // make sure the user is not deleting the url it uses + if(!strcmp(delete_url, pu->url->url)) { + info("Registry Delete Request: delete URL is the one currently accessed, person: '%s', machine '%s', url '%s', delete url '%s'", p->guid, m->guid, pu->url->url, delete_url); + return NULL; + } - registry_person_urls_lock(p); + registry_person_urls_lock(p); - PERSON_URL *dpu = dictionary_get(p->urls, delete_url); - if(!dpu) { - info("Registry Delete Request: URL not found for person: '%s', machine '%s', url '%s', delete url '%s'", p->guid, m->guid, pu->url->url, delete_url); - registry_person_urls_unlock(p); - return NULL; - } + PERSON_URL *dpu = dictionary_get(p->urls, delete_url); + if(!dpu) { + info("Registry Delete Request: URL not found for person: '%s', machine '%s', url '%s', delete url '%s'", p->guid, m->guid, pu->url->url, delete_url); + registry_person_urls_unlock(p); + return NULL; + } - registry_log('D', p, m, pu->url, dpu->url->url); + registry_log('D', p, m, pu->url, dpu->url->url); - dictionary_del(p->urls, dpu->url->url); - registry_url_unlink_nolock(dpu->url); - free(dpu); + dictionary_del(p->urls, dpu->url->url); + registry_url_unlink_nolock(dpu->url); + freez(dpu); - registry_person_urls_unlock(p); - return p; + registry_person_urls_unlock(p); + return p; } // a structure to pass to the dictionary_get_all() callback handler struct machine_request_callback_data { - MACHINE *find_this_machine; - PERSON_URL *result; + MACHINE *find_this_machine; + PERSON_URL *result; }; // the callback function // this will be run for every PERSON_URL of this PERSON int machine_request_callback(void *entry, void *data) { - PERSON_URL *mypu = (PERSON_URL *)entry; - struct machine_request_callback_data *myrdata = (struct machine_request_callback_data *)data; + PERSON_URL *mypu = (PERSON_URL *)entry; + struct machine_request_callback_data *myrdata = (struct machine_request_callback_data *)data; - if(mypu->machine == myrdata->find_this_machine) { - myrdata->result = mypu; - return -1; // this will also stop the walk through - } + if(mypu->machine == myrdata->find_this_machine) { + myrdata->result = mypu; + return -1; // this will also stop the walk through + } - return 0; // continue + return 0; // continue } MACHINE *registry_request_machine(char *person_guid, char *machine_guid, char *url, char *request_machine, time_t when) { - (void)when; + (void)when; - char mbuf[36 + 1]; + char mbuf[36 + 1]; - PERSON *p = NULL; - MACHINE *m = NULL; - PERSON_URL *pu = registry_verify_request(person_guid, machine_guid, url, &p, &m); - if(!pu || !p || !m) return NULL; + PERSON *p = NULL; + MACHINE *m = NULL; + PERSON_URL *pu = registry_verify_request(person_guid, machine_guid, url, &p, &m); + if(!pu || !p || !m) return NULL; - // make sure the machine GUID is valid - if(registry_regenerate_guid(request_machine, mbuf) == -1) { - info("Registry Machine URLs request: invalid machine GUID, person: '%s', machine '%s', url '%s', request machine '%s'", p->guid, m->guid, pu->url->url, request_machine); - return NULL; - } - request_machine = mbuf; + // make sure the machine GUID is valid + if(registry_regenerate_guid(request_machine, mbuf) == -1) { + info("Registry Machine URLs request: invalid machine GUID, person: '%s', machine '%s', url '%s', request machine '%s'", p->guid, m->guid, pu->url->url, request_machine); + return NULL; + } + request_machine = mbuf; - // make sure the machine exists - m = registry_machine_find(request_machine); - if(!m) { - info("Registry Machine URLs request: machine not found, person: '%s', machine '%s', url '%s', request machine '%s'", p->guid, m->guid, pu->url->url, request_machine); - return NULL; - } + // make sure the machine exists + m = registry_machine_find(request_machine); + if(!m) { + info("Registry Machine URLs request: machine not found, person: '%s', machine '%s', url '%s', request machine '%s'", p->guid, m->guid, pu->url->url, request_machine); + return NULL; + } - // Verify the user has in the past accessed this machine - // We will walk through the PERSON_URLs to find the machine - // linking to our machine + // Verify the user has in the past accessed this machine + // We will walk through the PERSON_URLs to find the machine + // linking to our machine - // a structure to pass to the dictionary_get_all() callback handler - struct machine_request_callback_data rdata = { m, NULL }; + // a structure to pass to the dictionary_get_all() callback handler + struct machine_request_callback_data rdata = { m, NULL }; - // request a walk through on the dictionary - // no need for locking here, the underlying dictionary has its own - dictionary_get_all(p->urls, machine_request_callback, &rdata); + // request a walk through on the dictionary + // no need for locking here, the underlying dictionary has its own + dictionary_get_all(p->urls, machine_request_callback, &rdata); - if(rdata.result) - return m; + if(rdata.result) + return m; - return NULL; + return NULL; } @@ -1011,231 +984,243 @@ MACHINE *registry_request_machine(char *person_guid, char *machine_guid, char *u #define REGISTRY_STATUS_FAILED "failed" #define REGISTRY_STATUS_DISABLED "disabled" -static inline void registry_set_person_cookie(struct web_client *w, PERSON *p) { - char edate[100]; - time_t et = time(NULL) + registry.persons_expiration; - struct tm etmbuf, *etm = gmtime_r(&et, &etmbuf); - strftime(edate, sizeof(edate), "%a, %d %b %Y %H:%M:%S %Z", etm); +int registry_verify_cookies_redirects(void) { + return registry.verify_cookies_redirects; +} - snprintfz(w->cookie1, COOKIE_MAX, NETDATA_REGISTRY_COOKIE_NAME "=%s; Expires=%s", p->guid, edate); +const char *registry_to_announce(void) { + return registry.registry_to_announce; +} - if(registry.registry_domain && registry.registry_domain[0]) - snprintfz(w->cookie2, COOKIE_MAX, NETDATA_REGISTRY_COOKIE_NAME "=%s; Domain=%s; Expires=%s", p->guid, registry.registry_domain, edate); +void registry_set_cookie(struct web_client *w, const char *guid) { + char edate[100]; + time_t et = time(NULL) + registry.persons_expiration; + struct tm etmbuf, *etm = gmtime_r(&et, &etmbuf); + strftime(edate, sizeof(edate), "%a, %d %b %Y %H:%M:%S %Z", etm); + + snprintfz(w->cookie1, COOKIE_MAX, NETDATA_REGISTRY_COOKIE_NAME "=%s; Expires=%s", guid, edate); + + if(registry.registry_domain && registry.registry_domain[0]) + snprintfz(w->cookie2, COOKIE_MAX, NETDATA_REGISTRY_COOKIE_NAME "=%s; Domain=%s; Expires=%s", guid, registry.registry_domain, edate); +} + +static inline void registry_set_person_cookie(struct web_client *w, PERSON *p) { + registry_set_cookie(w, p->guid); } static inline void registry_json_header(struct web_client *w, const char *action, const char *status) { - w->response.data->contenttype = CT_APPLICATION_JSON; - buffer_flush(w->response.data); - buffer_sprintf(w->response.data, "{\n\t\"action\": \"%s\",\n\t\"status\": \"%s\",\n\t\"hostname\": \"%s\",\n\t\"machine_guid\": \"%s\"", - action, status, registry.hostname, registry.machine_guid); + buffer_flush(w->response.data); + w->response.data->contenttype = CT_APPLICATION_JSON; + buffer_sprintf(w->response.data, "{\n\t\"action\": \"%s\",\n\t\"status\": \"%s\",\n\t\"hostname\": \"%s\",\n\t\"machine_guid\": \"%s\"", + action, status, registry.hostname, registry.machine_guid); } static inline void registry_json_footer(struct web_client *w) { - buffer_strcat(w->response.data, "\n}\n"); + buffer_strcat(w->response.data, "\n}\n"); } int registry_request_hello_json(struct web_client *w) { - registry_json_header(w, "hello", REGISTRY_STATUS_OK); + registry_json_header(w, "hello", REGISTRY_STATUS_OK); - buffer_sprintf(w->response.data, ",\n\t\"registry\": \"%s\"", - registry.registry_to_announce); + buffer_sprintf(w->response.data, ",\n\t\"registry\": \"%s\"", + registry.registry_to_announce); - registry_json_footer(w); - return 200; + registry_json_footer(w); + return 200; } static inline int registry_json_disabled(struct web_client *w, const char *action) { - registry_json_header(w, action, REGISTRY_STATUS_DISABLED); + registry_json_header(w, action, REGISTRY_STATUS_DISABLED); - buffer_sprintf(w->response.data, ",\n\t\"registry\": \"%s\"", - registry.registry_to_announce); + buffer_sprintf(w->response.data, ",\n\t\"registry\": \"%s\"", + registry.registry_to_announce); - registry_json_footer(w); - return 200; + registry_json_footer(w); + return 200; } // structure used be the callbacks below struct registry_json_walk_person_urls_callback { - PERSON *p; - MACHINE *m; - struct web_client *w; - int count; + PERSON *p; + MACHINE *m; + struct web_client *w; + int count; }; // callback for rendering PERSON_URLs static inline int registry_json_person_url_callback(void *entry, void *data) { - PERSON_URL *pu = (PERSON_URL *)entry; - struct registry_json_walk_person_urls_callback *c = (struct registry_json_walk_person_urls_callback *)data; - struct web_client *w = c->w; + PERSON_URL *pu = (PERSON_URL *)entry; + struct registry_json_walk_person_urls_callback *c = (struct registry_json_walk_person_urls_callback *)data; + struct web_client *w = c->w; - if(unlikely(c->count++)) - buffer_strcat(w->response.data, ","); + if(unlikely(c->count++)) + buffer_strcat(w->response.data, ","); - buffer_sprintf(w->response.data, "\n\t\t[ \"%s\", \"%s\", %u000, %u, \"%s\" ]", - pu->machine->guid, pu->url->url, pu->last_t, pu->usages, pu->name); + buffer_sprintf(w->response.data, "\n\t\t[ \"%s\", \"%s\", %u000, %u, \"%s\" ]", + pu->machine->guid, pu->url->url, pu->last_t, pu->usages, pu->name); - return 1; + return 1; } // callback for rendering MACHINE_URLs static inline int registry_json_machine_url_callback(void *entry, void *data) { - MACHINE_URL *mu = (MACHINE_URL *)entry; - struct registry_json_walk_person_urls_callback *c = (struct registry_json_walk_person_urls_callback *)data; - struct web_client *w = c->w; - MACHINE *m = c->m; + MACHINE_URL *mu = (MACHINE_URL *)entry; + struct registry_json_walk_person_urls_callback *c = (struct registry_json_walk_person_urls_callback *)data; + struct web_client *w = c->w; + MACHINE *m = c->m; - if(unlikely(c->count++)) - buffer_strcat(w->response.data, ","); + if(unlikely(c->count++)) + buffer_strcat(w->response.data, ","); - buffer_sprintf(w->response.data, "\n\t\t[ \"%s\", \"%s\", %u000, %u ]", - m->guid, mu->url->url, mu->last_t, mu->usages); + buffer_sprintf(w->response.data, "\n\t\t[ \"%s\", \"%s\", %u000, %u ]", + m->guid, mu->url->url, mu->last_t, mu->usages); - return 1; + return 1; } // the main method for registering an access int registry_request_access_json(struct web_client *w, char *person_guid, char *machine_guid, char *url, char *name, time_t when) { - if(!registry.enabled) - return registry_json_disabled(w, "access"); + if(!registry.enabled) + return registry_json_disabled(w, "access"); - PERSON *p = registry_request_access(person_guid, machine_guid, url, name, when); - if(!p) { - registry_json_header(w, "access", REGISTRY_STATUS_FAILED); - registry_json_footer(w); - return 412; - } + PERSON *p = registry_request_access(person_guid, machine_guid, url, name, when); + if(!p) { + registry_json_header(w, "access", REGISTRY_STATUS_FAILED); + registry_json_footer(w); + return 412; + } - // set the cookie - registry_set_person_cookie(w, p); + // set the cookie + registry_set_person_cookie(w, p); - // generate the response - registry_json_header(w, "access", REGISTRY_STATUS_OK); + // generate the response + registry_json_header(w, "access", REGISTRY_STATUS_OK); - buffer_sprintf(w->response.data, ",\n\t\"person_guid\": \"%s\",\n\t\"urls\": [", p->guid); - struct registry_json_walk_person_urls_callback c = { p, NULL, w, 0 }; - dictionary_get_all(p->urls, registry_json_person_url_callback, &c); - buffer_strcat(w->response.data, "\n\t]\n"); + buffer_sprintf(w->response.data, ",\n\t\"person_guid\": \"%s\",\n\t\"urls\": [", p->guid); + struct registry_json_walk_person_urls_callback c = { p, NULL, w, 0 }; + dictionary_get_all(p->urls, registry_json_person_url_callback, &c); + buffer_strcat(w->response.data, "\n\t]\n"); - registry_json_footer(w); - return 200; + registry_json_footer(w); + return 200; } // the main method for deleting a URL from a person int registry_request_delete_json(struct web_client *w, char *person_guid, char *machine_guid, char *url, char *delete_url, time_t when) { - if(!registry.enabled) - return registry_json_disabled(w, "delete"); - - PERSON *p = registry_request_delete(person_guid, machine_guid, url, delete_url, when); - if(!p) { - registry_json_header(w, "delete", REGISTRY_STATUS_FAILED); - registry_json_footer(w); - return 412; - } - - // generate the response - registry_json_header(w, "delete", REGISTRY_STATUS_OK); - registry_json_footer(w); - return 200; + if(!registry.enabled) + return registry_json_disabled(w, "delete"); + + PERSON *p = registry_request_delete(person_guid, machine_guid, url, delete_url, when); + if(!p) { + registry_json_header(w, "delete", REGISTRY_STATUS_FAILED); + registry_json_footer(w); + return 412; + } + + // generate the response + registry_json_header(w, "delete", REGISTRY_STATUS_OK); + registry_json_footer(w); + return 200; } // the main method for searching the URLs of a netdata int registry_request_search_json(struct web_client *w, char *person_guid, char *machine_guid, char *url, char *request_machine, time_t when) { - if(!registry.enabled) - return registry_json_disabled(w, "search"); + if(!registry.enabled) + return registry_json_disabled(w, "search"); - MACHINE *m = registry_request_machine(person_guid, machine_guid, url, request_machine, when); - if(!m) { - registry_json_header(w, "search", REGISTRY_STATUS_FAILED); - registry_json_footer(w); - return 404; - } + MACHINE *m = registry_request_machine(person_guid, machine_guid, url, request_machine, when); + if(!m) { + registry_json_header(w, "search", REGISTRY_STATUS_FAILED); + registry_json_footer(w); + return 404; + } - registry_json_header(w, "search", REGISTRY_STATUS_OK); + registry_json_header(w, "search", REGISTRY_STATUS_OK); - buffer_strcat(w->response.data, ",\n\t\"urls\": ["); - struct registry_json_walk_person_urls_callback c = { NULL, m, w, 0 }; - dictionary_get_all(m->urls, registry_json_machine_url_callback, &c); - buffer_strcat(w->response.data, "\n\t]\n"); + buffer_strcat(w->response.data, ",\n\t\"urls\": ["); + struct registry_json_walk_person_urls_callback c = { NULL, m, w, 0 }; + dictionary_get_all(m->urls, registry_json_machine_url_callback, &c); + buffer_strcat(w->response.data, "\n\t]\n"); - registry_json_footer(w); - return 200; + registry_json_footer(w); + return 200; } // structure used be the callbacks below struct registry_person_url_callback_verify_machine_exists_data { - MACHINE *m; - int count; + MACHINE *m; + int count; }; int registry_person_url_callback_verify_machine_exists(void *entry, void *data) { - struct registry_person_url_callback_verify_machine_exists_data *d = (struct registry_person_url_callback_verify_machine_exists_data *)data; - PERSON_URL *pu = (PERSON_URL *)entry; - MACHINE *m = d->m; + struct registry_person_url_callback_verify_machine_exists_data *d = (struct registry_person_url_callback_verify_machine_exists_data *)data; + PERSON_URL *pu = (PERSON_URL *)entry; + MACHINE *m = d->m; - if(pu->machine == m) - d->count++; + if(pu->machine == m) + d->count++; - return 0; + return 0; } // the main method for switching user identity int registry_request_switch_json(struct web_client *w, char *person_guid, char *machine_guid, char *url, char *new_person_guid, time_t when) { - (void)url; - (void)when; - - if(!registry.enabled) - return registry_json_disabled(w, "switch"); - - PERSON *op = registry_person_find(person_guid); - if(!op) { - registry_json_header(w, "switch", REGISTRY_STATUS_FAILED); - registry_json_footer(w); - return 430; - } - - PERSON *np = registry_person_find(new_person_guid); - if(!np) { - registry_json_header(w, "switch", REGISTRY_STATUS_FAILED); - registry_json_footer(w); - return 431; - } - - MACHINE *m = registry_machine_find(machine_guid); - if(!m) { - registry_json_header(w, "switch", REGISTRY_STATUS_FAILED); - registry_json_footer(w); - return 432; - } - - struct registry_person_url_callback_verify_machine_exists_data data = { m, 0 }; - - // verify the old person has access to this machine - dictionary_get_all(op->urls, registry_person_url_callback_verify_machine_exists, &data); - if(!data.count) { - registry_json_header(w, "switch", REGISTRY_STATUS_FAILED); - registry_json_footer(w); - return 433; - } - - // verify the new person has access to this machine - data.count = 0; - dictionary_get_all(np->urls, registry_person_url_callback_verify_machine_exists, &data); - if(!data.count) { - registry_json_header(w, "switch", REGISTRY_STATUS_FAILED); - registry_json_footer(w); - return 434; - } - - // set the cookie of the new person - // the user just switched identity - registry_set_person_cookie(w, np); - - // generate the response - registry_json_header(w, "switch", REGISTRY_STATUS_OK); - buffer_sprintf(w->response.data, ",\n\t\"person_guid\": \"%s\"", np->guid); - registry_json_footer(w); - return 200; + (void)url; + (void)when; + + if(!registry.enabled) + return registry_json_disabled(w, "switch"); + + PERSON *op = registry_person_find(person_guid); + if(!op) { + registry_json_header(w, "switch", REGISTRY_STATUS_FAILED); + registry_json_footer(w); + return 430; + } + + PERSON *np = registry_person_find(new_person_guid); + if(!np) { + registry_json_header(w, "switch", REGISTRY_STATUS_FAILED); + registry_json_footer(w); + return 431; + } + + MACHINE *m = registry_machine_find(machine_guid); + if(!m) { + registry_json_header(w, "switch", REGISTRY_STATUS_FAILED); + registry_json_footer(w); + return 432; + } + + struct registry_person_url_callback_verify_machine_exists_data data = { m, 0 }; + + // verify the old person has access to this machine + dictionary_get_all(op->urls, registry_person_url_callback_verify_machine_exists, &data); + if(!data.count) { + registry_json_header(w, "switch", REGISTRY_STATUS_FAILED); + registry_json_footer(w); + return 433; + } + + // verify the new person has access to this machine + data.count = 0; + dictionary_get_all(np->urls, registry_person_url_callback_verify_machine_exists, &data); + if(!data.count) { + registry_json_header(w, "switch", REGISTRY_STATUS_FAILED); + registry_json_footer(w); + return 434; + } + + // set the cookie of the new person + // the user just switched identity + registry_set_person_cookie(w, np); + + // generate the response + registry_json_header(w, "switch", REGISTRY_STATUS_OK); + buffer_sprintf(w->response.data, ",\n\t\"person_guid\": \"%s\"", np->guid); + registry_json_footer(w); + return 200; } @@ -1243,47 +1228,49 @@ int registry_request_switch_json(struct web_client *w, char *person_guid, char * // REGISTRY THIS MACHINE UNIQUE ID char *registry_get_this_machine_guid(void) { - if(likely(registry.machine_guid[0])) - return registry.machine_guid; - - // read it from disk - int fd = open(registry.machine_guid_filename, O_RDONLY); - if(fd != -1) { - char buf[36 + 1]; - if(read(fd, buf, 36) != 36) - error("Failed to read machine GUID from '%s'", registry.machine_guid_filename); - else { - buf[36] = '\0'; - if(registry_regenerate_guid(buf, registry.machine_guid) == -1) { - error("Failed to validate machine GUID '%s' from '%s'. Ignoring it - this might mean this netdata will appear as duplicate in the registry.", - buf, registry.machine_guid_filename); - - registry.machine_guid[0] = '\0'; - } - } - close(fd); - } - - // generate a new one? - if(!registry.machine_guid[0]) { - uuid_t uuid; - - uuid_generate_time(uuid); - uuid_unparse_lower(uuid, registry.machine_guid); - registry.machine_guid[36] = '\0'; - - // save it - fd = open(registry.machine_guid_filename, O_WRONLY|O_CREAT|O_TRUNC, 444); - if(fd == -1) - fatal("Cannot create unique machine id file '%s'. Please fix this.", registry.machine_guid_filename); - - if(write(fd, registry.machine_guid, 36) != 36) - fatal("Cannot write the unique machine id file '%s'. Please fix this.", registry.machine_guid_filename); - - close(fd); - } - - return registry.machine_guid; + if(likely(registry.machine_guid[0])) + return registry.machine_guid; + + // read it from disk + int fd = open(registry.machine_guid_filename, O_RDONLY); + if(fd != -1) { + char buf[36 + 1]; + if(read(fd, buf, 36) != 36) + error("Failed to read machine GUID from '%s'", registry.machine_guid_filename); + else { + buf[36] = '\0'; + if(registry_regenerate_guid(buf, registry.machine_guid) == -1) { + error("Failed to validate machine GUID '%s' from '%s'. Ignoring it - this might mean this netdata will appear as duplicate in the registry.", + buf, registry.machine_guid_filename); + + registry.machine_guid[0] = '\0'; + } + } + close(fd); + } + + // generate a new one? + if(!registry.machine_guid[0]) { + uuid_t uuid; + + uuid_generate_time(uuid); + uuid_unparse_lower(uuid, registry.machine_guid); + registry.machine_guid[36] = '\0'; + + // save it + fd = open(registry.machine_guid_filename, O_WRONLY|O_CREAT|O_TRUNC, 444); + if(fd == -1) + fatal("Cannot create unique machine id file '%s'. Please fix this.", registry.machine_guid_filename); + + if(write(fd, registry.machine_guid, 36) != 36) + fatal("Cannot write the unique machine id file '%s'. Please fix this.", registry.machine_guid_filename); + + close(fd); + } + + setenv("NETDATA_REGISTRY_UNIQUE_ID", registry.machine_guid, 1); + + return registry.machine_guid; } @@ -1291,548 +1278,557 @@ char *registry_get_this_machine_guid(void) { // REGISTRY LOAD/SAVE int registry_machine_save_url(void *entry, void *file) { - MACHINE_URL *mu = entry; - FILE *fp = file; + MACHINE_URL *mu = entry; + FILE *fp = file; - debug(D_REGISTRY, "Registry: registry_machine_save_url('%s')", mu->url->url); + debug(D_REGISTRY, "Registry: registry_machine_save_url('%s')", mu->url->url); - int ret = fprintf(fp, "V\t%08x\t%08x\t%08x\t%02x\t%s\n", - mu->first_t, - mu->last_t, - mu->usages, - mu->flags, - mu->url->url - ); + int ret = fprintf(fp, "V\t%08x\t%08x\t%08x\t%02x\t%s\n", + mu->first_t, + mu->last_t, + mu->usages, + mu->flags, + mu->url->url + ); - // error handling is done at registry_save() + // error handling is done at registry_save() - return ret; + return ret; } int registry_machine_save(void *entry, void *file) { - MACHINE *m = entry; - FILE *fp = file; + MACHINE *m = entry; + FILE *fp = file; - debug(D_REGISTRY, "Registry: registry_machine_save('%s')", m->guid); + debug(D_REGISTRY, "Registry: registry_machine_save('%s')", m->guid); - int ret = fprintf(fp, "M\t%08x\t%08x\t%08x\t%s\n", - m->first_t, - m->last_t, - m->usages, - m->guid - ); + int ret = fprintf(fp, "M\t%08x\t%08x\t%08x\t%s\n", + m->first_t, + m->last_t, + m->usages, + m->guid + ); - if(ret >= 0) { - int ret2 = dictionary_get_all(m->urls, registry_machine_save_url, fp); - if(ret2 < 0) return ret2; - ret += ret2; - } + if(ret >= 0) { + int ret2 = dictionary_get_all(m->urls, registry_machine_save_url, fp); + if(ret2 < 0) return ret2; + ret += ret2; + } - // error handling is done at registry_save() + // error handling is done at registry_save() - return ret; + return ret; } static inline int registry_person_save_url(void *entry, void *file) { - PERSON_URL *pu = entry; - FILE *fp = file; + PERSON_URL *pu = entry; + FILE *fp = file; - debug(D_REGISTRY, "Registry: registry_person_save_url('%s')", pu->url->url); + debug(D_REGISTRY, "Registry: registry_person_save_url('%s')", pu->url->url); - int ret = fprintf(fp, "U\t%08x\t%08x\t%08x\t%02x\t%s\t%s\t%s\n", - pu->first_t, - pu->last_t, - pu->usages, - pu->flags, - pu->machine->guid, - pu->name, - pu->url->url - ); + int ret = fprintf(fp, "U\t%08x\t%08x\t%08x\t%02x\t%s\t%s\t%s\n", + pu->first_t, + pu->last_t, + pu->usages, + pu->flags, + pu->machine->guid, + pu->name, + pu->url->url + ); - // error handling is done at registry_save() + // error handling is done at registry_save() - return ret; + return ret; } static inline int registry_person_save(void *entry, void *file) { - PERSON *p = entry; - FILE *fp = file; + PERSON *p = entry; + FILE *fp = file; - debug(D_REGISTRY, "Registry: registry_person_save('%s')", p->guid); + debug(D_REGISTRY, "Registry: registry_person_save('%s')", p->guid); - int ret = fprintf(fp, "P\t%08x\t%08x\t%08x\t%s\n", - p->first_t, - p->last_t, - p->usages, - p->guid - ); + int ret = fprintf(fp, "P\t%08x\t%08x\t%08x\t%s\n", + p->first_t, + p->last_t, + p->usages, + p->guid + ); - if(ret >= 0) { - int ret2 = dictionary_get_all(p->urls, registry_person_save_url, fp); - if (ret2 < 0) return ret2; - ret += ret2; - } + if(ret >= 0) { + int ret2 = dictionary_get_all(p->urls, registry_person_save_url, fp); + if (ret2 < 0) return ret2; + ret += ret2; + } - // error handling is done at registry_save() + // error handling is done at registry_save() - return ret; + return ret; } int registry_save(void) { - if(!registry.enabled) return -1; - - // make sure the log is not updated - registry_log_lock(); - - if(unlikely(!registry_should_save_db())) { - registry_log_unlock(); - return -2; - } - - char tmp_filename[FILENAME_MAX + 1]; - char old_filename[FILENAME_MAX + 1]; - - snprintfz(old_filename, FILENAME_MAX, "%s.old", registry.db_filename); - snprintfz(tmp_filename, FILENAME_MAX, "%s.tmp", registry.db_filename); - - debug(D_REGISTRY, "Registry: Creating file '%s'", tmp_filename); - FILE *fp = fopen(tmp_filename, "w"); - if(!fp) { - error("Registry: Cannot create file: %s", tmp_filename); - registry_log_unlock(); - return -1; - } - - // dictionary_get_all() has its own locking, so this is safe to do - - debug(D_REGISTRY, "Saving all machines"); - int bytes1 = dictionary_get_all(registry.machines, registry_machine_save, fp); - if(bytes1 < 0) { - error("Registry: Cannot save registry machines - return value %d", bytes1); - fclose(fp); - registry_log_unlock(); - return bytes1; - } - debug(D_REGISTRY, "Registry: saving machines took %d bytes", bytes1); - - debug(D_REGISTRY, "Saving all persons"); - int bytes2 = dictionary_get_all(registry.persons, registry_person_save, fp); - if(bytes2 < 0) { - error("Registry: Cannot save registry persons - return value %d", bytes2); - fclose(fp); - registry_log_unlock(); - return bytes2; - } - debug(D_REGISTRY, "Registry: saving persons took %d bytes", bytes2); - - // save the totals - fprintf(fp, "T\t%016llx\t%016llx\t%016llx\t%016llx\t%016llx\t%016llx\n", - registry.persons_count, - registry.machines_count, - registry.usages_count + 1, // this is required - it is lost on db rotation - registry.urls_count, - registry.persons_urls_count, - registry.machines_urls_count - ); - - fclose(fp); - - errno = 0; - - // remove the .old db - debug(D_REGISTRY, "Registry: Removing old db '%s'", old_filename); - if(unlink(old_filename) == -1 && errno != ENOENT) - error("Registry: cannot remove old registry file '%s'", old_filename); - - // rename the db to .old - debug(D_REGISTRY, "Registry: Link current db '%s' to .old: '%s'", registry.db_filename, old_filename); - if(link(registry.db_filename, old_filename) == -1 && errno != ENOENT) - error("Registry: cannot move file '%s' to '%s'. Saving registry DB failed!", tmp_filename, registry.db_filename); - - else { - // remove the database (it is saved in .old) - debug(D_REGISTRY, "Registry: removing db '%s'", registry.db_filename); - if (unlink(registry.db_filename) == -1 && errno != ENOENT) - error("Registry: cannot remove old registry file '%s'", registry.db_filename); - - // move the .tmp to make it active - debug(D_REGISTRY, "Registry: linking tmp db '%s' to active db '%s'", tmp_filename, registry.db_filename); - if (link(tmp_filename, registry.db_filename) == -1) { - error("Registry: cannot move file '%s' to '%s'. Saving registry DB failed!", tmp_filename, - registry.db_filename); - - // move the .old back - debug(D_REGISTRY, "Registry: linking old db '%s' to active db '%s'", old_filename, registry.db_filename); - if(link(old_filename, registry.db_filename) == -1) - error("Registry: cannot move file '%s' to '%s'. Recovering the old registry DB failed!", old_filename, registry.db_filename); - } - else { - debug(D_REGISTRY, "Registry: removing tmp db '%s'", tmp_filename); - if(unlink(tmp_filename) == -1) - error("Registry: cannot remove tmp registry file '%s'", tmp_filename); - - // it has been moved successfully - // discard the current registry log - registry_log_recreate_nolock(); - - registry.log_count = 0; - } - } - - // continue operations - registry_log_unlock(); - - return -1; + if(!registry.enabled) return -1; + + // make sure the log is not updated + registry_log_lock(); + + if(unlikely(!registry_should_save_db())) { + registry_log_unlock(); + return -2; + } + + char tmp_filename[FILENAME_MAX + 1]; + char old_filename[FILENAME_MAX + 1]; + + snprintfz(old_filename, FILENAME_MAX, "%s.old", registry.db_filename); + snprintfz(tmp_filename, FILENAME_MAX, "%s.tmp", registry.db_filename); + + debug(D_REGISTRY, "Registry: Creating file '%s'", tmp_filename); + FILE *fp = fopen(tmp_filename, "w"); + if(!fp) { + error("Registry: Cannot create file: %s", tmp_filename); + registry_log_unlock(); + return -1; + } + + // dictionary_get_all() has its own locking, so this is safe to do + + debug(D_REGISTRY, "Saving all machines"); + int bytes1 = dictionary_get_all(registry.machines, registry_machine_save, fp); + if(bytes1 < 0) { + error("Registry: Cannot save registry machines - return value %d", bytes1); + fclose(fp); + registry_log_unlock(); + return bytes1; + } + debug(D_REGISTRY, "Registry: saving machines took %d bytes", bytes1); + + debug(D_REGISTRY, "Saving all persons"); + int bytes2 = dictionary_get_all(registry.persons, registry_person_save, fp); + if(bytes2 < 0) { + error("Registry: Cannot save registry persons - return value %d", bytes2); + fclose(fp); + registry_log_unlock(); + return bytes2; + } + debug(D_REGISTRY, "Registry: saving persons took %d bytes", bytes2); + + // save the totals + fprintf(fp, "T\t%016llx\t%016llx\t%016llx\t%016llx\t%016llx\t%016llx\n", + registry.persons_count, + registry.machines_count, + registry.usages_count + 1, // this is required - it is lost on db rotation + registry.urls_count, + registry.persons_urls_count, + registry.machines_urls_count + ); + + fclose(fp); + + errno = 0; + + // remove the .old db + debug(D_REGISTRY, "Registry: Removing old db '%s'", old_filename); + if(unlink(old_filename) == -1 && errno != ENOENT) + error("Registry: cannot remove old registry file '%s'", old_filename); + + // rename the db to .old + debug(D_REGISTRY, "Registry: Link current db '%s' to .old: '%s'", registry.db_filename, old_filename); + if(link(registry.db_filename, old_filename) == -1 && errno != ENOENT) + error("Registry: cannot move file '%s' to '%s'. Saving registry DB failed!", tmp_filename, registry.db_filename); + + else { + // remove the database (it is saved in .old) + debug(D_REGISTRY, "Registry: removing db '%s'", registry.db_filename); + if (unlink(registry.db_filename) == -1 && errno != ENOENT) + error("Registry: cannot remove old registry file '%s'", registry.db_filename); + + // move the .tmp to make it active + debug(D_REGISTRY, "Registry: linking tmp db '%s' to active db '%s'", tmp_filename, registry.db_filename); + if (link(tmp_filename, registry.db_filename) == -1) { + error("Registry: cannot move file '%s' to '%s'. Saving registry DB failed!", tmp_filename, + registry.db_filename); + + // move the .old back + debug(D_REGISTRY, "Registry: linking old db '%s' to active db '%s'", old_filename, registry.db_filename); + if(link(old_filename, registry.db_filename) == -1) + error("Registry: cannot move file '%s' to '%s'. Recovering the old registry DB failed!", old_filename, registry.db_filename); + } + else { + debug(D_REGISTRY, "Registry: removing tmp db '%s'", tmp_filename); + if(unlink(tmp_filename) == -1) + error("Registry: cannot remove tmp registry file '%s'", tmp_filename); + + // it has been moved successfully + // discard the current registry log + registry_log_recreate_nolock(); + + registry.log_count = 0; + } + } + + // continue operations + registry_log_unlock(); + + return -1; } static inline size_t registry_load(void) { - char *s, buf[4096 + 1]; - PERSON *p = NULL; - MACHINE *m = NULL; - URL *u = NULL; - size_t line = 0; - - debug(D_REGISTRY, "Registry: loading active db from: '%s'", registry.db_filename); - FILE *fp = fopen(registry.db_filename, "r"); - if(!fp) { - error("Registry: cannot open registry file: '%s'", registry.db_filename); - return 0; - } - - size_t len = 0; - buf[4096] = '\0'; - while((s = fgets_trim_len(buf, 4096, fp, &len))) { - line++; - - debug(D_REGISTRY, "Registry: read line %zu to length %zu: %s", line, len, s); - switch(*s) { - case 'T': // totals - if(unlikely(len != 103 || s[1] != '\t' || s[18] != '\t' || s[35] != '\t' || s[52] != '\t' || s[69] != '\t' || s[86] != '\t' || s[103] != '\0')) { - error("Registry totals line %u is wrong (len = %zu).", line, len); - continue; - } - registry.persons_count = strtoull(&s[2], NULL, 16); - registry.machines_count = strtoull(&s[19], NULL, 16); - registry.usages_count = strtoull(&s[36], NULL, 16); - registry.urls_count = strtoull(&s[53], NULL, 16); - registry.persons_urls_count = strtoull(&s[70], NULL, 16); - registry.machines_urls_count = strtoull(&s[87], NULL, 16); - break; - - case 'P': // person - m = NULL; - // verify it is valid - if(unlikely(len != 65 || s[1] != '\t' || s[10] != '\t' || s[19] != '\t' || s[28] != '\t' || s[65] != '\0')) { - error("Registry person line %u is wrong (len = %zu).", line, len); - continue; - } - - s[1] = s[10] = s[19] = s[28] = '\0'; - p = registry_person_allocate(&s[29], strtoul(&s[2], NULL, 16)); - p->last_t = strtoul(&s[11], NULL, 16); - p->usages = strtoul(&s[20], NULL, 16); - debug(D_REGISTRY, "Registry loaded person '%s', first: %u, last: %u, usages: %u", p->guid, p->first_t, p->last_t, p->usages); - break; - - case 'M': // machine - p = NULL; - // verify it is valid - if(unlikely(len != 65 || s[1] != '\t' || s[10] != '\t' || s[19] != '\t' || s[28] != '\t' || s[65] != '\0')) { - error("Registry person line %u is wrong (len = %zu).", line, len); - continue; - } - - s[1] = s[10] = s[19] = s[28] = '\0'; - m = registry_machine_allocate(&s[29], strtoul(&s[2], NULL, 16)); - m->last_t = strtoul(&s[11], NULL, 16); - m->usages = strtoul(&s[20], NULL, 16); - debug(D_REGISTRY, "Registry loaded machine '%s', first: %u, last: %u, usages: %u", m->guid, m->first_t, m->last_t, m->usages); - break; - - case 'U': // person URL - if(unlikely(!p)) { - error("Registry: ignoring line %zu, no person loaded: %s", line, s); - continue; - } - - // verify it is valid - if(len < 69 || s[1] != '\t' || s[10] != '\t' || s[19] != '\t' || s[28] != '\t' || s[31] != '\t' || s[68] != '\t') { - error("Registry person URL line %u is wrong (len = %zu).", line, len); - continue; - } - - s[1] = s[10] = s[19] = s[28] = s[31] = s[68] = '\0'; - - // skip the name to find the url - char *url = &s[69]; - while(*url && *url != '\t') url++; - if(!*url) { - error("Registry person URL line %u does not have a url.", line); - continue; - } - *url++ = '\0'; - - u = registry_url_allocate_nolock(url, strlen(url)); - - time_t first_t = strtoul(&s[2], NULL, 16); - - m = registry_machine_find(&s[32]); - if(!m) m = registry_machine_allocate(&s[32], first_t); - - PERSON_URL *pu = registry_person_url_allocate(p, m, u, &s[69], strlen(&s[69]), first_t); - pu->last_t = strtoul(&s[11], NULL, 16); - pu->usages = strtoul(&s[20], NULL, 16); - pu->flags = strtoul(&s[29], NULL, 16); - debug(D_REGISTRY, "Registry loaded person URL '%s' with name '%s' of machine '%s', first: %u, last: %u, usages: %u, flags: %02x", u->url, pu->name, m->guid, pu->first_t, pu->last_t, pu->usages, pu->flags); - break; - - case 'V': // machine URL - if(unlikely(!m)) { - error("Registry: ignoring line %zu, no machine loaded: %s", line, s); - continue; - } - - // verify it is valid - if(len < 32 || s[1] != '\t' || s[10] != '\t' || s[19] != '\t' || s[28] != '\t' || s[31] != '\t') { - error("Registry person URL line %u is wrong (len = %zu).", line, len); - continue; - } - - s[1] = s[10] = s[19] = s[28] = s[31] = '\0'; - u = registry_url_allocate_nolock(&s[32], strlen(&s[32])); - - MACHINE_URL *mu = registry_machine_url_allocate(m, u, strtoul(&s[2], NULL, 16)); - mu->last_t = strtoul(&s[11], NULL, 16); - mu->usages = strtoul(&s[20], NULL, 16); - mu->flags = strtoul(&s[29], NULL, 16); - debug(D_REGISTRY, "Registry loaded machine URL '%s', machine '%s', first: %u, last: %u, usages: %u, flags: %02x", u->url, m->guid, mu->first_t, mu->last_t, mu->usages, mu->flags); - break; - - default: - error("Registry: ignoring line %zu of filename '%s': %s.", line, registry.db_filename, s); - break; - } - } - fclose(fp); - - return line; + char *s, buf[4096 + 1]; + PERSON *p = NULL; + MACHINE *m = NULL; + URL *u = NULL; + size_t line = 0; + + debug(D_REGISTRY, "Registry: loading active db from: '%s'", registry.db_filename); + FILE *fp = fopen(registry.db_filename, "r"); + if(!fp) { + error("Registry: cannot open registry file: '%s'", registry.db_filename); + return 0; + } + + size_t len = 0; + buf[4096] = '\0'; + while((s = fgets_trim_len(buf, 4096, fp, &len))) { + line++; + + debug(D_REGISTRY, "Registry: read line %zu to length %zu: %s", line, len, s); + switch(*s) { + case 'T': // totals + if(unlikely(len != 103 || s[1] != '\t' || s[18] != '\t' || s[35] != '\t' || s[52] != '\t' || s[69] != '\t' || s[86] != '\t' || s[103] != '\0')) { + error("Registry totals line %zu is wrong (len = %zu).", line, len); + continue; + } + registry.persons_count = strtoull(&s[2], NULL, 16); + registry.machines_count = strtoull(&s[19], NULL, 16); + registry.usages_count = strtoull(&s[36], NULL, 16); + registry.urls_count = strtoull(&s[53], NULL, 16); + registry.persons_urls_count = strtoull(&s[70], NULL, 16); + registry.machines_urls_count = strtoull(&s[87], NULL, 16); + break; + + case 'P': // person + m = NULL; + // verify it is valid + if(unlikely(len != 65 || s[1] != '\t' || s[10] != '\t' || s[19] != '\t' || s[28] != '\t' || s[65] != '\0')) { + error("Registry person line %zu is wrong (len = %zu).", line, len); + continue; + } + + s[1] = s[10] = s[19] = s[28] = '\0'; + p = registry_person_allocate(&s[29], strtoul(&s[2], NULL, 16)); + p->last_t = strtoul(&s[11], NULL, 16); + p->usages = strtoul(&s[20], NULL, 16); + debug(D_REGISTRY, "Registry loaded person '%s', first: %u, last: %u, usages: %u", p->guid, p->first_t, p->last_t, p->usages); + break; + + case 'M': // machine + p = NULL; + // verify it is valid + if(unlikely(len != 65 || s[1] != '\t' || s[10] != '\t' || s[19] != '\t' || s[28] != '\t' || s[65] != '\0')) { + error("Registry person line %zu is wrong (len = %zu).", line, len); + continue; + } + + s[1] = s[10] = s[19] = s[28] = '\0'; + m = registry_machine_allocate(&s[29], strtoul(&s[2], NULL, 16)); + m->last_t = strtoul(&s[11], NULL, 16); + m->usages = strtoul(&s[20], NULL, 16); + debug(D_REGISTRY, "Registry loaded machine '%s', first: %u, last: %u, usages: %u", m->guid, m->first_t, m->last_t, m->usages); + break; + + case 'U': // person URL + if(unlikely(!p)) { + error("Registry: ignoring line %zu, no person loaded: %s", line, s); + continue; + } + + // verify it is valid + if(len < 69 || s[1] != '\t' || s[10] != '\t' || s[19] != '\t' || s[28] != '\t' || s[31] != '\t' || s[68] != '\t') { + error("Registry person URL line %zu is wrong (len = %zu).", line, len); + continue; + } + + s[1] = s[10] = s[19] = s[28] = s[31] = s[68] = '\0'; + + // skip the name to find the url + char *url = &s[69]; + while(*url && *url != '\t') url++; + if(!*url) { + error("Registry person URL line %zu does not have a url.", line); + continue; + } + *url++ = '\0'; + + u = registry_url_allocate_nolock(url, strlen(url)); + + time_t first_t = strtoul(&s[2], NULL, 16); + + m = registry_machine_find(&s[32]); + if(!m) m = registry_machine_allocate(&s[32], first_t); + + PERSON_URL *pu = registry_person_url_allocate(p, m, u, &s[69], strlen(&s[69]), first_t); + pu->last_t = strtoul(&s[11], NULL, 16); + pu->usages = strtoul(&s[20], NULL, 16); + pu->flags = strtoul(&s[29], NULL, 16); + debug(D_REGISTRY, "Registry loaded person URL '%s' with name '%s' of machine '%s', first: %u, last: %u, usages: %u, flags: %02x", u->url, pu->name, m->guid, pu->first_t, pu->last_t, pu->usages, pu->flags); + break; + + case 'V': // machine URL + if(unlikely(!m)) { + error("Registry: ignoring line %zu, no machine loaded: %s", line, s); + continue; + } + + // verify it is valid + if(len < 32 || s[1] != '\t' || s[10] != '\t' || s[19] != '\t' || s[28] != '\t' || s[31] != '\t') { + error("Registry person URL line %zu is wrong (len = %zu).", line, len); + continue; + } + + s[1] = s[10] = s[19] = s[28] = s[31] = '\0'; + u = registry_url_allocate_nolock(&s[32], strlen(&s[32])); + + MACHINE_URL *mu = registry_machine_url_allocate(m, u, strtoul(&s[2], NULL, 16)); + mu->last_t = strtoul(&s[11], NULL, 16); + mu->usages = strtoul(&s[20], NULL, 16); + mu->flags = strtoul(&s[29], NULL, 16); + debug(D_REGISTRY, "Registry loaded machine URL '%s', machine '%s', first: %u, last: %u, usages: %u, flags: %02x", u->url, m->guid, mu->first_t, mu->last_t, mu->usages, mu->flags); + break; + + default: + error("Registry: ignoring line %zu of filename '%s': %s.", line, registry.db_filename, s); + break; + } + } + fclose(fp); + + return line; } // ---------------------------------------------------------------------------- // REGISTRY int registry_init(void) { - char filename[FILENAME_MAX + 1]; - - // registry enabled? - registry.enabled = config_get_boolean("registry", "enabled", 0); - - // pathnames - registry.pathname = config_get("registry", "registry db directory", VARLIB_DIR "/registry"); - if(mkdir(registry.pathname, 0755) == -1 && errno != EEXIST) { - error("Cannot create directory '%s'. Registry disabled.", registry.pathname); - registry.enabled = 0; - return -1; - } - - // filenames - snprintfz(filename, FILENAME_MAX, "%s/netdata.public.unique.id", registry.pathname); - registry.machine_guid_filename = config_get("registry", "netdata unique id file", filename); - registry_get_this_machine_guid(); - - snprintfz(filename, FILENAME_MAX, "%s/registry.db", registry.pathname); - registry.db_filename = config_get("registry", "registry db file", filename); - - snprintfz(filename, FILENAME_MAX, "%s/registry-log.db", registry.pathname); - registry.log_filename = config_get("registry", "registry log file", filename); - - // configuration options - registry.save_registry_every_entries = config_get_number("registry", "registry save db every new entries", 1000000); - registry.persons_expiration = config_get_number("registry", "registry expire idle persons days", 365) * 86400; - registry.registry_domain = config_get("registry", "registry domain", ""); - registry.registry_to_announce = config_get("registry", "registry to announce", "https://registry.my-netdata.io"); - registry.hostname = config_get("registry", "registry hostname", config_get("global", "hostname", hostname)); - - registry.max_url_length = config_get_number("registry", "max URL length", 1024); - registry.max_name_length = config_get_number("registry", "max URL name length", 50); - - - // initialize entries counters - registry.persons_count = 0; - registry.machines_count = 0; - registry.usages_count = 0; - registry.urls_count = 0; - registry.persons_urls_count = 0; - registry.machines_urls_count = 0; - - // initialize memory counters - registry.persons_memory = 0; - registry.machines_memory = 0; - registry.urls_memory = 0; - registry.persons_urls_memory = 0; - registry.machines_urls_memory = 0; - - // initialize locks - pthread_mutex_init(®istry.persons_lock, NULL); - pthread_mutex_init(®istry.machines_lock, NULL); - pthread_mutex_init(®istry.urls_lock, NULL); - pthread_mutex_init(®istry.person_urls_lock, NULL); - pthread_mutex_init(®istry.machine_urls_lock, NULL); - - // create dictionaries - registry.persons = dictionary_create(DICTIONARY_FLAGS); - registry.machines = dictionary_create(DICTIONARY_FLAGS); - registry.urls = dictionary_create(DICTIONARY_FLAGS); - - // load the registry database - if(registry.enabled) { - registry_log_open_nolock(); - registry_load(); - registry_log_load(); - } - - return 0; + char filename[FILENAME_MAX + 1]; + + // registry enabled? + registry.enabled = config_get_boolean("registry", "enabled", 0); + + // pathnames + registry.pathname = config_get("registry", "registry db directory", VARLIB_DIR "/registry"); + if(mkdir(registry.pathname, 0770) == -1 && errno != EEXIST) + fatal("Cannot create directory '%s'.", registry.pathname); + + // filenames + snprintfz(filename, FILENAME_MAX, "%s/netdata.public.unique.id", registry.pathname); + registry.machine_guid_filename = config_get("registry", "netdata unique id file", filename); + registry_get_this_machine_guid(); + + snprintfz(filename, FILENAME_MAX, "%s/registry.db", registry.pathname); + registry.db_filename = config_get("registry", "registry db file", filename); + + snprintfz(filename, FILENAME_MAX, "%s/registry-log.db", registry.pathname); + registry.log_filename = config_get("registry", "registry log file", filename); + + // configuration options + registry.save_registry_every_entries = config_get_number("registry", "registry save db every new entries", 1000000); + registry.persons_expiration = config_get_number("registry", "registry expire idle persons days", 365) * 86400; + registry.registry_domain = config_get("registry", "registry domain", ""); + registry.registry_to_announce = config_get("registry", "registry to announce", "https://registry.my-netdata.io"); + registry.hostname = config_get("registry", "registry hostname", config_get("global", "hostname", hostname)); + registry.verify_cookies_redirects = config_get_boolean("registry", "verify browser cookies support", 1); + + setenv("NETDATA_REGISTRY_HOSTNAME", registry.hostname, 1); + setenv("NETDATA_REGISTRY_URL", registry.registry_to_announce, 1); + + registry.max_url_length = config_get_number("registry", "max URL length", 1024); + if(registry.max_url_length < 10) { + registry.max_url_length = 10; + config_set_number("registry", "max URL length", registry.max_url_length); + } + + registry.max_name_length = config_get_number("registry", "max URL name length", 50); + if(registry.max_name_length < 10) { + registry.max_name_length = 10; + config_set_number("registry", "max URL name length", registry.max_name_length); + } + + // initialize entries counters + registry.persons_count = 0; + registry.machines_count = 0; + registry.usages_count = 0; + registry.urls_count = 0; + registry.persons_urls_count = 0; + registry.machines_urls_count = 0; + + // initialize memory counters + registry.persons_memory = 0; + registry.machines_memory = 0; + registry.urls_memory = 0; + registry.persons_urls_memory = 0; + registry.machines_urls_memory = 0; + + // initialize locks + pthread_mutex_init(®istry.persons_lock, NULL); + pthread_mutex_init(®istry.machines_lock, NULL); + pthread_mutex_init(®istry.urls_lock, NULL); + pthread_mutex_init(®istry.person_urls_lock, NULL); + pthread_mutex_init(®istry.machine_urls_lock, NULL); + + // create dictionaries + registry.persons = dictionary_create(DICTIONARY_FLAGS); + registry.machines = dictionary_create(DICTIONARY_FLAGS); + registry.urls = dictionary_create(DICTIONARY_FLAGS); + + // load the registry database + if(registry.enabled) { + registry_log_open_nolock(); + registry_load(); + registry_log_load(); + } + + return 0; } void registry_free(void) { - if(!registry.enabled) return; + if(!registry.enabled) return; - // we need to destroy the dictionaries ourselves - // since the dictionaries use memory we allocated + // we need to destroy the dictionaries ourselves + // since the dictionaries use memory we allocated - while(registry.persons->values_index.root) { - PERSON *p = ((NAME_VALUE *)registry.persons->values_index.root)->value; + while(registry.persons->values_index.root) { + PERSON *p = ((NAME_VALUE *)registry.persons->values_index.root)->value; - // fprintf(stderr, "\nPERSON: '%s', first: %u, last: %u, usages: %u\n", p->guid, p->first_t, p->last_t, p->usages); + // fprintf(stderr, "\nPERSON: '%s', first: %u, last: %u, usages: %u\n", p->guid, p->first_t, p->last_t, p->usages); - while(p->urls->values_index.root) { - PERSON_URL *pu = ((NAME_VALUE *)p->urls->values_index.root)->value; + while(p->urls->values_index.root) { + PERSON_URL *pu = ((NAME_VALUE *)p->urls->values_index.root)->value; - // fprintf(stderr, "\tURL: '%s', first: %u, last: %u, usages: %u, flags: 0x%02x\n", pu->url->url, pu->first_t, pu->last_t, pu->usages, pu->flags); + // fprintf(stderr, "\tURL: '%s', first: %u, last: %u, usages: %u, flags: 0x%02x\n", pu->url->url, pu->first_t, pu->last_t, pu->usages, pu->flags); - debug(D_REGISTRY, "Registry: deleting url '%s' from person '%s'", pu->url->url, p->guid); - dictionary_del(p->urls, pu->url->url); + debug(D_REGISTRY, "Registry: deleting url '%s' from person '%s'", pu->url->url, p->guid); + dictionary_del(p->urls, pu->url->url); - debug(D_REGISTRY, "Registry: unlinking url '%s' from person", pu->url->url); - registry_url_unlink_nolock(pu->url); + debug(D_REGISTRY, "Registry: unlinking url '%s' from person", pu->url->url); + registry_url_unlink_nolock(pu->url); - debug(D_REGISTRY, "Registry: freeing person url"); - free(pu); - } + debug(D_REGISTRY, "Registry: freeing person url"); + freez(pu); + } - debug(D_REGISTRY, "Registry: deleting person '%s' from persons registry", p->guid); - dictionary_del(registry.persons, p->guid); + debug(D_REGISTRY, "Registry: deleting person '%s' from persons registry", p->guid); + dictionary_del(registry.persons, p->guid); - debug(D_REGISTRY, "Registry: destroying URL dictionary of person '%s'", p->guid); - dictionary_destroy(p->urls); + debug(D_REGISTRY, "Registry: destroying URL dictionary of person '%s'", p->guid); + dictionary_destroy(p->urls); - debug(D_REGISTRY, "Registry: freeing person '%s'", p->guid); - free(p); - } + debug(D_REGISTRY, "Registry: freeing person '%s'", p->guid); + freez(p); + } - while(registry.machines->values_index.root) { - MACHINE *m = ((NAME_VALUE *)registry.machines->values_index.root)->value; + while(registry.machines->values_index.root) { + MACHINE *m = ((NAME_VALUE *)registry.machines->values_index.root)->value; - // fprintf(stderr, "\nMACHINE: '%s', first: %u, last: %u, usages: %u\n", m->guid, m->first_t, m->last_t, m->usages); + // fprintf(stderr, "\nMACHINE: '%s', first: %u, last: %u, usages: %u\n", m->guid, m->first_t, m->last_t, m->usages); - while(m->urls->values_index.root) { - MACHINE_URL *mu = ((NAME_VALUE *)m->urls->values_index.root)->value; + while(m->urls->values_index.root) { + MACHINE_URL *mu = ((NAME_VALUE *)m->urls->values_index.root)->value; - // fprintf(stderr, "\tURL: '%s', first: %u, last: %u, usages: %u, flags: 0x%02x\n", mu->url->url, mu->first_t, mu->last_t, mu->usages, mu->flags); + // fprintf(stderr, "\tURL: '%s', first: %u, last: %u, usages: %u, flags: 0x%02x\n", mu->url->url, mu->first_t, mu->last_t, mu->usages, mu->flags); - //debug(D_REGISTRY, "Registry: destroying persons dictionary from url '%s'", mu->url->url); - //dictionary_destroy(mu->persons); + //debug(D_REGISTRY, "Registry: destroying persons dictionary from url '%s'", mu->url->url); + //dictionary_destroy(mu->persons); - debug(D_REGISTRY, "Registry: deleting url '%s' from person '%s'", mu->url->url, m->guid); - dictionary_del(m->urls, mu->url->url); + debug(D_REGISTRY, "Registry: deleting url '%s' from person '%s'", mu->url->url, m->guid); + dictionary_del(m->urls, mu->url->url); - debug(D_REGISTRY, "Registry: unlinking url '%s' from machine", mu->url->url); - registry_url_unlink_nolock(mu->url); + debug(D_REGISTRY, "Registry: unlinking url '%s' from machine", mu->url->url); + registry_url_unlink_nolock(mu->url); - debug(D_REGISTRY, "Registry: freeing machine url"); - free(mu); - } + debug(D_REGISTRY, "Registry: freeing machine url"); + freez(mu); + } - debug(D_REGISTRY, "Registry: deleting machine '%s' from machines registry", m->guid); - dictionary_del(registry.machines, m->guid); + debug(D_REGISTRY, "Registry: deleting machine '%s' from machines registry", m->guid); + dictionary_del(registry.machines, m->guid); - debug(D_REGISTRY, "Registry: destroying URL dictionary of machine '%s'", m->guid); - dictionary_destroy(m->urls); + debug(D_REGISTRY, "Registry: destroying URL dictionary of machine '%s'", m->guid); + dictionary_destroy(m->urls); - debug(D_REGISTRY, "Registry: freeing machine '%s'", m->guid); - free(m); - } + debug(D_REGISTRY, "Registry: freeing machine '%s'", m->guid); + freez(m); + } - // and free the memory of remaining dictionary structures + // and free the memory of remaining dictionary structures - debug(D_REGISTRY, "Registry: destroying persons dictionary"); - dictionary_destroy(registry.persons); + debug(D_REGISTRY, "Registry: destroying persons dictionary"); + dictionary_destroy(registry.persons); - debug(D_REGISTRY, "Registry: destroying machines dictionary"); - dictionary_destroy(registry.machines); + debug(D_REGISTRY, "Registry: destroying machines dictionary"); + dictionary_destroy(registry.machines); - debug(D_REGISTRY, "Registry: destroying urls dictionary"); - dictionary_destroy(registry.urls); + debug(D_REGISTRY, "Registry: destroying urls dictionary"); + dictionary_destroy(registry.urls); } // ---------------------------------------------------------------------------- // STATISTICS void registry_statistics(void) { - if(!registry.enabled) return; - - static RRDSET *sts = NULL, *stc = NULL, *stm = NULL; - - if(!sts) sts = rrdset_find("netdata.registry_sessions"); - if(!sts) { - sts = rrdset_create("netdata", "registry_sessions", NULL, "registry", NULL, "NetData Registry Sessions", "session", 131000, rrd_update_every, RRDSET_TYPE_LINE); - - rrddim_add(sts, "sessions", NULL, 1, 1, RRDDIM_ABSOLUTE); - } - else rrdset_next(sts); - - rrddim_set(sts, "sessions", registry.usages_count); - rrdset_done(sts); - - // ------------------------------------------------------------------------ - - if(!stc) stc = rrdset_find("netdata.registry_entries"); - if(!stc) { - stc = rrdset_create("netdata", "registry_entries", NULL, "registry", NULL, "NetData Registry Entries", "entries", 131100, rrd_update_every, RRDSET_TYPE_LINE); - - rrddim_add(stc, "persons", NULL, 1, 1, RRDDIM_ABSOLUTE); - rrddim_add(stc, "machines", NULL, 1, 1, RRDDIM_ABSOLUTE); - rrddim_add(stc, "urls", NULL, 1, 1, RRDDIM_ABSOLUTE); - rrddim_add(stc, "persons_urls", NULL, 1, 1, RRDDIM_ABSOLUTE); - rrddim_add(stc, "machines_urls", NULL, 1, 1, RRDDIM_ABSOLUTE); - } - else rrdset_next(stc); - - rrddim_set(stc, "persons", registry.persons_count); - rrddim_set(stc, "machines", registry.machines_count); - rrddim_set(stc, "urls", registry.urls_count); - rrddim_set(stc, "persons_urls", registry.persons_urls_count); - rrddim_set(stc, "machines_urls", registry.machines_urls_count); - rrdset_done(stc); - - // ------------------------------------------------------------------------ - - if(!stm) stm = rrdset_find("netdata.registry_mem"); - if(!stm) { - stm = rrdset_create("netdata", "registry_mem", NULL, "registry", NULL, "NetData Registry Memory", "KB", 131300, rrd_update_every, RRDSET_TYPE_STACKED); - - rrddim_add(stm, "persons", NULL, 1, 1024, RRDDIM_ABSOLUTE); - rrddim_add(stm, "machines", NULL, 1, 1024, RRDDIM_ABSOLUTE); - rrddim_add(stm, "urls", NULL, 1, 1024, RRDDIM_ABSOLUTE); - rrddim_add(stm, "persons_urls", NULL, 1, 1024, RRDDIM_ABSOLUTE); - rrddim_add(stm, "machines_urls", NULL, 1, 1024, RRDDIM_ABSOLUTE); - } - else rrdset_next(stm); - - rrddim_set(stm, "persons", registry.persons_memory + registry.persons_count * sizeof(NAME_VALUE) + sizeof(DICTIONARY)); - rrddim_set(stm, "machines", registry.machines_memory + registry.machines_count * sizeof(NAME_VALUE) + sizeof(DICTIONARY)); - rrddim_set(stm, "urls", registry.urls_memory + registry.urls_count * sizeof(NAME_VALUE) + sizeof(DICTIONARY)); - rrddim_set(stm, "persons_urls", registry.persons_urls_memory + registry.persons_count * sizeof(DICTIONARY) + registry.persons_urls_count * sizeof(NAME_VALUE)); - rrddim_set(stm, "machines_urls", registry.machines_urls_memory + registry.machines_count * sizeof(DICTIONARY) + registry.machines_urls_count * sizeof(NAME_VALUE)); - rrdset_done(stm); + if(!registry.enabled) return; + + static RRDSET *sts = NULL, *stc = NULL, *stm = NULL; + + if(!sts) sts = rrdset_find("netdata.registry_sessions"); + if(!sts) { + sts = rrdset_create("netdata", "registry_sessions", NULL, "registry", NULL, "NetData Registry Sessions", "session", 131000, rrd_update_every, RRDSET_TYPE_LINE); + + rrddim_add(sts, "sessions", NULL, 1, 1, RRDDIM_ABSOLUTE); + } + else rrdset_next(sts); + + rrddim_set(sts, "sessions", registry.usages_count); + rrdset_done(sts); + + // ------------------------------------------------------------------------ + + if(!stc) stc = rrdset_find("netdata.registry_entries"); + if(!stc) { + stc = rrdset_create("netdata", "registry_entries", NULL, "registry", NULL, "NetData Registry Entries", "entries", 131100, rrd_update_every, RRDSET_TYPE_LINE); + + rrddim_add(stc, "persons", NULL, 1, 1, RRDDIM_ABSOLUTE); + rrddim_add(stc, "machines", NULL, 1, 1, RRDDIM_ABSOLUTE); + rrddim_add(stc, "urls", NULL, 1, 1, RRDDIM_ABSOLUTE); + rrddim_add(stc, "persons_urls", NULL, 1, 1, RRDDIM_ABSOLUTE); + rrddim_add(stc, "machines_urls", NULL, 1, 1, RRDDIM_ABSOLUTE); + } + else rrdset_next(stc); + + rrddim_set(stc, "persons", registry.persons_count); + rrddim_set(stc, "machines", registry.machines_count); + rrddim_set(stc, "urls", registry.urls_count); + rrddim_set(stc, "persons_urls", registry.persons_urls_count); + rrddim_set(stc, "machines_urls", registry.machines_urls_count); + rrdset_done(stc); + + // ------------------------------------------------------------------------ + + if(!stm) stm = rrdset_find("netdata.registry_mem"); + if(!stm) { + stm = rrdset_create("netdata", "registry_mem", NULL, "registry", NULL, "NetData Registry Memory", "KB", 131300, rrd_update_every, RRDSET_TYPE_STACKED); + + rrddim_add(stm, "persons", NULL, 1, 1024, RRDDIM_ABSOLUTE); + rrddim_add(stm, "machines", NULL, 1, 1024, RRDDIM_ABSOLUTE); + rrddim_add(stm, "urls", NULL, 1, 1024, RRDDIM_ABSOLUTE); + rrddim_add(stm, "persons_urls", NULL, 1, 1024, RRDDIM_ABSOLUTE); + rrddim_add(stm, "machines_urls", NULL, 1, 1024, RRDDIM_ABSOLUTE); + } + else rrdset_next(stm); + + rrddim_set(stm, "persons", registry.persons_memory + registry.persons_count * sizeof(NAME_VALUE) + sizeof(DICTIONARY)); + rrddim_set(stm, "machines", registry.machines_memory + registry.machines_count * sizeof(NAME_VALUE) + sizeof(DICTIONARY)); + rrddim_set(stm, "urls", registry.urls_memory + registry.urls_count * sizeof(NAME_VALUE) + sizeof(DICTIONARY)); + rrddim_set(stm, "persons_urls", registry.persons_urls_memory + registry.persons_count * sizeof(DICTIONARY) + registry.persons_urls_count * sizeof(NAME_VALUE)); + rrddim_set(stm, "machines_urls", registry.machines_urls_memory + registry.machines_count * sizeof(DICTIONARY) + registry.machines_urls_count * sizeof(NAME_VALUE)); + rrdset_done(stm); } diff --git a/src/registry.h b/src/registry.h index d95383b5d..c2b57a23d 100644 --- a/src/registry.h +++ b/src/registry.h @@ -1,10 +1,12 @@ -#include "web_client.h" - #ifndef NETDATA_REGISTRY_H #define NETDATA_REGISTRY_H 1 #define NETDATA_REGISTRY_COOKIE_NAME "netdata_registry_id" +extern void registry_set_cookie(struct web_client *w, const char *guid); +extern const char *registry_to_announce(void); +extern int registry_verify_cookies_redirects(void); + extern int registry_request_access_json(struct web_client *w, char *person_guid, char *machine_guid, char *url, char *name, time_t when); extern int registry_request_delete_json(struct web_client *w, char *person_guid, char *machine_guid, char *url, char *delete_url, time_t when); extern int registry_request_search_json(struct web_client *w, char *person_guid, char *machine_guid, char *url, char *request_machine, time_t when); @@ -1,24 +1,4 @@ -#ifdef HAVE_CONFIG_H -#include <config.h> -#endif -#include <stddef.h> -#include <string.h> -#include <unistd.h> -#include <time.h> -#include <sys/time.h> -#include <sys/mman.h> -#include <pthread.h> -#include <errno.h> -#include <ctype.h> -#include <sys/stat.h> -#include <sys/types.h> -#include <stdlib.h> - #include "common.h" -#include "log.h" -#include "appconfig.h" - -#include "rrd.h" #define RRD_DEFAULT_GAP_INTERPOLATIONS 1 @@ -32,36 +12,143 @@ int rrd_delete_unupdated_dimensions = 0; int rrd_update_every = UPDATE_EVERY; int rrd_default_history_entries = RRD_DEFAULT_HISTORY_ENTRIES; +int rrd_memory_mode = RRD_MEMORY_MODE_SAVE; -RRDSET *rrdset_root = NULL; -pthread_rwlock_t rrdset_root_rwlock = PTHREAD_RWLOCK_INITIALIZER; +static int rrdset_compare(void* a, void* b); +static int rrdset_compare_name(void* a, void* b); +static int rrdfamily_compare(void *a, void *b); -int rrd_memory_mode = RRD_MEMORY_MODE_SAVE; +// ---------------------------------------------------------------------------- +// RRDHOST + +RRDHOST localhost = { + .hostname = "localhost", + .rrdset_root = NULL, + .rrdset_root_rwlock = PTHREAD_RWLOCK_INITIALIZER, + .rrdset_root_index = { + { NULL, rrdset_compare }, + AVL_LOCK_INITIALIZER + }, + .rrdset_root_index_name = { + { NULL, rrdset_compare_name }, + AVL_LOCK_INITIALIZER + }, + .rrdfamily_root_index = { + { NULL, rrdfamily_compare }, + AVL_LOCK_INITIALIZER + }, + .variables_root_index = { + { NULL, rrdvar_compare }, + AVL_LOCK_INITIALIZER + }, + .health_log = { + .nextid = 1, + .count = 0, + .max = 1000, + .alarms = NULL, + .alarm_log_rwlock = PTHREAD_RWLOCK_INITIALIZER + } +}; + +void rrdhost_rwlock(RRDHOST *host) { + pthread_rwlock_wrlock(&host->rrdset_root_rwlock); +} + +void rrdhost_rdlock(RRDHOST *host) { + pthread_rwlock_rdlock(&host->rrdset_root_rwlock); +} + +void rrdhost_unlock(RRDHOST *host) { + pthread_rwlock_unlock(&host->rrdset_root_rwlock); +} + +void rrdhost_check_rdlock_int(RRDHOST *host, const char *file, const char *function, const unsigned long line) { + int ret = pthread_rwlock_trywrlock(&host->rrdset_root_rwlock); + + if(ret == 0) + fatal("RRDHOST '%s' should be read-locked, but it is not, at function %s() at line %lu of file '%s'", host->hostname, function, line, file); +} + +void rrdhost_check_wrlock_int(RRDHOST *host, const char *file, const char *function, const unsigned long line) { + int ret = pthread_rwlock_tryrdlock(&host->rrdset_root_rwlock); + + if(ret == 0) + fatal("RRDHOST '%s' should be write-locked, but it is not, at function %s() at line %lu of file '%s'", host->hostname, function, line, file); +} + +// ---------------------------------------------------------------------------- +// RRDFAMILY index + +static int rrdfamily_compare(void *a, void *b) { + if(((RRDFAMILY *)a)->hash_family < ((RRDFAMILY *)b)->hash_family) return -1; + else if(((RRDFAMILY *)a)->hash_family > ((RRDFAMILY *)b)->hash_family) return 1; + else return strcmp(((RRDFAMILY *)a)->family, ((RRDFAMILY *)b)->family); +} + +#define rrdfamily_index_add(host, rc) (RRDFAMILY *)avl_insert_lock(&((host)->rrdfamily_root_index), (avl *)(rc)) +#define rrdfamily_index_del(host, rc) (RRDFAMILY *)avl_remove_lock(&((host)->rrdfamily_root_index), (avl *)(rc)) + +static RRDFAMILY *rrdfamily_index_find(RRDHOST *host, const char *id, uint32_t hash) { + RRDFAMILY tmp; + tmp.family = id; + tmp.hash_family = (hash)?hash:simple_hash(tmp.family); + + return (RRDFAMILY *)avl_search_lock(&(host->rrdfamily_root_index), (avl *) &tmp); +} + +RRDFAMILY *rrdfamily_create(const char *id) { + RRDFAMILY *rc = rrdfamily_index_find(&localhost, id, 0); + if(!rc) { + rc = callocz(1, sizeof(RRDFAMILY)); + + rc->family = strdupz(id); + rc->hash_family = simple_hash(rc->family); + + // initialize the variables index + avl_init_lock(&rc->variables_root_index, rrdvar_compare); + RRDFAMILY *ret = rrdfamily_index_add(&localhost, rc); + if(ret != rc) + fatal("INTERNAL ERROR: Expected to INSERT RRDFAMILY '%s' into index, but inserted '%s'.", rc->family, (ret)?ret->family:"NONE"); + } + + rc->use_count++; + return rc; +} + +void rrdfamily_free(RRDFAMILY *rc) { + rc->use_count--; + if(!rc->use_count) { + RRDFAMILY *ret = rrdfamily_index_del(&localhost, rc); + if(ret != rc) + fatal("INTERNAL ERROR: Expected to DELETE RRDFAMILY '%s' from index, but deleted '%s'.", rc->family, (ret)?ret->family:"NONE"); + + if(rc->variables_root_index.avl_tree.root != NULL) + fatal("INTERNAL ERROR: Variables index of RRDFAMILY '%s' that is freed, is not empty.", rc->family); + + freez((void *)rc->family); + freez(rc); + } +} // ---------------------------------------------------------------------------- // RRDSET index static int rrdset_compare(void* a, void* b) { - if(((RRDSET *)a)->hash < ((RRDSET *)b)->hash) return -1; - else if(((RRDSET *)a)->hash > ((RRDSET *)b)->hash) return 1; - else return strcmp(((RRDSET *)a)->id, ((RRDSET *)b)->id); + if(((RRDSET *)a)->hash < ((RRDSET *)b)->hash) return -1; + else if(((RRDSET *)a)->hash > ((RRDSET *)b)->hash) return 1; + else return strcmp(((RRDSET *)a)->id, ((RRDSET *)b)->id); } -avl_tree_lock rrdset_root_index = { - { NULL, rrdset_compare }, - AVL_LOCK_INITIALIZER -}; - -#define rrdset_index_add(st) avl_insert_lock(&rrdset_root_index, (avl *)(st)) -#define rrdset_index_del(st) avl_remove_lock(&rrdset_root_index, (avl *)(st)) +#define rrdset_index_add(host, st) (RRDSET *)avl_insert_lock(&((host)->rrdset_root_index), (avl *)(st)) +#define rrdset_index_del(host, st) (RRDSET *)avl_remove_lock(&((host)->rrdset_root_index), (avl *)(st)) -static RRDSET *rrdset_index_find(const char *id, uint32_t hash) { - RRDSET tmp; - strncpyz(tmp.id, id, RRD_ID_LENGTH_MAX); - tmp.hash = (hash)?hash:simple_hash(tmp.id); +static RRDSET *rrdset_index_find(RRDHOST *host, const char *id, uint32_t hash) { + RRDSET tmp; + strncpyz(tmp.id, id, RRD_ID_LENGTH_MAX); + tmp.hash = (hash)?hash:simple_hash(tmp.id); - return (RRDSET *)avl_search_lock(&(rrdset_root_index), (avl *) &tmp); + return (RRDSET *)avl_search_lock(&(host->rrdset_root_index), (avl *) &tmp); } // ---------------------------------------------------------------------------- @@ -70,46 +157,50 @@ static RRDSET *rrdset_index_find(const char *id, uint32_t hash) { #define rrdset_from_avlname(avlname_ptr) ((RRDSET *)((avlname_ptr) - offsetof(RRDSET, avlname))) static int rrdset_compare_name(void* a, void* b) { - RRDSET *A = rrdset_from_avlname(a); - RRDSET *B = rrdset_from_avlname(b); + RRDSET *A = rrdset_from_avlname(a); + RRDSET *B = rrdset_from_avlname(b); - // fprintf(stderr, "COMPARING: %s with %s\n", A->name, B->name); + // fprintf(stderr, "COMPARING: %s with %s\n", A->name, B->name); - if(A->hash_name < B->hash_name) return -1; - else if(A->hash_name > B->hash_name) return 1; - else return strcmp(A->name, B->name); + if(A->hash_name < B->hash_name) return -1; + else if(A->hash_name > B->hash_name) return 1; + else return strcmp(A->name, B->name); } -avl_tree_lock rrdset_root_index_name = { - { NULL, rrdset_compare_name }, - AVL_LOCK_INITIALIZER -}; +RRDSET *rrdset_index_add_name(RRDHOST *host, RRDSET *st) { + void *result; + // fprintf(stderr, "ADDING: %s (name: %s)\n", st->id, st->name); + result = avl_insert_lock(&host->rrdset_root_index_name, (avl *) (&st->avlname)); + if(result) return rrdset_from_avlname(result); + return NULL; +} -int rrdset_index_add_name(RRDSET *st) { - // fprintf(stderr, "ADDING: %s (name: %s)\n", st->id, st->name); - return avl_insert_lock(&rrdset_root_index_name, (avl *) (&st->avlname)); +RRDSET *rrdset_index_del_name(RRDHOST *host, RRDSET *st) { + void *result; + // fprintf(stderr, "DELETING: %s (name: %s)\n", st->id, st->name); + result = (RRDSET *)avl_remove_lock(&((host)->rrdset_root_index_name), (avl *)(&st->avlname)); + if(result) return rrdset_from_avlname(result); + return NULL; } -#define rrdset_index_del_name(st) avl_remove_lock(&rrdset_root_index_name, (avl *)(&st->avlname)) - -static RRDSET *rrdset_index_find_name(const char *name, uint32_t hash) { - void *result = NULL; - RRDSET tmp; - tmp.name = name; - tmp.hash_name = (hash)?hash:simple_hash(tmp.name); - - // fprintf(stderr, "SEARCHING: %s\n", name); - result = avl_search_lock(&(rrdset_root_index_name), (avl *) (&(tmp.avlname))); - if(result) { - RRDSET *st = rrdset_from_avlname(result); - if(strcmp(st->magic, RRDSET_MAGIC)) - error("Search for RRDSET %s returned an invalid RRDSET %s (name %s)", name, st->id, st->name); - - // fprintf(stderr, "FOUND: %s\n", name); - return rrdset_from_avlname(result); - } - // fprintf(stderr, "NOT FOUND: %s\n", name); - return NULL; +static RRDSET *rrdset_index_find_name(RRDHOST *host, const char *name, uint32_t hash) { + void *result = NULL; + RRDSET tmp; + tmp.name = name; + tmp.hash_name = (hash)?hash:simple_hash(tmp.name); + + // fprintf(stderr, "SEARCHING: %s\n", name); + result = avl_search_lock(&host->rrdset_root_index_name, (avl *) (&(tmp.avlname))); + if(result) { + RRDSET *st = rrdset_from_avlname(result); + if(strcmp(st->magic, RRDSET_MAGIC)) + error("Search for RRDSET %s returned an invalid RRDSET %s (name %s)", name, st->id, st->name); + + // fprintf(stderr, "FOUND: %s\n", name); + return rrdset_from_avlname(result); + } + // fprintf(stderr, "NOT FOUND: %s\n", name); + return NULL; } @@ -117,20 +208,20 @@ static RRDSET *rrdset_index_find_name(const char *name, uint32_t hash) { // RRDDIM index static int rrddim_compare(void* a, void* b) { - if(((RRDDIM *)a)->hash < ((RRDDIM *)b)->hash) return -1; - else if(((RRDDIM *)a)->hash > ((RRDDIM *)b)->hash) return 1; - else return strcmp(((RRDDIM *)a)->id, ((RRDDIM *)b)->id); + if(((RRDDIM *)a)->hash < ((RRDDIM *)b)->hash) return -1; + else if(((RRDDIM *)a)->hash > ((RRDDIM *)b)->hash) return 1; + else return strcmp(((RRDDIM *)a)->id, ((RRDDIM *)b)->id); } #define rrddim_index_add(st, rd) avl_insert_lock(&((st)->dimensions_index), (avl *)(rd)) #define rrddim_index_del(st,rd ) avl_remove_lock(&((st)->dimensions_index), (avl *)(rd)) static RRDDIM *rrddim_index_find(RRDSET *st, const char *id, uint32_t hash) { - RRDDIM tmp; - strncpyz(tmp.id, id, RRD_ID_LENGTH_MAX); - tmp.hash = (hash)?hash:simple_hash(tmp.id); + RRDDIM tmp; + strncpyz(tmp.id, id, RRD_ID_LENGTH_MAX); + tmp.hash = (hash)?hash:simple_hash(tmp.id); - return (RRDDIM *)avl_search_lock(&(st->dimensions_index), (avl *) &tmp); + return (RRDDIM *)avl_search_lock(&(st->dimensions_index), (avl *) &tmp); } // ---------------------------------------------------------------------------- @@ -138,29 +229,29 @@ static RRDDIM *rrddim_index_find(RRDSET *st, const char *id, uint32_t hash) { int rrdset_type_id(const char *name) { - if(unlikely(strcmp(name, RRDSET_TYPE_AREA_NAME) == 0)) return RRDSET_TYPE_AREA; - else if(unlikely(strcmp(name, RRDSET_TYPE_STACKED_NAME) == 0)) return RRDSET_TYPE_STACKED; - else if(unlikely(strcmp(name, RRDSET_TYPE_LINE_NAME) == 0)) return RRDSET_TYPE_LINE; - return RRDSET_TYPE_LINE; + if(unlikely(strcmp(name, RRDSET_TYPE_AREA_NAME) == 0)) return RRDSET_TYPE_AREA; + else if(unlikely(strcmp(name, RRDSET_TYPE_STACKED_NAME) == 0)) return RRDSET_TYPE_STACKED; + else if(unlikely(strcmp(name, RRDSET_TYPE_LINE_NAME) == 0)) return RRDSET_TYPE_LINE; + return RRDSET_TYPE_LINE; } const char *rrdset_type_name(int chart_type) { - static char line[] = RRDSET_TYPE_LINE_NAME; - static char area[] = RRDSET_TYPE_AREA_NAME; - static char stacked[] = RRDSET_TYPE_STACKED_NAME; + static char line[] = RRDSET_TYPE_LINE_NAME; + static char area[] = RRDSET_TYPE_AREA_NAME; + static char stacked[] = RRDSET_TYPE_STACKED_NAME; - switch(chart_type) { - case RRDSET_TYPE_LINE: - return line; + switch(chart_type) { + case RRDSET_TYPE_LINE: + return line; - case RRDSET_TYPE_AREA: - return area; + case RRDSET_TYPE_AREA: + return area; - case RRDSET_TYPE_STACKED: - return stacked; - } - return line; + case RRDSET_TYPE_STACKED: + return stacked; + } + return line; } // ---------------------------------------------------------------------------- @@ -168,33 +259,33 @@ const char *rrdset_type_name(int chart_type) const char *rrd_memory_mode_name(int id) { - static const char ram[] = RRD_MEMORY_MODE_RAM_NAME; - static const char map[] = RRD_MEMORY_MODE_MAP_NAME; - static const char save[] = RRD_MEMORY_MODE_SAVE_NAME; + static const char ram[] = RRD_MEMORY_MODE_RAM_NAME; + static const char map[] = RRD_MEMORY_MODE_MAP_NAME; + static const char save[] = RRD_MEMORY_MODE_SAVE_NAME; - switch(id) { - case RRD_MEMORY_MODE_RAM: - return ram; + switch(id) { + case RRD_MEMORY_MODE_RAM: + return ram; - case RRD_MEMORY_MODE_MAP: - return map; + case RRD_MEMORY_MODE_MAP: + return map; - case RRD_MEMORY_MODE_SAVE: - default: - return save; - } + case RRD_MEMORY_MODE_SAVE: + default: + return save; + } - return save; + return save; } int rrd_memory_mode_id(const char *name) { - if(unlikely(!strcmp(name, RRD_MEMORY_MODE_RAM_NAME))) - return RRD_MEMORY_MODE_RAM; - else if(unlikely(!strcmp(name, RRD_MEMORY_MODE_MAP_NAME))) - return RRD_MEMORY_MODE_MAP; + if(unlikely(!strcmp(name, RRD_MEMORY_MODE_RAM_NAME))) + return RRD_MEMORY_MODE_RAM; + else if(unlikely(!strcmp(name, RRD_MEMORY_MODE_MAP_NAME))) + return RRD_MEMORY_MODE_MAP; - return RRD_MEMORY_MODE_SAVE; + return RRD_MEMORY_MODE_SAVE; } // ---------------------------------------------------------------------------- @@ -202,67 +293,73 @@ int rrd_memory_mode_id(const char *name) int rrddim_algorithm_id(const char *name) { - if(strcmp(name, RRDDIM_INCREMENTAL_NAME) == 0) return RRDDIM_INCREMENTAL; - if(strcmp(name, RRDDIM_ABSOLUTE_NAME) == 0) return RRDDIM_ABSOLUTE; - if(strcmp(name, RRDDIM_PCENT_OVER_ROW_TOTAL_NAME) == 0) return RRDDIM_PCENT_OVER_ROW_TOTAL; - if(strcmp(name, RRDDIM_PCENT_OVER_DIFF_TOTAL_NAME) == 0) return RRDDIM_PCENT_OVER_DIFF_TOTAL; - return RRDDIM_ABSOLUTE; + if(strcmp(name, RRDDIM_INCREMENTAL_NAME) == 0) return RRDDIM_INCREMENTAL; + if(strcmp(name, RRDDIM_ABSOLUTE_NAME) == 0) return RRDDIM_ABSOLUTE; + if(strcmp(name, RRDDIM_PCENT_OVER_ROW_TOTAL_NAME) == 0) return RRDDIM_PCENT_OVER_ROW_TOTAL; + if(strcmp(name, RRDDIM_PCENT_OVER_DIFF_TOTAL_NAME) == 0) return RRDDIM_PCENT_OVER_DIFF_TOTAL; + return RRDDIM_ABSOLUTE; } const char *rrddim_algorithm_name(int chart_type) { - static char absolute[] = RRDDIM_ABSOLUTE_NAME; - static char incremental[] = RRDDIM_INCREMENTAL_NAME; - static char percentage_of_absolute_row[] = RRDDIM_PCENT_OVER_ROW_TOTAL_NAME; - static char percentage_of_incremental_row[] = RRDDIM_PCENT_OVER_DIFF_TOTAL_NAME; + static char absolute[] = RRDDIM_ABSOLUTE_NAME; + static char incremental[] = RRDDIM_INCREMENTAL_NAME; + static char percentage_of_absolute_row[] = RRDDIM_PCENT_OVER_ROW_TOTAL_NAME; + static char percentage_of_incremental_row[] = RRDDIM_PCENT_OVER_DIFF_TOTAL_NAME; - switch(chart_type) { - case RRDDIM_ABSOLUTE: - return absolute; + switch(chart_type) { + case RRDDIM_ABSOLUTE: + return absolute; - case RRDDIM_INCREMENTAL: - return incremental; + case RRDDIM_INCREMENTAL: + return incremental; - case RRDDIM_PCENT_OVER_ROW_TOTAL: - return percentage_of_absolute_row; + case RRDDIM_PCENT_OVER_ROW_TOTAL: + return percentage_of_absolute_row; - case RRDDIM_PCENT_OVER_DIFF_TOTAL: - return percentage_of_incremental_row; - } - return absolute; + case RRDDIM_PCENT_OVER_DIFF_TOTAL: + return percentage_of_incremental_row; + } + return absolute; } // ---------------------------------------------------------------------------- // chart names -char *rrdset_strncpy_name(char *to, const char *from, int length) +char *rrdset_strncpyz_name(char *to, const char *from, size_t length) { - int i; - for(i = 0; i < length && from[i] ;i++) { - if(from[i] == '.' || isalpha(from[i]) || isdigit(from[i])) to[i] = from[i]; - else to[i] = '_'; - } - if(i < length) to[i] = '\0'; - to[length - 1] = '\0'; - - return to; + char c, *p = to; + + while (length-- && (c = *from++)) { + if(c != '.' && !isalnum(c)) + c = '_'; + + *p++ = c; + } + + *p = '\0'; + + return to; } void rrdset_set_name(RRDSET *st, const char *name) { - debug(D_RRD_CALLS, "rrdset_set_name() old: %s, new: %s", st->name, name); + debug(D_RRD_CALLS, "rrdset_set_name() old: %s, new: %s", st->name, name); - if(st->name) rrdset_index_del_name(st); + if(st->name) { + rrdset_index_del_name(&localhost, st); + rrdsetvar_rename_all(st); + } - char b[CONFIG_MAX_VALUE + 1]; - char n[RRD_ID_LENGTH_MAX + 1]; + char b[CONFIG_MAX_VALUE + 1]; + char n[RRD_ID_LENGTH_MAX + 1]; - snprintfz(n, RRD_ID_LENGTH_MAX, "%s.%s", st->type, name); - rrdset_strncpy_name(b, n, CONFIG_MAX_VALUE); - st->name = config_get(st->id, "name", b); - st->hash_name = simple_hash(st->name); + snprintfz(n, RRD_ID_LENGTH_MAX, "%s.%s", st->type, name); + rrdset_strncpyz_name(b, n, CONFIG_MAX_VALUE); + st->name = config_get(st->id, "name", b); + st->hash_name = simple_hash(st->name); - rrdset_index_add_name(st); + rrdset_index_add_name(&localhost, st); } // ---------------------------------------------------------------------------- @@ -270,25 +367,30 @@ void rrdset_set_name(RRDSET *st, const char *name) char *rrdset_cache_dir(const char *id) { - char *ret = NULL; - - static char *cache_dir = NULL; - if(!cache_dir) cache_dir = config_get("global", "cache directory", CACHE_DIR); - - char b[FILENAME_MAX + 1]; - char n[FILENAME_MAX + 1]; - rrdset_strncpy_name(b, id, FILENAME_MAX); - - snprintfz(n, FILENAME_MAX, "%s/%s", cache_dir, b); - ret = config_get(id, "cache directory", n); - - if(rrd_memory_mode == RRD_MEMORY_MODE_MAP || rrd_memory_mode == RRD_MEMORY_MODE_SAVE) { - int r = mkdir(ret, 0775); - if(r != 0 && errno != EEXIST) - error("Cannot create directory '%s'", ret); - } - - return ret; + char *ret = NULL; + + static char *cache_dir = NULL; + if(!cache_dir) { + cache_dir = config_get("global", "cache directory", CACHE_DIR); + int r = mkdir(cache_dir, 0755); + if(r != 0 && errno != EEXIST) + error("Cannot create directory '%s'", cache_dir); + } + + char b[FILENAME_MAX + 1]; + char n[FILENAME_MAX + 1]; + rrdset_strncpyz_name(b, id, FILENAME_MAX); + + snprintfz(n, FILENAME_MAX, "%s/%s", cache_dir, b); + ret = config_get(id, "cache directory", n); + + if(rrd_memory_mode == RRD_MEMORY_MODE_MAP || rrd_memory_mode == RRD_MEMORY_MODE_SAVE) { + int r = mkdir(ret, 0775); + if(r != 0 && errno != EEXIST) + error("Cannot create directory '%s'", ret); + } + + return ret; } // ---------------------------------------------------------------------------- @@ -296,1025 +398,1110 @@ char *rrdset_cache_dir(const char *id) void rrdset_reset(RRDSET *st) { - debug(D_RRD_CALLS, "rrdset_reset() %s", st->name); - - st->last_collected_time.tv_sec = 0; - st->last_collected_time.tv_usec = 0; - st->last_updated.tv_sec = 0; - st->last_updated.tv_usec = 0; - st->current_entry = 0; - st->counter = 0; - st->counter_done = 0; - - RRDDIM *rd; - for(rd = st->dimensions; rd ; rd = rd->next) { - rd->last_collected_time.tv_sec = 0; - rd->last_collected_time.tv_usec = 0; - rd->counter = 0; - bzero(rd->values, rd->entries * sizeof(storage_number)); - } + debug(D_RRD_CALLS, "rrdset_reset() %s", st->name); + + st->last_collected_time.tv_sec = 0; + st->last_collected_time.tv_usec = 0; + st->last_updated.tv_sec = 0; + st->last_updated.tv_usec = 0; + st->current_entry = 0; + st->counter = 0; + st->counter_done = 0; + + RRDDIM *rd; + for(rd = st->dimensions; rd ; rd = rd->next) { + rd->last_collected_time.tv_sec = 0; + rd->last_collected_time.tv_usec = 0; + rd->counter = 0; + bzero(rd->values, rd->entries * sizeof(storage_number)); + } } RRDSET *rrdset_create(const char *type, const char *id, const char *name, const char *family, const char *context, const char *title, const char *units, long priority, int update_every, int chart_type) { - if(!type || !type[0]) { - fatal("Cannot create rrd stats without a type."); - return NULL; - } - - if(!id || !id[0]) { - fatal("Cannot create rrd stats without an id."); - return NULL; - } - - char fullid[RRD_ID_LENGTH_MAX + 1]; - char fullfilename[FILENAME_MAX + 1]; - RRDSET *st = NULL; - - snprintfz(fullid, RRD_ID_LENGTH_MAX, "%s.%s", type, id); - - st = rrdset_find(fullid); - if(st) { - error("Cannot create rrd stats for '%s', it already exists.", fullid); - return st; - } - - long entries = config_get_number(fullid, "history", rrd_default_history_entries); - if(entries < 5) entries = config_set_number(fullid, "history", 5); - if(entries > RRD_HISTORY_ENTRIES_MAX) entries = config_set_number(fullid, "history", RRD_HISTORY_ENTRIES_MAX); - - int enabled = config_get_boolean(fullid, "enabled", 1); - if(!enabled) entries = 5; - - unsigned long size = sizeof(RRDSET); - char *cache_dir = rrdset_cache_dir(fullid); - - debug(D_RRD_CALLS, "Creating RRD_STATS for '%s.%s'.", type, id); - - snprintfz(fullfilename, FILENAME_MAX, "%s/main.db", cache_dir); - if(rrd_memory_mode != RRD_MEMORY_MODE_RAM) st = (RRDSET *)mymmap(fullfilename, size, ((rrd_memory_mode == RRD_MEMORY_MODE_MAP)?MAP_SHARED:MAP_PRIVATE), 0); - if(st) { - if(strcmp(st->magic, RRDSET_MAGIC) != 0) { - errno = 0; - info("Initializing file %s.", fullfilename); - bzero(st, size); - } - else if(strcmp(st->id, fullid) != 0) { - errno = 0; - error("File %s contents are not for chart %s. Clearing it.", fullfilename, fullid); - // munmap(st, size); - // st = NULL; - bzero(st, size); - } - else if(st->memsize != size || st->entries != entries) { - errno = 0; - error("File %s does not have the desired size. Clearing it.", fullfilename); - bzero(st, size); - } - else if(st->update_every != update_every) { - errno = 0; - error("File %s does not have the desired update frequency. Clearing it.", fullfilename); - bzero(st, size); - } - else if((time(NULL) - st->last_updated.tv_sec) > update_every * entries) { - errno = 0; - error("File %s is too old. Clearing it.", fullfilename); - bzero(st, size); - } - } - - if(st) { - st->name = NULL; - st->type = NULL; - st->family = NULL; - st->context = NULL; - st->title = NULL; - st->units = NULL; - st->dimensions = NULL; - st->next = NULL; - st->mapped = rrd_memory_mode; - } - else { - st = calloc(1, size); - if(!st) { - fatal("Cannot allocate memory for RRD_STATS %s.%s", type, id); - return NULL; - } - st->mapped = RRD_MEMORY_MODE_RAM; - } - st->memsize = size; - st->entries = entries; - st->update_every = update_every; - - strcpy(st->cache_filename, fullfilename); - strcpy(st->magic, RRDSET_MAGIC); - - strcpy(st->id, fullid); - st->hash = simple_hash(st->id); - - st->cache_dir = cache_dir; - - st->chart_type = rrdset_type_id(config_get(st->id, "chart type", rrdset_type_name(chart_type))); - st->type = config_get(st->id, "type", type); - st->family = config_get(st->id, "family", family?family:st->type); - st->context = config_get(st->id, "context", context?context:st->id); - st->units = config_get(st->id, "units", units?units:""); - - st->priority = config_get_number(st->id, "priority", priority); - st->enabled = enabled; - - st->isdetail = 0; - st->debug = 0; - - st->last_collected_time.tv_sec = 0; - st->last_collected_time.tv_usec = 0; - st->counter_done = 0; - - st->gap_when_lost_iterations_above = (int) ( - config_get_number(st->id, "gap when lost iterations above", RRD_DEFAULT_GAP_INTERPOLATIONS) + 2); - - avl_init_lock(&st->dimensions_index, rrddim_compare); - - pthread_rwlock_init(&st->rwlock, NULL); - pthread_rwlock_wrlock(&rrdset_root_rwlock); - - if(name && *name) rrdset_set_name(st, name); - else rrdset_set_name(st, id); - - { - char varvalue[CONFIG_MAX_VALUE + 1]; - snprintfz(varvalue, CONFIG_MAX_VALUE, "%s (%s)", title?title:"", st->name); - st->title = config_get(st->id, "title", varvalue); - } - - st->next = rrdset_root; - rrdset_root = st; - - rrdset_index_add(st); - - pthread_rwlock_unlock(&rrdset_root_rwlock); - - return(st); + if(!type || !type[0]) { + fatal("Cannot create rrd stats without a type."); + return NULL; + } + + if(!id || !id[0]) { + fatal("Cannot create rrd stats without an id."); + return NULL; + } + + char fullid[RRD_ID_LENGTH_MAX + 1]; + char fullfilename[FILENAME_MAX + 1]; + RRDSET *st = NULL; + + snprintfz(fullid, RRD_ID_LENGTH_MAX, "%s.%s", type, id); + + st = rrdset_find(fullid); + if(st) { + error("Cannot create rrd stats for '%s', it already exists.", fullid); + return st; + } + + long entries = config_get_number(fullid, "history", rrd_default_history_entries); + if(entries < 5) entries = config_set_number(fullid, "history", 5); + if(entries > RRD_HISTORY_ENTRIES_MAX) entries = config_set_number(fullid, "history", RRD_HISTORY_ENTRIES_MAX); + + int enabled = config_get_boolean(fullid, "enabled", 1); + if(!enabled) entries = 5; + + unsigned long size = sizeof(RRDSET); + char *cache_dir = rrdset_cache_dir(fullid); + + debug(D_RRD_CALLS, "Creating RRD_STATS for '%s.%s'.", type, id); + + snprintfz(fullfilename, FILENAME_MAX, "%s/main.db", cache_dir); + if(rrd_memory_mode != RRD_MEMORY_MODE_RAM) st = (RRDSET *)mymmap(fullfilename, size, ((rrd_memory_mode == RRD_MEMORY_MODE_MAP)?MAP_SHARED:MAP_PRIVATE), 0); + if(st) { + if(strcmp(st->magic, RRDSET_MAGIC) != 0) { + errno = 0; + info("Initializing file %s.", fullfilename); + bzero(st, size); + } + else if(strcmp(st->id, fullid) != 0) { + errno = 0; + error("File %s contents are not for chart %s. Clearing it.", fullfilename, fullid); + // munmap(st, size); + // st = NULL; + bzero(st, size); + } + else if(st->memsize != size || st->entries != entries) { + errno = 0; + error("File %s does not have the desired size. Clearing it.", fullfilename); + bzero(st, size); + } + else if(st->update_every != update_every) { + errno = 0; + error("File %s does not have the desired update frequency. Clearing it.", fullfilename); + bzero(st, size); + } + else if((time(NULL) - st->last_updated.tv_sec) > update_every * entries) { + errno = 0; + error("File %s is too old. Clearing it.", fullfilename); + bzero(st, size); + } + } + + if(st) { + st->name = NULL; + st->type = NULL; + st->family = NULL; + st->context = NULL; + st->title = NULL; + st->units = NULL; + st->dimensions = NULL; + st->next = NULL; + st->mapped = rrd_memory_mode; + st->variables = NULL; + } + else { + st = callocz(1, size); + st->mapped = RRD_MEMORY_MODE_RAM; + } + + st->memsize = size; + st->entries = entries; + st->update_every = update_every; + + if(st->current_entry >= st->entries) st->current_entry = 0; + + strcpy(st->cache_filename, fullfilename); + strcpy(st->magic, RRDSET_MAGIC); + + strcpy(st->id, fullid); + st->hash = simple_hash(st->id); + + st->cache_dir = cache_dir; + + st->chart_type = rrdset_type_id(config_get(st->id, "chart type", rrdset_type_name(chart_type))); + st->type = config_get(st->id, "type", type); + st->family = config_get(st->id, "family", family?family:st->type); + st->units = config_get(st->id, "units", units?units:""); + + st->context = config_get(st->id, "context", context?context:st->id); + st->hash_context = simple_hash(st->context); + + st->priority = config_get_number(st->id, "priority", priority); + st->enabled = enabled; + + st->isdetail = 0; + st->debug = 0; + + // if(!strcmp(st->id, "disk_util.dm-0")) { + // st->debug = 1; + // error("enabled debugging for '%s'", st->id); + // } + // else error("not enabled debugging for '%s'", st->id); + + st->green = NAN; + st->red = NAN; + + st->last_collected_time.tv_sec = 0; + st->last_collected_time.tv_usec = 0; + st->counter_done = 0; + + st->gap_when_lost_iterations_above = (int) ( + config_get_number(st->id, "gap when lost iterations above", RRD_DEFAULT_GAP_INTERPOLATIONS) + 2); + + avl_init_lock(&st->dimensions_index, rrddim_compare); + avl_init_lock(&st->variables_root_index, rrdvar_compare); + + pthread_rwlock_init(&st->rwlock, NULL); + rrdhost_rwlock(&localhost); + + if(name && *name) rrdset_set_name(st, name); + else rrdset_set_name(st, id); + + { + char varvalue[CONFIG_MAX_VALUE + 1]; + snprintfz(varvalue, CONFIG_MAX_VALUE, "%s (%s)", title?title:"", st->name); + st->title = config_get(st->id, "title", varvalue); + } + + st->rrdfamily = rrdfamily_create(st->family); + st->rrdhost = &localhost; + + st->next = localhost.rrdset_root; + localhost.rrdset_root = st; + + if(health_enabled) { + rrdsetvar_create(st, "last_collected_t", RRDVAR_TYPE_TIME_T, &st->last_collected_time.tv_sec, 0); + rrdsetvar_create(st, "collected_total_raw", RRDVAR_TYPE_TOTAL, &st->last_collected_total, 0); + rrdsetvar_create(st, "green", RRDVAR_TYPE_CALCULATED, &st->green, 0); + rrdsetvar_create(st, "red", RRDVAR_TYPE_CALCULATED, &st->red, 0); + rrdsetvar_create(st, "update_every", RRDVAR_TYPE_INT, &st->update_every, 0); + } + + rrdset_index_add(&localhost, st); + + rrdsetcalc_link_matching(st); + rrdcalctemplate_link_matching(st); + + rrdhost_unlock(&localhost); + + return(st); } RRDDIM *rrddim_add(RRDSET *st, const char *id, const char *name, long multiplier, long divisor, int algorithm) { - char filename[FILENAME_MAX + 1]; - char fullfilename[FILENAME_MAX + 1]; - - char varname[CONFIG_MAX_NAME + 1]; - RRDDIM *rd = NULL; - unsigned long size = sizeof(RRDDIM) + (st->entries * sizeof(storage_number)); - - debug(D_RRD_CALLS, "Adding dimension '%s/%s'.", st->id, id); - - rrdset_strncpy_name(filename, id, FILENAME_MAX); - snprintfz(fullfilename, FILENAME_MAX, "%s/%s.db", st->cache_dir, filename); - if(rrd_memory_mode != RRD_MEMORY_MODE_RAM) rd = (RRDDIM *)mymmap(fullfilename, size, ((rrd_memory_mode == RRD_MEMORY_MODE_MAP)?MAP_SHARED:MAP_PRIVATE), 1); - if(rd) { - struct timeval now; - gettimeofday(&now, NULL); - - if(strcmp(rd->magic, RRDDIMENSION_MAGIC) != 0) { - errno = 0; - info("Initializing file %s.", fullfilename); - bzero(rd, size); - } - else if(rd->memsize != size) { - errno = 0; - error("File %s does not have the desired size. Clearing it.", fullfilename); - bzero(rd, size); - } - else if(rd->multiplier != multiplier) { - errno = 0; - error("File %s does not have the same multiplier. Clearing it.", fullfilename); - bzero(rd, size); - } - else if(rd->divisor != divisor) { - errno = 0; - error("File %s does not have the same divisor. Clearing it.", fullfilename); - bzero(rd, size); - } - else if(rd->algorithm != algorithm) { - errno = 0; - error("File %s does not have the same algorithm. Clearing it.", fullfilename); - bzero(rd, size); - } - else if(rd->update_every != st->update_every) { - errno = 0; - error("File %s does not have the same refresh frequency. Clearing it.", fullfilename); - bzero(rd, size); - } - else if(usecdiff(&now, &rd->last_collected_time) > (rd->entries * rd->update_every * 1000000ULL)) { - errno = 0; - error("File %s is too old. Clearing it.", fullfilename); - bzero(rd, size); - } - else if(strcmp(rd->id, id) != 0) { - errno = 0; - error("File %s contents are not for dimension %s. Clearing it.", fullfilename, id); - // munmap(rd, size); - // rd = NULL; - bzero(rd, size); - } - } - - if(rd) { - // we have a file mapped for rd - rd->mapped = rrd_memory_mode; - rd->flags = 0x00000000; - rd->next = NULL; - rd->name = NULL; - } - else { - // if we didn't manage to get a mmap'd dimension, just create one - - rd = calloc(1, size); - if(!rd) { - fatal("Cannot allocate RRD_DIMENSION %s/%s.", st->id, id); - return NULL; - } - - rd->mapped = RRD_MEMORY_MODE_RAM; - } - rd->memsize = size; - - strcpy(rd->magic, RRDDIMENSION_MAGIC); - strcpy(rd->cache_filename, fullfilename); - strncpyz(rd->id, id, RRD_ID_LENGTH_MAX); - rd->hash = simple_hash(rd->id); - - snprintfz(varname, CONFIG_MAX_NAME, "dim %s name", rd->id); - rd->name = config_get(st->id, varname, (name && *name)?name:rd->id); - - snprintfz(varname, CONFIG_MAX_NAME, "dim %s algorithm", rd->id); - rd->algorithm = rrddim_algorithm_id(config_get(st->id, varname, rrddim_algorithm_name(algorithm))); - - snprintfz(varname, CONFIG_MAX_NAME, "dim %s multiplier", rd->id); - rd->multiplier = config_get_number(st->id, varname, multiplier); - - snprintfz(varname, CONFIG_MAX_NAME, "dim %s divisor", rd->id); - rd->divisor = config_get_number(st->id, varname, divisor); - if(!rd->divisor) rd->divisor = 1; - - rd->entries = st->entries; - rd->update_every = st->update_every; - - // prevent incremental calculation spikes - rd->counter = 0; - - // append this dimension - pthread_rwlock_wrlock(&st->rwlock); - if(!st->dimensions) - st->dimensions = rd; - else { - RRDDIM *td = st->dimensions; - for(; td->next; td = td->next) ; - td->next = rd; - } - pthread_rwlock_unlock(&st->rwlock); - - rrddim_index_add(st, rd); - - return(rd); + char filename[FILENAME_MAX + 1]; + char fullfilename[FILENAME_MAX + 1]; + + char varname[CONFIG_MAX_NAME + 1]; + RRDDIM *rd = NULL; + unsigned long size = sizeof(RRDDIM) + (st->entries * sizeof(storage_number)); + + debug(D_RRD_CALLS, "Adding dimension '%s/%s'.", st->id, id); + + rrdset_strncpyz_name(filename, id, FILENAME_MAX); + snprintfz(fullfilename, FILENAME_MAX, "%s/%s.db", st->cache_dir, filename); + if(rrd_memory_mode != RRD_MEMORY_MODE_RAM) rd = (RRDDIM *)mymmap(fullfilename, size, ((rrd_memory_mode == RRD_MEMORY_MODE_MAP)?MAP_SHARED:MAP_PRIVATE), 1); + if(rd) { + struct timeval now; + gettimeofday(&now, NULL); + + if(strcmp(rd->magic, RRDDIMENSION_MAGIC) != 0) { + errno = 0; + info("Initializing file %s.", fullfilename); + bzero(rd, size); + } + else if(rd->memsize != size) { + errno = 0; + error("File %s does not have the desired size. Clearing it.", fullfilename); + bzero(rd, size); + } + else if(rd->multiplier != multiplier) { + errno = 0; + error("File %s does not have the same multiplier. Clearing it.", fullfilename); + bzero(rd, size); + } + else if(rd->divisor != divisor) { + errno = 0; + error("File %s does not have the same divisor. Clearing it.", fullfilename); + bzero(rd, size); + } + else if(rd->algorithm != algorithm) { + errno = 0; + error("File %s does not have the same algorithm. Clearing it.", fullfilename); + bzero(rd, size); + } + else if(rd->update_every != st->update_every) { + errno = 0; + error("File %s does not have the same refresh frequency. Clearing it.", fullfilename); + bzero(rd, size); + } + else if(usec_dt(&now, &rd->last_collected_time) > (rd->entries * rd->update_every * 1000000ULL)) { + errno = 0; + error("File %s is too old. Clearing it.", fullfilename); + bzero(rd, size); + } + else if(strcmp(rd->id, id) != 0) { + errno = 0; + error("File %s contents are not for dimension %s. Clearing it.", fullfilename, id); + // munmap(rd, size); + // rd = NULL; + bzero(rd, size); + } + } + + if(rd) { + // we have a file mapped for rd + rd->mapped = rrd_memory_mode; + rd->flags = 0x00000000; + rd->variables = NULL; + rd->next = NULL; + rd->name = NULL; + } + else { + // if we didn't manage to get a mmap'd dimension, just create one + + rd = callocz(1, size); + rd->mapped = RRD_MEMORY_MODE_RAM; + } + rd->memsize = size; + + strcpy(rd->magic, RRDDIMENSION_MAGIC); + strcpy(rd->cache_filename, fullfilename); + strncpyz(rd->id, id, RRD_ID_LENGTH_MAX); + rd->hash = simple_hash(rd->id); + + snprintfz(varname, CONFIG_MAX_NAME, "dim %s name", rd->id); + rd->name = config_get(st->id, varname, (name && *name)?name:rd->id); + + snprintfz(varname, CONFIG_MAX_NAME, "dim %s algorithm", rd->id); + rd->algorithm = rrddim_algorithm_id(config_get(st->id, varname, rrddim_algorithm_name(algorithm))); + + snprintfz(varname, CONFIG_MAX_NAME, "dim %s multiplier", rd->id); + rd->multiplier = config_get_number(st->id, varname, multiplier); + + snprintfz(varname, CONFIG_MAX_NAME, "dim %s divisor", rd->id); + rd->divisor = config_get_number(st->id, varname, divisor); + if(!rd->divisor) rd->divisor = 1; + + rd->entries = st->entries; + rd->update_every = st->update_every; + + // prevent incremental calculation spikes + rd->counter = 0; + rd->updated = 0; + rd->calculated_value = 0; + rd->last_calculated_value = 0; + rd->collected_value = 0; + rd->last_collected_value = 0; + rd->collected_volume = 0; + rd->stored_volume = 0; + rd->last_stored_value = 0; + rd->values[st->current_entry] = pack_storage_number(0, SN_NOT_EXISTS); + rd->last_collected_time.tv_sec = 0; + rd->last_collected_time.tv_usec = 0; + rd->rrdset = st; + + // append this dimension + pthread_rwlock_wrlock(&st->rwlock); + if(!st->dimensions) + st->dimensions = rd; + else { + RRDDIM *td = st->dimensions; + for(; td->next; td = td->next) ; + td->next = rd; + } + + if(health_enabled) { + rrddimvar_create(rd, RRDVAR_TYPE_CALCULATED, NULL, NULL, &rd->last_stored_value, 0); + rrddimvar_create(rd, RRDVAR_TYPE_COLLECTED, NULL, "_raw", &rd->last_collected_value, 0); + rrddimvar_create(rd, RRDVAR_TYPE_TIME_T, NULL, "_last_collected_t", &rd->last_collected_time.tv_sec, 0); + } + + pthread_rwlock_unlock(&st->rwlock); + + rrddim_index_add(st, rd); + + return(rd); } void rrddim_set_name(RRDSET *st, RRDDIM *rd, const char *name) { - debug(D_RRD_CALLS, "rrddim_set_name() %s.%s", st->name, rd->name); + debug(D_RRD_CALLS, "rrddim_set_name() %s.%s", st->name, rd->name); + + char varname[CONFIG_MAX_NAME + 1]; + snprintfz(varname, CONFIG_MAX_NAME, "dim %s name", rd->id); + config_set_default(st->id, varname, name); - char varname[CONFIG_MAX_NAME + 1]; - snprintfz(varname, CONFIG_MAX_NAME, "dim %s name", rd->id); - config_set_default(st->id, varname, name); + rrddimvar_rename_all(rd); } void rrddim_free(RRDSET *st, RRDDIM *rd) { - debug(D_RRD_CALLS, "rrddim_free() %s.%s", st->name, rd->name); - - RRDDIM *i, *last = NULL; - for(i = st->dimensions; i && i != rd ; i = i->next) last = i; - - if(!i) { - error("Request to free dimension %s.%s but it is not linked.", st->id, rd->name); - return; - } - - if(last) last->next = rd->next; - else st->dimensions = rd->next; - rd->next = NULL; - - rrddim_index_del(st, rd); - - // free(rd->annotations); - if(rd->mapped == RRD_MEMORY_MODE_SAVE) { - debug(D_RRD_CALLS, "Saving dimension '%s' to '%s'.", rd->name, rd->cache_filename); - savememory(rd->cache_filename, rd, rd->memsize); - - debug(D_RRD_CALLS, "Unmapping dimension '%s'.", rd->name); - munmap(rd, rd->memsize); - } - else if(rd->mapped == RRD_MEMORY_MODE_MAP) { - debug(D_RRD_CALLS, "Unmapping dimension '%s'.", rd->name); - munmap(rd, rd->memsize); - } - else { - debug(D_RRD_CALLS, "Removing dimension '%s'.", rd->name); - free(rd); - } + debug(D_RRD_CALLS, "rrddim_free() %s.%s", st->name, rd->name); + + if(rd == st->dimensions) + st->dimensions = rd->next; + else { + RRDDIM *i; + for (i = st->dimensions; i && i->next != rd; i = i->next) ; + + if (i && i->next == rd) + i->next = rd->next; + else + error("Request to free dimension '%s.%s' but it is not linked.", st->id, rd->name); + } + rd->next = NULL; + + while(rd->variables) + rrddimvar_free(rd->variables); + + rrddim_index_del(st, rd); + + // free(rd->annotations); + if(rd->mapped == RRD_MEMORY_MODE_SAVE) { + debug(D_RRD_CALLS, "Saving dimension '%s' to '%s'.", rd->name, rd->cache_filename); + savememory(rd->cache_filename, rd, rd->memsize); + + debug(D_RRD_CALLS, "Unmapping dimension '%s'.", rd->name); + munmap(rd, rd->memsize); + } + else if(rd->mapped == RRD_MEMORY_MODE_MAP) { + debug(D_RRD_CALLS, "Unmapping dimension '%s'.", rd->name); + munmap(rd, rd->memsize); + } + else { + debug(D_RRD_CALLS, "Removing dimension '%s'.", rd->name); + freez(rd); + } } void rrdset_free_all(void) { - info("Freeing all memory..."); + info("Freeing all memory..."); - RRDSET *st; - for(st = rrdset_root; st ;) { - RRDSET *next = st->next; + rrdhost_rwlock(&localhost); - while(st->dimensions) - rrddim_free(st, st->dimensions); + RRDSET *st; + for(st = localhost.rrdset_root; st ;) { + RRDSET *next = st->next; - rrdset_index_del(st); + pthread_rwlock_wrlock(&st->rwlock); - if(st->mapped == RRD_MEMORY_MODE_SAVE) { - debug(D_RRD_CALLS, "Saving stats '%s' to '%s'.", st->name, st->cache_filename); - savememory(st->cache_filename, st, st->memsize); + while(st->variables) + rrdsetvar_free(st->variables); - debug(D_RRD_CALLS, "Unmapping stats '%s'.", st->name); - munmap(st, st->memsize); - } - else if(st->mapped == RRD_MEMORY_MODE_MAP) { - debug(D_RRD_CALLS, "Unmapping stats '%s'.", st->name); - munmap(st, st->memsize); - } - else - free(st); + while(st->alarms) + rrdsetcalc_unlink(st->alarms); - st = next; - } - rrdset_root = NULL; + while(st->dimensions) + rrddim_free(st, st->dimensions); - info("Memory cleanup completed..."); -} + rrdset_index_del(&localhost, st); -void rrdset_save_all(void) -{ - debug(D_RRD_CALLS, "rrdset_save_all()"); + st->rrdfamily->use_count--; + if(!st->rrdfamily->use_count) + rrdfamily_free(st->rrdfamily); + + pthread_rwlock_unlock(&st->rwlock); + + if(st->mapped == RRD_MEMORY_MODE_SAVE) { + debug(D_RRD_CALLS, "Saving stats '%s' to '%s'.", st->name, st->cache_filename); + savememory(st->cache_filename, st, st->memsize); + + debug(D_RRD_CALLS, "Unmapping stats '%s'.", st->name); + munmap(st, st->memsize); + } + else if(st->mapped == RRD_MEMORY_MODE_MAP) { + debug(D_RRD_CALLS, "Unmapping stats '%s'.", st->name); + munmap(st, st->memsize); + } + else + freez(st); + + st = next; + } + localhost.rrdset_root = NULL; + + rrdhost_unlock(&localhost); + + info("Memory cleanup completed..."); +} - // let it log a few error messages - error_log_limit_reset(); +void rrdset_save_all(void) { + info("Saving database..."); - RRDSET *st; - RRDDIM *rd; + RRDSET *st; + RRDDIM *rd; - pthread_rwlock_wrlock(&rrdset_root_rwlock); - for(st = rrdset_root; st ; st = st->next) { - pthread_rwlock_wrlock(&st->rwlock); + rrdhost_rwlock(&localhost); + for(st = localhost.rrdset_root; st ; st = st->next) { + pthread_rwlock_wrlock(&st->rwlock); - if(st->mapped == RRD_MEMORY_MODE_SAVE) { - debug(D_RRD_CALLS, "Saving stats '%s' to '%s'.", st->name, st->cache_filename); - savememory(st->cache_filename, st, st->memsize); - } + if(st->mapped == RRD_MEMORY_MODE_SAVE) { + debug(D_RRD_CALLS, "Saving stats '%s' to '%s'.", st->name, st->cache_filename); + savememory(st->cache_filename, st, st->memsize); + } - for(rd = st->dimensions; rd ; rd = rd->next) { - if(likely(rd->mapped == RRD_MEMORY_MODE_SAVE)) { - debug(D_RRD_CALLS, "Saving dimension '%s' to '%s'.", rd->name, rd->cache_filename); - savememory(rd->cache_filename, rd, rd->memsize); - } - } + for(rd = st->dimensions; rd ; rd = rd->next) { + if(likely(rd->mapped == RRD_MEMORY_MODE_SAVE)) { + debug(D_RRD_CALLS, "Saving dimension '%s' to '%s'.", rd->name, rd->cache_filename); + savememory(rd->cache_filename, rd, rd->memsize); + } + } - pthread_rwlock_unlock(&st->rwlock); - } - pthread_rwlock_unlock(&rrdset_root_rwlock); + pthread_rwlock_unlock(&st->rwlock); + } + rrdhost_unlock(&localhost); } RRDSET *rrdset_find(const char *id) { - debug(D_RRD_CALLS, "rrdset_find() for chart %s", id); + debug(D_RRD_CALLS, "rrdset_find() for chart %s", id); - RRDSET *st = rrdset_index_find(id, 0); - return(st); + RRDSET *st = rrdset_index_find(&localhost, id, 0); + return(st); } RRDSET *rrdset_find_bytype(const char *type, const char *id) { - debug(D_RRD_CALLS, "rrdset_find_bytype() for chart %s.%s", type, id); + debug(D_RRD_CALLS, "rrdset_find_bytype() for chart %s.%s", type, id); - char buf[RRD_ID_LENGTH_MAX + 1]; + char buf[RRD_ID_LENGTH_MAX + 1]; - strncpyz(buf, type, RRD_ID_LENGTH_MAX - 1); - strcat(buf, "."); - int len = (int) strlen(buf); - strncpyz(&buf[len], id, (size_t) (RRD_ID_LENGTH_MAX - len)); + strncpyz(buf, type, RRD_ID_LENGTH_MAX - 1); + strcat(buf, "."); + int len = (int) strlen(buf); + strncpyz(&buf[len], id, (size_t) (RRD_ID_LENGTH_MAX - len)); - return(rrdset_find(buf)); + return(rrdset_find(buf)); } RRDSET *rrdset_find_byname(const char *name) { - debug(D_RRD_CALLS, "rrdset_find_byname() for chart %s", name); + debug(D_RRD_CALLS, "rrdset_find_byname() for chart %s", name); - RRDSET *st = rrdset_index_find_name(name, 0); - return(st); + RRDSET *st = rrdset_index_find_name(&localhost, name, 0); + return(st); } RRDDIM *rrddim_find(RRDSET *st, const char *id) { - debug(D_RRD_CALLS, "rrddim_find() for chart %s, dimension %s", st->name, id); + debug(D_RRD_CALLS, "rrddim_find() for chart %s, dimension %s", st->name, id); - return rrddim_index_find(st, id, 0); + return rrddim_index_find(st, id, 0); } int rrddim_hide(RRDSET *st, const char *id) { - debug(D_RRD_CALLS, "rrddim_hide() for chart %s, dimension %s", st->name, id); + debug(D_RRD_CALLS, "rrddim_hide() for chart %s, dimension %s", st->name, id); - RRDDIM *rd = rrddim_find(st, id); - if(unlikely(!rd)) { - error("Cannot find dimension with id '%s' on stats '%s' (%s).", id, st->name, st->id); - return 1; - } + RRDDIM *rd = rrddim_find(st, id); + if(unlikely(!rd)) { + error("Cannot find dimension with id '%s' on stats '%s' (%s).", id, st->name, st->id); + return 1; + } - rd->flags |= RRDDIM_FLAG_HIDDEN; - return 0; + rd->flags |= RRDDIM_FLAG_HIDDEN; + return 0; } int rrddim_unhide(RRDSET *st, const char *id) { - debug(D_RRD_CALLS, "rrddim_unhide() for chart %s, dimension %s", st->name, id); + debug(D_RRD_CALLS, "rrddim_unhide() for chart %s, dimension %s", st->name, id); - RRDDIM *rd = rrddim_find(st, id); - if(unlikely(!rd)) { - error("Cannot find dimension with id '%s' on stats '%s' (%s).", id, st->name, st->id); - return 1; - } + RRDDIM *rd = rrddim_find(st, id); + if(unlikely(!rd)) { + error("Cannot find dimension with id '%s' on stats '%s' (%s).", id, st->name, st->id); + return 1; + } - if(rd->flags & RRDDIM_FLAG_HIDDEN) rd->flags ^= RRDDIM_FLAG_HIDDEN; - return 0; + if(rd->flags & RRDDIM_FLAG_HIDDEN) rd->flags ^= RRDDIM_FLAG_HIDDEN; + return 0; } collected_number rrddim_set_by_pointer(RRDSET *st, RRDDIM *rd, collected_number value) { - debug(D_RRD_CALLS, "rrddim_set_by_pointer() for chart %s, dimension %s, value " COLLECTED_NUMBER_FORMAT, st->name, rd->name, value); + debug(D_RRD_CALLS, "rrddim_set_by_pointer() for chart %s, dimension %s, value " COLLECTED_NUMBER_FORMAT, st->name, rd->name, value); - gettimeofday(&rd->last_collected_time, NULL); - rd->collected_value = value; - rd->updated = 1; - rd->counter++; + gettimeofday(&rd->last_collected_time, NULL); + rd->collected_value = value; + rd->updated = 1; + rd->counter++; - return rd->last_collected_value; + return rd->last_collected_value; } collected_number rrddim_set(RRDSET *st, const char *id, collected_number value) { - RRDDIM *rd = rrddim_find(st, id); - if(unlikely(!rd)) { - error("Cannot find dimension with id '%s' on stats '%s' (%s).", id, st->name, st->id); - return 0; - } + RRDDIM *rd = rrddim_find(st, id); + if(unlikely(!rd)) { + error("Cannot find dimension with id '%s' on stats '%s' (%s).", id, st->name, st->id); + return 0; + } - return rrddim_set_by_pointer(st, rd, value); + return rrddim_set_by_pointer(st, rd, value); } void rrdset_next_usec(RRDSET *st, unsigned long long microseconds) { - if(!microseconds) rrdset_next(st); - else { - debug(D_RRD_CALLS, "rrdset_next_usec() for chart %s with microseconds %llu", st->name, microseconds); + if(!microseconds) rrdset_next(st); + else { + debug(D_RRD_CALLS, "rrdset_next_usec() for chart %s with microseconds %llu", st->name, microseconds); - if(unlikely(st->debug)) debug(D_RRD_STATS, "%s: NEXT: %llu microseconds", st->name, microseconds); - st->usec_since_last_update = microseconds; - } + if(unlikely(st->debug)) debug(D_RRD_STATS, "%s: NEXT: %llu microseconds", st->name, microseconds); + st->usec_since_last_update = microseconds; + } } void rrdset_next(RRDSET *st) { - unsigned long long microseconds = 0; + unsigned long long microseconds = 0; - if(likely(st->last_collected_time.tv_sec)) { - struct timeval now; - gettimeofday(&now, NULL); - microseconds = usecdiff(&now, &st->last_collected_time); - } - // prevent infinite loop - else microseconds = st->update_every * 1000000ULL; + if(likely(st->last_collected_time.tv_sec)) { + struct timeval now; + gettimeofday(&now, NULL); + microseconds = usec_dt(&now, &st->last_collected_time); + } + // prevent infinite loop + else microseconds = st->update_every * 1000000ULL; - rrdset_next_usec(st, microseconds); + rrdset_next_usec(st, microseconds); } void rrdset_next_plugins(RRDSET *st) { - rrdset_next(st); + rrdset_next(st); } unsigned long long rrdset_done(RRDSET *st) { - debug(D_RRD_CALLS, "rrdset_done() for chart %s", st->name); - - RRDDIM *rd, *last; - int oldstate, store_this_entry = 1, first_entry = 0; - unsigned long long last_ut, now_ut, next_ut, stored_entries = 0; - - if(unlikely(pthread_setcancelstate(PTHREAD_CANCEL_DISABLE, &oldstate) != 0)) - error("Cannot set pthread cancel state to DISABLE."); - - // a read lock is OK here - pthread_rwlock_rdlock(&st->rwlock); - - // enable the chart, if it was disabled - if(unlikely(rrd_delete_unupdated_dimensions) && !st->enabled) - st->enabled = 1; - - // check if the chart has a long time to be updated - if(unlikely(st->usec_since_last_update > st->entries * st->update_every * 1000000ULL)) { - info("%s: took too long to be updated (%0.3Lf secs). Reseting it.", st->name, (long double)(st->usec_since_last_update / 1000000.0)); - rrdset_reset(st); - st->usec_since_last_update = st->update_every * 1000000ULL; - first_entry = 1; - } - if(unlikely(st->debug)) debug(D_RRD_STATS, "%s: microseconds since last update: %llu", st->name, st->usec_since_last_update); - - // set last_collected_time - if(unlikely(!st->last_collected_time.tv_sec)) { - // it is the first entry - // set the last_collected_time to now - gettimeofday(&st->last_collected_time, NULL); - - // the first entry should not be stored - store_this_entry = 0; - first_entry = 1; - - if(unlikely(st->debug)) debug(D_RRD_STATS, "%s: has not set last_collected_time. Setting it now. Will not store the next entry.", st->name); - } - else { - // it is not the first entry - // calculate the proper last_collected_time, using usec_since_last_update - unsigned long long ut = st->last_collected_time.tv_sec * 1000000ULL + st->last_collected_time.tv_usec + st->usec_since_last_update; - st->last_collected_time.tv_sec = (time_t) (ut / 1000000ULL); - st->last_collected_time.tv_usec = (useconds_t) (ut % 1000000ULL); - } - - // if this set has not been updated in the past - // we fake the last_update time to be = now - usec_since_last_update - if(unlikely(!st->last_updated.tv_sec)) { - // it has never been updated before - // set a fake last_updated, in the past using usec_since_last_update - unsigned long long ut = st->last_collected_time.tv_sec * 1000000ULL + st->last_collected_time.tv_usec - st->usec_since_last_update; - st->last_updated.tv_sec = (time_t) (ut / 1000000ULL); - st->last_updated.tv_usec = (useconds_t) (ut % 1000000ULL); - - // the first entry should not be stored - store_this_entry = 0; - first_entry = 1; - - if(unlikely(st->debug)) debug(D_RRD_STATS, "%s: initializing last_updated to now - %llu microseconds (%0.3Lf). Will not store the next entry.", st->name, st->usec_since_last_update, (long double)ut/1000000.0); - } - - // check if we will re-write the entire data set - if(unlikely(usecdiff(&st->last_collected_time, &st->last_updated) > st->update_every * st->entries * 1000000ULL)) { - info("%s: too old data (last updated at %u.%u, last collected at %u.%u). Reseting it. Will not store the next entry.", st->name, st->last_updated.tv_sec, st->last_updated.tv_usec, st->last_collected_time.tv_sec, st->last_collected_time.tv_usec); - rrdset_reset(st); - - st->usec_since_last_update = st->update_every * 1000000ULL; - - gettimeofday(&st->last_collected_time, NULL); - - unsigned long long ut = st->last_collected_time.tv_sec * 1000000ULL + st->last_collected_time.tv_usec - st->usec_since_last_update; - st->last_updated.tv_sec = (time_t) (ut / 1000000ULL); - st->last_updated.tv_usec = (useconds_t) (ut % 1000000ULL); - - // the first entry should not be stored - store_this_entry = 0; - first_entry = 1; - } - - // these are the 3 variables that will help us in interpolation - // last_ut = the last time we added a value to the storage - // now_ut = the time the current value is taken at - // next_ut = the time of the next interpolation point - last_ut = st->last_updated.tv_sec * 1000000ULL + st->last_updated.tv_usec; - now_ut = st->last_collected_time.tv_sec * 1000000ULL + st->last_collected_time.tv_usec; - next_ut = (st->last_updated.tv_sec + st->update_every) * 1000000ULL; - - if(unlikely(!first_entry && now_ut < next_ut)) { - if(unlikely(st->debug)) debug(D_RRD_STATS, "%s: THIS IS IN THE SAME INTERPOLATION POINT", st->name); - } - - if(unlikely(st->debug)) { - debug(D_RRD_STATS, "%s: last ut = %0.3Lf (last updated time)", st->name, (long double)last_ut/1000000.0); - debug(D_RRD_STATS, "%s: now ut = %0.3Lf (current update time)", st->name, (long double)now_ut/1000000.0); - debug(D_RRD_STATS, "%s: next ut = %0.3Lf (next interpolation point)", st->name, (long double)next_ut/1000000.0); - } - - if(unlikely(!st->counter_done)) { - store_this_entry = 0; - if(unlikely(st->debug)) debug(D_RRD_STATS, "%s: Will not store the next entry.", st->name); - } - st->counter_done++; - - // calculate totals and count the dimensions - int dimensions; - st->collected_total = 0; - for( rd = st->dimensions, dimensions = 0 ; likely(rd) ; rd = rd->next, dimensions++ ) - st->collected_total += rd->collected_value; - - uint32_t storage_flags = SN_EXISTS; - - // process all dimensions to calculate their values - // based on the collected figures only - // at this stage we do not interpolate anything - for( rd = st->dimensions ; likely(rd) ; rd = rd->next ) { - - if(unlikely(st->debug)) debug(D_RRD_STATS, "%s/%s: START " - " last_collected_value = " COLLECTED_NUMBER_FORMAT - " collected_value = " COLLECTED_NUMBER_FORMAT - " last_calculated_value = " CALCULATED_NUMBER_FORMAT - " calculated_value = " CALCULATED_NUMBER_FORMAT - , st->id, rd->name - , rd->last_collected_value - , rd->collected_value - , rd->last_calculated_value - , rd->calculated_value - ); - - switch(rd->algorithm) { - case RRDDIM_ABSOLUTE: - rd->calculated_value = (calculated_number)rd->collected_value - * (calculated_number)rd->multiplier - / (calculated_number)rd->divisor; - - if(unlikely(st->debug)) - debug(D_RRD_STATS, "%s/%s: CALC ABS/ABS-NO-IN " - CALCULATED_NUMBER_FORMAT " = " - COLLECTED_NUMBER_FORMAT - " * " CALCULATED_NUMBER_FORMAT - " / " CALCULATED_NUMBER_FORMAT - , st->id, rd->name - , rd->calculated_value - , rd->collected_value - , (calculated_number)rd->multiplier - , (calculated_number)rd->divisor - ); - break; - - case RRDDIM_PCENT_OVER_ROW_TOTAL: - if(unlikely(!st->collected_total)) rd->calculated_value = 0; - else - // the percentage of the current value - // over the total of all dimensions - rd->calculated_value = - (calculated_number)100 - * (calculated_number)rd->collected_value - / (calculated_number)st->collected_total; - - if(unlikely(st->debug)) - debug(D_RRD_STATS, "%s/%s: CALC PCENT-ROW " - CALCULATED_NUMBER_FORMAT " = 100" - " * " COLLECTED_NUMBER_FORMAT - " / " COLLECTED_NUMBER_FORMAT - , st->id, rd->name - , rd->calculated_value - , rd->collected_value - , st->collected_total - ); - break; - - case RRDDIM_INCREMENTAL: - if(unlikely(!rd->updated || rd->counter <= 1)) { - rd->calculated_value = 0; - continue; - } - - // if the new is smaller than the old (an overflow, or reset), set the old equal to the new - // to reset the calculation (it will give zero as the calculation for this second) - if(unlikely(rd->last_collected_value > rd->collected_value)) { - debug(D_RRD_STATS, "%s.%s: RESET or OVERFLOW. Last collected value = " COLLECTED_NUMBER_FORMAT ", current = " COLLECTED_NUMBER_FORMAT - , st->name, rd->name - , rd->last_collected_value - , rd->collected_value); - if(!(rd->flags & RRDDIM_FLAG_DONT_DETECT_RESETS_OR_OVERFLOWS)) storage_flags = SN_EXISTS_RESET; - rd->last_collected_value = rd->collected_value; - } - - rd->calculated_value = (calculated_number)(rd->collected_value - rd->last_collected_value) - * (calculated_number)rd->multiplier - / (calculated_number)rd->divisor; - - if(unlikely(st->debug)) - debug(D_RRD_STATS, "%s/%s: CALC INC PRE " - CALCULATED_NUMBER_FORMAT " = (" - COLLECTED_NUMBER_FORMAT " - " COLLECTED_NUMBER_FORMAT - ")" - " * " CALCULATED_NUMBER_FORMAT - " / " CALCULATED_NUMBER_FORMAT - , st->id, rd->name - , rd->calculated_value - , rd->collected_value, rd->last_collected_value - , (calculated_number)rd->multiplier - , (calculated_number)rd->divisor - ); - break; - - case RRDDIM_PCENT_OVER_DIFF_TOTAL: - if(unlikely(!rd->updated || rd->counter <= 1)) { - rd->calculated_value = 0; - continue; - } - - // the percentage of the current increment - // over the increment of all dimensions together - if(unlikely(st->collected_total == st->last_collected_total)) rd->calculated_value = rd->last_calculated_value; - else rd->calculated_value = - (calculated_number)100 - * (calculated_number)(rd->collected_value - rd->last_collected_value) - / (calculated_number)(st->collected_total - st->last_collected_total); - - if(unlikely(st->debug)) - debug(D_RRD_STATS, "%s/%s: CALC PCENT-DIFF " - CALCULATED_NUMBER_FORMAT " = 100" - " * (" COLLECTED_NUMBER_FORMAT " - " COLLECTED_NUMBER_FORMAT ")" - " / (" COLLECTED_NUMBER_FORMAT " - " COLLECTED_NUMBER_FORMAT ")" - , st->id, rd->name - , rd->calculated_value - , rd->collected_value, rd->last_collected_value - , st->collected_total, st->last_collected_total - ); - break; - - default: - // make the default zero, to make sure - // it gets noticed when we add new types - rd->calculated_value = 0; - - if(unlikely(st->debug)) - debug(D_RRD_STATS, "%s/%s: CALC " - CALCULATED_NUMBER_FORMAT " = 0" - , st->id, rd->name - , rd->calculated_value - ); - break; - } - - if(unlikely(st->debug)) debug(D_RRD_STATS, "%s/%s: PHASE2 " - " last_collected_value = " COLLECTED_NUMBER_FORMAT - " collected_value = " COLLECTED_NUMBER_FORMAT - " last_calculated_value = " CALCULATED_NUMBER_FORMAT - " calculated_value = " CALCULATED_NUMBER_FORMAT - , st->id, rd->name - , rd->last_collected_value - , rd->collected_value - , rd->last_calculated_value - , rd->calculated_value - ); - - } - - // at this point we have all the calculated values ready - // it is now time to interpolate values on a second boundary - - unsigned long long first_ut = last_ut; - long long iterations = (now_ut - last_ut) / (st->update_every * 1000000ULL); - if((now_ut % (st->update_every * 1000000ULL)) == 0) iterations++; - - for( ; likely(next_ut <= now_ut) ; next_ut += st->update_every * 1000000ULL, iterations-- ) { + if(unlikely(netdata_exit)) return 0; + + debug(D_RRD_CALLS, "rrdset_done() for chart %s", st->name); + + RRDDIM *rd, *last; + int oldstate, store_this_entry = 1, first_entry = 0; + unsigned long long last_ut, now_ut, next_ut, stored_entries = 0; + + if(unlikely(pthread_setcancelstate(PTHREAD_CANCEL_DISABLE, &oldstate) != 0)) + error("Cannot set pthread cancel state to DISABLE."); + + // a read lock is OK here + pthread_rwlock_rdlock(&st->rwlock); + + // enable the chart, if it was disabled + if(unlikely(rrd_delete_unupdated_dimensions) && !st->enabled) + st->enabled = 1; + + // check if the chart has a long time to be updated + if(unlikely(st->usec_since_last_update > st->entries * st->update_every * 1000000ULL)) { + info("%s: took too long to be updated (%0.3Lf secs). Reseting it.", st->name, (long double)(st->usec_since_last_update / 1000000.0)); + rrdset_reset(st); + st->usec_since_last_update = st->update_every * 1000000ULL; + first_entry = 1; + } + if(unlikely(st->debug)) debug(D_RRD_STATS, "%s: microseconds since last update: %llu", st->name, st->usec_since_last_update); + + // set last_collected_time + if(unlikely(!st->last_collected_time.tv_sec)) { + // it is the first entry + // set the last_collected_time to now + gettimeofday(&st->last_collected_time, NULL); + + // the first entry should not be stored + store_this_entry = 0; + first_entry = 1; + + if(unlikely(st->debug)) debug(D_RRD_STATS, "%s: has not set last_collected_time. Setting it now. Will not store the next entry.", st->name); + } + else { + // it is not the first entry + // calculate the proper last_collected_time, using usec_since_last_update + unsigned long long ut = st->last_collected_time.tv_sec * 1000000ULL + st->last_collected_time.tv_usec + st->usec_since_last_update; + st->last_collected_time.tv_sec = (time_t) (ut / 1000000ULL); + st->last_collected_time.tv_usec = (suseconds_t) (ut % 1000000ULL); + } + + // if this set has not been updated in the past + // we fake the last_update time to be = now - usec_since_last_update + if(unlikely(!st->last_updated.tv_sec)) { + // it has never been updated before + // set a fake last_updated, in the past using usec_since_last_update + unsigned long long ut = st->last_collected_time.tv_sec * 1000000ULL + st->last_collected_time.tv_usec - st->usec_since_last_update; + st->last_updated.tv_sec = (time_t) (ut / 1000000ULL); + st->last_updated.tv_usec = (suseconds_t) (ut % 1000000ULL); + + // the first entry should not be stored + store_this_entry = 0; + first_entry = 1; + + if(unlikely(st->debug)) debug(D_RRD_STATS, "%s: initializing last_updated to now - %llu microseconds (%0.3Lf). Will not store the next entry.", st->name, st->usec_since_last_update, (long double)ut/1000000.0); + } + + // check if we will re-write the entire data set + if(unlikely(usec_dt(&st->last_collected_time, &st->last_updated) > st->update_every * st->entries * 1000000ULL)) { + info("%s: too old data (last updated at %ld.%ld, last collected at %ld.%ld). Reseting it. Will not store the next entry.", st->name, st->last_updated.tv_sec, st->last_updated.tv_usec, st->last_collected_time.tv_sec, st->last_collected_time.tv_usec); + rrdset_reset(st); + + st->usec_since_last_update = st->update_every * 1000000ULL; + + gettimeofday(&st->last_collected_time, NULL); + + unsigned long long ut = st->last_collected_time.tv_sec * 1000000ULL + st->last_collected_time.tv_usec - st->usec_since_last_update; + st->last_updated.tv_sec = (time_t) (ut / 1000000ULL); + st->last_updated.tv_usec = (suseconds_t) (ut % 1000000ULL); + + // the first entry should not be stored + store_this_entry = 0; + first_entry = 1; + } + + // these are the 3 variables that will help us in interpolation + // last_ut = the last time we added a value to the storage + // now_ut = the time the current value is taken at + // next_ut = the time of the next interpolation point + last_ut = st->last_updated.tv_sec * 1000000ULL + st->last_updated.tv_usec; + now_ut = st->last_collected_time.tv_sec * 1000000ULL + st->last_collected_time.tv_usec; + next_ut = (st->last_updated.tv_sec + st->update_every) * 1000000ULL; + + if(unlikely(!first_entry && now_ut < next_ut)) { + if(unlikely(st->debug)) debug(D_RRD_STATS, "%s: THIS IS IN THE SAME INTERPOLATION POINT", st->name); + } + + if(unlikely(st->debug)) { + debug(D_RRD_STATS, "%s: last ut = %0.3Lf (last updated time)", st->name, (long double)last_ut/1000000.0); + debug(D_RRD_STATS, "%s: now ut = %0.3Lf (current update time)", st->name, (long double)now_ut/1000000.0); + debug(D_RRD_STATS, "%s: next ut = %0.3Lf (next interpolation point)", st->name, (long double)next_ut/1000000.0); + } + + if(unlikely(!st->counter_done)) { + store_this_entry = 0; + if(unlikely(st->debug)) debug(D_RRD_STATS, "%s: Will not store the next entry.", st->name); + } + st->counter_done++; + + // calculate totals and count the dimensions + int dimensions; + st->collected_total = 0; + for( rd = st->dimensions, dimensions = 0 ; likely(rd) ; rd = rd->next, dimensions++ ) + if(likely(rd->updated)) st->collected_total += rd->collected_value; + + uint32_t storage_flags = SN_EXISTS; + + // process all dimensions to calculate their values + // based on the collected figures only + // at this stage we do not interpolate anything + for( rd = st->dimensions ; likely(rd) ; rd = rd->next ) { + + if(unlikely(!rd->updated)) { + rd->calculated_value = 0; + continue; + } + + if(unlikely(st->debug)) debug(D_RRD_STATS, "%s/%s: START " + " last_collected_value = " COLLECTED_NUMBER_FORMAT + " collected_value = " COLLECTED_NUMBER_FORMAT + " last_calculated_value = " CALCULATED_NUMBER_FORMAT + " calculated_value = " CALCULATED_NUMBER_FORMAT + , st->id, rd->name + , rd->last_collected_value + , rd->collected_value + , rd->last_calculated_value + , rd->calculated_value + ); + + switch(rd->algorithm) { + case RRDDIM_ABSOLUTE: + rd->calculated_value = (calculated_number)rd->collected_value + * (calculated_number)rd->multiplier + / (calculated_number)rd->divisor; + + if(unlikely(st->debug)) + debug(D_RRD_STATS, "%s/%s: CALC ABS/ABS-NO-IN " + CALCULATED_NUMBER_FORMAT " = " + COLLECTED_NUMBER_FORMAT + " * " CALCULATED_NUMBER_FORMAT + " / " CALCULATED_NUMBER_FORMAT + , st->id, rd->name + , rd->calculated_value + , rd->collected_value + , (calculated_number)rd->multiplier + , (calculated_number)rd->divisor + ); + break; + + case RRDDIM_PCENT_OVER_ROW_TOTAL: + if(unlikely(!st->collected_total)) + rd->calculated_value = 0; + else + // the percentage of the current value + // over the total of all dimensions + rd->calculated_value = + (calculated_number)100 + * (calculated_number)rd->collected_value + / (calculated_number)st->collected_total; + + if(unlikely(st->debug)) + debug(D_RRD_STATS, "%s/%s: CALC PCENT-ROW " + CALCULATED_NUMBER_FORMAT " = 100" + " * " COLLECTED_NUMBER_FORMAT + " / " COLLECTED_NUMBER_FORMAT + , st->id, rd->name + , rd->calculated_value + , rd->collected_value + , st->collected_total + ); + break; + + case RRDDIM_INCREMENTAL: + if(unlikely(rd->counter <= 1)) { + rd->calculated_value = 0; + continue; + } + + // if the new is smaller than the old (an overflow, or reset), set the old equal to the new + // to reset the calculation (it will give zero as the calculation for this second) + if(unlikely(rd->last_collected_value > rd->collected_value)) { + debug(D_RRD_STATS, "%s.%s: RESET or OVERFLOW. Last collected value = " COLLECTED_NUMBER_FORMAT ", current = " COLLECTED_NUMBER_FORMAT + , st->name, rd->name + , rd->last_collected_value + , rd->collected_value); + if(!(rd->flags & RRDDIM_FLAG_DONT_DETECT_RESETS_OR_OVERFLOWS)) storage_flags = SN_EXISTS_RESET; + rd->last_collected_value = rd->collected_value; + } + + rd->calculated_value = + (calculated_number)(rd->collected_value - rd->last_collected_value) + * (calculated_number)rd->multiplier + / (calculated_number)rd->divisor; + + if(unlikely(st->debug)) + debug(D_RRD_STATS, "%s/%s: CALC INC PRE " + CALCULATED_NUMBER_FORMAT " = (" + COLLECTED_NUMBER_FORMAT " - " COLLECTED_NUMBER_FORMAT + ")" + " * " CALCULATED_NUMBER_FORMAT + " / " CALCULATED_NUMBER_FORMAT + , st->id, rd->name + , rd->calculated_value + , rd->collected_value, rd->last_collected_value + , (calculated_number)rd->multiplier + , (calculated_number)rd->divisor + ); + break; + + case RRDDIM_PCENT_OVER_DIFF_TOTAL: + if(unlikely(rd->counter <= 1)) { + rd->calculated_value = 0; + continue; + } + + // if the new is smaller than the old (an overflow, or reset), set the old equal to the new + // to reset the calculation (it will give zero as the calculation for this second) + if(unlikely(rd->last_collected_value > rd->collected_value)) { + debug(D_RRD_STATS, "%s.%s: RESET or OVERFLOW. Last collected value = " COLLECTED_NUMBER_FORMAT ", current = " COLLECTED_NUMBER_FORMAT + , st->name, rd->name + , rd->last_collected_value + , rd->collected_value); + if(!(rd->flags & RRDDIM_FLAG_DONT_DETECT_RESETS_OR_OVERFLOWS)) storage_flags = SN_EXISTS_RESET; + rd->last_collected_value = rd->collected_value; + } + + // the percentage of the current increment + // over the increment of all dimensions together + if(unlikely(st->collected_total == st->last_collected_total)) + rd->calculated_value = 0; + else + rd->calculated_value = + (calculated_number)100 + * (calculated_number)(rd->collected_value - rd->last_collected_value) + / (calculated_number)(st->collected_total - st->last_collected_total); + + if(unlikely(st->debug)) + debug(D_RRD_STATS, "%s/%s: CALC PCENT-DIFF " + CALCULATED_NUMBER_FORMAT " = 100" + " * (" COLLECTED_NUMBER_FORMAT " - " COLLECTED_NUMBER_FORMAT ")" + " / (" COLLECTED_NUMBER_FORMAT " - " COLLECTED_NUMBER_FORMAT ")" + , st->id, rd->name + , rd->calculated_value + , rd->collected_value, rd->last_collected_value + , st->collected_total, st->last_collected_total + ); + break; + + default: + // make the default zero, to make sure + // it gets noticed when we add new types + rd->calculated_value = 0; + + if(unlikely(st->debug)) + debug(D_RRD_STATS, "%s/%s: CALC " + CALCULATED_NUMBER_FORMAT " = 0" + , st->id, rd->name + , rd->calculated_value + ); + break; + } + + if(unlikely(st->debug)) debug(D_RRD_STATS, "%s/%s: PHASE2 " + " last_collected_value = " COLLECTED_NUMBER_FORMAT + " collected_value = " COLLECTED_NUMBER_FORMAT + " last_calculated_value = " CALCULATED_NUMBER_FORMAT + " calculated_value = " CALCULATED_NUMBER_FORMAT + , st->id, rd->name + , rd->last_collected_value + , rd->collected_value + , rd->last_calculated_value + , rd->calculated_value + ); + + } + + // at this point we have all the calculated values ready + // it is now time to interpolate values on a second boundary + + unsigned long long first_ut = last_ut; + long long iterations = (now_ut - last_ut) / (st->update_every * 1000000ULL); + if((now_ut % (st->update_every * 1000000ULL)) == 0) iterations++; + + for( ; likely(next_ut <= now_ut) ; next_ut += st->update_every * 1000000ULL, iterations-- ) { #ifdef NETDATA_INTERNAL_CHECKS - if(iterations < 0) { error("%s: iterations calculation wrapped! first_ut = %llu, last_ut = %llu, next_ut = %llu, now_ut = %llu", st->name, first_ut, last_ut, next_ut, now_ut); } + if(iterations < 0) { error("%s: iterations calculation wrapped! first_ut = %llu, last_ut = %llu, next_ut = %llu, now_ut = %llu", st->name, first_ut, last_ut, next_ut, now_ut); } #endif - if(unlikely(st->debug)) { - debug(D_RRD_STATS, "%s: last ut = %0.3Lf (last updated time)", st->name, (long double)last_ut/1000000.0); - debug(D_RRD_STATS, "%s: next ut = %0.3Lf (next interpolation point)", st->name, (long double)next_ut/1000000.0); - } - - st->last_updated.tv_sec = (time_t) (next_ut / 1000000ULL); - st->last_updated.tv_usec = 0; - - for( rd = st->dimensions ; likely(rd) ; rd = rd->next ) { - calculated_number new_value; - - switch(rd->algorithm) { - case RRDDIM_INCREMENTAL: - new_value = (calculated_number) - ( rd->calculated_value - * (calculated_number)(next_ut - last_ut) - / (calculated_number)(now_ut - last_ut) - ); - - if(unlikely(st->debug)) - debug(D_RRD_STATS, "%s/%s: CALC2 INC " - CALCULATED_NUMBER_FORMAT " = " - CALCULATED_NUMBER_FORMAT - " * %llu" - " / %llu" - , st->id, rd->name - , new_value - , rd->calculated_value - , (next_ut - last_ut) - , (now_ut - last_ut) - ); - - rd->calculated_value -= new_value; - new_value += rd->last_calculated_value; - rd->last_calculated_value = 0; - new_value /= (calculated_number)st->update_every; - break; - - case RRDDIM_ABSOLUTE: - case RRDDIM_PCENT_OVER_ROW_TOTAL: - case RRDDIM_PCENT_OVER_DIFF_TOTAL: - default: - if(iterations == 1) { - // this is the last iteration - // do not interpolate - // just show the calculated value - - new_value = rd->calculated_value; - } - else { - // we have missed an update - // interpolate in the middle values - - new_value = (calculated_number) - ( ( (rd->calculated_value - rd->last_calculated_value) - * (calculated_number)(next_ut - first_ut) - / (calculated_number)(now_ut - first_ut) - ) - + rd->last_calculated_value - ); - - if(unlikely(st->debug)) - debug(D_RRD_STATS, "%s/%s: CALC2 DEF " - CALCULATED_NUMBER_FORMAT " = (((" - "(" CALCULATED_NUMBER_FORMAT " - " CALCULATED_NUMBER_FORMAT ")" - " * %llu" - " / %llu) + " CALCULATED_NUMBER_FORMAT - , st->id, rd->name - , new_value - , rd->calculated_value, rd->last_calculated_value - , (next_ut - first_ut) - , (now_ut - first_ut), rd->last_calculated_value - ); - - // this is wrong - // it fades the value towards the target - // while we know the calculated value is different - // if(likely(next_ut + st->update_every * 1000000ULL > now_ut)) rd->calculated_value = new_value; - } - break; - } - - if(unlikely(!store_this_entry)) { - store_this_entry = 1; - continue; - } - - if(likely(rd->updated && rd->counter > 1 && iterations < st->gap_when_lost_iterations_above)) { - rd->values[st->current_entry] = pack_storage_number(new_value, storage_flags ); - - if(unlikely(st->debug)) - debug(D_RRD_STATS, "%s/%s: STORE[%ld] " - CALCULATED_NUMBER_FORMAT " = " CALCULATED_NUMBER_FORMAT - , st->id, rd->name - , st->current_entry - , unpack_storage_number(rd->values[st->current_entry]), new_value - ); - } - else { - if(unlikely(st->debug)) debug(D_RRD_STATS, "%s/%s: STORE[%ld] = NON EXISTING " - , st->id, rd->name - , st->current_entry - ); - rd->values[st->current_entry] = pack_storage_number(0, SN_NOT_EXISTS); - } - - stored_entries++; - - if(unlikely(st->debug)) { - calculated_number t1 = new_value * (calculated_number)rd->multiplier / (calculated_number)rd->divisor; - calculated_number t2 = unpack_storage_number(rd->values[st->current_entry]); - calculated_number accuracy = accuracy_loss(t1, t2); - debug(D_RRD_STATS, "%s/%s: UNPACK[%ld] = " CALCULATED_NUMBER_FORMAT " FLAGS=0x%08x (original = " CALCULATED_NUMBER_FORMAT ", accuracy loss = " CALCULATED_NUMBER_FORMAT "%%%s)" - , st->id, rd->name - , st->current_entry - , t2 - , get_storage_number_flags(rd->values[st->current_entry]) - , t1 - , accuracy - , (accuracy > ACCURACY_LOSS) ? " **TOO BIG** " : "" - ); - - rd->collected_volume += t1; - rd->stored_volume += t2; - accuracy = accuracy_loss(rd->collected_volume, rd->stored_volume); - debug(D_RRD_STATS, "%s/%s: VOLUME[%ld] = " CALCULATED_NUMBER_FORMAT ", calculated = " CALCULATED_NUMBER_FORMAT ", accuracy loss = " CALCULATED_NUMBER_FORMAT "%%%s" - , st->id, rd->name - , st->current_entry - , rd->stored_volume - , rd->collected_volume - , accuracy - , (accuracy > ACCURACY_LOSS) ? " **TOO BIG** " : "" - ); - - } - } - // reset the storage flags for the next point, if any; - storage_flags = SN_EXISTS; - - st->counter++; - st->current_entry = ((st->current_entry + 1) >= st->entries) ? 0 : st->current_entry + 1; - last_ut = next_ut; - } - - // align next interpolation to last collection point - if(likely(stored_entries || !store_this_entry)) { - st->last_updated.tv_sec = st->last_collected_time.tv_sec; - st->last_updated.tv_usec = st->last_collected_time.tv_usec; - } - - for( rd = st->dimensions; likely(rd) ; rd = rd->next ) { - if(unlikely(!rd->updated)) continue; - - if(likely(stored_entries || !store_this_entry)) { - if(unlikely(st->debug)) debug(D_RRD_STATS, "%s/%s: setting last_collected_value (old: " COLLECTED_NUMBER_FORMAT ") to last_collected_value (new: " COLLECTED_NUMBER_FORMAT ")", st->id, rd->name, rd->last_collected_value, rd->collected_value); - rd->last_collected_value = rd->collected_value; - - if(unlikely(st->debug)) debug(D_RRD_STATS, "%s/%s: setting last_calculated_value (old: " CALCULATED_NUMBER_FORMAT ") to last_calculated_value (new: " CALCULATED_NUMBER_FORMAT ")", st->id, rd->name, rd->last_calculated_value, rd->calculated_value); - rd->last_calculated_value = rd->calculated_value; - } - - rd->calculated_value = 0; - rd->collected_value = 0; - rd->updated = 0; - - if(unlikely(st->debug)) debug(D_RRD_STATS, "%s/%s: END " - " last_collected_value = " COLLECTED_NUMBER_FORMAT - " collected_value = " COLLECTED_NUMBER_FORMAT - " last_calculated_value = " CALCULATED_NUMBER_FORMAT - " calculated_value = " CALCULATED_NUMBER_FORMAT - , st->id, rd->name - , rd->last_collected_value - , rd->collected_value - , rd->last_calculated_value - , rd->calculated_value - ); - } - st->last_collected_total = st->collected_total; - - // ALL DONE ABOUT THE DATA UPDATE - // -------------------------------------------------------------------- - - // find if there are any obsolete dimensions (not updated recently) - if(unlikely(rrd_delete_unupdated_dimensions)) { - - for( rd = st->dimensions; likely(rd) ; rd = rd->next ) - if((rd->last_collected_time.tv_sec + (rrd_delete_unupdated_dimensions * st->update_every)) < st->last_collected_time.tv_sec) - break; - - if(unlikely(rd)) { - // there is dimension to free - // upgrade our read lock to a write lock - pthread_rwlock_unlock(&st->rwlock); - pthread_rwlock_wrlock(&st->rwlock); - - for( rd = st->dimensions, last = NULL ; likely(rd) ; ) { - // remove it only it is not updated in rrd_delete_unupdated_dimensions seconds - - if(unlikely((rd->last_collected_time.tv_sec + (rrd_delete_unupdated_dimensions * st->update_every)) < st->last_collected_time.tv_sec)) { - info("Removing obsolete dimension '%s' (%s) of '%s' (%s).", rd->name, rd->id, st->name, st->id); - - if(unlikely(!last)) { - st->dimensions = rd->next; - rd->next = NULL; - rrddim_free(st, rd); - rd = st->dimensions; - continue; - } - else { - last->next = rd->next; - rd->next = NULL; - rrddim_free(st, rd); - rd = last->next; - continue; - } - } - - last = rd; - rd = rd->next; - } - - if(unlikely(!st->dimensions)) { - info("Disabling chart %s (%s) since it does not have any dimensions", st->name, st->id); - st->enabled = 0; - } - } - } - - pthread_rwlock_unlock(&st->rwlock); - - if(unlikely(pthread_setcancelstate(oldstate, NULL) != 0)) - error("Cannot set pthread cancel state to RESTORE (%d).", oldstate); - - return(st->usec_since_last_update); + if(unlikely(st->debug)) { + debug(D_RRD_STATS, "%s: last ut = %0.3Lf (last updated time)", st->name, (long double)last_ut/1000000.0); + debug(D_RRD_STATS, "%s: next ut = %0.3Lf (next interpolation point)", st->name, (long double)next_ut/1000000.0); + } + + st->last_updated.tv_sec = (time_t) (next_ut / 1000000ULL); + st->last_updated.tv_usec = 0; + + for( rd = st->dimensions ; likely(rd) ; rd = rd->next ) { + calculated_number new_value; + + switch(rd->algorithm) { + case RRDDIM_INCREMENTAL: + new_value = (calculated_number) + ( rd->calculated_value + * (calculated_number)(next_ut - last_ut) + / (calculated_number)(now_ut - last_ut) + ); + + if(unlikely(st->debug)) + debug(D_RRD_STATS, "%s/%s: CALC2 INC " + CALCULATED_NUMBER_FORMAT " = " + CALCULATED_NUMBER_FORMAT + " * %llu" + " / %llu" + , st->id, rd->name + , new_value + , rd->calculated_value + , (next_ut - last_ut) + , (now_ut - last_ut) + ); + + rd->calculated_value -= new_value; + new_value += rd->last_calculated_value; + rd->last_calculated_value = 0; + new_value /= (calculated_number)st->update_every; + break; + + case RRDDIM_ABSOLUTE: + case RRDDIM_PCENT_OVER_ROW_TOTAL: + case RRDDIM_PCENT_OVER_DIFF_TOTAL: + default: + if(iterations == 1) { + // this is the last iteration + // do not interpolate + // just show the calculated value + + new_value = rd->calculated_value; + } + else { + // we have missed an update + // interpolate in the middle values + + new_value = (calculated_number) + ( ( (rd->calculated_value - rd->last_calculated_value) + * (calculated_number)(next_ut - first_ut) + / (calculated_number)(now_ut - first_ut) + ) + + rd->last_calculated_value + ); + + if(unlikely(st->debug)) + debug(D_RRD_STATS, "%s/%s: CALC2 DEF " + CALCULATED_NUMBER_FORMAT " = (((" + "(" CALCULATED_NUMBER_FORMAT " - " CALCULATED_NUMBER_FORMAT ")" + " * %llu" + " / %llu) + " CALCULATED_NUMBER_FORMAT + , st->id, rd->name + , new_value + , rd->calculated_value, rd->last_calculated_value + , (next_ut - first_ut) + , (now_ut - first_ut), rd->last_calculated_value + ); + + // this is wrong + // it fades the value towards the target + // while we know the calculated value is different + // if(likely(next_ut + st->update_every * 1000000ULL > now_ut)) rd->calculated_value = new_value; + } + break; + } + + if(unlikely(!store_this_entry)) { + // store_this_entry = 1; + continue; + } + + if(likely(rd->updated && rd->counter > 1 && iterations < st->gap_when_lost_iterations_above)) { + rd->values[st->current_entry] = pack_storage_number(new_value, storage_flags ); + rd->last_stored_value = new_value; + + if(unlikely(st->debug)) + debug(D_RRD_STATS, "%s/%s: STORE[%ld] " + CALCULATED_NUMBER_FORMAT " = " CALCULATED_NUMBER_FORMAT + , st->id, rd->name + , st->current_entry + , unpack_storage_number(rd->values[st->current_entry]), new_value + ); + } + else { + if(unlikely(st->debug)) debug(D_RRD_STATS, "%s/%s: STORE[%ld] = NON EXISTING " + , st->id, rd->name + , st->current_entry + ); + rd->values[st->current_entry] = pack_storage_number(0, SN_NOT_EXISTS); + rd->last_stored_value = NAN; + } + + stored_entries++; + + if(unlikely(st->debug)) { + calculated_number t1 = new_value * (calculated_number)rd->multiplier / (calculated_number)rd->divisor; + calculated_number t2 = unpack_storage_number(rd->values[st->current_entry]); + calculated_number accuracy = accuracy_loss(t1, t2); + debug(D_RRD_STATS, "%s/%s: UNPACK[%ld] = " CALCULATED_NUMBER_FORMAT " FLAGS=0x%08x (original = " CALCULATED_NUMBER_FORMAT ", accuracy loss = " CALCULATED_NUMBER_FORMAT "%%%s)" + , st->id, rd->name + , st->current_entry + , t2 + , get_storage_number_flags(rd->values[st->current_entry]) + , t1 + , accuracy + , (accuracy > ACCURACY_LOSS) ? " **TOO BIG** " : "" + ); + + rd->collected_volume += t1; + rd->stored_volume += t2; + accuracy = accuracy_loss(rd->collected_volume, rd->stored_volume); + debug(D_RRD_STATS, "%s/%s: VOLUME[%ld] = " CALCULATED_NUMBER_FORMAT ", calculated = " CALCULATED_NUMBER_FORMAT ", accuracy loss = " CALCULATED_NUMBER_FORMAT "%%%s" + , st->id, rd->name + , st->current_entry + , rd->stored_volume + , rd->collected_volume + , accuracy + , (accuracy > ACCURACY_LOSS) ? " **TOO BIG** " : "" + ); + + } + } + // reset the storage flags for the next point, if any; + storage_flags = SN_EXISTS; + + st->counter++; + st->current_entry = ((st->current_entry + 1) >= st->entries) ? 0 : st->current_entry + 1; + last_ut = next_ut; + } + + // align next interpolation to last collection point + if(likely(stored_entries || !store_this_entry)) { + st->last_updated.tv_sec = st->last_collected_time.tv_sec; + st->last_updated.tv_usec = st->last_collected_time.tv_usec; + st->last_collected_total = st->collected_total; + } + + for( rd = st->dimensions; likely(rd) ; rd = rd->next ) { + if(unlikely(!rd->updated)) continue; + + if(likely(stored_entries || !store_this_entry)) { + if(unlikely(st->debug)) debug(D_RRD_STATS, "%s/%s: setting last_collected_value (old: " COLLECTED_NUMBER_FORMAT ") to last_collected_value (new: " COLLECTED_NUMBER_FORMAT ")", st->id, rd->name, rd->last_collected_value, rd->collected_value); + rd->last_collected_value = rd->collected_value; + + if(unlikely(st->debug)) debug(D_RRD_STATS, "%s/%s: setting last_calculated_value (old: " CALCULATED_NUMBER_FORMAT ") to last_calculated_value (new: " CALCULATED_NUMBER_FORMAT ")", st->id, rd->name, rd->last_calculated_value, rd->calculated_value); + rd->last_calculated_value = rd->calculated_value; + } + + rd->calculated_value = 0; + rd->collected_value = 0; + rd->updated = 0; + + if(unlikely(st->debug)) debug(D_RRD_STATS, "%s/%s: END " + " last_collected_value = " COLLECTED_NUMBER_FORMAT + " collected_value = " COLLECTED_NUMBER_FORMAT + " last_calculated_value = " CALCULATED_NUMBER_FORMAT + " calculated_value = " CALCULATED_NUMBER_FORMAT + , st->id, rd->name + , rd->last_collected_value + , rd->collected_value + , rd->last_calculated_value + , rd->calculated_value + ); + } + + // ALL DONE ABOUT THE DATA UPDATE + // -------------------------------------------------------------------- + + // find if there are any obsolete dimensions (not updated recently) + if(unlikely(rrd_delete_unupdated_dimensions)) { + + for( rd = st->dimensions; likely(rd) ; rd = rd->next ) + if((rd->last_collected_time.tv_sec + (rrd_delete_unupdated_dimensions * st->update_every)) < st->last_collected_time.tv_sec) + break; + + if(unlikely(rd)) { + // there is dimension to free + // upgrade our read lock to a write lock + pthread_rwlock_unlock(&st->rwlock); + pthread_rwlock_wrlock(&st->rwlock); + + for( rd = st->dimensions, last = NULL ; likely(rd) ; ) { + // remove it only it is not updated in rrd_delete_unupdated_dimensions seconds + + if(unlikely((rd->last_collected_time.tv_sec + (rrd_delete_unupdated_dimensions * st->update_every)) < st->last_collected_time.tv_sec)) { + info("Removing obsolete dimension '%s' (%s) of '%s' (%s).", rd->name, rd->id, st->name, st->id); + + if(unlikely(!last)) { + st->dimensions = rd->next; + rd->next = NULL; + rrddim_free(st, rd); + rd = st->dimensions; + continue; + } + else { + last->next = rd->next; + rd->next = NULL; + rrddim_free(st, rd); + rd = last->next; + continue; + } + } + + last = rd; + rd = rd->next; + } + + if(unlikely(!st->dimensions)) { + info("Disabling chart %s (%s) since it does not have any dimensions", st->name, st->id); + st->enabled = 0; + } + } + } + + pthread_rwlock_unlock(&st->rwlock); + + if(unlikely(pthread_setcancelstate(oldstate, NULL) != 0)) + error("Cannot set pthread cancel state to RESTORE (%d).", oldstate); + + return(st->usec_since_last_update); } @@ -1,10 +1,3 @@ -#include <sys/time.h> -#include <pthread.h> -#include <stdint.h> - -#include "avl.h" -#include "storage_number.h" - #ifndef NETDATA_RRD_H #define NETDATA_RRD_H 1 @@ -20,10 +13,10 @@ extern int rrd_default_history_entries; // set to zero to disable this feature extern int rrd_delete_unupdated_dimensions; -#define RRD_ID_LENGTH_MAX 1024 +#define RRD_ID_LENGTH_MAX 400 -#define RRDSET_MAGIC "NETDATA RRD SET FILE V017" -#define RRDDIMENSION_MAGIC "NETDATA RRD DIMENSION FILE V017" +#define RRDSET_MAGIC "NETDATA RRD SET FILE V018" +#define RRDDIMENSION_MAGIC "NETDATA RRD DIMENSION FILE V018" typedef long long total_number; #define TOTAL_NUMBER_FORMAT "%lld" @@ -35,8 +28,8 @@ typedef long long total_number; #define RRDSET_TYPE_AREA_NAME "area" #define RRDSET_TYPE_STACKED_NAME "stacked" -#define RRDSET_TYPE_LINE 0 -#define RRDSET_TYPE_AREA 1 +#define RRDSET_TYPE_LINE 0 +#define RRDSET_TYPE_AREA 1 #define RRDSET_TYPE_STACKED 2 int rrdset_type_id(const char *name); @@ -63,15 +56,15 @@ extern int rrd_memory_mode_id(const char *name); // ---------------------------------------------------------------------------- // algorithms types -#define RRDDIM_ABSOLUTE_NAME "absolute" -#define RRDDIM_INCREMENTAL_NAME "incremental" -#define RRDDIM_PCENT_OVER_DIFF_TOTAL_NAME "percentage-of-incremental-row" -#define RRDDIM_PCENT_OVER_ROW_TOTAL_NAME "percentage-of-absolute-row" +#define RRDDIM_ABSOLUTE_NAME "absolute" +#define RRDDIM_INCREMENTAL_NAME "incremental" +#define RRDDIM_PCENT_OVER_DIFF_TOTAL_NAME "percentage-of-incremental-row" +#define RRDDIM_PCENT_OVER_ROW_TOTAL_NAME "percentage-of-absolute-row" -#define RRDDIM_ABSOLUTE 0 -#define RRDDIM_INCREMENTAL 1 -#define RRDDIM_PCENT_OVER_DIFF_TOTAL 2 -#define RRDDIM_PCENT_OVER_ROW_TOTAL 3 +#define RRDDIM_ABSOLUTE 0 +#define RRDDIM_INCREMENTAL 1 +#define RRDDIM_PCENT_OVER_DIFF_TOTAL 2 +#define RRDDIM_PCENT_OVER_ROW_TOTAL 3 extern int rrddim_algorithm_id(const char *name); extern const char *rrddim_algorithm_name(int chart_type); @@ -83,80 +76,103 @@ extern const char *rrddim_algorithm_name(int chart_type); #define RRDDIM_FLAG_DONT_DETECT_RESETS_OR_OVERFLOWS 0x00000002 // do not offer RESET or OVERFLOW info to callers // ---------------------------------------------------------------------------- +// RRD CONTEXT + +struct rrdfamily { + avl avl; + + const char *family; + uint32_t hash_family; + + size_t use_count; + + avl_tree_lock variables_root_index; +}; +typedef struct rrdfamily RRDFAMILY; + +// ---------------------------------------------------------------------------- // RRD DIMENSION struct rrddim { - // ------------------------------------------------------------------------ - // binary indexing structures + // ------------------------------------------------------------------------ + // binary indexing structures + + avl avl; // the binary index - this has to be first member! + + // ------------------------------------------------------------------------ + // the dimension definition + + char id[RRD_ID_LENGTH_MAX + 1]; // the id of this dimension (for internal identification) - avl avl; // the binary index - this has to be first member! + const char *name; // the name of this dimension (as presented to user) + // this is a pointer to the config structure + // since the config always has a higher priority + // (the user overwrites the name of the charts) - // ------------------------------------------------------------------------ - // the dimension definition + int algorithm; // the algorithm that is applied to add new collected values + long multiplier; // the multiplier of the collected values + long divisor; // the divider of the collected values - char id[RRD_ID_LENGTH_MAX + 1]; // the id of this dimension (for internal identification) + int mapped; // if set to non zero, this dimension is mapped to a file - const char *name; // the name of this dimension (as presented to user) - // this is a pointer to the config structure - // since the config always has a higher priority - // (the user overwrites the name of the charts) + // ------------------------------------------------------------------------ + // members for temporary data we need for calculations - int algorithm; // the algorithm that is applied to add new collected values - long multiplier; // the multiplier of the collected values - long divisor; // the divider of the collected values + uint32_t hash; // a simple hash of the id, to speed up searching / indexing + // instead of strcmp() every item in the binary index + // we first compare the hashes - int mapped; // if set to non zero, this dimension is mapped to a file + // FIXME + // we need the hash_name too! - // ------------------------------------------------------------------------ - // members for temporary data we need for calculations + uint32_t flags; - uint32_t hash; // a simple hash of the id, to speed up searching / indexing - // instead of strcmp() every item in the binary index - // we first compare the hashes + char cache_filename[FILENAME_MAX+1]; // the filename we load/save from/to this set - uint32_t flags; + unsigned long counter; // the number of times we added values to this rrdim - char cache_filename[FILENAME_MAX+1]; // the filename we load/save from/to this set + int updated; // set to 0 after each calculation, to 1 after each collected value + // we use this to detect that a dimension is not updated - unsigned long counter; // the number of times we added values to this rrdim + struct timeval last_collected_time; // when was this dimension last updated + // this is actual date time we updated the last_collected_value + // THIS IS DIFFERENT FROM THE SAME MEMBER OF RRDSET - int updated; // set to 0 after each calculation, to 1 after each collected value - // we use this to detect that a dimension is not updated + calculated_number calculated_value; // the current calculated value, after applying the algorithm - resets to zero after being used + calculated_number last_calculated_value; // the last calculated value processed - struct timeval last_collected_time; // when was this dimension last updated - // this is actual date time we updated the last_collected_value - // THIS IS DIFFERENT FROM THE SAME MEMBER OF RRDSET + calculated_number last_stored_value; // the last value as stored in the database (after interpolation) - calculated_number calculated_value; // the current calculated value, after applying the algorithm - calculated_number last_calculated_value; // the last calculated value + collected_number collected_value; // the current value, as collected - resets to 0 after being used + collected_number last_collected_value; // the last value that was collected, after being processed - collected_number collected_value; // the current value, as collected - collected_number last_collected_value; // the last value that was collected + // the *_volume members are used to calculate the accuracy of the rounding done by the + // storage number - they are printed to debug.log when debug is enabled for a set. + calculated_number collected_volume; // the sum of all collected values so far + calculated_number stored_volume; // the sum of all stored values so far - // the *_volume members are used to calculate the accuracy of the rounding done by the - // storage number - they are printed to debug.log when debug is enabled for a set. - calculated_number collected_volume; // the sum of all collected values so far - calculated_number stored_volume; // the sum of all stored values so far + struct rrddim *next; // linking of dimensions within the same data set + struct rrdset *rrdset; - struct rrddim *next; // linking of dimensions within the same data set + // ------------------------------------------------------------------------ + // members for checking the data when loading from disk - // ------------------------------------------------------------------------ - // members for checking the data when loading from disk + long entries; // how many entries this dimension has in ram + // this is the same to the entries of the data set + // we set it here, to check the data when we load it from disk. - long entries; // how many entries this dimension has in ram - // this is the same to the entries of the data set - // we set it here, to check the data when we load it from disk. + int update_every; // every how many seconds is this updated - int update_every; // every how many seconds is this updated + unsigned long memsize; // the memory allocated for this dimension - unsigned long memsize; // the memory allocated for this dimension + char magic[sizeof(RRDDIMENSION_MAGIC) + 1]; // a string to be saved, used to identify our data file - char magic[sizeof(RRDDIMENSION_MAGIC) + 1]; // a string to be saved, used to identify our data file + struct rrddimvar *variables; - // ------------------------------------------------------------------------ - // the values stored in this dimension, using our floating point numbers + // ------------------------------------------------------------------------ + // the values stored in this dimension, using our floating point numbers - storage_number values[]; // the array of values - THIS HAS TO BE THE LAST MEMBER + storage_number values[]; // the array of values - THIS HAS TO BE THE LAST MEMBER }; typedef struct rrddim RRDDIM; @@ -165,99 +181,155 @@ typedef struct rrddim RRDDIM; // RRDSET struct rrdset { - // ------------------------------------------------------------------------ - // binary indexing structures + // ------------------------------------------------------------------------ + // binary indexing structures + + avl avl; // the index, with key the id - this has to be first! + avl avlname; // the index, with key the name + + // ------------------------------------------------------------------------ + // the set configuration + + char id[RRD_ID_LENGTH_MAX + 1]; // id of the data set - avl avl; // the index, with key the id - this has to be first! - avl avlname; // the index, with key the name + const char *name; // the name of this dimension (as presented to user) + // this is a pointer to the config structure + // since the config always has a higher priority + // (the user overwrites the name of the charts) - // ------------------------------------------------------------------------ - // the set configuration + char *type; // the type of graph RRD_TYPE_* (a category, for determining graphing options) + char *family; // grouping sets under the same family + char *title; // title shown to user + char *units; // units of measurement - char id[RRD_ID_LENGTH_MAX + 1]; // id of the data set + char *context; // the template of this data set + uint32_t hash_context; - const char *name; // the name of this dimension (as presented to user) - // this is a pointer to the config structure - // since the config always has a higher priority - // (the user overwrites the name of the charts) + int chart_type; - char *type; // the type of graph RRD_TYPE_* (a category, for determining graphing options) - char *family; // grouping sets under the same family - char *context; // the template of this data set - char *title; // title shown to user - char *units; // units of measurement + int update_every; // every how many seconds is this updated? - int chart_type; + long entries; // total number of entries in the data set - int update_every; // every how many seconds is this updated? + long current_entry; // the entry that is currently being updated + // it goes around in a round-robin fashion - long entries; // total number of entries in the data set + int enabled; - long current_entry; // the entry that is currently being updated - // it goes around in a round-robin fashion + int gap_when_lost_iterations_above; // after how many lost iterations a gap should be stored + // netdata will interpolate values for gaps lower than this - int enabled; + long priority; - int gap_when_lost_iterations_above; // after how many lost iterations a gap should be stored - // netdata will interpolate values for gaps lower than this + int isdetail; // if set, the data set should be considered as a detail of another + // (the master data set should be the one that has the same family and is not detail) - long priority; + // ------------------------------------------------------------------------ + // members for temporary data we need for calculations - int isdetail; // if set, the data set should be considered as a detail of another - // (the master data set should be the one that has the same family and is not detail) + int mapped; // if set to 1, this is memory mapped - // ------------------------------------------------------------------------ - // members for temporary data we need for calculations + int debug; - int mapped; // if set to 1, this is memory mapped + char *cache_dir; // the directory to store dimensions + char cache_filename[FILENAME_MAX+1]; // the filename to store this set - int debug; + pthread_rwlock_t rwlock; - char *cache_dir; // the directory to store dimensions - char cache_filename[FILENAME_MAX+1]; // the filename to store this set + unsigned long counter; // the number of times we added values to this rrd + unsigned long counter_done; // the number of times we added values to this rrd - pthread_rwlock_t rwlock; + uint32_t hash; // a simple hash on the id, to speed up searching + // we first compare hashes, and only if the hashes are equal we do string comparisons - unsigned long counter; // the number of times we added values to this rrd - unsigned long counter_done; // the number of times we added values to this rrd + uint32_t hash_name; // a simple hash on the name - uint32_t hash; // a simple hash on the id, to speed up searching - // we first compare hashes, and only if the hashes are equal we do string comparisons + unsigned long long usec_since_last_update; // the time in microseconds since the last collection of data - uint32_t hash_name; // a simple hash on the name + struct timeval last_updated; // when this data set was last updated (updated every time the rrd_stats_done() function) + struct timeval last_collected_time; // when did this data set last collected values - unsigned long long usec_since_last_update; // the time in microseconds since the last collection of data + total_number collected_total; // used internally to calculate percentages + total_number last_collected_total; // used internally to calculate percentages - struct timeval last_updated; // when this data set was last updated (updated every time the rrd_stats_done() function) - struct timeval last_collected_time; // when did this data set last collected values + RRDFAMILY *rrdfamily; + struct rrdhost *rrdhost; - total_number collected_total; // used internally to calculate percentages - total_number last_collected_total; // used internally to calculate percentages + struct rrdset *next; // linking of rrdsets - struct rrdset *next; // linking of rrdsets + // ------------------------------------------------------------------------ + // local variables - // ------------------------------------------------------------------------ - // members for checking the data when loading from disk + calculated_number green; + calculated_number red; - unsigned long memsize; // how much mem we have allocated for this (without dimensions) + avl_tree_lock variables_root_index; + RRDSETVAR *variables; + RRDCALC *alarms; - char magic[sizeof(RRDSET_MAGIC) + 1]; // our magic + // ------------------------------------------------------------------------ + // members for checking the data when loading from disk - // ------------------------------------------------------------------------ - // the dimensions + unsigned long memsize; // how much mem we have allocated for this (without dimensions) + + char magic[sizeof(RRDSET_MAGIC) + 1]; // our magic + + // ------------------------------------------------------------------------ + // the dimensions + + avl_tree_lock dimensions_index; // the root of the dimensions index + RRDDIM *dimensions; // the actual data for every dimension - avl_tree_lock dimensions_index; // the root of the dimensions index - RRDDIM *dimensions; // the actual data for every dimension }; typedef struct rrdset RRDSET; -extern RRDSET *rrdset_root; -extern pthread_rwlock_t rrdset_root_rwlock; +// ---------------------------------------------------------------------------- +// RRD HOST + +struct rrdhost { + avl avl; + + char *hostname; + + RRDSET *rrdset_root; + pthread_rwlock_t rrdset_root_rwlock; + + avl_tree_lock rrdset_root_index; + avl_tree_lock rrdset_root_index_name; + + avl_tree_lock rrdfamily_root_index; + avl_tree_lock variables_root_index; + + // all RRDCALCs are primarily allocated and linked here + // RRDCALCs may be linked to charts at any point + // (charts may or may not exist when these are loaded) + RRDCALC *alarms; + ALARM_LOG health_log; + + RRDCALCTEMPLATE *templates; +}; +typedef struct rrdhost RRDHOST; +extern RRDHOST localhost; + +#ifdef NETDATA_INTERNAL_CHECKS +#define rrdhost_check_wrlock(host) rrdhost_check_wrlock_int(host, __FILE__, __FUNCTION__, __LINE__) +#define rrdhost_check_rdlock(host) rrdhost_check_rdlock_int(host, __FILE__, __FUNCTION__, __LINE__) +#else +#define rrdhost_check_rdlock(host) (void)0 +#define rrdhost_check_wrlock(host) (void)0 +#endif + +extern void rrdhost_check_wrlock_int(RRDHOST *host, const char *file, const char *function, const unsigned long line); +extern void rrdhost_check_rdlock_int(RRDHOST *host, const char *file, const char *function, const unsigned long line); + +extern void rrdhost_rwlock(RRDHOST *host); +extern void rrdhost_rdlock(RRDHOST *host); +extern void rrdhost_unlock(RRDHOST *host); // ---------------------------------------------------------------------------- // RRD SET functions -extern char *rrdset_strncpy_name(char *to, const char *from, int length); +extern char *rrdset_strncpyz_name(char *to, const char *from, size_t length); extern void rrdset_set_name(RRDSET *st, const char *name); extern char *rrdset_cache_dir(const char *id); @@ -265,15 +337,15 @@ extern char *rrdset_cache_dir(const char *id); extern void rrdset_reset(RRDSET *st); extern RRDSET *rrdset_create(const char *type - , const char *id - , const char *name - , const char *family - , const char *context - , const char *title - , const char *units - , long priority - , int update_every - , int chart_type); + , const char *id + , const char *name + , const char *family + , const char *context + , const char *title + , const char *units + , long priority + , int update_every + , int chart_type); extern void rrdset_free_all(void); extern void rrdset_save_all(void); @@ -306,20 +378,20 @@ extern unsigned long long rrdset_done(RRDSET *st); // get the slot of the round robin database, for the given timestamp (t) // it always returns a valid slot, although may not be for the time requested if the time is outside the round robin database #define rrdset_time2slot(st, t) ( \ - ( (time_t)(t) >= rrdset_last_entry_t(st)) ? ( rrdset_last_slot(st) ) : \ - ( ((time_t)(t) <= rrdset_first_entry_t(st)) ? rrdset_first_slot(st) : \ - ( (rrdset_last_slot(st) >= (unsigned long)((rrdset_last_entry_t(st) - (time_t)(t)) / (unsigned long)((st)->update_every)) ) ? \ - (rrdset_last_slot(st) - (unsigned long)((rrdset_last_entry_t(st) - (time_t)(t)) / (unsigned long)((st)->update_every)) ) : \ - (rrdset_last_slot(st) - (unsigned long)((rrdset_last_entry_t(st) - (time_t)(t)) / (unsigned long)((st)->update_every)) + (unsigned long)(st)->entries ) \ - ))) + ( (time_t)(t) >= rrdset_last_entry_t(st)) ? ( rrdset_last_slot(st) ) : \ + ( ((time_t)(t) <= rrdset_first_entry_t(st)) ? rrdset_first_slot(st) : \ + ( (rrdset_last_slot(st) >= (unsigned long)((rrdset_last_entry_t(st) - (time_t)(t)) / (unsigned long)((st)->update_every)) ) ? \ + (rrdset_last_slot(st) - (unsigned long)((rrdset_last_entry_t(st) - (time_t)(t)) / (unsigned long)((st)->update_every)) ) : \ + (rrdset_last_slot(st) - (unsigned long)((rrdset_last_entry_t(st) - (time_t)(t)) / (unsigned long)((st)->update_every)) + (unsigned long)(st)->entries ) \ + ))) // get the timestamp of a specific slot in the round robin database #define rrdset_slot2time(st, slot) ( rrdset_last_entry_t(st) - \ - ((unsigned long)(st)->update_every * ( \ - ( (unsigned long)(slot) > rrdset_last_slot(st)) ? \ - ( (rrdset_last_slot(st) - (unsigned long)(slot) + (unsigned long)(st)->entries) ) : \ - ( (rrdset_last_slot(st) - (unsigned long)(slot)) )) \ - )) + ((unsigned long)(st)->update_every * ( \ + ( (unsigned long)(slot) > rrdset_last_slot(st)) ? \ + ( (rrdset_last_slot(st) - (unsigned long)(slot) + (unsigned long)(st)->entries) ) : \ + ( (rrdset_last_slot(st) - (unsigned long)(slot)) )) \ + )) // ---------------------------------------------------------------------------- // RRD DIMENSION functions diff --git a/src/rrd2json.c b/src/rrd2json.c index e0bd06670..9009a8b1d 100644 --- a/src/rrd2json.c +++ b/src/rrd2json.c @@ -1,222 +1,215 @@ -#ifdef HAVE_CONFIG_H -#include <config.h> -#endif -#include <pthread.h> -#include <stdlib.h> -#include <string.h> -#include <stdint.h> -#include <math.h> - -#include "log.h" #include "common.h" -#include "rrd2json.h" #define HOSTNAME_MAX 1024 char *hostname = "unknown"; void rrd_stats_api_v1_chart(RRDSET *st, BUFFER *wb) { - pthread_rwlock_rdlock(&st->rwlock); - - buffer_sprintf(wb, - "\t\t{\n" - "\t\t\t\"id\": \"%s\",\n" - "\t\t\t\"name\": \"%s\",\n" - "\t\t\t\"type\": \"%s\",\n" - "\t\t\t\"family\": \"%s\",\n" - "\t\t\t\"context\": \"%s\",\n" - "\t\t\t\"title\": \"%s\",\n" - "\t\t\t\"priority\": %ld,\n" - "\t\t\t\"enabled\": %s,\n" - "\t\t\t\"units\": \"%s\",\n" - "\t\t\t\"data_url\": \"/api/v1/data?chart=%s\",\n" - "\t\t\t\"chart_type\": \"%s\",\n" - "\t\t\t\"duration\": %ld,\n" - "\t\t\t\"first_entry\": %lu,\n" - "\t\t\t\"last_entry\": %lu,\n" - "\t\t\t\"update_every\": %d,\n" - "\t\t\t\"dimensions\": {\n" - , st->id - , st->name - , st->type - , st->family - , st->context - , st->title - , st->priority - , st->enabled?"true":"false" - , st->units - , st->name - , rrdset_type_name(st->chart_type) - , st->entries * st->update_every - , rrdset_first_entry_t(st) - , rrdset_last_entry_t(st) - , st->update_every - ); - - unsigned long memory = st->memsize; - - int c = 0; - RRDDIM *rd; - for(rd = st->dimensions; rd ; rd = rd->next) { - if(rd->flags & RRDDIM_FLAG_HIDDEN) continue; - - memory += rd->memsize; - - buffer_sprintf(wb, - "%s" - "\t\t\t\t\"%s\": { \"name\": \"%s\" }" - , c?",\n":"" - , rd->id - , rd->name - ); - - c++; - } - - buffer_sprintf(wb, - "\n\t\t\t}\n" - "\t\t}" - ); - - pthread_rwlock_unlock(&st->rwlock); + pthread_rwlock_rdlock(&st->rwlock); + + buffer_sprintf(wb, + "\t\t{\n" + "\t\t\t\"id\": \"%s\",\n" + "\t\t\t\"name\": \"%s\",\n" + "\t\t\t\"type\": \"%s\",\n" + "\t\t\t\"family\": \"%s\",\n" + "\t\t\t\"context\": \"%s\",\n" + "\t\t\t\"title\": \"%s\",\n" + "\t\t\t\"priority\": %ld,\n" + "\t\t\t\"enabled\": %s,\n" + "\t\t\t\"units\": \"%s\",\n" + "\t\t\t\"data_url\": \"/api/v1/data?chart=%s\",\n" + "\t\t\t\"chart_type\": \"%s\",\n" + "\t\t\t\"duration\": %ld,\n" + "\t\t\t\"first_entry\": %ld,\n" + "\t\t\t\"last_entry\": %ld,\n" + "\t\t\t\"update_every\": %d,\n" + "\t\t\t\"dimensions\": {\n" + , st->id + , st->name + , st->type + , st->family + , st->context + , st->title + , st->priority + , st->enabled?"true":"false" + , st->units + , st->name + , rrdset_type_name(st->chart_type) + , st->entries * st->update_every + , rrdset_first_entry_t(st) + , rrdset_last_entry_t(st) + , st->update_every + ); + + unsigned long memory = st->memsize; + + int c = 0; + RRDDIM *rd; + for(rd = st->dimensions; rd ; rd = rd->next) { + if(rd->flags & RRDDIM_FLAG_HIDDEN) continue; + + memory += rd->memsize; + + buffer_sprintf(wb, + "%s" + "\t\t\t\t\"%s\": { \"name\": \"%s\" }" + , c?",\n":"" + , rd->id + , rd->name + ); + + c++; + } + + buffer_strcat(wb, "\n\t\t\t},\n\t\t\t\"green\": "); + buffer_rrd_value(wb, st->green); + buffer_strcat(wb, ",\n\t\t\t\"red\": "); + buffer_rrd_value(wb, st->red); + + buffer_sprintf(wb, + "\n\t\t}" + ); + + pthread_rwlock_unlock(&st->rwlock); } void rrd_stats_api_v1_charts(BUFFER *wb) { - long c; - RRDSET *st; - - buffer_sprintf(wb, "{\n" - "\t\"hostname\": \"%s\"" - ",\n\t\"update_every\": %d" - ",\n\t\"history\": %d" - ",\n\t\"charts\": {" - , hostname - , rrd_update_every - , rrd_default_history_entries - ); - - pthread_rwlock_rdlock(&rrdset_root_rwlock); - for(st = rrdset_root, c = 0; st ; st = st->next) { - if(st->enabled) { - if(c) buffer_strcat(wb, ","); - buffer_strcat(wb, "\n\t\t\""); - buffer_strcat(wb, st->id); - buffer_strcat(wb, "\": "); - rrd_stats_api_v1_chart(st, wb); - c++; - } - } - pthread_rwlock_unlock(&rrdset_root_rwlock); - - buffer_strcat(wb, "\n\t}\n}\n"); + long c; + RRDSET *st; + + buffer_sprintf(wb, "{\n" + "\t\"hostname\": \"%s\"" + ",\n\t\"update_every\": %d" + ",\n\t\"history\": %d" + ",\n\t\"charts\": {" + , hostname + , rrd_update_every + , rrd_default_history_entries + ); + + pthread_rwlock_rdlock(&localhost.rrdset_root_rwlock); + for(st = localhost.rrdset_root, c = 0; st ; st = st->next) { + if(st->enabled && st->dimensions) { + if(c) buffer_strcat(wb, ","); + buffer_strcat(wb, "\n\t\t\""); + buffer_strcat(wb, st->id); + buffer_strcat(wb, "\": "); + rrd_stats_api_v1_chart(st, wb); + c++; + } + } + pthread_rwlock_unlock(&localhost.rrdset_root_rwlock); + + buffer_strcat(wb, "\n\t}\n}\n"); } unsigned long rrd_stats_one_json(RRDSET *st, char *options, BUFFER *wb) { - time_t now = time(NULL); - - pthread_rwlock_rdlock(&st->rwlock); - - buffer_sprintf(wb, - "\t\t{\n" - "\t\t\t\"id\": \"%s\",\n" - "\t\t\t\"name\": \"%s\",\n" - "\t\t\t\"type\": \"%s\",\n" - "\t\t\t\"family\": \"%s\",\n" - "\t\t\t\"context\": \"%s\",\n" - "\t\t\t\"title\": \"%s\",\n" - "\t\t\t\"priority\": %ld,\n" - "\t\t\t\"enabled\": %d,\n" - "\t\t\t\"units\": \"%s\",\n" - "\t\t\t\"url\": \"/data/%s/%s\",\n" - "\t\t\t\"chart_type\": \"%s\",\n" - "\t\t\t\"counter\": %ld,\n" - "\t\t\t\"entries\": %ld,\n" - "\t\t\t\"first_entry_t\": %lu,\n" - "\t\t\t\"last_entry\": %ld,\n" - "\t\t\t\"last_entry_t\": %lu,\n" - "\t\t\t\"last_entry_secs_ago\": %lu,\n" - "\t\t\t\"update_every\": %d,\n" - "\t\t\t\"isdetail\": %d,\n" - "\t\t\t\"usec_since_last_update\": %llu,\n" - "\t\t\t\"collected_total\": " TOTAL_NUMBER_FORMAT ",\n" - "\t\t\t\"last_collected_total\": " TOTAL_NUMBER_FORMAT ",\n" - "\t\t\t\"dimensions\": [\n" - , st->id - , st->name - , st->type - , st->family - , st->context - , st->title - , st->priority - , st->enabled - , st->units - , st->name, options?options:"" - , rrdset_type_name(st->chart_type) - , st->counter - , st->entries - , rrdset_first_entry_t(st) - , rrdset_last_slot(st) - , rrdset_last_entry_t(st) - , (now < rrdset_last_entry_t(st)) ? (time_t)0 : now - rrdset_last_entry_t(st) - , st->update_every - , st->isdetail - , st->usec_since_last_update - , st->collected_total - , st->last_collected_total - ); - - unsigned long memory = st->memsize; - - RRDDIM *rd; - for(rd = st->dimensions; rd ; rd = rd->next) { - - memory += rd->memsize; - - buffer_sprintf(wb, - "\t\t\t\t{\n" - "\t\t\t\t\t\"id\": \"%s\",\n" - "\t\t\t\t\t\"name\": \"%s\",\n" - "\t\t\t\t\t\"entries\": %ld,\n" - "\t\t\t\t\t\"isHidden\": %d,\n" - "\t\t\t\t\t\"algorithm\": \"%s\",\n" - "\t\t\t\t\t\"multiplier\": %ld,\n" - "\t\t\t\t\t\"divisor\": %ld,\n" - "\t\t\t\t\t\"last_entry_t\": %lu,\n" - "\t\t\t\t\t\"collected_value\": " COLLECTED_NUMBER_FORMAT ",\n" - "\t\t\t\t\t\"calculated_value\": " CALCULATED_NUMBER_FORMAT ",\n" - "\t\t\t\t\t\"last_collected_value\": " COLLECTED_NUMBER_FORMAT ",\n" - "\t\t\t\t\t\"last_calculated_value\": " CALCULATED_NUMBER_FORMAT ",\n" - "\t\t\t\t\t\"memory\": %lu\n" - "\t\t\t\t}%s\n" - , rd->id - , rd->name - , rd->entries - , (rd->flags & RRDDIM_FLAG_HIDDEN)?1:0 - , rrddim_algorithm_name(rd->algorithm) - , rd->multiplier - , rd->divisor - , rd->last_collected_time.tv_sec - , rd->collected_value - , rd->calculated_value - , rd->last_collected_value - , rd->last_calculated_value - , rd->memsize - , rd->next?",":"" - ); - } - - buffer_sprintf(wb, - "\t\t\t],\n" - "\t\t\t\"memory\" : %lu\n" - "\t\t}" - , memory - ); - - pthread_rwlock_unlock(&st->rwlock); - return memory; + time_t now = time(NULL); + + pthread_rwlock_rdlock(&st->rwlock); + + buffer_sprintf(wb, + "\t\t{\n" + "\t\t\t\"id\": \"%s\",\n" + "\t\t\t\"name\": \"%s\",\n" + "\t\t\t\"type\": \"%s\",\n" + "\t\t\t\"family\": \"%s\",\n" + "\t\t\t\"context\": \"%s\",\n" + "\t\t\t\"title\": \"%s\",\n" + "\t\t\t\"priority\": %ld,\n" + "\t\t\t\"enabled\": %d,\n" + "\t\t\t\"units\": \"%s\",\n" + "\t\t\t\"url\": \"/data/%s/%s\",\n" + "\t\t\t\"chart_type\": \"%s\",\n" + "\t\t\t\"counter\": %lu,\n" + "\t\t\t\"entries\": %ld,\n" + "\t\t\t\"first_entry_t\": %ld,\n" + "\t\t\t\"last_entry\": %lu,\n" + "\t\t\t\"last_entry_t\": %ld,\n" + "\t\t\t\"last_entry_secs_ago\": %ld,\n" + "\t\t\t\"update_every\": %d,\n" + "\t\t\t\"isdetail\": %d,\n" + "\t\t\t\"usec_since_last_update\": %llu,\n" + "\t\t\t\"collected_total\": " TOTAL_NUMBER_FORMAT ",\n" + "\t\t\t\"last_collected_total\": " TOTAL_NUMBER_FORMAT ",\n" + "\t\t\t\"dimensions\": [\n" + , st->id + , st->name + , st->type + , st->family + , st->context + , st->title + , st->priority + , st->enabled + , st->units + , st->name, options?options:"" + , rrdset_type_name(st->chart_type) + , st->counter + , st->entries + , rrdset_first_entry_t(st) + , rrdset_last_slot(st) + , rrdset_last_entry_t(st) + , (now < rrdset_last_entry_t(st)) ? (time_t)0 : now - rrdset_last_entry_t(st) + , st->update_every + , st->isdetail + , st->usec_since_last_update + , st->collected_total + , st->last_collected_total + ); + + unsigned long memory = st->memsize; + + RRDDIM *rd; + for(rd = st->dimensions; rd ; rd = rd->next) { + + memory += rd->memsize; + + buffer_sprintf(wb, + "\t\t\t\t{\n" + "\t\t\t\t\t\"id\": \"%s\",\n" + "\t\t\t\t\t\"name\": \"%s\",\n" + "\t\t\t\t\t\"entries\": %ld,\n" + "\t\t\t\t\t\"isHidden\": %d,\n" + "\t\t\t\t\t\"algorithm\": \"%s\",\n" + "\t\t\t\t\t\"multiplier\": %ld,\n" + "\t\t\t\t\t\"divisor\": %ld,\n" + "\t\t\t\t\t\"last_entry_t\": %ld,\n" + "\t\t\t\t\t\"collected_value\": " COLLECTED_NUMBER_FORMAT ",\n" + "\t\t\t\t\t\"calculated_value\": " CALCULATED_NUMBER_FORMAT ",\n" + "\t\t\t\t\t\"last_collected_value\": " COLLECTED_NUMBER_FORMAT ",\n" + "\t\t\t\t\t\"last_calculated_value\": " CALCULATED_NUMBER_FORMAT ",\n" + "\t\t\t\t\t\"memory\": %lu\n" + "\t\t\t\t}%s\n" + , rd->id + , rd->name + , rd->entries + , (rd->flags & RRDDIM_FLAG_HIDDEN)?1:0 + , rrddim_algorithm_name(rd->algorithm) + , rd->multiplier + , rd->divisor + , rd->last_collected_time.tv_sec + , rd->collected_value + , rd->calculated_value + , rd->last_collected_value + , rd->last_calculated_value + , rd->memsize + , rd->next?",":"" + ); + } + + buffer_sprintf(wb, + "\t\t\t],\n" + "\t\t\t\"memory\" : %lu\n" + "\t\t}" + , memory + ); + + pthread_rwlock_unlock(&st->rwlock); + return memory; } #define RRD_GRAPH_JSON_HEADER "{\n\t\"charts\": [\n" @@ -224,40 +217,40 @@ unsigned long rrd_stats_one_json(RRDSET *st, char *options, BUFFER *wb) void rrd_stats_graph_json(RRDSET *st, char *options, BUFFER *wb) { - buffer_strcat(wb, RRD_GRAPH_JSON_HEADER); - rrd_stats_one_json(st, options, wb); - buffer_strcat(wb, RRD_GRAPH_JSON_FOOTER); + buffer_strcat(wb, RRD_GRAPH_JSON_HEADER); + rrd_stats_one_json(st, options, wb); + buffer_strcat(wb, RRD_GRAPH_JSON_FOOTER); } void rrd_stats_all_json(BUFFER *wb) { - unsigned long memory = 0; - long c; - RRDSET *st; - - buffer_strcat(wb, RRD_GRAPH_JSON_HEADER); - - pthread_rwlock_rdlock(&rrdset_root_rwlock); - for(st = rrdset_root, c = 0; st ; st = st->next) { - if(st->enabled) { - if(c) buffer_strcat(wb, ",\n"); - memory += rrd_stats_one_json(st, NULL, wb); - c++; - } - } - pthread_rwlock_unlock(&rrdset_root_rwlock); - - buffer_sprintf(wb, "\n\t],\n" - "\t\"hostname\": \"%s\",\n" - "\t\"update_every\": %d,\n" - "\t\"history\": %d,\n" - "\t\"memory\": %lu\n" - "}\n" - , hostname - , rrd_update_every - , rrd_default_history_entries - , memory - ); + unsigned long memory = 0; + long c; + RRDSET *st; + + buffer_strcat(wb, RRD_GRAPH_JSON_HEADER); + + pthread_rwlock_rdlock(&localhost.rrdset_root_rwlock); + for(st = localhost.rrdset_root, c = 0; st ; st = st->next) { + if(st->enabled && st->dimensions) { + if(c) buffer_strcat(wb, ",\n"); + memory += rrd_stats_one_json(st, NULL, wb); + c++; + } + } + pthread_rwlock_unlock(&localhost.rrdset_root_rwlock); + + buffer_sprintf(wb, "\n\t],\n" + "\t\"hostname\": \"%s\",\n" + "\t\"update_every\": %d,\n" + "\t\"history\": %d,\n" + "\t\"memory\": %lu\n" + "}\n" + , hostname + , rrd_update_every + , rrd_default_history_entries + , memory + ); } @@ -265,42 +258,42 @@ void rrd_stats_all_json(BUFFER *wb) // ---------------------------------------------------------------------------- // RRDR dimension options -#define RRDR_EMPTY 0x01 // the dimension contains / the value is empty (null) -#define RRDR_RESET 0x02 // the dimension contains / the value is reset -#define RRDR_HIDDEN 0x04 // the dimension contains / the value is hidden -#define RRDR_NONZERO 0x08 // the dimension contains / the value is non-zero +#define RRDR_EMPTY 0x01 // the dimension contains / the value is empty (null) +#define RRDR_RESET 0x02 // the dimension contains / the value is reset +#define RRDR_HIDDEN 0x04 // the dimension contains / the value is hidden +#define RRDR_NONZERO 0x08 // the dimension contains / the value is non-zero // RRDR result options #define RRDR_RESULT_OPTION_ABSOLUTE 0x00000001 #define RRDR_RESULT_OPTION_RELATIVE 0x00000002 typedef struct rrdresult { - RRDSET *st; // the chart this result refers to + RRDSET *st; // the chart this result refers to - uint32_t result_options; // RRDR_RESULT_OPTION_* + uint32_t result_options; // RRDR_RESULT_OPTION_* - int d; // the number of dimensions - long n; // the number of values in the arrays - long rows; // the number of rows used + int d; // the number of dimensions + long n; // the number of values in the arrays + long rows; // the number of rows used - uint8_t *od; // the options for the dimensions + uint8_t *od; // the options for the dimensions - time_t *t; // array of n timestamps - calculated_number *v; // array n x d values - uint8_t *o; // array n x d options + time_t *t; // array of n timestamps + calculated_number *v; // array n x d values + uint8_t *o; // array n x d options - long c; // current line ( -1 ~ n ), ( -1 = none, use rrdr_rows() to get number of rows ) + long c; // current line ( -1 ~ n ), ( -1 = none, use rrdr_rows() to get number of rows ) - long group; // how many collected values were grouped for each row - long update_every; // what is the suggested update frequency in seconds + long group; // how many collected values were grouped for each row + int update_every; // what is the suggested update frequency in seconds - calculated_number min; - calculated_number max; + calculated_number min; + calculated_number max; - time_t before; - time_t after; + time_t before; + time_t after; - int has_st_lock; // if st is read locked by us + int has_st_lock; // if st is read locked by us } RRDR; #define rrdr_rows(r) ((r)->rows) @@ -308,346 +301,347 @@ typedef struct rrdresult { /* static void rrdr_dump(RRDR *r) { - long c, i; - RRDDIM *d; - - fprintf(stderr, "\nCHART %s (%s)\n", r->st->id, r->st->name); - - for(c = 0, d = r->st->dimensions; d ;c++, d = d->next) { - fprintf(stderr, "DIMENSION %s (%s), %s%s%s%s\n" - , d->id - , d->name - , (r->od[c] & RRDR_EMPTY)?"EMPTY ":"" - , (r->od[c] & RRDR_RESET)?"RESET ":"" - , (r->od[c] & RRDR_HIDDEN)?"HIDDEN ":"" - , (r->od[c] & RRDR_NONZERO)?"NONZERO ":"" - ); - } - - if(r->rows <= 0) { - fprintf(stderr, "RRDR does not have any values in it.\n"); - return; - } - - fprintf(stderr, "RRDR includes %d values in it:\n", r->rows); - - // for each line in the array - for(i = 0; i < r->rows ;i++) { - calculated_number *cn = &r->v[ i * r->d ]; - uint8_t *co = &r->o[ i * r->d ]; - - // print the id and the timestamp of the line - fprintf(stderr, "%ld %ld ", i + 1, r->t[i]); - - // for each dimension - for(c = 0, d = r->st->dimensions; d ;c++, d = d->next) { - if(unlikely(r->od[c] & RRDR_HIDDEN)) continue; - if(unlikely(!(r->od[c] & RRDR_NONZERO))) continue; - - if(co[c] & RRDR_EMPTY) - fprintf(stderr, "null "); - else - fprintf(stderr, CALCULATED_NUMBER_FORMAT " %s%s%s%s " - , cn[c] - , (co[c] & RRDR_EMPTY)?"E":" " - , (co[c] & RRDR_RESET)?"R":" " - , (co[c] & RRDR_HIDDEN)?"H":" " - , (co[c] & RRDR_NONZERO)?"N":" " - ); - } - - fprintf(stderr, "\n"); - } + long c, i; + RRDDIM *d; + + fprintf(stderr, "\nCHART %s (%s)\n", r->st->id, r->st->name); + + for(c = 0, d = r->st->dimensions; d ;c++, d = d->next) { + fprintf(stderr, "DIMENSION %s (%s), %s%s%s%s\n" + , d->id + , d->name + , (r->od[c] & RRDR_EMPTY)?"EMPTY ":"" + , (r->od[c] & RRDR_RESET)?"RESET ":"" + , (r->od[c] & RRDR_HIDDEN)?"HIDDEN ":"" + , (r->od[c] & RRDR_NONZERO)?"NONZERO ":"" + ); + } + + if(r->rows <= 0) { + fprintf(stderr, "RRDR does not have any values in it.\n"); + return; + } + + fprintf(stderr, "RRDR includes %d values in it:\n", r->rows); + + // for each line in the array + for(i = 0; i < r->rows ;i++) { + calculated_number *cn = &r->v[ i * r->d ]; + uint8_t *co = &r->o[ i * r->d ]; + + // print the id and the timestamp of the line + fprintf(stderr, "%ld %ld ", i + 1, r->t[i]); + + // for each dimension + for(c = 0, d = r->st->dimensions; d ;c++, d = d->next) { + if(unlikely(r->od[c] & RRDR_HIDDEN)) continue; + if(unlikely(!(r->od[c] & RRDR_NONZERO))) continue; + + if(co[c] & RRDR_EMPTY) + fprintf(stderr, "null "); + else + fprintf(stderr, CALCULATED_NUMBER_FORMAT " %s%s%s%s " + , cn[c] + , (co[c] & RRDR_EMPTY)?"E":" " + , (co[c] & RRDR_RESET)?"R":" " + , (co[c] & RRDR_HIDDEN)?"H":" " + , (co[c] & RRDR_NONZERO)?"N":" " + ); + } + + fprintf(stderr, "\n"); + } } */ void rrdr_disable_not_selected_dimensions(RRDR *r, const char *dims) { - char b[strlen(dims) + 1]; - char *o = b, *tok; - strcpy(o, dims); - - long c; - RRDDIM *d; - - // disable all of them - for(c = 0, d = r->st->dimensions; d ;c++, d = d->next) - r->od[c] |= RRDR_HIDDEN; - - while(o && *o && (tok = mystrsep(&o, ", |"))) { - if(!*tok) continue; - - // find it and enable it - for(c = 0, d = r->st->dimensions; d ;c++, d = d->next) { - if(!strcmp(d->name, tok)) { - r->od[c] &= ~RRDR_HIDDEN; - - // since the user needs this dimension - // make it appear as NONZERO, to return it - // even if the dimension has only zeros - r->od[c] |= RRDR_NONZERO; - } - } - } + char b[strlen(dims) + 1]; + char *o = b, *tok; + strcpy(o, dims); + + long c; + RRDDIM *d; + + // disable all of them + for(c = 0, d = r->st->dimensions; d ;c++, d = d->next) + r->od[c] |= RRDR_HIDDEN; + + while(o && *o && (tok = mystrsep(&o, ",|"))) { + if(!*tok) continue; + + uint32_t hash = simple_hash(tok); + + // find it and enable it + for(c = 0, d = r->st->dimensions; d ;c++, d = d->next) { + if(unlikely((hash == d->hash && !strcmp(d->id, tok)) || !strcmp(d->name, tok))) { + r->od[c] &= ~RRDR_HIDDEN; + + // since the user needs this dimension + // make it appear as NONZERO, to return it + // even if the dimension has only zeros + r->od[c] |= RRDR_NONZERO; + } + } + } } void rrdr_buffer_print_format(BUFFER *wb, uint32_t format) { - switch(format) { - case DATASOURCE_JSON: - buffer_strcat(wb, DATASOURCE_FORMAT_JSON); - break; - - case DATASOURCE_DATATABLE_JSON: - buffer_strcat(wb, DATASOURCE_FORMAT_DATATABLE_JSON); - break; - - case DATASOURCE_DATATABLE_JSONP: - buffer_strcat(wb, DATASOURCE_FORMAT_DATATABLE_JSONP); - break; - - case DATASOURCE_JSONP: - buffer_strcat(wb, DATASOURCE_FORMAT_JSONP); - break; - - case DATASOURCE_SSV: - buffer_strcat(wb, DATASOURCE_FORMAT_SSV); - break; - - case DATASOURCE_CSV: - buffer_strcat(wb, DATASOURCE_FORMAT_CSV); - break; - - case DATASOURCE_TSV: - buffer_strcat(wb, DATASOURCE_FORMAT_TSV); - break; - - case DATASOURCE_HTML: - buffer_strcat(wb, DATASOURCE_FORMAT_HTML); - break; - - case DATASOURCE_JS_ARRAY: - buffer_strcat(wb, DATASOURCE_FORMAT_JS_ARRAY); - break; - - case DATASOURCE_SSV_COMMA: - buffer_strcat(wb, DATASOURCE_FORMAT_SSV_COMMA); - break; - - default: - buffer_strcat(wb, "unknown"); - break; - } + switch(format) { + case DATASOURCE_JSON: + buffer_strcat(wb, DATASOURCE_FORMAT_JSON); + break; + + case DATASOURCE_DATATABLE_JSON: + buffer_strcat(wb, DATASOURCE_FORMAT_DATATABLE_JSON); + break; + + case DATASOURCE_DATATABLE_JSONP: + buffer_strcat(wb, DATASOURCE_FORMAT_DATATABLE_JSONP); + break; + + case DATASOURCE_JSONP: + buffer_strcat(wb, DATASOURCE_FORMAT_JSONP); + break; + + case DATASOURCE_SSV: + buffer_strcat(wb, DATASOURCE_FORMAT_SSV); + break; + + case DATASOURCE_CSV: + buffer_strcat(wb, DATASOURCE_FORMAT_CSV); + break; + + case DATASOURCE_TSV: + buffer_strcat(wb, DATASOURCE_FORMAT_TSV); + break; + + case DATASOURCE_HTML: + buffer_strcat(wb, DATASOURCE_FORMAT_HTML); + break; + + case DATASOURCE_JS_ARRAY: + buffer_strcat(wb, DATASOURCE_FORMAT_JS_ARRAY); + break; + + case DATASOURCE_SSV_COMMA: + buffer_strcat(wb, DATASOURCE_FORMAT_SSV_COMMA); + break; + + default: + buffer_strcat(wb, "unknown"); + break; + } } uint32_t rrdr_check_options(RRDR *r, uint32_t options, const char *dims) { - if(options & RRDR_OPTION_NONZERO) { - long i; - - if(dims && *dims) { - // the caller wants specific dimensions - // disable NONZERO option - // to make sure we don't accidentally prevent - // the specific dimensions from being returned - i = 0; - } - else { - // find how many dimensions are not zero - long c; - RRDDIM *rd; - for(c = 0, i = 0, rd = r->st->dimensions; rd && c < r->d ; c++, rd = rd->next) { - if(unlikely(r->od[c] & RRDR_HIDDEN)) continue; - if(unlikely(!(r->od[c] & RRDR_NONZERO))) continue; - i++; - } - } - - // if with nonzero we get i = 0 (no dimensions will be returned) - // disable nonzero to show all dimensions - if(!i) options &= ~RRDR_OPTION_NONZERO; - } - - return options; + if(options & RRDR_OPTION_NONZERO) { + long i; + + if(dims && *dims) { + // the caller wants specific dimensions + // disable NONZERO option + // to make sure we don't accidentally prevent + // the specific dimensions from being returned + i = 0; + } + else { + // find how many dimensions are not zero + long c; + RRDDIM *rd; + for(c = 0, i = 0, rd = r->st->dimensions; rd && c < r->d ; c++, rd = rd->next) { + if(unlikely(r->od[c] & RRDR_HIDDEN)) continue; + if(unlikely(!(r->od[c] & RRDR_NONZERO))) continue; + i++; + } + } + + // if with nonzero we get i = 0 (no dimensions will be returned) + // disable nonzero to show all dimensions + if(!i) options &= ~RRDR_OPTION_NONZERO; + } + + return options; } void rrdr_json_wrapper_begin(RRDR *r, BUFFER *wb, uint32_t format, uint32_t options, int string_value) { - long rows = rrdr_rows(r); - long c, i; - RRDDIM *rd; - - //info("JSONWRAPPER(): %s: BEGIN", r->st->id); - char kq[2] = "", // key quote - sq[2] = ""; // string quote - - if( options & RRDR_OPTION_GOOGLE_JSON ) { - kq[0] = '\0'; - sq[0] = '\''; - } - else { - kq[0] = '"'; - sq[0] = '"'; - } - - buffer_sprintf(wb, "{\n" - " %sapi%s: 1,\n" - " %sid%s: %s%s%s,\n" - " %sname%s: %s%s%s,\n" - " %sview_update_every%s: %d,\n" - " %supdate_every%s: %d,\n" - " %sfirst_entry%s: %u,\n" - " %slast_entry%s: %u,\n" - " %sbefore%s: %u,\n" - " %safter%s: %u,\n" - " %sdimension_names%s: [" - , kq, kq - , kq, kq, sq, r->st->id, sq - , kq, kq, sq, r->st->name, sq - , kq, kq, r->update_every - , kq, kq, r->st->update_every - , kq, kq, rrdset_first_entry_t(r->st) - , kq, kq, rrdset_last_entry_t(r->st) - , kq, kq, r->before - , kq, kq, r->after - , kq, kq); - - for(c = 0, i = 0, rd = r->st->dimensions; rd && c < r->d ;c++, rd = rd->next) { - if(unlikely(r->od[c] & RRDR_HIDDEN)) continue; - if(unlikely((options & RRDR_OPTION_NONZERO) && !(r->od[c] & RRDR_NONZERO))) continue; - - if(i) buffer_strcat(wb, ", "); - buffer_strcat(wb, sq); - buffer_strcat(wb, rd->name); - buffer_strcat(wb, sq); - i++; - } - if(!i) { + long rows = rrdr_rows(r); + long c, i; + RRDDIM *rd; + + //info("JSONWRAPPER(): %s: BEGIN", r->st->id); + char kq[2] = "", // key quote + sq[2] = ""; // string quote + + if( options & RRDR_OPTION_GOOGLE_JSON ) { + kq[0] = '\0'; + sq[0] = '\''; + } + else { + kq[0] = '"'; + sq[0] = '"'; + } + + buffer_sprintf(wb, "{\n" + " %sapi%s: 1,\n" + " %sid%s: %s%s%s,\n" + " %sname%s: %s%s%s,\n" + " %sview_update_every%s: %d,\n" + " %supdate_every%s: %d,\n" + " %sfirst_entry%s: %u,\n" + " %slast_entry%s: %u,\n" + " %sbefore%s: %u,\n" + " %safter%s: %u,\n" + " %sdimension_names%s: [" + , kq, kq + , kq, kq, sq, r->st->id, sq + , kq, kq, sq, r->st->name, sq + , kq, kq, r->update_every + , kq, kq, r->st->update_every + , kq, kq, (uint32_t)rrdset_first_entry_t(r->st) + , kq, kq, (uint32_t)rrdset_last_entry_t(r->st) + , kq, kq, (uint32_t)r->before + , kq, kq, (uint32_t)r->after + , kq, kq); + + for(c = 0, i = 0, rd = r->st->dimensions; rd && c < r->d ;c++, rd = rd->next) { + if(unlikely(r->od[c] & RRDR_HIDDEN)) continue; + if(unlikely((options & RRDR_OPTION_NONZERO) && !(r->od[c] & RRDR_NONZERO))) continue; + + if(i) buffer_strcat(wb, ", "); + buffer_strcat(wb, sq); + buffer_strcat(wb, rd->name); + buffer_strcat(wb, sq); + i++; + } + if(!i) { #ifdef NETDATA_INTERNAL_CHECKS - error("RRDR is empty for %s (RRDR has %d dimensions, options is 0x%08x)", r->st->id, r->d, options); + error("RRDR is empty for %s (RRDR has %d dimensions, options is 0x%08x)", r->st->id, r->d, options); #endif - rows = 0; - buffer_strcat(wb, sq); - buffer_strcat(wb, "no data"); - buffer_strcat(wb, sq); - } - - buffer_sprintf(wb, "],\n" - " %sdimension_ids%s: [" - , kq, kq); - - for(c = 0, i = 0, rd = r->st->dimensions; rd && c < r->d ;c++, rd = rd->next) { - if(unlikely(r->od[c] & RRDR_HIDDEN)) continue; - if(unlikely((options & RRDR_OPTION_NONZERO) && !(r->od[c] & RRDR_NONZERO))) continue; - - if(i) buffer_strcat(wb, ", "); - buffer_strcat(wb, sq); - buffer_strcat(wb, rd->id); - buffer_strcat(wb, sq); - i++; - } - if(!i) { - rows = 0; - buffer_strcat(wb, sq); - buffer_strcat(wb, "no data"); - buffer_strcat(wb, sq); - } - - buffer_sprintf(wb, "],\n" - " %slatest_values%s: [" - , kq, kq); - - for(c = 0, i = 0, rd = r->st->dimensions; rd && c < r->d ;c++, rd = rd->next) { - if(unlikely(r->od[c] & RRDR_HIDDEN)) continue; - if(unlikely((options & RRDR_OPTION_NONZERO) && !(r->od[c] & RRDR_NONZERO))) continue; - - if(i) buffer_strcat(wb, ", "); - i++; - - storage_number n = rd->values[rrdset_last_slot(r->st)]; - - if(!does_storage_number_exist(n)) - buffer_strcat(wb, "null"); - else - buffer_rrd_value(wb, unpack_storage_number(n)); - } - if(!i) { - rows = 0; - buffer_strcat(wb, "null"); - } - - buffer_sprintf(wb, "],\n" - " %sview_latest_values%s: [" - , kq, kq); - - i = 0; - if(rows) { - for(c = 0, i = 0, rd = r->st->dimensions; rd && c < r->d ;c++, rd = rd->next) { - if(unlikely(r->od[c] & RRDR_HIDDEN)) continue; - if(unlikely((options & RRDR_OPTION_NONZERO) && !(r->od[c] & RRDR_NONZERO))) continue; - - if(i) buffer_strcat(wb, ", "); - i++; - - calculated_number *cn = &r->v[ (0) * r->d ]; - uint8_t *co = &r->o[ (0) * r->d ]; - - if(co[c] & RRDR_EMPTY) - buffer_strcat(wb, "null"); - else - buffer_rrd_value(wb, cn[c]); - } - } - if(!i) { - rows = 0; - buffer_strcat(wb, "null"); - } - - buffer_sprintf(wb, "],\n" - " %sdimensions%s: %d,\n" - " %spoints%s: %d,\n" - " %sformat%s: %s" - , kq, kq, i - , kq, kq, rows - , kq, kq, sq - ); - - rrdr_buffer_print_format(wb, format); - - buffer_sprintf(wb, "%s,\n" - " %sresult%s: " - , sq - , kq, kq - ); - - if(string_value) buffer_strcat(wb, sq); - //info("JSONWRAPPER(): %s: END", r->st->id); + rows = 0; + buffer_strcat(wb, sq); + buffer_strcat(wb, "no data"); + buffer_strcat(wb, sq); + } + + buffer_sprintf(wb, "],\n" + " %sdimension_ids%s: [" + , kq, kq); + + for(c = 0, i = 0, rd = r->st->dimensions; rd && c < r->d ;c++, rd = rd->next) { + if(unlikely(r->od[c] & RRDR_HIDDEN)) continue; + if(unlikely((options & RRDR_OPTION_NONZERO) && !(r->od[c] & RRDR_NONZERO))) continue; + + if(i) buffer_strcat(wb, ", "); + buffer_strcat(wb, sq); + buffer_strcat(wb, rd->id); + buffer_strcat(wb, sq); + i++; + } + if(!i) { + rows = 0; + buffer_strcat(wb, sq); + buffer_strcat(wb, "no data"); + buffer_strcat(wb, sq); + } + + buffer_sprintf(wb, "],\n" + " %slatest_values%s: [" + , kq, kq); + + for(c = 0, i = 0, rd = r->st->dimensions; rd && c < r->d ;c++, rd = rd->next) { + if(unlikely(r->od[c] & RRDR_HIDDEN)) continue; + if(unlikely((options & RRDR_OPTION_NONZERO) && !(r->od[c] & RRDR_NONZERO))) continue; + + if(i) buffer_strcat(wb, ", "); + i++; + + storage_number n = rd->values[rrdset_last_slot(r->st)]; + + if(!does_storage_number_exist(n)) + buffer_strcat(wb, "null"); + else + buffer_rrd_value(wb, unpack_storage_number(n)); + } + if(!i) { + rows = 0; + buffer_strcat(wb, "null"); + } + + buffer_sprintf(wb, "],\n" + " %sview_latest_values%s: [" + , kq, kq); + + i = 0; + if(rows) { + for(c = 0, i = 0, rd = r->st->dimensions; rd && c < r->d ;c++, rd = rd->next) { + if(unlikely(r->od[c] & RRDR_HIDDEN)) continue; + if(unlikely((options & RRDR_OPTION_NONZERO) && !(r->od[c] & RRDR_NONZERO))) continue; + + if(i) buffer_strcat(wb, ", "); + i++; + + calculated_number *cn = &r->v[ (0) * r->d ]; + uint8_t *co = &r->o[ (0) * r->d ]; + + if(co[c] & RRDR_EMPTY) + buffer_strcat(wb, "null"); + else + buffer_rrd_value(wb, cn[c]); + } + } + if(!i) { + rows = 0; + buffer_strcat(wb, "null"); + } + + buffer_sprintf(wb, "],\n" + " %sdimensions%s: %ld,\n" + " %spoints%s: %ld,\n" + " %sformat%s: %s" + , kq, kq, i + , kq, kq, rows + , kq, kq, sq + ); + + rrdr_buffer_print_format(wb, format); + + buffer_sprintf(wb, "%s,\n" + " %sresult%s: " + , sq + , kq, kq + ); + + if(string_value) buffer_strcat(wb, sq); + //info("JSONWRAPPER(): %s: END", r->st->id); } void rrdr_json_wrapper_end(RRDR *r, BUFFER *wb, uint32_t format, uint32_t options, int string_value) { - if(r) {;} - if(format) {;} - - char kq[2] = "", // key quote - sq[2] = ""; // string quote - - if( options & RRDR_OPTION_GOOGLE_JSON ) { - kq[0] = '\0'; - sq[0] = '\''; - } - else { - kq[0] = '"'; - sq[0] = '"'; - } - - if(string_value) buffer_strcat(wb, sq); - - buffer_sprintf(wb, ",\n %smin%s: ", kq, kq); - buffer_rrd_value(wb, r->min); - buffer_sprintf(wb, ",\n %smax%s: ", kq, kq); - buffer_rrd_value(wb, r->max); - buffer_strcat(wb, "\n}\n"); + (void)format; + + char kq[2] = "", // key quote + sq[2] = ""; // string quote + + if( options & RRDR_OPTION_GOOGLE_JSON ) { + kq[0] = '\0'; + sq[0] = '\''; + } + else { + kq[0] = '"'; + sq[0] = '"'; + } + + if(string_value) buffer_strcat(wb, sq); + + buffer_sprintf(wb, ",\n %smin%s: ", kq, kq); + buffer_rrd_value(wb, r->min); + buffer_sprintf(wb, ",\n %smax%s: ", kq, kq); + buffer_rrd_value(wb, r->max); + buffer_strcat(wb, "\n}\n"); } #define JSON_DATES_JS 1 @@ -655,1356 +649,1435 @@ void rrdr_json_wrapper_end(RRDR *r, BUFFER *wb, uint32_t format, uint32_t option static void rrdr2json(RRDR *r, BUFFER *wb, uint32_t options, int datatable) { - //info("RRD2JSON(): %s: BEGIN", r->st->id); - int row_annotations = 0, dates, dates_with_new = 0; - char kq[2] = "", // key quote - sq[2] = "", // string quote - pre_label[101] = "", // before each label - post_label[101] = "", // after each label - pre_date[101] = "", // the beginning of line, to the date - post_date[101] = "", // closing the date - pre_value[101] = "", // before each value - post_value[101] = "", // after each value - post_line[101] = "", // at the end of each row - normal_annotation[201] = "", // default row annotation - overflow_annotation[201] = "", // overflow row annotation - data_begin[101] = "", // between labels and values - finish[101] = ""; // at the end of everything - - if(datatable) { - dates = JSON_DATES_JS; - if( options & RRDR_OPTION_GOOGLE_JSON ) { - kq[0] = '\0'; - sq[0] = '\''; - } - else { - kq[0] = '"'; - sq[0] = '"'; - } - row_annotations = 1; - snprintfz(pre_date, 100, " {%sc%s:[{%sv%s:%s", kq, kq, kq, kq, sq); - snprintfz(post_date, 100, "%s}", sq); - snprintfz(pre_label, 100, ",\n {%sid%s:%s%s,%slabel%s:%s", kq, kq, sq, sq, kq, kq, sq); - snprintfz(post_label, 100, "%s,%spattern%s:%s%s,%stype%s:%snumber%s}", sq, kq, kq, sq, sq, kq, kq, sq, sq); - snprintfz(pre_value, 100, ",{%sv%s:", kq, kq); - strcpy(post_value, "}"); - strcpy(post_line, "]}"); - snprintfz(data_begin, 100, "\n ],\n %srows%s:\n [\n", kq, kq); - strcpy(finish, "\n ]\n}"); - - snprintfz(overflow_annotation, 200, ",{%sv%s:%sRESET OR OVERFLOW%s},{%sv%s:%sThe counters have been wrapped.%s}", kq, kq, sq, sq, kq, kq, sq, sq); - snprintfz(normal_annotation, 200, ",{%sv%s:null},{%sv%s:null}", kq, kq, kq, kq); - - buffer_sprintf(wb, "{\n %scols%s:\n [\n", kq, kq, kq, kq); - buffer_sprintf(wb, " {%sid%s:%s%s,%slabel%s:%stime%s,%spattern%s:%s%s,%stype%s:%sdatetime%s},\n", kq, kq, sq, sq, kq, kq, sq, sq, kq, kq, sq, sq, kq, kq, sq, sq); - buffer_sprintf(wb, " {%sid%s:%s%s,%slabel%s:%s%s,%spattern%s:%s%s,%stype%s:%sstring%s,%sp%s:{%srole%s:%sannotation%s}},\n", kq, kq, sq, sq, kq, kq, sq, sq, kq, kq, sq, sq, kq, kq, sq, sq, kq, kq, kq, kq, sq, sq); - buffer_sprintf(wb, " {%sid%s:%s%s,%slabel%s:%s%s,%spattern%s:%s%s,%stype%s:%sstring%s,%sp%s:{%srole%s:%sannotationText%s}}", kq, kq, sq, sq, kq, kq, sq, sq, kq, kq, sq, sq, kq, kq, sq, sq, kq, kq, kq, kq, sq, sq); - - // remove the valueobjects flag - // google wants its own keys - if(options & RRDR_OPTION_OBJECTSROWS) - options &= ~RRDR_OPTION_OBJECTSROWS; - } - else { - kq[0] = '"'; - sq[0] = '"'; - if((options & RRDR_OPTION_SECONDS) || (options & RRDR_OPTION_MILLISECONDS)) { - dates = JSON_DATES_TIMESTAMP; - dates_with_new = 0; - } - else { - dates = JSON_DATES_JS; - dates_with_new = 1; - } - if( options & RRDR_OPTION_OBJECTSROWS ) - strcpy(pre_date, " { "); - else - strcpy(pre_date, " [ "); - strcpy(pre_label, ", \""); - strcpy(post_label, "\""); - strcpy(pre_value, ", "); - if( options & RRDR_OPTION_OBJECTSROWS ) - strcpy(post_line, "}"); - else - strcpy(post_line, "]"); - snprintfz(data_begin, 100, "],\n %sdata%s:\n [\n", kq, kq); - strcpy(finish, "\n ]\n}"); - - buffer_sprintf(wb, "{\n %slabels%s: [", kq, kq); - buffer_sprintf(wb, "%stime%s", sq, sq); - } - - // ------------------------------------------------------------------------- - // print the JSON header - - long c, i; - RRDDIM *rd; - - // print the header lines - for(c = 0, i = 0, rd = r->st->dimensions; rd && c < r->d ;c++, rd = rd->next) { - if(unlikely(r->od[c] & RRDR_HIDDEN)) continue; - if(unlikely((options & RRDR_OPTION_NONZERO) && !(r->od[c] & RRDR_NONZERO))) continue; - - buffer_strcat(wb, pre_label); - buffer_strcat(wb, rd->name); - buffer_strcat(wb, post_label); - i++; - } - if(!i) { - buffer_strcat(wb, pre_label); - buffer_strcat(wb, "no data"); - buffer_strcat(wb, post_label); - } - - // print the begin of row data - buffer_strcat(wb, data_begin); - - // if all dimensions are hidden, print a null - if(!i) { - buffer_strcat(wb, finish); - return; - } - - long start = 0, end = rrdr_rows(r), step = 1; - if((options & RRDR_OPTION_REVERSED)) { - start = rrdr_rows(r) - 1; - end = -1; - step = -1; - } - - // for each line in the array - calculated_number total = 1; - for(i = start; i != end ;i += step) { - calculated_number *cn = &r->v[ i * r->d ]; - uint8_t *co = &r->o[ i * r->d ]; - - time_t now = r->t[i]; - - if(dates == JSON_DATES_JS) { - // generate the local date time - struct tm tmbuf, *tm = localtime_r(&now, &tmbuf); - if(!tm) { error("localtime_r() failed."); continue; } - - if(likely(i != start)) buffer_strcat(wb, ",\n"); - buffer_strcat(wb, pre_date); - - if( options & RRDR_OPTION_OBJECTSROWS ) - buffer_sprintf(wb, "%stime%s: ", kq, kq); - - if(dates_with_new) - buffer_strcat(wb, "new "); - - buffer_jsdate(wb, tm->tm_year + 1900, tm->tm_mon, tm->tm_mday, tm->tm_hour, tm->tm_min, tm->tm_sec); - - buffer_strcat(wb, post_date); - - if(row_annotations) { - // google supports one annotation per row - int annotation_found = 0; - for(c = 0, rd = r->st->dimensions; rd ;c++, rd = rd->next) { - if(co[c] & RRDR_RESET) { - buffer_strcat(wb, overflow_annotation); - annotation_found = 1; - break; - } - } - if(!annotation_found) - buffer_strcat(wb, normal_annotation); - } - } - else { - // print the timestamp of the line - if(likely(i != start)) buffer_strcat(wb, ",\n"); - buffer_strcat(wb, pre_date); - - if( options & RRDR_OPTION_OBJECTSROWS ) - buffer_sprintf(wb, "%stime%s: ", kq, kq); - - buffer_rrd_value(wb, (calculated_number)r->t[i]); - // in ms - if(options & RRDR_OPTION_MILLISECONDS) buffer_strcat(wb, "000"); - - buffer_strcat(wb, post_date); - } - - if(unlikely(options & RRDR_OPTION_PERCENTAGE)) { - total = 0; - for(c = 0, rd = r->st->dimensions; rd && c < r->d ;c++, rd = rd->next) { - calculated_number n = cn[c]; - - if(likely((options & RRDR_OPTION_ABSOLUTE) && n < 0)) - n = -n; - - total += n; - } - // prevent a division by zero - if(total == 0) total = 1; - } - - // for each dimension - for(c = 0, rd = r->st->dimensions; rd && c < r->d ;c++, rd = rd->next) { - if(unlikely(r->od[c] & RRDR_HIDDEN)) continue; - if(unlikely((options & RRDR_OPTION_NONZERO) && !(r->od[c] & RRDR_NONZERO))) continue; - - calculated_number n = cn[c]; - - buffer_strcat(wb, pre_value); - - if( options & RRDR_OPTION_OBJECTSROWS ) - buffer_sprintf(wb, "%s%s%s: ", kq, rd->name, kq); - - if(co[c] & RRDR_EMPTY) { - if(options & RRDR_OPTION_NULL2ZERO) - buffer_strcat(wb, "0"); - else - buffer_strcat(wb, "null"); - } - else { - if(unlikely((options & RRDR_OPTION_ABSOLUTE) && n < 0)) - n = -n; - - if(unlikely(options & RRDR_OPTION_PERCENTAGE)) - n = n * 100 / total; - - buffer_rrd_value(wb, n); - } - - buffer_strcat(wb, post_value); - } - - buffer_strcat(wb, post_line); - } - - buffer_strcat(wb, finish); - //info("RRD2JSON(): %s: END", r->st->id); + //info("RRD2JSON(): %s: BEGIN", r->st->id); + int row_annotations = 0, dates, dates_with_new = 0; + char kq[2] = "", // key quote + sq[2] = "", // string quote + pre_label[101] = "", // before each label + post_label[101] = "", // after each label + pre_date[101] = "", // the beginning of line, to the date + post_date[101] = "", // closing the date + pre_value[101] = "", // before each value + post_value[101] = "", // after each value + post_line[101] = "", // at the end of each row + normal_annotation[201] = "", // default row annotation + overflow_annotation[201] = "", // overflow row annotation + data_begin[101] = "", // between labels and values + finish[101] = ""; // at the end of everything + + if(datatable) { + dates = JSON_DATES_JS; + if( options & RRDR_OPTION_GOOGLE_JSON ) { + kq[0] = '\0'; + sq[0] = '\''; + } + else { + kq[0] = '"'; + sq[0] = '"'; + } + row_annotations = 1; + snprintfz(pre_date, 100, " {%sc%s:[{%sv%s:%s", kq, kq, kq, kq, sq); + snprintfz(post_date, 100, "%s}", sq); + snprintfz(pre_label, 100, ",\n {%sid%s:%s%s,%slabel%s:%s", kq, kq, sq, sq, kq, kq, sq); + snprintfz(post_label, 100, "%s,%spattern%s:%s%s,%stype%s:%snumber%s}", sq, kq, kq, sq, sq, kq, kq, sq, sq); + snprintfz(pre_value, 100, ",{%sv%s:", kq, kq); + strcpy(post_value, "}"); + strcpy(post_line, "]}"); + snprintfz(data_begin, 100, "\n ],\n %srows%s:\n [\n", kq, kq); + strcpy(finish, "\n ]\n}"); + + snprintfz(overflow_annotation, 200, ",{%sv%s:%sRESET OR OVERFLOW%s},{%sv%s:%sThe counters have been wrapped.%s}", kq, kq, sq, sq, kq, kq, sq, sq); + snprintfz(normal_annotation, 200, ",{%sv%s:null},{%sv%s:null}", kq, kq, kq, kq); + + buffer_sprintf(wb, "{\n %scols%s:\n [\n", kq, kq); + buffer_sprintf(wb, " {%sid%s:%s%s,%slabel%s:%stime%s,%spattern%s:%s%s,%stype%s:%sdatetime%s},\n", kq, kq, sq, sq, kq, kq, sq, sq, kq, kq, sq, sq, kq, kq, sq, sq); + buffer_sprintf(wb, " {%sid%s:%s%s,%slabel%s:%s%s,%spattern%s:%s%s,%stype%s:%sstring%s,%sp%s:{%srole%s:%sannotation%s}},\n", kq, kq, sq, sq, kq, kq, sq, sq, kq, kq, sq, sq, kq, kq, sq, sq, kq, kq, kq, kq, sq, sq); + buffer_sprintf(wb, " {%sid%s:%s%s,%slabel%s:%s%s,%spattern%s:%s%s,%stype%s:%sstring%s,%sp%s:{%srole%s:%sannotationText%s}}", kq, kq, sq, sq, kq, kq, sq, sq, kq, kq, sq, sq, kq, kq, sq, sq, kq, kq, kq, kq, sq, sq); + + // remove the valueobjects flag + // google wants its own keys + if(options & RRDR_OPTION_OBJECTSROWS) + options &= ~RRDR_OPTION_OBJECTSROWS; + } + else { + kq[0] = '"'; + sq[0] = '"'; + if((options & RRDR_OPTION_SECONDS) || (options & RRDR_OPTION_MILLISECONDS)) { + dates = JSON_DATES_TIMESTAMP; + dates_with_new = 0; + } + else { + dates = JSON_DATES_JS; + dates_with_new = 1; + } + if( options & RRDR_OPTION_OBJECTSROWS ) + strcpy(pre_date, " { "); + else + strcpy(pre_date, " [ "); + strcpy(pre_label, ", \""); + strcpy(post_label, "\""); + strcpy(pre_value, ", "); + if( options & RRDR_OPTION_OBJECTSROWS ) + strcpy(post_line, "}"); + else + strcpy(post_line, "]"); + snprintfz(data_begin, 100, "],\n %sdata%s:\n [\n", kq, kq); + strcpy(finish, "\n ]\n}"); + + buffer_sprintf(wb, "{\n %slabels%s: [", kq, kq); + buffer_sprintf(wb, "%stime%s", sq, sq); + } + + // ------------------------------------------------------------------------- + // print the JSON header + + long c, i; + RRDDIM *rd; + + // print the header lines + for(c = 0, i = 0, rd = r->st->dimensions; rd && c < r->d ;c++, rd = rd->next) { + if(unlikely(r->od[c] & RRDR_HIDDEN)) continue; + if(unlikely((options & RRDR_OPTION_NONZERO) && !(r->od[c] & RRDR_NONZERO))) continue; + + buffer_strcat(wb, pre_label); + buffer_strcat(wb, rd->name); + buffer_strcat(wb, post_label); + i++; + } + if(!i) { + buffer_strcat(wb, pre_label); + buffer_strcat(wb, "no data"); + buffer_strcat(wb, post_label); + } + + // print the begin of row data + buffer_strcat(wb, data_begin); + + // if all dimensions are hidden, print a null + if(!i) { + buffer_strcat(wb, finish); + return; + } + + long start = 0, end = rrdr_rows(r), step = 1; + if((options & RRDR_OPTION_REVERSED)) { + start = rrdr_rows(r) - 1; + end = -1; + step = -1; + } + + // for each line in the array + calculated_number total = 1; + for(i = start; i != end ;i += step) { + calculated_number *cn = &r->v[ i * r->d ]; + uint8_t *co = &r->o[ i * r->d ]; + + time_t now = r->t[i]; + + if(dates == JSON_DATES_JS) { + // generate the local date time + struct tm tmbuf, *tm = localtime_r(&now, &tmbuf); + if(!tm) { error("localtime_r() failed."); continue; } + + if(likely(i != start)) buffer_strcat(wb, ",\n"); + buffer_strcat(wb, pre_date); + + if( options & RRDR_OPTION_OBJECTSROWS ) + buffer_sprintf(wb, "%stime%s: ", kq, kq); + + if(dates_with_new) + buffer_strcat(wb, "new "); + + buffer_jsdate(wb, tm->tm_year + 1900, tm->tm_mon, tm->tm_mday, tm->tm_hour, tm->tm_min, tm->tm_sec); + + buffer_strcat(wb, post_date); + + if(row_annotations) { + // google supports one annotation per row + int annotation_found = 0; + for(c = 0, rd = r->st->dimensions; rd ;c++, rd = rd->next) { + if(co[c] & RRDR_RESET) { + buffer_strcat(wb, overflow_annotation); + annotation_found = 1; + break; + } + } + if(!annotation_found) + buffer_strcat(wb, normal_annotation); + } + } + else { + // print the timestamp of the line + if(likely(i != start)) buffer_strcat(wb, ",\n"); + buffer_strcat(wb, pre_date); + + if( options & RRDR_OPTION_OBJECTSROWS ) + buffer_sprintf(wb, "%stime%s: ", kq, kq); + + buffer_rrd_value(wb, (calculated_number)r->t[i]); + // in ms + if(options & RRDR_OPTION_MILLISECONDS) buffer_strcat(wb, "000"); + + buffer_strcat(wb, post_date); + } + + if(unlikely(options & RRDR_OPTION_PERCENTAGE)) { + total = 0; + for(c = 0, rd = r->st->dimensions; rd && c < r->d ;c++, rd = rd->next) { + calculated_number n = cn[c]; + + if(likely((options & RRDR_OPTION_ABSOLUTE) && n < 0)) + n = -n; + + total += n; + } + // prevent a division by zero + if(total == 0) total = 1; + } + + // for each dimension + for(c = 0, rd = r->st->dimensions; rd && c < r->d ;c++, rd = rd->next) { + if(unlikely(r->od[c] & RRDR_HIDDEN)) continue; + if(unlikely((options & RRDR_OPTION_NONZERO) && !(r->od[c] & RRDR_NONZERO))) continue; + + calculated_number n = cn[c]; + + buffer_strcat(wb, pre_value); + + if( options & RRDR_OPTION_OBJECTSROWS ) + buffer_sprintf(wb, "%s%s%s: ", kq, rd->name, kq); + + if(co[c] & RRDR_EMPTY) { + if(options & RRDR_OPTION_NULL2ZERO) + buffer_strcat(wb, "0"); + else + buffer_strcat(wb, "null"); + } + else { + if(unlikely((options & RRDR_OPTION_ABSOLUTE) && n < 0)) + n = -n; + + if(unlikely(options & RRDR_OPTION_PERCENTAGE)) + n = n * 100 / total; + + buffer_rrd_value(wb, n); + } + + buffer_strcat(wb, post_value); + } + + buffer_strcat(wb, post_line); + } + + buffer_strcat(wb, finish); + //info("RRD2JSON(): %s: END", r->st->id); } static void rrdr2csv(RRDR *r, BUFFER *wb, uint32_t options, const char *startline, const char *separator, const char *endline, const char *betweenlines) { - //info("RRD2CSV(): %s: BEGIN", r->st->id); - long c, i; - RRDDIM *d; - - // print the csv header - for(c = 0, i = 0, d = r->st->dimensions; d && c < r->d ;c++, d = d->next) { - if(unlikely(r->od[c] & RRDR_HIDDEN)) continue; - if(unlikely((options & RRDR_OPTION_NONZERO) && !(r->od[c] & RRDR_NONZERO))) continue; - - if(!i) { - buffer_strcat(wb, startline); - if(options & RRDR_OPTION_LABEL_QUOTES) buffer_strcat(wb, "\""); - buffer_strcat(wb, "time"); - if(options & RRDR_OPTION_LABEL_QUOTES) buffer_strcat(wb, "\""); - } - buffer_strcat(wb, separator); - if(options & RRDR_OPTION_LABEL_QUOTES) buffer_strcat(wb, "\""); - buffer_strcat(wb, d->name); - if(options & RRDR_OPTION_LABEL_QUOTES) buffer_strcat(wb, "\""); - i++; - } - buffer_strcat(wb, endline); - - if(!i) { - // no dimensions present - return; - } - - long start = 0, end = rrdr_rows(r), step = 1; - if((options & RRDR_OPTION_REVERSED)) { - start = rrdr_rows(r) - 1; - end = -1; - step = -1; - } - - // for each line in the array - calculated_number total = 1; - for(i = start; i != end ;i += step) { - calculated_number *cn = &r->v[ i * r->d ]; - uint8_t *co = &r->o[ i * r->d ]; - - buffer_strcat(wb, betweenlines); - buffer_strcat(wb, startline); - - time_t now = r->t[i]; - - if((options & RRDR_OPTION_SECONDS) || (options & RRDR_OPTION_MILLISECONDS)) { - // print the timestamp of the line - buffer_rrd_value(wb, (calculated_number)now); - // in ms - if(options & RRDR_OPTION_MILLISECONDS) buffer_strcat(wb, "000"); - } - else { - // generate the local date time - struct tm tmbuf, *tm = localtime_r(&now, &tmbuf); - if(!tm) { error("localtime() failed."); continue; } - buffer_date(wb, tm->tm_year + 1900, tm->tm_mon + 1, tm->tm_mday, tm->tm_hour, tm->tm_min, tm->tm_sec); - } - - if(unlikely(options & RRDR_OPTION_PERCENTAGE)) { - total = 0; - for(c = 0, d = r->st->dimensions; d && c < r->d ;c++, d = d->next) { - calculated_number n = cn[c]; - - if(likely((options & RRDR_OPTION_ABSOLUTE) && n < 0)) - n = -n; - - total += n; - } - // prevent a division by zero - if(total == 0) total = 1; - } - - // for each dimension - for(c = 0, d = r->st->dimensions; d && c < r->d ;c++, d = d->next) { - if(unlikely(r->od[c] & RRDR_HIDDEN)) continue; - if(unlikely((options & RRDR_OPTION_NONZERO) && !(r->od[c] & RRDR_NONZERO))) continue; - - buffer_strcat(wb, separator); - - calculated_number n = cn[c]; - - if(co[c] & RRDR_EMPTY) { - if(options & RRDR_OPTION_NULL2ZERO) - buffer_strcat(wb, "0"); - else - buffer_strcat(wb, "null"); - } - else { - if(unlikely((options & RRDR_OPTION_ABSOLUTE) && n < 0)) - n = -n; - - if(unlikely(options & RRDR_OPTION_PERCENTAGE)) - n = n * 100 / total; - - buffer_rrd_value(wb, n); - } - } - - buffer_strcat(wb, endline); - } - //info("RRD2CSV(): %s: END", r->st->id); + //info("RRD2CSV(): %s: BEGIN", r->st->id); + long c, i; + RRDDIM *d; + + // print the csv header + for(c = 0, i = 0, d = r->st->dimensions; d && c < r->d ;c++, d = d->next) { + if(unlikely(r->od[c] & RRDR_HIDDEN)) continue; + if(unlikely((options & RRDR_OPTION_NONZERO) && !(r->od[c] & RRDR_NONZERO))) continue; + + if(!i) { + buffer_strcat(wb, startline); + if(options & RRDR_OPTION_LABEL_QUOTES) buffer_strcat(wb, "\""); + buffer_strcat(wb, "time"); + if(options & RRDR_OPTION_LABEL_QUOTES) buffer_strcat(wb, "\""); + } + buffer_strcat(wb, separator); + if(options & RRDR_OPTION_LABEL_QUOTES) buffer_strcat(wb, "\""); + buffer_strcat(wb, d->name); + if(options & RRDR_OPTION_LABEL_QUOTES) buffer_strcat(wb, "\""); + i++; + } + buffer_strcat(wb, endline); + + if(!i) { + // no dimensions present + return; + } + + long start = 0, end = rrdr_rows(r), step = 1; + if((options & RRDR_OPTION_REVERSED)) { + start = rrdr_rows(r) - 1; + end = -1; + step = -1; + } + + // for each line in the array + calculated_number total = 1; + for(i = start; i != end ;i += step) { + calculated_number *cn = &r->v[ i * r->d ]; + uint8_t *co = &r->o[ i * r->d ]; + + buffer_strcat(wb, betweenlines); + buffer_strcat(wb, startline); + + time_t now = r->t[i]; + + if((options & RRDR_OPTION_SECONDS) || (options & RRDR_OPTION_MILLISECONDS)) { + // print the timestamp of the line + buffer_rrd_value(wb, (calculated_number)now); + // in ms + if(options & RRDR_OPTION_MILLISECONDS) buffer_strcat(wb, "000"); + } + else { + // generate the local date time + struct tm tmbuf, *tm = localtime_r(&now, &tmbuf); + if(!tm) { error("localtime() failed."); continue; } + buffer_date(wb, tm->tm_year + 1900, tm->tm_mon + 1, tm->tm_mday, tm->tm_hour, tm->tm_min, tm->tm_sec); + } + + if(unlikely(options & RRDR_OPTION_PERCENTAGE)) { + total = 0; + for(c = 0, d = r->st->dimensions; d && c < r->d ;c++, d = d->next) { + calculated_number n = cn[c]; + + if(likely((options & RRDR_OPTION_ABSOLUTE) && n < 0)) + n = -n; + + total += n; + } + // prevent a division by zero + if(total == 0) total = 1; + } + + // for each dimension + for(c = 0, d = r->st->dimensions; d && c < r->d ;c++, d = d->next) { + if(unlikely(r->od[c] & RRDR_HIDDEN)) continue; + if(unlikely((options & RRDR_OPTION_NONZERO) && !(r->od[c] & RRDR_NONZERO))) continue; + + buffer_strcat(wb, separator); + + calculated_number n = cn[c]; + + if(co[c] & RRDR_EMPTY) { + if(options & RRDR_OPTION_NULL2ZERO) + buffer_strcat(wb, "0"); + else + buffer_strcat(wb, "null"); + } + else { + if(unlikely((options & RRDR_OPTION_ABSOLUTE) && n < 0)) + n = -n; + + if(unlikely(options & RRDR_OPTION_PERCENTAGE)) + n = n * 100 / total; + + buffer_rrd_value(wb, n); + } + } + + buffer_strcat(wb, endline); + } + //info("RRD2CSV(): %s: END", r->st->id); +} + +inline static calculated_number rrdr2value(RRDR *r, long i, uint32_t options, int *all_values_are_null) { + long c; + RRDDIM *d; + + calculated_number *cn = &r->v[ i * r->d ]; + uint8_t *co = &r->o[ i * r->d ]; + + calculated_number sum = 0, min = 0, max = 0, v; + int all_null = 1, init = 1; + + calculated_number total = 1; + if(unlikely(options & RRDR_OPTION_PERCENTAGE)) { + total = 0; + for(c = 0, d = r->st->dimensions; d && c < r->d ;c++, d = d->next) { + calculated_number n = cn[c]; + + if(likely((options & RRDR_OPTION_ABSOLUTE) && n < 0)) + n = -n; + + total += n; + } + // prevent a division by zero + if(total == 0) total = 1; + } + + // for each dimension + for(c = 0, d = r->st->dimensions; d && c < r->d ;c++, d = d->next) { + if(unlikely(r->od[c] & RRDR_HIDDEN)) continue; + if(unlikely((options & RRDR_OPTION_NONZERO) && !(r->od[c] & RRDR_NONZERO))) continue; + + calculated_number n = cn[c]; + + if(likely((options & RRDR_OPTION_ABSOLUTE) && n < 0)) + n = -n; + + if(unlikely(options & RRDR_OPTION_PERCENTAGE)) + n = n * 100 / total; + + if(unlikely(init)) { + if(n > 0) { + min = 0; + max = n; + } + else { + min = n; + max = 0; + } + init = 0; + } + + if(likely(!(co[c] & RRDR_EMPTY))) { + all_null = 0; + sum += n; + } + + if(n < min) min = n; + if(n > max) max = n; + } + + if(unlikely(all_null)) { + if(likely(all_values_are_null)) + *all_values_are_null = 1; + return 0; + } + else { + if(likely(all_values_are_null)) + *all_values_are_null = 0; + } + + if(options & RRDR_OPTION_MIN2MAX) + v = max - min; + else + v = sum; + + return v; } static void rrdr2ssv(RRDR *r, BUFFER *wb, uint32_t options, const char *prefix, const char *separator, const char *suffix) { - //info("RRD2SSV(): %s: BEGIN", r->st->id); - long c, i; - RRDDIM *d; - - buffer_strcat(wb, prefix); - long start = 0, end = rrdr_rows(r), step = 1; - if((options & RRDR_OPTION_REVERSED)) { - start = rrdr_rows(r) - 1; - end = -1; - step = -1; - } - - // for each line in the array - calculated_number total = 1; - for(i = start; i != end ;i += step) { - - calculated_number *cn = &r->v[ i * r->d ]; - uint8_t *co = &r->o[ i * r->d ]; - - calculated_number sum = 0, min = 0, max = 0, v; - int all_null = 1, init = 1; - - if(unlikely(options & RRDR_OPTION_PERCENTAGE)) { - total = 0; - for(c = 0, d = r->st->dimensions; d && c < r->d ;c++, d = d->next) { - calculated_number n = cn[c]; - - if(likely((options & RRDR_OPTION_ABSOLUTE) && n < 0)) - n = -n; - - total += n; - } - // prevent a division by zero - if(total == 0) total = 1; - } - - // for each dimension - for(c = 0, d = r->st->dimensions; d && c < r->d ;c++, d = d->next) { - if(unlikely(r->od[c] & RRDR_HIDDEN)) continue; - if(unlikely((options & RRDR_OPTION_NONZERO) && !(r->od[c] & RRDR_NONZERO))) continue; - - calculated_number n = cn[c]; - - if(likely((options & RRDR_OPTION_ABSOLUTE) && n < 0)) - n = -n; - - if(unlikely(options & RRDR_OPTION_PERCENTAGE)) - n = n * 100 / total; - - if(unlikely(init)) { - if(n > 0) { - min = 0; - max = n; - } - else { - min = n; - max = 0; - } - init = 0; - } - - if(likely(!(co[c] & RRDR_EMPTY))) { - all_null = 0; - sum += n; - } - - if(n < min) min = n; - if(n > max) max = n; - } - - if(likely(i != start)) - buffer_strcat(wb, separator); - - if(all_null) { - if(options & RRDR_OPTION_NULL2ZERO) - buffer_strcat(wb, "0"); - else - buffer_strcat(wb, "null"); - } - else { - if(options & RRDR_OPTION_MIN2MAX) - v = max - min; - else - v = sum; - - if(likely(i != start)) { - if(r->min > v) r->min = v; - if(r->max < v) r->max = v; - } - else { - r->min = v; - r->max = v; - } - - buffer_rrd_value(wb, v); - } - } - buffer_strcat(wb, suffix); - //info("RRD2SSV(): %s: END", r->st->id); + //info("RRD2SSV(): %s: BEGIN", r->st->id); + long i; + + buffer_strcat(wb, prefix); + long start = 0, end = rrdr_rows(r), step = 1; + if((options & RRDR_OPTION_REVERSED)) { + start = rrdr_rows(r) - 1; + end = -1; + step = -1; + } + + // for each line in the array + for(i = start; i != end ;i += step) { + int all_values_are_null = 0; + calculated_number v = rrdr2value(r, i, options, &all_values_are_null); + + if(likely(i != start)) { + if(r->min > v) r->min = v; + if(r->max < v) r->max = v; + } + else { + r->min = v; + r->max = v; + } + + if(likely(i != start)) + buffer_strcat(wb, separator); + + if(all_values_are_null) { + if(options & RRDR_OPTION_NULL2ZERO) + buffer_strcat(wb, "0"); + else + buffer_strcat(wb, "null"); + } + else + buffer_rrd_value(wb, v); + } + buffer_strcat(wb, suffix); + //info("RRD2SSV(): %s: END", r->st->id); } inline static calculated_number *rrdr_line_values(RRDR *r) { - return &r->v[ r->c * r->d ]; + return &r->v[ r->c * r->d ]; } inline static uint8_t *rrdr_line_options(RRDR *r) { - return &r->o[ r->c * r->d ]; + return &r->o[ r->c * r->d ]; } inline static int rrdr_line_init(RRDR *r, time_t t) { - r->c++; + r->c++; - if(unlikely(r->c >= r->n)) { - error("requested to step above RRDR size for chart %s", r->st->name); - r->c = r->n - 1; - } + if(unlikely(r->c >= r->n)) { + error("requested to step above RRDR size for chart %s", r->st->name); + r->c = r->n - 1; + } - // save the time - r->t[r->c] = t; + // save the time + r->t[r->c] = t; - return 1; + return 1; } inline static void rrdr_lock_rrdset(RRDR *r) { - if(unlikely(!r)) { - error("NULL value given!"); - return; - } + if(unlikely(!r)) { + error("NULL value given!"); + return; + } - pthread_rwlock_rdlock(&r->st->rwlock); - r->has_st_lock = 1; + pthread_rwlock_rdlock(&r->st->rwlock); + r->has_st_lock = 1; } inline static void rrdr_unlock_rrdset(RRDR *r) { - if(unlikely(!r)) { - error("NULL value given!"); - return; - } - - if(likely(r->has_st_lock)) { - pthread_rwlock_unlock(&r->st->rwlock); - r->has_st_lock = 0; - } + if(unlikely(!r)) { + error("NULL value given!"); + return; + } + + if(likely(r->has_st_lock)) { + pthread_rwlock_unlock(&r->st->rwlock); + r->has_st_lock = 0; + } } inline static void rrdr_free(RRDR *r) { - if(unlikely(!r)) { - error("NULL value given!"); - return; - } - - rrdr_unlock_rrdset(r); - if(likely(r->t)) free(r->t); - if(likely(r->v)) free(r->v); - if(likely(r->o)) free(r->o); - if(likely(r->od)) free(r->od); - free(r); + if(unlikely(!r)) { + error("NULL value given!"); + return; + } + + rrdr_unlock_rrdset(r); + freez(r->t); + freez(r->v); + freez(r->o); + freez(r->od); + freez(r); } inline void rrdr_done(RRDR *r) { - r->rows = r->c + 1; - r->c = 0; + r->rows = r->c + 1; + r->c = 0; } static RRDR *rrdr_create(RRDSET *st, long n) { - if(unlikely(!st)) { - error("NULL value given!"); - return NULL; - } - - RRDR *r = calloc(1, sizeof(RRDR)); - if(unlikely(!r)) goto cleanup; - - r->st = st; - - rrdr_lock_rrdset(r); - - RRDDIM *rd; - for(rd = st->dimensions ; rd ; rd = rd->next) r->d++; + if(unlikely(!st)) { + error("NULL value given!"); + return NULL; + } - r->n = n; + RRDR *r = callocz(1, sizeof(RRDR)); + r->st = st; - r->t = malloc(n * sizeof(time_t)); - if(unlikely(!r->t)) goto cleanup; + rrdr_lock_rrdset(r); - r->v = malloc(n * r->d * sizeof(calculated_number)); - if(unlikely(!r->v)) goto cleanup; + RRDDIM *rd; + for(rd = st->dimensions ; rd ; rd = rd->next) r->d++; - r->o = malloc(n * r->d * sizeof(uint8_t)); - if(unlikely(!r->o)) goto cleanup; + r->n = n; - r->od = malloc(r->d * sizeof(uint8_t)); - if(unlikely(!r->od)) goto cleanup; + r->t = mallocz(n * sizeof(time_t)); + r->v = mallocz(n * r->d * sizeof(calculated_number)); + r->o = mallocz(n * r->d * sizeof(uint8_t)); + r->od = mallocz(r->d * sizeof(uint8_t)); - // set the hidden flag on hidden dimensions - int c; - for(c = 0, rd = st->dimensions ; rd ; c++, rd = rd->next) { - if(unlikely(rd->flags & RRDDIM_FLAG_HIDDEN)) r->od[c] = RRDR_HIDDEN; - else r->od[c] = 0; - } + // set the hidden flag on hidden dimensions + int c; + for(c = 0, rd = st->dimensions ; rd ; c++, rd = rd->next) { + if(unlikely(rd->flags & RRDDIM_FLAG_HIDDEN)) r->od[c] = RRDR_HIDDEN; + else r->od[c] = 0; + } - r->c = -1; + r->c = -1; + r->group = 1; + r->update_every = 1; - r->group = 1; - r->update_every = 1; - - return r; - -cleanup: - error("Cannot allocate RRDR memory for %d entries", n); - if(likely(r)) rrdr_free(r); - return NULL; + return r; } -RRDR *rrd2rrdr(RRDSET *st, long points, long long after, long long before, int group_method) +RRDR *rrd2rrdr(RRDSET *st, long points, long long after, long long before, int group_method, int aligned) { - int debug = st->debug; - int absolute_period_requested = -1; - - time_t first_entry_t = rrdset_first_entry_t(st); - time_t last_entry_t = rrdset_last_entry_t(st); - - if(before == 0 && after == 0) { - before = last_entry_t; - after = first_entry_t; - absolute_period_requested = 0; - } - - // allow relative for before and after - if(before <= st->update_every * st->entries) { - before = last_entry_t + before; - absolute_period_requested = 0; - } - - if(after <= st->update_every * st->entries) { - after = last_entry_t + after; - absolute_period_requested = 0; - } - - if(absolute_period_requested == -1) - absolute_period_requested = 1; + int debug = st->debug; + int absolute_period_requested = -1; + + time_t first_entry_t = rrdset_first_entry_t(st); + time_t last_entry_t = rrdset_last_entry_t(st); + + if(before == 0 && after == 0) { + before = last_entry_t; + after = first_entry_t; + absolute_period_requested = 0; + } + + // allow relative for before and after + if(((before < 0)?-before:before) <= (st->update_every * st->entries)) { + before = last_entry_t + before; + absolute_period_requested = 0; + } + + if(((after < 0)?-after:after) <= (st->update_every * st->entries)) { + if(after == 0) after = -st->update_every; + after = before + after; + absolute_period_requested = 0; + } + + if(absolute_period_requested == -1) + absolute_period_requested = 1; + + // make sure they are within our timeframe + if(before > last_entry_t) before = last_entry_t; + if(before < first_entry_t) before = first_entry_t; + + if(after > last_entry_t) after = last_entry_t; + if(after < first_entry_t) after = first_entry_t; + + // check if they are upside down + if(after > before) { + time_t tmp = before; + before = after; + after = tmp; + } + + // the duration of the chart + time_t duration = before - after; + long available_points = duration / st->update_every; + + if(duration <= 0 || available_points <= 0) + return rrdr_create(st, 1); + + // check the wanted points + if(points < 0) points = -points; + if(points > available_points) points = available_points; + if(points == 0) points = available_points; + + // calculate proper grouping of source data + long group = available_points / points; + if(group <= 0) group = 1; + + // round group to the closest integer + if(available_points % points > points / 2) group++; + + time_t after_new = (aligned) ? (after - (after % (group * st->update_every))) : after; + time_t before_new = (aligned) ? (before - (before % (group * st->update_every))) : before; + long points_new = (before_new - after_new) / st->update_every / group; + + // find the starting and ending slots in our round robin db + long start_at_slot = rrdset_time2slot(st, before_new), + stop_at_slot = rrdset_time2slot(st, after_new); - // make sure they are within our timeframe - if(before > last_entry_t) before = last_entry_t; - if(before < first_entry_t) before = first_entry_t; +#ifdef NETDATA_INTERNAL_CHECKS + if(after_new < first_entry_t) { + error("after_new %u is too small, minimum %u", (uint32_t)after_new, (uint32_t)first_entry_t); + } + if(after_new > last_entry_t) { + error("after_new %u is too big, maximum %u", (uint32_t)after_new, (uint32_t)last_entry_t); + } + if(before_new < first_entry_t) { + error("before_new %u is too small, minimum %u", (uint32_t)before_new, (uint32_t)first_entry_t); + } + if(before_new > last_entry_t) { + error("before_new %u is too big, maximum %u", (uint32_t)before_new, (uint32_t)last_entry_t); + } + if(start_at_slot < 0 || start_at_slot >= st->entries) { + error("start_at_slot is invalid %ld, expected 0 to %ld", start_at_slot, st->entries - 1); + } + if(stop_at_slot < 0 || stop_at_slot >= st->entries) { + error("stop_at_slot is invalid %ld, expected 0 to %ld", stop_at_slot, st->entries - 1); + } + if(points_new > (before_new - after_new) / group / st->update_every + 1) { + error("points_new %ld is more than points %ld", points_new, (before_new - after_new) / group / st->update_every + 1); + } +#endif - if(after > last_entry_t) after = last_entry_t; - if(after < first_entry_t) after = first_entry_t; + //info("RRD2RRDR(): %s: wanted %ld points, got %ld - group=%ld, wanted duration=%u, got %u - wanted %ld - %ld, got %ld - %ld", st->id, points, points_new, group, before - after, before_new - after_new, after, before, after_new, before_new); - // check if they are upside down - if(after > before) { - time_t tmp = before; - before = after; - after = tmp; - } + after = after_new; + before = before_new; + duration = before - after; + points = points_new; - // the duration of the chart - time_t duration = before - after; - long available_points = duration / st->update_every; + // Now we have: + // before = the end time of the calculation + // after = the start time of the calculation + // duration = the duration of the calculation + // group = the number of source points to aggregate / group together + // method = the method of grouping source points + // points = the number of points to generate - if(duration <= 0 || available_points <= 0) - return rrdr_create(st, 1); - // check the wanted points - if(points < 0) points = -points; - if(points > available_points) points = available_points; - if(points == 0) points = available_points; + // ------------------------------------------------------------------------- + // initialize our result set - // calculate proper grouping of source data - long group = available_points / points; - if(group <= 0) group = 1; + RRDR *r = rrdr_create(st, points); + if(!r) { +#ifdef NETDATA_INTERNAL_CHECKS + error("Cannot create RRDR for %s, after=%u, before=%u, duration=%u, points=%ld", st->id, (uint32_t)after, (uint32_t)before, (uint32_t)duration, points); +#endif + return NULL; + } + if(!r->d) { +#ifdef NETDATA_INTERNAL_CHECKS + error("Returning empty RRDR (no dimensions in RRDSET) for %s, after=%u, before=%u, duration=%u, points=%ld", st->id, (uint32_t)after, (uint32_t)before, (uint32_t)duration, points); +#endif + return r; + } + + if(absolute_period_requested == 1) + r->result_options |= RRDR_RESULT_OPTION_ABSOLUTE; + else + r->result_options |= RRDR_RESULT_OPTION_RELATIVE; + + // find how many dimensions we have + long dimensions = r->d; + + + // ------------------------------------------------------------------------- + // checks for debugging + + if(debug) debug(D_RRD_STATS, "INFO %s first_t: %u, last_t: %u, all_duration: %u, after: %u, before: %u, duration: %u, points: %ld, group: %ld" + , st->id + , (uint32_t)first_entry_t + , (uint32_t)last_entry_t + , (uint32_t)(last_entry_t - first_entry_t) + , (uint32_t)after + , (uint32_t)before + , (uint32_t)duration + , points + , group + ); + + + // ------------------------------------------------------------------------- + // temp arrays for keeping values per dimension + + calculated_number last_values[dimensions]; // keep the last value of each dimension + calculated_number group_values[dimensions]; // keep sums when grouping + long group_counts[dimensions]; // keep the number of values added to group_values + uint8_t group_options[dimensions]; + uint8_t found_non_zero[dimensions]; + + + // initialize them + RRDDIM *rd; + long c; + for( rd = st->dimensions, c = 0 ; rd && c < dimensions ; rd = rd->next, c++) { + last_values[c] = 0; + group_values[c] = (group_method == GROUP_MAX || group_method == GROUP_MIN)?NAN:0; + group_counts[c] = 0; + group_options[c] = 0; + found_non_zero[c] = 0; + } + + + // ------------------------------------------------------------------------- + // the main loop + + time_t now = rrdset_slot2time(st, start_at_slot), + dt = st->update_every, + group_start_t = 0; + + if(unlikely(debug)) debug(D_RRD_STATS, "BEGIN %s after_t: %u (stop_at_t: %ld), before_t: %u (start_at_t: %ld), start_t(now): %u, current_entry: %ld, entries: %ld" + , st->id + , (uint32_t)after + , stop_at_slot + , (uint32_t)before + , start_at_slot + , (uint32_t)now + , st->current_entry + , st->entries + ); + + r->group = group; + r->update_every = group * st->update_every; + r->before = now; + r->after = now; + + //info("RRD2RRDR(): %s: STARTING", st->id); + + long slot = start_at_slot, counter = 0, stop_now = 0, added = 0, group_count = 0, add_this = 0; + for(; !stop_now ; now -= dt, slot--, counter++) { + if(unlikely(slot < 0)) slot = st->entries - 1; + if(unlikely(slot == stop_at_slot)) stop_now = counter; + + if(unlikely(debug)) debug(D_RRD_STATS, "ROW %s slot: %ld, entries_counter: %ld, group_count: %ld, added: %ld, now: %ld, %s %s" + , st->id + , slot + , counter + , group_count + 1 + , added + , now + , (group_count + 1 == group)?"PRINT":" - " + , (now >= after && now <= before)?"RANGE":" - " + ); + + // make sure we return data in the proper time range + if(unlikely(now > before)) continue; + if(unlikely(now < after)) break; + + if(unlikely(group_count == 0)) { + group_start_t = now; + } + group_count++; + + if(unlikely(group_count == group)) { + if(unlikely(added >= points)) break; + add_this = 1; + } + + // do the calculations + for(rd = st->dimensions, c = 0 ; rd && c < dimensions ; rd = rd->next, c++) { + storage_number n = rd->values[slot]; + if(unlikely(!does_storage_number_exist(n))) continue; + + group_counts[c]++; + + calculated_number value = unpack_storage_number(n); + if(likely(value != 0.0)) { + group_options[c] |= RRDR_NONZERO; + found_non_zero[c] = 1; + } + + if(unlikely(did_storage_number_reset(n))) + group_options[c] |= RRDR_RESET; + + switch(group_method) { + case GROUP_MIN: + if(unlikely(isnan(group_values[c])) || + fabsl(value) < fabsl(group_values[c])) + group_values[c] = value; + break; + + case GROUP_MAX: + if(unlikely(isnan(group_values[c])) || + fabsl(value) > fabsl(group_values[c])) + group_values[c] = value; + break; + + default: + case GROUP_SUM: + case GROUP_AVERAGE: + case GROUP_UNDEFINED: + group_values[c] += value; + break; + + case GROUP_INCREMENTAL_SUM: + if(unlikely(slot == start_at_slot)) + last_values[c] = value; + + group_values[c] += last_values[c] - value; + last_values[c] = value; + break; + } + } + + // added it + if(unlikely(add_this)) { + if(unlikely(!rrdr_line_init(r, group_start_t))) break; + + r->after = now; + + calculated_number *cn = rrdr_line_values(r); + uint8_t *co = rrdr_line_options(r); + + for(rd = st->dimensions, c = 0 ; rd && c < dimensions ; rd = rd->next, c++) { + + // update the dimension options + if(likely(found_non_zero[c])) r->od[c] |= RRDR_NONZERO; + + // store the specific point options + co[c] = group_options[c]; + + // store the value + if(unlikely(group_counts[c] == 0)) { + cn[c] = 0.0; + co[c] |= RRDR_EMPTY; + group_values[c] = (group_method == GROUP_MAX || group_method == GROUP_MIN)?NAN:0; + } + else { + switch(group_method) { + case GROUP_MIN: + case GROUP_MAX: + if(unlikely(isnan(group_values[c]))) + cn[c] = 0; + else { + cn[c] = group_values[c]; + group_values[c] = NAN; + } + break; + + case GROUP_SUM: + case GROUP_INCREMENTAL_SUM: + cn[c] = group_values[c]; + group_values[c] = 0; + break; + + default: + case GROUP_AVERAGE: + case GROUP_UNDEFINED: + cn[c] = group_values[c] / group_counts[c]; + group_values[c] = 0; + break; + } + + if(cn[c] < r->min) r->min = cn[c]; + if(cn[c] > r->max) r->max = cn[c]; + } + + // reset for the next loop + group_counts[c] = 0; + group_options[c] = 0; + } + + added++; + group_count = 0; + add_this = 0; + } + } + + rrdr_done(r); + //info("RRD2RRDR(): %s: END %ld loops made, %ld points generated", st->id, counter, rrdr_rows(r)); + //error("SHIFT: %s: wanted %ld points, got %ld", st->id, points, rrdr_rows(r)); + return r; +} - // round group to the closest integer - if(available_points % points > points / 2) group++; +int rrd2value(RRDSET *st, BUFFER *wb, calculated_number *n, const char *dimensions, long points, long long after, long long before, int group_method, uint32_t options, time_t *db_after, time_t *db_before, int *value_is_null) +{ + RRDR *r = rrd2rrdr(st, points, after, before, group_method, !(options & RRDR_OPTION_NOT_ALIGNED)); + if(!r) { + if(value_is_null) *value_is_null = 1; + return 500; + } - time_t after_new = after - (after % (group * st->update_every)); - time_t before_new = before - (before % (group * st->update_every)); - long points_new = (before_new - after_new) / st->update_every / group; + if(rrdr_rows(r) == 0) { + rrdr_free(r); - // find the starting and ending slots in our round robin db - long start_at_slot = rrdset_time2slot(st, before_new), - stop_at_slot = rrdset_time2slot(st, after_new); + if(db_after) *db_after = 0; + if(db_before) *db_before = 0; + if(value_is_null) *value_is_null = 1; -#ifdef NETDATA_INTERNAL_CHECKS - if(after_new < first_entry_t) { - error("after_new %u is too small, minimum %u", after_new, first_entry_t); - } - if(after_new > last_entry_t) { - error("after_new %u is too big, maximum %u", after_new, last_entry_t); - } - if(before_new < first_entry_t) { - error("before_new %u is too small, minimum %u", before_new, first_entry_t); - } - if(before_new > last_entry_t) { - error("before_new %u is too big, maximum %u", before_new, last_entry_t); - } - if(start_at_slot < 0 || start_at_slot >= st->entries) { - error("start_at_slot is invalid %ld, expected %ld to %ld", start_at_slot, 0, st->entries - 1); - } - if(stop_at_slot < 0 || stop_at_slot >= st->entries) { - error("stop_at_slot is invalid %ld, expected %ld to %ld", stop_at_slot, 0, st->entries - 1); - } - if(points_new > (before_new - after_new) / group / st->update_every + 1) { - error("points_new %ld is more than points %ld", points_new, (before_new - after_new) / group / st->update_every + 1); - } -#endif + return 400; + } - //info("RRD2RRDR(): %s: wanted %ld points, got %ld - group=%ld, wanted duration=%u, got %u - wanted %ld - %ld, got %ld - %ld", st->id, points, points_new, group, before - after, before_new - after_new, after, before, after_new, before_new); + if(r->result_options & RRDR_RESULT_OPTION_RELATIVE) + wb->options |= WB_CONTENT_NO_CACHEABLE; + else if(r->result_options & RRDR_RESULT_OPTION_ABSOLUTE) + wb->options |= WB_CONTENT_CACHEABLE; - after = after_new; - before = before_new; - duration = before - after; - points = points_new; + options = rrdr_check_options(r, options, dimensions); - // Now we have: - // before = the end time of the calculation - // after = the start time of the calculation - // duration = the duration of the calculation - // group = the number of source points to aggregate / group together - // method = the method of grouping source points - // points = the number of points to generate + if(dimensions) + rrdr_disable_not_selected_dimensions(r, dimensions); + if(db_after) *db_after = r->after; + if(db_before) *db_before = r->before; - // ------------------------------------------------------------------------- - // initialize our result set + long i = (options & RRDR_OPTION_REVERSED)?rrdr_rows(r) - 1:0; + *n = rrdr2value(r, i, options, value_is_null); - RRDR *r = rrdr_create(st, points); - if(!r) { -#ifdef NETDATA_INTERNAL_CHECKS - error("Cannot create RRDR for %s, after=%u, before=%u, duration=%u, points=%d", st->id, after, before, duration, points); -#endif - return NULL; - } - if(!r->d) { -#ifdef NETDATA_INTERNAL_CHECKS - error("Returning empty RRDR (no dimensions in RRDSET) for %s, after=%u, before=%u, duration=%u, points=%d", st->id, after, before, duration, points); -#endif - return r; - } - - if(absolute_period_requested == 1) - r->result_options |= RRDR_RESULT_OPTION_ABSOLUTE; - else - r->result_options |= RRDR_RESULT_OPTION_RELATIVE; - - // find how many dimensions we have - long dimensions = r->d; - - - // ------------------------------------------------------------------------- - // checks for debugging - - if(debug) debug(D_RRD_STATS, "INFO %s first_t: %lu, last_t: %lu, all_duration: %lu, after: %lu, before: %lu, duration: %lu, points: %ld, group: %ld" - , st->id - , first_entry_t - , last_entry_t - , last_entry_t - first_entry_t - , after - , before - , duration - , points - , group - ); - - - // ------------------------------------------------------------------------- - // temp arrays for keeping values per dimension - - calculated_number group_values[dimensions]; // keep sums when grouping - long group_counts[dimensions]; // keep the number of values added to group_values - uint8_t group_options[dimensions]; - uint8_t found_non_zero[dimensions]; - - - // initialize them - RRDDIM *rd; - long c; - for( rd = st->dimensions, c = 0 ; rd && c < dimensions ; rd = rd->next, c++) { - group_values[c] = 0; - group_counts[c] = 0; - group_options[c] = 0; - found_non_zero[c] = 0; - } - - - // ------------------------------------------------------------------------- - // the main loop - - time_t now = rrdset_slot2time(st, start_at_slot), - dt = st->update_every, - group_start_t = 0; - - if(unlikely(debug)) debug(D_RRD_STATS, "BEGIN %s after_t: %lu (stop_at_t: %ld), before_t: %lu (start_at_t: %ld), start_t(now): %lu, current_entry: %ld, entries: %ld" - , st->id - , after - , stop_at_slot - , before - , start_at_slot - , now - , st->current_entry - , st->entries - ); - - r->group = group; - r->update_every = group * st->update_every; - r->before = now; - r->after = now; - - //info("RRD2RRDR(): %s: STARTING", st->id); - - long slot = start_at_slot, counter = 0, stop_now = 0, added = 0, group_count = 0, add_this = 0; - for(; !stop_now ; now -= dt, slot--, counter++) { - if(unlikely(slot < 0)) slot = st->entries - 1; - if(unlikely(slot == stop_at_slot)) stop_now = counter; - - if(unlikely(debug)) debug(D_RRD_STATS, "ROW %s slot: %ld, entries_counter: %ld, group_count: %ld, added: %ld, now: %lu, %s %s" - , st->id - , slot - , counter - , group_count + 1 - , added - , now - , (group_count + 1 == group)?"PRINT":" - " - , (now >= after && now <= before)?"RANGE":" - " - ); - - // make sure we return data in the proper time range - if(unlikely(now > before)) continue; - if(unlikely(now < after)) break; - - if(unlikely(group_count == 0)) { - group_start_t = now; - } - group_count++; - - if(unlikely(group_count == group)) { - if(unlikely(added >= points)) break; - add_this = 1; - } - - // do the calculations - for(rd = st->dimensions, c = 0 ; rd && c < dimensions ; rd = rd->next, c++) { - storage_number n = rd->values[slot]; - if(unlikely(!does_storage_number_exist(n))) continue; - - group_counts[c]++; - - calculated_number value = unpack_storage_number(n); - if(likely(value != 0.0)) { - group_options[c] |= RRDR_NONZERO; - found_non_zero[c] = 1; - } - - if(unlikely(did_storage_number_reset(n))) - group_options[c] |= RRDR_RESET; - - switch(group_method) { - case GROUP_MAX: - if(unlikely(fabsl(value) > fabsl(group_values[c]))) - group_values[c] = value; - break; - - default: - case GROUP_SUM: - case GROUP_AVERAGE: - group_values[c] += value; - break; - } - } - - // added it - if(unlikely(add_this)) { - if(unlikely(!rrdr_line_init(r, group_start_t))) break; - - r->after = now; - - calculated_number *cn = rrdr_line_values(r); - uint8_t *co = rrdr_line_options(r); - - for(rd = st->dimensions, c = 0 ; rd && c < dimensions ; rd = rd->next, c++) { - - // update the dimension options - if(likely(found_non_zero[c])) r->od[c] |= RRDR_NONZERO; - - // store the specific point options - co[c] = group_options[c]; - - // store the value - if(unlikely(group_counts[c] == 0)) { - cn[c] = 0.0; - co[c] |= RRDR_EMPTY; - } - else if(unlikely(group_method == GROUP_AVERAGE)) { - cn[c] = group_values[c] / group_counts[c]; - } - else { - cn[c] = group_values[c]; - } - - if(cn[c] < r->min) r->min = cn[c]; - if(cn[c] > r->max) r->max = cn[c]; - - // reset them for the next loop - group_values[c] = 0; - group_counts[c] = 0; - group_options[c] = 0; - } - - added++; - group_count = 0; - add_this = 0; - } - } - - rrdr_done(r); - //info("RRD2RRDR(): %s: END %ld loops made, %ld points generated", st->id, counter, rrdr_rows(r)); - //error("SHIFT: %s: wanted %ld points, got %ld", st->id, points, rrdr_rows(r)); - return r; + rrdr_free(r); + return 200; } int rrd2format(RRDSET *st, BUFFER *wb, BUFFER *dimensions, uint32_t format, long points, long long after, long long before, int group_method, uint32_t options, time_t *latest_timestamp) { - RRDR *r = rrd2rrdr(st, points, after, before, group_method); - if(!r) { - buffer_strcat(wb, "Cannot generate output with these parameters on this chart."); - return 500; - } - - if(r->result_options & RRDR_RESULT_OPTION_RELATIVE) - wb->options |= WB_CONTENT_NO_CACHEABLE; - else if(r->result_options & RRDR_RESULT_OPTION_ABSOLUTE) - wb->options |= WB_CONTENT_CACHEABLE; - - options = rrdr_check_options(r, options, (dimensions)?buffer_tostring(dimensions):NULL); - - if(dimensions) - rrdr_disable_not_selected_dimensions(r, buffer_tostring(dimensions)); - - if(latest_timestamp && rrdr_rows(r) > 0) - *latest_timestamp = r->before; - - switch(format) { - case DATASOURCE_SSV: - if(options & RRDR_OPTION_JSON_WRAP) { - wb->contenttype = CT_APPLICATION_JSON; - rrdr_json_wrapper_begin(r, wb, format, options, 1); - rrdr2ssv(r, wb, options, "", " ", ""); - rrdr_json_wrapper_end(r, wb, format, options, 1); - } - else { - wb->contenttype = CT_TEXT_PLAIN; - rrdr2ssv(r, wb, options, "", " ", ""); - } - break; - - case DATASOURCE_SSV_COMMA: - if(options & RRDR_OPTION_JSON_WRAP) { - wb->contenttype = CT_APPLICATION_JSON; - rrdr_json_wrapper_begin(r, wb, format, options, 1); - rrdr2ssv(r, wb, options, "", ",", ""); - rrdr_json_wrapper_end(r, wb, format, options, 1); - } - else { - wb->contenttype = CT_TEXT_PLAIN; - rrdr2ssv(r, wb, options, "", ",", ""); - } - break; - - case DATASOURCE_JS_ARRAY: - if(options & RRDR_OPTION_JSON_WRAP) { - wb->contenttype = CT_APPLICATION_JSON; - rrdr_json_wrapper_begin(r, wb, format, options, 0); - rrdr2ssv(r, wb, options, "[", ",", "]"); - rrdr_json_wrapper_end(r, wb, format, options, 0); - } - else { - wb->contenttype = CT_APPLICATION_JSON; - rrdr2ssv(r, wb, options, "[", ",", "]"); - } - break; - - case DATASOURCE_CSV: - if(options & RRDR_OPTION_JSON_WRAP) { - wb->contenttype = CT_APPLICATION_JSON; - rrdr_json_wrapper_begin(r, wb, format, options, 1); - rrdr2csv(r, wb, options, "", ",", "\\n", ""); - rrdr_json_wrapper_end(r, wb, format, options, 1); - } - else { - wb->contenttype = CT_TEXT_PLAIN; - rrdr2csv(r, wb, options, "", ",", "\r\n", ""); - } - break; - - case DATASOURCE_CSV_JSON_ARRAY: - wb->contenttype = CT_APPLICATION_JSON; - if(options & RRDR_OPTION_JSON_WRAP) { - rrdr_json_wrapper_begin(r, wb, format, options, 0); - buffer_strcat(wb, "[\n"); - rrdr2csv(r, wb, options + RRDR_OPTION_LABEL_QUOTES, "[", ",", "]", ",\n"); - buffer_strcat(wb, "\n]"); - rrdr_json_wrapper_end(r, wb, format, options, 0); - } - else { - wb->contenttype = CT_TEXT_PLAIN; - buffer_strcat(wb, "[\n"); - rrdr2csv(r, wb, options + RRDR_OPTION_LABEL_QUOTES, "[", ",", "]", ",\n"); - buffer_strcat(wb, "\n]"); - } - break; - - case DATASOURCE_TSV: - if(options & RRDR_OPTION_JSON_WRAP) { - wb->contenttype = CT_APPLICATION_JSON; - rrdr_json_wrapper_begin(r, wb, format, options, 1); - rrdr2csv(r, wb, options, "", "\t", "\\n", ""); - rrdr_json_wrapper_end(r, wb, format, options, 1); - } - else { - wb->contenttype = CT_TEXT_PLAIN; - rrdr2csv(r, wb, options, "", "\t", "\r\n", ""); - } - break; - - case DATASOURCE_HTML: - if(options & RRDR_OPTION_JSON_WRAP) { - wb->contenttype = CT_APPLICATION_JSON; - rrdr_json_wrapper_begin(r, wb, format, options, 1); - buffer_strcat(wb, "<html>\\n<center>\\n<table border=\\\"0\\\" cellpadding=\\\"5\\\" cellspacing=\\\"5\\\">\\n"); - rrdr2csv(r, wb, options, "<tr><td>", "</td><td>", "</td></tr>\\n", ""); - buffer_strcat(wb, "</table>\\n</center>\\n</html>\\n"); - rrdr_json_wrapper_end(r, wb, format, options, 1); - } - else { - wb->contenttype = CT_TEXT_HTML; - buffer_strcat(wb, "<html>\n<center>\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"5\">\n"); - rrdr2csv(r, wb, options, "<tr><td>", "</td><td>", "</td></tr>\n", ""); - buffer_strcat(wb, "</table>\n</center>\n</html>\n"); - } - break; - - case DATASOURCE_DATATABLE_JSONP: - wb->contenttype = CT_APPLICATION_X_JAVASCRIPT; - - if(options & RRDR_OPTION_JSON_WRAP) - rrdr_json_wrapper_begin(r, wb, format, options, 0); - - rrdr2json(r, wb, options, 1); - - if(options & RRDR_OPTION_JSON_WRAP) - rrdr_json_wrapper_end(r, wb, format, options, 0); - break; - - case DATASOURCE_DATATABLE_JSON: - wb->contenttype = CT_APPLICATION_JSON; - - if(options & RRDR_OPTION_JSON_WRAP) - rrdr_json_wrapper_begin(r, wb, format, options, 0); - - rrdr2json(r, wb, options, 1); - - if(options & RRDR_OPTION_JSON_WRAP) - rrdr_json_wrapper_end(r, wb, format, options, 0); - break; - - case DATASOURCE_JSONP: - wb->contenttype = CT_APPLICATION_X_JAVASCRIPT; - if(options & RRDR_OPTION_JSON_WRAP) - rrdr_json_wrapper_begin(r, wb, format, options, 0); - - rrdr2json(r, wb, options, 0); - - if(options & RRDR_OPTION_JSON_WRAP) - rrdr_json_wrapper_end(r, wb, format, options, 0); - break; - - case DATASOURCE_JSON: - default: - wb->contenttype = CT_APPLICATION_JSON; - - if(options & RRDR_OPTION_JSON_WRAP) - rrdr_json_wrapper_begin(r, wb, format, options, 0); - - rrdr2json(r, wb, options, 0); - - if(options & RRDR_OPTION_JSON_WRAP) - rrdr_json_wrapper_end(r, wb, format, options, 0); - break; - } - - rrdr_free(r); - return 200; + RRDR *r = rrd2rrdr(st, points, after, before, group_method, !(options & RRDR_OPTION_NOT_ALIGNED)); + if(!r) { + buffer_strcat(wb, "Cannot generate output with these parameters on this chart."); + return 500; + } + + if(r->result_options & RRDR_RESULT_OPTION_RELATIVE) + wb->options |= WB_CONTENT_NO_CACHEABLE; + else if(r->result_options & RRDR_RESULT_OPTION_ABSOLUTE) + wb->options |= WB_CONTENT_CACHEABLE; + + options = rrdr_check_options(r, options, (dimensions)?buffer_tostring(dimensions):NULL); + + if(dimensions) + rrdr_disable_not_selected_dimensions(r, buffer_tostring(dimensions)); + + if(latest_timestamp && rrdr_rows(r) > 0) + *latest_timestamp = r->before; + + switch(format) { + case DATASOURCE_SSV: + if(options & RRDR_OPTION_JSON_WRAP) { + wb->contenttype = CT_APPLICATION_JSON; + rrdr_json_wrapper_begin(r, wb, format, options, 1); + rrdr2ssv(r, wb, options, "", " ", ""); + rrdr_json_wrapper_end(r, wb, format, options, 1); + } + else { + wb->contenttype = CT_TEXT_PLAIN; + rrdr2ssv(r, wb, options, "", " ", ""); + } + break; + + case DATASOURCE_SSV_COMMA: + if(options & RRDR_OPTION_JSON_WRAP) { + wb->contenttype = CT_APPLICATION_JSON; + rrdr_json_wrapper_begin(r, wb, format, options, 1); + rrdr2ssv(r, wb, options, "", ",", ""); + rrdr_json_wrapper_end(r, wb, format, options, 1); + } + else { + wb->contenttype = CT_TEXT_PLAIN; + rrdr2ssv(r, wb, options, "", ",", ""); + } + break; + + case DATASOURCE_JS_ARRAY: + if(options & RRDR_OPTION_JSON_WRAP) { + wb->contenttype = CT_APPLICATION_JSON; + rrdr_json_wrapper_begin(r, wb, format, options, 0); + rrdr2ssv(r, wb, options, "[", ",", "]"); + rrdr_json_wrapper_end(r, wb, format, options, 0); + } + else { + wb->contenttype = CT_APPLICATION_JSON; + rrdr2ssv(r, wb, options, "[", ",", "]"); + } + break; + + case DATASOURCE_CSV: + if(options & RRDR_OPTION_JSON_WRAP) { + wb->contenttype = CT_APPLICATION_JSON; + rrdr_json_wrapper_begin(r, wb, format, options, 1); + rrdr2csv(r, wb, options, "", ",", "\\n", ""); + rrdr_json_wrapper_end(r, wb, format, options, 1); + } + else { + wb->contenttype = CT_TEXT_PLAIN; + rrdr2csv(r, wb, options, "", ",", "\r\n", ""); + } + break; + + case DATASOURCE_CSV_JSON_ARRAY: + wb->contenttype = CT_APPLICATION_JSON; + if(options & RRDR_OPTION_JSON_WRAP) { + rrdr_json_wrapper_begin(r, wb, format, options, 0); + buffer_strcat(wb, "[\n"); + rrdr2csv(r, wb, options + RRDR_OPTION_LABEL_QUOTES, "[", ",", "]", ",\n"); + buffer_strcat(wb, "\n]"); + rrdr_json_wrapper_end(r, wb, format, options, 0); + } + else { + wb->contenttype = CT_TEXT_PLAIN; + buffer_strcat(wb, "[\n"); + rrdr2csv(r, wb, options + RRDR_OPTION_LABEL_QUOTES, "[", ",", "]", ",\n"); + buffer_strcat(wb, "\n]"); + } + break; + + case DATASOURCE_TSV: + if(options & RRDR_OPTION_JSON_WRAP) { + wb->contenttype = CT_APPLICATION_JSON; + rrdr_json_wrapper_begin(r, wb, format, options, 1); + rrdr2csv(r, wb, options, "", "\t", "\\n", ""); + rrdr_json_wrapper_end(r, wb, format, options, 1); + } + else { + wb->contenttype = CT_TEXT_PLAIN; + rrdr2csv(r, wb, options, "", "\t", "\r\n", ""); + } + break; + + case DATASOURCE_HTML: + if(options & RRDR_OPTION_JSON_WRAP) { + wb->contenttype = CT_APPLICATION_JSON; + rrdr_json_wrapper_begin(r, wb, format, options, 1); + buffer_strcat(wb, "<html>\\n<center>\\n<table border=\\\"0\\\" cellpadding=\\\"5\\\" cellspacing=\\\"5\\\">\\n"); + rrdr2csv(r, wb, options, "<tr><td>", "</td><td>", "</td></tr>\\n", ""); + buffer_strcat(wb, "</table>\\n</center>\\n</html>\\n"); + rrdr_json_wrapper_end(r, wb, format, options, 1); + } + else { + wb->contenttype = CT_TEXT_HTML; + buffer_strcat(wb, "<html>\n<center>\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"5\">\n"); + rrdr2csv(r, wb, options, "<tr><td>", "</td><td>", "</td></tr>\n", ""); + buffer_strcat(wb, "</table>\n</center>\n</html>\n"); + } + break; + + case DATASOURCE_DATATABLE_JSONP: + wb->contenttype = CT_APPLICATION_X_JAVASCRIPT; + + if(options & RRDR_OPTION_JSON_WRAP) + rrdr_json_wrapper_begin(r, wb, format, options, 0); + + rrdr2json(r, wb, options, 1); + + if(options & RRDR_OPTION_JSON_WRAP) + rrdr_json_wrapper_end(r, wb, format, options, 0); + break; + + case DATASOURCE_DATATABLE_JSON: + wb->contenttype = CT_APPLICATION_JSON; + + if(options & RRDR_OPTION_JSON_WRAP) + rrdr_json_wrapper_begin(r, wb, format, options, 0); + + rrdr2json(r, wb, options, 1); + + if(options & RRDR_OPTION_JSON_WRAP) + rrdr_json_wrapper_end(r, wb, format, options, 0); + break; + + case DATASOURCE_JSONP: + wb->contenttype = CT_APPLICATION_X_JAVASCRIPT; + if(options & RRDR_OPTION_JSON_WRAP) + rrdr_json_wrapper_begin(r, wb, format, options, 0); + + rrdr2json(r, wb, options, 0); + + if(options & RRDR_OPTION_JSON_WRAP) + rrdr_json_wrapper_end(r, wb, format, options, 0); + break; + + case DATASOURCE_JSON: + default: + wb->contenttype = CT_APPLICATION_JSON; + + if(options & RRDR_OPTION_JSON_WRAP) + rrdr_json_wrapper_begin(r, wb, format, options, 0); + + rrdr2json(r, wb, options, 0); + + if(options & RRDR_OPTION_JSON_WRAP) + rrdr_json_wrapper_end(r, wb, format, options, 0); + break; + } + + rrdr_free(r); + return 200; } time_t rrd_stats_json(int type, RRDSET *st, BUFFER *wb, long points, long group, int group_method, time_t after, time_t before, int only_non_zero) { - int c; - pthread_rwlock_rdlock(&st->rwlock); + int c; + pthread_rwlock_rdlock(&st->rwlock); - // ------------------------------------------------------------------------- - // switch from JSON to google JSON + // ------------------------------------------------------------------------- + // switch from JSON to google JSON - char kq[2] = "\""; - char sq[2] = "\""; - switch(type) { - case DATASOURCE_DATATABLE_JSON: - case DATASOURCE_DATATABLE_JSONP: - kq[0] = '\0'; - sq[0] = '\''; - break; + char kq[2] = "\""; + char sq[2] = "\""; + switch(type) { + case DATASOURCE_DATATABLE_JSON: + case DATASOURCE_DATATABLE_JSONP: + kq[0] = '\0'; + sq[0] = '\''; + break; - case DATASOURCE_JSON: - default: - break; - } + case DATASOURCE_JSON: + default: + break; + } - // ------------------------------------------------------------------------- - // validate the parameters + // ------------------------------------------------------------------------- + // validate the parameters - if(points < 1) points = 1; - if(group < 1) group = 1; + if(points < 1) points = 1; + if(group < 1) group = 1; - if(before == 0 || before > rrdset_last_entry_t(st)) before = rrdset_last_entry_t(st); - if(after == 0 || after < rrdset_first_entry_t(st)) after = rrdset_first_entry_t(st); + if(before == 0 || before > rrdset_last_entry_t(st)) before = rrdset_last_entry_t(st); + if(after == 0 || after < rrdset_first_entry_t(st)) after = rrdset_first_entry_t(st); - // --- + // --- - // our return value (the last timestamp printed) - // this is required to detect re-transmit in google JSONP - time_t last_timestamp = 0; + // our return value (the last timestamp printed) + // this is required to detect re-transmit in google JSONP + time_t last_timestamp = 0; - // ------------------------------------------------------------------------- - // find how many dimensions we have + // ------------------------------------------------------------------------- + // find how many dimensions we have - int dimensions = 0; - RRDDIM *rd; - for( rd = st->dimensions ; rd ; rd = rd->next) dimensions++; - if(!dimensions) { - pthread_rwlock_unlock(&st->rwlock); - buffer_strcat(wb, "No dimensions yet."); - return 0; - } + int dimensions = 0; + RRDDIM *rd; + for( rd = st->dimensions ; rd ; rd = rd->next) dimensions++; + if(!dimensions) { + pthread_rwlock_unlock(&st->rwlock); + buffer_strcat(wb, "No dimensions yet."); + return 0; + } - // ------------------------------------------------------------------------- - // prepare various strings, to speed up the loop + // ------------------------------------------------------------------------- + // prepare various strings, to speed up the loop - char overflow_annotation[201]; snprintfz(overflow_annotation, 200, ",{%sv%s:%sRESET OR OVERFLOW%s},{%sv%s:%sThe counters have been wrapped.%s}", kq, kq, sq, sq, kq, kq, sq, sq); - char normal_annotation[201]; snprintfz(normal_annotation, 200, ",{%sv%s:null},{%sv%s:null}", kq, kq, kq, kq); - char pre_date[51]; snprintfz(pre_date, 50, " {%sc%s:[{%sv%s:%s", kq, kq, kq, kq, sq); - char post_date[21]; snprintfz(post_date, 20, "%s}", sq); - char pre_value[21]; snprintfz(pre_value, 20, ",{%sv%s:", kq, kq); - char post_value[21]; strcpy(post_value, "}"); + char overflow_annotation[201]; snprintfz(overflow_annotation, 200, ",{%sv%s:%sRESET OR OVERFLOW%s},{%sv%s:%sThe counters have been wrapped.%s}", kq, kq, sq, sq, kq, kq, sq, sq); + char normal_annotation[201]; snprintfz(normal_annotation, 200, ",{%sv%s:null},{%sv%s:null}", kq, kq, kq, kq); + char pre_date[51]; snprintfz(pre_date, 50, " {%sc%s:[{%sv%s:%s", kq, kq, kq, kq, sq); + char post_date[21]; snprintfz(post_date, 20, "%s}", sq); + char pre_value[21]; snprintfz(pre_value, 20, ",{%sv%s:", kq, kq); + char post_value[21]; strcpy(post_value, "}"); - // ------------------------------------------------------------------------- - // checks for debugging + // ------------------------------------------------------------------------- + // checks for debugging - if(st->debug) { - debug(D_RRD_STATS, "%s first_entry_t = %lu, last_entry_t = %lu, duration = %lu, after = %lu, before = %lu, duration = %lu, entries_to_show = %lu, group = %lu" - , st->id - , rrdset_first_entry_t(st) - , rrdset_last_entry_t(st) - , rrdset_last_entry_t(st) - rrdset_first_entry_t(st) - , after - , before - , before - after - , points - , group - ); + if(st->debug) { + debug(D_RRD_STATS, "%s first_entry_t = %ld, last_entry_t = %ld, duration = %ld, after = %ld, before = %ld, duration = %ld, entries_to_show = %ld, group = %ld" + , st->id + , rrdset_first_entry_t(st) + , rrdset_last_entry_t(st) + , rrdset_last_entry_t(st) - rrdset_first_entry_t(st) + , after + , before + , before - after + , points + , group + ); - if(before < after) - debug(D_RRD_STATS, "WARNING: %s The newest value in the database (%lu) is earlier than the oldest (%lu)", st->name, before, after); + if(before < after) + debug(D_RRD_STATS, "WARNING: %s The newest value in the database (%ld) is earlier than the oldest (%ld)", st->name, before, after); - if((before - after) > st->entries * st->update_every) - debug(D_RRD_STATS, "WARNING: %s The time difference between the oldest and the newest entries (%lu) is higher than the capacity of the database (%lu)", st->name, before - after, st->entries * st->update_every); - } - - - // ------------------------------------------------------------------------- - // temp arrays for keeping values per dimension - - calculated_number group_values[dimensions]; // keep sums when grouping - int print_hidden[dimensions]; // keep hidden flags - int found_non_zero[dimensions]; - int found_non_existing[dimensions]; + if((before - after) > st->entries * st->update_every) + debug(D_RRD_STATS, "WARNING: %s The time difference between the oldest and the newest entries (%ld) is higher than the capacity of the database (%ld)", st->name, before - after, st->entries * st->update_every); + } + + + // ------------------------------------------------------------------------- + // temp arrays for keeping values per dimension + + calculated_number group_values[dimensions]; // keep sums when grouping + int print_hidden[dimensions]; // keep hidden flags + int found_non_zero[dimensions]; + int found_non_existing[dimensions]; - // initialize them - for( rd = st->dimensions, c = 0 ; rd && c < dimensions ; rd = rd->next, c++) { - group_values[c] = 0; - print_hidden[c] = (rd->flags & RRDDIM_FLAG_HIDDEN)?1:0; - found_non_zero[c] = 0; - found_non_existing[c] = 0; - } + // initialize them + for( rd = st->dimensions, c = 0 ; rd && c < dimensions ; rd = rd->next, c++) { + group_values[c] = 0; + print_hidden[c] = (rd->flags & RRDDIM_FLAG_HIDDEN)?1:0; + found_non_zero[c] = 0; + found_non_existing[c] = 0; + } - // error("OLD: points=%d after=%d before=%d group=%d, duration=%d", entries_to_show, before - (st->update_every * group * entries_to_show), before, group, before - after + 1); - // rrd2array(st, entries_to_show, before - (st->update_every * group * entries_to_show), before, group_method, only_non_zero); - // rrd2rrdr(st, entries_to_show, before - (st->update_every * group * entries_to_show), before, group_method); - - // ------------------------------------------------------------------------- - // remove dimensions that contain only zeros - - int max_loop = 1; - if(only_non_zero) max_loop = 2; - - for(; max_loop ; max_loop--) { - - // ------------------------------------------------------------------------- - // print the JSON header - - buffer_sprintf(wb, "{\n %scols%s:\n [\n", kq, kq); - buffer_sprintf(wb, " {%sid%s:%s%s,%slabel%s:%stime%s,%spattern%s:%s%s,%stype%s:%sdatetime%s},\n", kq, kq, sq, sq, kq, kq, sq, sq, kq, kq, sq, sq, kq, kq, sq, sq); - buffer_sprintf(wb, " {%sid%s:%s%s,%slabel%s:%s%s,%spattern%s:%s%s,%stype%s:%sstring%s,%sp%s:{%srole%s:%sannotation%s}},\n", kq, kq, sq, sq, kq, kq, sq, sq, kq, kq, sq, sq, kq, kq, sq, sq, kq, kq, kq, kq, sq, sq); - buffer_sprintf(wb, " {%sid%s:%s%s,%slabel%s:%s%s,%spattern%s:%s%s,%stype%s:%sstring%s,%sp%s:{%srole%s:%sannotationText%s}}", kq, kq, sq, sq, kq, kq, sq, sq, kq, kq, sq, sq, kq, kq, sq, sq, kq, kq, kq, kq, sq, sq); - - // print the header for each dimension - // and update the print_hidden array for the dimensions that should be hidden - int pc = 0; - for( rd = st->dimensions, c = 0 ; rd && c < dimensions ; rd = rd->next, c++) { - if(!print_hidden[c]) { - pc++; - buffer_sprintf(wb, ",\n {%sid%s:%s%s,%slabel%s:%s%s%s,%spattern%s:%s%s,%stype%s:%snumber%s}", kq, kq, sq, sq, kq, kq, sq, rd->name, sq, kq, kq, sq, sq, kq, kq, sq, sq); - } - } - if(!pc) { - buffer_sprintf(wb, ",\n {%sid%s:%s%s,%slabel%s:%s%s%s,%spattern%s:%s%s,%stype%s:%snumber%s}", kq, kq, sq, sq, kq, kq, sq, "no data", sq, kq, kq, sq, sq, kq, kq, sq, sq); - } - - // print the begin of row data - buffer_sprintf(wb, "\n ],\n %srows%s:\n [\n", kq, kq); - - - // ------------------------------------------------------------------------- - // the main loop - - int annotate_reset = 0; - int annotation_count = 0; - - long t = rrdset_time2slot(st, before), - stop_at_t = rrdset_time2slot(st, after), - stop_now = 0; - - t -= t % group; - - time_t now = rrdset_slot2time(st, t), - dt = st->update_every; - - long count = 0, printed = 0, group_count = 0; - last_timestamp = 0; - - if(st->debug) debug(D_RRD_STATS, "%s: REQUEST after:%lu before:%lu, points:%d, group:%d, CHART cur:%ld first: %lu last:%lu, CALC start_t:%ld, stop_t:%ld" - , st->id - , after - , before - , points - , group - , st->current_entry - , rrdset_first_entry_t(st) - , rrdset_last_entry_t(st) - , t - , stop_at_t - ); - - long counter = 0; - for(; !stop_now ; now -= dt, t--, counter++) { - if(t < 0) t = st->entries - 1; - if(t == stop_at_t) stop_now = counter; - - int print_this = 0; - - if(st->debug) debug(D_RRD_STATS, "%s t = %ld, count = %ld, group_count = %ld, printed = %ld, now = %lu, %s %s" - , st->id - , t - , count + 1 - , group_count + 1 - , printed - , now - , (group_count + 1 == group)?"PRINT":" - " - , (now >= after && now <= before)?"RANGE":" - " - ); - - - // make sure we return data in the proper time range - if(now > before) continue; - if(now < after) break; - - //if(rrdset_slot2time(st, t) != now) - // error("%s: slot=%ld, now=%ld, slot2time=%ld, diff=%ld, last_entry_t=%ld, rrdset_last_slot=%ld", st->id, t, now, rrdset_slot2time(st,t), now - rrdset_slot2time(st,t), rrdset_last_entry_t(st), rrdset_last_slot(st)); - - count++; - group_count++; - - // check if we have to print this now - if(group_count == group) { - if(printed >= points) { - // debug(D_RRD_STATS, "Already printed all rows. Stopping."); - break; - } - - // generate the local date time - struct tm tmbuf, *tm = localtime_r(&now, &tmbuf); - if(!tm) { error("localtime() failed."); continue; } - if(now > last_timestamp) last_timestamp = now; - - if(printed) buffer_strcat(wb, "]},\n"); - buffer_strcat(wb, pre_date); - buffer_jsdate(wb, tm->tm_year + 1900, tm->tm_mon, tm->tm_mday, tm->tm_hour, tm->tm_min, tm->tm_sec); - buffer_strcat(wb, post_date); - - print_this = 1; - } - - // do the calculations - for(rd = st->dimensions, c = 0 ; rd && c < dimensions ; rd = rd->next, c++) { - storage_number n = rd->values[t]; - calculated_number value = unpack_storage_number(n); - - if(!does_storage_number_exist(n)) { - value = 0.0; - found_non_existing[c]++; - } - if(did_storage_number_reset(n)) annotate_reset = 1; - - switch(group_method) { - case GROUP_MAX: - if(abs(value) > abs(group_values[c])) group_values[c] = value; - break; - - case GROUP_SUM: - group_values[c] += value; - break; - - default: - case GROUP_AVERAGE: - group_values[c] += value; - if(print_this) group_values[c] /= ( group_count - found_non_existing[c] ); - break; - } - } - - if(print_this) { - if(annotate_reset) { - annotation_count++; - buffer_strcat(wb, overflow_annotation); - annotate_reset = 0; - } - else - buffer_strcat(wb, normal_annotation); - - pc = 0; - for(c = 0 ; c < dimensions ; c++) { - if(found_non_existing[c] == group_count) { - // all entries are non-existing - pc++; - buffer_strcat(wb, pre_value); - buffer_strcat(wb, "null"); - buffer_strcat(wb, post_value); - } - else if(!print_hidden[c]) { - pc++; - buffer_strcat(wb, pre_value); - buffer_rrd_value(wb, group_values[c]); - buffer_strcat(wb, post_value); - - if(group_values[c]) found_non_zero[c]++; - } - - // reset them for the next loop - group_values[c] = 0; - found_non_existing[c] = 0; - } - - // if all dimensions are hidden, print a null - if(!pc) { - buffer_strcat(wb, pre_value); - buffer_strcat(wb, "null"); - buffer_strcat(wb, post_value); - } - - printed++; - group_count = 0; - } - } - - if(printed) buffer_strcat(wb, "]}"); - buffer_strcat(wb, "\n ]\n}\n"); - - if(only_non_zero && max_loop > 1) { - int changed = 0; - for(rd = st->dimensions, c = 0 ; rd && c < dimensions ; rd = rd->next, c++) { - group_values[c] = 0; - found_non_existing[c] = 0; - - if(!print_hidden[c] && !found_non_zero[c]) { - changed = 1; - print_hidden[c] = 1; - } - } - - if(changed) buffer_flush(wb); - else break; - } - else break; - - } // max_loop - - debug(D_RRD_STATS, "RRD_STATS_JSON: %s total %ld bytes", st->name, wb->len); - - pthread_rwlock_unlock(&st->rwlock); - return last_timestamp; + // error("OLD: points=%d after=%d before=%d group=%d, duration=%d", entries_to_show, before - (st->update_every * group * entries_to_show), before, group, before - after + 1); + // rrd2array(st, entries_to_show, before - (st->update_every * group * entries_to_show), before, group_method, only_non_zero); + // rrd2rrdr(st, entries_to_show, before - (st->update_every * group * entries_to_show), before, group_method); + + // ------------------------------------------------------------------------- + // remove dimensions that contain only zeros + + int max_loop = 1; + if(only_non_zero) max_loop = 2; + + for(; max_loop ; max_loop--) { + + // ------------------------------------------------------------------------- + // print the JSON header + + buffer_sprintf(wb, "{\n %scols%s:\n [\n", kq, kq); + buffer_sprintf(wb, " {%sid%s:%s%s,%slabel%s:%stime%s,%spattern%s:%s%s,%stype%s:%sdatetime%s},\n", kq, kq, sq, sq, kq, kq, sq, sq, kq, kq, sq, sq, kq, kq, sq, sq); + buffer_sprintf(wb, " {%sid%s:%s%s,%slabel%s:%s%s,%spattern%s:%s%s,%stype%s:%sstring%s,%sp%s:{%srole%s:%sannotation%s}},\n", kq, kq, sq, sq, kq, kq, sq, sq, kq, kq, sq, sq, kq, kq, sq, sq, kq, kq, kq, kq, sq, sq); + buffer_sprintf(wb, " {%sid%s:%s%s,%slabel%s:%s%s,%spattern%s:%s%s,%stype%s:%sstring%s,%sp%s:{%srole%s:%sannotationText%s}}", kq, kq, sq, sq, kq, kq, sq, sq, kq, kq, sq, sq, kq, kq, sq, sq, kq, kq, kq, kq, sq, sq); + + // print the header for each dimension + // and update the print_hidden array for the dimensions that should be hidden + int pc = 0; + for( rd = st->dimensions, c = 0 ; rd && c < dimensions ; rd = rd->next, c++) { + if(!print_hidden[c]) { + pc++; + buffer_sprintf(wb, ",\n {%sid%s:%s%s,%slabel%s:%s%s%s,%spattern%s:%s%s,%stype%s:%snumber%s}", kq, kq, sq, sq, kq, kq, sq, rd->name, sq, kq, kq, sq, sq, kq, kq, sq, sq); + } + } + if(!pc) { + buffer_sprintf(wb, ",\n {%sid%s:%s%s,%slabel%s:%s%s%s,%spattern%s:%s%s,%stype%s:%snumber%s}", kq, kq, sq, sq, kq, kq, sq, "no data", sq, kq, kq, sq, sq, kq, kq, sq, sq); + } + + // print the begin of row data + buffer_sprintf(wb, "\n ],\n %srows%s:\n [\n", kq, kq); + + + // ------------------------------------------------------------------------- + // the main loop + + int annotate_reset = 0; + int annotation_count = 0; + + long t = rrdset_time2slot(st, before), + stop_at_t = rrdset_time2slot(st, after), + stop_now = 0; + + t -= t % group; + + time_t now = rrdset_slot2time(st, t), + dt = st->update_every; + + long count = 0, printed = 0, group_count = 0; + last_timestamp = 0; + + if(st->debug) debug(D_RRD_STATS, "%s: REQUEST after:%u before:%u, points:%ld, group:%ld, CHART cur:%ld first: %u last:%u, CALC start_t:%ld, stop_t:%ld" + , st->id + , (uint32_t)after + , (uint32_t)before + , points + , group + , st->current_entry + , (uint32_t)rrdset_first_entry_t(st) + , (uint32_t)rrdset_last_entry_t(st) + , t + , stop_at_t + ); + + long counter = 0; + for(; !stop_now ; now -= dt, t--, counter++) { + if(t < 0) t = st->entries - 1; + if(t == stop_at_t) stop_now = counter; + + int print_this = 0; + + if(st->debug) debug(D_RRD_STATS, "%s t = %ld, count = %ld, group_count = %ld, printed = %ld, now = %ld, %s %s" + , st->id + , t + , count + 1 + , group_count + 1 + , printed + , now + , (group_count + 1 == group)?"PRINT":" - " + , (now >= after && now <= before)?"RANGE":" - " + ); + + + // make sure we return data in the proper time range + if(now > before) continue; + if(now < after) break; + + //if(rrdset_slot2time(st, t) != now) + // error("%s: slot=%ld, now=%ld, slot2time=%ld, diff=%ld, last_entry_t=%ld, rrdset_last_slot=%ld", st->id, t, now, rrdset_slot2time(st,t), now - rrdset_slot2time(st,t), rrdset_last_entry_t(st), rrdset_last_slot(st)); + + count++; + group_count++; + + // check if we have to print this now + if(group_count == group) { + if(printed >= points) { + // debug(D_RRD_STATS, "Already printed all rows. Stopping."); + break; + } + + // generate the local date time + struct tm tmbuf, *tm = localtime_r(&now, &tmbuf); + if(!tm) { error("localtime() failed."); continue; } + if(now > last_timestamp) last_timestamp = now; + + if(printed) buffer_strcat(wb, "]},\n"); + buffer_strcat(wb, pre_date); + buffer_jsdate(wb, tm->tm_year + 1900, tm->tm_mon, tm->tm_mday, tm->tm_hour, tm->tm_min, tm->tm_sec); + buffer_strcat(wb, post_date); + + print_this = 1; + } + + // do the calculations + for(rd = st->dimensions, c = 0 ; rd && c < dimensions ; rd = rd->next, c++) { + storage_number n = rd->values[t]; + calculated_number value = unpack_storage_number(n); + + if(!does_storage_number_exist(n)) { + value = 0.0; + found_non_existing[c]++; + } + if(did_storage_number_reset(n)) annotate_reset = 1; + + switch(group_method) { + case GROUP_MAX: + if(abs(value) > abs(group_values[c])) group_values[c] = value; + break; + + case GROUP_SUM: + group_values[c] += value; + break; + + default: + case GROUP_AVERAGE: + group_values[c] += value; + if(print_this) group_values[c] /= ( group_count - found_non_existing[c] ); + break; + } + } + + if(print_this) { + if(annotate_reset) { + annotation_count++; + buffer_strcat(wb, overflow_annotation); + annotate_reset = 0; + } + else + buffer_strcat(wb, normal_annotation); + + pc = 0; + for(c = 0 ; c < dimensions ; c++) { + if(found_non_existing[c] == group_count) { + // all entries are non-existing + pc++; + buffer_strcat(wb, pre_value); + buffer_strcat(wb, "null"); + buffer_strcat(wb, post_value); + } + else if(!print_hidden[c]) { + pc++; + buffer_strcat(wb, pre_value); + buffer_rrd_value(wb, group_values[c]); + buffer_strcat(wb, post_value); + + if(group_values[c]) found_non_zero[c]++; + } + + // reset them for the next loop + group_values[c] = 0; + found_non_existing[c] = 0; + } + + // if all dimensions are hidden, print a null + if(!pc) { + buffer_strcat(wb, pre_value); + buffer_strcat(wb, "null"); + buffer_strcat(wb, post_value); + } + + printed++; + group_count = 0; + } + } + + if(printed) buffer_strcat(wb, "]}"); + buffer_strcat(wb, "\n ]\n}\n"); + + if(only_non_zero && max_loop > 1) { + int changed = 0; + for(rd = st->dimensions, c = 0 ; rd && c < dimensions ; rd = rd->next, c++) { + group_values[c] = 0; + found_non_existing[c] = 0; + + if(!print_hidden[c] && !found_non_zero[c]) { + changed = 1; + print_hidden[c] = 1; + } + } + + if(changed) buffer_flush(wb); + else break; + } + else break; + + } // max_loop + + debug(D_RRD_STATS, "RRD_STATS_JSON: %s total %lu bytes", st->name, wb->len); + + pthread_rwlock_unlock(&st->rwlock); + return last_timestamp; } diff --git a/src/rrd2json.h b/src/rrd2json.h index d1f850d4a..e3e5cb690 100644 --- a/src/rrd2json.h +++ b/src/rrd2json.h @@ -1,8 +1,3 @@ -#include <time.h> - -#include "web_buffer.h" -#include "rrd.h" - #ifndef NETDATA_RRD2JSON_H #define NETDATA_RRD2JSON_H 1 @@ -35,22 +30,26 @@ extern char *hostname; #define DATASOURCE_FORMAT_SSV_COMMA "ssvcomma" #define DATASOURCE_FORMAT_CSV_JSON_ARRAY "csvjsonarray" -#define GROUP_AVERAGE 0 -#define GROUP_MAX 1 -#define GROUP_SUM 2 +#define GROUP_UNDEFINED 0 +#define GROUP_AVERAGE 1 +#define GROUP_MIN 2 +#define GROUP_MAX 3 +#define GROUP_SUM 4 +#define GROUP_INCREMENTAL_SUM 5 -#define RRDR_OPTION_NONZERO 0x00000001 // don't output dimensions will just zero values -#define RRDR_OPTION_REVERSED 0x00000002 // output the rows in reverse order (oldest to newest) -#define RRDR_OPTION_ABSOLUTE 0x00000004 // values positive, for DATASOURCE_SSV before summing -#define RRDR_OPTION_MIN2MAX 0x00000008 // for DATASOURCE_SSV, out max - min, instead of sum -#define RRDR_OPTION_SECONDS 0x00000010 // output seconds, instead of dates -#define RRDR_OPTION_MILLISECONDS 0x00000020 // output milliseconds, instead of dates -#define RRDR_OPTION_NULL2ZERO 0x00000040 // do not show nulls, convert them to zeros -#define RRDR_OPTION_OBJECTSROWS 0x00000080 // each row of values should be an object, not an array -#define RRDR_OPTION_GOOGLE_JSON 0x00000100 // comply with google JSON/JSONP specs -#define RRDR_OPTION_JSON_WRAP 0x00000200 // wrap the response in a JSON header with info about the result -#define RRDR_OPTION_LABEL_QUOTES 0x00000400 // in CSV output, wrap header labels in double quotes -#define RRDR_OPTION_PERCENTAGE 0x00000800 // give values as percentage of total +#define RRDR_OPTION_NONZERO 0x00000001 // don't output dimensions will just zero values +#define RRDR_OPTION_REVERSED 0x00000002 // output the rows in reverse order (oldest to newest) +#define RRDR_OPTION_ABSOLUTE 0x00000004 // values positive, for DATASOURCE_SSV before summing +#define RRDR_OPTION_MIN2MAX 0x00000008 // when adding dimensions, use max - min, instead of sum +#define RRDR_OPTION_SECONDS 0x00000010 // output seconds, instead of dates +#define RRDR_OPTION_MILLISECONDS 0x00000020 // output milliseconds, instead of dates +#define RRDR_OPTION_NULL2ZERO 0x00000040 // do not show nulls, convert them to zeros +#define RRDR_OPTION_OBJECTSROWS 0x00000080 // each row of values should be an object, not an array +#define RRDR_OPTION_GOOGLE_JSON 0x00000100 // comply with google JSON/JSONP specs +#define RRDR_OPTION_JSON_WRAP 0x00000200 // wrap the response in a JSON header with info about the result +#define RRDR_OPTION_LABEL_QUOTES 0x00000400 // in CSV output, wrap header labels in double quotes +#define RRDR_OPTION_PERCENTAGE 0x00000800 // give values as percentage of total +#define RRDR_OPTION_NOT_ALIGNED 0x00001000 // do not align charts for persistant timeframes extern void rrd_stats_api_v1_chart(RRDSET *st, BUFFER *wb); extern void rrd_stats_api_v1_charts(BUFFER *wb); @@ -64,6 +63,6 @@ extern void rrd_stats_all_json(BUFFER *wb); extern time_t rrd_stats_json(int type, RRDSET *st, BUFFER *wb, long entries_to_show, long group, int group_method, time_t after, time_t before, int only_non_zero); extern int rrd2format(RRDSET *st, BUFFER *out, BUFFER *dimensions, uint32_t format, long points, long long after, long long before, int group_method, uint32_t options, time_t *latest_timestamp); - +extern int rrd2value(RRDSET *st, BUFFER *wb, calculated_number *n, const char *dimensions, long points, long long after, long long before, int group_method, uint32_t options, time_t *db_before, time_t *db_after, int *value_is_null); #endif /* NETDATA_RRD2JSON_H */ diff --git a/src/storage_number.c b/src/storage_number.c index b5c5f4067..27fe5f2c7 100644 --- a/src/storage_number.c +++ b/src/storage_number.c @@ -1,206 +1,168 @@ -#ifdef HAVE_CONFIG_H -#include <config.h> -#endif -#ifdef STORAGE_WITH_MATH -#include <math.h> -#endif - #include "common.h" -#include "log.h" -#include "storage_number.h" -#if __GNUC__ -#if __x86_64__ || __ppc64__ -#define ENVIRONMENT64 -#else -#define ENVIRONMENT32 -#endif -#endif +extern char *print_number_lu_r(char *str, unsigned long uvalue); +extern char *print_number_llu_r(char *str, unsigned long long uvalue); storage_number pack_storage_number(calculated_number value, uint32_t flags) { - // bit 32 = sign 0:positive, 1:negative - // bit 31 = 0:divide, 1:multiply - // bit 30, 29, 28 = (multiplier or divider) 0-6 (7 total) - // bit 27, 26, 25 flags - // bit 24 to bit 1 = the value - - storage_number r = get_storage_number_flags(flags); - if(!value) return r; - - int m = 0; - calculated_number n = value; - - // if the value is negative - // add the sign bit and make it positive - if(n < 0) { - r += (1 << 31); // the sign bit 32 - n = -n; - } - - // make its integer part fit in 0x00ffffff - // by dividing it by 10 up to 7 times - // and increasing the multiplier - while(m < 7 && n > (calculated_number)0x00ffffff) { - n /= 10; - m++; - } - - if(m) { - // the value was too big and we divided it - // so we add a multiplier to unpack it - r += (1 << 30) + (m << 27); // the multiplier m - - if(n > (calculated_number)0x00ffffff) { - error("Number " CALCULATED_NUMBER_FORMAT " is too big.", value); - r += 0x00ffffff; - return r; - } - } - else { - // 0x0019999e is the number that can be multiplied - // by 10 to give 0x00ffffff - // while the value is below 0x0019999e we can - // multiply it by 10, up to 7 times, increasing - // the multiplier - while(m < 7 && n < (calculated_number)0x0019999e) { - n *= 10; - m++; - } - - // the value was small enough and we multiplied it - // so we add a divider to unpack it - r += (0 << 30) + (m << 27); // the divider m - } + // bit 32 = sign 0:positive, 1:negative + // bit 31 = 0:divide, 1:multiply + // bit 30, 29, 28 = (multiplier or divider) 0-6 (7 total) + // bit 27, 26, 25 flags + // bit 24 to bit 1 = the value + + storage_number r = get_storage_number_flags(flags); + if(!value) return r; + + int m = 0; + calculated_number n = value; + + // if the value is negative + // add the sign bit and make it positive + if(n < 0) { + r += (1 << 31); // the sign bit 32 + n = -n; + } + + // make its integer part fit in 0x00ffffff + // by dividing it by 10 up to 7 times + // and increasing the multiplier + while(m < 7 && n > (calculated_number)0x00ffffff) { + n /= 10; + m++; + } + + if(m) { + // the value was too big and we divided it + // so we add a multiplier to unpack it + r += (1 << 30) + (m << 27); // the multiplier m + + if(n > (calculated_number)0x00ffffff) { + error("Number " CALCULATED_NUMBER_FORMAT " is too big.", value); + r += 0x00ffffff; + return r; + } + } + else { + // 0x0019999e is the number that can be multiplied + // by 10 to give 0x00ffffff + // while the value is below 0x0019999e we can + // multiply it by 10, up to 7 times, increasing + // the multiplier + while(m < 7 && n < (calculated_number)0x0019999e) { + n *= 10; + m++; + } + + // the value was small enough and we multiplied it + // so we add a divider to unpack it + r += (0 << 30) + (m << 27); // the divider m + } #ifdef STORAGE_WITH_MATH - // without this there are rounding problems - // example: 0.9 becomes 0.89 - r += lrint((double) n); + // without this there are rounding problems + // example: 0.9 becomes 0.89 + r += lrint((double) n); #else - r += (storage_number)n; + r += (storage_number)n; #endif - return r; + return r; } calculated_number unpack_storage_number(storage_number value) { - if(!value) return 0; - - int sign = 0, exp = 0; + if(!value) return 0; - value ^= get_storage_number_flags(value); + int sign = 0, exp = 0; - if(value & (1 << 31)) { - sign = 1; - value ^= 1 << 31; - } + value ^= get_storage_number_flags(value); - if(value & (1 << 30)) { - exp = 1; - value ^= 1 << 30; - } + if(value & (1 << 31)) { + sign = 1; + value ^= 1 << 31; + } - int mul = value >> 27; - value ^= mul << 27; + if(value & (1 << 30)) { + exp = 1; + value ^= 1 << 30; + } - calculated_number n = value; + int mul = value >> 27; + value ^= mul << 27; - // fprintf(stderr, "UNPACK: %08X, sign = %d, exp = %d, mul = %d, n = " CALCULATED_NUMBER_FORMAT "\n", value, sign, exp, mul, n); + calculated_number n = value; - while(mul > 0) { - if(exp) n *= 10; - else n /= 10; - mul--; - } + // fprintf(stderr, "UNPACK: %08X, sign = %d, exp = %d, mul = %d, n = " CALCULATED_NUMBER_FORMAT "\n", value, sign, exp, mul, n); - if(sign) n = -n; - return n; -} - -#ifdef ENVIRONMENT32 -// This trick seems to give an 80% speed increase in 32bit systems -// print_calculated_number_llu_r() will just print the digits up to the -// point the remaining value fits in 32 bits, and then calls -// print_calculated_number_lu_r() to print the rest with 32 bit arithmetic. - -static char *print_calculated_number_lu_r(char *str, unsigned long uvalue) { - char *wstr = str; + while(mul > 0) { + if(exp) n *= 10; + else n /= 10; + mul--; + } - // print each digit - do *wstr++ = (char)('0' + (uvalue % 10)); while(uvalue /= 10); - return wstr; + if(sign) n = -n; + return n; } -static char *print_calculated_number_llu_r(char *str, unsigned long long uvalue) { - char *wstr = str; - - // print each digit - do *wstr++ = (char)('0' + (uvalue % 10)); while((uvalue /= 10) && uvalue > (unsigned long long)0xffffffff); - if(uvalue) return print_calculated_number_lu_r(wstr, uvalue); - return wstr; -} -#endif - int print_calculated_number(char *str, calculated_number value) { - char *wstr = str; + char *wstr = str; - int sign = (value < 0) ? 1 : 0; - if(sign) value = -value; + int sign = (value < 0) ? 1 : 0; + if(sign) value = -value; #ifdef STORAGE_WITH_MATH - // without llrint() there are rounding problems - // for example 0.9 becomes 0.89 - unsigned long long uvalue = (unsigned long long int) llrint(value * (calculated_number)100000); + // without llrint() there are rounding problems + // for example 0.9 becomes 0.89 + unsigned long long uvalue = (unsigned long long int) llrint(value * (calculated_number)100000); #else - unsigned long long uvalue = value * (calculated_number)100000; + unsigned long long uvalue = value * (calculated_number)100000; #endif #ifdef ENVIRONMENT32 - if(uvalue > (unsigned long long)0xffffffff) - wstr = print_calculated_number_llu_r(str, uvalue); - else - wstr = print_calculated_number_lu_r(str, uvalue); + if(uvalue > (unsigned long long)0xffffffff) + wstr = print_number_llu_r(str, uvalue); + else + wstr = print_number_lu_r(str, uvalue); #else - do *wstr++ = (char)('0' + (uvalue % 10)); while(uvalue /= 10); + do *wstr++ = (char)('0' + (uvalue % 10)); while(uvalue /= 10); #endif - // make sure we have 6 bytes at least - while((wstr - str) < 6) *wstr++ = '0'; + // make sure we have 6 bytes at least + while((wstr - str) < 6) *wstr++ = '0'; - // put the sign back - if(sign) *wstr++ = '-'; + // put the sign back + if(sign) *wstr++ = '-'; - // reverse it + // reverse it char *begin = str, *end = --wstr, aux; while (end > begin) aux = *end, *end-- = *begin, *begin++ = aux; - // wstr--; - // strreverse(str, wstr); - - // remove trailing zeros - int decimal = 5; - while(decimal > 0 && *wstr == '0') { - *wstr-- = '\0'; - decimal--; - } - - // terminate it, one position to the right - // to let space for a dot - wstr[2] = '\0'; - - // make space for the dot - int i; - for(i = 0; i < decimal ;i++) { - wstr[1] = wstr[0]; - wstr--; - } - - // put the dot - if(wstr[2] == '\0') { wstr[1] = '\0'; decimal--; } - else wstr[1] = '.'; - - // return the buffer length - return (int) ((wstr - str) + 2 + decimal ); + // wstr--; + // strreverse(str, wstr); + + // remove trailing zeros + int decimal = 5; + while(decimal > 0 && *wstr == '0') { + *wstr-- = '\0'; + decimal--; + } + + // terminate it, one position to the right + // to let space for a dot + wstr[2] = '\0'; + + // make space for the dot + int i; + for(i = 0; i < decimal ;i++) { + wstr[1] = wstr[0]; + wstr--; + } + + // put the dot + if(wstr[2] == '\0') { wstr[1] = '\0'; decimal--; } + else wstr[1] = '.'; + + // return the buffer length + return (int) ((wstr - str) + 2 + decimal ); } diff --git a/src/storage_number.h b/src/storage_number.h index f35e99ddd..74d24a322 100644 --- a/src/storage_number.h +++ b/src/storage_number.h @@ -14,19 +14,18 @@ typedef long double collected_number; #define COLLECTED_NUMBER_FORMAT "%0.7Lf" */ -typedef int32_t storage_number; -typedef uint32_t ustorage_number; -#define STORAGE_NUMBER_FORMAT "%d" - -#define SN_NOT_EXISTS (0x0 << 24) -#define SN_EXISTS (0x1 << 24) -#define SN_EXISTS_RESET (0x2 << 24) -#define SN_EXISTS_UNDEF1 (0x3 << 24) -#define SN_EXISTS_UNDEF2 (0x4 << 24) -#define SN_EXISTS_UNDEF3 (0x5 << 24) -#define SN_EXISTS_UNDEF4 (0x6 << 24) - -#define SN_FLAGS_MASK (~(0x6 << 24)) +typedef uint32_t storage_number; +#define STORAGE_NUMBER_FORMAT "%u" + +#define SN_NOT_EXISTS (0x0 << 24) +#define SN_EXISTS (0x1 << 24) +#define SN_EXISTS_RESET (0x2 << 24) +#define SN_EXISTS_UNDEF1 (0x3 << 24) +#define SN_EXISTS_UNDEF2 (0x4 << 24) +#define SN_EXISTS_UNDEF3 (0x5 << 24) +#define SN_EXISTS_UNDEF4 (0x6 << 24) + +#define SN_FLAGS_MASK (~(0x6 << 24)) // extract the flags #define get_storage_number_flags(value) ((((storage_number)value) & (1 << 24)) | (((storage_number)value) & (2 << 24)) | (((storage_number)value) & (4 << 24))) diff --git a/src/sys_fs_cgroup.c b/src/sys_fs_cgroup.c index 9f3d3f0fd..892f737c1 100644 --- a/src/sys_fs_cgroup.c +++ b/src/sys_fs_cgroup.c @@ -1,22 +1,4 @@ -#ifdef HAVE_CONFIG_H -#include <config.h> -#endif - -#include <stdio.h> -#include <stdlib.h> -#include <inttypes.h> -#include <sys/types.h> -#include <dirent.h> -#include <string.h> - #include "common.h" -#include "appconfig.h" -#include "procfile.h" -#include "log.h" -#include "rrd.h" -#include "main.h" -#include "popen.h" -#include "proc_self_mountinfo.h" // ---------------------------------------------------------------------------- // cgroup globals @@ -24,164 +6,187 @@ static int cgroup_enable_cpuacct_stat = CONFIG_ONDEMAND_ONDEMAND; static int cgroup_enable_cpuacct_usage = CONFIG_ONDEMAND_ONDEMAND; static int cgroup_enable_memory = CONFIG_ONDEMAND_ONDEMAND; +static int cgroup_enable_devices = CONFIG_ONDEMAND_ONDEMAND; static int cgroup_enable_blkio = CONFIG_ONDEMAND_ONDEMAND; static int cgroup_enable_new_cgroups_detected_at_runtime = 1; static int cgroup_check_for_new_every = 10; static char *cgroup_cpuacct_base = NULL; static char *cgroup_blkio_base = NULL; static char *cgroup_memory_base = NULL; +static char *cgroup_devices_base = NULL; static int cgroup_root_count = 0; static int cgroup_root_max = 500; static int cgroup_max_depth = 0; void read_cgroup_plugin_configuration() { - cgroup_check_for_new_every = config_get_number("plugin:cgroups", "check for new cgroups every", cgroup_check_for_new_every); - - cgroup_enable_cpuacct_stat = config_get_boolean_ondemand("plugin:cgroups", "enable cpuacct stat", cgroup_enable_cpuacct_stat); - cgroup_enable_cpuacct_usage = config_get_boolean_ondemand("plugin:cgroups", "enable cpuacct usage", cgroup_enable_cpuacct_usage); - cgroup_enable_memory = config_get_boolean_ondemand("plugin:cgroups", "enable memory", cgroup_enable_memory); - cgroup_enable_blkio = config_get_boolean_ondemand("plugin:cgroups", "enable blkio", cgroup_enable_blkio); - - char filename[FILENAME_MAX + 1], *s; - struct mountinfo *mi, *root = mountinfo_read(); - - mi = mountinfo_find_by_filesystem_mount_source(root, "cgroup", "cpuacct"); - if(!mi) mi = mountinfo_find_by_filesystem_super_option(root, "cgroup", "cpuacct"); - if(!mi) s = "/sys/fs/cgroup/cpuacct"; - else s = mi->mount_point; - snprintfz(filename, FILENAME_MAX, "%s%s", global_host_prefix, s); - cgroup_cpuacct_base = config_get("plugin:cgroups", "path to /sys/fs/cgroup/cpuacct", filename); - - mi = mountinfo_find_by_filesystem_mount_source(root, "cgroup", "blkio"); - if(!mi) mi = mountinfo_find_by_filesystem_super_option(root, "cgroup", "blkio"); - if(!mi) s = "/sys/fs/cgroup/blkio"; - else s = mi->mount_point; - snprintfz(filename, FILENAME_MAX, "%s%s", global_host_prefix, s); - cgroup_blkio_base = config_get("plugin:cgroups", "path to /sys/fs/cgroup/blkio", filename); - - mi = mountinfo_find_by_filesystem_mount_source(root, "cgroup", "memory"); - if(!mi) mi = mountinfo_find_by_filesystem_super_option(root, "cgroup", "memory"); - if(!mi) s = "/sys/fs/cgroup/memory"; - else s = mi->mount_point; - snprintfz(filename, FILENAME_MAX, "%s%s", global_host_prefix, s); - cgroup_memory_base = config_get("plugin:cgroups", "path to /sys/fs/cgroup/memory", filename); - - cgroup_root_max = config_get_number("plugin:cgroups", "max cgroups to allow", cgroup_root_max); - cgroup_max_depth = config_get_number("plugin:cgroups", "max cgroups depth to monitor", cgroup_max_depth); - - cgroup_enable_new_cgroups_detected_at_runtime = config_get_boolean("plugin:cgroups", "enable new cgroups detected at run time", cgroup_enable_new_cgroups_detected_at_runtime); - - mountinfo_free(root); + cgroup_check_for_new_every = config_get_number("plugin:cgroups", "check for new cgroups every", cgroup_check_for_new_every); + + cgroup_enable_cpuacct_stat = config_get_boolean_ondemand("plugin:cgroups", "enable cpuacct stat", cgroup_enable_cpuacct_stat); + cgroup_enable_cpuacct_usage = config_get_boolean_ondemand("plugin:cgroups", "enable cpuacct usage", cgroup_enable_cpuacct_usage); + cgroup_enable_memory = config_get_boolean_ondemand("plugin:cgroups", "enable memory", cgroup_enable_memory); + cgroup_enable_blkio = config_get_boolean_ondemand("plugin:cgroups", "enable blkio", cgroup_enable_blkio); + + char filename[FILENAME_MAX + 1], *s; + struct mountinfo *mi, *root = mountinfo_read(); + + mi = mountinfo_find_by_filesystem_super_option(root, "cgroup", "cpuacct"); + if(!mi) mi = mountinfo_find_by_filesystem_mount_source(root, "cgroup", "cpuacct"); + if(!mi) { + error("Cannot find cgroup cpuacct mountinfo. Assuming default: /sys/fs/cgroup/cpuacct"); + s = "/sys/fs/cgroup/cpuacct"; + } + else s = mi->mount_point; + snprintfz(filename, FILENAME_MAX, "%s%s", global_host_prefix, s); + cgroup_cpuacct_base = config_get("plugin:cgroups", "path to /sys/fs/cgroup/cpuacct", filename); + + mi = mountinfo_find_by_filesystem_super_option(root, "cgroup", "blkio"); + if(!mi) mi = mountinfo_find_by_filesystem_mount_source(root, "cgroup", "blkio"); + if(!mi) { + error("Cannot find cgroup blkio mountinfo. Assuming default: /sys/fs/cgroup/blkio"); + s = "/sys/fs/cgroup/blkio"; + } + else s = mi->mount_point; + snprintfz(filename, FILENAME_MAX, "%s%s", global_host_prefix, s); + cgroup_blkio_base = config_get("plugin:cgroups", "path to /sys/fs/cgroup/blkio", filename); + + mi = mountinfo_find_by_filesystem_super_option(root, "cgroup", "memory"); + if(!mi) mi = mountinfo_find_by_filesystem_mount_source(root, "cgroup", "memory"); + if(!mi) { + error("Cannot find cgroup memory mountinfo. Assuming default: /sys/fs/cgroup/memory"); + s = "/sys/fs/cgroup/memory"; + } + else s = mi->mount_point; + snprintfz(filename, FILENAME_MAX, "%s%s", global_host_prefix, s); + cgroup_memory_base = config_get("plugin:cgroups", "path to /sys/fs/cgroup/memory", filename); + + mi = mountinfo_find_by_filesystem_super_option(root, "cgroup", "devices"); + if(!mi) mi = mountinfo_find_by_filesystem_mount_source(root, "cgroup", "devices"); + if(!mi) { + error("Cannot find cgroup devices mountinfo. Assuming default: /sys/fs/cgroup/devices"); + s = "/sys/fs/cgroup/devices"; + } + else s = mi->mount_point; + snprintfz(filename, FILENAME_MAX, "%s%s", global_host_prefix, s); + cgroup_devices_base = config_get("plugin:cgroups", "path to /sys/fs/cgroup/devices", filename); + + cgroup_root_max = config_get_number("plugin:cgroups", "max cgroups to allow", cgroup_root_max); + cgroup_max_depth = config_get_number("plugin:cgroups", "max cgroups depth to monitor", cgroup_max_depth); + + cgroup_enable_new_cgroups_detected_at_runtime = config_get_boolean("plugin:cgroups", "enable new cgroups detected at run time", cgroup_enable_new_cgroups_detected_at_runtime); + + mountinfo_free(root); } // ---------------------------------------------------------------------------- // cgroup objects struct blkio { - int updated; + int updated; - char *filename; + char *filename; - unsigned long long Read; - unsigned long long Write; + unsigned long long Read; + unsigned long long Write; /* - unsigned long long Sync; - unsigned long long Async; - unsigned long long Total; + unsigned long long Sync; + unsigned long long Async; + unsigned long long Total; */ }; // https://www.kernel.org/doc/Documentation/cgroup-v1/memory.txt struct memory { - int updated; - - char *filename; - - int has_dirty_swap; - - unsigned long long cache; - unsigned long long rss; - unsigned long long rss_huge; - unsigned long long mapped_file; - unsigned long long writeback; - unsigned long long dirty; - unsigned long long swap; - unsigned long long pgpgin; - unsigned long long pgpgout; - unsigned long long pgfault; - unsigned long long pgmajfault; + int updated; + + char *filename; + + int has_dirty_swap; + + unsigned long long cache; + unsigned long long rss; + unsigned long long rss_huge; + unsigned long long mapped_file; + unsigned long long writeback; + unsigned long long dirty; + unsigned long long swap; + unsigned long long pgpgin; + unsigned long long pgpgout; + unsigned long long pgfault; + unsigned long long pgmajfault; /* - unsigned long long inactive_anon; - unsigned long long active_anon; - unsigned long long inactive_file; - unsigned long long active_file; - unsigned long long unevictable; - unsigned long long hierarchical_memory_limit; - unsigned long long total_cache; - unsigned long long total_rss; - unsigned long long total_rss_huge; - unsigned long long total_mapped_file; - unsigned long long total_writeback; - unsigned long long total_dirty; - unsigned long long total_swap; - unsigned long long total_pgpgin; - unsigned long long total_pgpgout; - unsigned long long total_pgfault; - unsigned long long total_pgmajfault; - unsigned long long total_inactive_anon; - unsigned long long total_active_anon; - unsigned long long total_inactive_file; - unsigned long long total_active_file; - unsigned long long total_unevictable; + unsigned long long inactive_anon; + unsigned long long active_anon; + unsigned long long inactive_file; + unsigned long long active_file; + unsigned long long unevictable; + unsigned long long hierarchical_memory_limit; + unsigned long long total_cache; + unsigned long long total_rss; + unsigned long long total_rss_huge; + unsigned long long total_mapped_file; + unsigned long long total_writeback; + unsigned long long total_dirty; + unsigned long long total_swap; + unsigned long long total_pgpgin; + unsigned long long total_pgpgout; + unsigned long long total_pgfault; + unsigned long long total_pgmajfault; + unsigned long long total_inactive_anon; + unsigned long long total_active_anon; + unsigned long long total_inactive_file; + unsigned long long total_active_file; + unsigned long long total_unevictable; */ }; // https://www.kernel.org/doc/Documentation/cgroup-v1/cpuacct.txt struct cpuacct_stat { - int updated; + int updated; - char *filename; + char *filename; - unsigned long long user; - unsigned long long system; + unsigned long long user; + unsigned long long system; }; // https://www.kernel.org/doc/Documentation/cgroup-v1/cpuacct.txt struct cpuacct_usage { - int updated; + int updated; - char *filename; + char *filename; - unsigned int cpus; - unsigned long long *cpu_percpu; + unsigned int cpus; + unsigned long long *cpu_percpu; }; struct cgroup { - int available; // found in the filesystem - int enabled; // enabled in the config + int available; // found in the filesystem + int enabled; // enabled in the config + + char *id; + uint32_t hash; - char *id; - uint32_t hash; + char *chart_id; + uint32_t hash_chart; - char *chart_id; - char *chart_title; + char *chart_title; - struct cpuacct_stat cpuacct_stat; - struct cpuacct_usage cpuacct_usage; + struct cpuacct_stat cpuacct_stat; + struct cpuacct_usage cpuacct_usage; - struct memory memory; + struct memory memory; - struct blkio io_service_bytes; // bytes - struct blkio io_serviced; // operations + struct blkio io_service_bytes; // bytes + struct blkio io_serviced; // operations - struct blkio throttle_io_service_bytes; // bytes - struct blkio throttle_io_serviced; // operations + struct blkio throttle_io_service_bytes; // bytes + struct blkio throttle_io_serviced; // operations - struct blkio io_merged; // operations - struct blkio io_queued; // operations + struct blkio io_merged; // operations + struct blkio io_queued; // operations - struct cgroup *next; + struct cgroup *next; } *cgroup_root = NULL; @@ -189,393 +194,389 @@ struct cgroup { // read values from /sys void cgroup_read_cpuacct_stat(struct cpuacct_stat *cp) { - static procfile *ff = NULL; + static procfile *ff = NULL; - static uint32_t user_hash = 0; - static uint32_t system_hash = 0; + static uint32_t user_hash = 0; + static uint32_t system_hash = 0; - if(unlikely(user_hash == 0)) { - user_hash = simple_hash("user"); - system_hash = simple_hash("system"); - } + if(unlikely(user_hash == 0)) { + user_hash = simple_hash("user"); + system_hash = simple_hash("system"); + } - cp->updated = 0; - if(cp->filename) { - ff = procfile_reopen(ff, cp->filename, NULL, PROCFILE_FLAG_DEFAULT); - if(!ff) return; + cp->updated = 0; + if(cp->filename) { + ff = procfile_reopen(ff, cp->filename, NULL, PROCFILE_FLAG_DEFAULT); + if(!ff) return; - ff = procfile_readall(ff); - if(!ff) return; + ff = procfile_readall(ff); + if(!ff) return; - unsigned long i, lines = procfile_lines(ff); + unsigned long i, lines = procfile_lines(ff); - if(lines < 1) { - error("File '%s' should have 1+ lines.", cp->filename); - return; - } + if(lines < 1) { + error("File '%s' should have 1+ lines.", cp->filename); + return; + } - for(i = 0; i < lines ; i++) { - char *s = procfile_lineword(ff, i, 0); - uint32_t hash = simple_hash(s); + for(i = 0; i < lines ; i++) { + char *s = procfile_lineword(ff, i, 0); + uint32_t hash = simple_hash(s); - if(hash == user_hash && !strcmp(s, "user")) - cp->user = strtoull(procfile_lineword(ff, i, 1), NULL, 10); + if(hash == user_hash && !strcmp(s, "user")) + cp->user = strtoull(procfile_lineword(ff, i, 1), NULL, 10); - else if(hash == system_hash && !strcmp(s, "system")) - cp->system = strtoull(procfile_lineword(ff, i, 1), NULL, 10); - } + else if(hash == system_hash && !strcmp(s, "system")) + cp->system = strtoull(procfile_lineword(ff, i, 1), NULL, 10); + } - cp->updated = 1; + cp->updated = 1; - // fprintf(stderr, "READ '%s': user: %llu, system: %llu\n", cp->filename, cp->user, cp->system); - } + // fprintf(stderr, "READ '%s': user: %llu, system: %llu\n", cp->filename, cp->user, cp->system); + } } void cgroup_read_cpuacct_usage(struct cpuacct_usage *ca) { - static procfile *ff = NULL; - - ca->updated = 0; - if(ca->filename) { - ff = procfile_reopen(ff, ca->filename, NULL, PROCFILE_FLAG_DEFAULT); - if(!ff) return; - - ff = procfile_readall(ff); - if(!ff) return; - - if(procfile_lines(ff) < 1) { - error("File '%s' should have 1+ lines but has %d.", ca->filename, procfile_lines(ff)); - return; - } - - unsigned long i = procfile_linewords(ff, 0); - if(i <= 0) return; - - // we may have 1 more CPU reported - while(i > 0) { - char *s = procfile_lineword(ff, 0, i - 1); - if(!*s) i--; - else break; - } - - if(i != ca->cpus) { - free(ca->cpu_percpu); - - ca->cpu_percpu = malloc(sizeof(unsigned long long) * i); - if(!ca->cpu_percpu) - fatal("Cannot allocate memory (%z bytes)", sizeof(unsigned long long) * i); - - ca->cpus = i; - } - - for(i = 0; i < ca->cpus ;i++) { - ca->cpu_percpu[i] = strtoull(procfile_lineword(ff, 0, i), NULL, 10); - // fprintf(stderr, "READ '%s': cpu%d/%d: %llu ('%s')\n", ca->filename, i, ca->cpus, ca->cpu_percpu[i], procfile_lineword(ff, 0, i)); - } - - ca->updated = 1; - } + static procfile *ff = NULL; + + ca->updated = 0; + if(ca->filename) { + ff = procfile_reopen(ff, ca->filename, NULL, PROCFILE_FLAG_DEFAULT); + if(!ff) return; + + ff = procfile_readall(ff); + if(!ff) return; + + if(procfile_lines(ff) < 1) { + error("File '%s' should have 1+ lines but has %u.", ca->filename, procfile_lines(ff)); + return; + } + + unsigned long i = procfile_linewords(ff, 0); + if(i <= 0) return; + + // we may have 1 more CPU reported + while(i > 0) { + char *s = procfile_lineword(ff, 0, i - 1); + if(!*s) i--; + else break; + } + + if(i != ca->cpus) { + freez(ca->cpu_percpu); + ca->cpu_percpu = mallocz(sizeof(unsigned long long) * i); + ca->cpus = (unsigned int)i; + } + + for(i = 0; i < ca->cpus ;i++) { + ca->cpu_percpu[i] = strtoull(procfile_lineword(ff, 0, i), NULL, 10); + // fprintf(stderr, "READ '%s': cpu%d/%d: %llu ('%s')\n", ca->filename, i, ca->cpus, ca->cpu_percpu[i], procfile_lineword(ff, 0, i)); + } + + ca->updated = 1; + } } void cgroup_read_blkio(struct blkio *io) { - static procfile *ff = NULL; + static procfile *ff = NULL; - static uint32_t Read_hash = 0; - static uint32_t Write_hash = 0; + static uint32_t Read_hash = 0; + static uint32_t Write_hash = 0; /* - static uint32_t Sync_hash = 0; - static uint32_t Async_hash = 0; - static uint32_t Total_hash = 0; + static uint32_t Sync_hash = 0; + static uint32_t Async_hash = 0; + static uint32_t Total_hash = 0; */ - if(unlikely(Read_hash == 0)) { - Read_hash = simple_hash("Read"); - Write_hash = simple_hash("Write"); + if(unlikely(Read_hash == 0)) { + Read_hash = simple_hash("Read"); + Write_hash = simple_hash("Write"); /* - Sync_hash = simple_hash("Sync"); - Async_hash = simple_hash("Async"); - Total_hash = simple_hash("Total"); + Sync_hash = simple_hash("Sync"); + Async_hash = simple_hash("Async"); + Total_hash = simple_hash("Total"); */ - } + } - io->updated = 0; - if(io->filename) { - ff = procfile_reopen(ff, io->filename, NULL, PROCFILE_FLAG_DEFAULT); - if(!ff) return; + io->updated = 0; + if(io->filename) { + ff = procfile_reopen(ff, io->filename, NULL, PROCFILE_FLAG_DEFAULT); + if(!ff) return; - ff = procfile_readall(ff); - if(!ff) return; + ff = procfile_readall(ff); + if(!ff) return; - unsigned long i, lines = procfile_lines(ff); + unsigned long i, lines = procfile_lines(ff); - if(lines < 1) { - error("File '%s' should have 1+ lines.", io->filename); - return; - } + if(lines < 1) { + error("File '%s' should have 1+ lines.", io->filename); + return; + } - io->Read = 0; - io->Write = 0; + io->Read = 0; + io->Write = 0; /* - io->Sync = 0; - io->Async = 0; - io->Total = 0; + io->Sync = 0; + io->Async = 0; + io->Total = 0; */ - for(i = 0; i < lines ; i++) { - char *s = procfile_lineword(ff, i, 1); - uint32_t hash = simple_hash(s); + for(i = 0; i < lines ; i++) { + char *s = procfile_lineword(ff, i, 1); + uint32_t hash = simple_hash(s); - if(hash == Read_hash && !strcmp(s, "Read")) - io->Read += strtoull(procfile_lineword(ff, i, 2), NULL, 10); + if(hash == Read_hash && !strcmp(s, "Read")) + io->Read += strtoull(procfile_lineword(ff, i, 2), NULL, 10); - else if(hash == Write_hash && !strcmp(s, "Write")) - io->Write += strtoull(procfile_lineword(ff, i, 2), NULL, 10); + else if(hash == Write_hash && !strcmp(s, "Write")) + io->Write += strtoull(procfile_lineword(ff, i, 2), NULL, 10); /* - else if(hash == Sync_hash && !strcmp(s, "Sync")) - io->Sync += strtoull(procfile_lineword(ff, i, 2), NULL, 10); + else if(hash == Sync_hash && !strcmp(s, "Sync")) + io->Sync += strtoull(procfile_lineword(ff, i, 2), NULL, 10); - else if(hash == Async_hash && !strcmp(s, "Async")) - io->Async += strtoull(procfile_lineword(ff, i, 2), NULL, 10); + else if(hash == Async_hash && !strcmp(s, "Async")) + io->Async += strtoull(procfile_lineword(ff, i, 2), NULL, 10); - else if(hash == Total_hash && !strcmp(s, "Total")) - io->Total += strtoull(procfile_lineword(ff, i, 2), NULL, 10); + else if(hash == Total_hash && !strcmp(s, "Total")) + io->Total += strtoull(procfile_lineword(ff, i, 2), NULL, 10); */ - } + } - io->updated = 1; - // fprintf(stderr, "READ '%s': Read: %llu, Write: %llu, Sync: %llu, Async: %llu, Total: %llu\n", io->filename, io->Read, io->Write, io->Sync, io->Async, io->Total); - } + io->updated = 1; + // fprintf(stderr, "READ '%s': Read: %llu, Write: %llu, Sync: %llu, Async: %llu, Total: %llu\n", io->filename, io->Read, io->Write, io->Sync, io->Async, io->Total); + } } void cgroup_read_memory(struct memory *mem) { - static procfile *ff = NULL; - - static uint32_t cache_hash = 0; - static uint32_t rss_hash = 0; - static uint32_t rss_huge_hash = 0; - static uint32_t mapped_file_hash = 0; - static uint32_t writeback_hash = 0; - static uint32_t dirty_hash = 0; - static uint32_t swap_hash = 0; - static uint32_t pgpgin_hash = 0; - static uint32_t pgpgout_hash = 0; - static uint32_t pgfault_hash = 0; - static uint32_t pgmajfault_hash = 0; + static procfile *ff = NULL; + + static uint32_t cache_hash = 0; + static uint32_t rss_hash = 0; + static uint32_t rss_huge_hash = 0; + static uint32_t mapped_file_hash = 0; + static uint32_t writeback_hash = 0; + static uint32_t dirty_hash = 0; + static uint32_t swap_hash = 0; + static uint32_t pgpgin_hash = 0; + static uint32_t pgpgout_hash = 0; + static uint32_t pgfault_hash = 0; + static uint32_t pgmajfault_hash = 0; /* - static uint32_t inactive_anon_hash = 0; - static uint32_t active_anon_hash = 0; - static uint32_t inactive_file_hash = 0; - static uint32_t active_file_hash = 0; - static uint32_t unevictable_hash = 0; - static uint32_t hierarchical_memory_limit_hash = 0; - static uint32_t total_cache_hash = 0; - static uint32_t total_rss_hash = 0; - static uint32_t total_rss_huge_hash = 0; - static uint32_t total_mapped_file_hash = 0; - static uint32_t total_writeback_hash = 0; - static uint32_t total_dirty_hash = 0; - static uint32_t total_swap_hash = 0; - static uint32_t total_pgpgin_hash = 0; - static uint32_t total_pgpgout_hash = 0; - static uint32_t total_pgfault_hash = 0; - static uint32_t total_pgmajfault_hash = 0; - static uint32_t total_inactive_anon_hash = 0; - static uint32_t total_active_anon_hash = 0; - static uint32_t total_inactive_file_hash = 0; - static uint32_t total_active_file_hash = 0; - static uint32_t total_unevictable_hash = 0; + static uint32_t inactive_anon_hash = 0; + static uint32_t active_anon_hash = 0; + static uint32_t inactive_file_hash = 0; + static uint32_t active_file_hash = 0; + static uint32_t unevictable_hash = 0; + static uint32_t hierarchical_memory_limit_hash = 0; + static uint32_t total_cache_hash = 0; + static uint32_t total_rss_hash = 0; + static uint32_t total_rss_huge_hash = 0; + static uint32_t total_mapped_file_hash = 0; + static uint32_t total_writeback_hash = 0; + static uint32_t total_dirty_hash = 0; + static uint32_t total_swap_hash = 0; + static uint32_t total_pgpgin_hash = 0; + static uint32_t total_pgpgout_hash = 0; + static uint32_t total_pgfault_hash = 0; + static uint32_t total_pgmajfault_hash = 0; + static uint32_t total_inactive_anon_hash = 0; + static uint32_t total_active_anon_hash = 0; + static uint32_t total_inactive_file_hash = 0; + static uint32_t total_active_file_hash = 0; + static uint32_t total_unevictable_hash = 0; */ - if(unlikely(cache_hash == 0)) { - cache_hash = simple_hash("cache"); - rss_hash = simple_hash("rss"); - rss_huge_hash = simple_hash("rss_huge"); - mapped_file_hash = simple_hash("mapped_file"); - writeback_hash = simple_hash("writeback"); - dirty_hash = simple_hash("dirty"); - swap_hash = simple_hash("swap"); - pgpgin_hash = simple_hash("pgpgin"); - pgpgout_hash = simple_hash("pgpgout"); - pgfault_hash = simple_hash("pgfault"); - pgmajfault_hash = simple_hash("pgmajfault"); + if(unlikely(cache_hash == 0)) { + cache_hash = simple_hash("cache"); + rss_hash = simple_hash("rss"); + rss_huge_hash = simple_hash("rss_huge"); + mapped_file_hash = simple_hash("mapped_file"); + writeback_hash = simple_hash("writeback"); + dirty_hash = simple_hash("dirty"); + swap_hash = simple_hash("swap"); + pgpgin_hash = simple_hash("pgpgin"); + pgpgout_hash = simple_hash("pgpgout"); + pgfault_hash = simple_hash("pgfault"); + pgmajfault_hash = simple_hash("pgmajfault"); /* - inactive_anon_hash = simple_hash("inactive_anon"); - active_anon_hash = simple_hash("active_anon"); - inactive_file_hash = simple_hash("inactive_file"); - active_file_hash = simple_hash("active_file"); - unevictable_hash = simple_hash("unevictable"); - hierarchical_memory_limit_hash = simple_hash("hierarchical_memory_limit"); - total_cache_hash = simple_hash("total_cache"); - total_rss_hash = simple_hash("total_rss"); - total_rss_huge_hash = simple_hash("total_rss_huge"); - total_mapped_file_hash = simple_hash("total_mapped_file"); - total_writeback_hash = simple_hash("total_writeback"); - total_dirty_hash = simple_hash("total_dirty"); - total_swap_hash = simple_hash("total_swap"); - total_pgpgin_hash = simple_hash("total_pgpgin"); - total_pgpgout_hash = simple_hash("total_pgpgout"); - total_pgfault_hash = simple_hash("total_pgfault"); - total_pgmajfault_hash = simple_hash("total_pgmajfault"); - total_inactive_anon_hash = simple_hash("total_inactive_anon"); - total_active_anon_hash = simple_hash("total_active_anon"); - total_inactive_file_hash = simple_hash("total_inactive_file"); - total_active_file_hash = simple_hash("total_active_file"); - total_unevictable_hash = simple_hash("total_unevictable"); + inactive_anon_hash = simple_hash("inactive_anon"); + active_anon_hash = simple_hash("active_anon"); + inactive_file_hash = simple_hash("inactive_file"); + active_file_hash = simple_hash("active_file"); + unevictable_hash = simple_hash("unevictable"); + hierarchical_memory_limit_hash = simple_hash("hierarchical_memory_limit"); + total_cache_hash = simple_hash("total_cache"); + total_rss_hash = simple_hash("total_rss"); + total_rss_huge_hash = simple_hash("total_rss_huge"); + total_mapped_file_hash = simple_hash("total_mapped_file"); + total_writeback_hash = simple_hash("total_writeback"); + total_dirty_hash = simple_hash("total_dirty"); + total_swap_hash = simple_hash("total_swap"); + total_pgpgin_hash = simple_hash("total_pgpgin"); + total_pgpgout_hash = simple_hash("total_pgpgout"); + total_pgfault_hash = simple_hash("total_pgfault"); + total_pgmajfault_hash = simple_hash("total_pgmajfault"); + total_inactive_anon_hash = simple_hash("total_inactive_anon"); + total_active_anon_hash = simple_hash("total_active_anon"); + total_inactive_file_hash = simple_hash("total_inactive_file"); + total_active_file_hash = simple_hash("total_active_file"); + total_unevictable_hash = simple_hash("total_unevictable"); */ - } + } - mem->updated = 0; - if(mem->filename) { - ff = procfile_reopen(ff, mem->filename, NULL, PROCFILE_FLAG_DEFAULT); - if(!ff) return; + mem->updated = 0; + if(mem->filename) { + ff = procfile_reopen(ff, mem->filename, NULL, PROCFILE_FLAG_DEFAULT); + if(!ff) return; - ff = procfile_readall(ff); - if(!ff) return; + ff = procfile_readall(ff); + if(!ff) return; - unsigned long i, lines = procfile_lines(ff); + unsigned long i, lines = procfile_lines(ff); - if(lines < 1) { - error("File '%s' should have 1+ lines.", mem->filename); - return; - } + if(lines < 1) { + error("File '%s' should have 1+ lines.", mem->filename); + return; + } - for(i = 0; i < lines ; i++) { - char *s = procfile_lineword(ff, i, 0); - uint32_t hash = simple_hash(s); + for(i = 0; i < lines ; i++) { + char *s = procfile_lineword(ff, i, 0); + uint32_t hash = simple_hash(s); - if(hash == cache_hash && !strcmp(s, "cache")) - mem->cache = strtoull(procfile_lineword(ff, i, 1), NULL, 10); + if(hash == cache_hash && !strcmp(s, "cache")) + mem->cache = strtoull(procfile_lineword(ff, i, 1), NULL, 10); - else if(hash == rss_hash && !strcmp(s, "rss")) - mem->rss = strtoull(procfile_lineword(ff, i, 1), NULL, 10); + else if(hash == rss_hash && !strcmp(s, "rss")) + mem->rss = strtoull(procfile_lineword(ff, i, 1), NULL, 10); - else if(hash == rss_huge_hash && !strcmp(s, "rss_huge")) - mem->rss_huge = strtoull(procfile_lineword(ff, i, 1), NULL, 10); + else if(hash == rss_huge_hash && !strcmp(s, "rss_huge")) + mem->rss_huge = strtoull(procfile_lineword(ff, i, 1), NULL, 10); - else if(hash == mapped_file_hash && !strcmp(s, "mapped_file")) - mem->mapped_file = strtoull(procfile_lineword(ff, i, 1), NULL, 10); + else if(hash == mapped_file_hash && !strcmp(s, "mapped_file")) + mem->mapped_file = strtoull(procfile_lineword(ff, i, 1), NULL, 10); - else if(hash == writeback_hash && !strcmp(s, "writeback")) - mem->writeback = strtoull(procfile_lineword(ff, i, 1), NULL, 10); + else if(hash == writeback_hash && !strcmp(s, "writeback")) + mem->writeback = strtoull(procfile_lineword(ff, i, 1), NULL, 10); - else if(hash == dirty_hash && !strcmp(s, "dirty")) { - mem->dirty = strtoull(procfile_lineword(ff, i, 1), NULL, 10); - mem->has_dirty_swap = 1; - } + else if(hash == dirty_hash && !strcmp(s, "dirty")) { + mem->dirty = strtoull(procfile_lineword(ff, i, 1), NULL, 10); + mem->has_dirty_swap = 1; + } - else if(hash == swap_hash && !strcmp(s, "swap")) { - mem->swap = strtoull(procfile_lineword(ff, i, 1), NULL, 10); - mem->has_dirty_swap = 1; - } + else if(hash == swap_hash && !strcmp(s, "swap")) { + mem->swap = strtoull(procfile_lineword(ff, i, 1), NULL, 10); + mem->has_dirty_swap = 1; + } - else if(hash == pgpgin_hash && !strcmp(s, "pgpgin")) - mem->pgpgin = strtoull(procfile_lineword(ff, i, 1), NULL, 10); + else if(hash == pgpgin_hash && !strcmp(s, "pgpgin")) + mem->pgpgin = strtoull(procfile_lineword(ff, i, 1), NULL, 10); - else if(hash == pgpgout_hash && !strcmp(s, "pgpgout")) - mem->pgpgout = strtoull(procfile_lineword(ff, i, 1), NULL, 10); + else if(hash == pgpgout_hash && !strcmp(s, "pgpgout")) + mem->pgpgout = strtoull(procfile_lineword(ff, i, 1), NULL, 10); - else if(hash == pgfault_hash && !strcmp(s, "pgfault")) - mem->pgfault = strtoull(procfile_lineword(ff, i, 1), NULL, 10); + else if(hash == pgfault_hash && !strcmp(s, "pgfault")) + mem->pgfault = strtoull(procfile_lineword(ff, i, 1), NULL, 10); - else if(hash == pgmajfault_hash && !strcmp(s, "pgmajfault")) - mem->pgmajfault = strtoull(procfile_lineword(ff, i, 1), NULL, 10); + else if(hash == pgmajfault_hash && !strcmp(s, "pgmajfault")) + mem->pgmajfault = strtoull(procfile_lineword(ff, i, 1), NULL, 10); /* - else if(hash == inactive_anon_hash && !strcmp(s, "inactive_anon")) - mem->inactive_anon = strtoull(procfile_lineword(ff, i, 1), NULL, 10); + else if(hash == inactive_anon_hash && !strcmp(s, "inactive_anon")) + mem->inactive_anon = strtoull(procfile_lineword(ff, i, 1), NULL, 10); - else if(hash == active_anon_hash && !strcmp(s, "active_anon")) - mem->active_anon = strtoull(procfile_lineword(ff, i, 1), NULL, 10); + else if(hash == active_anon_hash && !strcmp(s, "active_anon")) + mem->active_anon = strtoull(procfile_lineword(ff, i, 1), NULL, 10); - else if(hash == inactive_file_hash && !strcmp(s, "inactive_file")) - mem->inactive_file = strtoull(procfile_lineword(ff, i, 1), NULL, 10); + else if(hash == inactive_file_hash && !strcmp(s, "inactive_file")) + mem->inactive_file = strtoull(procfile_lineword(ff, i, 1), NULL, 10); - else if(hash == active_file_hash && !strcmp(s, "active_file")) - mem->active_file = strtoull(procfile_lineword(ff, i, 1), NULL, 10); + else if(hash == active_file_hash && !strcmp(s, "active_file")) + mem->active_file = strtoull(procfile_lineword(ff, i, 1), NULL, 10); - else if(hash == unevictable_hash && !strcmp(s, "unevictable")) - mem->unevictable = strtoull(procfile_lineword(ff, i, 1), NULL, 10); + else if(hash == unevictable_hash && !strcmp(s, "unevictable")) + mem->unevictable = strtoull(procfile_lineword(ff, i, 1), NULL, 10); - else if(hash == hierarchical_memory_limit_hash && !strcmp(s, "hierarchical_memory_limit")) - mem->hierarchical_memory_limit = strtoull(procfile_lineword(ff, i, 1), NULL, 10); + else if(hash == hierarchical_memory_limit_hash && !strcmp(s, "hierarchical_memory_limit")) + mem->hierarchical_memory_limit = strtoull(procfile_lineword(ff, i, 1), NULL, 10); - else if(hash == total_cache_hash && !strcmp(s, "total_cache")) - mem->total_cache = strtoull(procfile_lineword(ff, i, 1), NULL, 10); + else if(hash == total_cache_hash && !strcmp(s, "total_cache")) + mem->total_cache = strtoull(procfile_lineword(ff, i, 1), NULL, 10); - else if(hash == total_rss_hash && !strcmp(s, "total_rss")) - mem->total_rss = strtoull(procfile_lineword(ff, i, 1), NULL, 10); + else if(hash == total_rss_hash && !strcmp(s, "total_rss")) + mem->total_rss = strtoull(procfile_lineword(ff, i, 1), NULL, 10); - else if(hash == total_rss_huge_hash && !strcmp(s, "total_rss_huge")) - mem->total_rss_huge = strtoull(procfile_lineword(ff, i, 1), NULL, 10); + else if(hash == total_rss_huge_hash && !strcmp(s, "total_rss_huge")) + mem->total_rss_huge = strtoull(procfile_lineword(ff, i, 1), NULL, 10); - else if(hash == total_mapped_file_hash && !strcmp(s, "total_mapped_file")) - mem->total_mapped_file = strtoull(procfile_lineword(ff, i, 1), NULL, 10); + else if(hash == total_mapped_file_hash && !strcmp(s, "total_mapped_file")) + mem->total_mapped_file = strtoull(procfile_lineword(ff, i, 1), NULL, 10); - else if(hash == total_writeback_hash && !strcmp(s, "total_writeback")) - mem->total_writeback = strtoull(procfile_lineword(ff, i, 1), NULL, 10); + else if(hash == total_writeback_hash && !strcmp(s, "total_writeback")) + mem->total_writeback = strtoull(procfile_lineword(ff, i, 1), NULL, 10); - else if(hash == total_dirty_hash && !strcmp(s, "total_dirty")) - mem->total_dirty = strtoull(procfile_lineword(ff, i, 1), NULL, 10); + else if(hash == total_dirty_hash && !strcmp(s, "total_dirty")) + mem->total_dirty = strtoull(procfile_lineword(ff, i, 1), NULL, 10); - else if(hash == total_swap_hash && !strcmp(s, "total_swap")) - mem->total_swap = strtoull(procfile_lineword(ff, i, 1), NULL, 10); + else if(hash == total_swap_hash && !strcmp(s, "total_swap")) + mem->total_swap = strtoull(procfile_lineword(ff, i, 1), NULL, 10); - else if(hash == total_pgpgin_hash && !strcmp(s, "total_pgpgin")) - mem->total_pgpgin = strtoull(procfile_lineword(ff, i, 1), NULL, 10); + else if(hash == total_pgpgin_hash && !strcmp(s, "total_pgpgin")) + mem->total_pgpgin = strtoull(procfile_lineword(ff, i, 1), NULL, 10); - else if(hash == total_pgpgout_hash && !strcmp(s, "total_pgpgout")) - mem->total_pgpgout = strtoull(procfile_lineword(ff, i, 1), NULL, 10); + else if(hash == total_pgpgout_hash && !strcmp(s, "total_pgpgout")) + mem->total_pgpgout = strtoull(procfile_lineword(ff, i, 1), NULL, 10); - else if(hash == total_pgfault_hash && !strcmp(s, "total_pgfault")) - mem->total_pgfault = strtoull(procfile_lineword(ff, i, 1), NULL, 10); + else if(hash == total_pgfault_hash && !strcmp(s, "total_pgfault")) + mem->total_pgfault = strtoull(procfile_lineword(ff, i, 1), NULL, 10); - else if(hash == total_pgmajfault_hash && !strcmp(s, "total_pgmajfault")) - mem->total_pgmajfault = strtoull(procfile_lineword(ff, i, 1), NULL, 10); + else if(hash == total_pgmajfault_hash && !strcmp(s, "total_pgmajfault")) + mem->total_pgmajfault = strtoull(procfile_lineword(ff, i, 1), NULL, 10); - else if(hash == total_inactive_anon_hash && !strcmp(s, "total_inactive_anon")) - mem->total_inactive_anon = strtoull(procfile_lineword(ff, i, 1), NULL, 10); + else if(hash == total_inactive_anon_hash && !strcmp(s, "total_inactive_anon")) + mem->total_inactive_anon = strtoull(procfile_lineword(ff, i, 1), NULL, 10); - else if(hash == total_active_anon_hash && !strcmp(s, "total_active_anon")) - mem->total_active_anon = strtoull(procfile_lineword(ff, i, 1), NULL, 10); + else if(hash == total_active_anon_hash && !strcmp(s, "total_active_anon")) + mem->total_active_anon = strtoull(procfile_lineword(ff, i, 1), NULL, 10); - else if(hash == total_inactive_file_hash && !strcmp(s, "total_inactive_file")) - mem->total_inactive_file = strtoull(procfile_lineword(ff, i, 1), NULL, 10); + else if(hash == total_inactive_file_hash && !strcmp(s, "total_inactive_file")) + mem->total_inactive_file = strtoull(procfile_lineword(ff, i, 1), NULL, 10); - else if(hash == total_active_file_hash && !strcmp(s, "total_active_file")) - mem->total_active_file = strtoull(procfile_lineword(ff, i, 1), NULL, 10); + else if(hash == total_active_file_hash && !strcmp(s, "total_active_file")) + mem->total_active_file = strtoull(procfile_lineword(ff, i, 1), NULL, 10); - else if(hash == total_unevictable_hash && !strcmp(s, "total_unevictable")) - mem->total_unevictable = strtoull(procfile_lineword(ff, i, 1), NULL, 10); + else if(hash == total_unevictable_hash && !strcmp(s, "total_unevictable")) + mem->total_unevictable = strtoull(procfile_lineword(ff, i, 1), NULL, 10); */ - } + } - // fprintf(stderr, "READ: '%s', cache: %llu, rss: %llu, rss_huge: %llu, mapped_file: %llu, writeback: %llu, dirty: %llu, swap: %llu, pgpgin: %llu, pgpgout: %llu, pgfault: %llu, pgmajfault: %llu, inactive_anon: %llu, active_anon: %llu, inactive_file: %llu, active_file: %llu, unevictable: %llu, hierarchical_memory_limit: %llu, total_cache: %llu, total_rss: %llu, total_rss_huge: %llu, total_mapped_file: %llu, total_writeback: %llu, total_dirty: %llu, total_swap: %llu, total_pgpgin: %llu, total_pgpgout: %llu, total_pgfault: %llu, total_pgmajfault: %llu, total_inactive_anon: %llu, total_active_anon: %llu, total_inactive_file: %llu, total_active_file: %llu, total_unevictable: %llu\n", mem->filename, mem->cache, mem->rss, mem->rss_huge, mem->mapped_file, mem->writeback, mem->dirty, mem->swap, mem->pgpgin, mem->pgpgout, mem->pgfault, mem->pgmajfault, mem->inactive_anon, mem->active_anon, mem->inactive_file, mem->active_file, mem->unevictable, mem->hierarchical_memory_limit, mem->total_cache, mem->total_rss, mem->total_rss_huge, mem->total_mapped_file, mem->total_writeback, mem->total_dirty, mem->total_swap, mem->total_pgpgin, mem->total_pgpgout, mem->total_pgfault, mem->total_pgmajfault, mem->total_inactive_anon, mem->total_active_anon, mem->total_inactive_file, mem->total_active_file, mem->total_unevictable); + // fprintf(stderr, "READ: '%s', cache: %llu, rss: %llu, rss_huge: %llu, mapped_file: %llu, writeback: %llu, dirty: %llu, swap: %llu, pgpgin: %llu, pgpgout: %llu, pgfault: %llu, pgmajfault: %llu, inactive_anon: %llu, active_anon: %llu, inactive_file: %llu, active_file: %llu, unevictable: %llu, hierarchical_memory_limit: %llu, total_cache: %llu, total_rss: %llu, total_rss_huge: %llu, total_mapped_file: %llu, total_writeback: %llu, total_dirty: %llu, total_swap: %llu, total_pgpgin: %llu, total_pgpgout: %llu, total_pgfault: %llu, total_pgmajfault: %llu, total_inactive_anon: %llu, total_active_anon: %llu, total_inactive_file: %llu, total_active_file: %llu, total_unevictable: %llu\n", mem->filename, mem->cache, mem->rss, mem->rss_huge, mem->mapped_file, mem->writeback, mem->dirty, mem->swap, mem->pgpgin, mem->pgpgout, mem->pgfault, mem->pgmajfault, mem->inactive_anon, mem->active_anon, mem->inactive_file, mem->active_file, mem->unevictable, mem->hierarchical_memory_limit, mem->total_cache, mem->total_rss, mem->total_rss_huge, mem->total_mapped_file, mem->total_writeback, mem->total_dirty, mem->total_swap, mem->total_pgpgin, mem->total_pgpgout, mem->total_pgfault, mem->total_pgmajfault, mem->total_inactive_anon, mem->total_active_anon, mem->total_inactive_file, mem->total_active_file, mem->total_unevictable); - mem->updated = 1; - } + mem->updated = 1; + } } void cgroup_read(struct cgroup *cg) { - debug(D_CGROUP, "reading metrics for cgroups '%s'", cg->id); - - cgroup_read_cpuacct_stat(&cg->cpuacct_stat); - cgroup_read_cpuacct_usage(&cg->cpuacct_usage); - cgroup_read_memory(&cg->memory); - cgroup_read_blkio(&cg->io_service_bytes); - cgroup_read_blkio(&cg->io_serviced); - cgroup_read_blkio(&cg->throttle_io_service_bytes); - cgroup_read_blkio(&cg->throttle_io_serviced); - cgroup_read_blkio(&cg->io_merged); - cgroup_read_blkio(&cg->io_queued); + debug(D_CGROUP, "reading metrics for cgroups '%s'", cg->id); + + cgroup_read_cpuacct_stat(&cg->cpuacct_stat); + cgroup_read_cpuacct_usage(&cg->cpuacct_usage); + cgroup_read_memory(&cg->memory); + cgroup_read_blkio(&cg->io_service_bytes); + cgroup_read_blkio(&cg->io_serviced); + cgroup_read_blkio(&cg->throttle_io_service_bytes); + cgroup_read_blkio(&cg->throttle_io_serviced); + cgroup_read_blkio(&cg->io_merged); + cgroup_read_blkio(&cg->io_queued); } void read_all_cgroups(struct cgroup *root) { - debug(D_CGROUP, "reading metrics for all cgroups"); + debug(D_CGROUP, "reading metrics for all cgroups"); - struct cgroup *cg; + struct cgroup *cg; - for(cg = root; cg ; cg = cg->next) - if(cg->enabled && cg->available) - cgroup_read(cg); + for(cg = root; cg ; cg = cg->next) + if(cg->enabled && cg->available) + cgroup_read(cg); } // ---------------------------------------------------------------------------- @@ -584,177 +585,189 @@ void read_all_cgroups(struct cgroup *root) { #define CGROUP_CHARTID_LINE_MAX 1024 void cgroup_get_chart_id(struct cgroup *cg) { - debug(D_CGROUP, "getting the name of cgroup '%s'", cg->id); - - pid_t cgroup_pid; - char buffer[CGROUP_CHARTID_LINE_MAX + 1]; - - snprintfz(buffer, CGROUP_CHARTID_LINE_MAX, "exec %s '%s'", - config_get("plugin:cgroups", "script to get cgroup names", PLUGINS_DIR "/cgroup-name.sh"), cg->chart_id); - - debug(D_CGROUP, "executing command '%s' for cgroup '%s'", buffer, cg->id); - FILE *fp = mypopen(buffer, &cgroup_pid); - if(!fp) { - error("CGROUP: Cannot popen(\"%s\", \"r\").", buffer); - return; - } - debug(D_CGROUP, "reading from command '%s' for cgroup '%s'", buffer, cg->id); - char *s = fgets(buffer, CGROUP_CHARTID_LINE_MAX, fp); - debug(D_CGROUP, "closing command for cgroup '%s'", cg->id); - mypclose(fp, cgroup_pid); - debug(D_CGROUP, "closed command for cgroup '%s'", cg->id); - - if(s && *s && *s != '\n') { - debug(D_CGROUP, "cgroup '%s' should be renamed to '%s'", cg->id, s); - - trim(s); - - free(cg->chart_title); - cg->chart_title = strdup(s); - if(!cg->chart_title) - fatal("CGROUP: Cannot allocate memory for chart name of cgroup '%s' chart name: '%s'", cg->id, s); - - netdata_fix_chart_name(cg->chart_title); - - free(cg->chart_id); - cg->chart_id = strdup(s); - if(!cg->chart_id) - fatal("CGROUP: Cannot allocate memory for chart id of cgroup '%s' chart id: '%s'", cg->id, s); - - netdata_fix_chart_id(cg->chart_id); - - debug(D_CGROUP, "cgroup '%s' renamed to '%s' (title: '%s')", cg->id, cg->chart_id, cg->chart_title); - } - else debug(D_CGROUP, "cgroup '%s' is not to be renamed (will be shown as '%s')", cg->id, cg->chart_id); + debug(D_CGROUP, "getting the name of cgroup '%s'", cg->id); + + pid_t cgroup_pid; + char buffer[CGROUP_CHARTID_LINE_MAX + 1]; + + snprintfz(buffer, CGROUP_CHARTID_LINE_MAX, "exec %s '%s'", + config_get("plugin:cgroups", "script to get cgroup names", PLUGINS_DIR "/cgroup-name.sh"), cg->chart_id); + + debug(D_CGROUP, "executing command '%s' for cgroup '%s'", buffer, cg->id); + FILE *fp = mypopen(buffer, &cgroup_pid); + if(!fp) { + error("CGROUP: Cannot popen(\"%s\", \"r\").", buffer); + return; + } + debug(D_CGROUP, "reading from command '%s' for cgroup '%s'", buffer, cg->id); + char *s = fgets(buffer, CGROUP_CHARTID_LINE_MAX, fp); + debug(D_CGROUP, "closing command for cgroup '%s'", cg->id); + mypclose(fp, cgroup_pid); + debug(D_CGROUP, "closed command for cgroup '%s'", cg->id); + + if(s && *s && *s != '\n') { + debug(D_CGROUP, "cgroup '%s' should be renamed to '%s'", cg->id, s); + + trim(s); + + freez(cg->chart_title); + cg->chart_title = strdupz(s); + netdata_fix_chart_name(cg->chart_title); + + freez(cg->chart_id); + cg->chart_id = strdupz(s); + netdata_fix_chart_id(cg->chart_id); + cg->hash_chart = simple_hash(cg->chart_id); + + debug(D_CGROUP, "cgroup '%s' renamed to '%s' (title: '%s')", cg->id, cg->chart_id, cg->chart_title); + } + else debug(D_CGROUP, "cgroup '%s' is not to be renamed (will be shown as '%s')", cg->id, cg->chart_id); } struct cgroup *cgroup_add(const char *id) { - debug(D_CGROUP, "adding cgroup '%s'", id); - - if(cgroup_root_count >= cgroup_root_max) { - info("Maximum number of cgroups reached (%d). Not adding cgroup '%s'", cgroup_root_count, id); - return NULL; - } - - int def = cgroup_enable_new_cgroups_detected_at_runtime; - const char *chart_id = id; - if(!*chart_id) { - chart_id = "/"; - - // disable by default the root cgroup - def = 0; - debug(D_CGROUP, "cgroup '%s' is the root container (by default %s)", id, (def)?"enabled":"disabled"); - } - else { - if(*chart_id == '/') chart_id++; - - size_t len = strlen(chart_id); - - // disable by default the parent cgroup - // for known cgroup managers - if(!strcmp(chart_id, "lxc") || - !strcmp(chart_id, "docker") || - !strcmp(chart_id, "libvirt") || - !strcmp(chart_id, "qemu") || - !strcmp(chart_id, "systemd") || - !strcmp(chart_id, "system.slice") || - !strcmp(chart_id, "machine.slice") || - !strcmp(chart_id, "user") || - !strcmp(chart_id, "system") || - !strcmp(chart_id, "machine") || - // starts with them - (len > 6 && !strncmp(chart_id, "user/", 6)) || - (len > 11 && !strncmp(chart_id, "user.slice/", 11)) || - // ends with them - (len > 5 && !strncmp(&chart_id[len - 5], ".user", 5)) || - (len > 5 && !strncmp(&chart_id[len - 5], ".swap", 5)) || - (len > 6 && !strncmp(&chart_id[len - 6], ".slice", 6)) || - (len > 6 && !strncmp(&chart_id[len - 6], ".mount", 6)) || - (len > 8 && !strncmp(&chart_id[len - 8], ".session", 8)) || - (len > 8 && !strncmp(&chart_id[len - 8], ".service", 8)) || - (len > 10 && !strncmp(&chart_id[len - 10], ".partition", 10)) - ) { - def = 0; - debug(D_CGROUP, "cgroup '%s' is %s (by default)", id, (def)?"enabled":"disabled"); - } - } - - struct cgroup *cg = calloc(1, sizeof(struct cgroup)); - if(!cg) fatal("Cannot allocate memory for cgroup '%s'", id); - - debug(D_CGROUP, "adding cgroup '%s'", id); - - cg->id = strdup(id); - if(!cg->id) fatal("Cannot allocate memory for cgroup '%s'", id); - - cg->hash = simple_hash(cg->id); - - cg->chart_id = strdup(chart_id); - if(!cg->chart_id) fatal("Cannot allocate memory for cgroup '%s'", id); - - cg->chart_title = strdup(chart_id); - if(!cg->chart_title) fatal("Cannot allocate memory for cgroup '%s'", id); - - if(!cgroup_root) - cgroup_root = cg; - else { - // append it - struct cgroup *e; - for(e = cgroup_root; e->next ;e = e->next) ; - e->next = cg; - } - - cgroup_root_count++; - - // fix the name by calling the external script - cgroup_get_chart_id(cg); - - char option[FILENAME_MAX + 1]; - snprintfz(option, FILENAME_MAX, "enable cgroup %s", cg->chart_title); - cg->enabled = config_get_boolean("plugin:cgroups", option, def); - - debug(D_CGROUP, "Added cgroup '%s' with chart id '%s' and title '%s' as %s (default was %s)", cg->id, cg->chart_id, cg->chart_title, (cg->enabled)?"enabled":"disabled", (def)?"enabled":"disabled"); - - return cg; + debug(D_CGROUP, "adding cgroup '%s'", id); + + if(cgroup_root_count >= cgroup_root_max) { + info("Maximum number of cgroups reached (%d). Not adding cgroup '%s'", cgroup_root_count, id); + return NULL; + } + + int def = cgroup_enable_new_cgroups_detected_at_runtime; + const char *chart_id = id; + if(!*chart_id) { + chart_id = "/"; + + // disable by default the root cgroup + def = 0; + debug(D_CGROUP, "cgroup '%s' is the root container (by default %s)", id, (def)?"enabled":"disabled"); + } + else { + if(*chart_id == '/') chart_id++; + + size_t len = strlen(chart_id); + + // disable by default the parent cgroup + // for known cgroup managers + if(!strcmp(chart_id, "lxc") || + !strcmp(chart_id, "docker") || + !strcmp(chart_id, "libvirt") || + !strcmp(chart_id, "qemu") || + !strcmp(chart_id, "systemd") || + !strcmp(chart_id, "system.slice") || + !strcmp(chart_id, "machine.slice") || + !strcmp(chart_id, "init.scope") || + !strcmp(chart_id, "user") || + !strcmp(chart_id, "system") || + !strcmp(chart_id, "machine") || + // starts with them + (len > 6 && !strncmp(chart_id, "user/", 6)) || + (len > 11 && !strncmp(chart_id, "user.slice/", 11)) || + // ends with them + (len > 5 && !strncmp(&chart_id[len - 5], ".user", 5)) || + (len > 5 && !strncmp(&chart_id[len - 5], ".swap", 5)) || + (len > 6 && !strncmp(&chart_id[len - 6], ".slice", 6)) || + (len > 6 && !strncmp(&chart_id[len - 6], ".mount", 6)) || + (len > 8 && !strncmp(&chart_id[len - 8], ".session", 8)) || + (len > 8 && !strncmp(&chart_id[len - 8], ".service", 8)) || + (len > 10 && !strncmp(&chart_id[len - 10], ".partition", 10)) + ) { + def = 0; + debug(D_CGROUP, "cgroup '%s' is %s (by default)", id, (def)?"enabled":"disabled"); + } + } + + struct cgroup *cg = callocz(1, sizeof(struct cgroup)); + + cg->id = strdupz(id); + cg->hash = simple_hash(cg->id); + + cg->chart_id = strdupz(chart_id); + netdata_fix_chart_id(cg->chart_id); + cg->hash_chart = simple_hash(cg->chart_id); + + cg->chart_title = strdupz(chart_id); + + if(!cgroup_root) + cgroup_root = cg; + else { + // append it + struct cgroup *e; + for(e = cgroup_root; e->next ;e = e->next) ; + e->next = cg; + } + + cgroup_root_count++; + + // fix the name by calling the external script + cgroup_get_chart_id(cg); + + debug(D_CGROUP, "adding cgroup '%s' with chart id '%s'", id, chart_id); + + char option[FILENAME_MAX + 1]; + snprintfz(option, FILENAME_MAX, "enable cgroup %s", cg->chart_title); + cg->enabled = config_get_boolean("plugin:cgroups", option, def); + + if(cg->enabled) { + struct cgroup *t; + for (t = cgroup_root; t; t = t->next) { + if (t != cg && t->enabled && t->hash_chart == cg->hash_chart && !strcmp(t->chart_id, cg->chart_id)) { + if (!strncmp(t->chart_id, "/system.slice/", 14) && !strncmp(cg->chart_id, "/init.scope/system.slice/", 25)) { + error("Control group with chart id '%s' already exists with id '%s' and is enabled. Swapping them by enabling cgroup with id '%s' and disabling cgroup with id '%s'.", + cg->chart_id, t->id, cg->id, t->id); + t->enabled = 0; + } else { + error("Control group with chart id '%s' already exists with id '%s' and is enabled. Disabling cgroup with id '%s'.", + cg->chart_id, t->id, cg->id); + cg->enabled = 0; + } + + break; + } + } + } + + debug(D_CGROUP, "Added cgroup '%s' with chart id '%s' and title '%s' as %s (default was %s)", cg->id, cg->chart_id, cg->chart_title, (cg->enabled)?"enabled":"disabled", (def)?"enabled":"disabled"); + + return cg; } void cgroup_free(struct cgroup *cg) { - debug(D_CGROUP, "Removing cgroup '%s' with chart id '%s' (was %s and %s)", cg->id, cg->chart_id, (cg->enabled)?"enabled":"disabled", (cg->available)?"available":"not available"); - - free(cg->cpuacct_usage.cpu_percpu); - - free(cg->cpuacct_stat.filename); - free(cg->cpuacct_usage.filename); - free(cg->memory.filename); - free(cg->io_service_bytes.filename); - free(cg->io_serviced.filename); - free(cg->throttle_io_service_bytes.filename); - free(cg->throttle_io_serviced.filename); - free(cg->io_merged.filename); - free(cg->io_queued.filename); - - free(cg->id); - free(cg->chart_id); - free(cg->chart_title); - free(cg); - - cgroup_root_count--; + debug(D_CGROUP, "Removing cgroup '%s' with chart id '%s' (was %s and %s)", cg->id, cg->chart_id, (cg->enabled)?"enabled":"disabled", (cg->available)?"available":"not available"); + + freez(cg->cpuacct_usage.cpu_percpu); + + freez(cg->cpuacct_stat.filename); + freez(cg->cpuacct_usage.filename); + freez(cg->memory.filename); + freez(cg->io_service_bytes.filename); + freez(cg->io_serviced.filename); + freez(cg->throttle_io_service_bytes.filename); + freez(cg->throttle_io_serviced.filename); + freez(cg->io_merged.filename); + freez(cg->io_queued.filename); + + freez(cg->id); + freez(cg->chart_id); + freez(cg->chart_title); + freez(cg); + + cgroup_root_count--; } // find if a given cgroup exists struct cgroup *cgroup_find(const char *id) { - debug(D_CGROUP, "searching for cgroup '%s'", id); + debug(D_CGROUP, "searching for cgroup '%s'", id); - uint32_t hash = simple_hash(id); + uint32_t hash = simple_hash(id); - struct cgroup *cg; - for(cg = cgroup_root; cg ; cg = cg->next) { - if(hash == cg->hash && strcmp(id, cg->id) == 0) - break; - } + struct cgroup *cg; + for(cg = cgroup_root; cg ; cg = cg->next) { + if(hash == cg->hash && strcmp(id, cg->id) == 0) + break; + } - debug(D_CGROUP, "cgroup_find('%s') %s", id, (cg)?"found":"not found"); - return cg; + debug(D_CGROUP, "cgroup_find('%s') %s", id, (cg)?"found":"not found"); + return cg; } // ---------------------------------------------------------------------------- @@ -762,198 +775,252 @@ struct cgroup *cgroup_find(const char *id) { // callback for find_file_in_subdirs() void found_subdir_in_dir(const char *dir) { - debug(D_CGROUP, "examining cgroup dir '%s'", dir); - - struct cgroup *cg = cgroup_find(dir); - if(!cg) { - if(*dir && cgroup_max_depth > 0) { - int depth = 0; - const char *s; - - for(s = dir; *s ;s++) - if(unlikely(*s == '/')) - depth++; - - if(depth > cgroup_max_depth) { - info("cgroup '%s' is too deep (%d, while max is %d)", dir, depth, cgroup_max_depth); - return; - } - } - debug(D_CGROUP, "will add dir '%s' as cgroup", dir); - cg = cgroup_add(dir); - } - - if(cg) cg->available = 1; + debug(D_CGROUP, "examining cgroup dir '%s'", dir); + + struct cgroup *cg = cgroup_find(dir); + if(!cg) { + if(*dir && cgroup_max_depth > 0) { + int depth = 0; + const char *s; + + for(s = dir; *s ;s++) + if(unlikely(*s == '/')) + depth++; + + if(depth > cgroup_max_depth) { + info("cgroup '%s' is too deep (%d, while max is %d)", dir, depth, cgroup_max_depth); + return; + } + } + debug(D_CGROUP, "will add dir '%s' as cgroup", dir); + cg = cgroup_add(dir); + } + + if(cg) cg->available = 1; } -void find_dir_in_subdirs(const char *base, const char *this, void (*callback)(const char *)) { - int enabled = -1; - if(!this) this = base; - size_t dirlen = strlen(this), baselen = strlen(base); - const char *relative_path = &this[baselen]; - - DIR *dir = opendir(this); - if(!dir) return; - - callback(relative_path); - - struct dirent *de = NULL; - while((de = readdir(dir))) { - if(de->d_type == DT_DIR - && ( - (de->d_name[0] == '.' && de->d_name[1] == '\0') - || (de->d_name[0] == '.' && de->d_name[1] == '.' && de->d_name[2] == '\0') - )) - continue; - - debug(D_CGROUP, "examining '%s/%s'", this, de->d_name); - - if(de->d_type == DT_DIR) { - if(enabled == -1) { - const char *r = relative_path; - if(*r == '\0') r = "/"; - else if (*r == '/') r++; - - // we check for this option here - // so that the config will not have settings - // for leaf directories - char option[FILENAME_MAX + 1]; - snprintfz(option, FILENAME_MAX, "search for cgroups under %s", r); - option[FILENAME_MAX] = '\0'; - enabled = config_get_boolean("plugin:cgroups", option, 1); - } - - if(enabled) { - char *s = malloc(dirlen + strlen(de->d_name) + 2); - if(s) { - strcpy(s, this); - strcat(s, "/"); - strcat(s, de->d_name); - find_dir_in_subdirs(base, s, callback); - free(s); - } - } - } - } - - closedir(dir); +int find_dir_in_subdirs(const char *base, const char *this, void (*callback)(const char *)) { + debug(D_CGROUP, "searching for directories in '%s'", base); + + int ret = -1; + int enabled = -1; + if(!this) this = base; + size_t dirlen = strlen(this), baselen = strlen(base); + const char *relative_path = &this[baselen]; + + DIR *dir = opendir(this); + if(!dir) { + error("Cannot read cgroups directory '%s'", base); + return ret; + } + ret = 1; + + callback(relative_path); + + struct dirent *de = NULL; + while((de = readdir(dir))) { + if(de->d_type == DT_DIR + && ( + (de->d_name[0] == '.' && de->d_name[1] == '\0') + || (de->d_name[0] == '.' && de->d_name[1] == '.' && de->d_name[2] == '\0') + )) + continue; + + debug(D_CGROUP, "examining '%s/%s'", this, de->d_name); + + if(de->d_type == DT_DIR) { + if(enabled == -1) { + const char *r = relative_path; + if(*r == '\0') r = "/"; + else if (*r == '/') r++; + + // we check for this option here + // so that the config will not have settings + // for leaf directories + char option[FILENAME_MAX + 1]; + snprintfz(option, FILENAME_MAX, "search for cgroups under %s", r); + option[FILENAME_MAX] = '\0'; + enabled = config_get_boolean("plugin:cgroups", option, 1); + } + + if(enabled) { + char *s = mallocz(dirlen + strlen(de->d_name) + 2); + strcpy(s, this); + strcat(s, "/"); + strcat(s, de->d_name); + int ret2 = find_dir_in_subdirs(base, s, callback); + if(ret2 > 0) ret += ret2; + freez(s); + } + } + } + + closedir(dir); + return ret; } void mark_all_cgroups_as_not_available() { - debug(D_CGROUP, "marking all cgroups as not available"); + debug(D_CGROUP, "marking all cgroups as not available"); - struct cgroup *cg; + struct cgroup *cg; - // mark all as not available - for(cg = cgroup_root; cg ; cg = cg->next) - cg->available = 0; + // mark all as not available + for(cg = cgroup_root; cg ; cg = cg->next) + cg->available = 0; } void cleanup_all_cgroups() { - struct cgroup *cg = cgroup_root, *last = NULL; - - for(; cg ;) { - if(!cg->available) { - - if(!last) - cgroup_root = cg->next; - else - last->next = cg->next; - - cgroup_free(cg); - - if(!last) - cg = cgroup_root; - else - cg = last->next; - } - else { - last = cg; - cg = cg->next; - } - } + struct cgroup *cg = cgroup_root, *last = NULL; + + for(; cg ;) { + if(!cg->available) { + + if(!last) + cgroup_root = cg->next; + else + last->next = cg->next; + + cgroup_free(cg); + + if(!last) + cg = cgroup_root; + else + cg = last->next; + } + else { + last = cg; + cg = cg->next; + } + } } void find_all_cgroups() { - debug(D_CGROUP, "searching for cgroups"); - - mark_all_cgroups_as_not_available(); - - if(cgroup_enable_cpuacct_stat || cgroup_enable_cpuacct_usage) - find_dir_in_subdirs(cgroup_cpuacct_base, NULL, found_subdir_in_dir); - - if(cgroup_enable_blkio) - find_dir_in_subdirs(cgroup_blkio_base, NULL, found_subdir_in_dir); - - if(cgroup_enable_memory) - find_dir_in_subdirs(cgroup_memory_base, NULL, found_subdir_in_dir); - - // remove any non-existing cgroups - cleanup_all_cgroups(); - - struct cgroup *cg; - for(cg = cgroup_root; cg ; cg = cg->next) { - // fprintf(stderr, " >>> CGROUP '%s' (%u - %s) with name '%s'\n", cg->id, cg->hash, cg->available?"available":"stopped", cg->name); - - if(unlikely(!cg->available)) - continue; - - debug(D_CGROUP, "checking paths for cgroup '%s'", cg->id); - - // check for newly added cgroups - // and update the filenames they read - char filename[FILENAME_MAX + 1]; - if(cgroup_enable_cpuacct_stat && !cg->cpuacct_stat.filename) { - snprintfz(filename, FILENAME_MAX, "%s%s/cpuacct.stat", cgroup_cpuacct_base, cg->id); - cg->cpuacct_stat.filename = strdup(filename); - debug(D_CGROUP, "cpuacct.stat filename for cgroup '%s': '%s'", cg->id, cg->cpuacct_stat.filename); - } - if(cgroup_enable_cpuacct_usage && !cg->cpuacct_usage.filename) { - snprintfz(filename, FILENAME_MAX, "%s%s/cpuacct.usage_percpu", cgroup_cpuacct_base, cg->id); - cg->cpuacct_usage.filename = strdup(filename); - debug(D_CGROUP, "cpuacct.usage_percpu filename for cgroup '%s': '%s'", cg->id, cg->cpuacct_usage.filename); - } - if(cgroup_enable_memory && !cg->memory.filename) { - snprintfz(filename, FILENAME_MAX, "%s%s/memory.stat", cgroup_memory_base, cg->id); - cg->memory.filename = strdup(filename); - debug(D_CGROUP, "memory.stat filename for cgroup '%s': '%s'", cg->id, cg->memory.filename); - } - if(cgroup_enable_blkio) { - if(!cg->io_service_bytes.filename) { - snprintfz(filename, FILENAME_MAX, "%s%s/blkio.io_service_bytes", cgroup_blkio_base, cg->id); - cg->io_service_bytes.filename = strdup(filename); - debug(D_CGROUP, "io_service_bytes filename for cgroup '%s': '%s'", cg->id, cg->io_service_bytes.filename); - } - if(!cg->io_serviced.filename) { - snprintfz(filename, FILENAME_MAX, "%s%s/blkio.io_serviced", cgroup_blkio_base, cg->id); - cg->io_serviced.filename = strdup(filename); - debug(D_CGROUP, "io_serviced filename for cgroup '%s': '%s'", cg->id, cg->io_serviced.filename); - } - if(!cg->throttle_io_service_bytes.filename) { - snprintfz(filename, FILENAME_MAX, "%s%s/blkio.throttle.io_service_bytes", cgroup_blkio_base, cg->id); - cg->throttle_io_service_bytes.filename = strdup(filename); - debug(D_CGROUP, "throttle_io_service_bytes filename for cgroup '%s': '%s'", cg->id, cg->throttle_io_service_bytes.filename); - } - if(!cg->throttle_io_serviced.filename) { - snprintfz(filename, FILENAME_MAX, "%s%s/blkio.throttle.io_serviced", cgroup_blkio_base, cg->id); - cg->throttle_io_serviced.filename = strdup(filename); - debug(D_CGROUP, "throttle_io_serviced filename for cgroup '%s': '%s'", cg->id, cg->throttle_io_serviced.filename); - } - if(!cg->io_merged.filename) { - snprintfz(filename, FILENAME_MAX, "%s%s/blkio.io_merged", cgroup_blkio_base, cg->id); - cg->io_merged.filename = strdup(filename); - debug(D_CGROUP, "io_merged filename for cgroup '%s': '%s'", cg->id, cg->io_merged.filename); - } - if(!cg->io_queued.filename) { - snprintfz(filename, FILENAME_MAX, "%s%s/blkio.io_queued", cgroup_blkio_base, cg->id); - cg->io_queued.filename = strdup(filename); - debug(D_CGROUP, "io_queued filename for cgroup '%s': '%s'", cg->id, cg->io_queued.filename); - } - } - } - - debug(D_CGROUP, "done searching for cgroups"); - return; + debug(D_CGROUP, "searching for cgroups"); + + mark_all_cgroups_as_not_available(); + + if(cgroup_enable_cpuacct_stat || cgroup_enable_cpuacct_usage) { + if (find_dir_in_subdirs(cgroup_cpuacct_base, NULL, found_subdir_in_dir) == -1) { + cgroup_enable_cpuacct_stat = cgroup_enable_cpuacct_usage = 0; + error("disabled cgroup cpu statistics."); + } + } + + if(cgroup_enable_blkio) { + if (find_dir_in_subdirs(cgroup_blkio_base, NULL, found_subdir_in_dir) == -1) { + cgroup_enable_blkio = 0; + error("disabled cgroup blkio statistics."); + } + } + + if(cgroup_enable_memory) { + if(find_dir_in_subdirs(cgroup_memory_base, NULL, found_subdir_in_dir) == -1) { + cgroup_enable_memory = 0; + error("disabled cgroup memory statistics."); + } + } + + if(cgroup_enable_devices) { + if(find_dir_in_subdirs(cgroup_devices_base, NULL, found_subdir_in_dir) == -1) { + cgroup_enable_devices = 0; + error("disabled cgroup devices statistics."); + } + } + + // remove any non-existing cgroups + cleanup_all_cgroups(); + + struct cgroup *cg; + struct stat buf; + for(cg = cgroup_root; cg ; cg = cg->next) { + // fprintf(stderr, " >>> CGROUP '%s' (%u - %s) with name '%s'\n", cg->id, cg->hash, cg->available?"available":"stopped", cg->name); + + if(unlikely(!cg->available)) + continue; + + debug(D_CGROUP, "checking paths for cgroup '%s'", cg->id); + + // check for newly added cgroups + // and update the filenames they read + char filename[FILENAME_MAX + 1]; + if(cgroup_enable_cpuacct_stat && !cg->cpuacct_stat.filename) { + snprintfz(filename, FILENAME_MAX, "%s%s/cpuacct.stat", cgroup_cpuacct_base, cg->id); + if(stat(filename, &buf) != -1) { + cg->cpuacct_stat.filename = strdupz(filename); + debug(D_CGROUP, "cpuacct.stat filename for cgroup '%s': '%s'", cg->id, cg->cpuacct_stat.filename); + } + else debug(D_CGROUP, "cpuacct.stat file for cgroup '%s': '%s' does not exist.", cg->id, filename); + } + if(cgroup_enable_cpuacct_usage && !cg->cpuacct_usage.filename) { + snprintfz(filename, FILENAME_MAX, "%s%s/cpuacct.usage_percpu", cgroup_cpuacct_base, cg->id); + if(stat(filename, &buf) != -1) { + cg->cpuacct_usage.filename = strdupz(filename); + debug(D_CGROUP, "cpuacct.usage_percpu filename for cgroup '%s': '%s'", cg->id, cg->cpuacct_usage.filename); + } + else debug(D_CGROUP, "cpuacct.usage_percpu file for cgroup '%s': '%s' does not exist.", cg->id, filename); + } + if(cgroup_enable_memory && !cg->memory.filename) { + snprintfz(filename, FILENAME_MAX, "%s%s/memory.stat", cgroup_memory_base, cg->id); + if(stat(filename, &buf) != -1) { + cg->memory.filename = strdupz(filename); + debug(D_CGROUP, "memory.stat filename for cgroup '%s': '%s'", cg->id, cg->memory.filename); + } + else debug(D_CGROUP, "memory.stat file for cgroup '%s': '%s' does not exist.", cg->id, filename); + } + if(cgroup_enable_blkio) { + if(!cg->io_service_bytes.filename) { + snprintfz(filename, FILENAME_MAX, "%s%s/blkio.io_service_bytes", cgroup_blkio_base, cg->id); + if(stat(filename, &buf) != -1) { + cg->io_service_bytes.filename = strdupz(filename); + debug(D_CGROUP, "io_service_bytes filename for cgroup '%s': '%s'", cg->id, cg->io_service_bytes.filename); + } + else debug(D_CGROUP, "io_service_bytes file for cgroup '%s': '%s' does not exist.", cg->id, filename); + } + if(!cg->io_serviced.filename) { + snprintfz(filename, FILENAME_MAX, "%s%s/blkio.io_serviced", cgroup_blkio_base, cg->id); + if(stat(filename, &buf) != -1) { + cg->io_serviced.filename = strdupz(filename); + debug(D_CGROUP, "io_serviced filename for cgroup '%s': '%s'", cg->id, cg->io_serviced.filename); + } + else debug(D_CGROUP, "io_serviced file for cgroup '%s': '%s' does not exist.", cg->id, filename); + } + if(!cg->throttle_io_service_bytes.filename) { + snprintfz(filename, FILENAME_MAX, "%s%s/blkio.throttle.io_service_bytes", cgroup_blkio_base, cg->id); + if(stat(filename, &buf) != -1) { + cg->throttle_io_service_bytes.filename = strdupz(filename); + debug(D_CGROUP, "throttle_io_service_bytes filename for cgroup '%s': '%s'", cg->id, cg->throttle_io_service_bytes.filename); + } + else debug(D_CGROUP, "throttle_io_service_bytes file for cgroup '%s': '%s' does not exist.", cg->id, filename); + } + if(!cg->throttle_io_serviced.filename) { + snprintfz(filename, FILENAME_MAX, "%s%s/blkio.throttle.io_serviced", cgroup_blkio_base, cg->id); + if(stat(filename, &buf) != -1) { + cg->throttle_io_serviced.filename = strdupz(filename); + debug(D_CGROUP, "throttle_io_serviced filename for cgroup '%s': '%s'", cg->id, cg->throttle_io_serviced.filename); + } + else debug(D_CGROUP, "throttle_io_serviced file for cgroup '%s': '%s' does not exist.", cg->id, filename); + } + if(!cg->io_merged.filename) { + snprintfz(filename, FILENAME_MAX, "%s%s/blkio.io_merged", cgroup_blkio_base, cg->id); + if(stat(filename, &buf) != -1) { + cg->io_merged.filename = strdupz(filename); + debug(D_CGROUP, "io_merged filename for cgroup '%s': '%s'", cg->id, cg->io_merged.filename); + } + else debug(D_CGROUP, "io_merged file for cgroup '%s': '%s' does not exist.", cg->id, filename); + } + if(!cg->io_queued.filename) { + snprintfz(filename, FILENAME_MAX, "%s%s/blkio.io_queued", cgroup_blkio_base, cg->id); + if(stat(filename, &buf) != -1) { + cg->io_queued.filename = strdupz(filename); + debug(D_CGROUP, "io_queued filename for cgroup '%s': '%s'", cg->id, cg->io_queued.filename); + } + else debug(D_CGROUP, "io_queued file for cgroup '%s': '%s' does not exist.", cg->id, filename); + } + } + } + + debug(D_CGROUP, "done searching for cgroups"); + return; } // ---------------------------------------------------------------------------- @@ -962,349 +1029,349 @@ void find_all_cgroups() { #define CHART_TITLE_MAX 300 void update_cgroup_charts(int update_every) { - debug(D_CGROUP, "updating cgroups charts"); - - char type[RRD_ID_LENGTH_MAX + 1]; - char title[CHART_TITLE_MAX + 1]; - - struct cgroup *cg; - RRDSET *st; - - for(cg = cgroup_root; cg ; cg = cg->next) { - if(!cg->available || !cg->enabled) - continue; - - if(cg->id[0] == '\0') - strcpy(type, "cgroup_root"); - else if(cg->id[0] == '/') - snprintfz(type, RRD_ID_LENGTH_MAX, "cgroup_%s", cg->chart_id); - else - snprintfz(type, RRD_ID_LENGTH_MAX, "cgroup_%s", cg->chart_id); - - netdata_fix_chart_id(type); - - if(cg->cpuacct_stat.updated) { - st = rrdset_find_bytype(type, "cpu"); - if(!st) { - snprintfz(title, CHART_TITLE_MAX, "CPU Usage for cgroup %s", cg->chart_title); - st = rrdset_create(type, "cpu", NULL, "cpu", "cgroup.cpu", title, "%", 40000, update_every, RRDSET_TYPE_STACKED); - - rrddim_add(st, "user", NULL, 100, hz, RRDDIM_INCREMENTAL); - rrddim_add(st, "system", NULL, 100, hz, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); - - rrddim_set(st, "user", cg->cpuacct_stat.user); - rrddim_set(st, "system", cg->cpuacct_stat.system); - rrdset_done(st); - } - - if(cg->cpuacct_usage.updated) { - char id[RRD_ID_LENGTH_MAX + 1]; - unsigned int i; - - st = rrdset_find_bytype(type, "cpu_per_core"); - if(!st) { - snprintfz(title, CHART_TITLE_MAX, "CPU Usage Per Core for cgroup %s", cg->chart_title); - st = rrdset_create(type, "cpu_per_core", NULL, "cpu", "cgroup.cpu_per_core", title, "%", 40100, update_every, RRDSET_TYPE_STACKED); - - for(i = 0; i < cg->cpuacct_usage.cpus ;i++) { - snprintfz(id, CHART_TITLE_MAX, "cpu%d", i); - rrddim_add(st, id, NULL, 100, 1000000, RRDDIM_INCREMENTAL); - } - } - else rrdset_next(st); - - for(i = 0; i < cg->cpuacct_usage.cpus ;i++) { - snprintfz(id, CHART_TITLE_MAX, "cpu%d", i); - rrddim_set(st, id, cg->cpuacct_usage.cpu_percpu[i]); - } - rrdset_done(st); - } - - if(cg->memory.updated) { - if(cg->memory.cache + cg->memory.rss + cg->memory.rss_huge + cg->memory.mapped_file > 0) { - st = rrdset_find_bytype(type, "mem"); - if(!st) { - snprintfz(title, CHART_TITLE_MAX, "Memory Usage for cgroup %s", cg->chart_title); - st = rrdset_create(type, "mem", NULL, "mem", "cgroup.mem", title, "MB", 40200, update_every, - RRDSET_TYPE_STACKED); - - rrddim_add(st, "cache", NULL, 1, 1024 * 1024, RRDDIM_ABSOLUTE); - rrddim_add(st, "rss", NULL, 1, 1024 * 1024, RRDDIM_ABSOLUTE); - if(cg->memory.has_dirty_swap) - rrddim_add(st, "swap", NULL, 1, 1024 * 1024, RRDDIM_ABSOLUTE); - rrddim_add(st, "rss_huge", NULL, 1, 1024 * 1024, RRDDIM_ABSOLUTE); - rrddim_add(st, "mapped_file", NULL, 1, 1024 * 1024, RRDDIM_ABSOLUTE); - } - else rrdset_next(st); - - rrddim_set(st, "cache", cg->memory.cache); - rrddim_set(st, "rss", cg->memory.rss); - if(cg->memory.has_dirty_swap) - rrddim_set(st, "swap", cg->memory.swap); - rrddim_set(st, "rss_huge", cg->memory.rss_huge); - rrddim_set(st, "mapped_file", cg->memory.mapped_file); - rrdset_done(st); - } - - st = rrdset_find_bytype(type, "writeback"); - if(!st) { - snprintfz(title, CHART_TITLE_MAX, "Writeback Memory for cgroup %s", cg->chart_title); - st = rrdset_create(type, "writeback", NULL, "mem", "cgroup.writeback", title, "MB", 40300, - update_every, RRDSET_TYPE_AREA); - - if(cg->memory.has_dirty_swap) - rrddim_add(st, "dirty", NULL, 1, 1024 * 1024, RRDDIM_ABSOLUTE); - rrddim_add(st, "writeback", NULL, 1, 1024 * 1024, RRDDIM_ABSOLUTE); - } - else rrdset_next(st); - - if(cg->memory.has_dirty_swap) - rrddim_set(st, "dirty", cg->memory.dirty); - rrddim_set(st, "writeback", cg->memory.writeback); - rrdset_done(st); - - if(cg->memory.pgpgin + cg->memory.pgpgout > 0) { - st = rrdset_find_bytype(type, "mem_activity"); - if(!st) { - snprintfz(title, CHART_TITLE_MAX, "Memory Activity for cgroup %s", cg->chart_title); - st = rrdset_create(type, "mem_activity", NULL, "mem", "cgroup.mem_activity", title, "MB/s", - 40400, update_every, RRDSET_TYPE_LINE); - - rrddim_add(st, "pgpgin", "in", sysconf(_SC_PAGESIZE), 1024 * 1024, RRDDIM_INCREMENTAL); - rrddim_add(st, "pgpgout", "out", -sysconf(_SC_PAGESIZE), 1024 * 1024, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); - - rrddim_set(st, "pgpgin", cg->memory.pgpgin); - rrddim_set(st, "pgpgout", cg->memory.pgpgout); - rrdset_done(st); - } - - if(cg->memory.pgfault + cg->memory.pgmajfault > 0) { - st = rrdset_find_bytype(type, "pgfaults"); - if(!st) { - snprintfz(title, CHART_TITLE_MAX, "Memory Page Faults for cgroup %s", cg->chart_title); - st = rrdset_create(type, "pgfaults", NULL, "mem", "cgroup.pgfaults", title, "MB/s", 40500, - update_every, RRDSET_TYPE_LINE); - - rrddim_add(st, "pgfault", NULL, sysconf(_SC_PAGESIZE), 1024 * 1024, RRDDIM_INCREMENTAL); - rrddim_add(st, "pgmajfault", "swap", -sysconf(_SC_PAGESIZE), 1024 * 1024, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); - - rrddim_set(st, "pgfault", cg->memory.pgfault); - rrddim_set(st, "pgmajfault", cg->memory.pgmajfault); - rrdset_done(st); - } - } - - if(cg->io_service_bytes.updated && cg->io_service_bytes.Read + cg->io_service_bytes.Write > 0) { - st = rrdset_find_bytype(type, "io"); - if(!st) { - snprintfz(title, CHART_TITLE_MAX, "I/O Bandwidth (all disks) for cgroup %s", cg->chart_title); - st = rrdset_create(type, "io", NULL, "disk", "cgroup.io", title, "KB/s", 41200, - update_every, RRDSET_TYPE_LINE); - - rrddim_add(st, "read", NULL, 1, 1024, RRDDIM_INCREMENTAL); - rrddim_add(st, "write", NULL, -1, 1024, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); - - rrddim_set(st, "read", cg->io_service_bytes.Read); - rrddim_set(st, "write", cg->io_service_bytes.Write); - rrdset_done(st); - } - - if(cg->io_serviced.updated && cg->io_serviced.Read + cg->io_serviced.Write > 0) { - st = rrdset_find_bytype(type, "serviced_ops"); - if(!st) { - snprintfz(title, CHART_TITLE_MAX, "Serviced I/O Operations (all disks) for cgroup %s", cg->chart_title); - st = rrdset_create(type, "serviced_ops", NULL, "disk", "cgroup.serviced_ops", title, "operations/s", 41200, - update_every, RRDSET_TYPE_LINE); - - rrddim_add(st, "read", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "write", NULL, -1, 1, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); - - rrddim_set(st, "read", cg->io_serviced.Read); - rrddim_set(st, "write", cg->io_serviced.Write); - rrdset_done(st); - } - - if(cg->throttle_io_service_bytes.updated && cg->throttle_io_service_bytes.Read + cg->throttle_io_service_bytes.Write > 0) { - st = rrdset_find_bytype(type, "io"); - if(!st) { - snprintfz(title, CHART_TITLE_MAX, "Throttle I/O Bandwidth (all disks) for cgroup %s", cg->chart_title); - st = rrdset_create(type, "io", NULL, "disk", "cgroup.io", title, "KB/s", 41200, - update_every, RRDSET_TYPE_LINE); - - rrddim_add(st, "read", NULL, 1, 1024, RRDDIM_INCREMENTAL); - rrddim_add(st, "write", NULL, -1, 1024, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); - - rrddim_set(st, "read", cg->throttle_io_service_bytes.Read); - rrddim_set(st, "write", cg->throttle_io_service_bytes.Write); - rrdset_done(st); - } - - - if(cg->throttle_io_serviced.updated && cg->throttle_io_serviced.Read + cg->throttle_io_serviced.Write > 0) { - st = rrdset_find_bytype(type, "throttle_serviced_ops"); - if(!st) { - snprintfz(title, CHART_TITLE_MAX, "Throttle Serviced I/O Operations (all disks) for cgroup %s", cg->chart_title); - st = rrdset_create(type, "throttle_serviced_ops", NULL, "disk", "cgroup.throttle_serviced_ops", title, "operations/s", 41200, - update_every, RRDSET_TYPE_LINE); - - rrddim_add(st, "read", NULL, 1, 1, RRDDIM_INCREMENTAL); - rrddim_add(st, "write", NULL, -1, 1, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); - - rrddim_set(st, "read", cg->throttle_io_serviced.Read); - rrddim_set(st, "write", cg->throttle_io_serviced.Write); - rrdset_done(st); - } - - if(cg->io_queued.updated) { - st = rrdset_find_bytype(type, "queued_ops"); - if(!st) { - snprintfz(title, CHART_TITLE_MAX, "Queued I/O Operations (all disks) for cgroup %s", cg->chart_title); - st = rrdset_create(type, "queued_ops", NULL, "disk", "cgroup.queued_ops", title, "operations", 42000, - update_every, RRDSET_TYPE_LINE); - - rrddim_add(st, "read", NULL, 1, 1, RRDDIM_ABSOLUTE); - rrddim_add(st, "write", NULL, -1, 1, RRDDIM_ABSOLUTE); - } - else rrdset_next(st); - - rrddim_set(st, "read", cg->io_queued.Read); - rrddim_set(st, "write", cg->io_queued.Write); - rrdset_done(st); - } - - if(cg->io_merged.updated && cg->io_merged.Read + cg->io_merged.Write > 0) { - st = rrdset_find_bytype(type, "merged_ops"); - if(!st) { - snprintfz(title, CHART_TITLE_MAX, "Merged I/O Operations (all disks) for cgroup %s", cg->chart_title); - st = rrdset_create(type, "merged_ops", NULL, "disk", "cgroup.merged_ops", title, "operations/s", 42100, - update_every, RRDSET_TYPE_LINE); - - rrddim_add(st, "read", NULL, 1, 1024, RRDDIM_INCREMENTAL); - rrddim_add(st, "write", NULL, -1, 1024, RRDDIM_INCREMENTAL); - } - else rrdset_next(st); - - rrddim_set(st, "read", cg->io_merged.Read); - rrddim_set(st, "write", cg->io_merged.Write); - rrdset_done(st); - } - } - - debug(D_CGROUP, "done updating cgroups charts"); + debug(D_CGROUP, "updating cgroups charts"); + + char type[RRD_ID_LENGTH_MAX + 1]; + char title[CHART_TITLE_MAX + 1]; + + struct cgroup *cg; + RRDSET *st; + + for(cg = cgroup_root; cg ; cg = cg->next) { + if(!cg->available || !cg->enabled) + continue; + + if(cg->id[0] == '\0') + strcpy(type, "cgroup_root"); + else if(cg->id[0] == '/') + snprintfz(type, RRD_ID_LENGTH_MAX, "cgroup_%s", cg->chart_id); + else + snprintfz(type, RRD_ID_LENGTH_MAX, "cgroup_%s", cg->chart_id); + + netdata_fix_chart_id(type); + + if(cg->cpuacct_stat.updated) { + st = rrdset_find_bytype(type, "cpu"); + if(!st) { + snprintfz(title, CHART_TITLE_MAX, "CPU Usage for cgroup %s", cg->chart_title); + st = rrdset_create(type, "cpu", NULL, "cpu", "cgroup.cpu", title, "%", 40000, update_every, RRDSET_TYPE_STACKED); + + rrddim_add(st, "user", NULL, 100, hz, RRDDIM_INCREMENTAL); + rrddim_add(st, "system", NULL, 100, hz, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); + + rrddim_set(st, "user", cg->cpuacct_stat.user); + rrddim_set(st, "system", cg->cpuacct_stat.system); + rrdset_done(st); + } + + if(cg->cpuacct_usage.updated) { + char id[RRD_ID_LENGTH_MAX + 1]; + unsigned int i; + + st = rrdset_find_bytype(type, "cpu_per_core"); + if(!st) { + snprintfz(title, CHART_TITLE_MAX, "CPU Usage Per Core for cgroup %s", cg->chart_title); + st = rrdset_create(type, "cpu_per_core", NULL, "cpu", "cgroup.cpu_per_core", title, "%", 40100, update_every, RRDSET_TYPE_STACKED); + + for(i = 0; i < cg->cpuacct_usage.cpus ;i++) { + snprintfz(id, CHART_TITLE_MAX, "cpu%u", i); + rrddim_add(st, id, NULL, 100, 1000000000, RRDDIM_INCREMENTAL); + } + } + else rrdset_next(st); + + for(i = 0; i < cg->cpuacct_usage.cpus ;i++) { + snprintfz(id, CHART_TITLE_MAX, "cpu%u", i); + rrddim_set(st, id, cg->cpuacct_usage.cpu_percpu[i]); + } + rrdset_done(st); + } + + if(cg->memory.updated) { + if(cg->memory.cache + cg->memory.rss + cg->memory.rss_huge + cg->memory.mapped_file > 0) { + st = rrdset_find_bytype(type, "mem"); + if(!st) { + snprintfz(title, CHART_TITLE_MAX, "Memory Usage for cgroup %s", cg->chart_title); + st = rrdset_create(type, "mem", NULL, "mem", "cgroup.mem", title, "MB", 40200, update_every, + RRDSET_TYPE_STACKED); + + rrddim_add(st, "cache", NULL, 1, 1024 * 1024, RRDDIM_ABSOLUTE); + rrddim_add(st, "rss", NULL, 1, 1024 * 1024, RRDDIM_ABSOLUTE); + if(cg->memory.has_dirty_swap) + rrddim_add(st, "swap", NULL, 1, 1024 * 1024, RRDDIM_ABSOLUTE); + rrddim_add(st, "rss_huge", NULL, 1, 1024 * 1024, RRDDIM_ABSOLUTE); + rrddim_add(st, "mapped_file", NULL, 1, 1024 * 1024, RRDDIM_ABSOLUTE); + } + else rrdset_next(st); + + rrddim_set(st, "cache", cg->memory.cache); + rrddim_set(st, "rss", cg->memory.rss); + if(cg->memory.has_dirty_swap) + rrddim_set(st, "swap", cg->memory.swap); + rrddim_set(st, "rss_huge", cg->memory.rss_huge); + rrddim_set(st, "mapped_file", cg->memory.mapped_file); + rrdset_done(st); + } + + st = rrdset_find_bytype(type, "writeback"); + if(!st) { + snprintfz(title, CHART_TITLE_MAX, "Writeback Memory for cgroup %s", cg->chart_title); + st = rrdset_create(type, "writeback", NULL, "mem", "cgroup.writeback", title, "MB", 40300, + update_every, RRDSET_TYPE_AREA); + + if(cg->memory.has_dirty_swap) + rrddim_add(st, "dirty", NULL, 1, 1024 * 1024, RRDDIM_ABSOLUTE); + rrddim_add(st, "writeback", NULL, 1, 1024 * 1024, RRDDIM_ABSOLUTE); + } + else rrdset_next(st); + + if(cg->memory.has_dirty_swap) + rrddim_set(st, "dirty", cg->memory.dirty); + rrddim_set(st, "writeback", cg->memory.writeback); + rrdset_done(st); + + if(cg->memory.pgpgin + cg->memory.pgpgout > 0) { + st = rrdset_find_bytype(type, "mem_activity"); + if(!st) { + snprintfz(title, CHART_TITLE_MAX, "Memory Activity for cgroup %s", cg->chart_title); + st = rrdset_create(type, "mem_activity", NULL, "mem", "cgroup.mem_activity", title, "MB/s", + 40400, update_every, RRDSET_TYPE_LINE); + + rrddim_add(st, "pgpgin", "in", sysconf(_SC_PAGESIZE), 1024 * 1024, RRDDIM_INCREMENTAL); + rrddim_add(st, "pgpgout", "out", -sysconf(_SC_PAGESIZE), 1024 * 1024, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); + + rrddim_set(st, "pgpgin", cg->memory.pgpgin); + rrddim_set(st, "pgpgout", cg->memory.pgpgout); + rrdset_done(st); + } + + if(cg->memory.pgfault + cg->memory.pgmajfault > 0) { + st = rrdset_find_bytype(type, "pgfaults"); + if(!st) { + snprintfz(title, CHART_TITLE_MAX, "Memory Page Faults for cgroup %s", cg->chart_title); + st = rrdset_create(type, "pgfaults", NULL, "mem", "cgroup.pgfaults", title, "MB/s", 40500, + update_every, RRDSET_TYPE_LINE); + + rrddim_add(st, "pgfault", NULL, sysconf(_SC_PAGESIZE), 1024 * 1024, RRDDIM_INCREMENTAL); + rrddim_add(st, "pgmajfault", "swap", -sysconf(_SC_PAGESIZE), 1024 * 1024, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); + + rrddim_set(st, "pgfault", cg->memory.pgfault); + rrddim_set(st, "pgmajfault", cg->memory.pgmajfault); + rrdset_done(st); + } + } + + if(cg->io_service_bytes.updated && cg->io_service_bytes.Read + cg->io_service_bytes.Write > 0) { + st = rrdset_find_bytype(type, "io"); + if(!st) { + snprintfz(title, CHART_TITLE_MAX, "I/O Bandwidth (all disks) for cgroup %s", cg->chart_title); + st = rrdset_create(type, "io", NULL, "disk", "cgroup.io", title, "KB/s", 41200, + update_every, RRDSET_TYPE_LINE); + + rrddim_add(st, "read", NULL, 1, 1024, RRDDIM_INCREMENTAL); + rrddim_add(st, "write", NULL, -1, 1024, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); + + rrddim_set(st, "read", cg->io_service_bytes.Read); + rrddim_set(st, "write", cg->io_service_bytes.Write); + rrdset_done(st); + } + + if(cg->io_serviced.updated && cg->io_serviced.Read + cg->io_serviced.Write > 0) { + st = rrdset_find_bytype(type, "serviced_ops"); + if(!st) { + snprintfz(title, CHART_TITLE_MAX, "Serviced I/O Operations (all disks) for cgroup %s", cg->chart_title); + st = rrdset_create(type, "serviced_ops", NULL, "disk", "cgroup.serviced_ops", title, "operations/s", 41200, + update_every, RRDSET_TYPE_LINE); + + rrddim_add(st, "read", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "write", NULL, -1, 1, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); + + rrddim_set(st, "read", cg->io_serviced.Read); + rrddim_set(st, "write", cg->io_serviced.Write); + rrdset_done(st); + } + + if(cg->throttle_io_service_bytes.updated && cg->throttle_io_service_bytes.Read + cg->throttle_io_service_bytes.Write > 0) { + st = rrdset_find_bytype(type, "io"); + if(!st) { + snprintfz(title, CHART_TITLE_MAX, "Throttle I/O Bandwidth (all disks) for cgroup %s", cg->chart_title); + st = rrdset_create(type, "io", NULL, "disk", "cgroup.io", title, "KB/s", 41200, + update_every, RRDSET_TYPE_LINE); + + rrddim_add(st, "read", NULL, 1, 1024, RRDDIM_INCREMENTAL); + rrddim_add(st, "write", NULL, -1, 1024, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); + + rrddim_set(st, "read", cg->throttle_io_service_bytes.Read); + rrddim_set(st, "write", cg->throttle_io_service_bytes.Write); + rrdset_done(st); + } + + + if(cg->throttle_io_serviced.updated && cg->throttle_io_serviced.Read + cg->throttle_io_serviced.Write > 0) { + st = rrdset_find_bytype(type, "throttle_serviced_ops"); + if(!st) { + snprintfz(title, CHART_TITLE_MAX, "Throttle Serviced I/O Operations (all disks) for cgroup %s", cg->chart_title); + st = rrdset_create(type, "throttle_serviced_ops", NULL, "disk", "cgroup.throttle_serviced_ops", title, "operations/s", 41200, + update_every, RRDSET_TYPE_LINE); + + rrddim_add(st, "read", NULL, 1, 1, RRDDIM_INCREMENTAL); + rrddim_add(st, "write", NULL, -1, 1, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); + + rrddim_set(st, "read", cg->throttle_io_serviced.Read); + rrddim_set(st, "write", cg->throttle_io_serviced.Write); + rrdset_done(st); + } + + if(cg->io_queued.updated) { + st = rrdset_find_bytype(type, "queued_ops"); + if(!st) { + snprintfz(title, CHART_TITLE_MAX, "Queued I/O Operations (all disks) for cgroup %s", cg->chart_title); + st = rrdset_create(type, "queued_ops", NULL, "disk", "cgroup.queued_ops", title, "operations", 42000, + update_every, RRDSET_TYPE_LINE); + + rrddim_add(st, "read", NULL, 1, 1, RRDDIM_ABSOLUTE); + rrddim_add(st, "write", NULL, -1, 1, RRDDIM_ABSOLUTE); + } + else rrdset_next(st); + + rrddim_set(st, "read", cg->io_queued.Read); + rrddim_set(st, "write", cg->io_queued.Write); + rrdset_done(st); + } + + if(cg->io_merged.updated && cg->io_merged.Read + cg->io_merged.Write > 0) { + st = rrdset_find_bytype(type, "merged_ops"); + if(!st) { + snprintfz(title, CHART_TITLE_MAX, "Merged I/O Operations (all disks) for cgroup %s", cg->chart_title); + st = rrdset_create(type, "merged_ops", NULL, "disk", "cgroup.merged_ops", title, "operations/s", 42100, + update_every, RRDSET_TYPE_LINE); + + rrddim_add(st, "read", NULL, 1, 1024, RRDDIM_INCREMENTAL); + rrddim_add(st, "write", NULL, -1, 1024, RRDDIM_INCREMENTAL); + } + else rrdset_next(st); + + rrddim_set(st, "read", cg->io_merged.Read); + rrddim_set(st, "write", cg->io_merged.Write); + rrdset_done(st); + } + } + + debug(D_CGROUP, "done updating cgroups charts"); } // ---------------------------------------------------------------------------- // cgroups main int do_sys_fs_cgroup(int update_every, unsigned long long dt) { - static int cgroup_global_config_read = 0; - static time_t last_run = 0; - time_t now = time(NULL); + (void)dt; - if(dt) {}; + static int cgroup_global_config_read = 0; + static time_t last_run = 0; + time_t now = time(NULL); - if(unlikely(!cgroup_global_config_read)) { - read_cgroup_plugin_configuration(); - cgroup_global_config_read = 1; - } + if(unlikely(!cgroup_global_config_read)) { + read_cgroup_plugin_configuration(); + cgroup_global_config_read = 1; + } - if(unlikely(cgroup_enable_new_cgroups_detected_at_runtime && now - last_run > cgroup_check_for_new_every)) { - find_all_cgroups(); - last_run = now; - } + if(unlikely(cgroup_enable_new_cgroups_detected_at_runtime && now - last_run > cgroup_check_for_new_every)) { + find_all_cgroups(); + last_run = now; + } - read_all_cgroups(cgroup_root); - update_cgroup_charts(update_every); + read_all_cgroups(cgroup_root); + update_cgroup_charts(update_every); - return 0; + return 0; } void *cgroups_main(void *ptr) { - if(ptr) { ; } + if(ptr) { ; } - info("CGROUP Plugin thread created with task id %d", gettid()); + info("CGROUP Plugin thread created with task id %d", gettid()); - if(pthread_setcanceltype(PTHREAD_CANCEL_DEFERRED, NULL) != 0) - error("Cannot set pthread cancel type to DEFERRED."); + if(pthread_setcanceltype(PTHREAD_CANCEL_DEFERRED, NULL) != 0) + error("Cannot set pthread cancel type to DEFERRED."); - if(pthread_setcancelstate(PTHREAD_CANCEL_ENABLE, NULL) != 0) - error("Cannot set pthread cancel state to ENABLE."); + if(pthread_setcancelstate(PTHREAD_CANCEL_ENABLE, NULL) != 0) + error("Cannot set pthread cancel state to ENABLE."); - struct rusage thread; + struct rusage thread; - // when ZERO, attempt to do it - int vdo_sys_fs_cgroup = 0; - int vdo_cpu_netdata = !config_get_boolean("plugin:cgroups", "cgroups plugin resources", 1); + // when ZERO, attempt to do it + int vdo_sys_fs_cgroup = 0; + int vdo_cpu_netdata = !config_get_boolean("plugin:cgroups", "cgroups plugin resources", 1); - // keep track of the time each module was called - unsigned long long sutime_sys_fs_cgroup = 0ULL; + // keep track of the time each module was called + unsigned long long sutime_sys_fs_cgroup = 0ULL; - // the next time we will run - aligned properly - unsigned long long sunext = (time(NULL) - (time(NULL) % rrd_update_every) + rrd_update_every) * 1000000ULL; - unsigned long long sunow; + // the next time we will run - aligned properly + unsigned long long sunext = (time(NULL) - (time(NULL) % rrd_update_every) + rrd_update_every) * 1000000ULL; + unsigned long long sunow; - RRDSET *stcpu_thread = NULL; + RRDSET *stcpu_thread = NULL; - for(;1;) { - if(unlikely(netdata_exit)) break; + for(;1;) { + if(unlikely(netdata_exit)) break; - // delay until it is our time to run - while((sunow = timems()) < sunext) - usleep((useconds_t)(sunext - sunow)); + // delay until it is our time to run + while((sunow = time_usec()) < sunext) + sleep_usec(sunext - sunow); - // find the next time we need to run - while(timems() > sunext) - sunext += rrd_update_every * 1000000ULL; + // find the next time we need to run + while(time_usec() > sunext) + sunext += rrd_update_every * 1000000ULL; - if(unlikely(netdata_exit)) break; + if(unlikely(netdata_exit)) break; - // BEGIN -- the job to be done + // BEGIN -- the job to be done - if(!vdo_sys_fs_cgroup) { - debug(D_PROCNETDEV_LOOP, "PROCNETDEV: calling do_sys_fs_cgroup()."); - sunow = timems(); - vdo_sys_fs_cgroup = do_sys_fs_cgroup(rrd_update_every, (sutime_sys_fs_cgroup > 0)?sunow - sutime_sys_fs_cgroup:0ULL); - sutime_sys_fs_cgroup = sunow; - } - if(unlikely(netdata_exit)) break; + if(!vdo_sys_fs_cgroup) { + debug(D_PROCNETDEV_LOOP, "PROCNETDEV: calling do_sys_fs_cgroup()."); + sunow = time_usec(); + vdo_sys_fs_cgroup = do_sys_fs_cgroup(rrd_update_every, (sutime_sys_fs_cgroup > 0)?sunow - sutime_sys_fs_cgroup:0ULL); + sutime_sys_fs_cgroup = sunow; + } + if(unlikely(netdata_exit)) break; - // END -- the job is done + // END -- the job is done - // -------------------------------------------------------------------- + // -------------------------------------------------------------------- - if(!vdo_cpu_netdata) { - getrusage(RUSAGE_THREAD, &thread); + if(!vdo_cpu_netdata) { + getrusage(RUSAGE_THREAD, &thread); - if(!stcpu_thread) stcpu_thread = rrdset_find("netdata.plugin_cgroups_cpu"); - if(!stcpu_thread) { - stcpu_thread = rrdset_create("netdata", "plugin_cgroups_cpu", NULL, "proc.internal", NULL, "NetData CGroups Plugin CPU usage", "milliseconds/s", 132000, rrd_update_every, RRDSET_TYPE_STACKED); + if(!stcpu_thread) stcpu_thread = rrdset_find("netdata.plugin_cgroups_cpu"); + if(!stcpu_thread) { + stcpu_thread = rrdset_create("netdata", "plugin_cgroups_cpu", NULL, "proc.internal", NULL, "NetData CGroups Plugin CPU usage", "milliseconds/s", 132000, rrd_update_every, RRDSET_TYPE_STACKED); - rrddim_add(stcpu_thread, "user", NULL, 1, 1000, RRDDIM_INCREMENTAL); - rrddim_add(stcpu_thread, "system", NULL, 1, 1000, RRDDIM_INCREMENTAL); - } - else rrdset_next(stcpu_thread); + rrddim_add(stcpu_thread, "user", NULL, 1, 1000, RRDDIM_INCREMENTAL); + rrddim_add(stcpu_thread, "system", NULL, 1, 1000, RRDDIM_INCREMENTAL); + } + else rrdset_next(stcpu_thread); - rrddim_set(stcpu_thread, "user" , thread.ru_utime.tv_sec * 1000000ULL + thread.ru_utime.tv_usec); - rrddim_set(stcpu_thread, "system", thread.ru_stime.tv_sec * 1000000ULL + thread.ru_stime.tv_usec); - rrdset_done(stcpu_thread); - } - } + rrddim_set(stcpu_thread, "user" , thread.ru_utime.tv_sec * 1000000ULL + thread.ru_utime.tv_usec); + rrddim_set(stcpu_thread, "system", thread.ru_stime.tv_sec * 1000000ULL + thread.ru_stime.tv_usec); + rrdset_done(stcpu_thread); + } + } - pthread_exit(NULL); - return NULL; + pthread_exit(NULL); + return NULL; } diff --git a/src/sys_kernel_mm_ksm.c b/src/sys_kernel_mm_ksm.c index 928ac8c62..8c51be1df 100644 --- a/src/sys_kernel_mm_ksm.c +++ b/src/sys_kernel_mm_ksm.c @@ -1,21 +1,9 @@ -#ifdef HAVE_CONFIG_H -#include <config.h> -#endif -#include <stdio.h> -#include <stdlib.h> -#include <string.h> - #include "common.h" -#include "log.h" -#include "appconfig.h" -#include "procfile.h" -#include "rrd.h" -#include "plugin_proc.h" -typedef struct name_value { - char filename[FILENAME_MAX + 1]; - unsigned long long value; -} NAME_VALUE; +typedef struct ksm_name_value { + char filename[FILENAME_MAX + 1]; + unsigned long long value; +} KSM_NAME_VALUE; #define PAGES_SHARED 0 #define PAGES_SHARING 1 @@ -23,128 +11,128 @@ typedef struct name_value { #define PAGES_VOLATILE 3 #define PAGES_TO_SCAN 4 -NAME_VALUE values[] = { - [PAGES_SHARED] = { "/sys/kernel/mm/ksm/pages_shared", 0ULL }, - [PAGES_SHARING] = { "/sys/kernel/mm/ksm/pages_sharing", 0ULL }, - [PAGES_UNSHARED] = { "/sys/kernel/mm/ksm/pages_unshared", 0ULL }, - [PAGES_VOLATILE] = { "/sys/kernel/mm/ksm/pages_volatile", 0ULL }, - [PAGES_TO_SCAN] = { "/sys/kernel/mm/ksm/pages_to_scan", 0ULL }, +KSM_NAME_VALUE values[] = { + [PAGES_SHARED] = { "/sys/kernel/mm/ksm/pages_shared", 0ULL }, + [PAGES_SHARING] = { "/sys/kernel/mm/ksm/pages_sharing", 0ULL }, + [PAGES_UNSHARED] = { "/sys/kernel/mm/ksm/pages_unshared", 0ULL }, + [PAGES_VOLATILE] = { "/sys/kernel/mm/ksm/pages_volatile", 0ULL }, + [PAGES_TO_SCAN] = { "/sys/kernel/mm/ksm/pages_to_scan", 0ULL }, }; int do_sys_kernel_mm_ksm(int update_every, unsigned long long dt) { - static procfile *ff_pages_shared = NULL, *ff_pages_sharing = NULL, *ff_pages_unshared = NULL, *ff_pages_volatile = NULL, *ff_pages_to_scan = NULL; - static long page_size = -1; + static procfile *ff_pages_shared = NULL, *ff_pages_sharing = NULL, *ff_pages_unshared = NULL, *ff_pages_volatile = NULL, *ff_pages_to_scan = NULL; + static long page_size = -1; - if(dt) {}; - - if(page_size == -1) - page_size = sysconf(_SC_PAGESIZE); - - if(!ff_pages_shared) { - snprintfz(values[PAGES_SHARED].filename, FILENAME_MAX, "%s%s", global_host_prefix, "/sys/kernel/mm/ksm/pages_shared"); - snprintfz(values[PAGES_SHARED].filename, FILENAME_MAX, "%s", config_get("plugin:proc:/sys/kernel/mm/ksm", "/sys/kernel/mm/ksm/pages_shared", values[PAGES_SHARED].filename)); - ff_pages_shared = procfile_open(values[PAGES_SHARED].filename, " \t:", PROCFILE_FLAG_DEFAULT); - } + if(dt) {}; + + if(page_size == -1) + page_size = sysconf(_SC_PAGESIZE); + + if(!ff_pages_shared) { + snprintfz(values[PAGES_SHARED].filename, FILENAME_MAX, "%s%s", global_host_prefix, "/sys/kernel/mm/ksm/pages_shared"); + snprintfz(values[PAGES_SHARED].filename, FILENAME_MAX, "%s", config_get("plugin:proc:/sys/kernel/mm/ksm", "/sys/kernel/mm/ksm/pages_shared", values[PAGES_SHARED].filename)); + ff_pages_shared = procfile_open(values[PAGES_SHARED].filename, " \t:", PROCFILE_FLAG_DEFAULT); + } - if(!ff_pages_sharing) { - snprintfz(values[PAGES_SHARING].filename, FILENAME_MAX, "%s%s", global_host_prefix, "/sys/kernel/mm/ksm/pages_sharing"); - snprintfz(values[PAGES_SHARING].filename, FILENAME_MAX, "%s", config_get("plugin:proc:/sys/kernel/mm/ksm", "/sys/kernel/mm/ksm/pages_sharing", values[PAGES_SHARING].filename)); - ff_pages_sharing = procfile_open(values[PAGES_SHARING].filename, " \t:", PROCFILE_FLAG_DEFAULT); - } + if(!ff_pages_sharing) { + snprintfz(values[PAGES_SHARING].filename, FILENAME_MAX, "%s%s", global_host_prefix, "/sys/kernel/mm/ksm/pages_sharing"); + snprintfz(values[PAGES_SHARING].filename, FILENAME_MAX, "%s", config_get("plugin:proc:/sys/kernel/mm/ksm", "/sys/kernel/mm/ksm/pages_sharing", values[PAGES_SHARING].filename)); + ff_pages_sharing = procfile_open(values[PAGES_SHARING].filename, " \t:", PROCFILE_FLAG_DEFAULT); + } - if(!ff_pages_unshared) { - snprintfz(values[PAGES_UNSHARED].filename, FILENAME_MAX, "%s%s", global_host_prefix, "/sys/kernel/mm/ksm/pages_unshared"); - snprintfz(values[PAGES_UNSHARED].filename, FILENAME_MAX, "%s", config_get("plugin:proc:/sys/kernel/mm/ksm", "/sys/kernel/mm/ksm/pages_unshared", values[PAGES_UNSHARED].filename)); - ff_pages_unshared = procfile_open(values[PAGES_UNSHARED].filename, " \t:", PROCFILE_FLAG_DEFAULT); - } + if(!ff_pages_unshared) { + snprintfz(values[PAGES_UNSHARED].filename, FILENAME_MAX, "%s%s", global_host_prefix, "/sys/kernel/mm/ksm/pages_unshared"); + snprintfz(values[PAGES_UNSHARED].filename, FILENAME_MAX, "%s", config_get("plugin:proc:/sys/kernel/mm/ksm", "/sys/kernel/mm/ksm/pages_unshared", values[PAGES_UNSHARED].filename)); + ff_pages_unshared = procfile_open(values[PAGES_UNSHARED].filename, " \t:", PROCFILE_FLAG_DEFAULT); + } - if(!ff_pages_volatile) { - snprintfz(values[PAGES_VOLATILE].filename, FILENAME_MAX, "%s%s", global_host_prefix, "/sys/kernel/mm/ksm/pages_volatile"); - snprintfz(values[PAGES_VOLATILE].filename, FILENAME_MAX, "%s", config_get("plugin:proc:/sys/kernel/mm/ksm", "/sys/kernel/mm/ksm/pages_volatile", values[PAGES_VOLATILE].filename)); - ff_pages_volatile = procfile_open(values[PAGES_VOLATILE].filename, " \t:", PROCFILE_FLAG_DEFAULT); - } + if(!ff_pages_volatile) { + snprintfz(values[PAGES_VOLATILE].filename, FILENAME_MAX, "%s%s", global_host_prefix, "/sys/kernel/mm/ksm/pages_volatile"); + snprintfz(values[PAGES_VOLATILE].filename, FILENAME_MAX, "%s", config_get("plugin:proc:/sys/kernel/mm/ksm", "/sys/kernel/mm/ksm/pages_volatile", values[PAGES_VOLATILE].filename)); + ff_pages_volatile = procfile_open(values[PAGES_VOLATILE].filename, " \t:", PROCFILE_FLAG_DEFAULT); + } - if(!ff_pages_to_scan) { - snprintfz(values[PAGES_TO_SCAN].filename, FILENAME_MAX, "%s%s", global_host_prefix, "/sys/kernel/mm/ksm/pages_to_scan"); - snprintfz(values[PAGES_TO_SCAN].filename, FILENAME_MAX, "%s", config_get("plugin:proc:/sys/kernel/mm/ksm", "/sys/kernel/mm/ksm/pages_to_scan", values[PAGES_TO_SCAN].filename)); - ff_pages_to_scan = procfile_open(values[PAGES_TO_SCAN].filename, " \t:", PROCFILE_FLAG_DEFAULT); - } + if(!ff_pages_to_scan) { + snprintfz(values[PAGES_TO_SCAN].filename, FILENAME_MAX, "%s%s", global_host_prefix, "/sys/kernel/mm/ksm/pages_to_scan"); + snprintfz(values[PAGES_TO_SCAN].filename, FILENAME_MAX, "%s", config_get("plugin:proc:/sys/kernel/mm/ksm", "/sys/kernel/mm/ksm/pages_to_scan", values[PAGES_TO_SCAN].filename)); + ff_pages_to_scan = procfile_open(values[PAGES_TO_SCAN].filename, " \t:", PROCFILE_FLAG_DEFAULT); + } - if(!ff_pages_shared || !ff_pages_sharing || !ff_pages_unshared || !ff_pages_volatile || !ff_pages_to_scan) return 1; + if(!ff_pages_shared || !ff_pages_sharing || !ff_pages_unshared || !ff_pages_volatile || !ff_pages_to_scan) return 1; - unsigned long long pages_shared = 0, pages_sharing = 0, pages_unshared = 0, pages_volatile = 0, pages_to_scan = 0, offered = 0, saved = 0; + unsigned long long pages_shared = 0, pages_sharing = 0, pages_unshared = 0, pages_volatile = 0, pages_to_scan = 0, offered = 0, saved = 0; - ff_pages_shared = procfile_readall(ff_pages_shared); - if(!ff_pages_shared) return 0; // we return 0, so that we will retry to open it next time - pages_shared = strtoull(procfile_lineword(ff_pages_shared, 0, 0), NULL, 10); + ff_pages_shared = procfile_readall(ff_pages_shared); + if(!ff_pages_shared) return 0; // we return 0, so that we will retry to open it next time + pages_shared = strtoull(procfile_lineword(ff_pages_shared, 0, 0), NULL, 10); - ff_pages_sharing = procfile_readall(ff_pages_sharing); - if(!ff_pages_sharing) return 0; // we return 0, so that we will retry to open it next time - pages_sharing = strtoull(procfile_lineword(ff_pages_sharing, 0, 0), NULL, 10); + ff_pages_sharing = procfile_readall(ff_pages_sharing); + if(!ff_pages_sharing) return 0; // we return 0, so that we will retry to open it next time + pages_sharing = strtoull(procfile_lineword(ff_pages_sharing, 0, 0), NULL, 10); - ff_pages_unshared = procfile_readall(ff_pages_unshared); - if(!ff_pages_unshared) return 0; // we return 0, so that we will retry to open it next time - pages_unshared = strtoull(procfile_lineword(ff_pages_unshared, 0, 0), NULL, 10); + ff_pages_unshared = procfile_readall(ff_pages_unshared); + if(!ff_pages_unshared) return 0; // we return 0, so that we will retry to open it next time + pages_unshared = strtoull(procfile_lineword(ff_pages_unshared, 0, 0), NULL, 10); - ff_pages_volatile = procfile_readall(ff_pages_volatile); - if(!ff_pages_volatile) return 0; // we return 0, so that we will retry to open it next time - pages_volatile = strtoull(procfile_lineword(ff_pages_volatile, 0, 0), NULL, 10); + ff_pages_volatile = procfile_readall(ff_pages_volatile); + if(!ff_pages_volatile) return 0; // we return 0, so that we will retry to open it next time + pages_volatile = strtoull(procfile_lineword(ff_pages_volatile, 0, 0), NULL, 10); - ff_pages_to_scan = procfile_readall(ff_pages_to_scan); - if(!ff_pages_to_scan) return 0; // we return 0, so that we will retry to open it next time - pages_to_scan = strtoull(procfile_lineword(ff_pages_to_scan, 0, 0), NULL, 10); + ff_pages_to_scan = procfile_readall(ff_pages_to_scan); + if(!ff_pages_to_scan) return 0; // we return 0, so that we will retry to open it next time + pages_to_scan = strtoull(procfile_lineword(ff_pages_to_scan, 0, 0), NULL, 10); - offered = pages_sharing + pages_unshared + pages_volatile; - saved = pages_sharing - pages_shared; + offered = pages_sharing + pages_shared + pages_unshared + pages_volatile; + saved = pages_sharing - pages_shared; - if(!offered || !pages_to_scan) return 0; + if(!offered || !pages_to_scan) return 0; - RRDSET *st; + RRDSET *st; - // -------------------------------------------------------------------- + // -------------------------------------------------------------------- - st = rrdset_find("mem.ksm"); - if(!st) { - st = rrdset_create("mem", "ksm", NULL, "ksm", NULL, "Kernel Same Page Merging", "MB", 5000, update_every, RRDSET_TYPE_AREA); + st = rrdset_find("mem.ksm"); + if(!st) { + st = rrdset_create("mem", "ksm", NULL, "ksm", NULL, "Kernel Same Page Merging", "MB", 5000, update_every, RRDSET_TYPE_AREA); - rrddim_add(st, "shared", NULL, 1, 1024 * 1024, RRDDIM_ABSOLUTE); - rrddim_add(st, "unshared", NULL, -1, 1024 * 1024, RRDDIM_ABSOLUTE); - rrddim_add(st, "sharing", NULL, 1, 1024 * 1024, RRDDIM_ABSOLUTE); - rrddim_add(st, "volatile", NULL, -1, 1024 * 1024, RRDDIM_ABSOLUTE); - rrddim_add(st, "to_scan", "to scan", -1, 1024 * 1024, RRDDIM_ABSOLUTE); - } - else rrdset_next(st); + rrddim_add(st, "shared", NULL, 1, 1024 * 1024, RRDDIM_ABSOLUTE); + rrddim_add(st, "unshared", NULL, -1, 1024 * 1024, RRDDIM_ABSOLUTE); + rrddim_add(st, "sharing", NULL, 1, 1024 * 1024, RRDDIM_ABSOLUTE); + rrddim_add(st, "volatile", NULL, -1, 1024 * 1024, RRDDIM_ABSOLUTE); + rrddim_add(st, "to_scan", "to scan", -1, 1024 * 1024, RRDDIM_ABSOLUTE); + } + else rrdset_next(st); - rrddim_set(st, "shared", pages_shared * page_size); - rrddim_set(st, "unshared", pages_unshared * page_size); - rrddim_set(st, "sharing", pages_sharing * page_size); - rrddim_set(st, "volatile", pages_volatile * page_size); - rrddim_set(st, "to_scan", pages_to_scan * page_size); - rrdset_done(st); + rrddim_set(st, "shared", pages_shared * page_size); + rrddim_set(st, "unshared", pages_unshared * page_size); + rrddim_set(st, "sharing", pages_sharing * page_size); + rrddim_set(st, "volatile", pages_volatile * page_size); + rrddim_set(st, "to_scan", pages_to_scan * page_size); + rrdset_done(st); - st = rrdset_find("mem.ksm_savings"); - if(!st) { - st = rrdset_create("mem", "ksm_savings", NULL, "ksm", NULL, "Kernel Same Page Merging Savings", "MB", 5001, update_every, RRDSET_TYPE_AREA); + st = rrdset_find("mem.ksm_savings"); + if(!st) { + st = rrdset_create("mem", "ksm_savings", NULL, "ksm", NULL, "Kernel Same Page Merging Savings", "MB", 5001, update_every, RRDSET_TYPE_AREA); - rrddim_add(st, "savings", NULL, -1, 1024 * 1024, RRDDIM_ABSOLUTE); - rrddim_add(st, "offered", NULL, 1, 1024 * 1024, RRDDIM_ABSOLUTE); - } - else rrdset_next(st); + rrddim_add(st, "savings", NULL, -1, 1024 * 1024, RRDDIM_ABSOLUTE); + rrddim_add(st, "offered", NULL, 1, 1024 * 1024, RRDDIM_ABSOLUTE); + } + else rrdset_next(st); - rrddim_set(st, "savings", saved * page_size); - rrddim_set(st, "offered", offered * page_size); - rrdset_done(st); + rrddim_set(st, "savings", saved * page_size); + rrddim_set(st, "offered", offered * page_size); + rrdset_done(st); - st = rrdset_find("mem.ksm_ratios"); - if(!st) { - st = rrdset_create("mem", "ksm_ratios", NULL, "ksm", NULL, "Kernel Same Page Merging Effectiveness", "percentage", 5002, update_every, RRDSET_TYPE_LINE); - - rrddim_add(st, "savings", NULL, 1, 10000, RRDDIM_ABSOLUTE); - } - else rrdset_next(st); - - rrddim_set(st, "savings", (saved * 1000000) / offered); - rrdset_done(st); + st = rrdset_find("mem.ksm_ratios"); + if(!st) { + st = rrdset_create("mem", "ksm_ratios", NULL, "ksm", NULL, "Kernel Same Page Merging Effectiveness", "percentage", 5002, update_every, RRDSET_TYPE_LINE); + + rrddim_add(st, "savings", NULL, 1, 10000, RRDDIM_ABSOLUTE); + } + else rrdset_next(st); + + rrddim_set(st, "savings", (saved * 1000000) / offered); + rrdset_done(st); - return 0; + return 0; } diff --git a/src/unit_test.c b/src/unit_test.c index 06b7afacb..a77fdb134 100644 --- a/src/unit_test.c +++ b/src/unit_test.c @@ -1,267 +1,258 @@ -#ifdef HAVE_CONFIG_H -#include <config.h> -#endif -#include <stdio.h> -#include <stdlib.h> -#include <string.h> -#include <sys/resource.h> - #include "common.h" -#include "storage_number.h" -#include "rrd.h" -#include "log.h" -#include "web_buffer.h" int check_storage_number(calculated_number n, int debug) { - char buffer[100]; - uint32_t flags = SN_EXISTS; - - storage_number s = pack_storage_number(n, flags); - calculated_number d = unpack_storage_number(s); - - if(!does_storage_number_exist(s)) { - fprintf(stderr, "Exists flags missing for number " CALCULATED_NUMBER_FORMAT "!\n", n); - return 5; - } - - calculated_number ddiff = d - n; - calculated_number dcdiff = ddiff * 100.0 / n; - - if(dcdiff < 0) dcdiff = -dcdiff; - - size_t len = print_calculated_number(buffer, d); - calculated_number p = strtold(buffer, NULL); - calculated_number pdiff = n - p; - calculated_number pcdiff = pdiff * 100.0 / n; - if(pcdiff < 0) pcdiff = -pcdiff; - - if(debug) { - fprintf(stderr, - CALCULATED_NUMBER_FORMAT " original\n" - CALCULATED_NUMBER_FORMAT " packed and unpacked, (stored as 0x%08X, diff " CALCULATED_NUMBER_FORMAT ", " CALCULATED_NUMBER_FORMAT "%%)\n" - "%s printed after unpacked (%zu bytes)\n" - CALCULATED_NUMBER_FORMAT " re-parsed from printed (diff " CALCULATED_NUMBER_FORMAT ", " CALCULATED_NUMBER_FORMAT "%%)\n\n", - n, - d, s, ddiff, dcdiff, - buffer, - len, p, pdiff, pcdiff - ); - if(len != strlen(buffer)) fprintf(stderr, "ERROR: printed number %s is reported to have length %zu but it has %zu\n", buffer, len, strlen(buffer)); - if(dcdiff > ACCURACY_LOSS) fprintf(stderr, "WARNING: packing number " CALCULATED_NUMBER_FORMAT " has accuracy loss %0.7Lf %%\n", n, dcdiff); - if(pcdiff > ACCURACY_LOSS) fprintf(stderr, "WARNING: re-parsing the packed, unpacked and printed number " CALCULATED_NUMBER_FORMAT " has accuracy loss %0.7Lf %%\n", n, pcdiff); - } - - if(len != strlen(buffer)) return 1; - if(dcdiff > ACCURACY_LOSS) return 3; - if(pcdiff > ACCURACY_LOSS) return 4; - return 0; + char buffer[100]; + uint32_t flags = SN_EXISTS; + + storage_number s = pack_storage_number(n, flags); + calculated_number d = unpack_storage_number(s); + + if(!does_storage_number_exist(s)) { + fprintf(stderr, "Exists flags missing for number " CALCULATED_NUMBER_FORMAT "!\n", n); + return 5; + } + + calculated_number ddiff = d - n; + calculated_number dcdiff = ddiff * 100.0 / n; + + if(dcdiff < 0) dcdiff = -dcdiff; + + size_t len = print_calculated_number(buffer, d); + calculated_number p = strtold(buffer, NULL); + calculated_number pdiff = n - p; + calculated_number pcdiff = pdiff * 100.0 / n; + if(pcdiff < 0) pcdiff = -pcdiff; + + if(debug) { + fprintf(stderr, + CALCULATED_NUMBER_FORMAT " original\n" + CALCULATED_NUMBER_FORMAT " packed and unpacked, (stored as 0x%08X, diff " CALCULATED_NUMBER_FORMAT ", " CALCULATED_NUMBER_FORMAT "%%)\n" + "%s printed after unpacked (%zu bytes)\n" + CALCULATED_NUMBER_FORMAT " re-parsed from printed (diff " CALCULATED_NUMBER_FORMAT ", " CALCULATED_NUMBER_FORMAT "%%)\n\n", + n, + d, s, ddiff, dcdiff, + buffer, + len, p, pdiff, pcdiff + ); + if(len != strlen(buffer)) fprintf(stderr, "ERROR: printed number %s is reported to have length %zu but it has %zu\n", buffer, len, strlen(buffer)); + if(dcdiff > ACCURACY_LOSS) fprintf(stderr, "WARNING: packing number " CALCULATED_NUMBER_FORMAT " has accuracy loss %0.7Lf %%\n", n, dcdiff); + if(pcdiff > ACCURACY_LOSS) fprintf(stderr, "WARNING: re-parsing the packed, unpacked and printed number " CALCULATED_NUMBER_FORMAT " has accuracy loss %0.7Lf %%\n", n, pcdiff); + } + + if(len != strlen(buffer)) return 1; + if(dcdiff > ACCURACY_LOSS) return 3; + if(pcdiff > ACCURACY_LOSS) return 4; + return 0; } void benchmark_storage_number(int loop, int multiplier) { - int i, j; - calculated_number n, d; - storage_number s; - unsigned long long user, system, total, mine, their; + int i, j; + calculated_number n, d; + storage_number s; + unsigned long long user, system, total, mine, their; - char buffer[100]; + char buffer[100]; - struct rusage now, last; + struct rusage now, last; - fprintf(stderr, "\n\nBenchmarking %d numbers, please wait...\n\n", loop); + fprintf(stderr, "\n\nBenchmarking %d numbers, please wait...\n\n", loop); - // ------------------------------------------------------------------------ + // ------------------------------------------------------------------------ - fprintf(stderr, "SYSTEM LONG DOUBLE SIZE: %zu bytes\n", sizeof(calculated_number)); - fprintf(stderr, "NETDATA FLOATING POINT SIZE: %zu bytes\n", sizeof(storage_number)); + fprintf(stderr, "SYSTEM LONG DOUBLE SIZE: %zu bytes\n", sizeof(calculated_number)); + fprintf(stderr, "NETDATA FLOATING POINT SIZE: %zu bytes\n", sizeof(storage_number)); - mine = (calculated_number)sizeof(storage_number) * (calculated_number)loop; - their = (calculated_number)sizeof(calculated_number) * (calculated_number)loop; + mine = (calculated_number)sizeof(storage_number) * (calculated_number)loop; + their = (calculated_number)sizeof(calculated_number) * (calculated_number)loop; - if(mine > their) { - fprintf(stderr, "\nNETDATA NEEDS %0.2Lf TIMES MORE MEMORY. Sorry!\n", (long double)(mine / their)); - } - else { - fprintf(stderr, "\nNETDATA INTERNAL FLOATING POINT ARITHMETICS NEEDS %0.2Lf TIMES LESS MEMORY.\n", (long double)(their / mine)); - } + if(mine > their) { + fprintf(stderr, "\nNETDATA NEEDS %0.2Lf TIMES MORE MEMORY. Sorry!\n", (long double)(mine / their)); + } + else { + fprintf(stderr, "\nNETDATA INTERNAL FLOATING POINT ARITHMETICS NEEDS %0.2Lf TIMES LESS MEMORY.\n", (long double)(their / mine)); + } - fprintf(stderr, "\nNETDATA FLOATING POINT\n"); - fprintf(stderr, "MIN POSITIVE VALUE " CALCULATED_NUMBER_FORMAT "\n", (calculated_number)STORAGE_NUMBER_POSITIVE_MIN); - fprintf(stderr, "MAX POSITIVE VALUE " CALCULATED_NUMBER_FORMAT "\n", (calculated_number)STORAGE_NUMBER_POSITIVE_MAX); - fprintf(stderr, "MIN NEGATIVE VALUE " CALCULATED_NUMBER_FORMAT "\n", (calculated_number)STORAGE_NUMBER_NEGATIVE_MIN); - fprintf(stderr, "MAX NEGATIVE VALUE " CALCULATED_NUMBER_FORMAT "\n", (calculated_number)STORAGE_NUMBER_NEGATIVE_MAX); - fprintf(stderr, "Maximum accuracy loss: " CALCULATED_NUMBER_FORMAT "%%\n\n\n", (calculated_number)ACCURACY_LOSS); + fprintf(stderr, "\nNETDATA FLOATING POINT\n"); + fprintf(stderr, "MIN POSITIVE VALUE " CALCULATED_NUMBER_FORMAT "\n", (calculated_number)STORAGE_NUMBER_POSITIVE_MIN); + fprintf(stderr, "MAX POSITIVE VALUE " CALCULATED_NUMBER_FORMAT "\n", (calculated_number)STORAGE_NUMBER_POSITIVE_MAX); + fprintf(stderr, "MIN NEGATIVE VALUE " CALCULATED_NUMBER_FORMAT "\n", (calculated_number)STORAGE_NUMBER_NEGATIVE_MIN); + fprintf(stderr, "MAX NEGATIVE VALUE " CALCULATED_NUMBER_FORMAT "\n", (calculated_number)STORAGE_NUMBER_NEGATIVE_MAX); + fprintf(stderr, "Maximum accuracy loss: " CALCULATED_NUMBER_FORMAT "%%\n\n\n", (calculated_number)ACCURACY_LOSS); - // ------------------------------------------------------------------------ + // ------------------------------------------------------------------------ - fprintf(stderr, "INTERNAL LONG DOUBLE PRINTING: "); - getrusage(RUSAGE_SELF, &last); + fprintf(stderr, "INTERNAL LONG DOUBLE PRINTING: "); + getrusage(RUSAGE_SELF, &last); - // do the job - for(j = 1; j < 11 ;j++) { - n = STORAGE_NUMBER_POSITIVE_MIN * j; + // do the job + for(j = 1; j < 11 ;j++) { + n = STORAGE_NUMBER_POSITIVE_MIN * j; - for(i = 0; i < loop ;i++) { - n *= multiplier; - if(n > STORAGE_NUMBER_POSITIVE_MAX) n = STORAGE_NUMBER_POSITIVE_MIN; + for(i = 0; i < loop ;i++) { + n *= multiplier; + if(n > STORAGE_NUMBER_POSITIVE_MAX) n = STORAGE_NUMBER_POSITIVE_MIN; - print_calculated_number(buffer, n); - } - } + print_calculated_number(buffer, n); + } + } - getrusage(RUSAGE_SELF, &now); - user = now.ru_utime.tv_sec * 1000000ULL + now.ru_utime.tv_usec - last.ru_utime.tv_sec * 1000000ULL + last.ru_utime.tv_usec; - system = now.ru_stime.tv_sec * 1000000ULL + now.ru_stime.tv_usec - last.ru_stime.tv_sec * 1000000ULL + last.ru_stime.tv_usec; - total = user + system; - mine = total; + getrusage(RUSAGE_SELF, &now); + user = now.ru_utime.tv_sec * 1000000ULL + now.ru_utime.tv_usec - last.ru_utime.tv_sec * 1000000ULL + last.ru_utime.tv_usec; + system = now.ru_stime.tv_sec * 1000000ULL + now.ru_stime.tv_usec - last.ru_stime.tv_sec * 1000000ULL + last.ru_stime.tv_usec; + total = user + system; + mine = total; - fprintf(stderr, "user %0.5Lf, system %0.5Lf, total %0.5Lf\n", (long double)(user / 1000000.0), (long double)(system / 1000000.0), (long double)(total / 1000000.0)); + fprintf(stderr, "user %0.5Lf, system %0.5Lf, total %0.5Lf\n", (long double)(user / 1000000.0), (long double)(system / 1000000.0), (long double)(total / 1000000.0)); - // ------------------------------------------------------------------------ + // ------------------------------------------------------------------------ - fprintf(stderr, "SYSTEM LONG DOUBLE PRINTING: "); - getrusage(RUSAGE_SELF, &last); + fprintf(stderr, "SYSTEM LONG DOUBLE PRINTING: "); + getrusage(RUSAGE_SELF, &last); - // do the job - for(j = 1; j < 11 ;j++) { - n = STORAGE_NUMBER_POSITIVE_MIN * j; + // do the job + for(j = 1; j < 11 ;j++) { + n = STORAGE_NUMBER_POSITIVE_MIN * j; - for(i = 0; i < loop ;i++) { - n *= multiplier; - if(n > STORAGE_NUMBER_POSITIVE_MAX) n = STORAGE_NUMBER_POSITIVE_MIN; - snprintfz(buffer, 100, CALCULATED_NUMBER_FORMAT, n); - } - } + for(i = 0; i < loop ;i++) { + n *= multiplier; + if(n > STORAGE_NUMBER_POSITIVE_MAX) n = STORAGE_NUMBER_POSITIVE_MIN; + snprintfz(buffer, 100, CALCULATED_NUMBER_FORMAT, n); + } + } - getrusage(RUSAGE_SELF, &now); - user = now.ru_utime.tv_sec * 1000000ULL + now.ru_utime.tv_usec - last.ru_utime.tv_sec * 1000000ULL + last.ru_utime.tv_usec; - system = now.ru_stime.tv_sec * 1000000ULL + now.ru_stime.tv_usec - last.ru_stime.tv_sec * 1000000ULL + last.ru_stime.tv_usec; - total = user + system; - their = total; + getrusage(RUSAGE_SELF, &now); + user = now.ru_utime.tv_sec * 1000000ULL + now.ru_utime.tv_usec - last.ru_utime.tv_sec * 1000000ULL + last.ru_utime.tv_usec; + system = now.ru_stime.tv_sec * 1000000ULL + now.ru_stime.tv_usec - last.ru_stime.tv_sec * 1000000ULL + last.ru_stime.tv_usec; + total = user + system; + their = total; - fprintf(stderr, "user %0.5Lf, system %0.5Lf, total %0.5Lf\n", (long double)(user / 1000000.0), (long double)(system / 1000000.0), (long double)(total / 1000000.0)); + fprintf(stderr, "user %0.5Lf, system %0.5Lf, total %0.5Lf\n", (long double)(user / 1000000.0), (long double)(system / 1000000.0), (long double)(total / 1000000.0)); - if(mine > total) { - fprintf(stderr, "NETDATA CODE IS SLOWER %0.2Lf %%\n", (long double)(mine * 100.0 / their - 100.0)); - } - else { - fprintf(stderr, "NETDATA CODE IS F A S T E R %0.2Lf %%\n", (long double)(their * 100.0 / mine - 100.0)); - } + if(mine > total) { + fprintf(stderr, "NETDATA CODE IS SLOWER %0.2Lf %%\n", (long double)(mine * 100.0 / their - 100.0)); + } + else { + fprintf(stderr, "NETDATA CODE IS F A S T E R %0.2Lf %%\n", (long double)(their * 100.0 / mine - 100.0)); + } - // ------------------------------------------------------------------------ + // ------------------------------------------------------------------------ - fprintf(stderr, "\nINTERNAL LONG DOUBLE PRINTING WITH PACK / UNPACK: "); - getrusage(RUSAGE_SELF, &last); + fprintf(stderr, "\nINTERNAL LONG DOUBLE PRINTING WITH PACK / UNPACK: "); + getrusage(RUSAGE_SELF, &last); - // do the job - for(j = 1; j < 11 ;j++) { - n = STORAGE_NUMBER_POSITIVE_MIN * j; + // do the job + for(j = 1; j < 11 ;j++) { + n = STORAGE_NUMBER_POSITIVE_MIN * j; - for(i = 0; i < loop ;i++) { - n *= multiplier; - if(n > STORAGE_NUMBER_POSITIVE_MAX) n = STORAGE_NUMBER_POSITIVE_MIN; + for(i = 0; i < loop ;i++) { + n *= multiplier; + if(n > STORAGE_NUMBER_POSITIVE_MAX) n = STORAGE_NUMBER_POSITIVE_MIN; - s = pack_storage_number(n, 1); - d = unpack_storage_number(s); - print_calculated_number(buffer, d); - } - } + s = pack_storage_number(n, 1); + d = unpack_storage_number(s); + print_calculated_number(buffer, d); + } + } - getrusage(RUSAGE_SELF, &now); - user = now.ru_utime.tv_sec * 1000000ULL + now.ru_utime.tv_usec - last.ru_utime.tv_sec * 1000000ULL + last.ru_utime.tv_usec; - system = now.ru_stime.tv_sec * 1000000ULL + now.ru_stime.tv_usec - last.ru_stime.tv_sec * 1000000ULL + last.ru_stime.tv_usec; - total = user + system; - mine = total; - - fprintf(stderr, "user %0.5Lf, system %0.5Lf, total %0.5Lf\n", (long double)(user / 1000000.0), (long double)(system / 1000000.0), (long double)(total / 1000000.0)); + getrusage(RUSAGE_SELF, &now); + user = now.ru_utime.tv_sec * 1000000ULL + now.ru_utime.tv_usec - last.ru_utime.tv_sec * 1000000ULL + last.ru_utime.tv_usec; + system = now.ru_stime.tv_sec * 1000000ULL + now.ru_stime.tv_usec - last.ru_stime.tv_sec * 1000000ULL + last.ru_stime.tv_usec; + total = user + system; + mine = total; + + fprintf(stderr, "user %0.5Lf, system %0.5Lf, total %0.5Lf\n", (long double)(user / 1000000.0), (long double)(system / 1000000.0), (long double)(total / 1000000.0)); - if(mine > their) { - fprintf(stderr, "WITH PACKING UNPACKING NETDATA CODE IS SLOWER %0.2Lf %%\n", (long double)(mine * 100.0 / their - 100.0)); - } - else { - fprintf(stderr, "EVEN WITH PACKING AND UNPACKING, NETDATA CODE IS F A S T E R %0.2Lf %%\n", (long double)(their * 100.0 / mine - 100.0)); - } + if(mine > their) { + fprintf(stderr, "WITH PACKING UNPACKING NETDATA CODE IS SLOWER %0.2Lf %%\n", (long double)(mine * 100.0 / their - 100.0)); + } + else { + fprintf(stderr, "EVEN WITH PACKING AND UNPACKING, NETDATA CODE IS F A S T E R %0.2Lf %%\n", (long double)(their * 100.0 / mine - 100.0)); + } - // ------------------------------------------------------------------------ + // ------------------------------------------------------------------------ } static int check_storage_number_exists() { - uint32_t flags = SN_EXISTS; - - - for(flags = 0; flags < 7 ; flags++) { - if(get_storage_number_flags(flags << 24) != flags << 24) { - fprintf(stderr, "Flag 0x%08x is not checked correctly. It became 0x%08x\n", flags << 24, get_storage_number_flags(flags << 24)); - return 1; - } - } - - flags = SN_EXISTS; - calculated_number n = 0.0; - - storage_number s = pack_storage_number(n, flags); - calculated_number d = unpack_storage_number(s); - if(get_storage_number_flags(s) != flags) { - fprintf(stderr, "Wrong flags. Given %08x, Got %08x!\n", flags, get_storage_number_flags(s)); - return 1; - } - if(n != d) { - fprintf(stderr, "Wrong number returned. Expected " CALCULATED_NUMBER_FORMAT ", returned " CALCULATED_NUMBER_FORMAT "!\n", n, d); - return 1; - } - - return 0; + uint32_t flags = SN_EXISTS; + + + for(flags = 0; flags < 7 ; flags++) { + if(get_storage_number_flags(flags << 24) != flags << 24) { + fprintf(stderr, "Flag 0x%08x is not checked correctly. It became 0x%08x\n", flags << 24, get_storage_number_flags(flags << 24)); + return 1; + } + } + + flags = SN_EXISTS; + calculated_number n = 0.0; + + storage_number s = pack_storage_number(n, flags); + calculated_number d = unpack_storage_number(s); + if(get_storage_number_flags(s) != flags) { + fprintf(stderr, "Wrong flags. Given %08x, Got %08x!\n", flags, get_storage_number_flags(s)); + return 1; + } + if(n != d) { + fprintf(stderr, "Wrong number returned. Expected " CALCULATED_NUMBER_FORMAT ", returned " CALCULATED_NUMBER_FORMAT "!\n", n, d); + return 1; + } + + return 0; } int unit_test_storage() { - if(check_storage_number_exists()) return 0; + if(check_storage_number_exists()) return 0; - calculated_number c, a = 0; - int i, j, g, r = 0; + calculated_number c, a = 0; + int i, j, g, r = 0; - for(g = -1; g <= 1 ; g++) { - a = 0; + for(g = -1; g <= 1 ; g++) { + a = 0; - if(!g) continue; + if(!g) continue; - for(j = 0; j < 9 ;j++) { - a += 0.0000001; - c = a * g; - for(i = 0; i < 21 ;i++, c *= 10) { - if(c > 0 && c < STORAGE_NUMBER_POSITIVE_MIN) continue; - if(c < 0 && c > STORAGE_NUMBER_NEGATIVE_MAX) continue; + for(j = 0; j < 9 ;j++) { + a += 0.0000001; + c = a * g; + for(i = 0; i < 21 ;i++, c *= 10) { + if(c > 0 && c < STORAGE_NUMBER_POSITIVE_MIN) continue; + if(c < 0 && c > STORAGE_NUMBER_NEGATIVE_MAX) continue; - if(check_storage_number(c, 1)) return 1; - } - } - } + if(check_storage_number(c, 1)) return 1; + } + } + } - benchmark_storage_number(1000000, 2); - return r; + benchmark_storage_number(1000000, 2); + return r; } // -------------------------------------------------------------------------------------------------------------------- struct feed_values { - unsigned long long microseconds; - calculated_number value; + unsigned long long microseconds; + collected_number value; }; struct test { - char name[100]; - char description[1024]; + char name[100]; + char description[1024]; + + int update_every; + unsigned long long multiplier; + unsigned long long divisor; + int algorithm; - int update_every; - unsigned long long multiplier; - unsigned long long divisor; - int algorithm; + unsigned long feed_entries; + unsigned long result_entries; + struct feed_values *feed; + calculated_number *results; - unsigned long feed_entries; - unsigned long result_entries; - struct feed_values *feed; - calculated_number *results; + collected_number *feed2; + calculated_number *results2; }; // -------------------------------------------------------------------------------------------------------------------- @@ -269,33 +260,35 @@ struct test { // test absolute values stored struct feed_values test1_feed[] = { - { 0, 10 }, - { 1000000, 20 }, - { 1000000, 30 }, - { 1000000, 40 }, - { 1000000, 50 }, - { 1000000, 60 }, - { 1000000, 70 }, - { 1000000, 80 }, - { 1000000, 90 }, - { 1000000, 100 }, + { 0, 10 }, + { 1000000, 20 }, + { 1000000, 30 }, + { 1000000, 40 }, + { 1000000, 50 }, + { 1000000, 60 }, + { 1000000, 70 }, + { 1000000, 80 }, + { 1000000, 90 }, + { 1000000, 100 }, }; calculated_number test1_results[] = { - 20, 30, 40, 50, 60, 70, 80, 90, 100 + 20, 30, 40, 50, 60, 70, 80, 90, 100 }; struct test test1 = { - "test1", // name - "test absolute values stored at exactly second boundaries", - 1, // update_every - 1, // multiplier - 1, // divisor - RRDDIM_ABSOLUTE, // algorithm - 10, // feed entries - 9, // result entries - test1_feed, // feed - test1_results // results + "test1", // name + "test absolute values stored at exactly second boundaries", + 1, // update_every + 1, // multiplier + 1, // divisor + RRDDIM_ABSOLUTE, // algorithm + 10, // feed entries + 9, // result entries + test1_feed, // feed + test1_results, // results + NULL, // feed2 + NULL // results2 }; // -------------------------------------------------------------------------------------------------------------------- @@ -303,497 +296,667 @@ struct test test1 = { // test absolute values stored in the middle of second boundaries struct feed_values test2_feed[] = { - { 500000, 10 }, - { 1000000, 20 }, - { 1000000, 30 }, - { 1000000, 40 }, - { 1000000, 50 }, - { 1000000, 60 }, - { 1000000, 70 }, - { 1000000, 80 }, - { 1000000, 90 }, - { 1000000, 100 }, + { 500000, 10 }, + { 1000000, 20 }, + { 1000000, 30 }, + { 1000000, 40 }, + { 1000000, 50 }, + { 1000000, 60 }, + { 1000000, 70 }, + { 1000000, 80 }, + { 1000000, 90 }, + { 1000000, 100 }, }; calculated_number test2_results[] = { - 20, 30, 40, 50, 60, 70, 80, 90, 100 + 20, 30, 40, 50, 60, 70, 80, 90, 100 }; struct test test2 = { - "test2", // name - "test absolute values stored in the middle of second boundaries", - 1, // update_every - 1, // multiplier - 1, // divisor - RRDDIM_ABSOLUTE, // algorithm - 10, // feed entries - 9, // result entries - test2_feed, // feed - test2_results // results + "test2", // name + "test absolute values stored in the middle of second boundaries", + 1, // update_every + 1, // multiplier + 1, // divisor + RRDDIM_ABSOLUTE, // algorithm + 10, // feed entries + 9, // result entries + test2_feed, // feed + test2_results, // results + NULL, // feed2 + NULL // results2 }; // -------------------------------------------------------------------------------------------------------------------- // test3 struct feed_values test3_feed[] = { - { 0, 10 }, - { 1000000, 20 }, - { 1000000, 30 }, - { 1000000, 40 }, - { 1000000, 50 }, - { 1000000, 60 }, - { 1000000, 70 }, - { 1000000, 80 }, - { 1000000, 90 }, - { 1000000, 100 }, + { 0, 10 }, + { 1000000, 20 }, + { 1000000, 30 }, + { 1000000, 40 }, + { 1000000, 50 }, + { 1000000, 60 }, + { 1000000, 70 }, + { 1000000, 80 }, + { 1000000, 90 }, + { 1000000, 100 }, }; calculated_number test3_results[] = { - 10, 10, 10, 10, 10, 10, 10, 10, 10 + 10, 10, 10, 10, 10, 10, 10, 10, 10 }; struct test test3 = { - "test3", // name - "test incremental values stored at exactly second boundaries", - 1, // update_every - 1, // multiplier - 1, // divisor - RRDDIM_INCREMENTAL, // algorithm - 10, // feed entries - 9, // result entries - test3_feed, // feed - test3_results // results + "test3", // name + "test incremental values stored at exactly second boundaries", + 1, // update_every + 1, // multiplier + 1, // divisor + RRDDIM_INCREMENTAL, // algorithm + 10, // feed entries + 9, // result entries + test3_feed, // feed + test3_results, // results + NULL, // feed2 + NULL // results2 }; // -------------------------------------------------------------------------------------------------------------------- // test4 struct feed_values test4_feed[] = { - { 500000, 10 }, - { 1000000, 20 }, - { 1000000, 30 }, - { 1000000, 40 }, - { 1000000, 50 }, - { 1000000, 60 }, - { 1000000, 70 }, - { 1000000, 80 }, - { 1000000, 90 }, - { 1000000, 100 }, + { 500000, 10 }, + { 1000000, 20 }, + { 1000000, 30 }, + { 1000000, 40 }, + { 1000000, 50 }, + { 1000000, 60 }, + { 1000000, 70 }, + { 1000000, 80 }, + { 1000000, 90 }, + { 1000000, 100 }, }; calculated_number test4_results[] = { - 5, 10, 10, 10, 10, 10, 10, 10, 10 + 5, 10, 10, 10, 10, 10, 10, 10, 10 }; struct test test4 = { - "test4", // name - "test incremental values stored in the middle of second boundaries", - 1, // update_every - 1, // multiplier - 1, // divisor - RRDDIM_INCREMENTAL, // algorithm - 10, // feed entries - 9, // result entries - test4_feed, // feed - test4_results // results + "test4", // name + "test incremental values stored in the middle of second boundaries", + 1, // update_every + 1, // multiplier + 1, // divisor + RRDDIM_INCREMENTAL, // algorithm + 10, // feed entries + 9, // result entries + test4_feed, // feed + test4_results, // results + NULL, // feed2 + NULL // results2 }; // -------------------------------------------------------------------------------------------------------------------- // test5 struct feed_values test5_feed[] = { - { 500000, 1000 }, - { 1000000, 2000 }, - { 1000000, 2000 }, - { 1000000, 2000 }, - { 1000000, 3000 }, - { 1000000, 2000 }, - { 1000000, 2000 }, - { 1000000, 2000 }, - { 1000000, 2000 }, - { 1000000, 2000 }, + { 500000, 1000 }, + { 1000000, 2000 }, + { 1000000, 2000 }, + { 1000000, 2000 }, + { 1000000, 3000 }, + { 1000000, 2000 }, + { 1000000, 2000 }, + { 1000000, 2000 }, + { 1000000, 2000 }, + { 1000000, 2000 }, }; calculated_number test5_results[] = { - 500, 500, 0, 500, 500, 0, 0, 0, 0 + 500, 500, 0, 500, 500, 0, 0, 0, 0 }; struct test test5 = { - "test5", // name - "test incremental values ups and downs", - 1, // update_every - 1, // multiplier - 1, // divisor - RRDDIM_INCREMENTAL, // algorithm - 10, // feed entries - 9, // result entries - test5_feed, // feed - test5_results // results + "test5", // name + "test incremental values ups and downs", + 1, // update_every + 1, // multiplier + 1, // divisor + RRDDIM_INCREMENTAL, // algorithm + 10, // feed entries + 9, // result entries + test5_feed, // feed + test5_results, // results + NULL, // feed2 + NULL // results2 }; // -------------------------------------------------------------------------------------------------------------------- // test6 struct feed_values test6_feed[] = { - { 250000, 1000 }, - { 250000, 2000 }, - { 250000, 3000 }, - { 250000, 4000 }, - { 250000, 5000 }, - { 250000, 6000 }, - { 250000, 7000 }, - { 250000, 8000 }, - { 250000, 9000 }, - { 250000, 10000 }, - { 250000, 11000 }, - { 250000, 12000 }, - { 250000, 13000 }, - { 250000, 14000 }, - { 250000, 15000 }, - { 250000, 16000 }, + { 250000, 1000 }, + { 250000, 2000 }, + { 250000, 3000 }, + { 250000, 4000 }, + { 250000, 5000 }, + { 250000, 6000 }, + { 250000, 7000 }, + { 250000, 8000 }, + { 250000, 9000 }, + { 250000, 10000 }, + { 250000, 11000 }, + { 250000, 12000 }, + { 250000, 13000 }, + { 250000, 14000 }, + { 250000, 15000 }, + { 250000, 16000 }, }; calculated_number test6_results[] = { - 3000, 4000, 4000, 4000 + 3000, 4000, 4000, 4000 }; struct test test6 = { - "test6", // name - "test incremental values updated within the same second", - 1, // update_every - 1, // multiplier - 1, // divisor - RRDDIM_INCREMENTAL, // algorithm - 16, // feed entries - 4, // result entries - test6_feed, // feed - test6_results // results + "test6", // name + "test incremental values updated within the same second", + 1, // update_every + 1, // multiplier + 1, // divisor + RRDDIM_INCREMENTAL, // algorithm + 16, // feed entries + 4, // result entries + test6_feed, // feed + test6_results, // results + NULL, // feed2 + NULL // results2 }; // -------------------------------------------------------------------------------------------------------------------- // test7 struct feed_values test7_feed[] = { - { 500000, 1000 }, - { 2000000, 2000 }, - { 2000000, 3000 }, - { 2000000, 4000 }, - { 2000000, 5000 }, - { 2000000, 6000 }, - { 2000000, 7000 }, - { 2000000, 8000 }, - { 2000000, 9000 }, - { 2000000, 10000 }, + { 500000, 1000 }, + { 2000000, 2000 }, + { 2000000, 3000 }, + { 2000000, 4000 }, + { 2000000, 5000 }, + { 2000000, 6000 }, + { 2000000, 7000 }, + { 2000000, 8000 }, + { 2000000, 9000 }, + { 2000000, 10000 }, }; calculated_number test7_results[] = { - 250, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500 + 250, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500 }; struct test test7 = { - "test7", // name - "test incremental values updated in long durations", - 1, // update_every - 1, // multiplier - 1, // divisor - RRDDIM_INCREMENTAL, // algorithm - 10, // feed entries - 18, // result entries - test7_feed, // feed - test7_results // results + "test7", // name + "test incremental values updated in long durations", + 1, // update_every + 1, // multiplier + 1, // divisor + RRDDIM_INCREMENTAL, // algorithm + 10, // feed entries + 18, // result entries + test7_feed, // feed + test7_results, // results + NULL, // feed2 + NULL // results2 }; // -------------------------------------------------------------------------------------------------------------------- // test8 struct feed_values test8_feed[] = { - { 500000, 1000 }, - { 2000000, 2000 }, - { 2000000, 3000 }, - { 2000000, 4000 }, - { 2000000, 5000 }, - { 2000000, 6000 }, + { 500000, 1000 }, + { 2000000, 2000 }, + { 2000000, 3000 }, + { 2000000, 4000 }, + { 2000000, 5000 }, + { 2000000, 6000 }, }; calculated_number test8_results[] = { - 1250, 2000, 2250, 3000, 3250, 4000, 4250, 5000, 5250, 6000 + 1250, 2000, 2250, 3000, 3250, 4000, 4250, 5000, 5250, 6000 }; struct test test8 = { - "test8", // name - "test absolute values updated in long durations", - 1, // update_every - 1, // multiplier - 1, // divisor - RRDDIM_ABSOLUTE, // algorithm - 6, // feed entries - 10, // result entries - test8_feed, // feed - test8_results // results + "test8", // name + "test absolute values updated in long durations", + 1, // update_every + 1, // multiplier + 1, // divisor + RRDDIM_ABSOLUTE, // algorithm + 6, // feed entries + 10, // result entries + test8_feed, // feed + test8_results, // results + NULL, // feed2 + NULL // results2 }; // -------------------------------------------------------------------------------------------------------------------- // test9 struct feed_values test9_feed[] = { - { 250000, 1000 }, - { 250000, 2000 }, - { 250000, 3000 }, - { 250000, 4000 }, - { 250000, 5000 }, - { 250000, 6000 }, - { 250000, 7000 }, - { 250000, 8000 }, - { 250000, 9000 }, - { 250000, 10000 }, - { 250000, 11000 }, - { 250000, 12000 }, - { 250000, 13000 }, - { 250000, 14000 }, - { 250000, 15000 }, - { 250000, 16000 }, + { 250000, 1000 }, + { 250000, 2000 }, + { 250000, 3000 }, + { 250000, 4000 }, + { 250000, 5000 }, + { 250000, 6000 }, + { 250000, 7000 }, + { 250000, 8000 }, + { 250000, 9000 }, + { 250000, 10000 }, + { 250000, 11000 }, + { 250000, 12000 }, + { 250000, 13000 }, + { 250000, 14000 }, + { 250000, 15000 }, + { 250000, 16000 }, }; calculated_number test9_results[] = { - 4000, 8000, 12000, 16000 + 4000, 8000, 12000, 16000 }; struct test test9 = { - "test9", // name - "test absolute values updated within the same second", - 1, // update_every - 1, // multiplier - 1, // divisor - RRDDIM_ABSOLUTE, // algorithm - 16, // feed entries - 4, // result entries - test9_feed, // feed - test9_results // results + "test9", // name + "test absolute values updated within the same second", + 1, // update_every + 1, // multiplier + 1, // divisor + RRDDIM_ABSOLUTE, // algorithm + 16, // feed entries + 4, // result entries + test9_feed, // feed + test9_results, // results + NULL, // feed2 + NULL // results2 }; // -------------------------------------------------------------------------------------------------------------------- // test10 struct feed_values test10_feed[] = { - { 500000, 1000 }, - { 600000, 1000 + 600 }, - { 200000, 1600 + 200 }, - { 1000000, 1800 + 1000 }, - { 200000, 2800 + 200 }, - { 2000000, 3000 + 2000 }, - { 600000, 5000 + 600 }, - { 400000, 5600 + 400 }, - { 900000, 6000 + 900 }, - { 1000000, 6900 + 1000 }, + { 500000, 1000 }, + { 600000, 1000 + 600 }, + { 200000, 1600 + 200 }, + { 1000000, 1800 + 1000 }, + { 200000, 2800 + 200 }, + { 2000000, 3000 + 2000 }, + { 600000, 5000 + 600 }, + { 400000, 5600 + 400 }, + { 900000, 6000 + 900 }, + { 1000000, 6900 + 1000 }, }; calculated_number test10_results[] = { - 500, 1000, 1000, 1000, 1000, 1000, 1000 + 500, 1000, 1000, 1000, 1000, 1000, 1000 }; struct test test10 = { - "test10", // name - "test incremental values updated in short and long durations", - 1, // update_every - 1, // multiplier - 1, // divisor - RRDDIM_INCREMENTAL, // algorithm - 10, // feed entries - 7, // result entries - test10_feed, // feed - test10_results // results + "test10", // name + "test incremental values updated in short and long durations", + 1, // update_every + 1, // multiplier + 1, // divisor + RRDDIM_INCREMENTAL, // algorithm + 10, // feed entries + 7, // result entries + test10_feed, // feed + test10_results, // results + NULL, // feed2 + NULL // results2 +}; + +// -------------------------------------------------------------------------------------------------------------------- +// test11 + +struct feed_values test11_feed[] = { + { 0, 10 }, + { 1000000, 20 }, + { 1000000, 30 }, + { 1000000, 40 }, + { 1000000, 50 }, + { 1000000, 60 }, + { 1000000, 70 }, + { 1000000, 80 }, + { 1000000, 90 }, + { 1000000, 100 }, +}; + +collected_number test11_feed2[] = { + 10, 20, 30, 40, 50, 60, 70, 80, 90, 100 +}; + +calculated_number test11_results[] = { + 50, 50, 50, 50, 50, 50, 50, 50, 50 +}; + +calculated_number test11_results2[] = { + 50, 50, 50, 50, 50, 50, 50, 50, 50 +}; + +struct test test11 = { + "test11", // name + "test percentage-of-incremental-row with equal values", + 1, // update_every + 1, // multiplier + 1, // divisor + RRDDIM_PCENT_OVER_DIFF_TOTAL, // algorithm + 10, // feed entries + 9, // result entries + test11_feed, // feed + test11_results, // results + test11_feed2, // feed2 + test11_results2 // results2 +}; + +// -------------------------------------------------------------------------------------------------------------------- +// test12 + +struct feed_values test12_feed[] = { + { 0, 10 }, + { 1000000, 20 }, + { 1000000, 30 }, + { 1000000, 40 }, + { 1000000, 50 }, + { 1000000, 60 }, + { 1000000, 70 }, + { 1000000, 80 }, + { 1000000, 90 }, + { 1000000, 100 }, +}; + +collected_number test12_feed2[] = { + 10*3, 20*3, 30*3, 40*3, 50*3, 60*3, 70*3, 80*3, 90*3, 100*3 +}; + +calculated_number test12_results[] = { + 25, 25, 25, 25, 25, 25, 25, 25, 25 +}; + +calculated_number test12_results2[] = { + 75, 75, 75, 75, 75, 75, 75, 75, 75 +}; + +struct test test12 = { + "test12", // name + "test percentage-of-incremental-row with equal values", + 1, // update_every + 1, // multiplier + 1, // divisor + RRDDIM_PCENT_OVER_DIFF_TOTAL, // algorithm + 10, // feed entries + 9, // result entries + test12_feed, // feed + test12_results, // results + test12_feed2, // feed2 + test12_results2 // results2 +}; + +// -------------------------------------------------------------------------------------------------------------------- +// test13 + +struct feed_values test13_feed[] = { + { 500000, 1000 }, + { 600000, 1000 + 600 }, + { 200000, 1600 + 200 }, + { 1000000, 1800 + 1000 }, + { 200000, 2800 + 200 }, + { 2000000, 3000 + 2000 }, + { 600000, 5000 + 600 }, + { 400000, 5600 + 400 }, + { 900000, 6000 + 900 }, + { 1000000, 6900 + 1000 }, +}; + +calculated_number test13_results[] = { + 83.3333300, 100, 100, 100, 100, 100, 100 +}; + +struct test test13 = { + "test13", // name + "test incremental values updated in short and long durations", + 1, // update_every + 1, // multiplier + 1, // divisor + RRDDIM_PCENT_OVER_DIFF_TOTAL, // algorithm + 10, // feed entries + 7, // result entries + test13_feed, // feed + test13_results, // results + NULL, // feed2 + NULL // results2 }; // -------------------------------------------------------------------------------------------------------------------- int run_test(struct test *test) { - fprintf(stderr, "\nRunning test '%s':\n%s\n", test->name, test->description); - - rrd_memory_mode = RRD_MEMORY_MODE_RAM; - rrd_update_every = test->update_every; - - char name[101]; - snprintfz(name, 100, "unittest-%s", test->name); - - // create the chart - RRDSET *st = rrdset_create("netdata", name, name, "netdata", NULL, "Unit Testing", "a value", 1, 1, RRDSET_TYPE_LINE); - RRDDIM *rd = rrddim_add(st, "dimension", NULL, test->multiplier, test->divisor, test->algorithm); - st->debug = 1; - - // feed it with the test data - unsigned long c; - for(c = 0; c < test->feed_entries; c++) { - if(debug_flags) fprintf(stderr, "\n\n"); - - if(c) { - fprintf(stderr, " > %s: feeding position %lu, after %llu microseconds, with value " CALCULATED_NUMBER_FORMAT "\n", test->name, c+1, test->feed[c].microseconds, test->feed[c].value); - rrdset_next_usec(st, test->feed[c].microseconds); - } - else { - fprintf(stderr, " > %s: feeding position %lu with value " CALCULATED_NUMBER_FORMAT "\n", test->name, c+1, test->feed[c].value); - } - - rrddim_set(st, "dimension", test->feed[c].value); - rrdset_done(st); - - // align the first entry to second boundary - if(!c) { - fprintf(stderr, " > %s: fixing first collection time to be %llu microseconds to second boundary\n", test->name, test->feed[c].microseconds); - rd->last_collected_time.tv_usec = st->last_collected_time.tv_usec = st->last_updated.tv_usec = test->feed[c].microseconds; - } - } - - // check the result - int errors = 0; - - if(st->counter != test->result_entries) { - fprintf(stderr, " %s stored %lu entries, but we were expecting %lu, ### E R R O R ###\n", test->name, st->counter, test->result_entries); - errors++; - } - - unsigned long max = (st->counter < test->result_entries)?st->counter:test->result_entries; - for(c = 0 ; c < max ; c++) { - calculated_number v = unpack_storage_number(rd->values[c]), n = test->results[c]; - fprintf(stderr, " %s: checking position %lu, expecting value " CALCULATED_NUMBER_FORMAT ", found " CALCULATED_NUMBER_FORMAT ", %s\n", test->name, c+1, n, v, (v == n)?"OK":"### E R R O R ###"); - if(v != n) errors++; - } - - return errors; + fprintf(stderr, "\nRunning test '%s':\n%s\n", test->name, test->description); + + rrd_memory_mode = RRD_MEMORY_MODE_RAM; + rrd_update_every = test->update_every; + + char name[101]; + snprintfz(name, 100, "unittest-%s", test->name); + + // create the chart + RRDSET *st = rrdset_create("netdata", name, name, "netdata", NULL, "Unit Testing", "a value", 1, 1, RRDSET_TYPE_LINE); + RRDDIM *rd = rrddim_add(st, "dim1", NULL, test->multiplier, test->divisor, test->algorithm); + + RRDDIM *rd2 = NULL; + if(test->feed2) + rd2 = rrddim_add(st, "dim2", NULL, test->multiplier, test->divisor, test->algorithm); + + st->debug = 1; + + // feed it with the test data + unsigned long c; + for(c = 0; c < test->feed_entries; c++) { + if(debug_flags) fprintf(stderr, "\n\n"); + + if(c) { + fprintf(stderr, " > %s: feeding position %lu, after %llu microseconds\n", test->name, c+1, test->feed[c].microseconds); + rrdset_next_usec(st, test->feed[c].microseconds); + } + else { + fprintf(stderr, " > %s: feeding position %lu\n", test->name, c+1); + } + + fprintf(stderr, " >> %s with value " COLLECTED_NUMBER_FORMAT "\n", rd->name, test->feed[c].value); + rrddim_set(st, "dim1", test->feed[c].value); + + if(rd2) { + fprintf(stderr, " >> %s with value " COLLECTED_NUMBER_FORMAT "\n", rd2->name, test->feed2[c]); + rrddim_set(st, "dim2", test->feed2[c]); + } + + rrdset_done(st); + + // align the first entry to second boundary + if(!c) { + fprintf(stderr, " > %s: fixing first collection time to be %llu microseconds to second boundary\n", test->name, test->feed[c].microseconds); + rd->last_collected_time.tv_usec = st->last_collected_time.tv_usec = st->last_updated.tv_usec = test->feed[c].microseconds; + } + } + + // check the result + int errors = 0; + + if(st->counter != test->result_entries) { + fprintf(stderr, " %s stored %lu entries, but we were expecting %lu, ### E R R O R ###\n", test->name, st->counter, test->result_entries); + errors++; + } + + unsigned long max = (st->counter < test->result_entries)?st->counter:test->result_entries; + for(c = 0 ; c < max ; c++) { + calculated_number v = unpack_storage_number(rd->values[c]); + calculated_number n = test->results[c]; + int same = (roundl(v * 10000000.0) == roundl(n * 10000000.0))?1:0; + fprintf(stderr, " %s/%s: checking position %lu, expecting value " CALCULATED_NUMBER_FORMAT ", found " CALCULATED_NUMBER_FORMAT ", %s\n", test->name, rd->name, c+1, n, v, (same)?"OK":"### E R R O R ###"); + if(!same) errors++; + + if(rd2) { + v = unpack_storage_number(rd2->values[c]); + n = test->results2[c]; + same = (roundl(v * 10000000.0) == roundl(n * 10000000.0))?1:0; + fprintf(stderr, " %s/%s: checking position %lu, expecting value " CALCULATED_NUMBER_FORMAT ", found " CALCULATED_NUMBER_FORMAT ", %s\n", test->name, rd2->name, c+1, n, v, (same)?"OK":"### E R R O R ###"); + if(!same) errors++; + } + } + + return errors; } int run_all_mockup_tests(void) { - if(run_test(&test1)) - return 1; + if(run_test(&test1)) + return 1; + + if(run_test(&test2)) + return 1; + + if(run_test(&test3)) + return 1; + + if(run_test(&test4)) + return 1; - if(run_test(&test2)) - return 1; + if(run_test(&test5)) + return 1; - if(run_test(&test3)) - return 1; + if(run_test(&test6)) + return 1; - if(run_test(&test4)) - return 1; + if(run_test(&test7)) + return 1; - if(run_test(&test5)) - return 1; + if(run_test(&test8)) + return 1; - if(run_test(&test6)) - return 1; + if(run_test(&test9)) + return 1; - if(run_test(&test7)) - return 1; + if(run_test(&test10)) + return 1; - if(run_test(&test8)) - return 1; + if(run_test(&test11)) + return 1; - if(run_test(&test9)) - return 1; + if(run_test(&test12)) + return 1; - if(run_test(&test10)) - return 1; + if(run_test(&test13)) + return 1; - return 0; + return 0; } int unit_test(long delay, long shift) { - static int repeat = 0; - repeat++; - - char name[101]; - snprintfz(name, 100, "unittest-%d-%ld-%ld", repeat, delay, shift); - - //debug_flags = 0xffffffff; - rrd_memory_mode = RRD_MEMORY_MODE_RAM; - rrd_update_every = 1; - - int do_abs = 1; - int do_inc = 1; - int do_abst = 0; - int do_absi = 0; - - RRDSET *st = rrdset_create("netdata", name, name, "netdata", NULL, "Unit Testing", "a value", 1, 1, RRDSET_TYPE_LINE); - st->debug = 1; - - RRDDIM *rdabs = NULL; - RRDDIM *rdinc = NULL; - RRDDIM *rdabst = NULL; - RRDDIM *rdabsi = NULL; - - if(do_abs) rdabs = rrddim_add(st, "absolute", "absolute", 1, 1, RRDDIM_ABSOLUTE); - if(do_inc) rdinc = rrddim_add(st, "incremental", "incremental", 1, 1, RRDDIM_INCREMENTAL); - if(do_abst) rdabst = rrddim_add(st, "percentage-of-absolute-row", "percentage-of-absolute-row", 1, 1, RRDDIM_PCENT_OVER_ROW_TOTAL); - if(do_absi) rdabsi = rrddim_add(st, "percentage-of-incremental-row", "percentage-of-incremental-row", 1, 1, RRDDIM_PCENT_OVER_DIFF_TOTAL); - - long increment = 1000; - collected_number i = 0; - - unsigned long c, dimensions = 0; - RRDDIM *rd; - for(rd = st->dimensions ; rd ; rd = rd->next) dimensions++; - - for(c = 0; c < 20 ;c++) { - i += increment; - - fprintf(stderr, "\n\nLOOP = %lu, DELAY = %ld, VALUE = " COLLECTED_NUMBER_FORMAT "\n", c, delay, i); - if(c) { - rrdset_next_usec(st, delay); - } - if(do_abs) rrddim_set(st, "absolute", i); - if(do_inc) rrddim_set(st, "incremental", i); - if(do_abst) rrddim_set(st, "percentage-of-absolute-row", i); - if(do_absi) rrddim_set(st, "percentage-of-incremental-row", i); - - if(!c) { - gettimeofday(&st->last_collected_time, NULL); - st->last_collected_time.tv_usec = shift; - } - - // prevent it from deleting the dimensions - for(rd = st->dimensions ; rd ; rd = rd->next) - rd->last_collected_time.tv_sec = st->last_collected_time.tv_sec; - - rrdset_done(st); - } - - unsigned long oincrement = increment; - increment = increment * st->update_every * 1000000 / delay; - fprintf(stderr, "\n\nORIGINAL INCREMENT: %lu, INCREMENT %lu, DELAY %lu, SHIFT %lu\n", oincrement * 10, increment * 10, delay, shift); - - int ret = 0; - storage_number sn; - calculated_number cn, v; - for(c = 0 ; c < st->counter ; c++) { - fprintf(stderr, "\nPOSITION: c = %lu, EXPECTED VALUE %lu\n", c, (oincrement + c * increment + increment * (1000000 - shift) / 1000000 )* 10); - - for(rd = st->dimensions ; rd ; rd = rd->next) { - sn = rd->values[c]; - cn = unpack_storage_number(sn); - fprintf(stderr, "\t %s " CALCULATED_NUMBER_FORMAT " (PACKED AS " STORAGE_NUMBER_FORMAT ") -> ", rd->id, cn, sn); - - if(rd == rdabs) v = - ( oincrement - // + (increment * (1000000 - shift) / 1000000) - + (c + 1) * increment - ); - - else if(rd == rdinc) v = (c?(increment):(increment * (1000000 - shift) / 1000000)); - else if(rd == rdabst) v = oincrement / dimensions / 10; - else if(rd == rdabsi) v = oincrement / dimensions / 10; - else v = 0; - - if(v == cn) fprintf(stderr, "passed.\n"); - else { - fprintf(stderr, "ERROR! (expected " CALCULATED_NUMBER_FORMAT ")\n", v); - ret = 1; - } - } - } - - if(ret) - fprintf(stderr, "\n\nUNIT TEST(%ld, %ld) FAILED\n\n", delay, shift); - - return ret; + static int repeat = 0; + repeat++; + + char name[101]; + snprintfz(name, 100, "unittest-%d-%ld-%ld", repeat, delay, shift); + + //debug_flags = 0xffffffff; + rrd_memory_mode = RRD_MEMORY_MODE_RAM; + rrd_update_every = 1; + + int do_abs = 1; + int do_inc = 1; + int do_abst = 0; + int do_absi = 0; + + RRDSET *st = rrdset_create("netdata", name, name, "netdata", NULL, "Unit Testing", "a value", 1, 1, RRDSET_TYPE_LINE); + st->debug = 1; + + RRDDIM *rdabs = NULL; + RRDDIM *rdinc = NULL; + RRDDIM *rdabst = NULL; + RRDDIM *rdabsi = NULL; + + if(do_abs) rdabs = rrddim_add(st, "absolute", "absolute", 1, 1, RRDDIM_ABSOLUTE); + if(do_inc) rdinc = rrddim_add(st, "incremental", "incremental", 1, 1, RRDDIM_INCREMENTAL); + if(do_abst) rdabst = rrddim_add(st, "percentage-of-absolute-row", "percentage-of-absolute-row", 1, 1, RRDDIM_PCENT_OVER_ROW_TOTAL); + if(do_absi) rdabsi = rrddim_add(st, "percentage-of-incremental-row", "percentage-of-incremental-row", 1, 1, RRDDIM_PCENT_OVER_DIFF_TOTAL); + + long increment = 1000; + collected_number i = 0; + + unsigned long c, dimensions = 0; + RRDDIM *rd; + for(rd = st->dimensions ; rd ; rd = rd->next) dimensions++; + + for(c = 0; c < 20 ;c++) { + i += increment; + + fprintf(stderr, "\n\nLOOP = %lu, DELAY = %ld, VALUE = " COLLECTED_NUMBER_FORMAT "\n", c, delay, i); + if(c) { + rrdset_next_usec(st, delay); + } + if(do_abs) rrddim_set(st, "absolute", i); + if(do_inc) rrddim_set(st, "incremental", i); + if(do_abst) rrddim_set(st, "percentage-of-absolute-row", i); + if(do_absi) rrddim_set(st, "percentage-of-incremental-row", i); + + if(!c) { + gettimeofday(&st->last_collected_time, NULL); + st->last_collected_time.tv_usec = shift; + } + + // prevent it from deleting the dimensions + for(rd = st->dimensions ; rd ; rd = rd->next) + rd->last_collected_time.tv_sec = st->last_collected_time.tv_sec; + + rrdset_done(st); + } + + unsigned long oincrement = increment; + increment = increment * st->update_every * 1000000 / delay; + fprintf(stderr, "\n\nORIGINAL INCREMENT: %lu, INCREMENT %ld, DELAY %ld, SHIFT %ld\n", oincrement * 10, increment * 10, delay, shift); + + int ret = 0; + storage_number sn; + calculated_number cn, v; + for(c = 0 ; c < st->counter ; c++) { + fprintf(stderr, "\nPOSITION: c = %lu, EXPECTED VALUE %lu\n", c, (oincrement + c * increment + increment * (1000000 - shift) / 1000000 )* 10); + + for(rd = st->dimensions ; rd ; rd = rd->next) { + sn = rd->values[c]; + cn = unpack_storage_number(sn); + fprintf(stderr, "\t %s " CALCULATED_NUMBER_FORMAT " (PACKED AS " STORAGE_NUMBER_FORMAT ") -> ", rd->id, cn, sn); + + if(rd == rdabs) v = + ( oincrement + // + (increment * (1000000 - shift) / 1000000) + + (c + 1) * increment + ); + + else if(rd == rdinc) v = (c?(increment):(increment * (1000000 - shift) / 1000000)); + else if(rd == rdabst) v = oincrement / dimensions / 10; + else if(rd == rdabsi) v = oincrement / dimensions / 10; + else v = 0; + + if(v == cn) fprintf(stderr, "passed.\n"); + else { + fprintf(stderr, "ERROR! (expected " CALCULATED_NUMBER_FORMAT ")\n", v); + ret = 1; + } + } + } + + if(ret) + fprintf(stderr, "\n\nUNIT TEST(%ld, %ld) FAILED\n\n", delay, shift); + + return ret; } @@ -1,13 +1,4 @@ -#ifdef HAVE_CONFIG_H -#include <config.h> -#endif -#include <stdlib.h> -#include <string.h> -#include <ctype.h> - #include "common.h" -#include "log.h" -#include "url.h" // ---------------------------------------------------------------------------- // URL encode / decode @@ -15,79 +6,72 @@ /* Converts a hex character to its integer value */ char from_hex(char ch) { - return (char)(isdigit(ch) ? ch - '0' : tolower(ch) - 'a' + 10); + return (char)(isdigit(ch) ? ch - '0' : tolower(ch) - 'a' + 10); } /* Converts an integer value to its hex character*/ char to_hex(char code) { - static char hex[] = "0123456789abcdef"; - return hex[code & 15]; + static char hex[] = "0123456789abcdef"; + return hex[code & 15]; } /* Returns a url-encoded version of str */ /* IMPORTANT: be sure to free() the returned string after use */ char *url_encode(char *str) { - char *buf, *pbuf; - - pbuf = buf = malloc(strlen(str) * 3 + 1); - - if(!buf) - fatal("Cannot allocate memory."); + char *buf, *pbuf; - while (*str) { - if (isalnum(*str) || *str == '-' || *str == '_' || *str == '.' || *str == '~') - *pbuf++ = *str; + pbuf = buf = mallocz(strlen(str) * 3 + 1); - else if (*str == ' ') - *pbuf++ = '+'; + while (*str) { + if (isalnum(*str) || *str == '-' || *str == '_' || *str == '.' || *str == '~') + *pbuf++ = *str; - else - *pbuf++ = '%', *pbuf++ = to_hex(*str >> 4), *pbuf++ = to_hex(*str & 15); + else if (*str == ' ') + *pbuf++ = '+'; - str++; - } - *pbuf = '\0'; + else + *pbuf++ = '%', *pbuf++ = to_hex(*str >> 4), *pbuf++ = to_hex(*str & 15); - // FIX: I think this is prudent. URLs can be as long as 2 KiB or more. - // We allocated 3 times more space to accomodate %NN encoding of - // non ASCII chars. If URL has none of these kind of chars we will - // end up with a big unused buffer. - // - // Try to shrink the buffer... - if (!!(pbuf = (char *)realloc(buf, strlen(buf)+1))) - buf = pbuf; + str++; + } + *pbuf = '\0'; - return buf; + pbuf = strdupz(buf); + freez(buf); + return pbuf; } /* Returns a url-decoded version of str */ /* IMPORTANT: be sure to free() the returned string after use */ char *url_decode(char *str) { - char *pstr = str, - *buf = malloc(strlen(str) + 1), - *pbuf = buf; + size_t size = strlen(str) + 1; - if(!buf) - fatal("Cannot allocate memory."); + char *buf = mallocz(size); + return url_decode_r(buf, str, size); +} - while (*pstr) { - if (*pstr == '%') { - if (pstr[1] && pstr[2]) { - *pbuf++ = from_hex(pstr[1]) << 4 | from_hex(pstr[2]); - pstr += 2; - } - } - else if (*pstr == '+') - *pbuf++ = ' '; +char *url_decode_r(char *to, char *url, size_t size) { + char *s = url, // source + *d = to, // destination + *e = &to[size - 1]; // destination end - else - *pbuf++ = *pstr; + while(*s && d < e) { + if(unlikely(*s == '%')) { + if(likely(s[1] && s[2])) { + *d++ = from_hex(s[1]) << 4 | from_hex(s[2]); + s += 2; + } + } + else if(unlikely(*s == '+')) + *d++ = ' '; - pstr++; - } + else + *d++ = *s; - *pbuf = '\0'; + s++; + } - return buf; -} + *d = '\0'; + return to; +} @@ -19,4 +19,6 @@ extern char *url_encode(char *str); /* IMPORTANT: be sure to free() the returned string after use */ extern char *url_decode(char *str); +extern char *url_decode_r(char *to, char *url, size_t size); + #endif /* NETDATA_URL_H */ diff --git a/src/web_buffer.c b/src/web_buffer.c index a0f153721..01a97ddc9 100644 --- a/src/web_buffer.c +++ b/src/web_buffer.c @@ -1,23 +1,11 @@ -#ifdef HAVE_CONFIG_H -#include <config.h> -#endif -#include <stdlib.h> -#include <string.h> - -#ifdef STORAGE_WITH_MATH -#include <math.h> -#endif - #include "common.h" -#include "web_buffer.h" -#include "log.h" #define BUFFER_OVERFLOW_EOF "EOF" static inline void buffer_overflow_init(BUFFER *b) { - b->buffer[b->size] = '\0'; - strcpy(&b->buffer[b->size + 1], BUFFER_OVERFLOW_EOF); + b->buffer[b->size] = '\0'; + strcpy(&b->buffer[b->size + 1], BUFFER_OVERFLOW_EOF); } #ifdef NETDATA_INTERNAL_CHECKS @@ -28,174 +16,224 @@ static inline void buffer_overflow_init(BUFFER *b) static inline void _buffer_overflow_check(BUFFER *b, const char *file, const char *function, const unsigned long line) { - if(b->len > b->size) { - error("BUFFER: length %ld is above size %ld, at line %lu, at function %s() of file '%s'.", b->len, b->size, line, function, file); - b->len = b->size; - } - - if(b->buffer[b->size] != '\0' || strcmp(&b->buffer[b->size + 1], BUFFER_OVERFLOW_EOF)) { - error("BUFFER: detected overflow at line %lu, at function %s() of file '%s'.", line, function, file); - buffer_overflow_init(b); - } + if(b->len > b->size) { + error("BUFFER: length %zu is above size %zu, at line %lu, at function %s() of file '%s'.", b->len, b->size, line, function, file); + b->len = b->size; + } + + if(b->buffer[b->size] != '\0' || strcmp(&b->buffer[b->size + 1], BUFFER_OVERFLOW_EOF)) { + error("BUFFER: detected overflow at line %lu, at function %s() of file '%s'.", line, function, file); + buffer_overflow_init(b); + } } void buffer_reset(BUFFER *wb) { - buffer_flush(wb); + buffer_flush(wb); - wb->contenttype = CT_TEXT_PLAIN; - wb->options = 0; - wb->date = 0; + wb->contenttype = CT_TEXT_PLAIN; + wb->options = 0; + wb->date = 0; - buffer_overflow_check(wb); + buffer_overflow_check(wb); } const char *buffer_tostring(BUFFER *wb) { - buffer_need_bytes(wb, 1); - wb->buffer[wb->len] = '\0'; + buffer_need_bytes(wb, 1); + wb->buffer[wb->len] = '\0'; - buffer_overflow_check(wb); + buffer_overflow_check(wb); - return(wb->buffer); + return(wb->buffer); } void buffer_char_replace(BUFFER *wb, char from, char to) { - char *s = wb->buffer, *end = &wb->buffer[wb->len]; + char *s = wb->buffer, *end = &wb->buffer[wb->len]; - while(s != end) { - if(*s == from) *s = to; - s++; - } + while(s != end) { + if(*s == from) *s = to; + s++; + } - buffer_overflow_check(wb); + buffer_overflow_check(wb); } +// This trick seems to give an 80% speed increase in 32bit systems +// print_calculated_number_llu_r() will just print the digits up to the +// point the remaining value fits in 32 bits, and then calls +// print_calculated_number_lu_r() to print the rest with 32 bit arithmetic. + +inline char *print_number_lu_r(char *str, unsigned long uvalue) { + char *wstr = str; + + // print each digit + do *wstr++ = (char)('0' + (uvalue % 10)); while(uvalue /= 10); + return wstr; +} + +inline char *print_number_llu_r(char *str, unsigned long long uvalue) { + char *wstr = str; + + // print each digit + do *wstr++ = (char)('0' + (uvalue % 10)); while((uvalue /= 10) && uvalue > (unsigned long long)0xffffffff); + if(uvalue) return print_number_lu_r(wstr, uvalue); + return wstr; +} + +void buffer_print_llu(BUFFER *wb, unsigned long long uvalue) +{ + buffer_need_bytes(wb, 50); + + char *str = &wb->buffer[wb->len]; + char *wstr = str; + +#ifdef ENVIRONMENT32 + if(uvalue > (unsigned long long)0xffffffff) + wstr = print_number_llu_r(wstr, uvalue); + else + wstr = print_number_lu_r(wstr, uvalue); +#else + do *wstr++ = (char)('0' + (uvalue % 10)); while(uvalue /= 10); +#endif + + // terminate it + *wstr = '\0'; + + // reverse it + char *begin = str, *end = wstr - 1, aux; + while (end > begin) aux = *end, *end-- = *begin, *begin++ = aux; + + // return the buffer length + wb->len += wstr - str; +} void buffer_strcat(BUFFER *wb, const char *txt) { - if(unlikely(!txt || !*txt)) return; - - buffer_need_bytes(wb, 1); - - char *s = &wb->buffer[wb->len], *start, *end = &wb->buffer[wb->size]; - long len = wb->len; - - start = s; - while(*txt && s != end) - *s++ = *txt++; - - len += s - start; - - wb->len = len; - buffer_overflow_check(wb); - - if(*txt) { - debug(D_WEB_BUFFER, "strcat(): increasing web_buffer at position %ld, size = %ld\n", wb->len, wb->size); - len = strlen(txt); - buffer_increase(wb, len); - buffer_strcat(wb, txt); - } - else { - // terminate the string - // without increasing the length - buffer_need_bytes(wb, (size_t)1); - wb->buffer[wb->len] = '\0'; - } + if(unlikely(!txt || !*txt)) return; + + buffer_need_bytes(wb, 1); + + char *s = &wb->buffer[wb->len], *start, *end = &wb->buffer[wb->size]; + size_t len = wb->len; + + start = s; + while(*txt && s != end) + *s++ = *txt++; + + len += s - start; + + wb->len = len; + buffer_overflow_check(wb); + + if(*txt) { + debug(D_WEB_BUFFER, "strcat(): increasing web_buffer at position %zu, size = %zu\n", wb->len, wb->size); + len = strlen(txt); + buffer_increase(wb, len); + buffer_strcat(wb, txt); + } + else { + // terminate the string + // without increasing the length + buffer_need_bytes(wb, (size_t)1); + wb->buffer[wb->len] = '\0'; + } } void buffer_snprintf(BUFFER *wb, size_t len, const char *fmt, ...) { - if(unlikely(!fmt || !*fmt)) return; + if(unlikely(!fmt || !*fmt)) return; - buffer_need_bytes(wb, len + 1); + buffer_need_bytes(wb, len + 1); - va_list args; - va_start(args, fmt); - wb->len += vsnprintfz(&wb->buffer[wb->len], len, fmt, args); - va_end(args); + va_list args; + va_start(args, fmt); + wb->len += vsnprintfz(&wb->buffer[wb->len], len, fmt, args); + va_end(args); - buffer_overflow_check(wb); + buffer_overflow_check(wb); - // the buffer is \0 terminated by vsnprintfz + // the buffer is \0 terminated by vsnprintfz } void buffer_vsprintf(BUFFER *wb, const char *fmt, va_list args) { - if(unlikely(!fmt || !*fmt)) return; + if(unlikely(!fmt || !*fmt)) return; - buffer_need_bytes(wb, 2); + buffer_need_bytes(wb, 2); - size_t len = wb->size - wb->len - 1; + size_t len = wb->size - wb->len - 1; - wb->len += vsnprintfz(&wb->buffer[wb->len], len, fmt, args); + wb->len += vsnprintfz(&wb->buffer[wb->len], len, fmt, args); - buffer_overflow_check(wb); + buffer_overflow_check(wb); - // the buffer is \0 terminated by vsnprintfz + // the buffer is \0 terminated by vsnprintfz } void buffer_sprintf(BUFFER *wb, const char *fmt, ...) { - if(unlikely(!fmt || !*fmt)) return; + if(unlikely(!fmt || !*fmt)) return; - buffer_need_bytes(wb, 2); + buffer_need_bytes(wb, 2); - size_t len = wb->size - wb->len - 1; - size_t wrote; + size_t len = wb->size - wb->len - 1; + size_t wrote; - va_list args; - va_start(args, fmt); - wrote = (size_t) vsnprintfz(&wb->buffer[wb->len], len, fmt, args); - va_end(args); + va_list args; + va_start(args, fmt); + wrote = (size_t) vsnprintfz(&wb->buffer[wb->len], len, fmt, args); + va_end(args); - if(unlikely(wrote >= len)) { - // there is bug in vsnprintf() and it returns - // a number higher to len, but it does not - // overflow the buffer. - // our buffer overflow detector will log it - // if it does. - buffer_overflow_check(wb); + if(unlikely(wrote >= len)) { + // truncated + buffer_overflow_check(wb); - debug(D_WEB_BUFFER, "web_buffer_sprintf(): increasing web_buffer at position %ld, size = %ld\n", wb->len, wb->size); - buffer_need_bytes(wb, len + WEB_DATA_LENGTH_INCREASE_STEP); + debug(D_WEB_BUFFER, "web_buffer_sprintf(): increasing web_buffer at position %zu, size = %zu\n", wb->len, wb->size); + buffer_need_bytes(wb, len + WEB_DATA_LENGTH_INCREASE_STEP); - va_start(args, fmt); - buffer_vsprintf(wb, fmt, args); - va_end(args); - } - else - wb->len += wrote; + va_start(args, fmt); + buffer_vsprintf(wb, fmt, args); + va_end(args); + } + else + wb->len += wrote; - // the buffer is \0 terminated by vsnprintf + // the buffer is \0 terminated by vsnprintf } void buffer_rrd_value(BUFFER *wb, calculated_number value) { - buffer_need_bytes(wb, 50); - wb->len += print_calculated_number(&wb->buffer[wb->len], value); + buffer_need_bytes(wb, 50); + + if(isnan(value) || isinf(value)) { + buffer_strcat(wb, "null"); + return; + } + else + wb->len += print_calculated_number(&wb->buffer[wb->len], value); - // terminate it - buffer_need_bytes(wb, 1); - wb->buffer[wb->len] = '\0'; + // terminate it + buffer_need_bytes(wb, 1); + wb->buffer[wb->len] = '\0'; - buffer_overflow_check(wb); + buffer_overflow_check(wb); } // generate a javascript date, the fastest possible way... void buffer_jsdate(BUFFER *wb, int year, int month, int day, int hours, int minutes, int seconds) { // 10 20 30 = 35 - // 01234567890123456789012345678901234 - // Date(2014,04,01,03,28,20) + // 01234567890123456789012345678901234 + // Date(2014,04,01,03,28,20) - buffer_need_bytes(wb, 30); + buffer_need_bytes(wb, 30); - char *b = &wb->buffer[wb->len], *p; + char *b = &wb->buffer[wb->len], *p; unsigned int *q = (unsigned int *)b; #if __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__ @@ -234,23 +272,23 @@ void buffer_jsdate(BUFFER *wb, int year, int month, int day, int hours, int minu *r++ = 0x2900; // ")\0" #endif - wb->len += (size_t)((char *)r - b - 1); + wb->len += (size_t)((char *)r - b - 1); - // terminate it - wb->buffer[wb->len] = '\0'; - buffer_overflow_check(wb); + // terminate it + wb->buffer[wb->len] = '\0'; + buffer_overflow_check(wb); } // generate a date, the fastest possible way... void buffer_date(BUFFER *wb, int year, int month, int day, int hours, int minutes, int seconds) { - // 10 20 30 = 35 - // 01234567890123456789012345678901234 - // 2014-04-01 03:28:20 + // 10 20 30 = 35 + // 01234567890123456789012345678901234 + // 2014-04-01 03:28:20 - buffer_need_bytes(wb, 36); + buffer_need_bytes(wb, 36); - char *b = &wb->buffer[wb->len]; + char *b = &wb->buffer[wb->len]; char *p = b; *p++ = '0' + year / 1000; year %= 1000; @@ -274,68 +312,56 @@ void buffer_date(BUFFER *wb, int year, int month, int day, int hours, int minute *p++ = '0' + seconds % 10; *p = '\0'; - wb->len += (size_t)(p - b); + wb->len += (size_t)(p - b); - // terminate it - wb->buffer[wb->len] = '\0'; - buffer_overflow_check(wb); + // terminate it + wb->buffer[wb->len] = '\0'; + buffer_overflow_check(wb); } -BUFFER *buffer_create(long size) +BUFFER *buffer_create(size_t size) { - BUFFER *b; - - debug(D_WEB_BUFFER, "Creating new web buffer of size %d.", size); - - b = calloc(1, sizeof(BUFFER)); - if(!b) { - error("Cannot allocate a web_buffer."); - return NULL; - } - - b->buffer = malloc(size + sizeof(BUFFER_OVERFLOW_EOF) + 2); - if(!b->buffer) { - error("Cannot allocate a buffer of size %u.", size + sizeof(BUFFER_OVERFLOW_EOF) + 2); - free(b); - return NULL; - } - b->buffer[0] = '\0'; - b->size = size; - b->contenttype = CT_TEXT_PLAIN; - buffer_overflow_init(b); - buffer_overflow_check(b); - - return(b); + BUFFER *b; + + debug(D_WEB_BUFFER, "Creating new web buffer of size %zu.", size); + + b = callocz(1, sizeof(BUFFER)); + b->buffer = mallocz(size + sizeof(BUFFER_OVERFLOW_EOF) + 2); + b->buffer[0] = '\0'; + b->size = size; + b->contenttype = CT_TEXT_PLAIN; + buffer_overflow_init(b); + buffer_overflow_check(b); + + return(b); } void buffer_free(BUFFER *b) { - buffer_overflow_check(b); + buffer_overflow_check(b); - debug(D_WEB_BUFFER, "Freeing web buffer of size %d.", b->size); + debug(D_WEB_BUFFER, "Freeing web buffer of size %zu.", b->size); - if(b->buffer) free(b->buffer); - free(b); + freez(b->buffer); + freez(b); } void buffer_increase(BUFFER *b, size_t free_size_required) { - buffer_overflow_check(b); - - size_t left = b->size - b->len; + buffer_overflow_check(b); - if(left >= free_size_required) return; + size_t left = b->size - b->len; - size_t increase = free_size_required - left; - if(increase < WEB_DATA_LENGTH_INCREASE_STEP) increase = WEB_DATA_LENGTH_INCREASE_STEP; + if(left >= free_size_required) return; - debug(D_WEB_BUFFER, "Increasing data buffer from size %d to %d.", b->size, b->size + increase); + size_t increase = free_size_required - left; + if(increase < WEB_DATA_LENGTH_INCREASE_STEP) increase = WEB_DATA_LENGTH_INCREASE_STEP; - b->buffer = realloc(b->buffer, b->size + increase + sizeof(BUFFER_OVERFLOW_EOF) + 2); - if(!b->buffer) fatal("Failed to increase data buffer from size %d to %d.", b->size + sizeof(BUFFER_OVERFLOW_EOF) + 2, b->size + increase + sizeof(BUFFER_OVERFLOW_EOF) + 2); + debug(D_WEB_BUFFER, "Increasing data buffer from size %zu to %zu.", b->size, b->size + increase); - b->size += increase; + b->buffer = reallocz(b->buffer, b->size + increase + sizeof(BUFFER_OVERFLOW_EOF) + 2); + b->size += increase; - buffer_overflow_init(b); - buffer_overflow_check(b); + buffer_overflow_init(b); + buffer_overflow_check(b); } diff --git a/src/web_buffer.h b/src/web_buffer.h index 73533f499..c4cd05632 100644 --- a/src/web_buffer.h +++ b/src/web_buffer.h @@ -1,47 +1,43 @@ -#include <stdarg.h> -#include <time.h> -#include "storage_number.h" - #ifndef NETDATA_WEB_BUFFER_H #define NETDATA_WEB_BUFFER_H 1 -#define WEB_DATA_LENGTH_INCREASE_STEP 16384 +#define WEB_DATA_LENGTH_INCREASE_STEP 1024 typedef struct web_buffer { - size_t size; // allocation size of buffer - size_t len; // current data length in buffer - char *buffer; // the buffer - uint8_t contenttype; - uint8_t options; - time_t date; // the date this content has been generated + size_t size; // allocation size of buffer + size_t len; // current data length in buffer + char *buffer; // the buffer + uint8_t contenttype; + uint8_t options; + time_t date; // the date this content has been generated } BUFFER; // options -#define WB_CONTENT_CACHEABLE 1 -#define WB_CONTENT_NO_CACHEABLE 2 +#define WB_CONTENT_CACHEABLE 1 +#define WB_CONTENT_NO_CACHEABLE 2 // content-types -#define CT_APPLICATION_JSON 1 -#define CT_TEXT_PLAIN 2 -#define CT_TEXT_HTML 3 -#define CT_APPLICATION_X_JAVASCRIPT 4 -#define CT_TEXT_CSS 5 -#define CT_TEXT_XML 6 -#define CT_APPLICATION_XML 7 -#define CT_TEXT_XSL 8 -#define CT_APPLICATION_OCTET_STREAM 9 -#define CT_APPLICATION_X_FONT_TRUETYPE 10 -#define CT_APPLICATION_X_FONT_OPENTYPE 11 -#define CT_APPLICATION_FONT_WOFF 12 -#define CT_APPLICATION_FONT_WOFF2 13 -#define CT_APPLICATION_VND_MS_FONTOBJ 14 -#define CT_IMAGE_SVG_XML 15 -#define CT_IMAGE_PNG 16 -#define CT_IMAGE_JPG 17 -#define CT_IMAGE_GIF 18 -#define CT_IMAGE_XICON 19 -#define CT_IMAGE_ICNS 20 -#define CT_IMAGE_BMP 21 +#define CT_APPLICATION_JSON 1 +#define CT_TEXT_PLAIN 2 +#define CT_TEXT_HTML 3 +#define CT_APPLICATION_X_JAVASCRIPT 4 +#define CT_TEXT_CSS 5 +#define CT_TEXT_XML 6 +#define CT_APPLICATION_XML 7 +#define CT_TEXT_XSL 8 +#define CT_APPLICATION_OCTET_STREAM 9 +#define CT_APPLICATION_X_FONT_TRUETYPE 10 +#define CT_APPLICATION_X_FONT_OPENTYPE 11 +#define CT_APPLICATION_FONT_WOFF 12 +#define CT_APPLICATION_FONT_WOFF2 13 +#define CT_APPLICATION_VND_MS_FONTOBJ 14 +#define CT_IMAGE_SVG_XML 15 +#define CT_IMAGE_PNG 16 +#define CT_IMAGE_JPG 17 +#define CT_IMAGE_GIF 18 +#define CT_IMAGE_XICON 19 +#define CT_IMAGE_ICNS 20 +#define CT_IMAGE_BMP 21 #define buffer_strlen(wb) ((wb)->len) extern const char *buffer_tostring(BUFFER *wb); @@ -57,14 +53,19 @@ extern void buffer_rrd_value(BUFFER *wb, calculated_number value); extern void buffer_date(BUFFER *wb, int year, int month, int day, int hours, int minutes, int seconds); extern void buffer_jsdate(BUFFER *wb, int year, int month, int day, int hours, int minutes, int seconds); -extern BUFFER *buffer_create(long size); +extern BUFFER *buffer_create(size_t size); extern void buffer_free(BUFFER *b); extern void buffer_increase(BUFFER *b, size_t free_size_required); -extern void buffer_snprintf(BUFFER *wb, size_t len, const char *fmt, ...); +extern void buffer_snprintf(BUFFER *wb, size_t len, const char *fmt, ...) __attribute__ (( format (printf, 3, 4))); extern void buffer_vsprintf(BUFFER *wb, const char *fmt, va_list args); -extern void buffer_sprintf(BUFFER *wb, const char *fmt, ...); +extern void buffer_sprintf(BUFFER *wb, const char *fmt, ...) __attribute__ (( format (printf, 2, 3))); extern void buffer_char_replace(BUFFER *wb, char from, char to); +extern char *print_number_lu_r(char *str, unsigned long uvalue); +extern char *print_number_llu_r(char *str, unsigned long long uvalue); + +extern void buffer_print_llu(BUFFER *wb, unsigned long long uvalue); + #endif /* NETDATA_WEB_BUFFER_H */ diff --git a/src/web_buffer_svg.c b/src/web_buffer_svg.c new file mode 100644 index 000000000..2f7627cc5 --- /dev/null +++ b/src/web_buffer_svg.c @@ -0,0 +1,616 @@ +#include "common.h" + +#define BADGE_HORIZONTAL_PADDING 4 +#define VERDANA_KERNING 0.2 +#define VERDANA_PADDING 1.0 + +/* + * verdana11_widths[] has been generated with this method: + * https://github.com/badges/shields/blob/master/measure-text.js +*/ + +double verdana11_widths[256] = { + [0] = 0.0, + [1] = 0.0, + [2] = 0.0, + [3] = 0.0, + [4] = 0.0, + [5] = 0.0, + [6] = 0.0, + [7] = 0.0, + [8] = 0.0, + [9] = 0.0, + [10] = 0.0, + [11] = 0.0, + [12] = 0.0, + [13] = 0.0, + [14] = 0.0, + [15] = 0.0, + [16] = 0.0, + [17] = 0.0, + [18] = 0.0, + [19] = 0.0, + [20] = 0.0, + [21] = 0.0, + [22] = 0.0, + [23] = 0.0, + [24] = 0.0, + [25] = 0.0, + [26] = 0.0, + [27] = 0.0, + [28] = 0.0, + [29] = 0.0, + [30] = 0.0, + [31] = 0.0, + [32] = 3.8671874999999996, // + [33] = 4.3291015625, // ! + [34] = 5.048828125, // " + [35] = 9.001953125, // # + [36] = 6.9931640625, // $ + [37] = 11.837890625, // % + [38] = 7.992187499999999, // & + [39] = 2.9541015625, // ' + [40] = 4.9951171875, // ( + [41] = 4.9951171875, // ) + [42] = 6.9931640625, // * + [43] = 9.001953125, // + + [44] = 4.00146484375, // , + [45] = 4.9951171875, // - + [46] = 4.00146484375, // . + [47] = 4.9951171875, // / + [48] = 6.9931640625, // 0 + [49] = 6.9931640625, // 1 + [50] = 6.9931640625, // 2 + [51] = 6.9931640625, // 3 + [52] = 6.9931640625, // 4 + [53] = 6.9931640625, // 5 + [54] = 6.9931640625, // 6 + [55] = 6.9931640625, // 7 + [56] = 6.9931640625, // 8 + [57] = 6.9931640625, // 9 + [58] = 4.9951171875, // : + [59] = 4.9951171875, // ; + [60] = 9.001953125, // < + [61] = 9.001953125, // = + [62] = 9.001953125, // > + [63] = 5.99951171875, // ? + [64] = 11.0, // @ + [65] = 7.51953125, // A + [66] = 7.541015625, // B + [67] = 7.680664062499999, // C + [68] = 8.4755859375, // D + [69] = 6.95556640625, // E + [70] = 6.32177734375, // F + [71] = 8.529296875, // G + [72] = 8.26611328125, // H + [73] = 4.6298828125, // I + [74] = 5.00048828125, // J + [75] = 7.62158203125, // K + [76] = 6.123046875, // L + [77] = 9.2705078125, // M + [78] = 8.228515625, // N + [79] = 8.658203125, // O + [80] = 6.63330078125, // P + [81] = 8.658203125, // Q + [82] = 7.6484375, // R + [83] = 7.51953125, // S + [84] = 6.7783203125, // T + [85] = 8.05126953125, // U + [86] = 7.51953125, // V + [87] = 10.87646484375, // W + [88] = 7.53564453125, // X + [89] = 6.767578125, // Y + [90] = 7.53564453125, // Z + [91] = 4.9951171875, // [ + [92] = 4.9951171875, // backslash + [93] = 4.9951171875, // ] + [94] = 9.001953125, // ^ + [95] = 6.9931640625, // _ + [96] = 6.9931640625, // ` + [97] = 6.6064453125, // a + [98] = 6.853515625, // b + [99] = 5.73095703125, // c + [100] = 6.853515625, // d + [101] = 6.552734375, // e + [102] = 3.8671874999999996, // f + [103] = 6.853515625, // g + [104] = 6.9609375, // h + [105] = 3.0185546875, // i + [106] = 3.78662109375, // j + [107] = 6.509765625, // k + [108] = 3.0185546875, // l + [109] = 10.69921875, // m + [110] = 6.9609375, // n + [111] = 6.67626953125, // o + [112] = 6.853515625, // p + [113] = 6.853515625, // q + [114] = 4.6943359375, // r + [115] = 5.73095703125, // s + [116] = 4.33447265625, // t + [117] = 6.9609375, // u + [118] = 6.509765625, // v + [119] = 9.001953125, // w + [120] = 6.509765625, // x + [121] = 6.509765625, // y + [122] = 5.779296875, // z + [123] = 6.982421875, // { + [124] = 4.9951171875, // | + [125] = 6.982421875, // } + [126] = 9.001953125, // ~ + [127] = 0.0, + [128] = 0.0, + [129] = 0.0, + [130] = 0.0, + [131] = 0.0, + [132] = 0.0, + [133] = 0.0, + [134] = 0.0, + [135] = 0.0, + [136] = 0.0, + [137] = 0.0, + [138] = 0.0, + [139] = 0.0, + [140] = 0.0, + [141] = 0.0, + [142] = 0.0, + [143] = 0.0, + [144] = 0.0, + [145] = 0.0, + [146] = 0.0, + [147] = 0.0, + [148] = 0.0, + [149] = 0.0, + [150] = 0.0, + [151] = 0.0, + [152] = 0.0, + [153] = 0.0, + [154] = 0.0, + [155] = 0.0, + [156] = 0.0, + [157] = 0.0, + [158] = 0.0, + [159] = 0.0, + [160] = 0.0, + [161] = 0.0, + [162] = 0.0, + [163] = 0.0, + [164] = 0.0, + [165] = 0.0, + [166] = 0.0, + [167] = 0.0, + [168] = 0.0, + [169] = 0.0, + [170] = 0.0, + [171] = 0.0, + [172] = 0.0, + [173] = 0.0, + [174] = 0.0, + [175] = 0.0, + [176] = 0.0, + [177] = 0.0, + [178] = 0.0, + [179] = 0.0, + [180] = 0.0, + [181] = 0.0, + [182] = 0.0, + [183] = 0.0, + [184] = 0.0, + [185] = 0.0, + [186] = 0.0, + [187] = 0.0, + [188] = 0.0, + [189] = 0.0, + [190] = 0.0, + [191] = 0.0, + [192] = 0.0, + [193] = 0.0, + [194] = 0.0, + [195] = 0.0, + [196] = 0.0, + [197] = 0.0, + [198] = 0.0, + [199] = 0.0, + [200] = 0.0, + [201] = 0.0, + [202] = 0.0, + [203] = 0.0, + [204] = 0.0, + [205] = 0.0, + [206] = 0.0, + [207] = 0.0, + [208] = 0.0, + [209] = 0.0, + [210] = 0.0, + [211] = 0.0, + [212] = 0.0, + [213] = 0.0, + [214] = 0.0, + [215] = 0.0, + [216] = 0.0, + [217] = 0.0, + [218] = 0.0, + [219] = 0.0, + [220] = 0.0, + [221] = 0.0, + [222] = 0.0, + [223] = 0.0, + [224] = 0.0, + [225] = 0.0, + [226] = 0.0, + [227] = 0.0, + [228] = 0.0, + [229] = 0.0, + [230] = 0.0, + [231] = 0.0, + [232] = 0.0, + [233] = 0.0, + [234] = 0.0, + [235] = 0.0, + [236] = 0.0, + [237] = 0.0, + [238] = 0.0, + [239] = 0.0, + [240] = 0.0, + [241] = 0.0, + [242] = 0.0, + [243] = 0.0, + [244] = 0.0, + [245] = 0.0, + [246] = 0.0, + [247] = 0.0, + [248] = 0.0, + [249] = 0.0, + [250] = 0.0, + [251] = 0.0, + [252] = 0.0, + [253] = 0.0, + [254] = 0.0, + [255] = 0.0 +}; + +// find the width of the string using the verdana 11points font +// re-write the string in place, skiping zero-length characters +static inline int verdana11_width(char *s) { + double w = 0.0; + char *d = s; + + while(*s) { + double t = verdana11_widths[(unsigned char)*s]; + if(t == 0.0) + s++; + else { + w += t + VERDANA_KERNING; + if(d != s) + *d++ = *s++; + else + d = ++s; + } + } + + *d = '\0'; + w -= VERDANA_KERNING; + w += VERDANA_PADDING; + return ceil(w); +} + +static inline size_t escape_xmlz(char *dst, const char *src, size_t len) { + size_t i = len; + + // required escapes from + // https://github.com/badges/shields/blob/master/badge.js + while(*src && i) { + switch(*src) { + case '\\': + *dst++ = '/'; + src++; + i--; + break; + + case '&': + if(i > 5) { + strcpy(dst, "&"); + i -= 5; + dst += 5; + src++; + } + else goto cleanup; + break; + + case '<': + if(i > 4) { + strcpy(dst, "<"); + i -= 4; + dst += 4; + src++; + } + else goto cleanup; + break; + + case '>': + if(i > 4) { + strcpy(dst, ">"); + i -= 4; + dst += 4; + src++; + } + else goto cleanup; + break; + + case '"': + if(i > 6) { + strcpy(dst, """); + i -= 6; + dst += 6; + src++; + } + else goto cleanup; + break; + + case '\'': + if(i > 6) { + strcpy(dst, "'"); + i -= 6; + dst += 6; + src++; + } + else goto cleanup; + break; + + default: + i--; + *dst++ = *src++; + break; + } + } + +cleanup: + *dst = '\0'; + return len - i; +} + +static inline const char *fix_units(const char *units) { + if(!units || !*units || !strcmp(units, "empty") || !strcmp(units, "null")) return ""; + if(!strcmp(units, "percentage") || !strcmp(units, "percent") || !strcmp(units, "pcent")) return "%"; + return units; +} + +static inline const char *color_map(const char *color) { + // colors from: + // https://github.com/badges/shields/blob/master/colorscheme.json + if(!strcmp(color, "brightgreen")) return "#4c1"; + else if(!strcmp(color, "green")) return "#97CA00"; + else if(!strcmp(color, "yellow")) return "#dfb317"; + else if(!strcmp(color, "yellowgreen")) return "#a4a61d"; + else if(!strcmp(color, "orange")) return "#fe7d37"; + else if(!strcmp(color, "red")) return "#e05d44"; + else if(!strcmp(color, "blue")) return "#007ec6"; + else if(!strcmp(color, "grey")) return "#555"; + else if(!strcmp(color, "gray")) return "#555"; + else if(!strcmp(color, "lightgrey")) return "#9f9f9f"; + else if(!strcmp(color, "lightgray")) return "#9f9f9f"; + return color; +} + +static inline void calc_colorz(const char *color, char *final, size_t len, calculated_number value, int value_is_null) { + char color_buffer[256 + 1] = ""; + char value_buffer[256 + 1] = ""; + char comparison = '>'; + + // example input: + // color<max|color>min|color:null... + + const char *c = color; + while(*c) { + char *dc = color_buffer, *dv = NULL; + size_t ci = 0, vi = 0; + + const char *t = c; + + while(*t && *t != '|') { + switch(*t) { + case ':': + comparison = '='; + dv = value_buffer; + break; + + case '}': + case ')': + case '>': + if(t[1] == '=') { + comparison = ')'; + t++; + } + else + comparison = '>'; + dv = value_buffer; + break; + + case '{': + case '(': + case '<': + if(t[1] == '=') { + comparison = '('; + t++; + } + else + comparison = '<'; + dv = value_buffer; + break; + + default: + if(dv) { + if(vi < 256) { + vi++; + *dv++ = *t; + } + } + else { + if(ci < 256) { + ci++; + *dc++ = *t; + } + } + break; + } + + t++; + } + + // prepare for next iteration + if(*t == '|') t++; + c = t; + + // do the math + *dc = '\0'; + if(dv) { + *dv = '\0'; + + if(value_is_null) { + if(!*value_buffer || !strcmp(value_buffer, "null")) + break; + } + else { + calculated_number v = strtold(value_buffer, NULL); + + if(comparison == '<' && value < v) break; + else if(comparison == '(' && value <= v) break; + else if(comparison == '>' && value > v) break; + else if(comparison == ')' && value >= v) break; + else if(comparison == '=' && value == v) break; + } + } + else + break; + } + + const char *b; + if(color_buffer[0]) + b = color_buffer; + else + b = color; + + strncpyz(final, b, len); +} + +// value + units +#define VALUE_STRING_SIZE 100 + +// label +#define LABEL_STRING_SIZE 200 + +// colors +#define COLOR_STRING_SIZE 100 + +void buffer_svg(BUFFER *wb, const char *label, calculated_number value, const char *units, const char *label_color, const char *value_color, int value_is_null, int precision) { + char label_buffer[LABEL_STRING_SIZE + 1] + , value_color_buffer[COLOR_STRING_SIZE + 1] + , value_string[VALUE_STRING_SIZE + 1] + , label_escaped[LABEL_STRING_SIZE + 1] + , value_escaped[VALUE_STRING_SIZE + 1] + , label_color_escaped[COLOR_STRING_SIZE + 1] + , value_color_escaped[COLOR_STRING_SIZE + 1]; + + int label_width, value_width, total_width; + + if(unlikely(!label_color || !*label_color)) + label_color = "#555"; + + if(unlikely(!value_color || !*value_color)) + value_color = (value_is_null)?"#999":"#4c1"; + + units = fix_units(units); + calc_colorz(value_color, value_color_buffer, COLOR_STRING_SIZE, value, value_is_null); + + char *separator = ""; + if(unlikely(isalnum(*units))) + separator = " "; + + if(unlikely(value_is_null)) + strcpy(value_string, "-"); + + else if(precision < 0) { + int len, l, lstop = 0; + + calculated_number abs = value; + if(isless(value, 0)) { + lstop = 1; + abs = -value; + } + + if(isgreaterequal(abs, 1000)) len = snprintfz(value_string, VALUE_STRING_SIZE, "%0.0Lf", (long double)value); + else if(isgreaterequal(abs, 100)) len = snprintfz(value_string, VALUE_STRING_SIZE, "%0.1Lf", (long double)value); + else if(isgreaterequal(abs, 1)) len = snprintfz(value_string, VALUE_STRING_SIZE, "%0.2Lf", (long double)value); + else if(isgreaterequal(abs, 0.1)) len = snprintfz(value_string, VALUE_STRING_SIZE, "%0.3Lf", (long double)value); + else len = snprintfz(value_string, VALUE_STRING_SIZE, "%0.4Lf", (long double)value); + + // remove trailing zeros + for(l = len - 1; l > lstop ; l--) { + if(likely(value_string[l] == '0')) { + value_string[l] = '\0'; + len--; + } + + else if(unlikely(value_string[l] == '.')) { + value_string[l] = '\0'; + len--; + break; + } + + else + break; + } + + if(len >= 0) + snprintfz(&value_string[len], VALUE_STRING_SIZE - len, "%s%s", separator, units); + } + else { + if(precision > 50) precision = 50; + snprintfz(value_string, VALUE_STRING_SIZE, "%0.*Lf%s%s", precision, (long double)value, separator, units); + } + + // we need to copy the label, since verdana11_width may write to it + strncpyz(label_buffer, label, LABEL_STRING_SIZE); + + label_width = verdana11_width(label_buffer) + (BADGE_HORIZONTAL_PADDING * 2); + value_width = verdana11_width(value_string) + (BADGE_HORIZONTAL_PADDING * 2); + total_width = label_width + value_width; + + escape_xmlz(label_escaped, label_buffer, LABEL_STRING_SIZE); + escape_xmlz(value_escaped, value_string, VALUE_STRING_SIZE); + escape_xmlz(label_color_escaped, color_map(label_color), COLOR_STRING_SIZE); + escape_xmlz(value_color_escaped, color_map(value_color_buffer), COLOR_STRING_SIZE); + + wb->contenttype = CT_IMAGE_SVG_XML; + + // svg template from: + // https://raw.githubusercontent.com/badges/shields/master/templates/flat-template.svg + buffer_sprintf(wb, + "<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"%d\" height=\"20\">" + "<linearGradient id=\"smooth\" x2=\"0\" y2=\"100%%\">" + "<stop offset=\"0\" stop-color=\"#bbb\" stop-opacity=\".1\"/>" + "<stop offset=\"1\" stop-opacity=\".1\"/>" + "</linearGradient>" + "<mask id=\"round\">" + "<rect width=\"%d\" height=\"20\" rx=\"3\" fill=\"#fff\"/>" + "</mask>" + "<g mask=\"url(#round)\">" + "<rect width=\"%d\" height=\"20\" fill=\"%s\"/>" + "<rect x=\"%d\" width=\"%d\" height=\"20\" fill=\"%s\"/>" + "<rect width=\"%d\" height=\"20\" fill=\"url(#smooth)\"/>" + "</g>" + "<g fill=\"#fff\" text-anchor=\"middle\" font-family=\"DejaVu Sans,Verdana,Geneva,sans-serif\" font-size=\"11\">" + "<text x=\"%d\" y=\"15\" fill=\"#010101\" fill-opacity=\".3\">%s</text>" + "<text x=\"%d\" y=\"14\">%s</text>" + "<text x=\"%d\" y=\"15\" fill=\"#010101\" fill-opacity=\".3\">%s</text>" + "<text x=\"%d\" y=\"14\">%s</text>" + "</g>" + "</svg>", + total_width, total_width, + label_width, label_color_escaped, + label_width, value_width, value_color_escaped, + total_width, + label_width / 2, label_escaped, + label_width / 2, label_escaped, + label_width + value_width / 2 -1, value_escaped, + label_width + value_width / 2 -1, value_escaped); +} diff --git a/src/web_buffer_svg.h b/src/web_buffer_svg.h new file mode 100644 index 000000000..1281847eb --- /dev/null +++ b/src/web_buffer_svg.h @@ -0,0 +1,6 @@ +#ifndef NETDATA_WEB_BUFFER_SVG_H +#define NETDATA_WEB_BUFFER_SVG_H 1 + +extern void buffer_svg(BUFFER *wb, const char *label, calculated_number value, const char *units, const char *label_color, const char *value_color, int value_is_null, int precision); + +#endif /* NETDATA_WEB_BUFFER_SVG_H */ diff --git a/src/web_client.c b/src/web_client.c index 601dda083..4036d4c81 100644 --- a/src/web_client.c +++ b/src/web_client.c @@ -1,2032 +1,2537 @@ -#ifdef HAVE_CONFIG_H -#include <config.h> -#endif -#include <unistd.h> -#include <stdlib.h> -#include <sys/types.h> -#include <sys/socket.h> -#include <netinet/in.h> -#include <arpa/inet.h> -#include <errno.h> -#include <pthread.h> -#include <sys/stat.h> -#include <fcntl.h> -#include <netinet/tcp.h> -#include <malloc.h> -#include <pwd.h> -#include <grp.h> -#include <ctype.h> - #include "common.h" -#include "log.h" -#include "appconfig.h" -#include "url.h" -#include "web_buffer.h" -#include "web_server.h" -#include "global_statistics.h" -#include "rrd.h" -#include "rrd2json.h" -#include "registry.h" - -#include "web_client.h" -#include "../config.h" #define INITIAL_WEB_DATA_LENGTH 16384 #define WEB_REQUEST_LENGTH 16384 #define TOO_BIG_REQUEST 16384 int web_client_timeout = DEFAULT_DISCONNECT_IDLE_WEB_CLIENTS_AFTER_SECONDS; -int web_enable_gzip = 1; +int web_donotrack_comply = 0; -extern int netdata_exit; +#ifdef NETDATA_WITH_ZLIB +int web_enable_gzip = 1, web_gzip_level = 3, web_gzip_strategy = Z_DEFAULT_STRATEGY; +#endif /* NETDATA_WITH_ZLIB */ struct web_client *web_clients = NULL; unsigned long long web_clients_count = 0; +inline int web_client_crock_socket(struct web_client *w) { +#ifdef TCP_CORK + if(likely(!w->tcp_cork && w->ofd != -1)) { + w->tcp_cork = 1; + if(unlikely(setsockopt(w->ofd, IPPROTO_TCP, TCP_CORK, (char *) &w->tcp_cork, sizeof(int)) != 0)) { + error("%llu: failed to enable TCP_CORK on socket.", w->id); + w->tcp_cork = 0; + return -1; + } + } +#endif /* TCP_CORK */ + + return 0; +} + +inline int web_client_uncrock_socket(struct web_client *w) { +#ifdef TCP_CORK + if(likely(w->tcp_cork && w->ofd != -1)) { + w->tcp_cork = 0; + if(unlikely(setsockopt(w->ofd, IPPROTO_TCP, TCP_CORK, (char *) &w->tcp_cork, sizeof(int)) != 0)) { + error("%llu: failed to disable TCP_CORK on socket.", w->id); + w->tcp_cork = 1; + return -1; + } + } +#endif /* TCP_CORK */ + + return 0; +} + struct web_client *web_client_create(int listener) { - struct web_client *w; - - w = calloc(1, sizeof(struct web_client)); - if(!w) { - error("Cannot allocate new web_client memory."); - return NULL; - } - - w->id = ++web_clients_count; - w->mode = WEB_CLIENT_MODE_NORMAL; - - { - struct sockaddr *sadr; - socklen_t addrlen; - - sadr = (struct sockaddr*) &w->clientaddr; - addrlen = sizeof(w->clientaddr); - - w->ifd = accept(listener, sadr, &addrlen); - if (w->ifd == -1) { - error("%llu: Cannot accept new incoming connection.", w->id); - free(w); - return NULL; - } - w->ofd = w->ifd; - - if(getnameinfo(sadr, addrlen, w->client_ip, NI_MAXHOST, w->client_port, NI_MAXSERV, NI_NUMERICHOST | NI_NUMERICSERV) != 0) { - error("Cannot getnameinfo() on received client connection."); - strncpyz(w->client_ip, "UNKNOWN", NI_MAXHOST); - strncpyz(w->client_port, "UNKNOWN", NI_MAXSERV); - } - w->client_ip[NI_MAXHOST] = '\0'; - w->client_port[NI_MAXSERV] = '\0'; - - switch(sadr->sa_family) { - - case AF_INET: - debug(D_WEB_CLIENT_ACCESS, "%llu: New IPv4 web client from %s port %s on socket %d.", w->id, w->client_ip, w->client_port, w->ifd); - break; - - case AF_INET6: - if(strncmp(w->client_ip, "::ffff:", 7) == 0) { - strcpy(w->client_ip, &w->client_ip[7]); - debug(D_WEB_CLIENT_ACCESS, "%llu: New IPv4 web client from %s port %s on socket %d.", w->id, w->client_ip, w->client_port, w->ifd); - } - debug(D_WEB_CLIENT_ACCESS, "%llu: New IPv6 web client from %s port %s on socket %d.", w->id, w->client_ip, w->client_port, w->ifd); - break; - - default: - debug(D_WEB_CLIENT_ACCESS, "%llu: New UNKNOWN web client from %s port %s on socket %d.", w->id, w->client_ip, w->client_port, w->ifd); - break; - } - - int flag = 1; - if(setsockopt(w->ifd, SOL_SOCKET, SO_KEEPALIVE, (char *) &flag, sizeof(int)) != 0) error("%llu: Cannot set SO_KEEPALIVE on socket.", w->id); - } - - w->response.data = buffer_create(INITIAL_WEB_DATA_LENGTH); - if(unlikely(!w->response.data)) { - // no need for error log - web_buffer_create already logged the error - close(w->ifd); - free(w); - return NULL; - } - - w->response.header = buffer_create(HTTP_RESPONSE_HEADER_SIZE); - if(unlikely(!w->response.header)) { - // no need for error log - web_buffer_create already logged the error - buffer_free(w->response.data); - close(w->ifd); - free(w); - return NULL; - } - - w->response.header_output = buffer_create(HTTP_RESPONSE_HEADER_SIZE); - if(unlikely(!w->response.header_output)) { - // no need for error log - web_buffer_create already logged the error - buffer_free(w->response.header); - buffer_free(w->response.data); - close(w->ifd); - free(w); - return NULL; - } - - w->origin[0] = '*'; - w->wait_receive = 1; - - if(web_clients) web_clients->prev = w; - w->next = web_clients; - web_clients = w; - - global_statistics.connected_clients++; - - return(w); + struct web_client *w; + + w = callocz(1, sizeof(struct web_client)); + w->id = ++web_clients_count; + w->mode = WEB_CLIENT_MODE_NORMAL; + + { + struct sockaddr *sadr; + socklen_t addrlen; + + sadr = (struct sockaddr*) &w->clientaddr; + addrlen = sizeof(w->clientaddr); + + w->ifd = accept4(listener, sadr, &addrlen, SOCK_NONBLOCK); + if (w->ifd == -1) { + error("%llu: Cannot accept new incoming connection.", w->id); + freez(w); + return NULL; + } + w->ofd = w->ifd; + + if(getnameinfo(sadr, addrlen, w->client_ip, NI_MAXHOST, w->client_port, NI_MAXSERV, NI_NUMERICHOST | NI_NUMERICSERV) != 0) { + error("Cannot getnameinfo() on received client connection."); + strncpyz(w->client_ip, "UNKNOWN", NI_MAXHOST); + strncpyz(w->client_port, "UNKNOWN", NI_MAXSERV); + } + w->client_ip[NI_MAXHOST] = '\0'; + w->client_port[NI_MAXSERV] = '\0'; + + switch(sadr->sa_family) { + case AF_INET: + debug(D_WEB_CLIENT_ACCESS, "%llu: New IPv4 web client from %s port %s on socket %d.", w->id, w->client_ip, w->client_port, w->ifd); + break; + + case AF_INET6: + if(strncmp(w->client_ip, "::ffff:", 7) == 0) { + memmove(w->client_ip, &w->client_ip[7], strlen(&w->client_ip[7]) + 1); + debug(D_WEB_CLIENT_ACCESS, "%llu: New IPv4 web client from %s port %s on socket %d.", w->id, w->client_ip, w->client_port, w->ifd); + } + else + debug(D_WEB_CLIENT_ACCESS, "%llu: New IPv6 web client from %s port %s on socket %d.", w->id, w->client_ip, w->client_port, w->ifd); + break; + + default: + debug(D_WEB_CLIENT_ACCESS, "%llu: New UNKNOWN web client from %s port %s on socket %d.", w->id, w->client_ip, w->client_port, w->ifd); + break; + } + + int flag = 1; + if(setsockopt(w->ofd, IPPROTO_TCP, TCP_NODELAY, (char *) &flag, sizeof(int)) != 0) + error("%llu: failed to enable TCP_NODELAY on socket.", w->id); + + flag = 1; + if(setsockopt(w->ifd, SOL_SOCKET, SO_KEEPALIVE, (char *) &flag, sizeof(int)) != 0) + error("%llu: Cannot set SO_KEEPALIVE on socket.", w->id); + } + + w->response.data = buffer_create(INITIAL_WEB_DATA_LENGTH); + w->response.header = buffer_create(HTTP_RESPONSE_HEADER_SIZE); + w->response.header_output = buffer_create(HTTP_RESPONSE_HEADER_SIZE); + w->origin[0] = '*'; + w->wait_receive = 1; + + if(web_clients) web_clients->prev = w; + w->next = web_clients; + web_clients = w; + + web_client_connected(); + + return(w); } -void web_client_reset(struct web_client *w) -{ - struct timeval tv; - gettimeofday(&tv, NULL); +void web_client_reset(struct web_client *w) { + web_client_uncrock_socket(w); + + debug(D_WEB_CLIENT, "%llu: Reseting client.", w->id); - long sent = (w->mode == WEB_CLIENT_MODE_FILECOPY)?w->response.rlen:w->response.data->len; + if(likely(w->last_url[0])) { + struct timeval tv; + gettimeofday(&tv, NULL); + size_t size = (w->mode == WEB_CLIENT_MODE_FILECOPY)?w->response.rlen:w->response.data->len; + size_t sent = size; #ifdef NETDATA_WITH_ZLIB - if(likely(w->response.zoutput)) sent = (long)w->response.zstream.total_out; + if(likely(w->response.zoutput)) sent = (size_t)w->response.zstream.total_out; #endif - long size = (w->mode == WEB_CLIENT_MODE_FILECOPY)?w->response.rlen:w->response.data->len; - - if(likely(w->last_url[0])) - log_access("%llu: (sent/all = %ld/%ld bytes %0.0f%%, prep/sent/total = %0.2f/%0.2f/%0.2f ms) %s: %d '%s'", - w->id, - sent, size, -((size>0)?((float)(size-sent)/(float)size * 100.0):0.0), - (float)usecdiff(&w->tv_ready, &w->tv_in) / 1000.0, - (float)usecdiff(&tv, &w->tv_ready) / 1000.0, - (float)usecdiff(&tv, &w->tv_in) / 1000.0, - (w->mode == WEB_CLIENT_MODE_FILECOPY)?"filecopy":((w->mode == WEB_CLIENT_MODE_OPTIONS)?"options":"data"), - w->response.code, - w->last_url - ); - - debug(D_WEB_CLIENT, "%llu: Reseting client.", w->id); - - if(unlikely(w->mode == WEB_CLIENT_MODE_FILECOPY)) { - debug(D_WEB_CLIENT, "%llu: Closing filecopy input file.", w->id); - close(w->ifd); - w->ifd = w->ofd; - } - - w->last_url[0] = '\0'; - w->cookie1[0] = '\0'; - w->cookie2[0] = '\0'; - w->origin[0] = '*'; - w->origin[1] = '\0'; - - w->mode = WEB_CLIENT_MODE_NORMAL; - w->enable_gzip = 0; - w->keepalive = 0; - if(w->decoded_url) { - free(w->decoded_url); - w->decoded_url = NULL; - } - - buffer_reset(w->response.header_output); - buffer_reset(w->response.header); - buffer_reset(w->response.data); - w->response.rlen = 0; - w->response.sent = 0; - w->response.code = 0; - - w->wait_receive = 1; - w->wait_send = 0; - - w->response.zoutput = 0; - - // if we had enabled compression, release it + // -------------------------------------------------------------------- + // global statistics + + finished_web_request_statistics(usec_dt(&tv, &w->tv_in), + w->stats_received_bytes, + w->stats_sent_bytes, + size, + sent); + + w->stats_received_bytes = 0; + w->stats_sent_bytes = 0; + + + // -------------------------------------------------------------------- + // access log + + log_access("%llu: (sent/all = %zu/%zu bytes %0.0f%%, prep/sent/total = %0.2f/%0.2f/%0.2f ms) %s: %d '%s'", + w->id, + sent, size, -((size > 0) ? ((size - sent) / (double) size * 100.0) : 0.0), + usec_dt(&w->tv_ready, &w->tv_in) / 1000.0, + usec_dt(&tv, &w->tv_ready) / 1000.0, + usec_dt(&tv, &w->tv_in) / 1000.0, + (w->mode == WEB_CLIENT_MODE_FILECOPY) ? "filecopy" : ((w->mode == WEB_CLIENT_MODE_OPTIONS) + ? "options" : "data"), + w->response.code, + w->last_url + ); + } + + if(unlikely(w->mode == WEB_CLIENT_MODE_FILECOPY)) { + if(w->ifd != w->ofd) { + debug(D_WEB_CLIENT, "%llu: Closing filecopy input file descriptor %d.", w->id, w->ifd); + if(w->ifd != -1) close(w->ifd); + w->ifd = w->ofd; + } + } + + w->last_url[0] = '\0'; + w->cookie1[0] = '\0'; + w->cookie2[0] = '\0'; + w->origin[0] = '*'; + w->origin[1] = '\0'; + + w->mode = WEB_CLIENT_MODE_NORMAL; + + w->tcp_cork = 0; + w->donottrack = 0; + w->tracking_required = 0; + w->keepalive = 0; + w->decoded_url[0] = '\0'; + + buffer_reset(w->response.header_output); + buffer_reset(w->response.header); + buffer_reset(w->response.data); + w->response.rlen = 0; + w->response.sent = 0; + w->response.code = 0; + + w->wait_receive = 1; + w->wait_send = 0; + + w->response.zoutput = 0; + + // if we had enabled compression, release it #ifdef NETDATA_WITH_ZLIB - if(w->response.zinitialized) { - debug(D_DEFLATE, "%llu: Reseting compression.", w->id); - deflateEnd(&w->response.zstream); - w->response.zsent = 0; - w->response.zhave = 0; - w->response.zstream.avail_in = 0; - w->response.zstream.avail_out = 0; - w->response.zstream.total_in = 0; - w->response.zstream.total_out = 0; - w->response.zinitialized = 0; - } + if(w->response.zinitialized) { + debug(D_DEFLATE, "%llu: Freeing compression resources.", w->id); + deflateEnd(&w->response.zstream); + w->response.zsent = 0; + w->response.zhave = 0; + w->response.zstream.avail_in = 0; + w->response.zstream.avail_out = 0; + w->response.zstream.total_in = 0; + w->response.zstream.total_out = 0; + w->response.zinitialized = 0; + } #endif // NETDATA_WITH_ZLIB } -struct web_client *web_client_free(struct web_client *w) -{ - struct web_client *n = w->next; - - debug(D_WEB_CLIENT_ACCESS, "%llu: Closing web client from %s port %s.", w->id, w->client_ip, w->client_port); +struct web_client *web_client_free(struct web_client *w) { + web_client_reset(w); - if(w->prev) w->prev->next = w->next; - if(w->next) w->next->prev = w->prev; + struct web_client *n = w->next; + if(w == web_clients) web_clients = n; - if(w == web_clients) web_clients = w->next; + debug(D_WEB_CLIENT_ACCESS, "%llu: Closing web client from %s port %s.", w->id, w->client_ip, w->client_port); - if(w->response.header_output) buffer_free(w->response.header_output); - if(w->response.header) buffer_free(w->response.header); - if(w->response.data) buffer_free(w->response.data); - close(w->ifd); - if(w->ofd != w->ifd) close(w->ofd); - free(w); + if(w->prev) w->prev->next = w->next; + if(w->next) w->next->prev = w->prev; + if(w->response.header_output) buffer_free(w->response.header_output); + if(w->response.header) buffer_free(w->response.header); + if(w->response.data) buffer_free(w->response.data); + if(w->ifd != -1) close(w->ifd); + if(w->ofd != -1 && w->ofd != w->ifd) close(w->ofd); + freez(w); - global_statistics.connected_clients--; + web_client_disconnected(); - return(n); + return(n); } -uid_t web_files_uid(void) -{ - static char *web_owner = NULL; - static uid_t owner_uid = 0; - - if(unlikely(!web_owner)) { - web_owner = config_get("global", "web files owner", config_get("global", "run as user", "")); - if(!web_owner || !*web_owner) - owner_uid = geteuid(); - else { - // getpwnam() is not thread safe, - // but we have called this function once - // while single threaded - struct passwd *pw = getpwnam(web_owner); - if(!pw) { - error("User %s is not present. Ignoring option.", web_owner); - owner_uid = geteuid(); - } - else { - debug(D_WEB_CLIENT, "Web files owner set to %s.\n", web_owner); - owner_uid = pw->pw_uid; - } - } - } - - return(owner_uid); +uid_t web_files_uid(void) { + static char *web_owner = NULL; + static uid_t owner_uid = 0; + + if(unlikely(!web_owner)) { + web_owner = config_get("global", "web files owner", config_get("global", "run as user", "")); + if(!web_owner || !*web_owner) + owner_uid = geteuid(); + else { + // getpwnam() is not thread safe, + // but we have called this function once + // while single threaded + struct passwd *pw = getpwnam(web_owner); + if(!pw) { + error("User '%s' is not present. Ignoring option.", web_owner); + owner_uid = geteuid(); + } + else { + debug(D_WEB_CLIENT, "Web files owner set to %s.", web_owner); + owner_uid = pw->pw_uid; + } + } + } + + return(owner_uid); } -gid_t web_files_gid(void) -{ - static char *web_group = NULL; - static gid_t owner_gid = 0; - - if(unlikely(!web_group)) { - web_group = config_get("global", "web files group", config_get("global", "web files owner", "")); - if(!web_group || !*web_group) - owner_gid = getegid(); - else { - // getgrnam() is not thread safe, - // but we have called this function once - // while single threaded - struct group *gr = getgrnam(web_group); - if(!gr) { - error("Group %s is not present. Ignoring option.", web_group); - owner_gid = getegid(); - } - else { - debug(D_WEB_CLIENT, "Web files group set to %s.\n", web_group); - owner_gid = gr->gr_gid; - } - } - } - - return(owner_gid); +gid_t web_files_gid(void) { + static char *web_group = NULL; + static gid_t owner_gid = 0; + + if(unlikely(!web_group)) { + web_group = config_get("global", "web files group", config_get("global", "web files owner", "")); + if(!web_group || !*web_group) + owner_gid = getegid(); + else { + // getgrnam() is not thread safe, + // but we have called this function once + // while single threaded + struct group *gr = getgrnam(web_group); + if(!gr) { + error("Group '%s' is not present. Ignoring option.", web_group); + owner_gid = getegid(); + } + else { + debug(D_WEB_CLIENT, "Web files group set to %s.", web_group); + owner_gid = gr->gr_gid; + } + } + } + + return(owner_gid); } int mysendfile(struct web_client *w, char *filename) { - static char *web_dir = NULL; - - // initialize our static data - if(unlikely(!web_dir)) web_dir = config_get("global", "web files directory", WEB_DIR); - - debug(D_WEB_CLIENT, "%llu: Looking for file '%s/%s'", w->id, web_dir, filename); - - // skip leading slashes - while (*filename == '/') filename++; - - // if the filename contain known paths, skip them - if(strncmp(filename, WEB_PATH_FILE "/", strlen(WEB_PATH_FILE) + 1) == 0) filename = &filename[strlen(WEB_PATH_FILE) + 1]; - - char *s; - for(s = filename; *s ;s++) { - if( !isalnum(*s) && *s != '/' && *s != '.' && *s != '-' && *s != '_') { - debug(D_WEB_CLIENT_ACCESS, "%llu: File '%s' is not acceptable.", w->id, filename); - buffer_sprintf(w->response.data, "File '%s' cannot be served. Filename contains invalid character '%c'", filename, *s); - return 400; - } - } - - // if the filename contains a .. refuse to serve it - if(strstr(filename, "..") != 0) { - debug(D_WEB_CLIENT_ACCESS, "%llu: File '%s' is not acceptable.", w->id, filename); - buffer_sprintf(w->response.data, "File '%s' cannot be served. Relative filenames with '..' in them are not supported.", filename); - return 400; - } - - // access the file - char webfilename[FILENAME_MAX + 1]; - snprintfz(webfilename, FILENAME_MAX, "%s/%s", web_dir, filename); - - // check if the file exists - struct stat stat; - if(lstat(webfilename, &stat) != 0) { - debug(D_WEB_CLIENT_ACCESS, "%llu: File '%s' is not found.", w->id, webfilename); - buffer_sprintf(w->response.data, "File '%s' does not exist, or is not accessible.", webfilename); - return 404; - } - - // check if the file is owned by expected user - if(stat.st_uid != web_files_uid()) { - error("%llu: File '%s' is owned by user %d (expected user %d). Access Denied.", w->id, webfilename, stat.st_uid, web_files_uid()); - buffer_sprintf(w->response.data, "Access to file '%s' is not permitted.", webfilename); - return 403; - } - - // check if the file is owned by expected group - if(stat.st_gid != web_files_gid()) { - error("%llu: File '%s' is owned by group %d (expected group %d). Access Denied.", w->id, webfilename, stat.st_gid, web_files_gid()); - buffer_sprintf(w->response.data, "Access to file '%s' is not permitted.", webfilename); - return 403; - } - - if((stat.st_mode & S_IFMT) == S_IFDIR) { - snprintfz(webfilename, FILENAME_MAX, "%s/index.html", filename); - return mysendfile(w, webfilename); - } - - if((stat.st_mode & S_IFMT) != S_IFREG) { - error("%llu: File '%s' is not a regular file. Access Denied.", w->id, webfilename); - buffer_sprintf(w->response.data, "Access to file '%s' is not permitted.", webfilename); - return 403; - } - - // open the file - w->ifd = open(webfilename, O_NONBLOCK, O_RDONLY); - if(w->ifd == -1) { - w->ifd = w->ofd; - - if(errno == EBUSY || errno == EAGAIN) { - error("%llu: File '%s' is busy, sending 307 Moved Temporarily to force retry.", w->id, webfilename); - buffer_sprintf(w->response.header, "Location: /" WEB_PATH_FILE "/%s\r\n", filename); - buffer_sprintf(w->response.data, "The file '%s' is currently busy. Please try again later.", webfilename); - return 307; - } - else { - error("%llu: Cannot open file '%s'.", w->id, webfilename); - buffer_sprintf(w->response.data, "Cannot open file '%s'.", webfilename); - return 404; - } - } - - // pick a Content-Type for the file - if(strstr(filename, ".html") != NULL) w->response.data->contenttype = CT_TEXT_HTML; - else if(strstr(filename, ".js") != NULL) w->response.data->contenttype = CT_APPLICATION_X_JAVASCRIPT; - else if(strstr(filename, ".css") != NULL) w->response.data->contenttype = CT_TEXT_CSS; - else if(strstr(filename, ".xml") != NULL) w->response.data->contenttype = CT_TEXT_XML; - else if(strstr(filename, ".xsl") != NULL) w->response.data->contenttype = CT_TEXT_XSL; - else if(strstr(filename, ".txt") != NULL) w->response.data->contenttype = CT_TEXT_PLAIN; - else if(strstr(filename, ".svg") != NULL) w->response.data->contenttype = CT_IMAGE_SVG_XML; - else if(strstr(filename, ".ttf") != NULL) w->response.data->contenttype = CT_APPLICATION_X_FONT_TRUETYPE; - else if(strstr(filename, ".otf") != NULL) w->response.data->contenttype = CT_APPLICATION_X_FONT_OPENTYPE; - else if(strstr(filename, ".woff2")!= NULL) w->response.data->contenttype = CT_APPLICATION_FONT_WOFF2; - else if(strstr(filename, ".woff") != NULL) w->response.data->contenttype = CT_APPLICATION_FONT_WOFF; - else if(strstr(filename, ".eot") != NULL) w->response.data->contenttype = CT_APPLICATION_VND_MS_FONTOBJ; - else if(strstr(filename, ".png") != NULL) w->response.data->contenttype = CT_IMAGE_PNG; - else if(strstr(filename, ".jpg") != NULL) w->response.data->contenttype = CT_IMAGE_JPG; - else if(strstr(filename, ".jpeg") != NULL) w->response.data->contenttype = CT_IMAGE_JPG; - else if(strstr(filename, ".gif") != NULL) w->response.data->contenttype = CT_IMAGE_GIF; - else if(strstr(filename, ".bmp") != NULL) w->response.data->contenttype = CT_IMAGE_BMP; - else if(strstr(filename, ".ico") != NULL) w->response.data->contenttype = CT_IMAGE_XICON; - else if(strstr(filename, ".icns") != NULL) w->response.data->contenttype = CT_IMAGE_ICNS; - else w->response.data->contenttype = CT_APPLICATION_OCTET_STREAM; - - debug(D_WEB_CLIENT_ACCESS, "%llu: Sending file '%s' (%ld bytes, ifd %d, ofd %d).", w->id, webfilename, stat.st_size, w->ifd, w->ofd); - - w->mode = WEB_CLIENT_MODE_FILECOPY; - w->wait_receive = 1; - w->wait_send = 0; - buffer_flush(w->response.data); - w->response.rlen = stat.st_size; - w->response.data->date = stat.st_mtim.tv_sec; - - return 200; + static char *web_dir = NULL; + + // initialize our static data + if(unlikely(!web_dir)) web_dir = config_get("global", "web files directory", WEB_DIR); + + debug(D_WEB_CLIENT, "%llu: Looking for file '%s/%s'", w->id, web_dir, filename); + + // skip leading slashes + while (*filename == '/') filename++; + + // if the filename contain known paths, skip them + if(strncmp(filename, WEB_PATH_FILE "/", strlen(WEB_PATH_FILE) + 1) == 0) + filename = &filename[strlen(WEB_PATH_FILE) + 1]; + + char *s; + for(s = filename; *s ;s++) { + if( !isalnum(*s) && *s != '/' && *s != '.' && *s != '-' && *s != '_') { + debug(D_WEB_CLIENT_ACCESS, "%llu: File '%s' is not acceptable.", w->id, filename); + buffer_sprintf(w->response.data, "File '%s' cannot be served. Filename contains invalid character '%c'", filename, *s); + return 400; + } + } + + // if the filename contains a .. refuse to serve it + if(strstr(filename, "..") != 0) { + debug(D_WEB_CLIENT_ACCESS, "%llu: File '%s' is not acceptable.", w->id, filename); + buffer_sprintf(w->response.data, "File '%s' cannot be served. Relative filenames with '..' in them are not supported.", filename); + return 400; + } + + // access the file + char webfilename[FILENAME_MAX + 1]; + snprintfz(webfilename, FILENAME_MAX, "%s/%s", web_dir, filename); + + // check if the file exists + struct stat stat; + if(lstat(webfilename, &stat) != 0) { + debug(D_WEB_CLIENT_ACCESS, "%llu: File '%s' is not found.", w->id, webfilename); + buffer_sprintf(w->response.data, "File '%s' does not exist, or is not accessible.", webfilename); + return 404; + } + + // check if the file is owned by expected user + if(stat.st_uid != web_files_uid()) { + error("%llu: File '%s' is owned by user %u (expected user %u). Access Denied.", w->id, webfilename, stat.st_uid, web_files_uid()); + buffer_sprintf(w->response.data, "Access to file '%s' is not permitted.", webfilename); + return 403; + } + + // check if the file is owned by expected group + if(stat.st_gid != web_files_gid()) { + error("%llu: File '%s' is owned by group %u (expected group %u). Access Denied.", w->id, webfilename, stat.st_gid, web_files_gid()); + buffer_sprintf(w->response.data, "Access to file '%s' is not permitted.", webfilename); + return 403; + } + + if((stat.st_mode & S_IFMT) == S_IFDIR) { + snprintfz(webfilename, FILENAME_MAX, "%s/index.html", filename); + return mysendfile(w, webfilename); + } + + if((stat.st_mode & S_IFMT) != S_IFREG) { + error("%llu: File '%s' is not a regular file. Access Denied.", w->id, webfilename); + buffer_sprintf(w->response.data, "Access to file '%s' is not permitted.", webfilename); + return 403; + } + + // open the file + w->ifd = open(webfilename, O_NONBLOCK, O_RDONLY); + if(w->ifd == -1) { + w->ifd = w->ofd; + + if(errno == EBUSY || errno == EAGAIN) { + error("%llu: File '%s' is busy, sending 307 Moved Temporarily to force retry.", w->id, webfilename); + buffer_sprintf(w->response.header, "Location: /" WEB_PATH_FILE "/%s\r\n", filename); + buffer_sprintf(w->response.data, "The file '%s' is currently busy. Please try again later.", webfilename); + return 307; + } + else { + error("%llu: Cannot open file '%s'.", w->id, webfilename); + buffer_sprintf(w->response.data, "Cannot open file '%s'.", webfilename); + return 404; + } + } + if(fcntl(w->ifd, F_SETFL, O_NONBLOCK) < 0) + error("%llu: Cannot set O_NONBLOCK on file '%s'.", w->id, webfilename); + + // pick a Content-Type for the file + if(strstr(filename, ".html") != NULL) w->response.data->contenttype = CT_TEXT_HTML; + else if(strstr(filename, ".js") != NULL) w->response.data->contenttype = CT_APPLICATION_X_JAVASCRIPT; + else if(strstr(filename, ".css") != NULL) w->response.data->contenttype = CT_TEXT_CSS; + else if(strstr(filename, ".xml") != NULL) w->response.data->contenttype = CT_TEXT_XML; + else if(strstr(filename, ".xsl") != NULL) w->response.data->contenttype = CT_TEXT_XSL; + else if(strstr(filename, ".txt") != NULL) w->response.data->contenttype = CT_TEXT_PLAIN; + else if(strstr(filename, ".svg") != NULL) w->response.data->contenttype = CT_IMAGE_SVG_XML; + else if(strstr(filename, ".ttf") != NULL) w->response.data->contenttype = CT_APPLICATION_X_FONT_TRUETYPE; + else if(strstr(filename, ".otf") != NULL) w->response.data->contenttype = CT_APPLICATION_X_FONT_OPENTYPE; + else if(strstr(filename, ".woff2")!= NULL) w->response.data->contenttype = CT_APPLICATION_FONT_WOFF2; + else if(strstr(filename, ".woff") != NULL) w->response.data->contenttype = CT_APPLICATION_FONT_WOFF; + else if(strstr(filename, ".eot") != NULL) w->response.data->contenttype = CT_APPLICATION_VND_MS_FONTOBJ; + else if(strstr(filename, ".png") != NULL) w->response.data->contenttype = CT_IMAGE_PNG; + else if(strstr(filename, ".jpg") != NULL) w->response.data->contenttype = CT_IMAGE_JPG; + else if(strstr(filename, ".jpeg") != NULL) w->response.data->contenttype = CT_IMAGE_JPG; + else if(strstr(filename, ".gif") != NULL) w->response.data->contenttype = CT_IMAGE_GIF; + else if(strstr(filename, ".bmp") != NULL) w->response.data->contenttype = CT_IMAGE_BMP; + else if(strstr(filename, ".ico") != NULL) w->response.data->contenttype = CT_IMAGE_XICON; + else if(strstr(filename, ".icns") != NULL) w->response.data->contenttype = CT_IMAGE_ICNS; + else w->response.data->contenttype = CT_APPLICATION_OCTET_STREAM; + + debug(D_WEB_CLIENT_ACCESS, "%llu: Sending file '%s' (%ld bytes, ifd %d, ofd %d).", w->id, webfilename, stat.st_size, w->ifd, w->ofd); + + w->mode = WEB_CLIENT_MODE_FILECOPY; + w->wait_receive = 1; + w->wait_send = 0; + buffer_flush(w->response.data); + w->response.rlen = stat.st_size; + w->response.data->date = stat.st_mtim.tv_sec; + + return 200; } #ifdef NETDATA_WITH_ZLIB -void web_client_enable_deflate(struct web_client *w) { - if(w->response.zinitialized == 1) { - error("%llu: Compression has already be initialized for this client.", w->id); - return; - } - - if(w->response.sent) { - error("%llu: Cannot enable compression in the middle of a conversation.", w->id); - return; - } - - w->response.zstream.zalloc = Z_NULL; - w->response.zstream.zfree = Z_NULL; - w->response.zstream.opaque = Z_NULL; - - w->response.zstream.next_in = (Bytef *)w->response.data->buffer; - w->response.zstream.avail_in = 0; - w->response.zstream.total_in = 0; - - w->response.zstream.next_out = w->response.zbuffer; - w->response.zstream.avail_out = 0; - w->response.zstream.total_out = 0; - - w->response.zstream.zalloc = Z_NULL; - w->response.zstream.zfree = Z_NULL; - w->response.zstream.opaque = Z_NULL; - -// if(deflateInit(&w->response.zstream, Z_DEFAULT_COMPRESSION) != Z_OK) { -// error("%llu: Failed to initialize zlib. Proceeding without compression.", w->id); -// return; -// } - - // Select GZIP compression: windowbits = 15 + 16 = 31 - if(deflateInit2(&w->response.zstream, Z_DEFAULT_COMPRESSION, Z_DEFLATED, 31, 8, Z_DEFAULT_STRATEGY) != Z_OK) { - error("%llu: Failed to initialize zlib. Proceeding without compression.", w->id); - return; - } - - w->response.zsent = 0; - w->response.zoutput = 1; - w->response.zinitialized = 1; - - debug(D_DEFLATE, "%llu: Initialized compression.", w->id); +void web_client_enable_deflate(struct web_client *w, int gzip) { + if(unlikely(w->response.zinitialized)) { + error("%llu: Compression has already be initialized for this client.", w->id); + return; + } + + if(unlikely(w->response.sent)) { + error("%llu: Cannot enable compression in the middle of a conversation.", w->id); + return; + } + + w->response.zstream.zalloc = Z_NULL; + w->response.zstream.zfree = Z_NULL; + w->response.zstream.opaque = Z_NULL; + + w->response.zstream.next_in = (Bytef *)w->response.data->buffer; + w->response.zstream.avail_in = 0; + w->response.zstream.total_in = 0; + + w->response.zstream.next_out = w->response.zbuffer; + w->response.zstream.avail_out = 0; + w->response.zstream.total_out = 0; + + w->response.zstream.zalloc = Z_NULL; + w->response.zstream.zfree = Z_NULL; + w->response.zstream.opaque = Z_NULL; + +// if(deflateInit(&w->response.zstream, Z_DEFAULT_COMPRESSION) != Z_OK) { +// error("%llu: Failed to initialize zlib. Proceeding without compression.", w->id); +// return; +// } + + // Select GZIP compression: windowbits = 15 + 16 = 31 + if(deflateInit2(&w->response.zstream, web_gzip_level, Z_DEFLATED, 15 + ((gzip)?16:0), 8, web_gzip_strategy) != Z_OK) { + error("%llu: Failed to initialize zlib. Proceeding without compression.", w->id); + return; + } + + w->response.zsent = 0; + w->response.zoutput = 1; + w->response.zinitialized = 1; + + debug(D_DEFLATE, "%llu: Initialized compression.", w->id); } #endif // NETDATA_WITH_ZLIB +void buffer_data_options2string(BUFFER *wb, uint32_t options) { + int count = 0; + + if(options & RRDR_OPTION_NONZERO) { + if(count++) buffer_strcat(wb, " "); + buffer_strcat(wb, "nonzero"); + } + + if(options & RRDR_OPTION_REVERSED) { + if(count++) buffer_strcat(wb, " "); + buffer_strcat(wb, "flip"); + } + + if(options & RRDR_OPTION_JSON_WRAP) { + if(count++) buffer_strcat(wb, " "); + buffer_strcat(wb, "jsonwrap"); + } + + if(options & RRDR_OPTION_MIN2MAX) { + if(count++) buffer_strcat(wb, " "); + buffer_strcat(wb, "min2max"); + } + + if(options & RRDR_OPTION_MILLISECONDS) { + if(count++) buffer_strcat(wb, " "); + buffer_strcat(wb, "ms"); + } + + if(options & RRDR_OPTION_ABSOLUTE) { + if(count++) buffer_strcat(wb, " "); + buffer_strcat(wb, "abs"); + } + + if(options & RRDR_OPTION_SECONDS) { + if(count++) buffer_strcat(wb, " "); + buffer_strcat(wb, "seconds"); + } + + if(options & RRDR_OPTION_NULL2ZERO) { + if(count++) buffer_strcat(wb, " "); + buffer_strcat(wb, "null2zero"); + } + + if(options & RRDR_OPTION_OBJECTSROWS) { + if(count++) buffer_strcat(wb, " "); + buffer_strcat(wb, "objectrows"); + } + + if(options & RRDR_OPTION_GOOGLE_JSON) { + if(count++) buffer_strcat(wb, " "); + buffer_strcat(wb, "google_json"); + } + + if(options & RRDR_OPTION_PERCENTAGE) { + if(count++) buffer_strcat(wb, " "); + buffer_strcat(wb, "percentage"); + } + + if(options & RRDR_OPTION_NOT_ALIGNED) { + if(count++) buffer_strcat(wb, " "); + buffer_strcat(wb, "unaligned"); + } +} + uint32_t web_client_api_request_v1_data_options(char *o) { - uint32_t ret = 0x00000000; - char *tok; - - while(o && *o && (tok = mystrsep(&o, ", |"))) { - if(!*tok) continue; - - if(!strcmp(tok, "nonzero")) - ret |= RRDR_OPTION_NONZERO; - else if(!strcmp(tok, "flip") || !strcmp(tok, "reversed") || !strcmp(tok, "reverse")) - ret |= RRDR_OPTION_REVERSED; - else if(!strcmp(tok, "jsonwrap")) - ret |= RRDR_OPTION_JSON_WRAP; - else if(!strcmp(tok, "min2max")) - ret |= RRDR_OPTION_MIN2MAX; - else if(!strcmp(tok, "ms") || !strcmp(tok, "milliseconds")) - ret |= RRDR_OPTION_MILLISECONDS; - else if(!strcmp(tok, "abs") || !strcmp(tok, "absolute") || !strcmp(tok, "absolute_sum") || !strcmp(tok, "absolute-sum")) - ret |= RRDR_OPTION_ABSOLUTE; - else if(!strcmp(tok, "seconds")) - ret |= RRDR_OPTION_SECONDS; - else if(!strcmp(tok, "null2zero")) - ret |= RRDR_OPTION_NULL2ZERO; - else if(!strcmp(tok, "objectrows")) - ret |= RRDR_OPTION_OBJECTSROWS; - else if(!strcmp(tok, "google_json")) - ret |= RRDR_OPTION_GOOGLE_JSON; - else if(!strcmp(tok, "percentage")) - ret |= RRDR_OPTION_PERCENTAGE; - } - - return ret; + uint32_t ret = 0x00000000; + char *tok; + + while(o && *o && (tok = mystrsep(&o, ", |"))) { + if(!*tok) continue; + + if(!strcmp(tok, "nonzero")) + ret |= RRDR_OPTION_NONZERO; + else if(!strcmp(tok, "flip") || !strcmp(tok, "reversed") || !strcmp(tok, "reverse")) + ret |= RRDR_OPTION_REVERSED; + else if(!strcmp(tok, "jsonwrap")) + ret |= RRDR_OPTION_JSON_WRAP; + else if(!strcmp(tok, "min2max")) + ret |= RRDR_OPTION_MIN2MAX; + else if(!strcmp(tok, "ms") || !strcmp(tok, "milliseconds")) + ret |= RRDR_OPTION_MILLISECONDS; + else if(!strcmp(tok, "abs") || !strcmp(tok, "absolute") || !strcmp(tok, "absolute_sum") || !strcmp(tok, "absolute-sum")) + ret |= RRDR_OPTION_ABSOLUTE; + else if(!strcmp(tok, "seconds")) + ret |= RRDR_OPTION_SECONDS; + else if(!strcmp(tok, "null2zero")) + ret |= RRDR_OPTION_NULL2ZERO; + else if(!strcmp(tok, "objectrows")) + ret |= RRDR_OPTION_OBJECTSROWS; + else if(!strcmp(tok, "google_json")) + ret |= RRDR_OPTION_GOOGLE_JSON; + else if(!strcmp(tok, "percentage")) + ret |= RRDR_OPTION_PERCENTAGE; + else if(!strcmp(tok, "unaligned")) + ret |= RRDR_OPTION_NOT_ALIGNED; + } + + return ret; } uint32_t web_client_api_request_v1_data_format(char *name) { - if(!strcmp(name, DATASOURCE_FORMAT_DATATABLE_JSON)) // datatable - return DATASOURCE_DATATABLE_JSON; + if(!strcmp(name, DATASOURCE_FORMAT_DATATABLE_JSON)) // datatable + return DATASOURCE_DATATABLE_JSON; - else if(!strcmp(name, DATASOURCE_FORMAT_DATATABLE_JSONP)) // datasource - return DATASOURCE_DATATABLE_JSONP; + else if(!strcmp(name, DATASOURCE_FORMAT_DATATABLE_JSONP)) // datasource + return DATASOURCE_DATATABLE_JSONP; - else if(!strcmp(name, DATASOURCE_FORMAT_JSON)) // json - return DATASOURCE_JSON; + else if(!strcmp(name, DATASOURCE_FORMAT_JSON)) // json + return DATASOURCE_JSON; - else if(!strcmp(name, DATASOURCE_FORMAT_JSONP)) // jsonp - return DATASOURCE_JSONP; + else if(!strcmp(name, DATASOURCE_FORMAT_JSONP)) // jsonp + return DATASOURCE_JSONP; - else if(!strcmp(name, DATASOURCE_FORMAT_SSV)) // ssv - return DATASOURCE_SSV; + else if(!strcmp(name, DATASOURCE_FORMAT_SSV)) // ssv + return DATASOURCE_SSV; - else if(!strcmp(name, DATASOURCE_FORMAT_CSV)) // csv - return DATASOURCE_CSV; + else if(!strcmp(name, DATASOURCE_FORMAT_CSV)) // csv + return DATASOURCE_CSV; - else if(!strcmp(name, DATASOURCE_FORMAT_TSV) || !strcmp(name, "tsv-excel")) // tsv - return DATASOURCE_TSV; + else if(!strcmp(name, DATASOURCE_FORMAT_TSV) || !strcmp(name, "tsv-excel")) // tsv + return DATASOURCE_TSV; - else if(!strcmp(name, DATASOURCE_FORMAT_HTML)) // html - return DATASOURCE_HTML; + else if(!strcmp(name, DATASOURCE_FORMAT_HTML)) // html + return DATASOURCE_HTML; - else if(!strcmp(name, DATASOURCE_FORMAT_JS_ARRAY)) // array - return DATASOURCE_JS_ARRAY; + else if(!strcmp(name, DATASOURCE_FORMAT_JS_ARRAY)) // array + return DATASOURCE_JS_ARRAY; - else if(!strcmp(name, DATASOURCE_FORMAT_SSV_COMMA)) // ssvcomma - return DATASOURCE_SSV_COMMA; + else if(!strcmp(name, DATASOURCE_FORMAT_SSV_COMMA)) // ssvcomma + return DATASOURCE_SSV_COMMA; - else if(!strcmp(name, DATASOURCE_FORMAT_CSV_JSON_ARRAY)) // csvjsonarray - return DATASOURCE_CSV_JSON_ARRAY; + else if(!strcmp(name, DATASOURCE_FORMAT_CSV_JSON_ARRAY)) // csvjsonarray + return DATASOURCE_CSV_JSON_ARRAY; - return DATASOURCE_JSON; + return DATASOURCE_JSON; } uint32_t web_client_api_request_v1_data_google_format(char *name) { - if(!strcmp(name, "json")) - return DATASOURCE_DATATABLE_JSONP; + if(!strcmp(name, "json")) + return DATASOURCE_DATATABLE_JSONP; - else if(!strcmp(name, "html")) - return DATASOURCE_HTML; + else if(!strcmp(name, "html")) + return DATASOURCE_HTML; - else if(!strcmp(name, "csv")) - return DATASOURCE_CSV; + else if(!strcmp(name, "csv")) + return DATASOURCE_CSV; - else if(!strcmp(name, "tsv-excel")) - return DATASOURCE_TSV; + else if(!strcmp(name, "tsv-excel")) + return DATASOURCE_TSV; - return DATASOURCE_JSON; + return DATASOURCE_JSON; } -int web_client_api_request_v1_data_group(char *name) +const char *group_method2string(int group) { + switch(group) { + case GROUP_UNDEFINED: + return ""; + + case GROUP_AVERAGE: + return "average"; + + case GROUP_MIN: + return "min"; + + case GROUP_MAX: + return "max"; + + case GROUP_SUM: + return "sum"; + + case GROUP_INCREMENTAL_SUM: + return "incremental-sum"; + + default: + return "unknown-group-method"; + } +} + +int web_client_api_request_v1_data_group(char *name, int def) { - if(!strcmp(name, "max")) - return GROUP_MAX; + if(!strcmp(name, "average")) + return GROUP_AVERAGE; + + else if(!strcmp(name, "min")) + return GROUP_MIN; + + else if(!strcmp(name, "max")) + return GROUP_MAX; + + else if(!strcmp(name, "sum")) + return GROUP_SUM; - else if(!strcmp(name, "average")) - return GROUP_AVERAGE; + else if(!strcmp(name, "incremental-sum")) + return GROUP_INCREMENTAL_SUM; - return GROUP_MAX; + return def; +} + +int web_client_api_request_v1_alarms(struct web_client *w, char *url) +{ + int all = 0; + + while(url) { + char *value = mystrsep(&url, "?&[]"); + if (!value || !*value) continue; + + if(!strcmp(value, "all")) all = 1; + else if(!strcmp(value, "active")) all = 0; + } + + buffer_flush(w->response.data); + w->response.data->contenttype = CT_APPLICATION_JSON; + health_alarms2json(&localhost, w->response.data, all); + return 200; +} + +int web_client_api_request_v1_alarm_log(struct web_client *w, char *url) +{ + (void)url; + + buffer_flush(w->response.data); + w->response.data->contenttype = CT_APPLICATION_JSON; + health_alarm_log2json(&localhost, w->response.data); + return 200; } int web_client_api_request_v1_charts(struct web_client *w, char *url) { - if(url) { ; } + if(url) { ; } - buffer_flush(w->response.data); - w->response.data->contenttype = CT_APPLICATION_JSON; - rrd_stats_api_v1_charts(w->response.data); - return 200; + buffer_flush(w->response.data); + w->response.data->contenttype = CT_APPLICATION_JSON; + rrd_stats_api_v1_charts(w->response.data); + return 200; } int web_client_api_request_v1_chart(struct web_client *w, char *url) { - int ret = 400; - char *chart = NULL; - - buffer_flush(w->response.data); - - while(url) { - char *value = mystrsep(&url, "?&[]"); - if(!value || !*value) continue; - - char *name = mystrsep(&value, "="); - if(!name || !*name) continue; - if(!value || !*value) continue; - - // name and value are now the parameters - // they are not null and not empty - - if(!strcmp(name, "chart")) chart = value; - //else { - /// buffer_sprintf(w->response.data, "Unknown parameter '%s' in request.", name); - // goto cleanup; - //} - } - - if(!chart || !*chart) { - buffer_sprintf(w->response.data, "No chart id is given at the request."); - goto cleanup; - } - - RRDSET *st = rrdset_find(chart); - if(!st) st = rrdset_find_byname(chart); - if(!st) { - buffer_sprintf(w->response.data, "Chart '%s' is not found.", chart); - ret = 404; - goto cleanup; - } - - w->response.data->contenttype = CT_APPLICATION_JSON; - rrd_stats_api_v1_chart(st, w->response.data); - return 200; + int ret = 400; + char *chart = NULL; + + buffer_flush(w->response.data); + + while(url) { + char *value = mystrsep(&url, "?&[]"); + if(!value || !*value) continue; + + char *name = mystrsep(&value, "="); + if(!name || !*name) continue; + if(!value || !*value) continue; + + // name and value are now the parameters + // they are not null and not empty + + if(!strcmp(name, "chart")) chart = value; + //else { + /// buffer_sprintf(w->response.data, "Unknown parameter '%s' in request.", name); + // goto cleanup; + //} + } + + if(!chart || !*chart) { + buffer_sprintf(w->response.data, "No chart id is given at the request."); + goto cleanup; + } + + RRDSET *st = rrdset_find(chart); + if(!st) st = rrdset_find_byname(chart); + if(!st) { + buffer_sprintf(w->response.data, "Chart '%s' is not found.", chart); + ret = 404; + goto cleanup; + } + + w->response.data->contenttype = CT_APPLICATION_JSON; + rrd_stats_api_v1_chart(st, w->response.data); + return 200; cleanup: - return ret; + return ret; +} + +int web_client_api_request_v1_badge(struct web_client *w, char *url) { + int ret = 400; + buffer_flush(w->response.data); + + BUFFER *dimensions = NULL; + + const char *chart = NULL + , *before_str = NULL + , *after_str = NULL + , *points_str = NULL + , *multiply_str = NULL + , *divide_str = NULL + , *label = NULL + , *units = NULL + , *label_color = NULL + , *value_color = NULL + , *refresh_str = NULL + , *precision_str = NULL + , *alarm = NULL; + + int group = GROUP_AVERAGE; + uint32_t options = 0x00000000; + + while(url) { + char *value = mystrsep(&url, "/?&[]"); + if(!value || !*value) continue; + + char *name = mystrsep(&value, "="); + if(!name || !*name) continue; + if(!value || !*value) continue; + + debug(D_WEB_CLIENT, "%llu: API v1 badge.svg query param '%s' with value '%s'", w->id, name, value); + + // name and value are now the parameters + // they are not null and not empty + + if(!strcmp(name, "chart")) chart = value; + else if(!strcmp(name, "dimension") || !strcmp(name, "dim") || !strcmp(name, "dimensions") || !strcmp(name, "dims")) { + if(!dimensions) + dimensions = buffer_create(100); + + buffer_strcat(dimensions, "|"); + buffer_strcat(dimensions, value); + } + else if(!strcmp(name, "after")) after_str = value; + else if(!strcmp(name, "before")) before_str = value; + else if(!strcmp(name, "points")) points_str = value; + else if(!strcmp(name, "group")) { + group = web_client_api_request_v1_data_group(value, GROUP_AVERAGE); + } + else if(!strcmp(name, "options")) { + options |= web_client_api_request_v1_data_options(value); + } + else if(!strcmp(name, "label")) label = value; + else if(!strcmp(name, "units")) units = value; + else if(!strcmp(name, "label_color")) label_color = value; + else if(!strcmp(name, "value_color")) value_color = value; + else if(!strcmp(name, "multiply")) multiply_str = value; + else if(!strcmp(name, "divide")) divide_str = value; + else if(!strcmp(name, "refresh")) refresh_str = value; + else if(!strcmp(name, "precision")) precision_str = value; + else if(!strcmp(name, "alarm")) alarm = value; + } + + if(!chart || !*chart) { + buffer_sprintf(w->response.data, "No chart id is given at the request."); + goto cleanup; + } + + RRDSET *st = rrdset_find(chart); + if(!st) st = rrdset_find_byname(chart); + if(!st) { + buffer_svg(w->response.data, "chart not found", 0, "", NULL, NULL, 1, -1); + ret = 200; + goto cleanup; + } + + RRDCALC *rc = NULL; + if(alarm) { + rc = rrdcalc_find(st, alarm); + if (!rc) { + buffer_svg(w->response.data, "alarm not found", 0, "", NULL, NULL, 1, -1); + ret = 200; + goto cleanup; + } + } + + long long multiply = (multiply_str && *multiply_str )?atol(multiply_str):1; + long long divide = (divide_str && *divide_str )?atol(divide_str):1; + long long before = (before_str && *before_str )?atol(before_str):0; + long long after = (after_str && *after_str )?atol(after_str):-st->update_every; + int points = (points_str && *points_str )?atoi(points_str):1; + int precision = (precision_str && *precision_str)?atoi(precision_str):-1; + + if(!multiply) multiply = 1; + if(!divide) divide = 1; + + int refresh = 0; + if(refresh_str && *refresh_str) { + if(!strcmp(refresh_str, "auto")) { + if(rc) refresh = rc->update_every; + else if(options & RRDR_OPTION_NOT_ALIGNED) + refresh = st->update_every; + else { + refresh = (before - after); + if(refresh < 0) refresh = -refresh; + } + } + else { + refresh = atoi(refresh_str); + if(refresh < 0) refresh = -refresh; + } + } + + if(!label) { + if(alarm) { + char *s = (char *)alarm; + while(*s) { + if(*s == '_') *s = ' '; + s++; + } + label = alarm; + } + else if(dimensions) { + const char *dim = buffer_tostring(dimensions); + if(*dim == '|') dim++; + label = dim; + } + else + label = st->name; + } + if(!units) { + if(alarm) { + if(rc->units) + units = rc->units; + else + units = ""; + } + else if(options & RRDR_OPTION_PERCENTAGE) + units = "%"; + else + units = st->units; + } + + debug(D_WEB_CLIENT, "%llu: API command 'badge.svg' for chart '%s', alarm '%s', dimensions '%s', after '%lld', before '%lld', points '%d', group '%d', options '0x%08x'" + , w->id + , chart + , alarm?alarm:"" + , (dimensions)?buffer_tostring(dimensions):"" + , after + , before + , points + , group + , options + ); + + if(rc) { + calculated_number n = rc->value; + if(isnan(n) || isinf(n)) n = 0; + + if (refresh > 0) + buffer_sprintf(w->response.header, "Refresh: %d\r\n", refresh); + + if(!value_color) { + switch(rc->status) { + case RRDCALC_STATUS_CRITICAL: + value_color = "red"; + break; + + case RRDCALC_STATUS_WARNING: + value_color = "orange"; + break; + + case RRDCALC_STATUS_CLEAR: + value_color = "brightgreen"; + break; + + case RRDCALC_STATUS_UNDEFINED: + value_color = "lightgrey"; + break; + + case RRDCALC_STATUS_UNINITIALIZED: + value_color = "#000"; + break; + + default: + value_color = "grey"; + break; + } + } + + buffer_svg(w->response.data, label, rc->value * multiply / divide, units, label_color, value_color, 0, precision); + ret = 200; + } + else { + time_t latest_timestamp = 0; + int value_is_null = 1; + calculated_number n = 0; + ret = 500; + + // if the collected value is too old, don't calculate its value + if (rrdset_last_entry_t(st) >= (time(NULL) - (st->update_every * st->gap_when_lost_iterations_above))) + ret = rrd2value(st, w->response.data, &n, (dimensions) ? buffer_tostring(dimensions) : NULL, points, after, + before, group, options, NULL, &latest_timestamp, &value_is_null); + + // if the value cannot be calculated, show empty badge + if (ret != 200) { + value_is_null = 1; + n = 0; + ret = 200; + } + else if (refresh > 0) + buffer_sprintf(w->response.header, "Refresh: %d\r\n", refresh); + + // render the badge + buffer_svg(w->response.data, label, n * multiply / divide, units, label_color, value_color, value_is_null, + precision); + } + +cleanup: + if(dimensions) + buffer_free(dimensions); + return ret; } // returns the HTTP code int web_client_api_request_v1_data(struct web_client *w, char *url) { - debug(D_WEB_CLIENT, "%llu: API v1 data with URL '%s'", w->id, url); - - int ret = 400; - BUFFER *dimensions = NULL; - - buffer_flush(w->response.data); - - char *google_version = "0.6", - *google_reqId = "0", - *google_sig = "0", - *google_out = "json", - *responseHandler = NULL, - *outFileName = NULL; - - time_t last_timestamp_in_data = 0, google_timestamp = 0; - - char *chart = NULL - , *before_str = NULL - , *after_str = NULL - , *points_str = NULL; - - int group = GROUP_MAX; - uint32_t format = DATASOURCE_JSON; - uint32_t options = 0x00000000; - - while(url) { - char *value = mystrsep(&url, "?&[]"); - if(!value || !*value) continue; - - char *name = mystrsep(&value, "="); - if(!name || !*name) continue; - if(!value || !*value) continue; - - debug(D_WEB_CLIENT, "%llu: API v1 data query param '%s' with value '%s'", w->id, name, value); - - // name and value are now the parameters - // they are not null and not empty - - if(!strcmp(name, "chart")) chart = value; - else if(!strcmp(name, "dimension") || !strcmp(name, "dim") || !strcmp(name, "dimensions") || !strcmp(name, "dims")) { - if(!dimensions) dimensions = buffer_create(strlen(value)); - if(dimensions) { - buffer_strcat(dimensions, "|"); - buffer_strcat(dimensions, value); - } - } - else if(!strcmp(name, "after")) after_str = value; - else if(!strcmp(name, "before")) before_str = value; - else if(!strcmp(name, "points")) points_str = value; - else if(!strcmp(name, "group")) { - group = web_client_api_request_v1_data_group(value); - } - else if(!strcmp(name, "format")) { - format = web_client_api_request_v1_data_format(value); - } - else if(!strcmp(name, "options")) { - options |= web_client_api_request_v1_data_options(value); - } - else if(!strcmp(name, "callback")) { - responseHandler = value; - } - else if(!strcmp(name, "filename")) { - outFileName = value; - } - else if(!strcmp(name, "tqx")) { - // parse Google Visualization API options - // https://developers.google.com/chart/interactive/docs/dev/implementing_data_source - char *tqx_name, *tqx_value; - - while(value) { - tqx_value = mystrsep(&value, ";"); - if(!tqx_value || !*tqx_value) continue; - - tqx_name = mystrsep(&tqx_value, ":"); - if(!tqx_name || !*tqx_name) continue; - if(!tqx_value || !*tqx_value) continue; - - if(!strcmp(tqx_name, "version")) - google_version = tqx_value; - else if(!strcmp(tqx_name, "reqId")) - google_reqId = tqx_value; - else if(!strcmp(tqx_name, "sig")) { - google_sig = tqx_value; - google_timestamp = strtoul(google_sig, NULL, 0); - } - else if(!strcmp(tqx_name, "out")) { - google_out = tqx_value; - format = web_client_api_request_v1_data_google_format(google_out); - } - else if(!strcmp(tqx_name, "responseHandler")) - responseHandler = tqx_value; - else if(!strcmp(tqx_name, "outFileName")) - outFileName = tqx_value; - } - } - } - - if(!chart || !*chart) { - buffer_sprintf(w->response.data, "No chart id is given at the request."); - goto cleanup; - } - - RRDSET *st = rrdset_find(chart); - if(!st) st = rrdset_find_byname(chart); - if(!st) { - buffer_sprintf(w->response.data, "Chart '%s' is not found.", chart); - ret = 404; - goto cleanup; - } - - long long before = (before_str && *before_str)?atol(before_str):0; - long long after = (after_str && *after_str) ?atol(after_str):0; - int points = (points_str && *points_str)?atoi(points_str):0; - - debug(D_WEB_CLIENT, "%llu: API command 'data' for chart '%s', dimensions '%s', after '%lld', before '%lld', points '%d', group '%u', format '%u', options '0x%08x'" - , w->id - , chart - , (dimensions)?buffer_tostring(dimensions):"" - , after - , before - , points - , group - , format - , options - ); - - if(outFileName && *outFileName) { - buffer_sprintf(w->response.header, "Content-Disposition: attachment; filename=\"%s\"\r\n", outFileName); - error("generating outfilename header: '%s'", outFileName); - } - - if(format == DATASOURCE_DATATABLE_JSONP) { - if(responseHandler == NULL) - responseHandler = "google.visualization.Query.setResponse"; - - debug(D_WEB_CLIENT_ACCESS, "%llu: GOOGLE JSON/JSONP: version = '%s', reqId = '%s', sig = '%s', out = '%s', responseHandler = '%s', outFileName = '%s'", - w->id, google_version, google_reqId, google_sig, google_out, responseHandler, outFileName - ); - - buffer_sprintf(w->response.data, - "%s({version:'%s',reqId:'%s',status:'ok',sig:'%lu',table:", - responseHandler, google_version, google_reqId, st->last_updated.tv_sec); - } - else if(format == DATASOURCE_JSONP) { - if(responseHandler == NULL) - responseHandler = "callback"; - - buffer_strcat(w->response.data, responseHandler); - buffer_strcat(w->response.data, "("); - } - - ret = rrd2format(st, w->response.data, dimensions, format, points, after, before, group, options, &last_timestamp_in_data); - - if(format == DATASOURCE_DATATABLE_JSONP) { - if(google_timestamp < last_timestamp_in_data) - buffer_strcat(w->response.data, "});"); - - else { - // the client already has the latest data - buffer_flush(w->response.data); - buffer_sprintf(w->response.data, - "%s({version:'%s',reqId:'%s',status:'error',errors:[{reason:'not_modified',message:'Data not modified'}]});", - responseHandler, google_version, google_reqId); - } - } - else if(format == DATASOURCE_JSONP) - buffer_strcat(w->response.data, ");"); + debug(D_WEB_CLIENT, "%llu: API v1 data with URL '%s'", w->id, url); + + int ret = 400; + BUFFER *dimensions = NULL; + + buffer_flush(w->response.data); + + char *google_version = "0.6", + *google_reqId = "0", + *google_sig = "0", + *google_out = "json", + *responseHandler = NULL, + *outFileName = NULL; + + time_t last_timestamp_in_data = 0, google_timestamp = 0; + + char *chart = NULL + , *before_str = NULL + , *after_str = NULL + , *points_str = NULL; + + int group = GROUP_AVERAGE; + uint32_t format = DATASOURCE_JSON; + uint32_t options = 0x00000000; + + while(url) { + char *value = mystrsep(&url, "?&[]"); + if(!value || !*value) continue; + + char *name = mystrsep(&value, "="); + if(!name || !*name) continue; + if(!value || !*value) continue; + + debug(D_WEB_CLIENT, "%llu: API v1 data query param '%s' with value '%s'", w->id, name, value); + + // name and value are now the parameters + // they are not null and not empty + + if(!strcmp(name, "chart")) chart = value; + else if(!strcmp(name, "dimension") || !strcmp(name, "dim") || !strcmp(name, "dimensions") || !strcmp(name, "dims")) { + if(!dimensions) dimensions = buffer_create(100); + buffer_strcat(dimensions, "|"); + buffer_strcat(dimensions, value); + } + else if(!strcmp(name, "after")) after_str = value; + else if(!strcmp(name, "before")) before_str = value; + else if(!strcmp(name, "points")) points_str = value; + else if(!strcmp(name, "group")) { + group = web_client_api_request_v1_data_group(value, GROUP_AVERAGE); + } + else if(!strcmp(name, "format")) { + format = web_client_api_request_v1_data_format(value); + } + else if(!strcmp(name, "options")) { + options |= web_client_api_request_v1_data_options(value); + } + else if(!strcmp(name, "callback")) { + responseHandler = value; + } + else if(!strcmp(name, "filename")) { + outFileName = value; + } + else if(!strcmp(name, "tqx")) { + // parse Google Visualization API options + // https://developers.google.com/chart/interactive/docs/dev/implementing_data_source + char *tqx_name, *tqx_value; + + while(value) { + tqx_value = mystrsep(&value, ";"); + if(!tqx_value || !*tqx_value) continue; + + tqx_name = mystrsep(&tqx_value, ":"); + if(!tqx_name || !*tqx_name) continue; + if(!tqx_value || !*tqx_value) continue; + + if(!strcmp(tqx_name, "version")) + google_version = tqx_value; + else if(!strcmp(tqx_name, "reqId")) + google_reqId = tqx_value; + else if(!strcmp(tqx_name, "sig")) { + google_sig = tqx_value; + google_timestamp = strtoul(google_sig, NULL, 0); + } + else if(!strcmp(tqx_name, "out")) { + google_out = tqx_value; + format = web_client_api_request_v1_data_google_format(google_out); + } + else if(!strcmp(tqx_name, "responseHandler")) + responseHandler = tqx_value; + else if(!strcmp(tqx_name, "outFileName")) + outFileName = tqx_value; + } + } + } + + if(!chart || !*chart) { + buffer_sprintf(w->response.data, "No chart id is given at the request."); + goto cleanup; + } + + RRDSET *st = rrdset_find(chart); + if(!st) st = rrdset_find_byname(chart); + if(!st) { + buffer_sprintf(w->response.data, "Chart '%s' is not found.", chart); + ret = 404; + goto cleanup; + } + + long long before = (before_str && *before_str)?atol(before_str):0; + long long after = (after_str && *after_str) ?atol(after_str):0; + int points = (points_str && *points_str)?atoi(points_str):0; + + debug(D_WEB_CLIENT, "%llu: API command 'data' for chart '%s', dimensions '%s', after '%lld', before '%lld', points '%d', group '%d', format '%u', options '0x%08x'" + , w->id + , chart + , (dimensions)?buffer_tostring(dimensions):"" + , after + , before + , points + , group + , format + , options + ); + + if(outFileName && *outFileName) { + buffer_sprintf(w->response.header, "Content-Disposition: attachment; filename=\"%s\"\r\n", outFileName); + debug(D_WEB_CLIENT, "%llu: generating outfilename header: '%s'", w->id, outFileName); + } + + if(format == DATASOURCE_DATATABLE_JSONP) { + if(responseHandler == NULL) + responseHandler = "google.visualization.Query.setResponse"; + + debug(D_WEB_CLIENT_ACCESS, "%llu: GOOGLE JSON/JSONP: version = '%s', reqId = '%s', sig = '%s', out = '%s', responseHandler = '%s', outFileName = '%s'", + w->id, google_version, google_reqId, google_sig, google_out, responseHandler, outFileName + ); + + buffer_sprintf(w->response.data, + "%s({version:'%s',reqId:'%s',status:'ok',sig:'%ld',table:", + responseHandler, google_version, google_reqId, st->last_updated.tv_sec); + } + else if(format == DATASOURCE_JSONP) { + if(responseHandler == NULL) + responseHandler = "callback"; + + buffer_strcat(w->response.data, responseHandler); + buffer_strcat(w->response.data, "("); + } + + ret = rrd2format(st, w->response.data, dimensions, format, points, after, before, group, options, &last_timestamp_in_data); + + if(format == DATASOURCE_DATATABLE_JSONP) { + if(google_timestamp < last_timestamp_in_data) + buffer_strcat(w->response.data, "});"); + + else { + // the client already has the latest data + buffer_flush(w->response.data); + buffer_sprintf(w->response.data, + "%s({version:'%s',reqId:'%s',status:'error',errors:[{reason:'not_modified',message:'Data not modified'}]});", + responseHandler, google_version, google_reqId); + } + } + else if(format == DATASOURCE_JSONP) + buffer_strcat(w->response.data, ");"); cleanup: - if(dimensions) buffer_free(dimensions); - return ret; + if(dimensions) buffer_free(dimensions); + return ret; } int web_client_api_request_v1_registry(struct web_client *w, char *url) { - char person_guid[36 + 1] = ""; - - debug(D_WEB_CLIENT, "%llu: API v1 registry with URL '%s'", w->id, url); - - // FIXME - // The browser may send multiple cookies with our id - - char *cookie = strstr(w->response.data->buffer, " " NETDATA_REGISTRY_COOKIE_NAME "="); - if(cookie) - strncpyz(person_guid, &cookie[sizeof(NETDATA_REGISTRY_COOKIE_NAME) + 1], 36); - - char action = '\0'; - char *machine_guid = NULL, - *machine_url = NULL, - *url_name = NULL, - *search_machine_guid = NULL, - *delete_url = NULL, - *to_person_guid = NULL; - - while(url) { - char *value = mystrsep(&url, "?&[]"); - if (!value || !*value) continue; - - char *name = mystrsep(&value, "="); - if (!name || !*name) continue; - if (!value || !*value) continue; - - debug(D_WEB_CLIENT, "%llu: API v1 registry query param '%s' with value '%s'", w->id, name, value); - - if(!strcmp(name, "action")) { - if(!strcmp(value, "access")) action = 'A'; - else if(!strcmp(value, "hello")) action = 'H'; - else if(!strcmp(value, "delete")) action = 'D'; - else if(!strcmp(value, "search")) action = 'S'; - else if(!strcmp(value, "switch")) action = 'W'; - } - else if(!strcmp(name, "machine")) - machine_guid = value; - - else if(!strcmp(name, "url")) - machine_url = value; - - else if(action == 'A') { - if(!strcmp(name, "name")) - url_name = value; - } - else if(action == 'D') { - if(!strcmp(name, "delete_url")) - delete_url = value; - } - else if(action == 'S') { - if(!strcmp(name, "for")) - search_machine_guid = value; - } - else if(action == 'W') { - if(!strcmp(name, "to")) - to_person_guid = value; - } - } - - if(action == 'A' && (!machine_guid || !machine_url || !url_name)) { - buffer_flush(w->response.data); - buffer_sprintf(w->response.data, "Invalid registry request - access requires these parameters: machine ('%s'), url ('%s'), name ('%s')", - machine_guid?machine_guid:"UNSET", machine_url?machine_url:"UNSET", url_name?url_name:"UNSET"); - return 400; - } - else if(action == 'D' && (!machine_guid || !machine_url || !delete_url)) { - buffer_flush(w->response.data); - buffer_sprintf(w->response.data, "Invalid registry request - delete requires these parameters: machine ('%s'), url ('%s'), delete_url ('%s')", - machine_guid?machine_guid:"UNSET", machine_url?machine_url:"UNSET", delete_url?delete_url:"UNSET"); - return 400; - } - else if(action == 'S' && (!machine_guid || !machine_url || !search_machine_guid)) { - buffer_flush(w->response.data); - buffer_sprintf(w->response.data, "Invalid registry request - search requires these parameters: machine ('%s'), url ('%s'), for ('%s')", - machine_guid?machine_guid:"UNSET", machine_url?machine_url:"UNSET", search_machine_guid?search_machine_guid:"UNSET"); - return 400; - } - else if(action == 'W' && (!machine_guid || !machine_url || !to_person_guid)) { - buffer_flush(w->response.data); - buffer_sprintf(w->response.data, "Invalid registry request - switching identity requires these parameters: machine ('%s'), url ('%s'), to ('%s')", - machine_guid?machine_guid:"UNSET", machine_url?machine_url:"UNSET", to_person_guid?to_person_guid:"UNSET"); - return 400; - } - - switch(action) { - case 'A': - return registry_request_access_json(w, person_guid, machine_guid, machine_url, url_name, time(NULL)); - - case 'D': - return registry_request_delete_json(w, person_guid, machine_guid, machine_url, delete_url, time(NULL)); - - case 'S': - return registry_request_search_json(w, person_guid, machine_guid, machine_url, search_machine_guid, time(NULL)); - - case 'W': - return registry_request_switch_json(w, person_guid, machine_guid, machine_url, to_person_guid, time(NULL)); - - case 'H': - return registry_request_hello_json(w); - - default: - buffer_flush(w->response.data); - buffer_sprintf(w->response.data, "Invalid registry request - you need to set an action: hello, access, delete, search"); - return 400; - } - - buffer_flush(w->response.data); - buffer_sprintf(w->response.data, "Invalid or no registry action."); - return 400; + static uint32_t hash_action = 0, hash_access = 0, hash_hello = 0, hash_delete = 0, hash_search = 0, + hash_switch = 0, hash_machine = 0, hash_url = 0, hash_name = 0, hash_delete_url = 0, hash_for = 0, + hash_to = 0 /*, hash_redirects = 0 */; + + if(unlikely(!hash_action)) { + hash_action = simple_hash("action"); + hash_access = simple_hash("access"); + hash_hello = simple_hash("hello"); + hash_delete = simple_hash("delete"); + hash_search = simple_hash("search"); + hash_switch = simple_hash("switch"); + hash_machine = simple_hash("machine"); + hash_url = simple_hash("url"); + hash_name = simple_hash("name"); + hash_delete_url = simple_hash("delete_url"); + hash_for = simple_hash("for"); + hash_to = simple_hash("to"); +/* + hash_redirects = simple_hash("redirects"); +*/ + } + + char person_guid[36 + 1] = ""; + + debug(D_WEB_CLIENT, "%llu: API v1 registry with URL '%s'", w->id, url); + + // FIXME + // The browser may send multiple cookies with our id + + char *cookie = strstr(w->response.data->buffer, NETDATA_REGISTRY_COOKIE_NAME "="); + if(cookie) + strncpyz(person_guid, &cookie[sizeof(NETDATA_REGISTRY_COOKIE_NAME)], 36); + + char action = '\0'; + char *machine_guid = NULL, + *machine_url = NULL, + *url_name = NULL, + *search_machine_guid = NULL, + *delete_url = NULL, + *to_person_guid = NULL; +/* + int redirects = 0; +*/ + + while(url) { + char *value = mystrsep(&url, "?&[]"); + if (!value || !*value) continue; + + char *name = mystrsep(&value, "="); + if (!name || !*name) continue; + if (!value || !*value) continue; + + debug(D_WEB_CLIENT, "%llu: API v1 registry query param '%s' with value '%s'", w->id, name, value); + + uint32_t hash = simple_hash(name); + + if(hash == hash_action && !strcmp(name, "action")) { + uint32_t vhash = simple_hash(value); + + if(vhash == hash_access && !strcmp(value, "access")) action = 'A'; + else if(vhash == hash_hello && !strcmp(value, "hello")) action = 'H'; + else if(vhash == hash_delete && !strcmp(value, "delete")) action = 'D'; + else if(vhash == hash_search && !strcmp(value, "search")) action = 'S'; + else if(vhash == hash_switch && !strcmp(value, "switch")) action = 'W'; +#ifdef NETDATA_INTERNAL_CHECKS + else error("unknown registry action '%s'", value); +#endif /* NETDATA_INTERNAL_CHECKS */ + } +/* + else if(hash == hash_redirects && !strcmp(name, "redirects")) + redirects = atoi(value); +*/ + else if(hash == hash_machine && !strcmp(name, "machine")) + machine_guid = value; + + else if(hash == hash_url && !strcmp(name, "url")) + machine_url = value; + + else if(action == 'A') { + if(hash == hash_name && !strcmp(name, "name")) + url_name = value; + } + else if(action == 'D') { + if(hash == hash_delete_url && !strcmp(name, "delete_url")) + delete_url = value; + } + else if(action == 'S') { + if(hash == hash_for && !strcmp(name, "for")) + search_machine_guid = value; + } + else if(action == 'W') { + if(hash == hash_to && !strcmp(name, "to")) + to_person_guid = value; + } +#ifdef NETDATA_INTERNAL_CHECKS + else error("unused registry URL parameter '%s' with value '%s'", name, value); +#endif /* NETDATA_INTERNAL_CHECKS */ + } + + if(web_donotrack_comply && w->donottrack) { + buffer_flush(w->response.data); + buffer_sprintf(w->response.data, "Your web browser is sending 'DNT: 1' (Do Not Track). The registry requires persistent cookies on your browser to work."); + return 400; + } + + if(action == 'A' && (!machine_guid || !machine_url || !url_name)) { + buffer_flush(w->response.data); + buffer_sprintf(w->response.data, "Invalid registry request - access requires these parameters: machine ('%s'), url ('%s'), name ('%s')", + machine_guid?machine_guid:"UNSET", machine_url?machine_url:"UNSET", url_name?url_name:"UNSET"); + return 400; + } + else if(action == 'D' && (!machine_guid || !machine_url || !delete_url)) { + buffer_flush(w->response.data); + buffer_sprintf(w->response.data, "Invalid registry request - delete requires these parameters: machine ('%s'), url ('%s'), delete_url ('%s')", + machine_guid?machine_guid:"UNSET", machine_url?machine_url:"UNSET", delete_url?delete_url:"UNSET"); + return 400; + } + else if(action == 'S' && (!machine_guid || !machine_url || !search_machine_guid)) { + buffer_flush(w->response.data); + buffer_sprintf(w->response.data, "Invalid registry request - search requires these parameters: machine ('%s'), url ('%s'), for ('%s')", + machine_guid?machine_guid:"UNSET", machine_url?machine_url:"UNSET", search_machine_guid?search_machine_guid:"UNSET"); + return 400; + } + else if(action == 'W' && (!machine_guid || !machine_url || !to_person_guid)) { + buffer_flush(w->response.data); + buffer_sprintf(w->response.data, "Invalid registry request - switching identity requires these parameters: machine ('%s'), url ('%s'), to ('%s')", + machine_guid?machine_guid:"UNSET", machine_url?machine_url:"UNSET", to_person_guid?to_person_guid:"UNSET"); + return 400; + } + + switch(action) { + case 'A': + w->tracking_required = 1; + if(registry_verify_cookies_redirects() > 0 && (!cookie || !person_guid[0])) { + buffer_flush(w->response.data); + + registry_set_cookie(w, "give-me-back-this-cookie-please"); + w->response.data->contenttype = CT_APPLICATION_JSON; + buffer_sprintf(w->response.data, "{ \"status\": \"redirect\", \"registry\": \"%s\" }", registry_to_announce()); + return 200; + +/* + * it seems that web browsers are ignoring 307 (Moved Temporarily) + * under certain conditions, when using CORS + * so this is commented and we use application level redirects instead + * + redirects++; + + if(redirects > registry_verify_cookies_redirects()) { + buffer_flush(w->response.data); + buffer_sprintf(w->response.data, "Your browser does not support cookies"); + return 400; + } + + char *encoded_url = url_encode(machine_url); + if(!encoded_url) { + error("%llu: Cannot URL encode string '%s'", w->id, machine_url); + return 500; + } + + char *encoded_name = url_encode(url_name); + if(!encoded_name) { + free(encoded_url); + error("%llu: Cannot URL encode string '%s'", w->id, url_name); + return 500; + } + + char *encoded_guid = url_encode(machine_guid); + if(!encoded_guid) { + free(encoded_url); + free(encoded_name); + error("%llu: Cannot URL encode string '%s'", w->id, machine_guid); + return 500; + } + + buffer_sprintf(w->response.header, "Location: %s/api/v1/registry?action=access&machine=%s&name=%s&url=%s&redirects=%d\r\n", + registry_to_announce(), encoded_guid, encoded_name, encoded_url, redirects); + + free(encoded_guid); + free(encoded_name); + free(encoded_url); + return 307 +*/ + } + return registry_request_access_json(w, person_guid, machine_guid, machine_url, url_name, time(NULL)); + + case 'D': + w->tracking_required = 1; + return registry_request_delete_json(w, person_guid, machine_guid, machine_url, delete_url, time(NULL)); + + case 'S': + w->tracking_required = 1; + return registry_request_search_json(w, person_guid, machine_guid, machine_url, search_machine_guid, time(NULL)); + + case 'W': + w->tracking_required = 1; + return registry_request_switch_json(w, person_guid, machine_guid, machine_url, to_person_guid, time(NULL)); + + case 'H': + return registry_request_hello_json(w); + + default: + buffer_flush(w->response.data); + buffer_strcat(w->response.data, "Invalid registry request - you need to set an action: hello, access, delete, search"); + return 400; + } } -int web_client_api_request_v1(struct web_client *w, char *url) -{ - static uint32_t data_hash = 0, chart_hash = 0, charts_hash = 0, registry_hash = 0; - - if(unlikely(data_hash == 0)) { - data_hash = simple_hash("data"); - chart_hash = simple_hash("chart"); - charts_hash = simple_hash("charts"); - registry_hash = simple_hash("registry"); - } - - // get the command - char *tok = mystrsep(&url, "/?&"); - if(tok && *tok) { - debug(D_WEB_CLIENT, "%llu: Searching for API v1 command '%s'.", w->id, tok); - uint32_t hash = simple_hash(tok); - - if(hash == data_hash && !strcmp(tok, "data")) - return web_client_api_request_v1_data(w, url); - - else if(hash == chart_hash && !strcmp(tok, "chart")) - return web_client_api_request_v1_chart(w, url); - - else if(hash == charts_hash && !strcmp(tok, "charts")) - return web_client_api_request_v1_charts(w, url); - - else if(hash == registry_hash && !strcmp(tok, "registry")) - return web_client_api_request_v1_registry(w, url); - - else { - buffer_flush(w->response.data); - buffer_sprintf(w->response.data, "Unsupported v1 API command: %s", tok); - return 404; - } - } - else { - buffer_flush(w->response.data); - buffer_sprintf(w->response.data, "API v1 command?"); - return 400; - } +int web_client_api_request_v1(struct web_client *w, char *url) { + static uint32_t hash_data = 0, hash_chart = 0, hash_charts = 0, hash_registry = 0, hash_badge = 0, hash_alarms = 0, hash_alarm_log = 0; + + if(unlikely(hash_data == 0)) { + hash_data = simple_hash("data"); + hash_chart = simple_hash("chart"); + hash_charts = simple_hash("charts"); + hash_registry = simple_hash("registry"); + hash_badge = simple_hash("badge.svg"); + hash_alarms = simple_hash("alarms"); + hash_alarm_log = simple_hash("alarm_log"); + } + + // get the command + char *tok = mystrsep(&url, "/?&"); + if(tok && *tok) { + debug(D_WEB_CLIENT, "%llu: Searching for API v1 command '%s'.", w->id, tok); + uint32_t hash = simple_hash(tok); + + if(hash == hash_data && !strcmp(tok, "data")) + return web_client_api_request_v1_data(w, url); + + else if(hash == hash_chart && !strcmp(tok, "chart")) + return web_client_api_request_v1_chart(w, url); + + else if(hash == hash_charts && !strcmp(tok, "charts")) + return web_client_api_request_v1_charts(w, url); + + else if(hash == hash_registry && !strcmp(tok, "registry")) + return web_client_api_request_v1_registry(w, url); + + else if(hash == hash_badge && !strcmp(tok, "badge.svg")) + return web_client_api_request_v1_badge(w, url); + + else if(hash == hash_alarms && !strcmp(tok, "alarms")) + return web_client_api_request_v1_alarms(w, url); + + else if(hash == hash_alarm_log && !strcmp(tok, "alarm_log")) + return web_client_api_request_v1_alarm_log(w, url); + + else { + buffer_flush(w->response.data); + buffer_sprintf(w->response.data, "Unsupported v1 API command: %s", tok); + return 404; + } + } + else { + buffer_flush(w->response.data); + buffer_sprintf(w->response.data, "API v1 command?"); + return 400; + } } int web_client_api_request(struct web_client *w, char *url) { - // get the api version - char *tok = mystrsep(&url, "/?&"); - if(tok && *tok) { - debug(D_WEB_CLIENT, "%llu: Searching for API version '%s'.", w->id, tok); - if(strcmp(tok, "v1") == 0) - return web_client_api_request_v1(w, url); - else { - buffer_flush(w->response.data); - buffer_sprintf(w->response.data, "Unsupported API version: %s", tok); - return 404; - } - } - else { - buffer_flush(w->response.data); - buffer_sprintf(w->response.data, "Which API version?"); - return 400; - } + // get the api version + char *tok = mystrsep(&url, "/?&"); + if(tok && *tok) { + debug(D_WEB_CLIENT, "%llu: Searching for API version '%s'.", w->id, tok); + if(strcmp(tok, "v1") == 0) + return web_client_api_request_v1(w, url); + else { + buffer_flush(w->response.data); + buffer_sprintf(w->response.data, "Unsupported API version: %s", tok); + return 404; + } + } + else { + buffer_flush(w->response.data); + buffer_sprintf(w->response.data, "Which API version?"); + return 400; + } } -int web_client_data_request(struct web_client *w, char *url, int datasource_type) +int web_client_api_old_data_request(struct web_client *w, char *url, int datasource_type) { - RRDSET *st = NULL; - - char *args = strchr(url, '?'); - if(args) { - *args='\0'; - args = &args[1]; - } - - // get the name of the data to show - char *tok = mystrsep(&url, "/"); - - // do we have such a data set? - if(tok && *tok) { - debug(D_WEB_CLIENT, "%llu: Searching for RRD data with name '%s'.", w->id, tok); - st = rrdset_find_byname(tok); - if(!st) st = rrdset_find(tok); - } - - if(!st) { - // we don't have it - // try to send a file with that name - buffer_flush(w->response.data); - return(mysendfile(w, tok)); - } - - // we have it - debug(D_WEB_CLIENT, "%llu: Found RRD data with name '%s'.", w->id, tok); - - // how many entries does the client want? - long lines = rrd_default_history_entries; - long group_count = 1; - time_t after = 0, before = 0; - int group_method = GROUP_AVERAGE; - int nonzero = 0; - - if(url) { - // parse the lines required - tok = mystrsep(&url, "/"); - if(tok) lines = atoi(tok); - if(lines < 1) lines = 1; - } - if(url) { - // parse the group count required - tok = mystrsep(&url, "/"); - if(tok && *tok) group_count = atoi(tok); - if(group_count < 1) group_count = 1; - //if(group_count > save_history / 20) group_count = save_history / 20; - } - if(url) { - // parse the grouping method required - tok = mystrsep(&url, "/"); - if(tok && *tok) { - if(strcmp(tok, "max") == 0) group_method = GROUP_MAX; - else if(strcmp(tok, "average") == 0) group_method = GROUP_AVERAGE; - else if(strcmp(tok, "sum") == 0) group_method = GROUP_SUM; - else debug(D_WEB_CLIENT, "%llu: Unknown group method '%s'", w->id, tok); - } - } - if(url) { - // parse after time - tok = mystrsep(&url, "/"); - if(tok && *tok) after = strtoul(tok, NULL, 10); - if(after < 0) after = 0; - } - if(url) { - // parse before time - tok = mystrsep(&url, "/"); - if(tok && *tok) before = strtoul(tok, NULL, 10); - if(before < 0) before = 0; - } - if(url) { - // parse nonzero - tok = mystrsep(&url, "/"); - if(tok && *tok && strcmp(tok, "nonzero") == 0) nonzero = 1; - } - - w->response.data->contenttype = CT_APPLICATION_JSON; - buffer_flush(w->response.data); - - char *google_version = "0.6"; - char *google_reqId = "0"; - char *google_sig = "0"; - char *google_out = "json"; - char *google_responseHandler = "google.visualization.Query.setResponse"; - char *google_outFileName = NULL; - time_t last_timestamp_in_data = 0; - if(datasource_type == DATASOURCE_DATATABLE_JSON || datasource_type == DATASOURCE_DATATABLE_JSONP) { - - w->response.data->contenttype = CT_APPLICATION_X_JAVASCRIPT; - - while(args) { - tok = mystrsep(&args, "&"); - if(tok && *tok) { - char *name = mystrsep(&tok, "="); - if(name && *name && strcmp(name, "tqx") == 0) { - char *key = mystrsep(&tok, ":"); - char *value = mystrsep(&tok, ";"); - if(key && value && *key && *value) { - if(strcmp(key, "version") == 0) - google_version = value; - - else if(strcmp(key, "reqId") == 0) - google_reqId = value; - - else if(strcmp(key, "sig") == 0) - google_sig = value; - - else if(strcmp(key, "out") == 0) - google_out = value; - - else if(strcmp(key, "responseHandler") == 0) - google_responseHandler = value; - - else if(strcmp(key, "outFileName") == 0) - google_outFileName = value; - } - } - } - } - - debug(D_WEB_CLIENT_ACCESS, "%llu: GOOGLE JSONP: version = '%s', reqId = '%s', sig = '%s', out = '%s', responseHandler = '%s', outFileName = '%s'", - w->id, google_version, google_reqId, google_sig, google_out, google_responseHandler, google_outFileName - ); - - if(datasource_type == DATASOURCE_DATATABLE_JSONP) { - last_timestamp_in_data = strtoul(google_sig, NULL, 0); - - // check the client wants json - if(strcmp(google_out, "json") != 0) { - buffer_sprintf(w->response.data, - "%s({version:'%s',reqId:'%s',status:'error',errors:[{reason:'invalid_query',message:'output format is not supported',detailed_message:'the format %s requested is not supported by netdata.'}]});", - google_responseHandler, google_version, google_reqId, google_out); - return 200; - } - } - } - - if(datasource_type == DATASOURCE_DATATABLE_JSONP) { - buffer_sprintf(w->response.data, - "%s({version:'%s',reqId:'%s',status:'ok',sig:'%lu',table:", - google_responseHandler, google_version, google_reqId, st->last_updated.tv_sec); - } - - debug(D_WEB_CLIENT_ACCESS, "%llu: Sending RRD data '%s' (id %s, %d lines, %d group, %d group_method, %lu after, %lu before).", w->id, st->name, st->id, lines, group_count, group_method, after, before); - time_t timestamp_in_data = rrd_stats_json(datasource_type, st, w->response.data, lines, group_count, group_method, after, before, nonzero); - - if(datasource_type == DATASOURCE_DATATABLE_JSONP) { - if(timestamp_in_data > last_timestamp_in_data) - buffer_strcat(w->response.data, "});"); - - else { - // the client already has the latest data - buffer_flush(w->response.data); - buffer_sprintf(w->response.data, - "%s({version:'%s',reqId:'%s',status:'error',errors:[{reason:'not_modified',message:'Data not modified'}]});", - google_responseHandler, google_version, google_reqId); - } - } - - return 200; + RRDSET *st = NULL; + + char *args = strchr(url, '?'); + if(args) { + *args='\0'; + args = &args[1]; + } + + // get the name of the data to show + char *tok = mystrsep(&url, "/"); + if(!tok) tok = ""; + + // do we have such a data set? + if(*tok) { + debug(D_WEB_CLIENT, "%llu: Searching for RRD data with name '%s'.", w->id, tok); + st = rrdset_find_byname(tok); + if(!st) st = rrdset_find(tok); + } + + if(!st) { + // we don't have it + // try to send a file with that name + buffer_flush(w->response.data); + return(mysendfile(w, tok)); + } + + // we have it + debug(D_WEB_CLIENT, "%llu: Found RRD data with name '%s'.", w->id, tok); + + // how many entries does the client want? + int lines = rrd_default_history_entries; + int group_count = 1; + time_t after = 0, before = 0; + int group_method = GROUP_AVERAGE; + int nonzero = 0; + + if(url) { + // parse the lines required + tok = mystrsep(&url, "/"); + if(tok) lines = atoi(tok); + if(lines < 1) lines = 1; + } + if(url) { + // parse the group count required + tok = mystrsep(&url, "/"); + if(tok && *tok) group_count = atoi(tok); + if(group_count < 1) group_count = 1; + //if(group_count > save_history / 20) group_count = save_history / 20; + } + if(url) { + // parse the grouping method required + tok = mystrsep(&url, "/"); + if(tok && *tok) { + if(strcmp(tok, "max") == 0) group_method = GROUP_MAX; + else if(strcmp(tok, "average") == 0) group_method = GROUP_AVERAGE; + else if(strcmp(tok, "sum") == 0) group_method = GROUP_SUM; + else debug(D_WEB_CLIENT, "%llu: Unknown group method '%s'", w->id, tok); + } + } + if(url) { + // parse after time + tok = mystrsep(&url, "/"); + if(tok && *tok) after = strtoul(tok, NULL, 10); + if(after < 0) after = 0; + } + if(url) { + // parse before time + tok = mystrsep(&url, "/"); + if(tok && *tok) before = strtoul(tok, NULL, 10); + if(before < 0) before = 0; + } + if(url) { + // parse nonzero + tok = mystrsep(&url, "/"); + if(tok && *tok && strcmp(tok, "nonzero") == 0) nonzero = 1; + } + + w->response.data->contenttype = CT_APPLICATION_JSON; + buffer_flush(w->response.data); + + char *google_version = "0.6"; + char *google_reqId = "0"; + char *google_sig = "0"; + char *google_out = "json"; + char *google_responseHandler = "google.visualization.Query.setResponse"; + char *google_outFileName = NULL; + time_t last_timestamp_in_data = 0; + if(datasource_type == DATASOURCE_DATATABLE_JSON || datasource_type == DATASOURCE_DATATABLE_JSONP) { + + w->response.data->contenttype = CT_APPLICATION_X_JAVASCRIPT; + + while(args) { + tok = mystrsep(&args, "&"); + if(tok && *tok) { + char *name = mystrsep(&tok, "="); + if(name && *name && strcmp(name, "tqx") == 0) { + char *key = mystrsep(&tok, ":"); + char *value = mystrsep(&tok, ";"); + if(key && value && *key && *value) { + if(strcmp(key, "version") == 0) + google_version = value; + + else if(strcmp(key, "reqId") == 0) + google_reqId = value; + + else if(strcmp(key, "sig") == 0) + google_sig = value; + + else if(strcmp(key, "out") == 0) + google_out = value; + + else if(strcmp(key, "responseHandler") == 0) + google_responseHandler = value; + + else if(strcmp(key, "outFileName") == 0) + google_outFileName = value; + } + } + } + } + + debug(D_WEB_CLIENT_ACCESS, "%llu: GOOGLE JSONP: version = '%s', reqId = '%s', sig = '%s', out = '%s', responseHandler = '%s', outFileName = '%s'", + w->id, google_version, google_reqId, google_sig, google_out, google_responseHandler, google_outFileName + ); + + if(datasource_type == DATASOURCE_DATATABLE_JSONP) { + last_timestamp_in_data = strtoul(google_sig, NULL, 0); + + // check the client wants json + if(strcmp(google_out, "json") != 0) { + buffer_sprintf(w->response.data, + "%s({version:'%s',reqId:'%s',status:'error',errors:[{reason:'invalid_query',message:'output format is not supported',detailed_message:'the format %s requested is not supported by netdata.'}]});", + google_responseHandler, google_version, google_reqId, google_out); + return 200; + } + } + } + + if(datasource_type == DATASOURCE_DATATABLE_JSONP) { + buffer_sprintf(w->response.data, + "%s({version:'%s',reqId:'%s',status:'ok',sig:'%ld',table:", + google_responseHandler, google_version, google_reqId, st->last_updated.tv_sec); + } + + debug(D_WEB_CLIENT_ACCESS, "%llu: Sending RRD data '%s' (id %s, %d lines, %d group, %d group_method, %ld after, %ld before).", + w->id, st->name, st->id, lines, group_count, group_method, after, before); + + time_t timestamp_in_data = rrd_stats_json(datasource_type, st, w->response.data, lines, group_count, group_method, (unsigned long)after, (unsigned long)before, nonzero); + + if(datasource_type == DATASOURCE_DATATABLE_JSONP) { + if(timestamp_in_data > last_timestamp_in_data) + buffer_strcat(w->response.data, "});"); + + else { + // the client already has the latest data + buffer_flush(w->response.data); + buffer_sprintf(w->response.data, + "%s({version:'%s',reqId:'%s',status:'error',errors:[{reason:'not_modified',message:'Data not modified'}]});", + google_responseHandler, google_version, google_reqId); + } + } + + return 200; } -/* -int web_client_parse_request(struct web_client *w) { - // protocol - // hostname - // path - // query string name-value - // http version - // method - // http request headers name-value +const char *web_content_type_to_string(uint8_t contenttype) { + switch(contenttype) { + case CT_TEXT_HTML: + return "text/html; charset=utf-8"; - web_client_clean_request(w); + case CT_APPLICATION_XML: + return "application/xml; charset=utf-8"; - debug(D_WEB_DATA, "%llu: Processing data buffer of %d bytes: '%s'.", w->id, w->response.data->bytes, w->response.data->buffer); + case CT_APPLICATION_JSON: + return "application/json; charset=utf-8"; - char *buf = w->response.data->buffer; - char *line, *tok; + case CT_APPLICATION_X_JAVASCRIPT: + return "application/x-javascript; charset=utf-8"; - // ------------------------------------------------------------------------ - // the first line + case CT_TEXT_CSS: + return "text/css; charset=utf-8"; - if(buf && (line = strsep(&buf, "\r\n"))) { - // method - if(line && (tok = strsep(&line, " "))) { - w->request.protocol = strdup(tok); - } - else goto cleanup; + case CT_TEXT_XML: + return "text/xml; charset=utf-8"; - // url - } - else goto cleanup; + case CT_TEXT_XSL: + return "text/xsl; charset=utf-8"; - // ------------------------------------------------------------------------ - // the rest of the lines + case CT_APPLICATION_OCTET_STREAM: + return "application/octet-stream"; - while(buf && (line = strsep(&buf, "\r\n"))) { - while(line && (tok = strsep(&line, ": "))) { - } - } + case CT_IMAGE_SVG_XML: + return "image/svg+xml"; - char *url = NULL; + case CT_APPLICATION_X_FONT_TRUETYPE: + return "application/x-font-truetype"; + case CT_APPLICATION_X_FONT_OPENTYPE: + return "application/x-font-opentype"; -cleanup: - web_client_clean_request(w); - return 0; + case CT_APPLICATION_FONT_WOFF: + return "application/font-woff"; + + case CT_APPLICATION_FONT_WOFF2: + return "application/font-woff2"; + + case CT_APPLICATION_VND_MS_FONTOBJ: + return "application/vnd.ms-fontobject"; + + case CT_IMAGE_PNG: + return "image/png"; + + case CT_IMAGE_JPG: + return "image/jpeg"; + + case CT_IMAGE_GIF: + return "image/gif"; + + case CT_IMAGE_XICON: + return "image/x-icon"; + + case CT_IMAGE_BMP: + return "image/bmp"; + + case CT_IMAGE_ICNS: + return "image/icns"; + + default: + case CT_TEXT_PLAIN: + return "text/plain; charset=utf-8"; + } } -*/ -static inline char *http_header_parse(struct web_client *w, char *s) { - char *e = s; +const char *web_response_code_to_string(int code) { + switch(code) { + case 200: + return "OK"; + + case 307: + return "Temporary Redirect"; - // find the : - while(*e && *e != ':') e++; - if(!*e || e[1] != ' ') return e; + case 400: + return "Bad Request"; - // get the name - *e = '\0'; + case 403: + return "Forbidden"; - // find the value - char *v, *ve; - v = ve = e + 2; + case 404: + return "Not Found"; - // find the \r - while(*ve && *ve != '\r') ve++; - if(!*ve || ve[1] != '\n') { - *e = ':'; - return ve; - } + case 412: + return "Preconditions Failed"; - // terminate the value - *ve = '\0'; + default: + if(code >= 100 && code < 200) + return "Informational"; - // fprintf(stderr, "HEADER: '%s' = '%s'\n", s, v); + if(code >= 200 && code < 300) + return "Successful"; - if(!strcasecmp(s, "Origin")) - strncpyz(w->origin, v, ORIGIN_MAX); + if(code >= 300 && code < 400) + return "Redirection"; + + if(code >= 400 && code < 500) + return "Bad Request"; + + if(code >= 500 && code < 600) + return "Server Error"; + + return "Undefined Error"; + } +} - else if(!strcasecmp(s, "Connection")) { - if(strcasestr(v, "keep-alive")) - w->keepalive = 1; - } +static inline char *http_header_parse(struct web_client *w, char *s) { + static uint32_t hash_origin = 0, hash_connection = 0, hash_accept_encoding = 0, hash_donottrack = 0; + + if(unlikely(!hash_origin)) { + hash_origin = simple_uhash("Origin"); + hash_connection = simple_uhash("Connection"); + hash_accept_encoding = simple_uhash("Accept-Encoding"); + hash_donottrack = simple_uhash("DNT"); + } + + char *e = s; + + // find the : + while(*e && *e != ':') e++; + if(!*e) return e; + + // get the name + *e = '\0'; + + // find the value + char *v = e + 1, *ve; + + // skip leading spaces from value + while(*v == ' ') v++; + ve = v; + + // find the \r + while(*ve && *ve != '\r') ve++; + if(!*ve || ve[1] != '\n') { + *e = ':'; + return ve; + } + + // terminate the value + *ve = '\0'; + + // fprintf(stderr, "HEADER: '%s' = '%s'\n", s, v); + uint32_t hash = simple_uhash(s); + + if(hash == hash_origin && !strcasecmp(s, "Origin")) + strncpyz(w->origin, v, ORIGIN_MAX); + + else if(hash == hash_connection && !strcasecmp(s, "Connection")) { + if(strcasestr(v, "keep-alive")) + w->keepalive = 1; + } + else if(web_donotrack_comply && hash == hash_donottrack && !strcasecmp(s, "DNT")) { + if(*v == '0') w->donottrack = 0; + else if(*v == '1') w->donottrack = 1; + } #ifdef NETDATA_WITH_ZLIB - else if(!strcasecmp(s, "Accept-Encoding")) { - if(web_enable_gzip && strcasestr(v, "gzip")) { - w->enable_gzip = 1; - } - } + else if(hash == hash_accept_encoding && !strcasecmp(s, "Accept-Encoding")) { + if(web_enable_gzip) { + if(strcasestr(v, "gzip")) + web_client_enable_deflate(w, 1); + // + // does not seem to work + // else if(strcasestr(v, "deflate")) + // web_client_enable_deflate(w, 0); + } + } #endif /* NETDATA_WITH_ZLIB */ - *e = ':'; - *ve = '\r'; - return ve; + *e = ':'; + *ve = '\r'; + return ve; } // http_request_validate() // returns: // = 0 : all good, process the request -// > 0 : request is complete, but is not supported +// > 0 : request is not supported // < 0 : request is incomplete - wait for more data static inline int http_request_validate(struct web_client *w) { - char *s = w->response.data->buffer, *encoded_url = NULL; - - // is is a valid request? - if(!strncmp(s, "GET ", 4)) { - encoded_url = s = &s[4]; - w->mode = WEB_CLIENT_MODE_NORMAL; - } - else if(!strncmp(s, "OPTIONS ", 8)) { - encoded_url = s = &s[8]; - w->mode = WEB_CLIENT_MODE_OPTIONS; - } - else { - w->wait_receive = 0; - return 1; - } - - // find the SPACE + "HTTP/" - while(*s) { - // find the space - while (*s && *s != ' ') s++; - - // is it SPACE + "HTTP/" ? - if(*s && !strncmp(s, " HTTP/", 6)) break; - else s++; - } - - // incomplete requests - if(!*s) { - w->wait_receive = 1; - return -2; - } - - // we have the end of encoded_url - remember it - char *ue = s; - - while(*s) { - // find a line feed - while (*s && *s != '\r') s++; - - // did we reach the end? - if(unlikely(!*s)) break; - - // is it \r\n ? - if (likely(s[1] == '\n')) { - - // is it again \r\n ? (header end) - if(unlikely(s[2] == '\r' && s[3] == '\n')) { - // a valid complete HTTP request found - - *ue = '\0'; - w->decoded_url = url_decode(encoded_url); - *ue = ' '; - - w->wait_receive = 0; - return 0; - } - - // another header line - s = http_header_parse(w, &s[2]); - } - else s++; - } - - // incomplete request - w->wait_receive = 1; - return -3; + char *s = w->response.data->buffer, *encoded_url = NULL; + + // is is a valid request? + if(!strncmp(s, "GET ", 4)) { + encoded_url = s = &s[4]; + w->mode = WEB_CLIENT_MODE_NORMAL; + } + else if(!strncmp(s, "OPTIONS ", 8)) { + encoded_url = s = &s[8]; + w->mode = WEB_CLIENT_MODE_OPTIONS; + } + else { + w->wait_receive = 0; + return 1; + } + + // find the SPACE + "HTTP/" + while(*s) { + // find the next space + while (*s && *s != ' ') s++; + + // is it SPACE + "HTTP/" ? + if(*s && !strncmp(s, " HTTP/", 6)) break; + else s++; + } + + // incomplete requests + if(unlikely(!*s)) { + w->wait_receive = 1; + return -2; + } + + // we have the end of encoded_url - remember it + char *ue = s; + + // make sure we have complete request + // complete requests contain: \r\n\r\n + while(*s) { + // find a line feed + while(*s && *s++ != '\r'); + + // did we reach the end? + if(unlikely(!*s)) break; + + // is it \r\n ? + if(likely(*s++ == '\n')) { + + // is it again \r\n ? (header end) + if(unlikely(*s == '\r' && s[1] == '\n')) { + // a valid complete HTTP request found + + *ue = '\0'; + url_decode_r(w->decoded_url, encoded_url, URL_MAX + 1); + *ue = ' '; + + // copy the URL - we are going to overwrite parts of it + // FIXME -- we should avoid it + strncpyz(w->last_url, w->decoded_url, URL_MAX); + + w->wait_receive = 0; + return 0; + } + + // another header line + s = http_header_parse(w, s); + } + } + + // incomplete request + w->wait_receive = 1; + return -3; } void web_client_process(struct web_client *w) { - int code = 500; - ssize_t bytes; - - int what_to_do = http_request_validate(w); - - // wait for more data - if(what_to_do < 0) { - if(w->response.data->len > TOO_BIG_REQUEST) { - strcpy(w->last_url, "too big request"); - - debug(D_WEB_CLIENT_ACCESS, "%llu: Received request is too big (%zd bytes).", w->id, w->response.data->len); - - code = 400; - buffer_flush(w->response.data); - buffer_sprintf(w->response.data, "Received request is too big (%zd bytes).\r\n", w->response.data->len); - } - else { - // wait for more data - return; - } - } - else if(what_to_do > 0) { - strcpy(w->last_url, "not a valid response"); - - debug(D_WEB_CLIENT_ACCESS, "%llu: Cannot understand '%s'.", w->id, w->response.data->buffer); - - code = 500; - buffer_flush(w->response.data); - buffer_strcat(w->response.data, "I don't understand you...\r\n"); - } - else { // what_to_do == 0 - gettimeofday(&w->tv_in, NULL); - - global_statistics_lock(); - global_statistics.web_requests++; - global_statistics_unlock(); - - // copy the URL - we are going to overwrite parts of it - // FIXME -- we should avoid it - strncpyz(w->last_url, w->decoded_url, URL_MAX); - - if(w->mode == WEB_CLIENT_MODE_OPTIONS) { - code = 200; - w->response.data->contenttype = CT_TEXT_PLAIN; - buffer_flush(w->response.data); - buffer_strcat(w->response.data, "OK"); - } - else { -#ifdef NETDATA_WITH_ZLIB - if(w->enable_gzip) - web_client_enable_deflate(w); -#endif - - char *url = w->decoded_url; - char *tok = mystrsep(&url, "/?"); - if(tok && *tok) { - debug(D_WEB_CLIENT, "%llu: Processing command '%s'.", w->id, tok); - - if(strcmp(tok, "api") == 0) { - // the client is requesting api access - code = web_client_api_request(w, url); - } - else if(strcmp(tok, "netdata.conf") == 0) { - code = 200; - debug(D_WEB_CLIENT_ACCESS, "%llu: Sending netdata.conf ...", w->id); - - w->response.data->contenttype = CT_TEXT_PLAIN; - buffer_flush(w->response.data); - generate_config(w->response.data, 0); - } - else if(strcmp(tok, WEB_PATH_DATA) == 0) { // "data" - // the client is requesting rrd data -- OLD API - code = web_client_data_request(w, url, DATASOURCE_JSON); - } - else if(strcmp(tok, WEB_PATH_DATASOURCE) == 0) { // "datasource" - // the client is requesting google datasource -- OLD API - code = web_client_data_request(w, url, DATASOURCE_DATATABLE_JSONP); - } - else if(strcmp(tok, WEB_PATH_GRAPH) == 0) { // "graph" - // the client is requesting an rrd graph -- OLD API - - // get the name of the data to show - tok = mystrsep(&url, "/?&"); - if(tok && *tok) { - debug(D_WEB_CLIENT, "%llu: Searching for RRD data with name '%s'.", w->id, tok); - - // do we have such a data set? - RRDSET *st = rrdset_find_byname(tok); - if(!st) st = rrdset_find(tok); - if(!st) { - // we don't have it - // try to send a file with that name - buffer_flush(w->response.data); - code = mysendfile(w, tok); - } - else { - code = 200; - debug(D_WEB_CLIENT_ACCESS, "%llu: Sending %s.json of RRD_STATS...", w->id, st->name); - w->response.data->contenttype = CT_APPLICATION_JSON; - buffer_flush(w->response.data); - rrd_stats_graph_json(st, url, w->response.data); - } - } - else { - code = 400; - buffer_flush(w->response.data); - buffer_strcat(w->response.data, "Graph name?\r\n"); - } - } - else if(strcmp(tok, "list") == 0) { - // OLD API - code = 200; - - debug(D_WEB_CLIENT_ACCESS, "%llu: Sending list of RRD_STATS...", w->id); - - buffer_flush(w->response.data); - RRDSET *st = rrdset_root; - - for ( ; st ; st = st->next ) - buffer_sprintf(w->response.data, "%s\n", st->name); - } - else if(strcmp(tok, "all.json") == 0) { - // OLD API - code = 200; - debug(D_WEB_CLIENT_ACCESS, "%llu: Sending JSON list of all monitors of RRD_STATS...", w->id); - - w->response.data->contenttype = CT_APPLICATION_JSON; - buffer_flush(w->response.data); - rrd_stats_all_json(w->response.data); - } + static uint32_t hash_api = 0, hash_netdata_conf = 0, hash_data = 0, hash_datasource = 0, hash_graph = 0, + hash_list = 0, hash_all_json = 0, hash_exit = 0, hash_debug = 0, hash_mirror = 0; + + // start timing us + gettimeofday(&w->tv_in, NULL); + + if(unlikely(!hash_api)) { + hash_api = simple_hash("api"); + hash_netdata_conf = simple_hash("netdata.conf"); + hash_data = simple_hash(WEB_PATH_DATA); + hash_datasource = simple_hash(WEB_PATH_DATASOURCE); + hash_graph = simple_hash(WEB_PATH_GRAPH); + hash_list = simple_hash("list"); + hash_all_json = simple_hash("all.json"); + hash_exit = simple_hash("exit"); + hash_debug = simple_hash("debug"); + hash_mirror = simple_hash("mirror"); + } + + int code = 500; + ssize_t bytes; + + int what_to_do = http_request_validate(w); + + // wait for more data + if(what_to_do < 0) { + if(w->response.data->len > TOO_BIG_REQUEST) { + strcpy(w->last_url, "too big request"); + + debug(D_WEB_CLIENT_ACCESS, "%llu: Received request is too big (%zu bytes).", w->id, w->response.data->len); + + code = 400; + buffer_flush(w->response.data); + buffer_sprintf(w->response.data, "Received request is too big (%zu bytes).\r\n", w->response.data->len); + } + else { + // wait for more data + return; + } + } + else if(what_to_do > 0) { + // strcpy(w->last_url, "not a valid request"); + + debug(D_WEB_CLIENT_ACCESS, "%llu: Cannot understand '%s'.", w->id, w->response.data->buffer); + + code = 500; + buffer_flush(w->response.data); + buffer_strcat(w->response.data, "I don't understand you...\r\n"); + } + else { // what_to_do == 0 + if(w->mode == WEB_CLIENT_MODE_OPTIONS) { + code = 200; + w->response.data->contenttype = CT_TEXT_PLAIN; + buffer_flush(w->response.data); + buffer_strcat(w->response.data, "OK"); + } + else { + char *url = w->decoded_url; + char *tok = mystrsep(&url, "/?"); + if(tok && *tok) { + uint32_t hash = simple_hash(tok); + debug(D_WEB_CLIENT, "%llu: Processing command '%s'.", w->id, tok); + + if(hash == hash_api && strcmp(tok, "api") == 0) { + // the client is requesting api access + code = web_client_api_request(w, url); + } + else if(hash == hash_netdata_conf && strcmp(tok, "netdata.conf") == 0) { + code = 200; + debug(D_WEB_CLIENT_ACCESS, "%llu: Sending netdata.conf ...", w->id); + + w->response.data->contenttype = CT_TEXT_PLAIN; + buffer_flush(w->response.data); + generate_config(w->response.data, 0); + } + else if(hash == hash_data && strcmp(tok, WEB_PATH_DATA) == 0) { // "data" + // the client is requesting rrd data -- OLD API + code = web_client_api_old_data_request(w, url, DATASOURCE_JSON); + } + else if(hash == hash_datasource && strcmp(tok, WEB_PATH_DATASOURCE) == 0) { // "datasource" + // the client is requesting google datasource -- OLD API + code = web_client_api_old_data_request(w, url, DATASOURCE_DATATABLE_JSONP); + } + else if(hash == hash_graph && strcmp(tok, WEB_PATH_GRAPH) == 0) { // "graph" + // the client is requesting an rrd graph -- OLD API + + // get the name of the data to show + tok = mystrsep(&url, "/?&"); + if(tok && *tok) { + debug(D_WEB_CLIENT, "%llu: Searching for RRD data with name '%s'.", w->id, tok); + + // do we have such a data set? + RRDSET *st = rrdset_find_byname(tok); + if(!st) st = rrdset_find(tok); + if(!st) { + // we don't have it + // try to send a file with that name + buffer_flush(w->response.data); + code = mysendfile(w, tok); + } + else { + code = 200; + debug(D_WEB_CLIENT_ACCESS, "%llu: Sending %s.json of RRD_STATS...", w->id, st->name); + w->response.data->contenttype = CT_APPLICATION_JSON; + buffer_flush(w->response.data); + rrd_stats_graph_json(st, url, w->response.data); + } + } + else { + code = 400; + buffer_flush(w->response.data); + buffer_strcat(w->response.data, "Graph name?\r\n"); + } + } + else if(hash == hash_list && strcmp(tok, "list") == 0) { + // OLD API + code = 200; + + debug(D_WEB_CLIENT_ACCESS, "%llu: Sending list of RRD_STATS...", w->id); + + buffer_flush(w->response.data); + RRDSET *st = localhost.rrdset_root; + + for ( ; st ; st = st->next ) + buffer_sprintf(w->response.data, "%s\n", st->name); + } + else if(hash == hash_all_json && strcmp(tok, "all.json") == 0) { + // OLD API + code = 200; + debug(D_WEB_CLIENT_ACCESS, "%llu: Sending JSON list of all monitors of RRD_STATS...", w->id); + + w->response.data->contenttype = CT_APPLICATION_JSON; + buffer_flush(w->response.data); + rrd_stats_all_json(w->response.data); + } #ifdef NETDATA_INTERNAL_CHECKS - else if(strcmp(tok, "exit") == 0) { - code = 200; - w->response.data->contenttype = CT_TEXT_PLAIN; - buffer_flush(w->response.data); - - if(!netdata_exit) - buffer_strcat(w->response.data, "ok, will do..."); - else - buffer_strcat(w->response.data, "I am doing it already"); - - netdata_exit = 1; - } - else if(strcmp(tok, "debug") == 0) { - buffer_flush(w->response.data); - - // get the name of the data to show - tok = mystrsep(&url, "/?&"); - if(tok && *tok) { - debug(D_WEB_CLIENT, "%llu: Searching for RRD data with name '%s'.", w->id, tok); - - // do we have such a data set? - RRDSET *st = rrdset_find_byname(tok); - if(!st) st = rrdset_find(tok); - if(!st) { - code = 404; - buffer_sprintf(w->response.data, "Chart %s is not found.\r\n", tok); - debug(D_WEB_CLIENT_ACCESS, "%llu: %s is not found.", w->id, tok); - } - else { - code = 200; - debug_flags |= D_RRD_STATS; - st->debug = !st->debug; - buffer_sprintf(w->response.data, "Chart %s has now debug %s.\r\n", tok, st->debug?"enabled":"disabled"); - debug(D_WEB_CLIENT_ACCESS, "%llu: debug for %s is %s.", w->id, tok, st->debug?"enabled":"disabled"); - } - } - else { - code = 500; - buffer_flush(w->response.data); - buffer_strcat(w->response.data, "debug which chart?\r\n"); - } - } - else if(strcmp(tok, "mirror") == 0) { - code = 200; - - debug(D_WEB_CLIENT_ACCESS, "%llu: Mirroring...", w->id); - - // replace the zero bytes with spaces - buffer_char_replace(w->response.data, '\0', ' '); - - // just leave the buffer as is - // it will be copied back to the client - } -#endif /* NETDATA_INTERNAL_CHECKS */ - else { - char filename[FILENAME_MAX+1]; - url = filename; - strncpyz(filename, w->last_url, FILENAME_MAX); - tok = mystrsep(&url, "?"); - buffer_flush(w->response.data); - code = mysendfile(w, (tok && *tok)?tok:"/"); - } - } - else { - char filename[FILENAME_MAX+1]; - url = filename; - strncpyz(filename, w->last_url, FILENAME_MAX); - tok = mystrsep(&url, "?"); - buffer_flush(w->response.data); - code = mysendfile(w, (tok && *tok)?tok:"/"); - } - } - } - - gettimeofday(&w->tv_ready, NULL); - w->response.data->date = time(NULL); - w->response.sent = 0; - w->response.code = code; - - // prepare the HTTP response header - debug(D_WEB_CLIENT, "%llu: Generating HTTP header with response %d.", w->id, code); - - char *content_type_string; - switch(w->response.data->contenttype) { - case CT_TEXT_HTML: - content_type_string = "text/html; charset=utf-8"; - break; - - case CT_APPLICATION_XML: - content_type_string = "application/xml; charset=utf-8"; - break; - - case CT_APPLICATION_JSON: - content_type_string = "application/json; charset=utf-8"; - break; - - case CT_APPLICATION_X_JAVASCRIPT: - content_type_string = "application/x-javascript; charset=utf-8"; - break; - - case CT_TEXT_CSS: - content_type_string = "text/css; charset=utf-8"; - break; - - case CT_TEXT_XML: - content_type_string = "text/xml; charset=utf-8"; - break; - - case CT_TEXT_XSL: - content_type_string = "text/xsl; charset=utf-8"; - break; - - case CT_APPLICATION_OCTET_STREAM: - content_type_string = "application/octet-stream"; - break; - - case CT_IMAGE_SVG_XML: - content_type_string = "image/svg+xml"; - break; - - case CT_APPLICATION_X_FONT_TRUETYPE: - content_type_string = "application/x-font-truetype"; - break; - - case CT_APPLICATION_X_FONT_OPENTYPE: - content_type_string = "application/x-font-opentype"; - break; - - case CT_APPLICATION_FONT_WOFF: - content_type_string = "application/font-woff"; - break; - - case CT_APPLICATION_FONT_WOFF2: - content_type_string = "application/font-woff2"; - break; - - case CT_APPLICATION_VND_MS_FONTOBJ: - content_type_string = "application/vnd.ms-fontobject"; - break; - - case CT_IMAGE_PNG: - content_type_string = "image/png"; - break; - - case CT_IMAGE_JPG: - content_type_string = "image/jpeg"; - break; - - case CT_IMAGE_GIF: - content_type_string = "image/gif"; - break; - - case CT_IMAGE_XICON: - content_type_string = "image/x-icon"; - break; - - case CT_IMAGE_BMP: - content_type_string = "image/bmp"; - break; - - case CT_IMAGE_ICNS: - content_type_string = "image/icns"; - break; - - default: - case CT_TEXT_PLAIN: - content_type_string = "text/plain; charset=utf-8"; - break; - } - - char *code_msg; - switch(code) { - case 200: - code_msg = "OK"; - break; - - case 307: - code_msg = "Temporary Redirect"; - break; - - case 400: - code_msg = "Bad Request"; - break; - - case 403: - code_msg = "Forbidden"; - break; - - case 404: - code_msg = "Not Found"; - break; - - case 412: - code_msg = "Preconditions Failed"; - break; - - default: - code_msg = "Internal Server Error"; - break; - } - - char date[100]; - struct tm tmbuf, *tm = gmtime_r(&w->response.data->date, &tmbuf); - strftime(date, sizeof(date), "%a, %d %b %Y %H:%M:%S %Z", tm); - - buffer_sprintf(w->response.header_output, - "HTTP/1.1 %d %s\r\n" - "Connection: %s\r\n" - "Server: NetData Embedded HTTP Server\r\n" - "Access-Control-Allow-Origin: %s\r\n" - "Access-Control-Allow-Credentials: true\r\n" - "Content-Type: %s\r\n" - "Date: %s\r\n" - , code, code_msg - , w->keepalive?"keep-alive":"close" - , w->origin - , content_type_string - , date - ); - - if(w->cookie1[0]) { - buffer_sprintf(w->response.header_output, - "Set-Cookie: %s\r\n", - w->cookie1); - } - - if(w->cookie2[0]) { - buffer_sprintf(w->response.header_output, - "Set-Cookie: %s\r\n", - w->cookie2); - } - - if(w->mode == WEB_CLIENT_MODE_OPTIONS) { - buffer_strcat(w->response.header_output, - "Access-Control-Allow-Methods: GET, OPTIONS\r\n" - "Access-Control-Allow-Headers: accept, x-requested-with, origin, content-type, cookie\r\n" - "Access-Control-Max-Age: 1209600\r\n" // 86400 * 14 - ); - } - - if(buffer_strlen(w->response.header)) - buffer_strcat(w->response.header_output, buffer_tostring(w->response.header)); - - if(w->mode == WEB_CLIENT_MODE_NORMAL && (w->response.data->options & WB_CONTENT_NO_CACHEABLE)) { - buffer_sprintf(w->response.header_output, - "Expires: %s\r\n" - "Cache-Control: no-cache\r\n" - , date); - } - else if(w->mode != WEB_CLIENT_MODE_OPTIONS) { - char edate[100]; - time_t et = w->response.data->date + (86400 * 14); - struct tm etmbuf, *etm = gmtime_r(&et, &etmbuf); - strftime(edate, sizeof(edate), "%a, %d %b %Y %H:%M:%S %Z", etm); - - buffer_sprintf(w->response.header_output, - "Expires: %s\r\n" - "Cache-Control: public\r\n" - , edate); - } - - // if we know the content length, put it - if(!w->response.zoutput && (w->response.data->len || w->response.rlen)) - buffer_sprintf(w->response.header_output, - "Content-Length: %ld\r\n" - , w->response.data->len? w->response.data->len: w->response.rlen - ); - else if(!w->response.zoutput) - w->keepalive = 0; // content-length is required for keep-alive - - if(w->response.zoutput) { - buffer_strcat(w->response.header_output, - "Content-Encoding: gzip\r\n" - "Transfer-Encoding: chunked\r\n" - ); - } - - buffer_strcat(w->response.header_output, "\r\n"); - - // disable TCP_NODELAY, to buffer the header - int flag = 0; - if(setsockopt(w->ofd, IPPROTO_TCP, TCP_NODELAY, (char *) &flag, sizeof(int)) != 0) - error("%llu: failed to disable TCP_NODELAY on socket.", w->id); - - // sent the HTTP header - debug(D_WEB_DATA, "%llu: Sending response HTTP header of size %d: '%s'" - , w->id - , buffer_strlen(w->response.header_output) - , buffer_tostring(w->response.header_output) - ); - - bytes = send(w->ofd, buffer_tostring(w->response.header_output), buffer_strlen(w->response.header_output), 0); - if(bytes != (ssize_t) buffer_strlen(w->response.header_output)) - error("%llu: HTTP Header failed to be sent (I sent %d bytes but the system sent %d bytes)." - , w->id - , buffer_strlen(w->response.header_output) - , bytes); - else { - global_statistics_lock(); - global_statistics.bytes_sent += bytes; - global_statistics_unlock(); - } - - // enable TCP_NODELAY, to send all data immediately at the next send() - flag = 1; - if(setsockopt(w->ofd, IPPROTO_TCP, TCP_NODELAY, (char *) &flag, sizeof(int)) != 0) error("%llu: failed to enable TCP_NODELAY on socket.", w->id); - - // enable sending immediately if we have data - if(w->response.data->len) w->wait_send = 1; - else w->wait_send = 0; - - // pretty logging - switch(w->mode) { - case WEB_CLIENT_MODE_OPTIONS: - debug(D_WEB_CLIENT, "%llu: Done preparing the OPTIONS response. Sending data (%d bytes) to client.", w->id, w->response.data->len); - break; - - case WEB_CLIENT_MODE_NORMAL: - debug(D_WEB_CLIENT, "%llu: Done preparing the response. Sending data (%d bytes) to client.", w->id, w->response.data->len); - break; - - case WEB_CLIENT_MODE_FILECOPY: - if(w->response.rlen) { - debug(D_WEB_CLIENT, "%llu: Done preparing the response. Will be sending data file of %d bytes to client.", w->id, w->response.rlen); - w->wait_receive = 1; - - /* - // utilize the kernel sendfile() for copying the file to the socket. - // this block of code can be commented, without anything missing. - // when it is commented, the program will copy the data using async I/O. - { - long len = sendfile(w->ofd, w->ifd, NULL, w->response.data->rbytes); - if(len != w->response.data->rbytes) error("%llu: sendfile() should copy %ld bytes, but copied %ld. Falling back to manual copy.", w->id, w->response.data->rbytes, len); - else web_client_reset(w); - } - */ - } - else - debug(D_WEB_CLIENT, "%llu: Done preparing the response. Will be sending an unknown amount of bytes to client.", w->id); - break; - - default: - fatal("%llu: Unknown client mode %d.", w->id, w->mode); - break; - } + else if(hash == hash_exit && strcmp(tok, "exit") == 0) { + code = 200; + w->response.data->contenttype = CT_TEXT_PLAIN; + buffer_flush(w->response.data); + + if(!netdata_exit) + buffer_strcat(w->response.data, "ok, will do..."); + else + buffer_strcat(w->response.data, "I am doing it already"); + + error("web request to exit received."); + netdata_cleanup_and_exit(0); + netdata_exit = 1; + } + else if(hash == hash_debug && strcmp(tok, "debug") == 0) { + buffer_flush(w->response.data); + + // get the name of the data to show + tok = mystrsep(&url, "/?&"); + if(tok && *tok) { + debug(D_WEB_CLIENT, "%llu: Searching for RRD data with name '%s'.", w->id, tok); + + // do we have such a data set? + RRDSET *st = rrdset_find_byname(tok); + if(!st) st = rrdset_find(tok); + if(!st) { + code = 404; + buffer_sprintf(w->response.data, "Chart %s is not found.\r\n", tok); + debug(D_WEB_CLIENT_ACCESS, "%llu: %s is not found.", w->id, tok); + } + else { + code = 200; + debug_flags |= D_RRD_STATS; + st->debug = !st->debug; + buffer_sprintf(w->response.data, "Chart %s has now debug %s.\r\n", tok, st->debug?"enabled":"disabled"); + debug(D_WEB_CLIENT_ACCESS, "%llu: debug for %s is %s.", w->id, tok, st->debug?"enabled":"disabled"); + } + } + else { + code = 500; + buffer_flush(w->response.data); + buffer_strcat(w->response.data, "debug which chart?\r\n"); + } + } + else if(hash == hash_mirror && strcmp(tok, "mirror") == 0) { + code = 200; + + debug(D_WEB_CLIENT_ACCESS, "%llu: Mirroring...", w->id); + + // replace the zero bytes with spaces + buffer_char_replace(w->response.data, '\0', ' '); + + // just leave the buffer as is + // it will be copied back to the client + } +#endif /* NETDATA_INTERNAL_CHECKS */ + else { + char filename[FILENAME_MAX+1]; + url = filename; + strncpyz(filename, w->last_url, FILENAME_MAX); + tok = mystrsep(&url, "?"); + buffer_flush(w->response.data); + code = mysendfile(w, (tok && *tok)?tok:"/"); + } + } + else { + char filename[FILENAME_MAX+1]; + url = filename; + strncpyz(filename, w->last_url, FILENAME_MAX); + tok = mystrsep(&url, "?"); + buffer_flush(w->response.data); + code = mysendfile(w, (tok && *tok)?tok:"/"); + } + } + } + + gettimeofday(&w->tv_ready, NULL); + w->response.data->date = time(NULL); + w->response.sent = 0; + w->response.code = code; + + // prepare the HTTP response header + debug(D_WEB_CLIENT, "%llu: Generating HTTP header with response %d.", w->id, code); + + const char *content_type_string = web_content_type_to_string(w->response.data->contenttype); + const char *code_msg = web_response_code_to_string(code); + + char date[32]; + struct tm tmbuf, *tm = gmtime_r(&w->response.data->date, &tmbuf); + strftime(date, sizeof(date), "%a, %d %b %Y %H:%M:%S %Z", tm); + + buffer_sprintf(w->response.header_output, + "HTTP/1.1 %d %s\r\n" + "Connection: %s\r\n" + "Server: NetData Embedded HTTP Server\r\n" + "Access-Control-Allow-Origin: %s\r\n" + "Access-Control-Allow-Credentials: true\r\n" + "Content-Type: %s\r\n" + "Date: %s\r\n" + , code, code_msg + , w->keepalive?"keep-alive":"close" + , w->origin + , content_type_string + , date + ); + + if(w->cookie1[0] || w->cookie2[0]) { + if(w->cookie1[0]) { + buffer_sprintf(w->response.header_output, + "Set-Cookie: %s\r\n", + w->cookie1); + } + + if(w->cookie2[0]) { + buffer_sprintf(w->response.header_output, + "Set-Cookie: %s\r\n", + w->cookie2); + } + + if(web_donotrack_comply) + buffer_sprintf(w->response.header_output, + "Tk: T;cookies\r\n"); + } + else { + if(web_donotrack_comply) { + if(w->tracking_required) + buffer_sprintf(w->response.header_output, + "Tk: T;cookies\r\n"); + else + buffer_sprintf(w->response.header_output, + "Tk: N\r\n"); + } + } + + if(w->mode == WEB_CLIENT_MODE_OPTIONS) { + buffer_strcat(w->response.header_output, + "Access-Control-Allow-Methods: GET, OPTIONS\r\n" + "Access-Control-Allow-Headers: accept, x-requested-with, origin, content-type, cookie\r\n" + "Access-Control-Max-Age: 1209600\r\n" // 86400 * 14 + ); + } + + if(buffer_strlen(w->response.header)) + buffer_strcat(w->response.header_output, buffer_tostring(w->response.header)); + + if(w->mode == WEB_CLIENT_MODE_NORMAL && (w->response.data->options & WB_CONTENT_NO_CACHEABLE)) { + buffer_sprintf(w->response.header_output, + "Expires: %s\r\n" + "Cache-Control: no-cache\r\n" + , date); + } + else if(w->mode != WEB_CLIENT_MODE_OPTIONS) { + char edate[32]; + time_t et = w->response.data->date + (86400 * 14); + struct tm etmbuf, *etm = gmtime_r(&et, &etmbuf); + strftime(edate, sizeof(edate), "%a, %d %b %Y %H:%M:%S %Z", etm); + + buffer_sprintf(w->response.header_output, + "Expires: %s\r\n" + "Cache-Control: public\r\n" + , edate); + } + + // if we know the content length, put it + if(!w->response.zoutput && (w->response.data->len || w->response.rlen)) + buffer_sprintf(w->response.header_output, + "Content-Length: %zu\r\n" + , w->response.data->len? w->response.data->len: w->response.rlen + ); + else if(!w->response.zoutput) + w->keepalive = 0; // content-length is required for keep-alive + + if(w->response.zoutput) { + buffer_strcat(w->response.header_output, + "Content-Encoding: gzip\r\n" + "Transfer-Encoding: chunked\r\n" + ); + } + + buffer_strcat(w->response.header_output, "\r\n"); + + // sent the HTTP header + debug(D_WEB_DATA, "%llu: Sending response HTTP header of size %zu: '%s'" + , w->id + , buffer_strlen(w->response.header_output) + , buffer_tostring(w->response.header_output) + ); + + web_client_crock_socket(w); + + bytes = send(w->ofd, buffer_tostring(w->response.header_output), buffer_strlen(w->response.header_output), 0); + if(bytes != (ssize_t) buffer_strlen(w->response.header_output)) { + if(bytes > 0) + w->stats_sent_bytes += bytes; + + debug(D_WEB_CLIENT, "%llu: HTTP Header failed to be sent (I sent %zu bytes but the system sent %zd bytes). Closing web client." + , w->id + , buffer_strlen(w->response.header_output) + , bytes); + + WEB_CLIENT_IS_DEAD(w); + return; + } + else + w->stats_sent_bytes += bytes; + + // enable sending immediately if we have data + if(w->response.data->len) w->wait_send = 1; + else w->wait_send = 0; + + // pretty logging + switch(w->mode) { + case WEB_CLIENT_MODE_OPTIONS: + debug(D_WEB_CLIENT, "%llu: Done preparing the OPTIONS response. Sending data (%zu bytes) to client.", w->id, w->response.data->len); + break; + + case WEB_CLIENT_MODE_NORMAL: + debug(D_WEB_CLIENT, "%llu: Done preparing the response. Sending data (%zu bytes) to client.", w->id, w->response.data->len); + break; + + case WEB_CLIENT_MODE_FILECOPY: + if(w->response.rlen) { + debug(D_WEB_CLIENT, "%llu: Done preparing the response. Will be sending data file of %zu bytes to client.", w->id, w->response.rlen); + w->wait_receive = 1; + + /* + // utilize the kernel sendfile() for copying the file to the socket. + // this block of code can be commented, without anything missing. + // when it is commented, the program will copy the data using async I/O. + { + long len = sendfile(w->ofd, w->ifd, NULL, w->response.data->rbytes); + if(len != w->response.data->rbytes) + error("%llu: sendfile() should copy %ld bytes, but copied %ld. Falling back to manual copy.", w->id, w->response.data->rbytes, len); + else + web_client_reset(w); + } + */ + } + else + debug(D_WEB_CLIENT, "%llu: Done preparing the response. Will be sending an unknown amount of bytes to client.", w->id); + break; + + default: + fatal("%llu: Unknown client mode %d.", w->id, w->mode); + break; + } } -long web_client_send_chunk_header(struct web_client *w, long len) +ssize_t web_client_send_chunk_header(struct web_client *w, size_t len) { - debug(D_DEFLATE, "%llu: OPEN CHUNK of %d bytes (hex: %x).", w->id, len, len); - char buf[1024]; - sprintf(buf, "%lX\r\n", len); - ssize_t bytes = send(w->ofd, buf, strlen(buf), MSG_DONTWAIT); - - if(bytes > 0) debug(D_DEFLATE, "%llu: Sent chunk header %d bytes.", w->id, bytes); - else if(bytes == 0) debug(D_DEFLATE, "%llu: Did not send chunk header to the client.", w->id); - else debug(D_DEFLATE, "%llu: Failed to send chunk header to client.", w->id); - - return bytes; + debug(D_DEFLATE, "%llu: OPEN CHUNK of %zu bytes (hex: %zx).", w->id, len, len); + char buf[24]; + sprintf(buf, "%zX\r\n", len); + + ssize_t bytes = send(w->ofd, buf, strlen(buf), 0); + if(bytes > 0) { + debug(D_DEFLATE, "%llu: Sent chunk header %zd bytes.", w->id, bytes); + w->stats_sent_bytes += bytes; + } + + else if(bytes == 0) { + debug(D_WEB_CLIENT, "%llu: Did not send chunk header to the client.", w->id); + WEB_CLIENT_IS_DEAD(w); + } + else { + debug(D_WEB_CLIENT, "%llu: Failed to send chunk header to client.", w->id); + WEB_CLIENT_IS_DEAD(w); + } + + return bytes; } -long web_client_send_chunk_close(struct web_client *w) +ssize_t web_client_send_chunk_close(struct web_client *w) { - //debug(D_DEFLATE, "%llu: CLOSE CHUNK.", w->id); - - ssize_t bytes = send(w->ofd, "\r\n", 2, MSG_DONTWAIT); - - if(bytes > 0) debug(D_DEFLATE, "%llu: Sent chunk suffix %d bytes.", w->id, bytes); - else if(bytes == 0) debug(D_DEFLATE, "%llu: Did not send chunk suffix to the client.", w->id); - else debug(D_DEFLATE, "%llu: Failed to send chunk suffix to client.", w->id); - - return bytes; + //debug(D_DEFLATE, "%llu: CLOSE CHUNK.", w->id); + + ssize_t bytes = send(w->ofd, "\r\n", 2, 0); + if(bytes > 0) { + debug(D_DEFLATE, "%llu: Sent chunk suffix %zd bytes.", w->id, bytes); + w->stats_sent_bytes += bytes; + } + + else if(bytes == 0) { + debug(D_WEB_CLIENT, "%llu: Did not send chunk suffix to the client.", w->id); + WEB_CLIENT_IS_DEAD(w); + } + else { + debug(D_WEB_CLIENT, "%llu: Failed to send chunk suffix to client.", w->id); + WEB_CLIENT_IS_DEAD(w); + } + + return bytes; } -long web_client_send_chunk_finalize(struct web_client *w) +ssize_t web_client_send_chunk_finalize(struct web_client *w) { - //debug(D_DEFLATE, "%llu: FINALIZE CHUNK.", w->id); - - ssize_t bytes = send(w->ofd, "\r\n0\r\n\r\n", 7, MSG_DONTWAIT); - - if(bytes > 0) debug(D_DEFLATE, "%llu: Sent chunk suffix %d bytes.", w->id, bytes); - else if(bytes == 0) debug(D_DEFLATE, "%llu: Did not send chunk suffix to the client.", w->id); - else debug(D_DEFLATE, "%llu: Failed to send chunk suffix to client.", w->id); - - return bytes; + //debug(D_DEFLATE, "%llu: FINALIZE CHUNK.", w->id); + + ssize_t bytes = send(w->ofd, "\r\n0\r\n\r\n", 7, 0); + if(bytes > 0) { + debug(D_DEFLATE, "%llu: Sent chunk suffix %zd bytes.", w->id, bytes); + w->stats_sent_bytes += bytes; + } + + else if(bytes == 0) { + debug(D_WEB_CLIENT, "%llu: Did not send chunk finalize suffix to the client.", w->id); + WEB_CLIENT_IS_DEAD(w); + } + else { + debug(D_WEB_CLIENT, "%llu: Failed to send chunk finalize suffix to client.", w->id); + WEB_CLIENT_IS_DEAD(w); + } + + return bytes; } #ifdef NETDATA_WITH_ZLIB -long web_client_send_deflate(struct web_client *w) +ssize_t web_client_send_deflate(struct web_client *w) { - long len = 0, t = 0; - - // when using compression, - // w->response.sent is the amount of bytes passed through compression - - debug(D_DEFLATE, "%llu: web_client_send_deflate(): w->response.data->len = %d, w->response.sent = %d, w->response.zhave = %d, w->response.zsent = %d, w->response.zstream.avail_in = %d, w->response.zstream.avail_out = %d, w->response.zstream.total_in = %d, w->response.zstream.total_out = %d.", w->id, w->response.data->len, w->response.sent, w->response.zhave, w->response.zsent, w->response.zstream.avail_in, w->response.zstream.avail_out, w->response.zstream.total_in, w->response.zstream.total_out); - - if(w->response.data->len - w->response.sent == 0 && w->response.zstream.avail_in == 0 && w->response.zhave == w->response.zsent && w->response.zstream.avail_out != 0) { - // there is nothing to send - - debug(D_WEB_CLIENT, "%llu: Out of output data.", w->id); - - // finalize the chunk - if(w->response.sent != 0) - t += web_client_send_chunk_finalize(w); - - // there can be two cases for this - // A. we have done everything - // B. we temporarily have nothing to send, waiting for the buffer to be filled by ifd - - if(w->mode == WEB_CLIENT_MODE_FILECOPY && w->wait_receive && w->ifd != w->ofd && w->response.rlen && w->response.rlen > w->response.data->len) { - // we have to wait, more data will come - debug(D_WEB_CLIENT, "%llu: Waiting for more data to become available.", w->id); - w->wait_send = 0; - return(0); - } - - if(w->keepalive == 0) { - debug(D_WEB_CLIENT, "%llu: Closing (keep-alive is not enabled). %ld bytes sent.", w->id, w->response.sent); - errno = 0; - return(-1); - } - - // reset the client - web_client_reset(w); - debug(D_WEB_CLIENT, "%llu: Done sending all data on socket. Waiting for next request on the same socket.", w->id); - return(0); - } - - if(w->response.zhave == w->response.zsent) { - // compress more input data - - // close the previous open chunk - if(w->response.sent != 0) t += web_client_send_chunk_close(w); - - debug(D_DEFLATE, "%llu: Compressing %d new bytes starting from %d (and %d left behind).", w->id, (w->response.data->len - w->response.sent), w->response.sent, w->response.zstream.avail_in); - - // give the compressor all the data not passed through the compressor yet - if(w->response.data->len > w->response.sent) { -#ifdef NETDATA_INTERNAL_CHECKS - if((long)w->response.sent - (long)w->response.zstream.avail_in < 0) - error("internal error: avail_in is corrupted."); -#endif - w->response.zstream.next_in = (Bytef *)&w->response.data->buffer[w->response.sent - w->response.zstream.avail_in]; - w->response.zstream.avail_in += (uInt) (w->response.data->len - w->response.sent); - } - - // reset the compressor output buffer - w->response.zstream.next_out = w->response.zbuffer; - w->response.zstream.avail_out = ZLIB_CHUNK; - - // ask for FINISH if we have all the input - int flush = Z_SYNC_FLUSH; - if(w->mode == WEB_CLIENT_MODE_NORMAL - || (w->mode == WEB_CLIENT_MODE_FILECOPY && !w->wait_receive && w->response.data->len == w->response.rlen)) { - flush = Z_FINISH; - debug(D_DEFLATE, "%llu: Requesting Z_FINISH, if possible.", w->id); - } - else { - debug(D_DEFLATE, "%llu: Requesting Z_SYNC_FLUSH.", w->id); - } - - // compress - if(deflate(&w->response.zstream, flush) == Z_STREAM_ERROR) { - error("%llu: Compression failed. Closing down client.", w->id); - web_client_reset(w); - return(-1); - } - - w->response.zhave = ZLIB_CHUNK - w->response.zstream.avail_out; - w->response.zsent = 0; - - // keep track of the bytes passed through the compressor - w->response.sent = w->response.data->len; - - debug(D_DEFLATE, "%llu: Compression produced %d bytes.", w->id, w->response.zhave); - - // open a new chunk - t += web_client_send_chunk_header(w, w->response.zhave); - } - - debug(D_WEB_CLIENT, "%llu: Sending %d bytes of data (+%d of chunk header).", w->id, w->response.zhave - w->response.zsent, t); - - len = send(w->ofd, &w->response.zbuffer[w->response.zsent], (size_t) (w->response.zhave - w->response.zsent), MSG_DONTWAIT); - if(len > 0) { - w->response.zsent += len; - if(t > 0) len += t; - debug(D_WEB_CLIENT, "%llu: Sent %d bytes.", w->id, len); - } - else if(len == 0) debug(D_WEB_CLIENT, "%llu: Did not send any bytes to the client (zhave = %ld, zsent = %ld, need to send = %ld).", w->id, w->response.zhave, w->response.zsent, w->response.zhave - w->response.zsent); - else debug(D_WEB_CLIENT, "%llu: Failed to send data to client. Reason: %s", w->id, strerror(errno)); - - return(len); + ssize_t len = 0, t = 0; + + // when using compression, + // w->response.sent is the amount of bytes passed through compression + + debug(D_DEFLATE, "%llu: web_client_send_deflate(): w->response.data->len = %zu, w->response.sent = %zu, w->response.zhave = %zu, w->response.zsent = %zu, w->response.zstream.avail_in = %u, w->response.zstream.avail_out = %u, w->response.zstream.total_in = %lu, w->response.zstream.total_out = %lu.", + w->id, w->response.data->len, w->response.sent, w->response.zhave, w->response.zsent, w->response.zstream.avail_in, w->response.zstream.avail_out, w->response.zstream.total_in, w->response.zstream.total_out); + + if(w->response.data->len - w->response.sent == 0 && w->response.zstream.avail_in == 0 && w->response.zhave == w->response.zsent && w->response.zstream.avail_out != 0) { + // there is nothing to send + + debug(D_WEB_CLIENT, "%llu: Out of output data.", w->id); + + // finalize the chunk + if(w->response.sent != 0) { + t = web_client_send_chunk_finalize(w); + if(t < 0) return t; + } + + if(w->mode == WEB_CLIENT_MODE_FILECOPY && w->wait_receive && w->response.rlen && w->response.rlen > w->response.data->len) { + // we have to wait, more data will come + debug(D_WEB_CLIENT, "%llu: Waiting for more data to become available.", w->id); + w->wait_send = 0; + return t; + } + + if(unlikely(!w->keepalive)) { + debug(D_WEB_CLIENT, "%llu: Closing (keep-alive is not enabled). %zu bytes sent.", w->id, w->response.sent); + WEB_CLIENT_IS_DEAD(w); + return t; + } + + // reset the client + web_client_reset(w); + debug(D_WEB_CLIENT, "%llu: Done sending all data on socket.", w->id); + return t; + } + + if(w->response.zhave == w->response.zsent) { + // compress more input data + + // close the previous open chunk + if(w->response.sent != 0) { + t = web_client_send_chunk_close(w); + if(t < 0) return t; + } + + debug(D_DEFLATE, "%llu: Compressing %zu new bytes starting from %zu (and %u left behind).", w->id, (w->response.data->len - w->response.sent), w->response.sent, w->response.zstream.avail_in); + + // give the compressor all the data not passed through the compressor yet + if(w->response.data->len > w->response.sent) { + w->response.zstream.next_in = (Bytef *)&w->response.data->buffer[w->response.sent - w->response.zstream.avail_in]; + w->response.zstream.avail_in += (uInt) (w->response.data->len - w->response.sent); + } + + // reset the compressor output buffer + w->response.zstream.next_out = w->response.zbuffer; + w->response.zstream.avail_out = ZLIB_CHUNK; + + // ask for FINISH if we have all the input + int flush = Z_SYNC_FLUSH; + if(w->mode == WEB_CLIENT_MODE_NORMAL + || (w->mode == WEB_CLIENT_MODE_FILECOPY && !w->wait_receive && w->response.data->len == w->response.rlen)) { + flush = Z_FINISH; + debug(D_DEFLATE, "%llu: Requesting Z_FINISH, if possible.", w->id); + } + else { + debug(D_DEFLATE, "%llu: Requesting Z_SYNC_FLUSH.", w->id); + } + + // compress + if(deflate(&w->response.zstream, flush) == Z_STREAM_ERROR) { + error("%llu: Compression failed. Closing down client.", w->id); + web_client_reset(w); + return(-1); + } + + w->response.zhave = ZLIB_CHUNK - w->response.zstream.avail_out; + w->response.zsent = 0; + + // keep track of the bytes passed through the compressor + w->response.sent = w->response.data->len; + + debug(D_DEFLATE, "%llu: Compression produced %zu bytes.", w->id, w->response.zhave); + + // open a new chunk + ssize_t t2 = web_client_send_chunk_header(w, w->response.zhave); + if(t2 < 0) return t2; + t += t2; + } + + debug(D_WEB_CLIENT, "%llu: Sending %zu bytes of data (+%zd of chunk header).", w->id, w->response.zhave - w->response.zsent, t); + + len = send(w->ofd, &w->response.zbuffer[w->response.zsent], (size_t) (w->response.zhave - w->response.zsent), MSG_DONTWAIT); + if(len > 0) { + w->stats_sent_bytes += len; + w->response.zsent += len; + len += t; + debug(D_WEB_CLIENT, "%llu: Sent %zd bytes.", w->id, len); + } + else if(len == 0) { + debug(D_WEB_CLIENT, "%llu: Did not send any bytes to the client (zhave = %zu, zsent = %zu, need to send = %zu).", + w->id, w->response.zhave, w->response.zsent, w->response.zhave - w->response.zsent); + + WEB_CLIENT_IS_DEAD(w); + } + else { + debug(D_WEB_CLIENT, "%llu: Failed to send data to client.", w->id); + WEB_CLIENT_IS_DEAD(w); + } + + return(len); } #endif // NETDATA_WITH_ZLIB -long web_client_send(struct web_client *w) -{ +ssize_t web_client_send(struct web_client *w) { #ifdef NETDATA_WITH_ZLIB - if(likely(w->response.zoutput)) return web_client_send_deflate(w); + if(likely(w->response.zoutput)) return web_client_send_deflate(w); #endif // NETDATA_WITH_ZLIB - long bytes; - - if(unlikely(w->response.data->len - w->response.sent == 0)) { - // there is nothing to send - - debug(D_WEB_CLIENT, "%llu: Out of output data.", w->id); - - // there can be two cases for this - // A. we have done everything - // B. we temporarily have nothing to send, waiting for the buffer to be filled by ifd - - if(w->mode == WEB_CLIENT_MODE_FILECOPY && w->wait_receive && w->ifd != w->ofd && w->response.rlen && w->response.rlen > w->response.data->len) { - // we have to wait, more data will come - debug(D_WEB_CLIENT, "%llu: Waiting for more data to become available.", w->id); - w->wait_send = 0; - return(0); - } - - if(unlikely(w->keepalive == 0)) { - debug(D_WEB_CLIENT, "%llu: Closing (keep-alive is not enabled). %ld bytes sent.", w->id, w->response.sent); - errno = 0; - return(-1); - } - - web_client_reset(w); - debug(D_WEB_CLIENT, "%llu: Done sending all data on socket. Waiting for next request on the same socket.", w->id); - return(0); - } - - bytes = send(w->ofd, &w->response.data->buffer[w->response.sent], w->response.data->len - w->response.sent, MSG_DONTWAIT); - if(likely(bytes > 0)) { - w->response.sent += bytes; - debug(D_WEB_CLIENT, "%llu: Sent %d bytes.", w->id, bytes); - } - else if(likely(bytes == 0)) debug(D_WEB_CLIENT, "%llu: Did not send any bytes to the client.", w->id); - else debug(D_WEB_CLIENT, "%llu: Failed to send data to client.", w->id); - - return(bytes); + ssize_t bytes; + + if(unlikely(w->response.data->len - w->response.sent == 0)) { + // there is nothing to send + + debug(D_WEB_CLIENT, "%llu: Out of output data.", w->id); + + // there can be two cases for this + // A. we have done everything + // B. we temporarily have nothing to send, waiting for the buffer to be filled by ifd + + if(w->mode == WEB_CLIENT_MODE_FILECOPY && w->wait_receive && w->response.rlen && w->response.rlen > w->response.data->len) { + // we have to wait, more data will come + debug(D_WEB_CLIENT, "%llu: Waiting for more data to become available.", w->id); + w->wait_send = 0; + return 0; + } + + if(unlikely(!w->keepalive)) { + debug(D_WEB_CLIENT, "%llu: Closing (keep-alive is not enabled). %zu bytes sent.", w->id, w->response.sent); + WEB_CLIENT_IS_DEAD(w); + return 0; + } + + web_client_reset(w); + debug(D_WEB_CLIENT, "%llu: Done sending all data on socket. Waiting for next request on the same socket.", w->id); + return 0; + } + + bytes = send(w->ofd, &w->response.data->buffer[w->response.sent], w->response.data->len - w->response.sent, MSG_DONTWAIT); + if(likely(bytes > 0)) { + w->stats_sent_bytes += bytes; + w->response.sent += bytes; + debug(D_WEB_CLIENT, "%llu: Sent %zd bytes.", w->id, bytes); + } + else if(likely(bytes == 0)) { + debug(D_WEB_CLIENT, "%llu: Did not send any bytes to the client.", w->id); + WEB_CLIENT_IS_DEAD(w); + } + else { + debug(D_WEB_CLIENT, "%llu: Failed to send data to client.", w->id); + WEB_CLIENT_IS_DEAD(w); + } + + return(bytes); } -long web_client_receive(struct web_client *w) +ssize_t web_client_receive(struct web_client *w) { - // do we have any space for more data? - buffer_need_bytes(w->response.data, WEB_REQUEST_LENGTH); - - long left = w->response.data->size - w->response.data->len; - long bytes; - - if(unlikely(w->mode == WEB_CLIENT_MODE_FILECOPY)) - bytes = read(w->ifd, &w->response.data->buffer[w->response.data->len], (size_t) (left - 1)); - else - bytes = recv(w->ifd, &w->response.data->buffer[w->response.data->len], (size_t) (left - 1), MSG_DONTWAIT); - - if(likely(bytes > 0)) { - size_t old = w->response.data->len; - w->response.data->len += bytes; - w->response.data->buffer[w->response.data->len] = '\0'; - - debug(D_WEB_CLIENT, "%llu: Received %d bytes.", w->id, bytes); - debug(D_WEB_DATA, "%llu: Received data: '%s'.", w->id, &w->response.data->buffer[old]); - - if(w->mode == WEB_CLIENT_MODE_FILECOPY) { - w->wait_send = 1; - if(w->response.rlen && w->response.data->len >= w->response.rlen) w->wait_receive = 0; - } - } - else if(likely(bytes == 0)) { - debug(D_WEB_CLIENT, "%llu: Out of input data.", w->id); - - // if we cannot read, it means we have an error on input. - // if however, we are copying a file from ifd to ofd, we should not return an error. - // in this case, the error should be generated when the file has been sent to the client. - - if(w->mode == WEB_CLIENT_MODE_FILECOPY) { - // we are copying data from ifd to ofd - // let it finish copying... - w->wait_receive = 0; - debug(D_WEB_CLIENT, "%llu: Disabling input.", w->id); - } - else { - bytes = -1; - errno = 0; - } - } - - return(bytes); + // do we have any space for more data? + buffer_need_bytes(w->response.data, WEB_REQUEST_LENGTH); + + ssize_t left = w->response.data->size - w->response.data->len; + ssize_t bytes; + + if(unlikely(w->mode == WEB_CLIENT_MODE_FILECOPY)) + bytes = read(w->ifd, &w->response.data->buffer[w->response.data->len], (size_t) (left - 1)); + else + bytes = recv(w->ifd, &w->response.data->buffer[w->response.data->len], (size_t) (left - 1), MSG_DONTWAIT); + + if(likely(bytes > 0)) { + if(w->mode != WEB_CLIENT_MODE_FILECOPY) + w->stats_received_bytes += bytes; + + size_t old = w->response.data->len; + w->response.data->len += bytes; + w->response.data->buffer[w->response.data->len] = '\0'; + + debug(D_WEB_CLIENT, "%llu: Received %zd bytes.", w->id, bytes); + debug(D_WEB_DATA, "%llu: Received data: '%s'.", w->id, &w->response.data->buffer[old]); + + if(w->mode == WEB_CLIENT_MODE_FILECOPY) { + w->wait_send = 1; + + if(w->response.rlen && w->response.data->len >= w->response.rlen) + w->wait_receive = 0; + } + } + else if(likely(bytes == 0)) { + debug(D_WEB_CLIENT, "%llu: Out of input data.", w->id); + + // if we cannot read, it means we have an error on input. + // if however, we are copying a file from ifd to ofd, we should not return an error. + // in this case, the error should be generated when the file has been sent to the client. + + if(w->mode == WEB_CLIENT_MODE_FILECOPY) { + // we are copying data from ifd to ofd + // let it finish copying... + w->wait_receive = 0; + + debug(D_WEB_CLIENT, "%llu: Read the whole file.", w->id); + if(w->ifd != w->ofd) close(w->ifd); + w->ifd = w->ofd; + } + else { + debug(D_WEB_CLIENT, "%llu: failed to receive data.", w->id); + WEB_CLIENT_IS_DEAD(w); + } + } + else { + debug(D_WEB_CLIENT, "%llu: receive data failed.", w->id); + WEB_CLIENT_IS_DEAD(w); + } + + return(bytes); } @@ -2040,104 +2545,131 @@ long web_client_receive(struct web_client *w) void *web_client_main(void *ptr) { - if(pthread_setcanceltype(PTHREAD_CANCEL_DEFERRED, NULL) != 0) - error("Cannot set pthread cancel type to DEFERRED."); - - if(pthread_setcancelstate(PTHREAD_CANCEL_ENABLE, NULL) != 0) - error("Cannot set pthread cancel state to ENABLE."); - - struct timeval tv; - struct web_client *w = ptr; - int retval; - fd_set ifds, ofds, efds; - int fdmax = 0; - - log_access("%llu: %s port %s connected on thread task id %d", w->id, w->client_ip, w->client_port, gettid()); - - for(;;) { - FD_ZERO (&ifds); - FD_ZERO (&ofds); - FD_ZERO (&efds); - - FD_SET(w->ifd, &efds); - - if(w->ifd != w->ofd) - FD_SET(w->ofd, &efds); - - if (w->wait_receive) { - FD_SET(w->ifd, &ifds); - if(w->ifd > fdmax) fdmax = w->ifd; - } - - if (w->wait_send) { - FD_SET(w->ofd, &ofds); - if(w->ofd > fdmax) fdmax = w->ofd; - } - - tv.tv_sec = web_client_timeout; - tv.tv_usec = 0; - - debug(D_WEB_CLIENT, "%llu: Waiting socket async I/O for %s %s", w->id, w->wait_receive?"INPUT":"", w->wait_send?"OUTPUT":""); - retval = select(fdmax+1, &ifds, &ofds, &efds, &tv); - - if(retval == -1) { - debug(D_WEB_CLIENT_ACCESS, "%llu: LISTENER: select() failed.", w->id); - continue; - } - else if(!retval) { - // timeout - debug(D_WEB_CLIENT_ACCESS, "%llu: LISTENER: timeout.", w->id); - break; - } - - if(FD_ISSET(w->ifd, &efds)) { - debug(D_WEB_CLIENT_ACCESS, "%llu: Received error on input socket.", w->id); - break; - } - - if(FD_ISSET(w->ofd, &efds)) { - debug(D_WEB_CLIENT_ACCESS, "%llu: Received error on output socket.", w->id); - break; - } - - if(w->wait_send && FD_ISSET(w->ofd, &ofds)) { - long bytes; - if((bytes = web_client_send(w)) < 0) { - debug(D_WEB_CLIENT, "%llu: Cannot send data to client. Closing client.", w->id); - errno = 0; - break; - } - - global_statistics_lock(); - global_statistics.bytes_sent += bytes; - global_statistics_unlock(); - } - - if(w->wait_receive && FD_ISSET(w->ifd, &ifds)) { - long bytes; - if((bytes = web_client_receive(w)) < 0) { - debug(D_WEB_CLIENT, "%llu: Cannot receive data from client. Closing client.", w->id); - errno = 0; - break; - } - - if(w->mode == WEB_CLIENT_MODE_NORMAL) { - debug(D_WEB_CLIENT, "%llu: Attempting to process received data (%ld bytes).", w->id, bytes); - // info("%llu: Attempting to process received data (%ld bytes).", w->id, bytes); - web_client_process(w); - } - - global_statistics_lock(); - global_statistics.bytes_received += bytes; - global_statistics_unlock(); - } - } - - log_access("%llu: %s port %s disconnected from thread task id %d", w->id, w->client_ip, w->client_port, gettid()); - debug(D_WEB_CLIENT, "%llu: done...", w->id); - - web_client_reset(w); - w->obsolete = 1; - - return NULL; + if(pthread_setcanceltype(PTHREAD_CANCEL_DEFERRED, NULL) != 0) + error("Cannot set pthread cancel type to DEFERRED."); + + if(pthread_setcancelstate(PTHREAD_CANCEL_ENABLE, NULL) != 0) + error("Cannot set pthread cancel state to ENABLE."); + + struct web_client *w = ptr; + struct pollfd fds[2], *ifd, *ofd; + int retval, fdmax = 0, timeout; + + log_access("%llu: %s port %s connected on thread task id %d", w->id, w->client_ip, w->client_port, gettid()); + + for(;;) { + if(unlikely(w->dead)) { + debug(D_WEB_CLIENT, "%llu: client is dead.", w->id); + break; + } + else if(unlikely(!w->wait_receive && !w->wait_send)) { + debug(D_WEB_CLIENT, "%llu: client is not set for neither receiving nor sending data.", w->id); + break; + } + + if(unlikely(w->ifd < 0 || w->ofd < 0)) { + error("%llu: invalid file descriptor, ifd = %d, ofd = %d (required 0 <= fd", w->id, w->ifd, w->ofd); + break; + } + + if(w->ifd == w->ofd) { + fds[0].fd = w->ifd; + fds[0].events = 0; + fds[0].revents = 0; + + if(w->wait_receive) fds[0].events |= POLLIN; + if(w->wait_send) fds[0].events |= POLLOUT; + + fds[1].fd = -1; + fds[1].events = 0; + fds[1].revents = 0; + + ifd = ofd = &fds[0]; + + fdmax = 1; + } + else { + fds[0].fd = w->ifd; + fds[0].events = 0; + fds[0].revents = 0; + if(w->wait_receive) fds[0].events |= POLLIN; + ifd = &fds[0]; + + fds[1].fd = w->ofd; + fds[1].events = 0; + fds[1].revents = 0; + if(w->wait_send) fds[1].events |= POLLOUT; + ofd = &fds[1]; + + fdmax = 2; + } + + debug(D_WEB_CLIENT, "%llu: Waiting socket async I/O for %s %s", w->id, w->wait_receive?"INPUT":"", w->wait_send?"OUTPUT":""); + errno = 0; + timeout = web_client_timeout * 1000; + retval = poll(fds, fdmax, timeout); + + if(unlikely(retval == -1)) { + if(errno == EAGAIN || errno == EINTR) { + debug(D_WEB_CLIENT, "%llu: EAGAIN received.", w->id); + continue; + } + + debug(D_WEB_CLIENT, "%llu: LISTENER: poll() failed (input fd = %d, output fd = %d). Closing client.", w->id, w->ifd, w->ofd); + break; + } + else if(unlikely(!retval)) { + debug(D_WEB_CLIENT, "%llu: Timeout while waiting socket async I/O for %s %s", w->id, w->wait_receive?"INPUT":"", w->wait_send?"OUTPUT":""); + break; + } + + int used = 0; + if(w->wait_send && ofd->revents & POLLOUT) { + used++; + if(web_client_send(w) < 0) { + debug(D_WEB_CLIENT, "%llu: Cannot send data to client. Closing client.", w->id); + break; + } + } + + if(w->wait_receive && (ifd->revents & POLLIN || ifd->revents & POLLPRI)) { + used++; + if(web_client_receive(w) < 0) { + debug(D_WEB_CLIENT, "%llu: Cannot receive data from client. Closing client.", w->id); + break; + } + + if(w->mode == WEB_CLIENT_MODE_NORMAL) { + debug(D_WEB_CLIENT, "%llu: Attempting to process received data.", w->id); + web_client_process(w); + } + } + + if(unlikely(!used)) { + debug(D_WEB_CLIENT_ACCESS, "%llu: Received error on socket.", w->id); + break; + } + } + + web_client_reset(w); + + log_access("%llu: %s port %s disconnected from thread task id %d", w->id, w->client_ip, w->client_port, gettid()); + debug(D_WEB_CLIENT, "%llu: done...", w->id); + + // close the sockets/files now + // to free file descriptors + if(w->ifd == w->ofd) { + if(w->ifd != -1) close(w->ifd); + } + else { + if(w->ifd != -1) close(w->ifd); + if(w->ofd != -1) close(w->ofd); + } + w->ifd = -1; + w->ofd = -1; + + w->obsolete = 1; + + pthread_exit(NULL); + return NULL; } diff --git a/src/web_client.h b/src/web_client.h index f663be4a1..2555a0c24 100644 --- a/src/web_client.h +++ b/src/web_client.h @@ -1,92 +1,94 @@ - -#ifdef NETDATA_WITH_ZLIB -#include <zlib.h> -#endif - -#include <sys/time.h> -#include <string.h> -#include <sys/socket.h> -#include <netinet/in.h> -#include <arpa/inet.h> -#include <netdb.h> - -#include "web_buffer.h" -#include "dictionary.h" +#ifndef NETDATA_WEB_CLIENT_H +#define NETDATA_WEB_CLIENT_H 1 #define DEFAULT_DISCONNECT_IDLE_WEB_CLIENTS_AFTER_SECONDS 60 extern int web_client_timeout; -extern int web_enable_gzip; -#ifndef NETDATA_WEB_CLIENT_H -#define NETDATA_WEB_CLIENT_H 1 +#ifdef NETDATA_WITH_ZLIB +extern int web_enable_gzip, web_gzip_level, web_gzip_strategy, web_donotrack_comply; +#endif /* NETDATA_WITH_ZLIB */ -#define WEB_CLIENT_MODE_NORMAL 0 -#define WEB_CLIENT_MODE_FILECOPY 1 -#define WEB_CLIENT_MODE_OPTIONS 2 +#define WEB_CLIENT_MODE_NORMAL 0 +#define WEB_CLIENT_MODE_FILECOPY 1 +#define WEB_CLIENT_MODE_OPTIONS 2 #define URL_MAX 8192 -#define ZLIB_CHUNK 16384 +#define ZLIB_CHUNK 16384 #define HTTP_RESPONSE_HEADER_SIZE 4096 #define COOKIE_MAX 1024 #define ORIGIN_MAX 1024 struct response { - BUFFER *header; // our response header - BUFFER *header_output; // internal use - BUFFER *data; // our response data buffer + BUFFER *header; // our response header + BUFFER *header_output; // internal use + BUFFER *data; // our response data buffer - int code; // the HTTP response code + int code; // the HTTP response code - size_t rlen; // if non-zero, the excepted size of ifd (input) - size_t sent; // current data length sent to output + size_t rlen; // if non-zero, the excepted size of ifd (input of firecopy) + size_t sent; // current data length sent to output - int zoutput; // if set to 1, web_client_send() will send compressed data + int zoutput; // if set to 1, web_client_send() will send compressed data #ifdef NETDATA_WITH_ZLIB - z_stream zstream; // zlib stream for sending compressed output to client - Bytef zbuffer[ZLIB_CHUNK]; // temporary buffer for storing compressed output - long zsent; // the compressed bytes we have sent to the client - long zhave; // the compressed bytes that we have to send - int zinitialized; -#endif + z_stream zstream; // zlib stream for sending compressed output to client + Bytef zbuffer[ZLIB_CHUNK]; // temporary buffer for storing compressed output + size_t zsent; // the compressed bytes we have sent to the client + size_t zhave; // the compressed bytes that we have received from zlib + int zinitialized:1; +#endif /* NETDATA_WITH_ZLIB */ }; struct web_client { - unsigned long long id; + unsigned long long id; + + uint8_t obsolete:1; // if set to 1, the listener will remove this client + // after setting this to 1, you should not touch + // this web_client + + uint8_t dead:1; // if set to 1, this client is dead + + uint8_t keepalive:1; // if set to 1, the web client will be re-used - char client_ip[NI_MAXHOST+1]; - char client_port[NI_MAXSERV+1]; + uint8_t mode:3; // the operational mode of the client - char last_url[URL_MAX+1]; + uint8_t wait_receive:1; // 1 = we are waiting more input data + uint8_t wait_send:1; // 1 = we have data to send to the client - struct timeval tv_in, tv_ready; + uint8_t donottrack:1; // 1 = we should not set cookies on this client + uint8_t tracking_required:1; // 1 = if the request requires cookies - char cookie1[COOKIE_MAX+1]; - char cookie2[COOKIE_MAX+1]; - char origin[ORIGIN_MAX+1]; + int tcp_cork; // 1 = we have a cork on the socket - int mode; - int keepalive; - int enable_gzip; - char *decoded_url; + int ifd; + int ofd; - struct sockaddr_storage clientaddr; + char client_ip[NI_MAXHOST+1]; + char client_port[NI_MAXSERV+1]; - pthread_t thread; // the thread servicing this client - int obsolete; // if set to 1, the listener will remove this client + char decoded_url[URL_MAX + 1]; // we decode the URL in this buffer + char last_url[URL_MAX+1]; // we keep a copy of the decoded URL here - int ifd; - int ofd; + struct timeval tv_in, tv_ready; - struct response response; + char cookie1[COOKIE_MAX+1]; + char cookie2[COOKIE_MAX+1]; + char origin[ORIGIN_MAX+1]; - int wait_receive; - int wait_send; + struct sockaddr_storage clientaddr; + struct response response; - struct web_client *prev; - struct web_client *next; + size_t stats_received_bytes; + size_t stats_sent_bytes; + + pthread_t thread; // the thread servicing this client + + struct web_client *prev; + struct web_client *next; }; +#define WEB_CLIENT_IS_DEAD(w) (w)->dead=1 + extern struct web_client *web_clients; extern uid_t web_files_uid(void); @@ -94,7 +96,15 @@ extern uid_t web_files_gid(void); extern struct web_client *web_client_create(int listener); extern struct web_client *web_client_free(struct web_client *w); +extern ssize_t web_client_send(struct web_client *w); +extern ssize_t web_client_receive(struct web_client *w); +extern void web_client_process(struct web_client *w); +extern void web_client_reset(struct web_client *w); extern void *web_client_main(void *ptr); +extern int web_client_api_request_v1_data_group(char *name, int def); +extern const char *group_method2string(int group); + +extern void buffer_data_options2string(BUFFER *wb, uint32_t options); #endif diff --git a/src/web_server.c b/src/web_server.c index 0da72b5be..cf3687f3e 100644 --- a/src/web_server.c +++ b/src/web_server.c @@ -1,270 +1,601 @@ -#ifdef HAVE_CONFIG_H -#include <config.h> -#endif -#include <unistd.h> -#include <stdlib.h> -#include <sys/types.h> -#include <sys/socket.h> -#include <netinet/in.h> -#include <arpa/inet.h> -#include <errno.h> -#include <pthread.h> -#include <sys/stat.h> -#include <fcntl.h> -#include <netinet/tcp.h> -#include <malloc.h> - #include "common.h" -#include "log.h" -#include "appconfig.h" -#include "url.h" -#include "web_buffer.h" -#include "web_client.h" -#include "web_server.h" -#include "global_statistics.h" -#include "rrd.h" -#include "rrd2json.h" -#include "../config.h" int listen_backlog = LISTEN_BACKLOG; - -int listen_fd = -1; +size_t listen_fds_count = 0; +int listen_fds[MAX_LISTEN_FDS] = { [0 ... 99] = -1 }; +char *listen_fds_names[MAX_LISTEN_FDS] = { [0 ... 99] = NULL }; int listen_port = LISTEN_PORT; +int web_server_mode = WEB_SERVER_MODE_MULTI_THREADED; #ifdef NETDATA_INTERNAL_CHECKS static void log_allocations(void) { - static int mem = 0; + static int mem = 0; - struct mallinfo mi; + struct mallinfo mi; - mi = mallinfo(); - if(mi.uordblks > mem) { - int clients = 0; - struct web_client *w; - for(w = web_clients; w ; w = w->next) clients++; + mi = mallinfo(); + if(mi.uordblks > mem) { + int clients = 0; + struct web_client *w; + for(w = web_clients; w ; w = w->next) clients++; - info("Allocated memory increased from %d to %d (increased by %d bytes). There are %d web clients connected.", mem, mi.uordblks, mi.uordblks - mem, clients); - mem = mi.uordblks; - } + info("Allocated memory increased from %d to %d (increased by %d bytes). There are %d web clients connected.", mem, mi.uordblks, mi.uordblks - mem, clients); + mem = mi.uordblks; + } } #endif -static int is_ip_anything(const char *ip) -{ - if(!ip || !*ip - || !strcmp(ip, "any") - || !strcmp(ip, "all") - || !strcmp(ip, "*") - || !strcmp(ip, "::") - || !strcmp(ip, "0.0.0.0") - ) return 1; - - return 0; +#ifndef HAVE_ACCEPT4 +int accept4(int sock, struct sockaddr *addr, socklen_t *addrlen, int flags) { + int fd = accept(sock, addr, addrlen); + int newflags = 0; + + if (fd < 0) return fd; + + if (flags & SOCK_NONBLOCK) { + newflags |= O_NONBLOCK; + flags &= ~SOCK_NONBLOCK; + } + + if (flags & SOCK_CLOEXEC) { + newflags |= O_CLOEXEC; + flags &= ~SOCK_CLOEXEC; + } + + if (flags) { + errno = -EINVAL; + return -1; + } + + if (fcntl(fd, F_SETFL, newflags) < 0) { + int saved_errno = errno; + close(fd); + errno = saved_errno; + return -1; + } + + return fd; } +#endif -int create_listen_socket4(const char *ip, int port, int listen_backlog) -{ - int sock; - int sockopt = 1; - - debug(D_LISTENER, "IPv4 creating new listening socket on port %d", port); - - sock = socket(AF_INET, SOCK_STREAM, 0); - if(sock < 0) { - error("IPv4 socket() failed."); - return -1; - } - - /* avoid "address already in use" */ - setsockopt(sock, SOL_SOCKET, SO_REUSEADDR, (void*)&sockopt, sizeof(sockopt)); - - struct sockaddr_in name; - memset(&name, 0, sizeof(struct sockaddr_in)); - name.sin_family = AF_INET; - name.sin_port = htons (port); - - if(is_ip_anything(ip)) { - name.sin_addr.s_addr = htonl(INADDR_ANY); - info("Listening on any IPs (IPv4)."); - } - else { - int ret = inet_pton(AF_INET, ip, (void *)&name.sin_addr.s_addr); - if(ret != 1) { - error("Failed to convert IP '%s' to a valid IPv4 address.", ip); - close(sock); - return -1; - } - info("Listening on IP '%s' (IPv4).", ip); - } - - if(bind (sock, (struct sockaddr *) &name, sizeof (name)) < 0) { - close(sock); - error("IPv4 bind() failed."); - return -1; - } - - if(listen(sock, listen_backlog) < 0) { - close(sock); - fatal("IPv4 listen() failed."); - return -1; - } - - debug(D_LISTENER, "IPv4 listening port %d created", port); - return sock; +int create_listen_socket4(const char *ip, int port, int listen_backlog) { + int sock; + int sockopt = 1; + + debug(D_LISTENER, "IPv4 creating new listening socket on ip '%s' port %d", ip, port); + + sock = socket(AF_INET, SOCK_STREAM, 0); + if(sock < 0) { + error("IPv4 socket() on ip '%s' port %d failed.", ip, port); + return -1; + } + + /* avoid "address already in use" */ + if(setsockopt(sock, SOL_SOCKET, SO_REUSEADDR, (void*)&sockopt, sizeof(sockopt)) != 0) + error("Cannot set SO_REUSEADDR on ip '%s' port's %d.", ip, port); + + struct sockaddr_in name; + memset(&name, 0, sizeof(struct sockaddr_in)); + name.sin_family = AF_INET; + name.sin_port = htons (port); + + int ret = inet_pton(AF_INET, ip, (void *)&name.sin_addr.s_addr); + if(ret != 1) { + error("Failed to convert IP '%s' to a valid IPv4 address.", ip); + close(sock); + return -1; + } + + if(bind (sock, (struct sockaddr *) &name, sizeof (name)) < 0) { + close(sock); + error("IPv4 bind() on ip '%s' port %d failed.", ip, port); + return -1; + } + + if(listen(sock, listen_backlog) < 0) { + close(sock); + fatal("IPv4 listen() on ip '%s' port %d failed.", ip, port); + return -1; + } + + debug(D_LISTENER, "Listening on IPv4 ip '%s' port %d", ip, port); + return sock; } -int create_listen_socket6(const char *ip, int port, int listen_backlog) -{ - int sock = -1; - int sockopt = 1; - - debug(D_LISTENER, "IPv6 creating new listening socket on port %d", port); - - sock = socket(AF_INET6, SOCK_STREAM, 0); - if (sock < 0) { - error("IPv6 socket() failed. Disabling IPv6."); - return -1; - } - - /* avoid "address already in use" */ - setsockopt(sock, SOL_SOCKET, SO_REUSEADDR, (void*)&sockopt, sizeof(sockopt)); - - struct sockaddr_in6 name; - memset(&name, 0, sizeof(struct sockaddr_in6)); - name.sin6_family = AF_INET6; - name.sin6_port = htons ((uint16_t) port); - - if(is_ip_anything(ip)) { - name.sin6_addr = in6addr_any; - info("Listening on all IPs (IPv6 and IPv4)"); - } - else { - int ret = inet_pton(AF_INET6, ip, (void *)&name.sin6_addr.s6_addr); - if(ret != 1) { - error("Failed to convert IP '%s' to a valid IPv6 address. Disabling IPv6.", ip); - close(sock); - return -1; - } - info("Listening on IP '%s' (IPv6)", ip); - } - - name.sin6_scope_id = 0; - - if (bind (sock, (struct sockaddr *) &name, sizeof (name)) < 0) { - close(sock); - error("IPv6 bind() failed. Disabling IPv6."); - return -1; - } - - if (listen(sock, listen_backlog) < 0) { - close(sock); - error("IPv6 listen() failed. Disabling IPv6."); - return -1; - } - - debug(D_LISTENER, "IPv6 listening port %d created", port); - return sock; +int create_listen_socket6(const char *ip, int port, int listen_backlog) { + int sock = -1; + int sockopt = 1; + int ipv6only = 1; + + debug(D_LISTENER, "IPv6 creating new listening socket on ip '%s' port %d", ip, port); + + sock = socket(AF_INET6, SOCK_STREAM, 0); + if (sock < 0) { + error("IPv6 socket() on ip '%s' port %d failed.", ip, port); + return -1; + } + + /* avoid "address already in use" */ + if(setsockopt(sock, SOL_SOCKET, SO_REUSEADDR, (void*)&sockopt, sizeof(sockopt)) != 0) + error("Cannot set SO_REUSEADDR on ip '%s' port's %d.", ip, port); + + /* IPv6 only */ + if(setsockopt(sock, IPPROTO_IPV6, IPV6_V6ONLY, (void*)&ipv6only, sizeof(ipv6only)) != 0) + error("Cannot set IPV6_V6ONLY on ip '%s' port's %d.", ip, port); + + struct sockaddr_in6 name; + memset(&name, 0, sizeof(struct sockaddr_in6)); + name.sin6_family = AF_INET6; + name.sin6_port = htons ((uint16_t) port); + + int ret = inet_pton(AF_INET6, ip, (void *)&name.sin6_addr.s6_addr); + if(ret != 1) { + error("Failed to convert IP '%s' to a valid IPv6 address.", ip); + close(sock); + return -1; + } + + name.sin6_scope_id = 0; + + if (bind (sock, (struct sockaddr *) &name, sizeof (name)) < 0) { + close(sock); + error("IPv6 bind() on ip '%s' port %d failed.", ip, port); + return -1; + } + + if (listen(sock, listen_backlog) < 0) { + close(sock); + error("IPv6 listen() on ip '%s' port %d failed.", ip, port); + return -1; + } + + debug(D_LISTENER, "Listening on IPv6 ip '%s' port %d", ip, port); + return sock; +} + +static inline int add_listen_socket(int fd, const char *ip, int port) { + if(listen_fds_count >= MAX_LISTEN_FDS) { + error("Too many listening sockets. Failed to add listening socket at ip '%s' port %d", ip, port); + close(fd); + return -1; + } + + listen_fds[listen_fds_count] = fd; + + char buffer[100 + 1]; + snprintfz(buffer, 100, "[%s]:%d", ip, port); + listen_fds_names[listen_fds_count] = strdupz(buffer); + + listen_fds_count++; + return 0; +} + +int is_listen_socket(int fd) { + size_t i; + for(i = 0; i < listen_fds_count ;i++) + if(listen_fds[i] == fd) return 1; + + return 0; +} + +static inline void close_listen_sockets(void) { + size_t i; + for(i = 0; i < listen_fds_count ;i++) { + close(listen_fds[i]); + listen_fds[i] = -1; + + freez(listen_fds_names[i]); + listen_fds_names[i] = NULL; + } + + listen_fds_count = 0; +} + +static inline int bind_to_one(const char *definition, int default_port, int listen_backlog) { + int added = 0; + struct addrinfo hints; + struct addrinfo *result = NULL, *rp = NULL; + + char buffer[strlen(definition) + 1]; + strcpy(buffer, definition); + + char buffer2[10 + 1]; + snprintfz(buffer2, 10, "%d", default_port); + + char *ip = buffer, *port = buffer2; + + char *e = ip; + if(*e == '[') { + e = ++ip; + while(*e && *e != ']') e++; + if(*e == ']') { + *e = '\0'; + e++; + } + } + else { + while(*e && *e != ':') e++; + } + + if(*e == ':') { + port = e + 1; + *e = '\0'; + } + + if(!*ip || *ip == '*' || !strcmp(ip, "any") || !strcmp(ip, "all")) + ip = NULL; + if(!*port) + port = buffer2; + + memset(&hints, 0, sizeof(struct addrinfo)); + hints.ai_family = AF_UNSPEC; /* Allow IPv4 or IPv6 */ + hints.ai_socktype = SOCK_DGRAM; /* Datagram socket */ + hints.ai_flags = AI_PASSIVE; /* For wildcard IP address */ + hints.ai_protocol = 0; /* Any protocol */ + hints.ai_canonname = NULL; + hints.ai_addr = NULL; + hints.ai_next = NULL; + + int r = getaddrinfo(ip, port, &hints, &result); + if (r != 0) { + error("getaddrinfo('%s', '%s'): %s\n", ip, port, gai_strerror(r)); + return -1; + } + + for (rp = result; rp != NULL; rp = rp->ai_next) { + int fd = -1; + + char rip[INET_ADDRSTRLEN + INET6_ADDRSTRLEN] = "INVALID"; + int rport = default_port; + + switch (rp->ai_addr->sa_family) { + case AF_INET: { + struct sockaddr_in *sin = (struct sockaddr_in *) rp->ai_addr; + inet_ntop(AF_INET, &sin->sin_addr, rip, INET_ADDRSTRLEN); + rport = ntohs(sin->sin_port); + fd = create_listen_socket4(rip, rport, listen_backlog); + break; + } + + case AF_INET6: { + struct sockaddr_in6 *sin6 = (struct sockaddr_in6 *) rp->ai_addr; + inet_ntop(AF_INET6, &sin6->sin6_addr, rip, INET6_ADDRSTRLEN); + rport = ntohs(sin6->sin6_port); + fd = create_listen_socket6(rip, rport, listen_backlog); + break; + } + } + + if (fd == -1) + error("Cannot bind to ip '%s', port %d", rip, default_port); + else { + add_listen_socket(fd, rip, rport); + added++; + } + } + + freeaddrinfo(result); + + return added; } +int create_listen_sockets(void) { + listen_backlog = (int) config_get_number("global", "http port listen backlog", LISTEN_BACKLOG); + + if(config_exists("global", "bind socket to IP") && !config_exists("global", "bind to")) + config_rename("global", "bind socket to IP", "bind to"); + + if(config_exists("global", "port") && !config_exists("global", "default port")) + config_rename("global", "port", "default port"); + + listen_port = (int) config_get_number("global", "default port", LISTEN_PORT); + if(listen_port < 1 || listen_port > 65535) { + error("Invalid listen port %d given. Defaulting to %d.", listen_port, LISTEN_PORT); + listen_port = (int) config_set_number("global", "default port", LISTEN_PORT); + } + debug(D_OPTIONS, "Default listen port set to %d.", listen_port); + + char *s = config_get("global", "bind to", "*"); + while(*s) { + char *e = s; + + // skip separators, moving both s(tart) and e(nd) + while(isspace(*e) || *e == ',') s = ++e; + + // move e(nd) to the first separator + while(*e && !isspace(*e) && *e != ',') e++; + + // is there anything? + if(!*s || s == e) break; + + char buf[e - s + 1]; + strncpyz(buf, s, e - s); + bind_to_one(buf, listen_port, listen_backlog); + + s = e; + } + + if(!listen_fds_count) + fatal("Cannot listen on any socket. Exiting..."); + + return (int)listen_fds_count; +} // -------------------------------------------------------------------------------------- // the main socket listener +static inline void cleanup_web_clients(void) { + struct web_client *w; + + for (w = web_clients; w;) { + if (w->obsolete) { + debug(D_WEB_CLIENT, "%llu: Removing client.", w->id); + // pthread_cancel(w->thread); + // pthread_join(w->thread, NULL); + w = web_client_free(w); +#ifdef NETDATA_INTERNAL_CHECKS + log_allocations(); +#endif + } + else w = w->next; + } +} + // 1. it accepts new incoming requests on our port // 2. creates a new web_client for each connection received // 3. spawns a new pthread to serve the client (this is optimal for keep-alive clients) // 4. cleans up old web_clients that their pthreads have been exited -void *socket_listen_main(void *ptr) -{ - if(ptr) { ; } - - info("WEB SERVER thread created with task id %d", gettid()); - - struct web_client *w; - struct timeval tv; - int retval; - - if(ptr) { ; } - - if(pthread_setcanceltype(PTHREAD_CANCEL_DEFERRED, NULL) != 0) - error("Cannot set pthread cancel type to DEFERRED."); - - if(pthread_setcancelstate(PTHREAD_CANCEL_ENABLE, NULL) != 0) - error("Cannot set pthread cancel state to ENABLE."); - - web_client_timeout = (int) config_get_number("global", "disconnect idle web clients after seconds", DEFAULT_DISCONNECT_IDLE_WEB_CLIENTS_AFTER_SECONDS); - web_enable_gzip = config_get_boolean("global", "enable web responses gzip compression", web_enable_gzip); - - if(listen_fd < 0) fatal("LISTENER: Listen socket is not ready."); - - fd_set ifds, ofds, efds; - int fdmax = listen_fd; - - FD_ZERO (&ifds); - FD_ZERO (&ofds); - FD_ZERO (&efds); - - for(;;) { - tv.tv_sec = 0; - tv.tv_usec = 200000; - - if(listen_fd >= 0) { - FD_SET(listen_fd, &ifds); - FD_SET(listen_fd, &efds); - } - - // debug(D_WEB_CLIENT, "LISTENER: Waiting..."); - retval = select(fdmax+1, &ifds, &ofds, &efds, &tv); - - if(retval == -1) { - error("LISTENER: select() failed."); - continue; - } - else if(retval) { - // check for new incoming connections - if(FD_ISSET(listen_fd, &ifds)) { - w = web_client_create(listen_fd); - if(unlikely(!w)) { - // no need for error log - web_client_create already logged the error - continue; - } - - if(pthread_create(&w->thread, NULL, web_client_main, w) != 0) { - error("%llu: failed to create new thread for web client."); - w->obsolete = 1; - } - else if(pthread_detach(w->thread) != 0) { - error("%llu: Cannot request detach of newly created web client thread.", w->id); - w->obsolete = 1; - } - } - else debug(D_WEB_CLIENT, "LISTENER: select() didn't do anything."); - - } - //else { - // debug(D_WEB_CLIENT, "LISTENER: select() timeout."); - //} - - // cleanup unused clients - for(w = web_clients; w ; w = w?w->next:NULL) { - if(w->obsolete) { - debug(D_WEB_CLIENT, "%llu: Removing client.", w->id); - // pthread_join(w->thread, NULL); - w = web_client_free(w); -#ifdef NETDATA_INTERNAL_CHECKS - log_allocations(); -#endif - } - } - } +#define CLEANUP_EVERY_EVENTS 100 + +void *socket_listen_main_multi_threaded(void *ptr) { + (void)ptr; + + web_server_mode = WEB_SERVER_MODE_MULTI_THREADED; + info("Multi-threaded WEB SERVER thread created with task id %d", gettid()); + + struct web_client *w; + int retval, counter = 0; + + if(pthread_setcanceltype(PTHREAD_CANCEL_DEFERRED, NULL) != 0) + error("Cannot set pthread cancel type to DEFERRED."); + + if(pthread_setcancelstate(PTHREAD_CANCEL_ENABLE, NULL) != 0) + error("Cannot set pthread cancel state to ENABLE."); + + if(!listen_fds_count) + fatal("LISTENER: No sockets to listen to."); + + struct pollfd *fds = callocz(sizeof(struct pollfd), listen_fds_count); + + size_t i; + for(i = 0; i < listen_fds_count ;i++) { + fds[i].fd = listen_fds[i]; + fds[i].events = POLLIN; + fds[i].revents = 0; + + info("Listening on '%s'", (listen_fds_names[i])?listen_fds_names[i]:"UNKNOWN"); + } + + int timeout = 10 * 1000; + + for(;;) { + // debug(D_WEB_CLIENT, "LISTENER: Waiting..."); + retval = poll(fds, listen_fds_count, timeout); + + if(unlikely(retval == -1)) { + error("LISTENER: poll() failed."); + continue; + } + else if(unlikely(!retval)) { + debug(D_WEB_CLIENT, "LISTENER: select() timeout."); + counter = 0; + cleanup_web_clients(); + continue; + } + + for(i = 0 ; i < listen_fds_count ; i++) { + short int revents = fds[i].revents; + + // check for new incoming connections + if(revents & POLLIN || revents & POLLPRI) { + fds[i].revents = 0; + + w = web_client_create(fds[i].fd); + if(unlikely(!w)) { + // no need for error log - web_client_create already logged the error + continue; + } + + if(pthread_create(&w->thread, NULL, web_client_main, w) != 0) { + error("%llu: failed to create new thread for web client.", w->id); + w->obsolete = 1; + } + else if(pthread_detach(w->thread) != 0) { + error("%llu: Cannot request detach of newly created web client thread.", w->id); + w->obsolete = 1; + } + } + } + + // cleanup unused clients + counter++; + if(counter >= CLEANUP_EVERY_EVENTS) { + counter = 0; + cleanup_web_clients(); + } + } + + debug(D_WEB_CLIENT, "LISTENER: exit!"); + close_listen_sockets(); + + return NULL; +} - error("LISTENER: exit!"); +struct web_client *single_threaded_clients[FD_SETSIZE]; - if(listen_fd >= 0) close(listen_fd); - exit(2); +static inline int single_threaded_link_client(struct web_client *w, fd_set *ifds, fd_set *ofds, fd_set *efds, int *max) { + if(unlikely(w->obsolete || w->dead || (!w->wait_receive && !w->wait_send))) + return 1; - return NULL; + if(unlikely(w->ifd < 0 || w->ifd >= FD_SETSIZE || w->ofd < 0 || w->ofd >= FD_SETSIZE)) { + error("%llu: invalid file descriptor, ifd = %d, ofd = %d (required 0 <= fd < FD_SETSIZE (%d)", w->id, w->ifd, w->ofd, FD_SETSIZE); + return 1; + } + + FD_SET(w->ifd, efds); + if(unlikely(*max < w->ifd)) *max = w->ifd; + + if(unlikely(w->ifd != w->ofd)) { + if(*max < w->ofd) *max = w->ofd; + FD_SET(w->ofd, efds); + } + + if(w->wait_receive) FD_SET(w->ifd, ifds); + if(w->wait_send) FD_SET(w->ofd, ofds); + + single_threaded_clients[w->ifd] = w; + single_threaded_clients[w->ofd] = w; + + return 0; } +static inline int single_threaded_unlink_client(struct web_client *w, fd_set *ifds, fd_set *ofds, fd_set *efds) { + FD_CLR(w->ifd, efds); + if(unlikely(w->ifd != w->ofd)) FD_CLR(w->ofd, efds); + + if(w->wait_receive) FD_CLR(w->ifd, ifds); + if(w->wait_send) FD_CLR(w->ofd, ofds); + + single_threaded_clients[w->ifd] = NULL; + single_threaded_clients[w->ofd] = NULL; + + if(unlikely(w->obsolete || w->dead || (!w->wait_receive && !w->wait_send))) + return 1; + + return 0; +} + +void *socket_listen_main_single_threaded(void *ptr) { + (void)ptr; + + web_server_mode = WEB_SERVER_MODE_SINGLE_THREADED; + + info("Single-threaded WEB SERVER thread created with task id %d", gettid()); + + struct web_client *w; + int retval; + + if(pthread_setcanceltype(PTHREAD_CANCEL_DEFERRED, NULL) != 0) + error("Cannot set pthread cancel type to DEFERRED."); + + if(pthread_setcancelstate(PTHREAD_CANCEL_ENABLE, NULL) != 0) + error("Cannot set pthread cancel state to ENABLE."); + + if(!listen_fds_count) + fatal("LISTENER: no listen sockets available."); + + size_t i; + for(i = 0; i < FD_SETSIZE ; i++) + single_threaded_clients[i] = NULL; + + fd_set ifds, ofds, efds, rifds, rofds, refds; + FD_ZERO (&ifds); + FD_ZERO (&ofds); + FD_ZERO (&efds); + int fdmax = 0; + + for(i = 0; i < listen_fds_count ; i++) { + if (listen_fds[i] < 0 || listen_fds[i] >= FD_SETSIZE) + fatal("LISTENER: Listen socket %d is not ready, or invalid.", listen_fds[i]); + + info("Listening on '%s'", (listen_fds_names[i])?listen_fds_names[i]:"UNKNOWN"); + + FD_SET(listen_fds[i], &ifds); + FD_SET(listen_fds[i], &efds); + if(fdmax < listen_fds[i]) + fdmax = listen_fds[i]; + } + + for(;;) { + debug(D_WEB_CLIENT_ACCESS, "LISTENER: single threaded web server waiting (fdmax = %d)...", fdmax); + + struct timeval tv = { .tv_sec = 1, .tv_usec = 0 }; + rifds = ifds; + rofds = ofds; + refds = efds; + retval = select(fdmax+1, &rifds, &rofds, &refds, &tv); + + if(unlikely(retval == -1)) { + error("LISTENER: select() failed."); + continue; + } + else if(likely(retval)) { + debug(D_WEB_CLIENT_ACCESS, "LISTENER: got something."); + + for(i = 0; i < listen_fds_count ; i++) { + if (FD_ISSET(listen_fds[i], &rifds)) { + debug(D_WEB_CLIENT_ACCESS, "LISTENER: new connection."); + w = web_client_create(listen_fds[i]); + if (single_threaded_link_client(w, &ifds, &ofds, &ifds, &fdmax) != 0) { + web_client_free(w); + } + } + } + + for(i = 0 ; i <= (size_t)fdmax ; i++) { + if(likely(!FD_ISSET(i, &rifds) && !FD_ISSET(i, &rofds) && !FD_ISSET(i, &refds))) + continue; + + w = single_threaded_clients[i]; + if(unlikely(!w)) + continue; + + if(unlikely(single_threaded_unlink_client(w, &ifds, &ofds, &efds) != 0)) { + web_client_free(w); + continue; + } + + if (unlikely(FD_ISSET(w->ifd, &refds) || FD_ISSET(w->ofd, &refds))) { + web_client_free(w); + continue; + } + + if (unlikely(w->wait_receive && FD_ISSET(w->ifd, &rifds))) { + if (unlikely(web_client_receive(w) < 0)) { + web_client_free(w); + continue; + } + + if (w->mode != WEB_CLIENT_MODE_FILECOPY) { + debug(D_WEB_CLIENT, "%llu: Processing received data.", w->id); + web_client_process(w); + } + } + + if (unlikely(w->wait_send && FD_ISSET(w->ofd, &rofds))) { + if (unlikely(web_client_send(w) < 0)) { + debug(D_WEB_CLIENT, "%llu: Cannot send data to client. Closing client.", w->id); + web_client_free(w); + continue; + } + } + + if(unlikely(single_threaded_link_client(w, &ifds, &ofds, &efds, &fdmax) != 0)) { + web_client_free(w); + } + } + } + else { + debug(D_WEB_CLIENT_ACCESS, "LISTENER: single threaded web server timeout."); +#ifdef NETDATA_INTERNAL_CHECKS + log_allocations(); +#endif + } + } + + debug(D_WEB_CLIENT, "LISTENER: exit!"); + close_listen_sockets(); + return NULL; +} diff --git a/src/web_server.h b/src/web_server.h index 39f3bad91..93adc5b28 100644 --- a/src/web_server.h +++ b/src/web_server.h @@ -1,20 +1,38 @@ #ifndef NETDATA_WEB_SERVER_H #define NETDATA_WEB_SERVER_H 1 -#define WEB_PATH_FILE "file" -#define WEB_PATH_DATA "data" -#define WEB_PATH_DATASOURCE "datasource" -#define WEB_PATH_GRAPH "graph" +#define WEB_PATH_FILE "file" +#define WEB_PATH_DATA "data" +#define WEB_PATH_DATASOURCE "datasource" +#define WEB_PATH_GRAPH "graph" #define LISTEN_PORT 19999 #define LISTEN_BACKLOG 100 -extern int listen_backlog; -extern int listen_fd; -extern int listen_port; +#ifndef MAX_LISTEN_FDS +#define MAX_LISTEN_FDS 100 +#endif -extern int create_listen_socket4(const char *ip, int port, int listen_backlog); -extern int create_listen_socket6(const char *ip, int port, int listen_backlog); -extern void *socket_listen_main(void *ptr); +#define WEB_SERVER_MODE_MULTI_THREADED 0 +#define WEB_SERVER_MODE_SINGLE_THREADED 1 +extern int web_server_mode; + +extern void *socket_listen_main_multi_threaded(void *ptr); +extern void *socket_listen_main_single_threaded(void *ptr); +extern int create_listen_sockets(void); +extern int is_listen_socket(int fd); + +#ifndef HAVE_ACCEPT4 +extern int accept4(int sock, struct sockaddr *addr, socklen_t *addrlen, int flags); + +#ifndef SOCK_NONBLOCK +#define SOCK_NONBLOCK 00004000 +#endif /* #ifndef SOCK_NONBLOCK */ + +#ifndef SOCK_CLOEXEC +#define SOCK_CLOEXEC 02000000 +#endif /* #ifndef SOCK_CLOEXEC */ + +#endif /* #ifndef HAVE_ACCEPT4 */ #endif /* NETDATA_WEB_SERVER_H */ diff --git a/system/Makefile.am b/system/Makefile.am index f16a720e2..b2e49c5af 100644 --- a/system/Makefile.am +++ b/system/Makefile.am @@ -7,6 +7,7 @@ CLEANFILES = \ netdata.logrotate \ netdata.service \ netdata-init-d \ + netdata-lsb \ $(NULL) include $(top_srcdir)/build/subst.inc @@ -18,6 +19,7 @@ nodist_noinst_DATA = \ netdata.logrotate \ netdata.service \ netdata-init-d \ + netdata-lsb \ $(NULL) dist_noinst_DATA = \ @@ -25,5 +27,6 @@ dist_noinst_DATA = \ netdata.logrotate.in \ netdata.service.in \ netdata-init-d.in \ + netdata-lsb.in \ netdata.conf \ $(NULL) diff --git a/system/Makefile.in b/system/Makefile.in index e8f5eb3c8..75d2e753f 100644 --- a/system/Makefile.in +++ b/system/Makefile.in @@ -233,6 +233,8 @@ pluginsdir = @pluginsdir@ prefix = @prefix@ program_transform_name = @program_transform_name@ psdir = @psdir@ +pythondir = @pythondir@ +registrydir = @registrydir@ sbindir = @sbindir@ sharedstatedir = @sharedstatedir@ srcdir = @srcdir@ @@ -253,6 +255,7 @@ CLEANFILES = \ netdata.logrotate \ netdata.service \ netdata-init-d \ + netdata-lsb \ $(NULL) SUFFIXES = .in @@ -261,6 +264,7 @@ nodist_noinst_DATA = \ netdata.logrotate \ netdata.service \ netdata-init-d \ + netdata-lsb \ $(NULL) dist_noinst_DATA = \ @@ -268,6 +272,7 @@ dist_noinst_DATA = \ netdata.logrotate.in \ netdata.service.in \ netdata-init-d.in \ + netdata-lsb.in \ netdata.conf \ $(NULL) @@ -463,6 +468,7 @@ uninstall-am: -e 's#[@]localstatedir_POST@#$(localstatedir)#g' \ -e 's#[@]sbindir_POST@#$(sbindir)#g' \ -e 's#[@]sysconfdir_POST@#$(sysconfdir)#g' \ + -e 's#[@]pythondir_POST@#$(pythondir)#g' \ $< > $@.tmp; then \ mv "$@.tmp" "$@"; \ else \ diff --git a/system/netdata-init-d.in b/system/netdata-init-d.in index c317d1021..edda9950f 100755..100644 --- a/system/netdata-init-d.in +++ b/system/netdata-init-d.in @@ -1,14 +1,20 @@ -#!/bin/bash +#!/bin/sh +# +# netdata Real-time performance monitoring, done right # chkconfig: 345 99 01 -# description: startup script +# description: Netdata is a daemon that collects data in real-time (per second) +# and presents a web site to view and analyze them. The presentation +# is also real-time and full of interactive charts that precisely +# render all collected values. +# processname: netdata # Source functions . /etc/rc.d/init.d/functions DAEMON="netdata" DAEMON_PATH=@sbindir_POST@ -PIDFILE=@localstatedir_POST@/$DAEMON.pid -DAEMONOPTS="-pidfile $PIDFILE" +PIDFILE=@localstatedir_POST@/run/$DAEMON.pid +DAEMONOPTS="-P $PIDFILE" STOP_TIMEOUT="10" service_start() @@ -32,6 +38,17 @@ service_stop() return $RETVAL } +condrestart() +{ + if ! service_status > /dev/null; then + RETVAL=$1 + return $RETVAL + fi + + service_stop + service_start +} + service_status() { status -p ${PIDFILE} $DAEMON_PATH/$DAEMON @@ -41,9 +58,6 @@ case "$1" in start) service_start ;; -status) - service_status -;; stop) service_stop ;; @@ -51,7 +65,16 @@ restart) service_stop service_start ;; +try-restart) + condrestart 0 + ;; +force-reload) + condrestart 7 +;; +status) + service_status +;; *) - echo "Usage: $0 {status|start|stop|restart}" - exit 1 + echo "Usage: $0 {start|stop|restart|try-restart|force-reload|status}" + exit 3 esac diff --git a/system/netdata-lsb.in b/system/netdata-lsb.in new file mode 100644 index 000000000..d81659775 --- /dev/null +++ b/system/netdata-lsb.in @@ -0,0 +1,100 @@ +#!/bin/bash +# +### BEGIN INIT INFO +# Provides: netdata +# Required-Start: $local_fs $remote_fs $network $named $time apache2 httpd squid nginx mysql named opensips upsd hostapd postfix lm_sensors +# Required-Stop: $local_fs $remote_fs $network $named $time apache2 httpd squid nginx mysql named opensips upsd hostapd postfix lm_sensors +# Should-Start: $local_fs $network $named $remote_fs $time $all +# Should-Stop: $local_fs $network $named $remote_fs $time $all +# Default-Start: 2 3 4 5 +# Default-Stop: 0 1 6 +# Short-Description: Start and stop the netdata real-time monitoring server daemon +# Description: Controls the main netdata monitoring server daemon "netdata". +# and all its plugins. +### END INIT INFO +# +set -e +set -u +${DEBIAN_SCRIPT_DEBUG:+ set -v -x} + +DAEMON="netdata" +DAEMON_PATH=@sbindir_POST@ +PIDFILE=@localstatedir_POST@/$DAEMON.pid +DAEMONOPTS="-P $PIDFILE" + +test -x $DAEMON_PATH/$DAEMON || exit 0 + +. /lib/lsb/init-functions + +# Safeguard (relative paths, core dumps..) +cd / +umask 022 + +service_start() { + log_daemon_msg "Starting real-time performance monitoring" "netdata" + start_daemon -p $PIDFILE $DAEMON_PATH/$DAEMON $DAEMONOPTS + RETVAL=$? + log_end_msg $RETVAL + return $RETVAL +} + +service_stop() { + log_daemon_msg "Stopping real-time performance monitoring" "netdata" + killproc -p ${PIDFILE} $DAEMON_PATH/$DAEMON + RETVAL=$? + log_end_msg $RETVAL + + if [ $RETVAL -eq 0 ]; then + rm -f ${PIDFILE} + fi + return $RETVAL +} + +condrestart() { + if ! service_status > /dev/null; then + RETVAL=$1 + return + fi + + service_stop + service_start +} + +service_status() { + status_of_proc -p $PIDFILE $DAEMON_PATH/$DAEMON netdata +} + + +# +# main() +# + +case "${1:-''}" in + 'start') + service_start + ;; + + 'stop') + service_stop + ;; + + 'restart') + service_stop + service_start + ;; + + 'try-restart') + condrestart 0 + ;; + + 'force-reload') + condrestart 7 + ;; + + 'status') + service_status && exit 0 || exit $? + ;; + *) + echo "Usage: $0 {start|stop|restart|try-restart|force-reload|status}" + exit 1 +esac diff --git a/system/netdata-openrc.in b/system/netdata-openrc.in index 299db5699..55808364c 100755..100644 --- a/system/netdata-openrc.in +++ b/system/netdata-openrc.in @@ -1,4 +1,4 @@ -#!/sbin/runscript +#!/sbin/openrc-run # The user netdata is configured to run as. # If you edit its configuration file to set a different @@ -25,7 +25,7 @@ extra_started_commands="getconf" pidfile="/run/netdata.pid" command="${NETDATA_INSTALL_PATH}/usr/sbin/netdata" command_background="yes" -command_args="-pidfile ${pidfile} ${NETDATA_EXTRA_ARGS}" +command_args="-P ${pidfile} ${NETDATA_EXTRA_ARGS}" # start_stop_daemon_args="-u ${NETDATA_OWNER}" start_stop_daemon_args="" diff --git a/system/netdata.conf b/system/netdata.conf index df7ea1abe..6108d9086 100644 --- a/system/netdata.conf +++ b/system/netdata.conf @@ -9,3 +9,6 @@ run as user = netdata web files owner = root web files group = netdata + # Netdata is not designed to be exposed to potentially hostile networks + # See https://github.com/firehol/netdata/issues/164 + bind to = localhost diff --git a/system/netdata.logrotate.in b/system/netdata.logrotate.in index e77d5ff72..a026a8a51 100644 --- a/system/netdata.logrotate.in +++ b/system/netdata.logrotate.in @@ -6,16 +6,8 @@ delaycompress notifempty sharedscripts - # - # if you add netdata to your init.d/system.d - # comment su & copytruncate and uncomment postrotate - # to have netdata restart when logs are rotated - su netdata - copytruncate - # - #postrotate - # if service netdata status > /dev/null ; then \ - # service netdata restart > /dev/null; \ - # fi; - #endscript + create 0664 netdata netdata + postrotate + /bin/kill -HUP `pidof netdata 2>/dev/null` 2>/dev/null || true + endscript } diff --git a/system/netdata.service.in b/system/netdata.service.in index 91db6122d..e260e2738 100644 --- a/system/netdata.service.in +++ b/system/netdata.service.in @@ -1,17 +1,15 @@ [Unit] -Description=Linux real time system monitoring, over the web +Description=Real time performance monitoring After=network.target httpd.service squid.service nfs-server.service mysqld.service named.service postfix.service [Service] -Type=forking -WorkingDirectory=/tmp -User=root -Group=root -PIDFile=@localstatedir_POST@/run/netdata.pid -ExecStart=@sbindir_POST@/netdata -pidfile @localstatedir_POST@/run/netdata.pid -KillMode=mixed -KillSignal=SIGTERM -TimeoutStopSec=30 +Type=simple +User=netdata +Group=netdata +ExecStart=@sbindir_POST@/netdata -D + +# saving a big db on slow disks may need some time +TimeoutStopSec=60 [Install] WantedBy=multi-user.target diff --git a/web/.well-known/dnt/cookies b/web/.well-known/dnt/cookies new file mode 100644 index 000000000..dc2fe0b4a --- /dev/null +++ b/web/.well-known/dnt/cookies @@ -0,0 +1,14 @@ +{
+ "tracking": "T",
+ "compliance": ["https://github.com/firehol/netdata/wiki/cookies#compliance"],
+ "qualifiers": "afc",
+ "controller": ["https://github.com/firehol/netdata/wiki/cookies#controller"],
+ "same-party": [
+ "my-netdata.io",
+ "mynetdata.io",
+ "netdata.online",
+ "netdata.rocks",
+ "registry.my-netdata.io"
+ ],
+ "policy": "https://github.com/firehol/netdata/wiki/cookies#policy",
+}
diff --git a/web/Makefile.am b/web/Makefile.am index 174ef229b..0432f8a58 100644 --- a/web/Makefile.am +++ b/web/Makefile.am @@ -16,6 +16,7 @@ dist_web_DATA = \ netdata-swagger.yaml \ netdata-swagger.json \ robots.txt \ + registry.html \ tv.html \ version.txt \ $(NULL) @@ -77,6 +78,8 @@ dist_webfonts_DATA = \ webimagesdir=$(webdir)/images dist_webimages_DATA = \ + images/animated.gif \ + images/post.png \ images/seo-performance-16.png \ images/seo-performance-24.png \ images/seo-performance-32.png \ @@ -91,6 +94,16 @@ dist_webimages_DATA = \ images/seo-performance-multi-size.icns \ $(NULL) + +webwellknowndir=$(webdir)/.well-known +dist_webwellknown_DATA = \ + $(NULL) + +webdntdir=$(webdir)/.well-known/dnt +dist_webdnt_DATA = \ + .well-known/dnt/cookies \ + $(NULL) + version.txt: if test -d "$(top_srcdir)/.git"; then \ git --git-dir="$(top_srcdir)/.git" log -n 1 --format=%H; \ diff --git a/web/Makefile.in b/web/Makefile.in index e95290128..6e5e46a91 100644 --- a/web/Makefile.in +++ b/web/Makefile.in @@ -80,8 +80,10 @@ build_triplet = @build@ host_triplet = @host@ subdir = web DIST_COMMON = $(srcdir)/Makefile.in $(srcdir)/Makefile.am \ - $(dist_web_DATA) $(dist_webcss_DATA) $(dist_webfonts_DATA) \ - $(dist_webimages_DATA) $(dist_weblib_DATA) $(dist_webold_DATA) + $(dist_web_DATA) $(dist_webcss_DATA) $(dist_webdnt_DATA) \ + $(dist_webfonts_DATA) $(dist_webimages_DATA) \ + $(dist_weblib_DATA) $(dist_webold_DATA) \ + $(dist_webwellknown_DATA) ACLOCAL_M4 = $(top_srcdir)/aclocal.m4 am__aclocal_m4_deps = $(top_srcdir)/m4/ax_pthread.m4 \ $(top_srcdir)/configure.ac @@ -138,10 +140,13 @@ am__uninstall_files_from_dir = { \ $(am__cd) "$$dir" && rm -f $$files; }; \ } am__installdirs = "$(DESTDIR)$(webdir)" "$(DESTDIR)$(webcssdir)" \ - "$(DESTDIR)$(webfontsdir)" "$(DESTDIR)$(webimagesdir)" \ - "$(DESTDIR)$(weblibdir)" "$(DESTDIR)$(webolddir)" -DATA = $(dist_web_DATA) $(dist_webcss_DATA) $(dist_webfonts_DATA) \ - $(dist_webimages_DATA) $(dist_weblib_DATA) $(dist_webold_DATA) + "$(DESTDIR)$(webdntdir)" "$(DESTDIR)$(webfontsdir)" \ + "$(DESTDIR)$(webimagesdir)" "$(DESTDIR)$(weblibdir)" \ + "$(DESTDIR)$(webolddir)" "$(DESTDIR)$(webwellknowndir)" +DATA = $(dist_web_DATA) $(dist_webcss_DATA) $(dist_webdnt_DATA) \ + $(dist_webfonts_DATA) $(dist_webimages_DATA) \ + $(dist_weblib_DATA) $(dist_webold_DATA) \ + $(dist_webwellknown_DATA) am__tagged_files = $(HEADERS) $(SOURCES) $(TAGS_FILES) $(LISP) DISTFILES = $(DIST_COMMON) $(DIST_SOURCES) $(TEXINFOS) $(EXTRA_DIST) ACLOCAL = @ACLOCAL@ @@ -265,6 +270,8 @@ pluginsdir = @pluginsdir@ prefix = @prefix@ program_transform_name = @program_transform_name@ psdir = @psdir@ +pythondir = @pythondir@ +registrydir = @registrydir@ sbindir = @sbindir@ sharedstatedir = @sharedstatedir@ srcdir = @srcdir@ @@ -293,6 +300,7 @@ dist_web_DATA = \ netdata-swagger.yaml \ netdata-swagger.json \ robots.txt \ + registry.html \ tv.html \ version.txt \ $(NULL) @@ -354,6 +362,8 @@ dist_webfonts_DATA = \ webimagesdir = $(webdir)/images dist_webimages_DATA = \ + images/animated.gif \ + images/post.png \ images/seo-performance-16.png \ images/seo-performance-24.png \ images/seo-performance-32.png \ @@ -368,6 +378,15 @@ dist_webimages_DATA = \ images/seo-performance-multi-size.icns \ $(NULL) +webwellknowndir = $(webdir)/.well-known +dist_webwellknown_DATA = \ + $(NULL) + +webdntdir = $(webdir)/.well-known/dnt +dist_webdnt_DATA = \ + .well-known/dnt/cookies \ + $(NULL) + all: all-am .SUFFIXES: @@ -443,6 +462,27 @@ uninstall-dist_webcssDATA: @list='$(dist_webcss_DATA)'; test -n "$(webcssdir)" || list=; \ files=`for p in $$list; do echo $$p; done | sed -e 's|^.*/||'`; \ dir='$(DESTDIR)$(webcssdir)'; $(am__uninstall_files_from_dir) +install-dist_webdntDATA: $(dist_webdnt_DATA) + @$(NORMAL_INSTALL) + @list='$(dist_webdnt_DATA)'; test -n "$(webdntdir)" || list=; \ + if test -n "$$list"; then \ + echo " $(MKDIR_P) '$(DESTDIR)$(webdntdir)'"; \ + $(MKDIR_P) "$(DESTDIR)$(webdntdir)" || exit 1; \ + fi; \ + for p in $$list; do \ + if test -f "$$p"; then d=; else d="$(srcdir)/"; fi; \ + echo "$$d$$p"; \ + done | $(am__base_list) | \ + while read files; do \ + echo " $(INSTALL_DATA) $$files '$(DESTDIR)$(webdntdir)'"; \ + $(INSTALL_DATA) $$files "$(DESTDIR)$(webdntdir)" || exit $$?; \ + done + +uninstall-dist_webdntDATA: + @$(NORMAL_UNINSTALL) + @list='$(dist_webdnt_DATA)'; test -n "$(webdntdir)" || list=; \ + files=`for p in $$list; do echo $$p; done | sed -e 's|^.*/||'`; \ + dir='$(DESTDIR)$(webdntdir)'; $(am__uninstall_files_from_dir) install-dist_webfontsDATA: $(dist_webfonts_DATA) @$(NORMAL_INSTALL) @list='$(dist_webfonts_DATA)'; test -n "$(webfontsdir)" || list=; \ @@ -527,6 +567,27 @@ uninstall-dist_weboldDATA: @list='$(dist_webold_DATA)'; test -n "$(webolddir)" || list=; \ files=`for p in $$list; do echo $$p; done | sed -e 's|^.*/||'`; \ dir='$(DESTDIR)$(webolddir)'; $(am__uninstall_files_from_dir) +install-dist_webwellknownDATA: $(dist_webwellknown_DATA) + @$(NORMAL_INSTALL) + @list='$(dist_webwellknown_DATA)'; test -n "$(webwellknowndir)" || list=; \ + if test -n "$$list"; then \ + echo " $(MKDIR_P) '$(DESTDIR)$(webwellknowndir)'"; \ + $(MKDIR_P) "$(DESTDIR)$(webwellknowndir)" || exit 1; \ + fi; \ + for p in $$list; do \ + if test -f "$$p"; then d=; else d="$(srcdir)/"; fi; \ + echo "$$d$$p"; \ + done | $(am__base_list) | \ + while read files; do \ + echo " $(INSTALL_DATA) $$files '$(DESTDIR)$(webwellknowndir)'"; \ + $(INSTALL_DATA) $$files "$(DESTDIR)$(webwellknowndir)" || exit $$?; \ + done + +uninstall-dist_webwellknownDATA: + @$(NORMAL_UNINSTALL) + @list='$(dist_webwellknown_DATA)'; test -n "$(webwellknowndir)" || list=; \ + files=`for p in $$list; do echo $$p; done | sed -e 's|^.*/||'`; \ + dir='$(DESTDIR)$(webwellknowndir)'; $(am__uninstall_files_from_dir) tags TAGS: ctags CTAGS: @@ -568,7 +629,7 @@ check-am: all-am check: check-am all-am: Makefile $(DATA) installdirs: - for dir in "$(DESTDIR)$(webdir)" "$(DESTDIR)$(webcssdir)" "$(DESTDIR)$(webfontsdir)" "$(DESTDIR)$(webimagesdir)" "$(DESTDIR)$(weblibdir)" "$(DESTDIR)$(webolddir)"; do \ + for dir in "$(DESTDIR)$(webdir)" "$(DESTDIR)$(webcssdir)" "$(DESTDIR)$(webdntdir)" "$(DESTDIR)$(webfontsdir)" "$(DESTDIR)$(webimagesdir)" "$(DESTDIR)$(weblibdir)" "$(DESTDIR)$(webolddir)" "$(DESTDIR)$(webwellknowndir)"; do \ test -z "$$dir" || $(MKDIR_P) "$$dir"; \ done install: install-am @@ -623,8 +684,9 @@ info: info-am info-am: install-data-am: install-dist_webDATA install-dist_webcssDATA \ - install-dist_webfontsDATA install-dist_webimagesDATA \ - install-dist_weblibDATA install-dist_weboldDATA + install-dist_webdntDATA install-dist_webfontsDATA \ + install-dist_webimagesDATA install-dist_weblibDATA \ + install-dist_weboldDATA install-dist_webwellknownDATA install-dvi: install-dvi-am @@ -669,8 +731,9 @@ ps: ps-am ps-am: uninstall-am: uninstall-dist_webDATA uninstall-dist_webcssDATA \ - uninstall-dist_webfontsDATA uninstall-dist_webimagesDATA \ - uninstall-dist_weblibDATA uninstall-dist_weboldDATA + uninstall-dist_webdntDATA uninstall-dist_webfontsDATA \ + uninstall-dist_webimagesDATA uninstall-dist_weblibDATA \ + uninstall-dist_weboldDATA uninstall-dist_webwellknownDATA .MAKE: install-am install-strip @@ -678,17 +741,20 @@ uninstall-am: uninstall-dist_webDATA uninstall-dist_webcssDATA \ ctags-am distclean distclean-generic distdir dvi dvi-am html \ html-am info info-am install install-am install-data \ install-data-am install-dist_webDATA install-dist_webcssDATA \ - install-dist_webfontsDATA install-dist_webimagesDATA \ - install-dist_weblibDATA install-dist_weboldDATA install-dvi \ - install-dvi-am install-exec install-exec-am install-html \ - install-html-am install-info install-info-am install-man \ - install-pdf install-pdf-am install-ps install-ps-am \ - install-strip installcheck installcheck-am installdirs \ - maintainer-clean maintainer-clean-generic mostlyclean \ - mostlyclean-generic pdf pdf-am ps ps-am tags-am uninstall \ - uninstall-am uninstall-dist_webDATA uninstall-dist_webcssDATA \ + install-dist_webdntDATA install-dist_webfontsDATA \ + install-dist_webimagesDATA install-dist_weblibDATA \ + install-dist_weboldDATA install-dist_webwellknownDATA \ + install-dvi install-dvi-am install-exec install-exec-am \ + install-html install-html-am install-info install-info-am \ + install-man install-pdf install-pdf-am install-ps \ + install-ps-am install-strip installcheck installcheck-am \ + installdirs maintainer-clean maintainer-clean-generic \ + mostlyclean mostlyclean-generic pdf pdf-am ps ps-am tags-am \ + uninstall uninstall-am uninstall-dist_webDATA \ + uninstall-dist_webcssDATA uninstall-dist_webdntDATA \ uninstall-dist_webfontsDATA uninstall-dist_webimagesDATA \ - uninstall-dist_weblibDATA uninstall-dist_weboldDATA + uninstall-dist_weblibDATA uninstall-dist_weboldDATA \ + uninstall-dist_webwellknownDATA version.txt: diff --git a/web/dashboard.css b/web/dashboard.css index 63e2b905f..eb2ed1b0d 100644 --- a/web/dashboard.css +++ b/web/dashboard.css @@ -6,6 +6,19 @@ body { font-variant: normal; } +.morelink { + color: #765d9c; + text-decoration: none; +} +.morelink:hover { + color: #563d7c; + text-decoration: none; +} +.morelink:focus { + color: #765d9c; + text-decoration: none; +} + .netdata-chart-alignment { margin-left: 55px; } diff --git a/web/dashboard.html b/web/dashboard.html index 2b6c80684..e5d01353f 100644 --- a/web/dashboard.html +++ b/web/dashboard.html @@ -1,15 +1,24 @@ <!DOCTYPE html> <html lang="en"> <head> - <title>NetData Dashboard</title> - - <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> - <meta charset="utf-8"> - <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1"> - <meta name="viewport" content="width=device-width, initial-scale=1"> - <meta name="apple-mobile-web-app-capable" content="yes"> - <meta name="apple-mobile-web-app-status-bar-style" content="black-translucent"> - <meta name="author" content="costa@tsaousis.gr"> + <title>NetData Dashboard</title> + <meta name="application-name" content="netdata"> + + <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> + <meta charset="utf-8"> + <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1"> + <meta name="viewport" content="width=device-width, initial-scale=1"> + <meta name="apple-mobile-web-app-capable" content="yes"> + <meta name="apple-mobile-web-app-status-bar-style" content="black-translucent"> + <meta name="author" content="costa@tsaousis.gr"> + + <meta property="og:locale" content="en_US" /> + <meta property="og:image" content="https://my-netdata.io/images/seo-performance-512.png"/> + <meta property="og:url" content="http://my-netdata.io/"/> + <meta property="og:type" content="website"/> + <meta property="og:site_name" content="netdata"/> + <meta property="og:title" content="netdata - real-time performance monitoring, done right!"/> + <meta property="og:description" content="Stunning real-time dashboards, blazingly fast and extremely interactive. Zero configuration, zero dependencies, zero maintenance." /> </head> <body> @@ -23,33 +32,33 @@ This is a template for building custom dashboards. To build a dashboard you just <!DOCTYPE html> <html lang="en"> <head> - <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> - <meta charset="utf-8"> - <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1"> - <meta name="viewport" content="width=device-width, initial-scale=1"> - <meta name="apple-mobile-web-app-capable" content="yes"> - <meta name="apple-mobile-web-app-status-bar-style" content="black-translucent"> + <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> + <meta charset="utf-8"> + <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1"> + <meta name="viewport" content="width=device-width, initial-scale=1"> + <meta name="apple-mobile-web-app-capable" content="yes"> + <meta name="apple-mobile-web-app-status-bar-style" content="black-translucent"> </head> <body> - <div data-netdata="system.processes" - data-chart-library="dygraph" - data-width="600" - data-height="200" - data-after="-600" - ></div> + <div data-netdata="system.processes" + data-chart-library="dygraph" + data-width="600" + data-height="200" + data-after="-600" + ></div> </body> <script type="text/javascript" src="http://netdata.server:19999/dashboard.js"></script> </html> </pre> <ul> - <li>You can host your dashboard anywhere.</li> - <li>You can add as many charts as you like.</li> - <li>You can have charts from many different netdata servers (add <pre>data-host="http://another.netdata.server:19999/"</pre> to each chart).</li> - <li>You can use different chart libraries on the same page: <b>peity</b>, <b>sparkline</b>, <b>dygraph</b>, <b>google</b>, <b>morris</b></li> - <li>You can customize each chart to your preferences. For each chart library most of their attributes can be given in <b>data-</b> attributes.</li> - <li>Each chart can have each own duration - it is controlled with the <b>data-after</b> attribute to give that many seconds of data.</li> - <li>Depending on the width of the chart and <b>data-after</b> attribute, netdata will automatically refresh the chart when it needs to be updated. For example giving 600 pixels for width for -600 seconds of data, using a chart library that needs 3 pixels per point, will yeld in a chart updated once every 3 seconds.</li> + <li>You can host your dashboard anywhere.</li> + <li>You can add as many charts as you like.</li> + <li>You can have charts from many different netdata servers (add <pre>data-host="http://another.netdata.server:19999/"</pre> to each chart).</li> + <li>You can use different chart libraries on the same page: <b>peity</b>, <b>sparkline</b>, <b>dygraph</b>, <b>google</b>, <b>morris</b></li> + <li>You can customize each chart to your preferences. For each chart library most of their attributes can be given in <b>data-</b> attributes.</li> + <li>Each chart can have each own duration - it is controlled with the <b>data-after</b> attribute to give that many seconds of data.</li> + <li>Depending on the width of the chart and <b>data-after</b> attribute, netdata will automatically refresh the chart when it needs to be updated. For example giving 600 pixels for width for -600 seconds of data, using a chart library that needs 3 pixels per point, will yeld in a chart updated once every 3 seconds.</li> </ul> @@ -60,61 +69,61 @@ Sparkline charts stretch the values to show the variations between values in mor They also have mouse-hover support. <br/> <b>Sparklines are fantastic</b>. You can inline charts in text. For example this - <div data-netdata="system.cpu" - data-chart-library="sparkline" - data-width="5%" - data-height="20" - data-after="-30" - ></div> is my current cpu usage (last 30 seconds), - while this - <div data-netdata="netdata.net" - data-dimensions="out" - data-chart-library="sparkline" - data-width="10%" - data-height="20" - data-after="-60" - ></div> is the bandwidth my netdata server is currently transmitting (last minute) - and this - <div data-netdata="netdata.requests" - data-chart-library="sparkline" - data-width="20%" - data-height="20" - data-after="-180" - ></div> is the requests/sec it serves (last 3 minutes). + <div data-netdata="system.cpu" + data-chart-library="sparkline" + data-width="5%" + data-height="20" + data-after="-30" + ></div> is my current cpu usage (last 30 seconds), + while this + <div data-netdata="netdata.net" + data-dimensions="out" + data-chart-library="sparkline" + data-width="10%" + data-height="20" + data-after="-60" + ></div> is the bandwidth my netdata server is currently transmitting (last minute) + and this + <div data-netdata="netdata.requests" + data-chart-library="sparkline" + data-width="20%" + data-height="20" + data-after="-180" + ></div> is the requests/sec it serves (last 3 minutes). <br/> <div style="width: 33%; display: inline-block;"> - <div data-netdata="system.processes" - data-chart-library="sparkline" - data-width="100%" - data-height="30" - data-after="-300" - data-dt-element-name="time101" - ></div> - <br/> - <small>rendered in <span id="time101">X</span> ms</small> + <div data-netdata="system.processes" + data-chart-library="sparkline" + data-width="100%" + data-height="30" + data-after="-300" + data-dt-element-name="time101" + ></div> + <br/> + <small>rendered in <span id="time101">X</span> ms</small> </div> <div style="width: 33%; display: inline-block;"> - <div data-netdata="system.ipv4" - data-chart-library="sparkline" - data-width="100%" - data-height="30" - data-after="-300" - data-dt-element-name="time102" - ></div> - <br/> - <small>rendered in <span id="time102">X</span> ms</small> + <div data-netdata="system.ipv4" + data-chart-library="sparkline" + data-width="100%" + data-height="30" + data-after="-300" + data-dt-element-name="time102" + ></div> + <br/> + <small>rendered in <span id="time102">X</span> ms</small> </div> <div style="width: 33%; display: inline-block;"> - <div data-netdata="system.cpu" - data-chart-library="sparkline" - data-width="100%" - data-height="30" - data-after="-300" - data-dt-element-name="time103" - ></div> - <br/> - <small>rendered in <span id="time103">X</span> ms</small> + <div data-netdata="system.cpu" + data-chart-library="sparkline" + data-width="100%" + data-height="30" + data-after="-300" + data-dt-element-name="time103" + ></div> + <br/> + <small>rendered in <span id="time103">X</span> ms</small> </div> @@ -126,37 +135,37 @@ Peity charts cannot have multiple dimensions on the charts - so netdata will use the total of all dimensions. <br/> <div style="width: 33%; display: inline-block;"> - <div data-netdata="system.processes" - data-chart-library="peity" - data-width="100%" - data-height="30" - data-after="-300" - data-dt-element-name="time001" - ></div> - <br/> - <small>rendered in <span id="time001">X</span> ms</small> + <div data-netdata="system.processes" + data-chart-library="peity" + data-width="100%" + data-height="30" + data-after="-300" + data-dt-element-name="time001" + ></div> + <br/> + <small>rendered in <span id="time001">X</span> ms</small> </div> <div style="width: 33%; display: inline-block;"> - <div data-netdata="system.ipv4" - data-chart-library="peity" - data-width="100%" - data-height="30" - data-after="-300" - data-dt-element-name="time002" - ></div> - <br/> - <small>rendered in <span id="time002">X</span> ms</small> + <div data-netdata="system.ipv4" + data-chart-library="peity" + data-width="100%" + data-height="30" + data-after="-300" + data-dt-element-name="time002" + ></div> + <br/> + <small>rendered in <span id="time002">X</span> ms</small> </div> <div style="width: 33%; display: inline-block;"> - <div data-netdata="system.cpu" - data-chart-library="peity" - data-width="100%" - data-height="30" - data-after="-300" - data-dt-element-name="time003" - ></div> - <br/> - <small>rendered in <span id="time003">X</span> ms</small> + <div data-netdata="system.cpu" + data-chart-library="peity" + data-width="100%" + data-height="30" + data-after="-300" + data-dt-element-name="time003" + ></div> + <br/> + <small>rendered in <span id="time003">X</span> ms</small> </div> @@ -169,173 +178,173 @@ The charts are zoomable (drag their contents to pan, shift with mouse wheel to z <b>Netdata magic!</b> Realtime charts on your web page! <br/> Sparklines using dygraphs - <div data-netdata="system.processes" - data-chart-library="dygraph" - data-dygraph-theme="sparkline" - data-legend="no" - data-width="15%" - data-height="20" - data-after="-300" - ></div> - are also possible! This - <div data-netdata="system.ipv4" - data-chart-library="dygraph" - data-dygraph-theme="sparkline" - data-legend="no" - data-width="15%" - data-height="20" - data-after="-300" - ></div> - is an area chart, while this - <div data-netdata="system.cpu" - data-chart-library="dygraph" - data-dygraph-theme="sparkline" - data-legend="no" - data-width="15%" - data-height="20" - data-after="-300" - ></div> is a stacked area chart! + <div data-netdata="system.processes" + data-chart-library="dygraph" + data-dygraph-theme="sparkline" + data-legend="no" + data-width="15%" + data-height="20" + data-after="-300" + ></div> + are also possible! This + <div data-netdata="system.ipv4" + data-chart-library="dygraph" + data-dygraph-theme="sparkline" + data-legend="no" + data-width="15%" + data-height="20" + data-after="-300" + ></div> + is an area chart, while this + <div data-netdata="system.cpu" + data-chart-library="dygraph" + data-dygraph-theme="sparkline" + data-legend="no" + data-width="15%" + data-height="20" + data-after="-300" + ></div> is a stacked area chart! <br/> <div style="width: 33%; display: inline-block;"> - <div data-netdata="system.processes" - data-chart-library="dygraph" - data-dygraph-theme="sparkline" - data-width="100%" - data-height="30" - data-after="-300" - data-dt-element-name="time501" - ></div> - <br/> - <small>rendered in <span id="time501">X</span> ms</small> + <div data-netdata="system.processes" + data-chart-library="dygraph" + data-dygraph-theme="sparkline" + data-width="100%" + data-height="30" + data-after="-300" + data-dt-element-name="time501" + ></div> + <br/> + <small>rendered in <span id="time501">X</span> ms</small> </div> <div style="width: 33%; display: inline-block;"> - <div data-netdata="system.ipv4" - data-chart-library="dygraph" - data-dygraph-theme="sparkline" - data-width="100%" - data-height="30" - data-after="-300" - data-dt-element-name="time502" - ></div> - <br/> - <small>rendered in <span id="time502">X</span> ms</small> + <div data-netdata="system.ipv4" + data-chart-library="dygraph" + data-dygraph-theme="sparkline" + data-width="100%" + data-height="30" + data-after="-300" + data-dt-element-name="time502" + ></div> + <br/> + <small>rendered in <span id="time502">X</span> ms</small> </div> <div style="width: 33%; display: inline-block;"> - <div data-netdata="system.cpu" - data-chart-library="dygraph" - data-dygraph-theme="sparkline" - data-width="100%" - data-height="30" - data-after="-300" - data-dt-element-name="time503" - ></div> - <br/> - <small>rendered in <span id="time503">X</span> ms</small> + <div data-netdata="system.cpu" + data-chart-library="dygraph" + data-dygraph-theme="sparkline" + data-width="100%" + data-height="30" + data-after="-300" + data-dt-element-name="time503" + ></div> + <br/> + <small>rendered in <span id="time503">X</span> ms</small> </div> <br/> <div style="width: 33%; display: inline-block;"> - <div data-netdata="system.processes" - data-chart-library="dygraph" - data-width="100%" - data-height="200" - data-after="-300" - data-dt-element-name="time201" - ></div> - <br/> - <small>rendered in <span id="time201">X</span> ms</small> -</div> -<div style="width: 33%; display: inline-block;"> - <div data-netdata="system.ipv4" - data-chart-library="dygraph" - data-width="100%" - data-height="200" - data-after="-300" - data-dt-element-name="time202" - ></div> - <br/> - <small>rendered in <span id="time202">X</span> ms</small> -</div> -<div style="width: 33%; display: inline-block;"> - <div data-netdata="system.cpu" - data-chart-library="dygraph" - data-width="100%" - data-height="200" - data-after="-300" - data-dt-element-name="time203" - ></div> - <br/> - <small>rendered in <span id="time203">X</span> ms</small> -</div> -<div style="width: 33%; display: inline-block;"> - <div data-netdata="system.io" - data-chart-library="dygraph" - data-width="100%" - data-height="200" - data-after="-300" - data-dt-element-name="time204" - ></div> - <br/> - <small>rendered in <span id="time204">X</span> ms</small> -</div> -<div style="width: 33%; display: inline-block;"> - <div data-netdata="apps.cpu" - data-chart-library="dygraph" - data-width="100%" - data-height="200" - data-after="-300" - data-dt-element-name="time205" - ></div> - <br/> - <small>rendered in <span id="time205">X</span> ms</small> -</div> -<div style="width: 33%; display: inline-block;"> - <div data-netdata="netdata.net" - data-chart-library="dygraph" - data-width="100%" - data-height="200" - data-after="-300" - data-dt-element-name="time206" - ></div> - <br/> - <small>rendered in <span id="time206">X</span> ms</small> -</div> -<div style="width: 33%; display: inline-block;"> - <div data-netdata="netdata.server_cpu" - data-chart-library="dygraph" - data-width="100%" - data-height="200" - data-after="-300" - data-dt-element-name="time207" - ></div> - <br/> - <small>rendered in <span id="time207">X</span> ms</small> -</div> -<div style="width: 33%; display: inline-block;"> - <div data-netdata="netdata.requests" - data-chart-library="dygraph" - data-width="100%" - data-height="200" - data-after="-300" - data-dt-element-name="time208" - ></div> - <br/> - <small>rendered in <span id="time208">X</span> ms</small> -</div> -<div style="width: 33%; display: inline-block;"> - <div data-netdata="net.gstag0" - data-chart-library="dygraph" - data-width="100%" - data-height="200" - data-after="-300" - data-dt-element-name="time209" - ></div> - <br/> - <small>rendered in <span id="time209">X</span> ms</small> + <div data-netdata="system.processes" + data-chart-library="dygraph" + data-width="100%" + data-height="200" + data-after="-300" + data-dt-element-name="time201" + ></div> + <br/> + <small>rendered in <span id="time201">X</span> ms</small> +</div> +<div style="width: 33%; display: inline-block;"> + <div data-netdata="system.ipv4" + data-chart-library="dygraph" + data-width="100%" + data-height="200" + data-after="-300" + data-dt-element-name="time202" + ></div> + <br/> + <small>rendered in <span id="time202">X</span> ms</small> +</div> +<div style="width: 33%; display: inline-block;"> + <div data-netdata="system.cpu" + data-chart-library="dygraph" + data-width="100%" + data-height="200" + data-after="-300" + data-dt-element-name="time203" + ></div> + <br/> + <small>rendered in <span id="time203">X</span> ms</small> +</div> +<div style="width: 33%; display: inline-block;"> + <div data-netdata="system.io" + data-chart-library="dygraph" + data-width="100%" + data-height="200" + data-after="-300" + data-dt-element-name="time204" + ></div> + <br/> + <small>rendered in <span id="time204">X</span> ms</small> +</div> +<div style="width: 33%; display: inline-block;"> + <div data-netdata="apps.cpu" + data-chart-library="dygraph" + data-width="100%" + data-height="200" + data-after="-300" + data-dt-element-name="time205" + ></div> + <br/> + <small>rendered in <span id="time205">X</span> ms</small> +</div> +<div style="width: 33%; display: inline-block;"> + <div data-netdata="netdata.net" + data-chart-library="dygraph" + data-width="100%" + data-height="200" + data-after="-300" + data-dt-element-name="time206" + ></div> + <br/> + <small>rendered in <span id="time206">X</span> ms</small> +</div> +<div style="width: 33%; display: inline-block;"> + <div data-netdata="netdata.server_cpu" + data-chart-library="dygraph" + data-width="100%" + data-height="200" + data-after="-300" + data-dt-element-name="time207" + ></div> + <br/> + <small>rendered in <span id="time207">X</span> ms</small> +</div> +<div style="width: 33%; display: inline-block;"> + <div data-netdata="netdata.requests" + data-chart-library="dygraph" + data-width="100%" + data-height="200" + data-after="-300" + data-dt-element-name="time208" + ></div> + <br/> + <small>rendered in <span id="time208">X</span> ms</small> +</div> +<div style="width: 33%; display: inline-block;"> + <div data-netdata="net.gstag0" + data-chart-library="dygraph" + data-width="100%" + data-height="200" + data-after="-300" + data-dt-element-name="time209" + ></div> + <br/> + <small>rendered in <span id="time209">X</span> ms</small> </div> @@ -344,82 +353,82 @@ Sparklines using dygraphs <h1>EasyPieChart</h1> <br/> <div style="width: 33%; display: inline-block;"> - <div style="display: inline-block; position: relative;"> - <div data-netdata="system.processes" - data-chart-library="easypiechart" - data-width="200" - data-height="200" - data-after="-300" - data-points="300" - data-dt-element-name="time601" - ></div> - <br/> - <small>rendered in <span id="time601">X</span> ms</small> - </div> - <div style="display: inline-block; position: relative;"> - <div data-netdata="system.processes" - data-chart-library="easypiechart" - data-width="150" - data-height="150" - data-after="-300" - data-points="150" - data-dt-element-name="time601a" - ></div> - <br/> - <small>rendered in <span id="time601a">X</span> ms</small> - </div> -</div> -<div style="width: 33%; display: inline-block;"> - <div style="display: inline-block; position: relative;"> - <div data-netdata="system.ipv4" - data-chart-library="easypiechart" - data-width="200" - data-height="200" - data-after="-300" - data-points="300" - data-dt-element-name="time602" - ></div> - <br/> - <small>rendered in <span id="time602">X</span> ms</small> - </div> - <div style="display: inline-block; position: relative;"> - <div data-netdata="system.ipv4" - data-chart-library="easypiechart" - data-width="100" - data-height="100" - data-after="-300" - data-points="150" - data-dt-element-name="time602a" - ></div> - <br/> - <small>rendered in <span id="time602a">X</span> ms</small> - </div> -</div> -<div style="width: 33%; display: inline-block;"> - <div style="display: inline-block; position: relative;"> - <div data-netdata="system.cpu" - data-chart-library="easypiechart" - data-width="200" - data-height="200" - data-after="-300" - data-points="300" - data-dt-element-name="time603" - ></div> - <br/> - <small>rendered in <span id="time603">X</span> ms</small> - </div> - <div style="display: inline-block; position: relative;"> - <div data-netdata="system.cpu" - data-chart-library="easypiechart" - data-width="75" - data-height="75" - data-after="-300" - data-points="150" - data-dt-element-name="time603a" - ></div> - <br/> - <small>rendered in <span id="time603a">X</span> ms</small> - </div> + <div style="display: inline-block; position: relative;"> + <div data-netdata="system.processes" + data-chart-library="easypiechart" + data-width="200" + data-height="200" + data-after="-300" + data-points="300" + data-dt-element-name="time601" + ></div> + <br/> + <small>rendered in <span id="time601">X</span> ms</small> + </div> + <div style="display: inline-block; position: relative;"> + <div data-netdata="system.processes" + data-chart-library="easypiechart" + data-width="150" + data-height="150" + data-after="-300" + data-points="150" + data-dt-element-name="time601a" + ></div> + <br/> + <small>rendered in <span id="time601a">X</span> ms</small> + </div> +</div> +<div style="width: 33%; display: inline-block;"> + <div style="display: inline-block; position: relative;"> + <div data-netdata="system.ipv4" + data-chart-library="easypiechart" + data-width="200" + data-height="200" + data-after="-300" + data-points="300" + data-dt-element-name="time602" + ></div> + <br/> + <small>rendered in <span id="time602">X</span> ms</small> + </div> + <div style="display: inline-block; position: relative;"> + <div data-netdata="system.ipv4" + data-chart-library="easypiechart" + data-width="100" + data-height="100" + data-after="-300" + data-points="150" + data-dt-element-name="time602a" + ></div> + <br/> + <small>rendered in <span id="time602a">X</span> ms</small> + </div> +</div> +<div style="width: 33%; display: inline-block;"> + <div style="display: inline-block; position: relative;"> + <div data-netdata="system.cpu" + data-chart-library="easypiechart" + data-width="200" + data-height="200" + data-after="-300" + data-points="300" + data-dt-element-name="time603" + ></div> + <br/> + <small>rendered in <span id="time603">X</span> ms</small> + </div> + <div style="display: inline-block; position: relative;"> + <div data-netdata="system.cpu" + data-chart-library="easypiechart" + data-width="75" + data-height="75" + data-after="-300" + data-points="150" + data-dt-element-name="time603a" + ></div> + <br/> + <small>rendered in <span id="time603a">X</span> ms</small> + </div> </div> @@ -427,82 +436,82 @@ Sparklines using dygraphs <h1>Gauge.js</h1> <br/> <div style="width: 33%; display: inline-block;"> - <div style="display: inline-block; position: relative;"> - <div data-netdata="system.processes" - data-chart-library="gauge" - data-width="250" - data-height="200" - data-after="-300" - data-points="300" - data-dt-element-name="time701" - ></div> - <br/> - <small>rendered in <span id="time701">X</span> ms</small> - </div> - <div style="display: inline-block; position: relative;"> - <div data-netdata="system.processes" - data-chart-library="gauge" - data-width="125" - data-height="100" - data-after="-300" - data-points="150" - data-dt-element-name="time701a" - ></div> - <br/> - <small>rendered in <span id="time701a">X</span> ms</small> - </div> -</div> -<div style="width: 33%; display: inline-block;"> - <div style="display: inline-block; position: relative;"> - <div data-netdata="system.ipv4" - data-chart-library="gauge" - data-width="250" - data-height="200" - data-after="-300" - data-points="300" - data-dt-element-name="time702" - ></div> - <br/> - <small>rendered in <span id="time702">X</span> ms</small> - </div> - <div style="display: inline-block; position: relative;"> - <div data-netdata="system.ipv4" - data-chart-library="gauge" - data-width="125" - data-height="100" - data-after="-300" - data-points="150" - data-dt-element-name="time702a" - ></div> - <br/> - <small>rendered in <span id="time702a">X</span> ms</small> - </div> -</div> -<div style="width: 33%; display: inline-block;"> - <div style="display: inline-block; position: relative;"> - <div data-netdata="system.cpu" - data-chart-library="gauge" - data-width="250" - data-height="200" - data-after="-300" - data-points="300" - data-dt-element-name="time703" - ></div> - <br/> - <small>rendered in <span id="time703">X</span> ms</small> - </div> - <div style="display: inline-block; position: relative;"> - <div data-netdata="system.cpu" - data-chart-library="gauge" - data-width="125" - data-height="100" - data-after="-300" - data-points="150" - data-dt-element-name="time703a" - ></div> - <br/> - <small>rendered in <span id="time703a">X</span> ms</small> - </div> + <div style="display: inline-block; position: relative;"> + <div data-netdata="system.processes" + data-chart-library="gauge" + data-width="250" + data-height="200" + data-after="-300" + data-points="300" + data-dt-element-name="time701" + ></div> + <br/> + <small>rendered in <span id="time701">X</span> ms</small> + </div> + <div style="display: inline-block; position: relative;"> + <div data-netdata="system.processes" + data-chart-library="gauge" + data-width="125" + data-height="100" + data-after="-300" + data-points="150" + data-dt-element-name="time701a" + ></div> + <br/> + <small>rendered in <span id="time701a">X</span> ms</small> + </div> +</div> +<div style="width: 33%; display: inline-block;"> + <div style="display: inline-block; position: relative;"> + <div data-netdata="system.ipv4" + data-chart-library="gauge" + data-width="250" + data-height="200" + data-after="-300" + data-points="300" + data-dt-element-name="time702" + ></div> + <br/> + <small>rendered in <span id="time702">X</span> ms</small> + </div> + <div style="display: inline-block; position: relative;"> + <div data-netdata="system.ipv4" + data-chart-library="gauge" + data-width="125" + data-height="100" + data-after="-300" + data-points="150" + data-dt-element-name="time702a" + ></div> + <br/> + <small>rendered in <span id="time702a">X</span> ms</small> + </div> +</div> +<div style="width: 33%; display: inline-block;"> + <div style="display: inline-block; position: relative;"> + <div data-netdata="system.cpu" + data-chart-library="gauge" + data-width="250" + data-height="200" + data-after="-300" + data-points="300" + data-dt-element-name="time703" + ></div> + <br/> + <small>rendered in <span id="time703">X</span> ms</small> + </div> + <div style="display: inline-block; position: relative;"> + <div data-netdata="system.cpu" + data-chart-library="gauge" + data-width="125" + data-height="100" + data-after="-300" + data-points="150" + data-dt-element-name="time703a" + ></div> + <br/> + <small>rendered in <span id="time703a">X</span> ms</small> + </div> </div> @@ -512,37 +521,37 @@ NetData was originaly developed with Google Charts. NetData is a complete Google Visualization API provider. <br/> <div style="width: 33%; display: inline-block;"> - <div data-netdata="system.processes" - data-chart-library="google" - data-width="100%" - data-height="200" - data-after="-300" - data-dt-element-name="time301" - ></div> - <br/> - <small>rendered in <span id="time301">X</span> ms</small> + <div data-netdata="system.processes" + data-chart-library="google" + data-width="100%" + data-height="200" + data-after="-300" + data-dt-element-name="time301" + ></div> + <br/> + <small>rendered in <span id="time301">X</span> ms</small> </div> <div style="width: 33%; display: inline-block;"> - <div data-netdata="system.ipv4" - data-chart-library="google" - data-width="100%" - data-height="200" - data-after="-300" - data-dt-element-name="time302" - ></div> - <br/> - <small>rendered in <span id="time302">X</span> ms</small> + <div data-netdata="system.ipv4" + data-chart-library="google" + data-width="100%" + data-height="200" + data-after="-300" + data-dt-element-name="time302" + ></div> + <br/> + <small>rendered in <span id="time302">X</span> ms</small> </div> <div style="width: 33%; display: inline-block;"> - <div data-netdata="system.cpu" - data-chart-library="google" - data-width="100%" - data-height="200" - data-after="-300" - data-dt-element-name="time303" - ></div> - <br/> - <small>rendered in <span id="time303">X</span> ms</small> + <div data-netdata="system.cpu" + data-chart-library="google" + data-width="100%" + data-height="200" + data-after="-300" + data-dt-element-name="time303" + ></div> + <br/> + <small>rendered in <span id="time303">X</span> ms</small> </div> @@ -555,37 +564,37 @@ NetData is a complete Google Visualization API provider. Unfortunatelly, Morris Charts are very slow. Here we force them to lower their detail to get acceptable results. <br/> <div style="width: 33%; display: inline-block;"> - <div data-netdata="system.processes" - data-chart-library="morris" - data-width="100%" - data-height="200" - data-after="-300" - data-dt-element-name="time401" - ></div> - <br/> - <small>rendered in <span id="time401">X</span> ms</small> + <div data-netdata="system.processes" + data-chart-library="morris" + data-width="100%" + data-height="200" + data-after="-300" + data-dt-element-name="time401" + ></div> + <br/> + <small>rendered in <span id="time401">X</span> ms</small> </div> <div style="width: 33%; display: inline-block;"> - <div data-netdata="system.ipv4" - data-chart-library="morris" - data-width="100%" - data-height="200" - data-after="-300" - data-dt-element-name="time402" - ></div> - <br/> - <small>rendered in <span id="time402">X</span> ms</small> + <div data-netdata="system.ipv4" + data-chart-library="morris" + data-width="100%" + data-height="200" + data-after="-300" + data-dt-element-name="time402" + ></div> + <br/> + <small>rendered in <span id="time402">X</span> ms</small> </div> <div style="width: 33%; display: inline-block;"> - <div data-netdata="system.cpu" - data-chart-library="morris" - data-width="100%" - data-height="200" - data-after="-300" - data-dt-element-name="time403" - ></div> - <br/> - <small>rendered in <span id="time403">X</span> ms</small> + <div data-netdata="system.cpu" + data-chart-library="morris" + data-width="100%" + data-height="200" + data-after="-300" + data-dt-element-name="time403" + ></div> + <br/> + <small>rendered in <span id="time403">X</span> ms</small> </div> @@ -601,37 +610,37 @@ C3 charts are not usable in large scale. They suffer from the following issues: So, to avoid flashing the charts, we destroy and re-create the charts on each update. Also, since they manipulate the data with javascript we were forced to lower the detail they render to get acceptable speeds. <br/> <div style="width: 33%; display: inline-block;"> - <div data-netdata="system.processes" - data-chart-library="c3" - data-width="100%" - data-height="200" - data-after="-300" - data-dt-element-name="time801" - ></div> - <br/> - <small>rendered in <span id="time801">X</span> ms</small> + <div data-netdata="system.processes" + data-chart-library="c3" + data-width="100%" + data-height="200" + data-after="-300" + data-dt-element-name="time801" + ></div> + <br/> + <small>rendered in <span id="time801">X</span> ms</small> </div> <div style="width: 33%; display: inline-block;"> - <div data-netdata="system.ipv4" - data-chart-library="c3" - data-width="100%" - data-height="200" - data-after="-300" - data-dt-element-name="time802" - ></div> - <br/> - <small>rendered in <span id="time802">X</span> ms</small> + <div data-netdata="system.ipv4" + data-chart-library="c3" + data-width="100%" + data-height="200" + data-after="-300" + data-dt-element-name="time802" + ></div> + <br/> + <small>rendered in <span id="time802">X</span> ms</small> </div> <div style="width: 33%; display: inline-block;"> - <div data-netdata="system.cpu" - data-chart-library="c3" - data-width="100%" - data-height="200" - data-after="-300" - data-dt-element-name="time803" - ></div> - <br/> - <small>rendered in <span id="time803">X</span> ms</small> + <div data-netdata="system.cpu" + data-chart-library="c3" + data-width="100%" + data-height="200" + data-after="-300" + data-dt-element-name="time803" + ></div> + <br/> + <small>rendered in <span id="time803">X</span> ms</small> </div> @@ -641,9 +650,9 @@ So, to avoid flashing the charts, we destroy and re-create the charts on each up </div> <!-- 1st container --> </body> </html> - <!-- you can set your netdata server globally, by ucommenting this --> - <!-- you can also give a different server per chart, with the attribute: data-host="http://netdata.server:19999" --> - <!-- <script> netdataServer = "http://box:19999"; </script> --> + <!-- you can set your netdata server globally, by ucommenting this --> + <!-- you can also give a different server per chart, with the attribute: data-host="http://netdata.server:19999" --> + <!-- <script> netdataServer = "http://box:19999"; </script> --> - <!-- load the dashboard manager - it will do the rest --> - <script type="text/javascript" src="dashboard.js?v37"></script> + <!-- load the dashboard manager - it will do the rest --> + <script type="text/javascript" src="dashboard.js?v42"></script> diff --git a/web/dashboard.js b/web/dashboard.js index 27847a243..67fb00b8b 100644 --- a/web/dashboard.js +++ b/web/dashboard.js @@ -1,20 +1,22 @@ // You can set the following variables before loading this script: // -// var netdataNoDygraphs = true; // do not use dygraph -// var netdataNoSparklines = true; // do not use sparkline -// var netdataNoPeitys = true; // do not use peity -// var netdataNoGoogleCharts = true; // do not use google -// var netdataNoMorris = true; // do not use morris -// var netdataNoEasyPieChart = true; // do not use easy pie chart -// var netdataNoGauge = true; // do not use gauge.js -// var netdataNoD3 = true; // do not use D3 -// var netdataNoC3 = true; // do not use C3 -// var netdataNoBootstrap = true; // do not load bootstrap -// var netdataDontStart = true; // do not start the thread to process the charts -// var netdataErrorCallback = null; // Callback function that will be invoked upon error -// var netdataNoRegistry = true; // Don't update the registry for this access -// var netdataRegistryCallback = null; // Callback function that will be invoked with one param, +// var netdataNoDygraphs = true; // do not use dygraph +// var netdataNoSparklines = true; // do not use sparkline +// var netdataNoPeitys = true; // do not use peity +// var netdataNoGoogleCharts = true; // do not use google +// var netdataNoMorris = true; // do not use morris +// var netdataNoEasyPieChart = true; // do not use easy pie chart +// var netdataNoGauge = true; // do not use gauge.js +// var netdataNoD3 = true; // do not use D3 +// var netdataNoC3 = true; // do not use C3 +// var netdataNoBootstrap = true; // do not load bootstrap +// var netdataDontStart = true; // do not start the thread to process the charts +// var netdataErrorCallback = null; // Callback function that will be invoked upon error +// var netdataNoRegistry = true; // Don't update the registry for this access +// var netdataRegistryCallback = null; // Callback function that will be invoked with one param, // the URLs from the registry +// var netdataShowHelp = true; // enable/disable help +// var netdataShowAlarms = true; // enable/disable help // // You can also set the default netdata server, using the following. // When this variable is not set, we assume the page is hosted on your @@ -23,5685 +25,5844 @@ //(function(window, document, undefined) { - // ------------------------------------------------------------------------ - // compatibility fixes - - // fix IE issue with console - if(!window.console) { window.console = { log: function(){} }; } - - // if string.endsWith is not defined, define it - if(typeof String.prototype.endsWith !== 'function') { - String.prototype.endsWith = function(s) { - if(s.length > this.length) return false; - return this.slice(-s.length) === s; - }; - } - - // if string.startsWith is not defined, define it - if(typeof String.prototype.startsWith !== 'function') { - String.prototype.startsWith = function(s) { - if(s.length > this.length) return false; - return this.slice(s.length) === s; - }; - } - - // global namespace - var NETDATA = window.NETDATA || {}; - - // ---------------------------------------------------------------------------------------------------------------- - // Detect the netdata server - - // http://stackoverflow.com/questions/984510/what-is-my-script-src-url - // http://stackoverflow.com/questions/6941533/get-protocol-domain-and-port-from-url - NETDATA._scriptSource = function() { - var script = null; - - if(typeof document.currentScript !== 'undefined') { - script = document.currentScript; - } - else { - var all_scripts = document.getElementsByTagName('script'); - script = all_scripts[all_scripts.length - 1]; - } - - if (typeof script.getAttribute.length !== 'undefined') - script = script.src; - else - script = script.getAttribute('src', -1); - - return script; - }; - - if(typeof netdataServer !== 'undefined') - NETDATA.serverDefault = netdataServer; - else { - var s = NETDATA._scriptSource(); - if(s) NETDATA.serverDefault = s.replace(/\/dashboard.js(\?.*)*$/g, ""); - else { - console.log('WARNING: Cannot detect the URL of the netdata server.'); - NETDATA.serverDefault = null; - } - } - - if(NETDATA.serverDefault === null) - NETDATA.serverDefault = ''; - else if(NETDATA.serverDefault.slice(-1) !== '/') - NETDATA.serverDefault += '/'; - - // default URLs for all the external files we need - // make them RELATIVE so that the whole thing can also be - // installed under a web server - NETDATA.jQuery = NETDATA.serverDefault + 'lib/jquery-1.12.0.min.js'; - NETDATA.peity_js = NETDATA.serverDefault + 'lib/jquery.peity.min.js'; - NETDATA.sparkline_js = NETDATA.serverDefault + 'lib/jquery.sparkline.min.js'; - NETDATA.easypiechart_js = NETDATA.serverDefault + 'lib/jquery.easypiechart.min.js'; - NETDATA.gauge_js = NETDATA.serverDefault + 'lib/gauge.min.js'; - NETDATA.dygraph_js = NETDATA.serverDefault + 'lib/dygraph-combined.js'; - NETDATA.dygraph_smooth_js = NETDATA.serverDefault + 'lib/dygraph-smooth-plotter.js'; - NETDATA.raphael_js = NETDATA.serverDefault + 'lib/raphael-min.js'; - NETDATA.morris_js = NETDATA.serverDefault + 'lib/morris.min.js'; - NETDATA.d3_js = NETDATA.serverDefault + 'lib/d3.min.js'; - NETDATA.c3_js = NETDATA.serverDefault + 'lib/c3.min.js'; - NETDATA.c3_css = NETDATA.serverDefault + 'css/c3.min.css'; - NETDATA.morris_css = NETDATA.serverDefault + 'css/morris.css'; - NETDATA.google_js = 'https://www.google.com/jsapi'; - - NETDATA.themes = { - white: { - bootstrap_css: NETDATA.serverDefault + 'css/bootstrap.min.css', - dashboard_css: NETDATA.serverDefault + 'dashboard.css', - background: '#FFFFFF', - foreground: '#000000', - grid: '#DDDDDD', - axis: '#CCCCCC', - colors: [ '#3366CC', '#DC3912', '#109618', '#FF9900', '#990099', '#DD4477', - '#3B3EAC', '#66AA00', '#0099C6', '#B82E2E', '#AAAA11', '#5574A6', - '#994499', '#22AA99', '#6633CC', '#E67300', '#316395', '#8B0707', - '#329262', '#3B3EAC' ], - easypiechart_track: '#f0f0f0', - easypiechart_scale: '#dfe0e0', - gauge_pointer: '#C0C0C0', - gauge_stroke: '#F0F0F0', - gauge_gradient: false - }, - slate: { - bootstrap_css: NETDATA.serverDefault + 'css/bootstrap.slate.min.css', - dashboard_css: NETDATA.serverDefault + 'dashboard.slate.css', - background: '#272b30', - foreground: '#C8C8C8', - grid: '#373b40', - axis: '#373b40', -/* colors: [ '#55bb33', '#ff2222', '#0099C6', '#faa11b', '#adbce0', '#DDDD00', - '#4178ba', '#f58122', '#a5cc39', '#f58667', '#f5ef89', '#cf93c0', - '#a5d18a', '#b8539d', '#3954a3', '#c8a9cf', '#c7de8a', '#fad20a', - '#a6a479', '#a66da8' ], + // ------------------------------------------------------------------------ + // compatibility fixes + + // fix IE issue with console + if(!window.console) { window.console = { log: function(){} }; } + + // if string.endsWith is not defined, define it + if(typeof String.prototype.endsWith !== 'function') { + String.prototype.endsWith = function(s) { + if(s.length > this.length) return false; + return this.slice(-s.length) === s; + }; + } + + // if string.startsWith is not defined, define it + if(typeof String.prototype.startsWith !== 'function') { + String.prototype.startsWith = function(s) { + if(s.length > this.length) return false; + return this.slice(s.length) === s; + }; + } + + // global namespace + var NETDATA = window.NETDATA || {}; + + // ---------------------------------------------------------------------------------------------------------------- + // Detect the netdata server + + // http://stackoverflow.com/questions/984510/what-is-my-script-src-url + // http://stackoverflow.com/questions/6941533/get-protocol-domain-and-port-from-url + NETDATA._scriptSource = function() { + var script = null; + + if(typeof document.currentScript !== 'undefined') { + script = document.currentScript; + } + else { + var all_scripts = document.getElementsByTagName('script'); + script = all_scripts[all_scripts.length - 1]; + } + + if (typeof script.getAttribute.length !== 'undefined') + script = script.src; + else + script = script.getAttribute('src', -1); + + return script; + }; + + if(typeof netdataServer !== 'undefined') + NETDATA.serverDefault = netdataServer; + else { + var s = NETDATA._scriptSource(); + if(s) NETDATA.serverDefault = s.replace(/\/dashboard.js(\?.*)*$/g, ""); + else { + console.log('WARNING: Cannot detect the URL of the netdata server.'); + NETDATA.serverDefault = null; + } + } + + if(NETDATA.serverDefault === null) + NETDATA.serverDefault = ''; + else if(NETDATA.serverDefault.slice(-1) !== '/') + NETDATA.serverDefault += '/'; + + // default URLs for all the external files we need + // make them RELATIVE so that the whole thing can also be + // installed under a web server + NETDATA.jQuery = NETDATA.serverDefault + 'lib/jquery-1.12.0.min.js'; + NETDATA.peity_js = NETDATA.serverDefault + 'lib/jquery.peity.min.js'; + NETDATA.sparkline_js = NETDATA.serverDefault + 'lib/jquery.sparkline.min.js'; + NETDATA.easypiechart_js = NETDATA.serverDefault + 'lib/jquery.easypiechart.min.js'; + NETDATA.gauge_js = NETDATA.serverDefault + 'lib/gauge.min.js'; + NETDATA.dygraph_js = NETDATA.serverDefault + 'lib/dygraph-combined.js'; + NETDATA.dygraph_smooth_js = NETDATA.serverDefault + 'lib/dygraph-smooth-plotter.js'; + NETDATA.raphael_js = NETDATA.serverDefault + 'lib/raphael-min.js'; + NETDATA.morris_js = NETDATA.serverDefault + 'lib/morris.min.js'; + NETDATA.d3_js = NETDATA.serverDefault + 'lib/d3.min.js'; + NETDATA.c3_js = NETDATA.serverDefault + 'lib/c3.min.js'; + NETDATA.c3_css = NETDATA.serverDefault + 'css/c3.min.css'; + NETDATA.morris_css = NETDATA.serverDefault + 'css/morris.css'; + NETDATA.google_js = 'https://www.google.com/jsapi'; + + NETDATA.themes = { + white: { + bootstrap_css: NETDATA.serverDefault + 'css/bootstrap.min.css', + dashboard_css: NETDATA.serverDefault + 'dashboard.css', + background: '#FFFFFF', + foreground: '#000000', + grid: '#DDDDDD', + axis: '#CCCCCC', + colors: [ '#3366CC', '#DC3912', '#109618', '#FF9900', '#990099', '#DD4477', + '#3B3EAC', '#66AA00', '#0099C6', '#B82E2E', '#AAAA11', '#5574A6', + '#994499', '#22AA99', '#6633CC', '#E67300', '#316395', '#8B0707', + '#329262', '#3B3EAC' ], + easypiechart_track: '#f0f0f0', + easypiechart_scale: '#dfe0e0', + gauge_pointer: '#C0C0C0', + gauge_stroke: '#F0F0F0', + gauge_gradient: false + }, + slate: { + bootstrap_css: NETDATA.serverDefault + 'css/bootstrap.slate.min.css', + dashboard_css: NETDATA.serverDefault + 'dashboard.slate.css', + background: '#272b30', + foreground: '#C8C8C8', + grid: '#373b40', + axis: '#373b40', +/* colors: [ '#55bb33', '#ff2222', '#0099C6', '#faa11b', '#adbce0', '#DDDD00', + '#4178ba', '#f58122', '#a5cc39', '#f58667', '#f5ef89', '#cf93c0', + '#a5d18a', '#b8539d', '#3954a3', '#c8a9cf', '#c7de8a', '#fad20a', + '#a6a479', '#a66da8' ], */ - colors: [ '#66AA00', '#FE3912', '#3366CC', '#D66300', '#0099C6', '#DDDD00', - '#3B3EAC', '#EE9911', '#BB44CC', '#C83E3E', '#990099', '#CC7700', - '#22AA99', '#109618', '#6633CC', '#DD4477', '#316395', '#8B0707', - '#329262', '#3B3EFF' ], - easypiechart_track: '#373b40', - easypiechart_scale: '#373b40', - gauge_pointer: '#474b50', - gauge_stroke: '#373b40', - gauge_gradient: false - } - }; - - if(typeof netdataTheme !== 'undefined' && typeof NETDATA.themes[netdataTheme] !== 'undefined') - NETDATA.themes.current = NETDATA.themes[netdataTheme]; - else - NETDATA.themes.current = NETDATA.themes.white; - - NETDATA.colors = NETDATA.themes.current.colors; - - // these are the colors Google Charts are using - // we have them here to attempt emulate their look and feel on the other chart libraries - // http://there4.io/2012/05/02/google-chart-color-list/ - //NETDATA.colors = [ '#3366CC', '#DC3912', '#FF9900', '#109618', '#990099', '#3B3EAC', '#0099C6', - // '#DD4477', '#66AA00', '#B82E2E', '#316395', '#994499', '#22AA99', '#AAAA11', - // '#6633CC', '#E67300', '#8B0707', '#329262', '#5574A6', '#3B3EAC' ]; + colors: [ '#66AA00', '#FE3912', '#3366CC', '#D66300', '#0099C6', '#DDDD00', + '#5054e6', '#EE9911', '#BB44CC', '#e45757', '#ef0aef', '#CC7700', + '#22AA99', '#109618', '#905bfd', '#f54882', '#4381bf', '#ff3737', + '#329262', '#3B3EFF' ], + easypiechart_track: '#373b40', + easypiechart_scale: '#373b40', + gauge_pointer: '#474b50', + gauge_stroke: '#373b40', + gauge_gradient: false + } + }; + + if(typeof netdataTheme !== 'undefined' && typeof NETDATA.themes[netdataTheme] !== 'undefined') + NETDATA.themes.current = NETDATA.themes[netdataTheme]; + else + NETDATA.themes.current = NETDATA.themes.white; + + if(typeof netdataShowHelp === 'undefined') + netdataShowHelp = true; + + if(typeof netdataShowAlarms === 'undefined') + netdataShowAlarms = true; + + NETDATA.colors = NETDATA.themes.current.colors; + + // these are the colors Google Charts are using + // we have them here to attempt emulate their look and feel on the other chart libraries + // http://there4.io/2012/05/02/google-chart-color-list/ + //NETDATA.colors = [ '#3366CC', '#DC3912', '#FF9900', '#109618', '#990099', '#3B3EAC', '#0099C6', + // '#DD4477', '#66AA00', '#B82E2E', '#316395', '#994499', '#22AA99', '#AAAA11', + // '#6633CC', '#E67300', '#8B0707', '#329262', '#5574A6', '#3B3EAC' ]; - // an alternative set - // http://www.mulinblog.com/a-color-palette-optimized-for-data-visualization/ - // (blue) (red) (orange) (green) (pink) (brown) (purple) (yellow) (gray) - //NETDATA.colors = [ '#5DA5DA', '#F15854', '#FAA43A', '#60BD68', '#F17CB0', '#B2912F', '#B276B2', '#DECF3F', '#4D4D4D' ]; + // an alternative set + // http://www.mulinblog.com/a-color-palette-optimized-for-data-visualization/ + // (blue) (red) (orange) (green) (pink) (brown) (purple) (yellow) (gray) + //NETDATA.colors = [ '#5DA5DA', '#F15854', '#FAA43A', '#60BD68', '#F17CB0', '#B2912F', '#B276B2', '#DECF3F', '#4D4D4D' ]; - // ---------------------------------------------------------------------------------------------------------------- - // the defaults for all charts + // ---------------------------------------------------------------------------------------------------------------- + // the defaults for all charts - // if the user does not specify any of these, the following will be used + // if the user does not specify any of these, the following will be used - NETDATA.chartDefaults = { - host: NETDATA.serverDefault, // the server to get data from - width: '100%', // the chart width - can be null - height: '100%', // the chart height - can be null - min_width: null, // the chart minimum width - can be null - library: 'dygraph', // the graphing library to use - method: 'average', // the grouping method - before: 0, // panning - after: -600, // panning - pixels_per_point: 1, // the detail of the chart - fill_luminance: 0.8 // luminance of colors in solit areas - }; + NETDATA.chartDefaults = { + host: NETDATA.serverDefault, // the server to get data from + width: '100%', // the chart width - can be null + height: '100%', // the chart height - can be null + min_width: null, // the chart minimum width - can be null + library: 'dygraph', // the graphing library to use + method: 'average', // the grouping method + before: 0, // panning + after: -600, // panning + pixels_per_point: 1, // the detail of the chart + fill_luminance: 0.8 // luminance of colors in solit areas + }; - // ---------------------------------------------------------------------------------------------------------------- - // global options + // ---------------------------------------------------------------------------------------------------------------- + // global options - NETDATA.options = { - pauseCallback: null, // a callback when we are really paused + NETDATA.options = { + pauseCallback: null, // a callback when we are really paused - pause: false, // when enabled we don't auto-refresh the charts + pause: false, // when enabled we don't auto-refresh the charts - targets: null, // an array of all the state objects that are - // currently active (independently of their - // viewport visibility) + targets: null, // an array of all the state objects that are + // currently active (independently of their + // viewport visibility) - updated_dom: true, // when true, the DOM has been updated with - // new elements we have to check. + updated_dom: true, // when true, the DOM has been updated with + // new elements we have to check. - auto_refresher_fast_weight: 0, // this is the current time in ms, spent - // rendering charts continiously. - // used with .current.fast_render_timeframe + auto_refresher_fast_weight: 0, // this is the current time in ms, spent + // rendering charts continiously. + // used with .current.fast_render_timeframe - page_is_visible: true, // when true, this page is visible + page_is_visible: true, // when true, this page is visible - auto_refresher_stop_until: 0, // timestamp in ms - used internaly, to stop the - // auto-refresher for some time (when a chart is - // performing pan or zoom, we need to stop refreshing - // all other charts, to have the maximum speed for - // rendering the chart that is panned or zoomed). - // Used with .current.global_pan_sync_time + auto_refresher_stop_until: 0, // timestamp in ms - used internaly, to stop the + // auto-refresher for some time (when a chart is + // performing pan or zoom, we need to stop refreshing + // all other charts, to have the maximum speed for + // rendering the chart that is panned or zoomed). + // Used with .current.global_pan_sync_time - last_resized: new Date().getTime(), // the timestamp of the last resize request + last_resized: new Date().getTime(), // the timestamp of the last resize request - last_page_scroll: 0, // the timestamp the last time the page was scrolled + last_page_scroll: 0, // the timestamp the last time the page was scrolled - // the current profile - // we may have many... - current: { - pixels_per_point: 1, // the minimum pixels per point for all charts - // increase this to speed javascript up - // each chart library has its own limit too - // the max of this and the chart library is used - // the final is calculated every time, so a change - // here will have immediate effect on the next chart - // update + // the current profile + // we may have many... + current: { + pixels_per_point: 1, // the minimum pixels per point for all charts + // increase this to speed javascript up + // each chart library has its own limit too + // the max of this and the chart library is used + // the final is calculated every time, so a change + // here will have immediate effect on the next chart + // update - idle_between_charts: 100, // ms - how much time to wait between chart updates + idle_between_charts: 100, // ms - how much time to wait between chart updates - fast_render_timeframe: 200, // ms - render continously until this time of continious - // rendering has been reached - // this setting is used to make it render e.g. 10 - // charts at once, sleep idle_between_charts time - // and continue for another 10 charts. + fast_render_timeframe: 200, // ms - render continously until this time of continious + // rendering has been reached + // this setting is used to make it render e.g. 10 + // charts at once, sleep idle_between_charts time + // and continue for another 10 charts. - idle_between_loops: 500, // ms - if all charts have been updated, wait this - // time before starting again. + idle_between_loops: 500, // ms - if all charts have been updated, wait this + // time before starting again. - idle_parallel_loops: 100, // ms - the time between parallel refresher updates + idle_parallel_loops: 100, // ms - the time between parallel refresher updates - idle_lost_focus: 500, // ms - when the window does not have focus, check - // if focus has been regained, every this time + idle_lost_focus: 500, // ms - when the window does not have focus, check + // if focus has been regained, every this time - global_pan_sync_time: 1000, // ms - when you pan or zoon a chart, the background - // autorefreshing of charts is paused for this amount - // of time + global_pan_sync_time: 1000, // ms - when you pan or zoon a chart, the background + // autorefreshing of charts is paused for this amount + // of time - sync_selection_delay: 1500, // ms - when you pan or zoom a chart, wait this amount - // of time before setting up synchronized selections - // on hover. - - sync_selection: true, // enable or disable selection sync - - pan_and_zoom_delay: 50, // when panning or zooming, how ofter to update the chart - - sync_pan_and_zoom: true, // enable or disable pan and zoom sync - - pan_and_zoom_data_padding: true, // fetch more data for the master chart when panning or zooming - - update_only_visible: true, // enable or disable visibility management - - parallel_refresher: true, // enable parallel refresh of charts - - concurrent_refreshes: true, // when parallel_refresher is enabled, sync also the charts - - destroy_on_hide: false, // destroy charts when they are not visible - - show_help: true, // when enabled the charts will show some help - show_help_delay_show_ms: 500, - show_help_delay_hide_ms: 0, - - eliminate_zero_dimensions: true, // do not show dimensions with just zeros - - stop_updates_when_focus_is_lost: true, // boolean - shall we stop auto-refreshes when document does not have user focus - stop_updates_while_resizing: 1000, // ms - time to stop auto-refreshes while resizing the charts - - double_click_speed: 500, // ms - time between clicks / taps to detect double click/tap - - smooth_plot: true, // enable smooth plot, where possible - - charts_selection_animation_delay: 50, // delay to animate charts when syncing selection - - color_fill_opacity_line: 1.0, - color_fill_opacity_area: 0.2, - color_fill_opacity_stacked: 0.8, - - pan_and_zoom_factor: 0.25, // the increment when panning and zooming with the toolbox - pan_and_zoom_factor_multiplier_control: 2.0, - pan_and_zoom_factor_multiplier_shift: 3.0, - pan_and_zoom_factor_multiplier_alt: 4.0, - - setOptionCallback: function() { ; } - }, - - debug: { - show_boxes: false, - main_loop: false, - focus: false, - visibility: false, - chart_data_url: false, - chart_errors: false, // FIXME - chart_timing: false, - chart_calls: false, - libraries: false, - dygraph: false - } - }; - - NETDATA.statistics = { - refreshes_total: 0, - refreshes_active: 0, - refreshes_active_max: 0 - }; - - - // ---------------------------------------------------------------------------------------------------------------- - // local storage options - - NETDATA.localStorage = { - default: {}, - current: {}, - callback: {} // only used for resetting back to defaults - }; - - NETDATA.localStorageGet = function(key, def, callback) { - var ret = def; - - if(typeof NETDATA.localStorage.default[key.toString()] === 'undefined') { - NETDATA.localStorage.default[key.toString()] = def; - NETDATA.localStorage.callback[key.toString()] = callback; - } - - if(typeof Storage !== "undefined" && typeof localStorage === 'object') { - try { - // console.log('localStorage: loading "' + key.toString() + '"'); - ret = localStorage.getItem(key.toString()); - if(ret === null || ret === 'undefined') { - // console.log('localStorage: cannot load it, saving "' + key.toString() + '" with value "' + JSON.stringify(def) + '"'); - localStorage.setItem(key.toString(), JSON.stringify(def)); - ret = def; - } - else { - // console.log('localStorage: got "' + key.toString() + '" with value "' + ret + '"'); - ret = JSON.parse(ret); - // console.log('localStorage: loaded "' + key.toString() + '" as value ' + ret + ' of type ' + typeof(ret)); - } - } - catch(error) { - console.log('localStorage: failed to read "' + key.toString() + '", using default: "' + def.toString() + '"'); - ret = def; - } - } - - if(typeof ret === 'undefined' || ret === 'undefined') { - console.log('localStorage: LOADED UNDEFINED "' + key.toString() + '" as value ' + ret + ' of type ' + typeof(ret)); - ret = def; - } - - NETDATA.localStorage.current[key.toString()] = ret; - return ret; - }; - - NETDATA.localStorageSet = function(key, value, callback) { - if(typeof value === 'undefined' || value === 'undefined') { - console.log('localStorage: ATTEMPT TO SET UNDEFINED "' + key.toString() + '" as value ' + value + ' of type ' + typeof(value)); - } - - if(typeof NETDATA.localStorage.default[key.toString()] === 'undefined') { - NETDATA.localStorage.default[key.toString()] = value; - NETDATA.localStorage.current[key.toString()] = value; - NETDATA.localStorage.callback[key.toString()] = callback; - } - - if(typeof Storage !== "undefined" && typeof localStorage === 'object') { - // console.log('localStorage: saving "' + key.toString() + '" with value "' + JSON.stringify(value) + '"'); - try { - localStorage.setItem(key.toString(), JSON.stringify(value)); - } - catch(e) { - console.log('localStorage: failed to save "' + key.toString() + '" with value: "' + value.toString() + '"'); - } - } - - NETDATA.localStorage.current[key.toString()] = value; - return value; - }; - - NETDATA.localStorageGetRecursive = function(obj, prefix, callback) { - for(var i in obj) { - if(typeof obj[i] === 'object') { - //console.log('object ' + prefix + '.' + i.toString()); - NETDATA.localStorageGetRecursive(obj[i], prefix + '.' + i.toString(), callback); - continue; - } - - obj[i] = NETDATA.localStorageGet(prefix + '.' + i.toString(), obj[i], callback); - } - }; - - NETDATA.setOption = function(key, value) { - if(key.toString() === 'setOptionCallback') { - if(typeof NETDATA.options.current.setOptionCallback === 'function') { - NETDATA.options.current[key.toString()] = value; - NETDATA.options.current.setOptionCallback(); - } - } - else if(NETDATA.options.current[key.toString()] !== value) { - var name = 'options.' + key.toString(); - - if(typeof NETDATA.localStorage.default[name.toString()] === 'undefined') - console.log('localStorage: setOption() on unsaved option: "' + name.toString() + '", value: ' + value); - - //console.log(NETDATA.localStorage); - //console.log('setOption: setting "' + key.toString() + '" to "' + value + '" of type ' + typeof(value) + ' original type ' + typeof(NETDATA.options.current[key.toString()])); - //console.log(NETDATA.options); - NETDATA.options.current[key.toString()] = NETDATA.localStorageSet(name.toString(), value, null); - - if(typeof NETDATA.options.current.setOptionCallback === 'function') - NETDATA.options.current.setOptionCallback(); - } - - return true; - }; - - NETDATA.getOption = function(key) { - return NETDATA.options.current[key.toString()]; - }; - - // read settings from local storage - NETDATA.localStorageGetRecursive(NETDATA.options.current, 'options', null); - - // always start with this option enabled. - NETDATA.setOption('stop_updates_when_focus_is_lost', true); - - NETDATA.resetOptions = function() { - for(var i in NETDATA.localStorage.default) { - var a = i.split('.'); - - if(a[0] === 'options') { - if(a[1] === 'setOptionCallback') continue; - if(typeof NETDATA.localStorage.default[i] === 'undefined') continue; - if(NETDATA.options.current[i] === NETDATA.localStorage.default[i]) continue; - - NETDATA.setOption(a[1], NETDATA.localStorage.default[i]); - } - else if(a[0] === 'chart_heights') { - if(typeof NETDATA.localStorage.callback[i] === 'function' && typeof NETDATA.localStorage.default[i] !== 'undefined') { - NETDATA.localStorage.callback[i](NETDATA.localStorage.default[i]); - } - } - } - } - - // ---------------------------------------------------------------------------------------------------------------- - - if(NETDATA.options.debug.main_loop === true) - console.log('welcome to NETDATA'); - - NETDATA.onresize = function() { - NETDATA.options.last_resized = new Date().getTime(); - NETDATA.onscroll(); - }; - - NETDATA.onscroll = function() { - // console.log('onscroll'); - - NETDATA.options.last_page_scroll = new Date().getTime(); - if(NETDATA.options.targets === null) return; - - // when the user scrolls he sees that we have - // hidden all the not-visible charts - // using this little function we try to switch - // the charts back to visible quickly - var targets = NETDATA.options.targets; - var len = targets.length; - while(len--) targets[len].isVisible(); - }; - - window.onresize = NETDATA.onresize; - window.onscroll = NETDATA.onscroll; - - // ---------------------------------------------------------------------------------------------------------------- - // Error Handling - - NETDATA.errorCodes = { - 100: { message: "Cannot load chart library", alert: true }, - 101: { message: "Cannot load jQuery", alert: true }, - 402: { message: "Chart library not found", alert: false }, - 403: { message: "Chart library not enabled/is failed", alert: false }, - 404: { message: "Chart not found", alert: false }, - 405: { message: "Cannot download charts index from server", alert: true }, - 406: { message: "Invalid charts index downloaded from server", alert: true }, - 407: { message: "Cannot HELLO netdata server", alert: false }, - 408: { message: "Netdata servers sent invalid response to HELLO", alert: false }, - 409: { message: "Cannot ACCESS netdata registry", alert: false }, - 410: { message: "Netdata registry ACCESS failed", alert: false }, - 411: { message: "Netdata registry server send invalid response to DELETE ", alert: false }, - 412: { message: "Netdata registry DELETE failed", alert: false }, - 413: { message: "Netdata registry server send invalid response to SWITCH ", alert: false }, - 414: { message: "Netdata registry SWITCH failed", alert: false } - }; - NETDATA.errorLast = { - code: 0, - message: "", - datetime: 0 - }; - - NETDATA.error = function(code, msg) { - NETDATA.errorLast.code = code; - NETDATA.errorLast.message = msg; - NETDATA.errorLast.datetime = new Date().getTime(); - - console.log("ERROR " + code + ": " + NETDATA.errorCodes[code].message + ": " + msg); - - var ret = true; - if(typeof netdataErrorCallback === 'function') { - ret = netdataErrorCallback('system', code, msg); - } - - if(ret && NETDATA.errorCodes[code].alert) - alert("ERROR " + code + ": " + NETDATA.errorCodes[code].message + ": " + msg); - }; - - NETDATA.errorReset = function() { - NETDATA.errorLast.code = 0; - NETDATA.errorLast.message = "You are doing fine!"; - NETDATA.errorLast.datetime = 0; - }; - - // ---------------------------------------------------------------------------------------------------------------- - // Chart Registry - - // When multiple charts need the same chart, we avoid downloading it - // multiple times (and having it in browser memory multiple time) - // by using this registry. - - // Every time we download a chart definition, we save it here with .add() - // Then we try to get it back with .get(). If that fails, we download it. - - NETDATA.chartRegistry = { - charts: {}, - - fixid: function(id) { - return id.replace(/:/g, "_").replace(/\//g, "_"); - }, - - add: function(host, id, data) { - host = this.fixid(host); - id = this.fixid(id); - - if(typeof this.charts[host] === 'undefined') - this.charts[host] = {}; - - //console.log('added ' + host + '/' + id); - this.charts[host][id] = data; - }, - - get: function(host, id) { - host = this.fixid(host); - id = this.fixid(id); - - if(typeof this.charts[host] === 'undefined') - return null; - - if(typeof this.charts[host][id] === 'undefined') - return null; - - //console.log('cached ' + host + '/' + id); - return this.charts[host][id]; - }, - - downloadAll: function(host, callback) { - while(host.slice(-1) === '/') - host = host.substring(0, host.length - 1); - - var self = this; - - $.ajax({ - url: host + '/api/v1/charts', - async: true, - cache: false - }) - .done(function(data) { - if(data !== null) { - var h = NETDATA.chartRegistry.fixid(host); - self.charts[h] = data.charts; - } - else NETDATA.error(406, host + '/api/v1/charts'); - - if(typeof callback === 'function') - callback(data); - }) - .fail(function() { - NETDATA.error(405, host + '/api/v1/charts'); - - if(typeof callback === 'function') - callback(null); - }); - } - }; - - // ---------------------------------------------------------------------------------------------------------------- - // Global Pan and Zoom on charts - - // Using this structure are synchronize all the charts, so that - // when you pan or zoom one, all others are automatically refreshed - // to the same timespan. - - NETDATA.globalPanAndZoom = { - seq: 0, // timestamp ms - // every time a chart is panned or zoomed - // we set the timestamp here - // then we use it as a sequence number - // to find if other charts are syncronized - // to this timerange - - master: null, // the master chart (state), to which all others - // are synchronized - - force_before_ms: null, // the timespan to sync all other charts - force_after_ms: null, - - // set a new master - setMaster: function(state, after, before) { - if(NETDATA.options.current.sync_pan_and_zoom === false) - return; - - if(this.master !== null && this.master !== state) - this.master.resetChart(true, true); - - var now = new Date().getTime(); - this.master = state; - this.seq = now; - this.force_after_ms = after; - this.force_before_ms = before; - NETDATA.options.auto_refresher_stop_until = now + NETDATA.options.current.global_pan_sync_time; - }, - - // clear the master - clearMaster: function() { - if(this.master !== null) { - var st = this.master; - this.master = null; - st.resetChart(); - } - - this.master = null; - this.seq = 0; - this.force_after_ms = null; - this.force_before_ms = null; - NETDATA.options.auto_refresher_stop_until = 0; - }, - - // is the given state the master of the global - // pan and zoom sync? - isMaster: function(state) { - if(this.master === state) return true; - return false; - }, - - // are we currently have a global pan and zoom sync? - isActive: function() { - if(this.master !== null && this.force_before_ms !== null && this.force_after_ms !== null && this.seq !== 0) return true; - return false; - }, - - // check if a chart, other than the master - // needs to be refreshed, due to the global pan and zoom - shouldBeAutoRefreshed: function(state) { - if(this.master === null || this.seq === 0) - return false; - - //if(state.needsRecreation()) - // return true; - - if(state.tm.pan_and_zoom_seq === this.seq) - return false; - - return true; - } - }; - - // ---------------------------------------------------------------------------------------------------------------- - // dimensions selection - - // FIXME - // move color assignment to dimensions, here - - dimensionStatus = function(parent, label, name_div, value_div, color) { - this.enabled = false; - this.parent = parent; - this.label = label; - this.name_div = null; - this.value_div = null; - this.color = NETDATA.themes.current.foreground; - - if(parent.selected_count > parent.unselected_count) - this.selected = true; - else - this.selected = false; - - this.setOptions(name_div, value_div, color); - }; - - dimensionStatus.prototype.invalidate = function() { - this.name_div = null; - this.value_div = null; - this.enabled = false; - }; - - dimensionStatus.prototype.setOptions = function(name_div, value_div, color) { - this.color = color; - - if(this.name_div != name_div) { - this.name_div = name_div; - this.name_div.title = this.label; - this.name_div.style.color = this.color; - if(this.selected === false) - this.name_div.className = 'netdata-legend-name not-selected'; - else - this.name_div.className = 'netdata-legend-name selected'; - } - - if(this.value_div != value_div) { - this.value_div = value_div; - this.value_div.title = this.label; - this.value_div.style.color = this.color; - if(this.selected === false) - this.value_div.className = 'netdata-legend-value not-selected'; - else - this.value_div.className = 'netdata-legend-value selected'; - } - - this.enabled = true; - this.setHandler(); - }; - - dimensionStatus.prototype.setHandler = function() { - if(this.enabled === false) return; - - var ds = this; - - // this.name_div.onmousedown = this.value_div.onmousedown = function(e) { - this.name_div.onclick = this.value_div.onclick = function(e) { - e.preventDefault(); - if(ds.isSelected()) { - // this is selected - if(e.shiftKey === true || e.ctrlKey === true) { - // control or shift key is pressed -> unselect this (except is none will remain selected, in which case select all) - ds.unselect(); - - if(ds.parent.countSelected() === 0) - ds.parent.selectAll(); - } - else { - // no key is pressed -> select only this (except if it is the only selected already, in which case select all) - if(ds.parent.countSelected() === 1) { - ds.parent.selectAll(); - } - else { - ds.parent.selectNone(); - ds.select(); - } - } - } - else { - // this is not selected - if(e.shiftKey === true || e.ctrlKey === true) { - // control or shift key is pressed -> select this too - ds.select(); - } - else { - // no key is pressed -> select only this - ds.parent.selectNone(); - ds.select(); - } - } - - ds.parent.state.redrawChart(); - } - }; - - dimensionStatus.prototype.select = function() { - if(this.enabled === false) return; - - this.name_div.className = 'netdata-legend-name selected'; - this.value_div.className = 'netdata-legend-value selected'; - this.selected = true; - }; - - dimensionStatus.prototype.unselect = function() { - if(this.enabled === false) return; - - this.name_div.className = 'netdata-legend-name not-selected'; - this.value_div.className = 'netdata-legend-value hidden'; - this.selected = false; - }; - - dimensionStatus.prototype.isSelected = function() { - return(this.enabled === true && this.selected === true); - }; - - // ---------------------------------------------------------------------------------------------------------------- - - dimensionsVisibility = function(state) { - this.state = state; - this.len = 0; - this.dimensions = {}; - this.selected_count = 0; - this.unselected_count = 0; - }; - - dimensionsVisibility.prototype.dimensionAdd = function(label, name_div, value_div, color) { - if(typeof this.dimensions[label] === 'undefined') { - this.len++; - this.dimensions[label] = new dimensionStatus(this, label, name_div, value_div, color); - } - else - this.dimensions[label].setOptions(name_div, value_div, color); - - return this.dimensions[label]; - }; - - dimensionsVisibility.prototype.dimensionGet = function(label) { - return this.dimensions[label]; - }; - - dimensionsVisibility.prototype.invalidateAll = function() { - for(var d in this.dimensions) - this.dimensions[d].invalidate(); - }; - - dimensionsVisibility.prototype.selectAll = function() { - for(var d in this.dimensions) - this.dimensions[d].select(); - }; - - dimensionsVisibility.prototype.countSelected = function() { - var i = 0; - for(var d in this.dimensions) - if(this.dimensions[d].isSelected()) i++; - - return i; - }; - - dimensionsVisibility.prototype.selectNone = function() { - for(var d in this.dimensions) - this.dimensions[d].unselect(); - }; - - dimensionsVisibility.prototype.selected2BooleanArray = function(array) { - var ret = new Array(); - this.selected_count = 0; - this.unselected_count = 0; - - for(var i = 0, len = array.length; i < len ; i++) { - var ds = this.dimensions[array[i]]; - if(typeof ds === 'undefined') { - // console.log(array[i] + ' is not found'); - ret.push(false); - continue; - } - - if(ds.isSelected()) { - ret.push(true); - this.selected_count++; - } - else { - ret.push(false); - this.unselected_count++; - } - } - - if(this.selected_count === 0 && this.unselected_count !== 0) { - this.selectAll(); - return this.selected2BooleanArray(array); - } - - return ret; - }; - - - // ---------------------------------------------------------------------------------------------------------------- - // global selection sync - - NETDATA.globalSelectionSync = { - state: null, - dont_sync_before: 0, - last_t: 0, - slaves: [], - - stop: function() { - if(this.state !== null) - this.state.globalSelectionSyncStop(); - }, - - delay: function() { - if(this.state !== null) { - this.state.globalSelectionSyncDelay(); - } - } - }; - - // ---------------------------------------------------------------------------------------------------------------- - // Our state object, where all per-chart values are stored - - chartState = function(element) { - var self = $(element); - this.element = element; - - // IMPORTANT: - // all private functions should use 'that', instead of 'this' - var that = this; - - /* error() - private - * show an error instead of the chart - */ - var error = function(msg) { - var ret = true; - - if(typeof netdataErrorCallback === 'function') { - ret = netdataErrorCallback('chart', that.id, msg); - } - - if(ret) { - that.element.innerHTML = that.id + ': ' + msg; - that.enabled = false; - that.current = that.pan; - } - }; - - // GUID - a unique identifier for the chart - this.uuid = NETDATA.guid(); - - // string - the name of chart - this.id = self.data('netdata'); - - // string - the key for localStorage settings - this.settings_id = self.data('id') || null; - - // the user given dimensions of the element - this.width = self.data('width') || NETDATA.chartDefaults.width; - this.height = self.data('height') || NETDATA.chartDefaults.height; - - if(this.settings_id !== null) { - this.height = NETDATA.localStorageGet('chart_heights.' + this.settings_id, this.height, function(height) { - // this is the callback that will be called - // if and when the user resets all localStorage variables - // to their defaults - - resizeChartToHeight(height); - }); - } - - // string - the netdata server URL, without any path - this.host = self.data('host') || NETDATA.chartDefaults.host; - - // make sure the host does not end with / - // all netdata API requests use absolute paths - while(this.host.slice(-1) === '/') - this.host = this.host.substring(0, this.host.length - 1); - - // string - the grouping method requested by the user - this.method = self.data('method') || NETDATA.chartDefaults.method; - - // the time-range requested by the user - this.after = self.data('after') || NETDATA.chartDefaults.after; - this.before = self.data('before') || NETDATA.chartDefaults.before; - - // the pixels per point requested by the user - this.pixels_per_point = self.data('pixels-per-point') || 1; - this.points = self.data('points') || null; - - // the dimensions requested by the user - this.dimensions = self.data('dimensions') || null; - - // the chart library requested by the user - this.library_name = self.data('chart-library') || NETDATA.chartDefaults.library; - - // object - the chart library used - this.library = null; - - // color management - this.colors = null; - this.colors_assigned = {}; - this.colors_available = null; - - // the element already created by the user - this.element_message = null; - - // the element with the chart - this.element_chart = null; - - // the element with the legend of the chart (if created by us) - this.element_legend = null; - this.element_legend_childs = { - hidden: null, - title_date: null, - title_time: null, - title_units: null, - nano: null, - nano_options: null, - series: null - }; - - this.chart_url = null; // string - the url to download chart info - this.chart = null; // object - the chart as downloaded from the server - - this.title = self.data('title') || null; // the title of the chart - this.units = self.data('units') || null; // the units of the chart dimensions - this.append_options = self.data('append-options') || null; // the units of the chart dimensions - - this.running = false; // boolean - true when the chart is being refreshed now - this.validated = false; // boolean - has the chart been validated? - this.enabled = true; // boolean - is the chart enabled for refresh? - this.paused = false; // boolean - is the chart paused for any reason? - this.selected = false; // boolean - is the chart shown a selection? - this.debug = false; // boolean - console.log() debug info about this chart - - this.netdata_first = 0; // milliseconds - the first timestamp in netdata - this.netdata_last = 0; // milliseconds - the last timestamp in netdata - this.requested_after = null; // milliseconds - the timestamp of the request after param - this.requested_before = null; // milliseconds - the timestamp of the request before param - this.requested_padding = null; - this.view_after = 0; - this.view_before = 0; - - this.auto = { - name: 'auto', - autorefresh: true, - force_update_at: 0, // the timestamp to force the update at - force_before_ms: null, - force_after_ms: null - }; - this.pan = { - name: 'pan', - autorefresh: false, - force_update_at: 0, // the timestamp to force the update at - force_before_ms: null, - force_after_ms: null - }; - this.zoom = { - name: 'zoom', - autorefresh: false, - force_update_at: 0, // the timestamp to force the update at - force_before_ms: null, - force_after_ms: null - }; - - // this is a pointer to one of the sub-classes below - // auto, pan, zoom - this.current = this.auto; - - // check the requested library is available - // we don't initialize it here - it will be initialized when - // this chart will be first used - if(typeof NETDATA.chartLibraries[that.library_name] === 'undefined') { - NETDATA.error(402, that.library_name); - error('chart library "' + that.library_name + '" is not found'); - return; - } - else if(NETDATA.chartLibraries[that.library_name].enabled === false) { - NETDATA.error(403, that.library_name); - error('chart library "' + that.library_name + '" is not enabled'); - return; - } - else - that.library = NETDATA.chartLibraries[that.library_name]; - - // milliseconds - the time the last refresh took - this.refresh_dt_ms = 0; - - // if we need to report the rendering speed - // find the element that needs to be updated - var refresh_dt_element_name = self.data('dt-element-name') || null; // string - the element to print refresh_dt_ms - - if(refresh_dt_element_name !== null) - this.refresh_dt_element = document.getElementById(refresh_dt_element_name) || null; - else - this.refresh_dt_element = null; - - this.dimensions_visibility = new dimensionsVisibility(this); - - this._updating = false; - - // ============================================================================================================ - // PRIVATE FUNCTIONS - - var createDOM = function() { - if(that.enabled === false) return; - - if(that.element_message !== null) that.element_message.innerHTML = ''; - if(that.element_legend !== null) that.element_legend.innerHTML = ''; - if(that.element_chart !== null) that.element_chart.innerHTML = ''; - - that.element.innerHTML = ''; - - that.element_message = document.createElement('div'); - that.element_message.className = ' netdata-message hidden'; - that.element.appendChild(that.element_message); - - that.element_chart = document.createElement('div'); - that.element_chart.id = that.library_name + '-' + that.uuid + '-chart'; - that.element.appendChild(that.element_chart); - - if(that.hasLegend() === true) { - that.element.className = "netdata-container-with-legend"; - that.element_chart.className = 'netdata-chart-with-legend-right netdata-' + that.library_name + '-chart-with-legend-right'; - - that.element_legend = document.createElement('div'); - that.element_legend.className = 'netdata-chart-legend netdata-' + that.library_name + '-legend'; - that.element.appendChild(that.element_legend); - } - else { - that.element.className = "netdata-container"; - that.element_chart.className = ' netdata-chart netdata-' + that.library_name + '-chart'; - - that.element_legend = null; - } - that.element_legend_childs.series = null; - - if(typeof(that.width) === 'string') - $(that.element).css('width', that.width); - else if(typeof(that.width) === 'number') - $(that.element).css('width', that.width + 'px'); - - if(typeof(that.library.aspect_ratio) === 'undefined') { - if(typeof(that.height) === 'string') - $(that.element).css('height', that.height); - else if(typeof(that.height) === 'number') - $(that.element).css('height', that.height + 'px'); - } - else { - var w = that.element.offsetWidth; - if(w === null || w === 0) { - // the div is hidden - // this will resize the chart when next viewed - that.tm.last_resized = 0; - } - else - $(that.element).css('height', (that.element.offsetWidth * that.library.aspect_ratio / 100).toString() + 'px'); - } - - if(NETDATA.chartDefaults.min_width !== null) - $(that.element).css('min-width', NETDATA.chartDefaults.min_width); - - that.tm.last_dom_created = new Date().getTime(); - - showLoading(); - }; - - /* init() private - * initialize state variables - * destroy all (possibly) created state elements - * create the basic DOM for a chart - */ - var init = function() { - if(that.enabled === false) return; - - that.paused = false; - that.selected = false; - - that.chart_created = false; // boolean - is the library.create() been called? - that.updates_counter = 0; // numeric - the number of refreshes made so far - that.updates_since_last_unhide = 0; // numeric - the number of refreshes made since the last time the chart was unhidden - that.updates_since_last_creation = 0; // numeric - the number of refreshes made since the last time the chart was created - - that.tm = { - last_initialized: 0, // milliseconds - the timestamp it was last initialized - last_dom_created: 0, // milliseconds - the timestamp its DOM was last created - last_mode_switch: 0, // milliseconds - the timestamp it switched modes - - last_info_downloaded: 0, // milliseconds - the timestamp we downloaded the chart - last_updated: 0, // the timestamp the chart last updated with data - pan_and_zoom_seq: 0, // the sequence number of the global synchronization - // between chart. - // Used with NETDATA.globalPanAndZoom.seq - last_visible_check: 0, // the time we last checked if it is visible - last_resized: 0, // the time the chart was resized - last_hidden: 0, // the time the chart was hidden - last_unhidden: 0, // the time the chart was unhidden - last_autorefreshed: 0 // the time the chart was last refreshed - }; - - that.data = null; // the last data as downloaded from the netdata server - that.data_url = 'invalid://'; // string - the last url used to update the chart - that.data_points = 0; // number - the number of points returned from netdata - that.data_after = 0; // milliseconds - the first timestamp of the data - that.data_before = 0; // milliseconds - the last timestamp of the data - that.data_update_every = 0; // milliseconds - the frequency to update the data - - that.tm.last_initialized = new Date().getTime(); - createDOM(); - - that.setMode('auto'); - }; - - var maxMessageFontSize = function() { - // normally we want a font size, as tall as the element - var h = that.element_message.clientHeight; - - // but give it some air, 20% let's say, or 5 pixels min - var lost = Math.max(h * 0.2, 5); - h -= lost; - - // center the text, vertically - var paddingTop = (lost - 5) / 2; - - // but check the width too - // it should fit 10 characters in it - var w = that.element_message.clientWidth / 10; - if(h > w) { - paddingTop += (h - w) / 2; - h = w; - } - - // and don't make it too huge - // 5% of the screen size is good - if(h > screen.height / 20) { - paddingTop += (h - (screen.height / 20)) / 2; - h = screen.height / 20; - } - - // set it - that.element_message.style.fontSize = h.toString() + 'px'; - that.element_message.style.paddingTop = paddingTop.toString() + 'px'; - }; - - var showMessage = function(msg) { - that.element_message.className = 'netdata-message'; - that.element_message.innerHTML = msg; - that.element_message.style.fontSize = 'x-small'; - that.element_message.style.paddingTop = '0px'; - that.___messageHidden___ = undefined; - }; - - var showMessageIcon = function(icon) { - that.element_message.innerHTML = icon; - that.element_message.className = 'netdata-message icon'; - maxMessageFontSize(); - that.___messageHidden___ = undefined; - }; - - var hideMessage = function() { - if(typeof that.___messageHidden___ === 'undefined') { - that.___messageHidden___ = true; - that.element_message.className = 'netdata-message hidden'; - } - }; - - var showRendering = function() { - var icon; - if(that.chart !== null) { - if(that.chart.chart_type === 'line') - icon = '<i class="fa fa-line-chart"></i>'; - else - icon = '<i class="fa fa-area-chart"></i>'; - } - else - icon = '<i class="fa fa-area-chart"></i>'; - - showMessageIcon(icon + ' netdata'); - }; - - var showLoading = function() { - if(that.chart_created === false) { - showMessageIcon('<i class="fa fa-refresh"></i> netdata'); - return true; - } - return false; - }; - - var isHidden = function() { - if(typeof that.___chartIsHidden___ !== 'undefined') - return true; - - return false; - }; - - // hide the chart, when it is not visible - called from isVisible() - var hideChart = function() { - // hide it, if it is not already hidden - if(isHidden() === true) return; - - if(that.chart_created === true) { - if(NETDATA.options.current.destroy_on_hide === true) { - // we should destroy it - init(); - } - else { - showRendering(); - that.element_chart.style.display = 'none'; - if(that.element_legend !== null) that.element_legend.style.display = 'none'; - that.tm.last_hidden = new Date().getTime(); - - // de-allocate data - // This works, but I not sure there are no corner cases somewhere - // so it is commented - if the user has memory issues he can - // set Destroy on Hide for all charts - // that.data = null; - } - } - - that.___chartIsHidden___ = true; - }; - - // unhide the chart, when it is visible - called from isVisible() - var unhideChart = function() { - if(isHidden() === false) return; - - that.___chartIsHidden___ = undefined; - that.updates_since_last_unhide = 0; - - if(that.chart_created === false) { - // we need to re-initialize it, to show our background - // logo in bootstrap tabs, until the chart loads - init(); - } - else { - that.tm.last_unhidden = new Date().getTime(); - that.element_chart.style.display = ''; - if(that.element_legend !== null) that.element_legend.style.display = ''; - resizeChart(); - hideMessage(); - } - }; - - var canBeRendered = function() { - if(isHidden() === true || that.isVisible(true) === false) - return false; - - return true; - }; - - // https://github.com/petkaantonov/bluebird/wiki/Optimization-killers - var callChartLibraryUpdateSafely = function(data) { - var status; - - if(canBeRendered() === false) - return false; - - if(NETDATA.options.debug.chart_errors === true) - status = that.library.update(that, data); - else { - try { - status = that.library.update(that, data); - } - catch(err) { - status = false; - } - } - - if(status === false) { - error('chart failed to be updated as ' + that.library_name); - return false; - } - - return true; - }; - - // https://github.com/petkaantonov/bluebird/wiki/Optimization-killers - var callChartLibraryCreateSafely = function(data) { - var status; - - if(canBeRendered() === false) - return false; - - if(NETDATA.options.debug.chart_errors === true) - status = that.library.create(that, data); - else { - try { - status = that.library.create(that, data); - } - catch(err) { - status = false; - } - } - - if(status === false) { - error('chart failed to be created as ' + that.library_name); - return false; - } - - that.chart_created = true; - that.updates_since_last_creation = 0; - return true; - }; - - // ---------------------------------------------------------------------------------------------------------------- - // Chart Resize - - // resizeChart() - private - // to be called just before the chart library to make sure that - // a properly sized dom is available - var resizeChart = function() { - if(that.isVisible() === true && that.tm.last_resized < NETDATA.options.last_resized) { - if(that.chart_created === false) return; - - if(that.needsRecreation()) { - init(); - } - else if(typeof that.library.resize === 'function') { - that.library.resize(that); - - if(that.element_legend_childs.nano !== null && that.element_legend_childs.nano_options !== null) - $(that.element_legend_childs.nano).nanoScroller(); - - maxMessageFontSize(); - } - - that.tm.last_resized = new Date().getTime(); - } - }; - - // this is the actual chart resize algorithm - // it will: - // - resize the entire container - // - update the internal states - // - resize the chart as the div changes height - // - update the scrollbar of the legend - var resizeChartToHeight = function(h) { - // console.log(h); - that.element.style.height = h; - - if(that.settings_id !== null) - NETDATA.localStorageSet('chart_heights.' + that.settings_id, h); - - var now = new Date().getTime(); - NETDATA.options.last_page_scroll = now; - NETDATA.options.auto_refresher_stop_until = now + NETDATA.options.current.stop_updates_while_resizing; - - // force a resize - that.tm.last_resized = 0; - resizeChart(); - }; - - this.resizeHandler = function(e) { - e.preventDefault(); - - if(typeof this.event_resize === 'undefined' - || this.event_resize.chart_original_w === 'undefined' - || this.event_resize.chart_original_h === 'undefined') - this.event_resize = { - chart_original_w: this.element.clientWidth, - chart_original_h: this.element.clientHeight, - last: 0 - }; - - if(e.type === 'touchstart') { - this.event_resize.mouse_start_x = e.touches.item(0).pageX; - this.event_resize.mouse_start_y = e.touches.item(0).pageY; - } - else { - this.event_resize.mouse_start_x = e.clientX; - this.event_resize.mouse_start_y = e.clientY; - } - - this.event_resize.chart_start_w = this.element.clientWidth; - this.event_resize.chart_start_h = this.element.clientHeight; - this.event_resize.chart_last_w = this.element.clientWidth; - this.event_resize.chart_last_h = this.element.clientHeight; - - var now = new Date().getTime(); - if(now - this.event_resize.last <= NETDATA.options.current.double_click_speed) { - // double click / double tap event - - // the optimal height of the chart - // showing the entire legend - var optimal = this.event_resize.chart_last_h - + this.element_legend_childs.content.scrollHeight - - this.element_legend_childs.content.clientHeight; - - // if we are not optimal, be optimal - if(this.event_resize.chart_last_h != optimal) - resizeChartToHeight(optimal.toString() + 'px'); - - // else if we do not have the original height - // reset to the original height - else if(this.event_resize.chart_last_h != this.event_resize.chart_original_h) - resizeChartToHeight(this.event_resize.chart_original_h.toString() + 'px'); - } - else { - this.event_resize.last = now; - - // process movement event - document.onmousemove = - document.ontouchmove = - this.element_legend_childs.resize_handler.onmousemove = - this.element_legend_childs.resize_handler.ontouchmove = - function(e) { - var y = null; - - switch(e.type) { - case 'mousemove': y = e.clientY; break; - case 'touchmove': y = e.touches.item(e.touches - 1).pageY; break; - } - - if(y !== null) { - var newH = that.event_resize.chart_start_h + y - that.event_resize.mouse_start_y; - - if(newH >= 70 && newH !== that.event_resize.chart_last_h) { - resizeChartToHeight(newH.toString() + 'px'); - that.event_resize.chart_last_h = newH; - } - } - }; - - // process end event - document.onmouseup = - document.ontouchend = - this.element_legend_childs.resize_handler.onmouseup = - this.element_legend_childs.resize_handler.ontouchend = - function(e) { - // remove all the hooks - document.onmouseup = - document.onmousemove = - document.ontouchmove = - document.ontouchend = - that.element_legend_childs.resize_handler.onmousemove = - that.element_legend_childs.resize_handler.ontouchmove = - that.element_legend_childs.resize_handler.onmouseout = - that.element_legend_childs.resize_handler.onmouseup = - that.element_legend_childs.resize_handler.ontouchend = - null; - - // allow auto-refreshes - NETDATA.options.auto_refresher_stop_until = 0; - }; - } - }; - - - var noDataToShow = function() { - showMessageIcon('<i class="fa fa-warning"></i> empty'); - that.legendUpdateDOM(); - that.tm.last_autorefreshed = new Date().getTime(); - // that.data_update_every = 30 * 1000; - //that.element_chart.style.display = 'none'; - //if(that.element_legend !== null) that.element_legend.style.display = 'none'; - //that.___chartIsHidden___ = true; - }; - - // ============================================================================================================ - // PUBLIC FUNCTIONS - - this.error = function(msg) { - error(msg); - }; - - this.setMode = function(m) { - if(this.current !== null && this.current.name === m) return; - - if(m === 'auto') - this.current = this.auto; - else if(m === 'pan') - this.current = this.pan; - else if(m === 'zoom') - this.current = this.zoom; - else - this.current = this.auto; - - this.current.force_update_at = 0; - this.current.force_before_ms = null; - this.current.force_after_ms = null; - - this.tm.last_mode_switch = new Date().getTime(); - }; - - // ---------------------------------------------------------------------------------------------------------------- - // global selection sync - - // prevent to global selection sync for some time - this.globalSelectionSyncDelay = function(ms) { - if(NETDATA.options.current.sync_selection === false) - return; - - if(typeof ms === 'number') - NETDATA.globalSelectionSync.dont_sync_before = new Date().getTime() + ms; - else - NETDATA.globalSelectionSync.dont_sync_before = new Date().getTime() + NETDATA.options.current.sync_selection_delay; - }; - - // can we globally apply selection sync? - this.globalSelectionSyncAbility = function() { - if(NETDATA.options.current.sync_selection === false) - return false; - - if(NETDATA.globalSelectionSync.dont_sync_before > new Date().getTime()) - return false; - - return true; - }; - - this.globalSelectionSyncIsMaster = function() { - if(NETDATA.globalSelectionSync.state === this) - return true; - else - return false; - }; - - // this chart is the master of the global selection sync - this.globalSelectionSyncBeMaster = function() { - // am I the master? - if(this.globalSelectionSyncIsMaster()) { - if(this.debug === true) - this.log('sync: I am the master already.'); - - return; - } - - if(NETDATA.globalSelectionSync.state) { - if(this.debug === true) - this.log('sync: I am not the sync master. Resetting global sync.'); - - this.globalSelectionSyncStop(); - } - - // become the master - if(this.debug === true) - this.log('sync: becoming sync master.'); - - this.selected = true; - NETDATA.globalSelectionSync.state = this; - - // find the all slaves - var targets = NETDATA.options.targets; - var len = targets.length; - while(len--) { - st = targets[len]; - - if(st === this) { - if(this.debug === true) - st.log('sync: not adding me to sync'); - } - else if(st.globalSelectionSyncIsEligible()) { - if(this.debug === true) - st.log('sync: adding to sync as slave'); - - st.globalSelectionSyncBeSlave(); - } - } - - // this.globalSelectionSyncDelay(100); - }; - - // can the chart participate to the global selection sync as a slave? - this.globalSelectionSyncIsEligible = function() { - if(this.enabled === true - && this.library !== null - && typeof this.library.setSelection === 'function' - && this.isVisible() === true - && this.chart_created === true) - return true; - - return false; - }; - - // this chart becomes a slave of the global selection sync - this.globalSelectionSyncBeSlave = function() { - if(NETDATA.globalSelectionSync.state !== this) - NETDATA.globalSelectionSync.slaves.push(this); - }; - - // sync all the visible charts to the given time - // this is to be called from the chart libraries - this.globalSelectionSync = function(t) { - if(this.globalSelectionSyncAbility() === false) { - if(this.debug === true) - this.log('sync: cannot sync (yet?).'); - - return; - } - - if(this.globalSelectionSyncIsMaster() === false) { - if(this.debug === true) - this.log('sync: trying to be sync master.'); - - this.globalSelectionSyncBeMaster(); - - if(this.globalSelectionSyncAbility() === false) { - if(this.debug === true) - this.log('sync: cannot sync (yet?).'); - - return; - } - } - - NETDATA.globalSelectionSync.last_t = t; - $.each(NETDATA.globalSelectionSync.slaves, function(i, st) { - st.setSelection(t); - }); - }; - - // stop syncing all charts to the given time - this.globalSelectionSyncStop = function() { - if(NETDATA.globalSelectionSync.slaves.length) { - if(this.debug === true) - this.log('sync: cleaning up...'); - - $.each(NETDATA.globalSelectionSync.slaves, function(i, st) { - if(st === that) { - if(that.debug === true) - st.log('sync: not adding me to sync stop'); - } - else { - if(that.debug === true) - st.log('sync: removed slave from sync'); - - st.clearSelection(); - } - }); - - NETDATA.globalSelectionSync.last_t = 0; - NETDATA.globalSelectionSync.slaves = []; - NETDATA.globalSelectionSync.state = null; - } - - this.clearSelection(); - }; - - this.setSelection = function(t) { - if(typeof this.library.setSelection === 'function') { - if(this.library.setSelection(this, t) === true) - this.selected = true; - else - this.selected = false; - } - else this.selected = true; - - if(this.selected === true && this.debug === true) - this.log('selection set to ' + t.toString()); - - return this.selected; - }; - - this.clearSelection = function() { - if(this.selected === true) { - if(typeof this.library.clearSelection === 'function') { - if(this.library.clearSelection(this) === true) - this.selected = false; - else - this.selected = true; - } - else this.selected = false; - - if(this.selected === false && this.debug === true) - this.log('selection cleared'); - - this.legendReset(); - } - - return this.selected; - }; - - // find if a timestamp (ms) is shown in the current chart - this.timeIsVisible = function(t) { - if(t >= this.data_after && t <= this.data_before) - return true; - return false; - }; - - this.calculateRowForTime = function(t) { - if(this.timeIsVisible(t) === false) return -1; - return Math.floor((t - this.data_after) / this.data_update_every); - }; - - // ---------------------------------------------------------------------------------------------------------------- - - // console logging - this.log = function(msg) { - console.log(this.id + ' (' + this.library_name + ' ' + this.uuid + '): ' + msg); - }; - - this.pauseChart = function() { - if(this.paused === false) { - if(this.debug === true) - this.log('pauseChart()'); - - this.paused = true; - } - }; - - this.unpauseChart = function() { - if(this.paused === true) { - if(this.debug === true) - this.log('unpauseChart()'); - - this.paused = false; - } - }; - - this.resetChart = function(dont_clear_master, dont_update) { - if(this.debug === true) - this.log('resetChart(' + dont_clear_master + ', ' + dont_update + ') called'); - - if(typeof dont_clear_master === 'undefined') - dont_clear_master = false; - - if(typeof dont_update === 'undefined') - dont_update = false; - - if(dont_clear_master !== true && NETDATA.globalPanAndZoom.isMaster(this) === true) { - if(this.debug === true) - this.log('resetChart() diverting to clearMaster().'); - // this will call us back with master === true - NETDATA.globalPanAndZoom.clearMaster(); - return; - } - - this.clearSelection(); - - this.tm.pan_and_zoom_seq = 0; - - this.setMode('auto'); - this.current.force_update_at = 0; - this.current.force_before_ms = null; - this.current.force_after_ms = null; - this.tm.last_autorefreshed = 0; - this.paused = false; - this.selected = false; - this.enabled = true; - // this.debug = false; - - // do not update the chart here - // or the chart will flip-flop when it is the master - // of a selection sync and another chart becomes - // the new master - - if(dont_update !== true && this.isVisible() === true) { - this.updateChart(); - } - }; - - this.updateChartPanOrZoom = function(after, before) { - var logme = 'updateChartPanOrZoom(' + after + ', ' + before + '): '; - var ret = true; - - if(this.debug === true) - this.log(logme); - - if(before < after) { - if(this.debug === true) - this.log(logme + 'flipped parameters, rejecting it.'); - - return false; - } - - if(typeof this.fixed_min_duration === 'undefined') - this.fixed_min_duration = Math.round((this.chartWidth() / 30) * this.chart.update_every * 1000); - - var min_duration = this.fixed_min_duration; - var current_duration = Math.round(this.view_before - this.view_after); - - // round the numbers - after = Math.round(after); - before = Math.round(before); - - // align them to update_every - // stretching them further away - after -= after % this.data_update_every; - before += this.data_update_every - (before % this.data_update_every); - - // the final wanted duration - var wanted_duration = before - after; - - // to allow panning, accept just a point below our minimum - if((current_duration - this.data_update_every) < min_duration) - min_duration = current_duration - this.data_update_every; - - // we do it, but we adjust to minimum size and return false - // when the wanted size is below the current and the minimum - // and we zoom - if(wanted_duration < current_duration && wanted_duration < min_duration) { - if(this.debug === true) - this.log(logme + 'too small: min_duration: ' + (min_duration / 1000).toString() + ', wanted: ' + (wanted_duration / 1000).toString()); - - min_duration = this.fixed_min_duration; - - var dt = (min_duration - wanted_duration) / 2; - before += dt; - after -= dt; - wanted_duration = before - after; - ret = false; - } - - var tolerance = this.data_update_every * 2; - var movement = Math.abs(before - this.view_before); - - if(Math.abs(current_duration - wanted_duration) <= tolerance && movement <= tolerance && ret === true) { - if(this.debug === true) - this.log(logme + 'REJECTING UPDATE: current/min duration: ' + (current_duration / 1000).toString() + '/' + (this.fixed_min_duration / 1000).toString() + ', wanted duration: ' + (wanted_duration / 1000).toString() + ', duration diff: ' + (Math.round(Math.abs(current_duration - wanted_duration) / 1000)).toString() + ', movement: ' + (movement / 1000).toString() + ', tolerance: ' + (tolerance / 1000).toString() + ', returning: ' + false); - return false; - } - - if(this.current.name === 'auto') { - this.log(logme + 'caller called me with mode: ' + this.current.name); - this.setMode('pan'); - } - - if(this.debug === true) - this.log(logme + 'ACCEPTING UPDATE: current/min duration: ' + (current_duration / 1000).toString() + '/' + (this.fixed_min_duration / 1000).toString() + ', wanted duration: ' + (wanted_duration / 1000).toString() + ', duration diff: ' + (Math.round(Math.abs(current_duration - wanted_duration) / 1000)).toString() + ', movement: ' + (movement / 1000).toString() + ', tolerance: ' + (tolerance / 1000).toString() + ', returning: ' + ret); - - this.current.force_update_at = new Date().getTime() + NETDATA.options.current.pan_and_zoom_delay; - this.current.force_after_ms = after; - this.current.force_before_ms = before; - NETDATA.globalPanAndZoom.setMaster(this, after, before); - return ret; - }; - - this.legendFormatValue = function(value) { - if(value === null || value === 'undefined') return '-'; - if(typeof value !== 'number') return value; - - var abs = Math.abs(value); - if(abs >= 1000) return (Math.round(value)).toLocaleString(); - if(abs >= 100 ) return (Math.round(value * 10) / 10).toLocaleString(); - if(abs >= 1 ) return (Math.round(value * 100) / 100).toLocaleString(); - if(abs >= 0.1 ) return (Math.round(value * 1000) / 1000).toLocaleString(); - return (Math.round(value * 10000) / 10000).toLocaleString(); - }; - - this.legendSetLabelValue = function(label, value) { - var series = this.element_legend_childs.series[label]; - if(typeof series === 'undefined') return; - if(series.value === null && series.user === null) return; - - // if the value has not changed, skip DOM update - //if(series.last === value) return; - - var s, r; - if(typeof value === 'number') { - var v = Math.abs(value); - s = r = this.legendFormatValue(value); - - if(typeof series.last === 'number') { - if(v > series.last) s += '<i class="fa fa-angle-up" style="width: 8px; text-align: center; overflow: hidden; vertical-align: middle;"></i>'; - else if(v < series.last) s += '<i class="fa fa-angle-down" style="width: 8px; text-align: center; overflow: hidden; vertical-align: middle;"></i>'; - else s += '<i class="fa fa-angle-left" style="width: 8px; text-align: center; overflow: hidden; vertical-align: middle;"></i>'; - } - else s += '<i class="fa fa-angle-right" style="width: 8px; text-align: center; overflow: hidden; vertical-align: middle;"></i>'; - series.last = v; - } - else { - s = r = value; - series.last = value; - } - - if(series.value !== null) series.value.innerHTML = s; - if(series.user !== null) series.user.innerHTML = r; - }; - - this.legendSetDate = function(ms) { - if(typeof ms !== 'number') { - this.legendShowUndefined(); - return; - } - - var d = new Date(ms); - - if(this.element_legend_childs.title_date) - this.element_legend_childs.title_date.innerHTML = d.toLocaleDateString(); - - if(this.element_legend_childs.title_time) - this.element_legend_childs.title_time.innerHTML = d.toLocaleTimeString(); - - if(this.element_legend_childs.title_units) - this.element_legend_childs.title_units.innerHTML = this.units; - }; - - this.legendShowUndefined = function() { - if(this.element_legend_childs.title_date) - this.element_legend_childs.title_date.innerHTML = ' '; - - if(this.element_legend_childs.title_time) - this.element_legend_childs.title_time.innerHTML = this.chart.name; - - if(this.element_legend_childs.title_units) - this.element_legend_childs.title_units.innerHTML = ' '; - - if(this.data && this.element_legend_childs.series !== null) { - var labels = this.data.dimension_names; - var i = labels.length; - while(i--) { - var label = labels[i]; - - if(typeof label === 'undefined') continue; - if(typeof this.element_legend_childs.series[label] === 'undefined') continue; - this.legendSetLabelValue(label, null); - } - } - }; - - this.legendShowLatestValues = function() { - if(this.chart === null) return; - if(this.selected) return; - - if(this.data === null || this.element_legend_childs.series === null) { - this.legendShowUndefined(); - return; - } - - var show_undefined = true; - if(Math.abs(this.netdata_last - this.view_before) <= this.data_update_every) - show_undefined = false; - - if(show_undefined) { - this.legendShowUndefined(); - return; - } - - this.legendSetDate(this.view_before); - - var labels = this.data.dimension_names; - var i = labels.length; - while(i--) { - var label = labels[i]; - - if(typeof label === 'undefined') continue; - if(typeof this.element_legend_childs.series[label] === 'undefined') continue; - - if(show_undefined) - this.legendSetLabelValue(label, null); - else - this.legendSetLabelValue(label, this.data.view_latest_values[i]); - } - }; - - this.legendReset = function() { - this.legendShowLatestValues(); - }; - - // this should be called just ONCE per dimension per chart - this._chartDimensionColor = function(label) { - if(this.colors === null) this.chartColors(); - - if(typeof this.colors_assigned[label] === 'undefined') { - if(this.colors_available.length === 0) { - for(var i = 0, len = NETDATA.themes.current.colors.length; i < len ; i++) - this.colors_available.push(NETDATA.themes.current.colors[i]); - } - - this.colors_assigned[label] = this.colors_available.shift(); - - if(this.debug === true) - this.log('label "' + label + '" got color "' + this.colors_assigned[label]); - } - else { - if(this.debug === true) - this.log('label "' + label + '" already has color "' + this.colors_assigned[label] + '"'); - } - - this.colors.push(this.colors_assigned[label]); - return this.colors_assigned[label]; - }; - - this.chartColors = function() { - if(this.colors !== null) return this.colors; - - this.colors = new Array(); - this.colors_available = new Array(); - var i, len; - - var c = $(this.element).data('colors'); - // this.log('read colors: ' + c); - if(typeof c !== 'undefined' && c !== null && c.length > 0) { - if(typeof c !== 'string') { - this.log('invalid color given: ' + c + ' (give a space separated list of colors)'); - } - else { - c = c.split(' '); - var added = 0; - - while(added < 20) { - for(i = 0, len = c.length; i < len ; i++) { - added++; - this.colors_available.push(c[i]); - // this.log('adding color: ' + c[i]); - } - } - } - } - - // push all the standard colors too - for(i = 0, len = NETDATA.themes.current.colors.length; i < len ; i++) - this.colors_available.push(NETDATA.themes.current.colors[i]); - - return this.colors; - }; - - this.legendUpdateDOM = function() { - var needed = false; - - // check that the legend DOM is up to date for the downloaded dimensions - if(typeof this.element_legend_childs.series !== 'object' || this.element_legend_childs.series === null) { - // this.log('the legend does not have any series - requesting legend update'); - needed = true; - } - else if(this.data === null) { - // this.log('the chart does not have any data - requesting legend update'); - needed = true; - } - else if(typeof this.element_legend_childs.series.labels_key === 'undefined') { - needed = true; - } - else { - var labels = this.data.dimension_names.toString(); - if(labels !== this.element_legend_childs.series.labels_key) { - needed = true; - - if(this.debug === true) - this.log('NEW LABELS: "' + labels + '" NOT EQUAL OLD LABELS: "' + this.element_legend_childs.series.labels_key + '"'); - } - } - - if(needed === false) { - // make sure colors available - this.chartColors(); - - // do we have to update the current values? - // we do this, only when the visible chart is current - if(Math.abs(this.netdata_last - this.view_before) <= this.data_update_every) { - if(this.debug === true) - this.log('chart is in latest position... updating values on legend...'); - - //var labels = this.data.dimension_names; - //var i = labels.length; - //while(i--) - // this.legendSetLabelValue(labels[i], this.data.latest_values[i]); - } - return; - } - if(this.colors === null) { - // this is the first time we update the chart - // let's assign colors to all dimensions - if(this.library.track_colors() === true) - for(var dim in this.chart.dimensions) - this._chartDimensionColor(this.chart.dimensions[dim].name); - } - // we will re-generate the colors for the chart - // based on the selected dimensions - this.colors = null; - - if(this.debug === true) - this.log('updating Legend DOM'); - - // mark all dimensions as invalid - this.dimensions_visibility.invalidateAll(); - - var genLabel = function(state, parent, name, count) { - var color = state._chartDimensionColor(name); - - var user_element = null; - var user_id = self.data('show-value-of-' + name + '-at') || null; - if(user_id !== null) { - user_element = document.getElementById(user_id) || null; - if(user_element === null) - state.log('Cannot find element with id: ' + user_id); - } - - state.element_legend_childs.series[name] = { - name: document.createElement('span'), - value: document.createElement('span'), - user: user_element, - last: null - }; - - var label = state.element_legend_childs.series[name]; - - // create the dimension visibility tracking for this label - state.dimensions_visibility.dimensionAdd(name, label.name, label.value, color); - - var rgb = NETDATA.colorHex2Rgb(color); - label.name.innerHTML = '<table class="netdata-legend-name-table-' - + state.chart.chart_type - + '" style="background-color: ' - + 'rgba(' + rgb.r + ',' + rgb.g + ',' + rgb.b + ',' + NETDATA.options.current['color_fill_opacity_' + state.chart.chart_type] + ')' - + '"><tr class="netdata-legend-name-tr"><td class="netdata-legend-name-td"></td></tr></table>' - - var text = document.createTextNode(' ' + name); - label.name.appendChild(text); - - if(count > 0) - parent.appendChild(document.createElement('br')); - - parent.appendChild(label.name); - parent.appendChild(label.value); - }; - - var content = document.createElement('div'); - - if(this.hasLegend()) { - this.element_legend_childs = { - content: content, - resize_handler: document.createElement('div'), - toolbox: document.createElement('div'), - toolbox_left: document.createElement('div'), - toolbox_right: document.createElement('div'), - toolbox_reset: document.createElement('div'), - toolbox_zoomin: document.createElement('div'), - toolbox_zoomout: document.createElement('div'), - toolbox_volume: document.createElement('div'), - title_date: document.createElement('span'), - title_time: document.createElement('span'), - title_units: document.createElement('span'), - nano: document.createElement('div'), - nano_options: { - paneClass: 'netdata-legend-series-pane', - sliderClass: 'netdata-legend-series-slider', - contentClass: 'netdata-legend-series-content', - enabledClass: '__enabled', - flashedClass: '__flashed', - activeClass: '__active', - tabIndex: -1, - alwaysVisible: true, - sliderMinHeight: 10 - }, - series: {} - }; - - this.element_legend.innerHTML = ''; - - if(this.library.toolboxPanAndZoom !== null) { - - function get_pan_and_zoom_step(event) { - if (event.ctrlKey) - return NETDATA.options.current.pan_and_zoom_factor * NETDATA.options.current.pan_and_zoom_factor_multiplier_control; - - else if (event.shiftKey) - return NETDATA.options.current.pan_and_zoom_factor * NETDATA.options.current.pan_and_zoom_factor_multiplier_shift; - - else if (event.altKey) - return NETDATA.options.current.pan_and_zoom_factor * NETDATA.options.current.pan_and_zoom_factor_multiplier_alt; - - else - return NETDATA.options.current.pan_and_zoom_factor; - } - - this.element_legend_childs.toolbox.className += ' netdata-legend-toolbox'; - this.element.appendChild(this.element_legend_childs.toolbox); - - this.element_legend_childs.toolbox_left.className += ' netdata-legend-toolbox-button'; - this.element_legend_childs.toolbox_left.innerHTML = '<i class="fa fa-backward"></i>'; - this.element_legend_childs.toolbox.appendChild(this.element_legend_childs.toolbox_left); - this.element_legend_childs.toolbox_left.onclick = function(e) { - e.preventDefault(); - - var step = (that.view_before - that.view_after) * get_pan_and_zoom_step(e); - var before = that.view_before - step; - var after = that.view_after - step; - if(after >= that.netdata_first) - that.library.toolboxPanAndZoom(that, after, before); - }; - if(NETDATA.options.current.show_help === true) - $(this.element_legend_childs.toolbox_left).popover({ - container: "body", - animation: false, - html: true, - trigger: 'hover', - placement: 'bottom', - delay: { show: NETDATA.options.current.show_help_delay_show_ms, hide: NETDATA.options.current.show_help_delay_hide_ms }, - title: 'Pan Left', - content: 'Pan the chart to the left. You can also <b>drag it</b> with your mouse or your finger (on touch devices).<br/><small>Help, can be disabled from the settings.</small>' - }); - - - this.element_legend_childs.toolbox_reset.className += ' netdata-legend-toolbox-button'; - this.element_legend_childs.toolbox_reset.innerHTML = '<i class="fa fa-play"></i>'; - this.element_legend_childs.toolbox.appendChild(this.element_legend_childs.toolbox_reset); - this.element_legend_childs.toolbox_reset.onclick = function(e) { - e.preventDefault(); - NETDATA.resetAllCharts(that); - }; - if(NETDATA.options.current.show_help === true) - $(this.element_legend_childs.toolbox_reset).popover({ - container: "body", - animation: false, - html: true, - trigger: 'hover', - placement: 'bottom', - delay: { show: NETDATA.options.current.show_help_delay_show_ms, hide: NETDATA.options.current.show_help_delay_hide_ms }, - title: 'Chart Reset', - content: 'Reset all the charts to their default auto-refreshing state. You can also <b>double click</b> the chart contents with your mouse or your finger (on touch devices).<br/><small>Help, can be disabled from the settings.</small>' - }); - - this.element_legend_childs.toolbox_right.className += ' netdata-legend-toolbox-button'; - this.element_legend_childs.toolbox_right.innerHTML = '<i class="fa fa-forward"></i>'; - this.element_legend_childs.toolbox.appendChild(this.element_legend_childs.toolbox_right); - this.element_legend_childs.toolbox_right.onclick = function(e) { - e.preventDefault(); - var step = (that.view_before - that.view_after) * get_pan_and_zoom_step(e); - var before = that.view_before + step; - var after = that.view_after + step; - if(before <= that.netdata_last) - that.library.toolboxPanAndZoom(that, after, before); - }; - if(NETDATA.options.current.show_help === true) - $(this.element_legend_childs.toolbox_right).popover({ - container: "body", - animation: false, - html: true, - trigger: 'hover', - placement: 'bottom', - delay: { show: NETDATA.options.current.show_help_delay_show_ms, hide: NETDATA.options.current.show_help_delay_hide_ms }, - title: 'Pan Right', - content: 'Pan the chart to the right. You can also <b>drag it</b> with your mouse or your finger (on touch devices).<br/><small>Help, can be disabled from the settings.</small>' - }); - - - this.element_legend_childs.toolbox_zoomin.className += ' netdata-legend-toolbox-button'; - this.element_legend_childs.toolbox_zoomin.innerHTML = '<i class="fa fa-plus"></i>'; - this.element_legend_childs.toolbox.appendChild(this.element_legend_childs.toolbox_zoomin); - this.element_legend_childs.toolbox_zoomin.onclick = function(e) { - e.preventDefault(); - var dt = ((that.view_before - that.view_after) * (get_pan_and_zoom_step(e) * 0.8) / 2); - var before = that.view_before - dt; - var after = that.view_after + dt; - that.library.toolboxPanAndZoom(that, after, before); - }; - if(NETDATA.options.current.show_help === true) - $(this.element_legend_childs.toolbox_zoomin).popover({ - container: "body", - animation: false, - html: true, - trigger: 'hover', - placement: 'bottom', - delay: { show: NETDATA.options.current.show_help_delay_show_ms, hide: NETDATA.options.current.show_help_delay_hide_ms }, - title: 'Chart Zoom In', - content: 'Zoom in the chart. You can also press SHIFT and select an area of the chart to zoom in. On Chrome and Opera, you can press the SHIFT or the ALT keys and then use the mouse wheel to zoom in or out.<br/><small>Help, can be disabled from the settings.</small>' - }); - - this.element_legend_childs.toolbox_zoomout.className += ' netdata-legend-toolbox-button'; - this.element_legend_childs.toolbox_zoomout.innerHTML = '<i class="fa fa-minus"></i>'; - this.element_legend_childs.toolbox.appendChild(this.element_legend_childs.toolbox_zoomout); - this.element_legend_childs.toolbox_zoomout.onclick = function(e) { - e.preventDefault(); - var dt = (((that.view_before - that.view_after) / (1.0 - (get_pan_and_zoom_step(e) * 0.8)) - (that.view_before - that.view_after)) / 2); - var before = that.view_before + dt; - var after = that.view_after - dt; - - that.library.toolboxPanAndZoom(that, after, before); - }; - if(NETDATA.options.current.show_help === true) - $(this.element_legend_childs.toolbox_zoomout).popover({ - container: "body", - animation: false, - html: true, - trigger: 'hover', - placement: 'bottom', - delay: { show: NETDATA.options.current.show_help_delay_show_ms, hide: NETDATA.options.current.show_help_delay_hide_ms }, - title: 'Chart Zoom Out', - content: 'Zoom out the chart. On Chrome and Opera, you can also press the SHIFT or the ALT keys and then use the mouse wheel to zoom in or out.<br/><small>Help, can be disabled from the settings.</small>' - }); - - //this.element_legend_childs.toolbox_volume.className += ' netdata-legend-toolbox-button'; - //this.element_legend_childs.toolbox_volume.innerHTML = '<i class="fa fa-sort-amount-desc"></i>'; - //this.element_legend_childs.toolbox_volume.title = 'Visible Volume'; - //this.element_legend_childs.toolbox.appendChild(this.element_legend_childs.toolbox_volume); - //this.element_legend_childs.toolbox_volume.onclick = function(e) { - //e.preventDefault(); - //alert('clicked toolbox_volume on ' + that.id); - //} - } - else { - this.element_legend_childs.toolbox = null; - this.element_legend_childs.toolbox_left = null; - this.element_legend_childs.toolbox_reset = null; - this.element_legend_childs.toolbox_right = null; - this.element_legend_childs.toolbox_zoomin = null; - this.element_legend_childs.toolbox_zoomout = null; - this.element_legend_childs.toolbox_volume = null; - } - - this.element_legend_childs.resize_handler.className += " netdata-legend-resize-handler"; - this.element_legend_childs.resize_handler.innerHTML = '<i class="fa fa-chevron-up"></i><i class="fa fa-chevron-down"></i>'; - this.element.appendChild(this.element_legend_childs.resize_handler); - if(NETDATA.options.current.show_help === true) - $(this.element_legend_childs.resize_handler).popover({ - container: "body", - animation: false, - html: true, - trigger: 'hover', - placement: 'bottom', - delay: { show: NETDATA.options.current.show_help_delay_show_ms, hide: NETDATA.options.current.show_help_delay_hide_ms }, - title: 'Chart Resize', - content: 'Drag this point with your mouse or your finger (on touch devices), to resize the chart vertically. You can also <b>double click it</b> or <b>double tap it</b> to reset between 2 states: the default and the one that fits all the values.<br/><small>Help, can be disabled from the settings.</small>' - }); - - // mousedown event - this.element_legend_childs.resize_handler.onmousedown = - function(e) { - that.resizeHandler(e); - }; - - // touchstart event - this.element_legend_childs.resize_handler.addEventListener('touchstart', function(e) { - that.resizeHandler(e); - }, false); - - this.element_legend_childs.title_date.className += " netdata-legend-title-date"; - this.element_legend.appendChild(this.element_legend_childs.title_date); - - this.element_legend.appendChild(document.createElement('br')); - - this.element_legend_childs.title_time.className += " netdata-legend-title-time"; - this.element_legend.appendChild(this.element_legend_childs.title_time); - - this.element_legend.appendChild(document.createElement('br')); - - this.element_legend_childs.title_units.className += " netdata-legend-title-units"; - this.element_legend.appendChild(this.element_legend_childs.title_units); - - this.element_legend.appendChild(document.createElement('br')); - - this.element_legend_childs.nano.className = 'netdata-legend-series'; - this.element_legend.appendChild(this.element_legend_childs.nano); - - content.className = 'netdata-legend-series-content'; - this.element_legend_childs.nano.appendChild(content); - - if(NETDATA.options.current.show_help === true) - $(content).popover({ - container: "body", - animation: false, - html: true, - trigger: 'hover', - placement: 'bottom', - title: 'Chart Legend', - delay: { show: NETDATA.options.current.show_help_delay_show_ms, hide: NETDATA.options.current.show_help_delay_hide_ms }, - content: 'You can click or tap on the values or the labels to select dimentions. By pressing SHIFT or CONTROL, you can enable or disable multiple dimensions.<br/><small>Help, can be disabled from the settings.</small>' - }); - } - else { - this.element_legend_childs = { - content: content, - resize_handler: null, - toolbox: null, - toolbox_left: null, - toolbox_right: null, - toolbox_reset: null, - toolbox_zoomin: null, - toolbox_zoomout: null, - toolbox_volume: null, - title_date: null, - title_time: null, - title_units: null, - nano: null, - nano_options: null, - series: {} - }; - } - - if(this.data) { - this.element_legend_childs.series.labels_key = this.data.dimension_names.toString(); - if(this.debug === true) - this.log('labels from data: "' + this.element_legend_childs.series.labels_key + '"'); - - for(var i = 0, len = this.data.dimension_names.length; i < len ;i++) { - genLabel(this, content, this.data.dimension_names[i], i); - } - } - else { - var tmp = new Array(); - for(var dim in this.chart.dimensions) { - tmp.push(this.chart.dimensions[dim].name); - genLabel(this, content, this.chart.dimensions[dim].name, i); - } - this.element_legend_childs.series.labels_key = tmp.toString(); - if(this.debug === true) - this.log('labels from chart: "' + this.element_legend_childs.series.labels_key + '"'); - } - - // create a hidden div to be used for hidding - // the original legend of the chart library - var el = document.createElement('div'); - if(this.element_legend !== null) - this.element_legend.appendChild(el); - el.style.display = 'none'; - - this.element_legend_childs.hidden = document.createElement('div'); - el.appendChild(this.element_legend_childs.hidden); - - if(this.element_legend_childs.nano !== null && this.element_legend_childs.nano_options !== null) - $(this.element_legend_childs.nano).nanoScroller(this.element_legend_childs.nano_options); - - this.legendShowLatestValues(); - }; - - this.hasLegend = function() { - if(typeof this.___hasLegendCache___ !== 'undefined') - return this.___hasLegendCache___; - - var leg = false; - if(this.library && this.library.legend(this) === 'right-side') { - var legend = $(this.element).data('legend') || 'yes'; - if(legend === 'yes') leg = true; - } - - this.___hasLegendCache___ = leg; - return leg; - }; - - this.legendWidth = function() { - return (this.hasLegend())?140:0; - }; - - this.legendHeight = function() { - return $(this.element).height(); - }; - - this.chartWidth = function() { - return $(this.element).width() - this.legendWidth(); - }; - - this.chartHeight = function() { - return $(this.element).height(); - }; - - this.chartPixelsPerPoint = function() { - // force an options provided detail - var px = this.pixels_per_point; - - if(this.library && px < this.library.pixels_per_point(this)) - px = this.library.pixels_per_point(this); - - if(px < NETDATA.options.current.pixels_per_point) - px = NETDATA.options.current.pixels_per_point; - - return px; - }; - - this.needsRecreation = function() { - return ( - this.chart_created === true - && this.library - && this.library.autoresize() === false - && this.tm.last_resized < NETDATA.options.last_resized - ); - }; - - this.chartURL = function() { - var after, before, points_multiplier = 1; - if(NETDATA.globalPanAndZoom.isActive() && NETDATA.globalPanAndZoom.isMaster(this) === false) { - this.tm.pan_and_zoom_seq = NETDATA.globalPanAndZoom.seq; - - after = Math.round(NETDATA.globalPanAndZoom.force_after_ms / 1000); - before = Math.round(NETDATA.globalPanAndZoom.force_before_ms / 1000); - this.view_after = after * 1000; - this.view_before = before * 1000; - - this.requested_padding = null; - points_multiplier = 1; - } - else if(this.current.force_before_ms !== null && this.current.force_after_ms !== null) { - this.tm.pan_and_zoom_seq = 0; - - before = Math.round(this.current.force_before_ms / 1000); - after = Math.round(this.current.force_after_ms / 1000); - this.view_after = after * 1000; - this.view_before = before * 1000; - - if(NETDATA.options.current.pan_and_zoom_data_padding === true) { - this.requested_padding = Math.round((before - after) / 2); - after -= this.requested_padding; - before += this.requested_padding; - this.requested_padding *= 1000; - points_multiplier = 2; - } - - this.current.force_before_ms = null; - this.current.force_after_ms = null; - } - else { - this.tm.pan_and_zoom_seq = 0; - - before = this.before; - after = this.after; - this.view_after = after * 1000; - this.view_before = before * 1000; - - this.requested_padding = null; - points_multiplier = 1; - } - - this.requested_after = after * 1000; - this.requested_before = before * 1000; - - this.data_points = this.points || Math.round(this.chartWidth() / this.chartPixelsPerPoint()); - - // build the data URL - this.data_url = this.host + this.chart.data_url; - this.data_url += "&format=" + this.library.format(); - this.data_url += "&points=" + (this.data_points * points_multiplier).toString(); - this.data_url += "&group=" + this.method; - this.data_url += "&options=" + this.library.options(this); - this.data_url += '|jsonwrap'; - - if(NETDATA.options.current.eliminate_zero_dimensions === true) - this.data_url += '|nonzero'; - - if(this.append_options !== null) - this.data_url += '|' + this.append_options.toString(); - - if(after) - this.data_url += "&after=" + after.toString(); - - if(before) - this.data_url += "&before=" + before.toString(); - - if(this.dimensions) - this.data_url += "&dimensions=" + this.dimensions; - - if(NETDATA.options.debug.chart_data_url === true || this.debug === true) - this.log('chartURL(): ' + this.data_url + ' WxH:' + this.chartWidth() + 'x' + this.chartHeight() + ' points: ' + this.data_points + ' library: ' + this.library_name); - }; - - this.redrawChart = function() { - if(this.data !== null) - this.updateChartWithData(this.data); - }; - - this.updateChartWithData = function(data) { - if(this.debug === true) - this.log('updateChartWithData() called.'); - - // this may force the chart to be re-created - resizeChart(); - - this.data = data; - this.updates_counter++; - this.updates_since_last_unhide++; - this.updates_since_last_creation++; - - var started = new Date().getTime(); - - // if the result is JSON, find the latest update-every - this.data_update_every = data.view_update_every * 1000; - this.data_after = data.after * 1000; - this.data_before = data.before * 1000; - this.netdata_first = data.first_entry * 1000; - this.netdata_last = data.last_entry * 1000; - this.data_points = data.points; - data.state = this; - - if(NETDATA.options.current.pan_and_zoom_data_padding === true && this.requested_padding !== null) { - if(this.view_after < this.data_after) { - // console.log('adusting view_after from ' + this.view_after + ' to ' + this.data_after); - this.view_after = this.data_after; - } - - if(this.view_before > this.data_before) { - // console.log('adusting view_before from ' + this.view_before + ' to ' + this.data_before); - this.view_before = this.data_before; - } - } - else { - this.view_after = this.data_after; - this.view_before = this.data_before; - } - - if(this.debug === true) { - this.log('UPDATE No ' + this.updates_counter + ' COMPLETED'); - - if(this.current.force_after_ms) - this.log('STATUS: forced : ' + (this.current.force_after_ms / 1000).toString() + ' - ' + (this.current.force_before_ms / 1000).toString()); - else - this.log('STATUS: forced : unset'); - - this.log('STATUS: requested : ' + (this.requested_after / 1000).toString() + ' - ' + (this.requested_before / 1000).toString()); - this.log('STATUS: downloaded: ' + (this.data_after / 1000).toString() + ' - ' + (this.data_before / 1000).toString()); - this.log('STATUS: rendered : ' + (this.view_after / 1000).toString() + ' - ' + (this.view_before / 1000).toString()); - this.log('STATUS: points : ' + (this.data_points).toString()); - } - - if(this.data_points === 0) { - noDataToShow(); - return; - } - - if(this.updates_since_last_creation >= this.library.max_updates_to_recreate()) { - if(this.debug === true) - this.log('max updates of ' + this.updates_since_last_creation.toString() + ' reached. Forcing re-generation.'); - - this.chart_created = false; - } - - // check and update the legend - this.legendUpdateDOM(); - - if(this.chart_created === true - && typeof this.library.update === 'function') { - - if(this.debug === true) - this.log('updating chart...'); - - if(callChartLibraryUpdateSafely(data) === false) - return; - } - else { - if(this.debug === true) - this.log('creating chart...'); - - if(callChartLibraryCreateSafely(data) === false) - return; - } - hideMessage(); - this.legendShowLatestValues(); - if(this.selected === true) - NETDATA.globalSelectionSync.stop(); - - // update the performance counters - var now = new Date().getTime(); - this.tm.last_updated = now; - - // don't update last_autorefreshed if this chart is - // forced to be updated with global PanAndZoom - if(NETDATA.globalPanAndZoom.isActive()) - this.tm.last_autorefreshed = 0; - else { - if(NETDATA.options.current.parallel_refresher === true && NETDATA.options.current.concurrent_refreshes === true) - this.tm.last_autorefreshed = now - (now % this.data_update_every); - else - this.tm.last_autorefreshed = now; - } - - this.refresh_dt_ms = now - started; - NETDATA.options.auto_refresher_fast_weight += this.refresh_dt_ms; - - if(this.refresh_dt_element !== null) - this.refresh_dt_element.innerHTML = this.refresh_dt_ms.toString(); - }; - - this.updateChart = function(callback) { - if(this.debug === true) - this.log('updateChart() called.'); - - if(this._updating === true) { - if(this.debug === true) - this.log('I am already updating...'); - - if(typeof callback === 'function') callback(); - return false; - } - - // due to late initialization of charts and libraries - // we need to check this too - if(this.enabled === false) { - if(this.debug === true) - this.log('I am not enabled'); - - if(typeof callback === 'function') callback(); - return false; - } - - if(canBeRendered() === false) { - if(typeof callback === 'function') callback(); - return false; - } - - if(this.chart === null) { - this.getChart(function() { that.updateChart(callback); }); - return false; - } - - if(this.library.initialized === false) { - if(this.library.enabled === true) { - this.library.initialize(function() { that.updateChart(callback); }); - return false; - } - else { - error('chart library "' + this.library_name + '" is not available.'); - if(typeof callback === 'function') callback(); - return false; - } - } - - this.clearSelection(); - this.chartURL(); - - if(this.debug === true) - this.log('updating from ' + this.data_url); - - NETDATA.statistics.refreshes_total++; - NETDATA.statistics.refreshes_active++; - - if(NETDATA.statistics.refreshes_active > NETDATA.statistics.refreshes_active_max) - NETDATA.statistics.refreshes_active_max = NETDATA.statistics.refreshes_active; - - this._updating = true; - - this.xhr = $.ajax( { - url: this.data_url, - cache: false, - async: true - }) - .success(function(data) { - if(that.debug === true) - that.log('data received. updating chart.'); - - that.updateChartWithData(data); - }) - .fail(function() { - error('data download failed for url: ' + that.data_url); - }) - .always(function() { - NETDATA.statistics.refreshes_active--; - that._updating = false; - if(typeof callback === 'function') callback(); - }); - - return true; - }; - - this.isVisible = function(nocache) { - if(typeof nocache === 'undefined') - nocache = false; - - // this.log('last_visible_check: ' + this.tm.last_visible_check + ', last_page_scroll: ' + NETDATA.options.last_page_scroll); - - // caching - we do not evaluate the charts visibility - // if the page has not been scrolled since the last check - if(nocache === false && this.tm.last_visible_check > NETDATA.options.last_page_scroll) - return this.___isVisible___; - - this.tm.last_visible_check = new Date().getTime(); - - var wh = window.innerHeight; - var x = this.element.getBoundingClientRect(); - var ret = 0; - var tolerance = 0; - - if(x.width === 0 || x.height === 0) { - hideChart(); - this.___isVisible___ = false; - return this.___isVisible___; - } - - if(x.top < 0 && -x.top > x.height) { - // the chart is entirely above - ret = -x.top - x.height; - } - else if(x.top > wh) { - // the chart is entirely below - ret = x.top - wh; - } - - if(ret > tolerance) { - // the chart is too far - - hideChart(); - this.___isVisible___ = false; - return this.___isVisible___; - } - else { - // the chart is inside or very close - - unhideChart(); - this.___isVisible___ = true; - return this.___isVisible___; - } - }; - - this.isAutoRefreshable = function() { - return (this.current.autorefresh); - }; - - this.canBeAutoRefreshed = function() { - var now = new Date().getTime(); - - if(this.running === true) { - if(this.debug === true) - this.log('I am already running'); - - return false; - } - - if(this.enabled === false) { - if(this.debug === true) - this.log('I am not enabled'); - - return false; - } - - if(this.library === null || this.library.enabled === false) { - error('charting library "' + this.library_name + '" is not available'); - if(this.debug === true) - this.log('My chart library ' + this.library_name + ' is not available'); - - return false; - } - - if(this.isVisible() === false) { - if(NETDATA.options.debug.visibility === true || this.debug === true) - this.log('I am not visible'); - - return false; - } - - if(this.current.force_update_at !== 0 && this.current.force_update_at < now) { - if(this.debug === true) - this.log('timed force update detected - allowing this update'); - - this.current.force_update_at = 0; - return true; - } - - if(this.isAutoRefreshable() === true) { - // allow the first update, even if the page is not visible - if(this.updates_counter && this.updates_since_last_unhide && NETDATA.options.page_is_visible === false) { - if(NETDATA.options.debug.focus === true || this.debug === true) - this.log('canBeAutoRefreshed(): page does not have focus'); - - return false; - } - - if(this.needsRecreation() === true) { - if(this.debug === true) - this.log('canBeAutoRefreshed(): needs re-creation.'); - - return true; - } - - // options valid only for autoRefresh() - if(NETDATA.options.auto_refresher_stop_until === 0 || NETDATA.options.auto_refresher_stop_until < now) { - if(NETDATA.globalPanAndZoom.isActive()) { - if(NETDATA.globalPanAndZoom.shouldBeAutoRefreshed(this)) { - if(this.debug === true) - this.log('canBeAutoRefreshed(): global panning: I need an update.'); - - return true; - } - else { - if(this.debug === true) - this.log('canBeAutoRefreshed(): global panning: I am already up to date.'); - - return false; - } - } - - if(this.selected === true) { - if(this.debug === true) - this.log('canBeAutoRefreshed(): I have a selection in place.'); - - return false; - } - - if(this.paused === true) { - if(this.debug === true) - this.log('canBeAutoRefreshed(): I am paused.'); - - return false; - } - - if(now - this.tm.last_autorefreshed >= this.data_update_every) { - if(this.debug === true) - this.log('canBeAutoRefreshed(): It is time to update me.'); - - return true; - } - } - } - - return false; - }; - - this.autoRefresh = function(callback) { - if(this.canBeAutoRefreshed() === true && this.running === false) { - var state = this; - - state.running = true; - state.updateChart(function() { - state.running = false; - - if(typeof callback !== 'undefined') - callback(); - }); - } - else { - if(typeof callback !== 'undefined') - callback(); - } - }; - - this._defaultsFromDownloadedChart = function(chart) { - this.chart = chart; - this.chart_url = chart.url; - this.data_update_every = chart.update_every * 1000; - this.data_points = Math.round(this.chartWidth() / this.chartPixelsPerPoint()); - this.tm.last_info_downloaded = new Date().getTime(); - - if(this.title === null) - this.title = chart.title; - - if(this.units === null) - this.units = chart.units; - }; - - // fetch the chart description from the netdata server - this.getChart = function(callback) { - this.chart = NETDATA.chartRegistry.get(this.host, this.id); - if(this.chart) { - this._defaultsFromDownloadedChart(this.chart); - if(typeof callback === 'function') callback(); - } - else { - this.chart_url = "/api/v1/chart?chart=" + this.id; - - if(this.debug === true) - this.log('downloading ' + this.chart_url); - - $.ajax( { - url: this.host + this.chart_url, - cache: false, - async: true - }) - .done(function(chart) { - chart.url = that.chart_url; - that._defaultsFromDownloadedChart(chart); - NETDATA.chartRegistry.add(that.host, that.id, chart); - }) - .fail(function() { - NETDATA.error(404, that.chart_url); - error('chart not found on url "' + that.chart_url + '"'); - }) - .always(function() { - if(typeof callback === 'function') callback(); - }); - } - }; - - // ============================================================================================================ - // INITIALIZATION - - init(); - }; - - NETDATA.resetAllCharts = function(state) { - // first clear the global selection sync - // to make sure no chart is in selected state - state.globalSelectionSyncStop(); - - // there are 2 possibilities here - // a. state is the global Pan and Zoom master - // b. state is not the global Pan and Zoom master - var master = true; - if(NETDATA.globalPanAndZoom.isMaster(state) === false) - master = false; - - // clear the global Pan and Zoom - // this will also refresh the master - // and unblock any charts currently mirroring the master - NETDATA.globalPanAndZoom.clearMaster(); - - // if we were not the master, reset our status too - // this is required because most probably the mouse - // is over this chart, blocking it from auto-refreshing - if(master === false && (state.paused === true || state.selected === true)) - state.resetChart(); - }; - - // get or create a chart state, given a DOM element - NETDATA.chartState = function(element) { - var state = $(element).data('netdata-state-object') || null; - if(state === null) { - state = new chartState(element); - $(element).data('netdata-state-object', state); - } - return state; - }; - - // ---------------------------------------------------------------------------------------------------------------- - // Library functions - - // Load a script without jquery - // This is used to load jquery - after it is loaded, we use jquery - NETDATA._loadjQuery = function(callback) { - if(typeof jQuery === 'undefined') { - if(NETDATA.options.debug.main_loop === true) - console.log('loading ' + NETDATA.jQuery); - - var script = document.createElement('script'); - script.type = 'text/javascript'; - script.async = true; - script.src = NETDATA.jQuery; - - // script.onabort = onError; - script.onerror = function(err, t) { NETDATA.error(101, NETDATA.jQuery); }; - if(typeof callback === "function") - script.onload = callback; - - var s = document.getElementsByTagName('script')[0]; - s.parentNode.insertBefore(script, s); - } - else if(typeof callback === "function") - callback(); - }; - - NETDATA._loadCSS = function(filename) { - // don't use jQuery here - // styles are loaded before jQuery - // to eliminate showing an unstyled page to the user - - var fileref = document.createElement("link"); - fileref.setAttribute("rel", "stylesheet"); - fileref.setAttribute("type", "text/css"); - fileref.setAttribute("href", filename); - - if (typeof fileref !== 'undefined') - document.getElementsByTagName("head")[0].appendChild(fileref); - }; - - NETDATA.colorHex2Rgb = function(hex) { - // Expand shorthand form (e.g. "03F") to full form (e.g. "0033FF") - var shorthandRegex = /^#?([a-f\d])([a-f\d])([a-f\d])$/i; - hex = hex.replace(shorthandRegex, function(m, r, g, b) { - return r + r + g + g + b + b; - }); - - var result = /^#?([a-f\d]{2})([a-f\d]{2})([a-f\d]{2})$/i.exec(hex); - return result ? { - r: parseInt(result[1], 16), - g: parseInt(result[2], 16), - b: parseInt(result[3], 16) - } : null; - }; - - NETDATA.colorLuminance = function(hex, lum) { - // validate hex string - hex = String(hex).replace(/[^0-9a-f]/gi, ''); - if (hex.length < 6) - hex = hex[0]+hex[0]+hex[1]+hex[1]+hex[2]+hex[2]; - - lum = lum || 0; - - // convert to decimal and change luminosity - var rgb = "#", c, i; - for (i = 0; i < 3; i++) { - c = parseInt(hex.substr(i*2,2), 16); - c = Math.round(Math.min(Math.max(0, c + (c * lum)), 255)).toString(16); - rgb += ("00"+c).substr(c.length); - } - - return rgb; - }; - - NETDATA.guid = function() { - function s4() { - return Math.floor((1 + Math.random()) * 0x10000) - .toString(16) - .substring(1); - } - - return s4() + s4() + '-' + s4() + '-' + s4() + '-' + s4() + '-' + s4() + s4() + s4(); - }; - - NETDATA.zeropad = function(x) { - if(x > -10 && x < 10) return '0' + x.toString(); - else return x.toString(); - }; - - // user function to signal us the DOM has been - // updated. - NETDATA.updatedDom = function() { - NETDATA.options.updated_dom = true; - }; - - NETDATA.ready = function(callback) { - NETDATA.options.pauseCallback = callback; - }; - - NETDATA.pause = function(callback) { - if(NETDATA.options.pause === true) - callback(); - else - NETDATA.options.pauseCallback = callback; - }; - - NETDATA.unpause = function() { - NETDATA.options.pauseCallback = null; - NETDATA.options.updated_dom = true; - NETDATA.options.pause = false; - }; - - // ---------------------------------------------------------------------------------------------------------------- - - // this is purely sequencial charts refresher - // it is meant to be autonomous - NETDATA.chartRefresherNoParallel = function(index) { - if(NETDATA.options.debug.mail_loop === true) - console.log('NETDATA.chartRefresherNoParallel(' + index + ')'); - - if(NETDATA.options.updated_dom === true) { - // the dom has been updated - // get the dom parts again - NETDATA.parseDom(NETDATA.chartRefresher); - return; - } - if(index >= NETDATA.options.targets.length) { - if(NETDATA.options.debug.main_loop === true) - console.log('waiting to restart main loop...'); - - NETDATA.options.auto_refresher_fast_weight = 0; - - setTimeout(function() { - NETDATA.chartRefresher(); - }, NETDATA.options.current.idle_between_loops); - } - else { - var state = NETDATA.options.targets[index]; - - if(NETDATA.options.auto_refresher_fast_weight < NETDATA.options.current.fast_render_timeframe) { - if(NETDATA.options.debug.main_loop === true) - console.log('fast rendering...'); - - state.autoRefresh(function() { - NETDATA.chartRefresherNoParallel(++index); - }); - } - else { - if(NETDATA.options.debug.main_loop === true) console.log('waiting for next refresh...'); - NETDATA.options.auto_refresher_fast_weight = 0; - - setTimeout(function() { - state.autoRefresh(function() { - NETDATA.chartRefresherNoParallel(++index); - }); - }, NETDATA.options.current.idle_between_charts); - } - } - }; - - // this is part of the parallel refresher - // its cause is to refresh sequencially all the charts - // that depend on chart library initialization - // it will call the parallel refresher back - // as soon as it sees a chart that its chart library - // is initialized - NETDATA.chartRefresher_uninitialized = function() { - if(NETDATA.options.updated_dom === true) { - // the dom has been updated - // get the dom parts again - NETDATA.parseDom(NETDATA.chartRefresher); - return; - } - - if(NETDATA.options.sequencial.length === 0) - NETDATA.chartRefresher(); - else { - var state = NETDATA.options.sequencial.pop(); - if(state.library.initialized === true) - NETDATA.chartRefresher(); - else - state.autoRefresh(NETDATA.chartRefresher_uninitialized); - } - }; - - NETDATA.chartRefresherWaitTime = function() { - return NETDATA.options.current.idle_parallel_loops; - }; - - // the default refresher - // it will create 2 sets of charts: - // - the ones that can be refreshed in parallel - // - the ones that depend on something else - // the first set will be executed in parallel - // the second will be given to NETDATA.chartRefresher_uninitialized() - NETDATA.chartRefresher = function() { - if(NETDATA.options.pause === true) { - // console.log('auto-refresher is paused'); - setTimeout(NETDATA.chartRefresher, - NETDATA.chartRefresherWaitTime()); - return; - } - - if(typeof NETDATA.options.pauseCallback === 'function') { - // console.log('auto-refresher is calling pauseCallback'); - NETDATA.options.pause = true; - NETDATA.options.pauseCallback(); - NETDATA.chartRefresher(); - return; - } - - if(NETDATA.options.current.parallel_refresher === false) { - NETDATA.chartRefresherNoParallel(0); - return; - } - - if(NETDATA.options.updated_dom === true) { - // the dom has been updated - // get the dom parts again - NETDATA.parseDom(NETDATA.chartRefresher); - return; - } - - var parallel = new Array(); - var targets = NETDATA.options.targets; - var len = targets.length; - var state; - while(len--) { - state = targets[len]; - if(state.isVisible() === false || state.running === true) - continue; - - if(state.library.initialized === false) { - if(state.library.enabled === true) { - state.library.initialize(NETDATA.chartRefresher); - return; - } - else { - state.error('chart library "' + state.library_name + '" is not enabled.'); - } - } - - parallel.unshift(state); - } - - if(parallel.length > 0) { - // this will execute the jobs in parallel - $(parallel).each(function() { - this.autoRefresh(); - }) - } - - // run the next refresh iteration - setTimeout(NETDATA.chartRefresher, - NETDATA.chartRefresherWaitTime()); - }; - - NETDATA.parseDom = function(callback) { - NETDATA.options.last_page_scroll = new Date().getTime(); - NETDATA.options.updated_dom = false; - - var targets = $('div[data-netdata]'); //.filter(':visible'); - - if(NETDATA.options.debug.main_loop === true) - console.log('DOM updated - there are ' + targets.length + ' charts on page.'); - - NETDATA.options.targets = new Array(); - var len = targets.length; - while(len--) { - // the initialization will take care of sizing - // and the "loading..." message - NETDATA.options.targets.push(NETDATA.chartState(targets[len])); - } - - if(typeof callback === 'function') callback(); - }; - - // this is the main function - where everything starts - NETDATA.start = function() { - // this should be called only once - - NETDATA.options.page_is_visible = true; - - $(window).blur(function() { - if(NETDATA.options.current.stop_updates_when_focus_is_lost === true) { - NETDATA.options.page_is_visible = false; - if(NETDATA.options.debug.focus === true) - console.log('Lost Focus!'); - } - }); - - $(window).focus(function() { - if(NETDATA.options.current.stop_updates_when_focus_is_lost === true) { - NETDATA.options.page_is_visible = true; - if(NETDATA.options.debug.focus === true) - console.log('Focus restored!'); - } - }); - - if(typeof document.hasFocus === 'function' && !document.hasFocus()) { - if(NETDATA.options.current.stop_updates_when_focus_is_lost === true) { - NETDATA.options.page_is_visible = false; - if(NETDATA.options.debug.focus === true) - console.log('Document has no focus!'); - } - } - - // bootstrap tab switching - $('a[data-toggle="tab"]').on('shown.bs.tab', NETDATA.onscroll); - - // bootstrap modal switching - $('.modal').on('hidden.bs.modal', NETDATA.onscroll); - $('.modal').on('shown.bs.modal', NETDATA.onscroll); - - // bootstrap collapse switching - $('.collapse').on('hidden.bs.collapse', NETDATA.onscroll); - $('.collapse').on('shown.bs.collapse', NETDATA.onscroll); - - NETDATA.parseDom(NETDATA.chartRefresher); - - // Registry initialization - setTimeout(NETDATA.registry.init, 3000); - }; - - // ---------------------------------------------------------------------------------------------------------------- - // peity - - NETDATA.peityInitialize = function(callback) { - if(typeof netdataNoPeitys === 'undefined' || !netdataNoPeitys) { - $.ajax({ - url: NETDATA.peity_js, - cache: true, - dataType: "script" - }) - .done(function() { - NETDATA.registerChartLibrary('peity', NETDATA.peity_js); - }) - .fail(function() { - NETDATA.chartLibraries.peity.enabled = false; - NETDATA.error(100, NETDATA.peity_js); - }) - .always(function() { - if(typeof callback === "function") - callback(); - }); - } - else { - NETDATA.chartLibraries.peity.enabled = false; - if(typeof callback === "function") - callback(); - } - }; - - NETDATA.peityChartUpdate = function(state, data) { - state.peity_instance.innerHTML = data.result; - - if(state.peity_options.stroke !== state.chartColors()[0]) { - state.peity_options.stroke = state.chartColors()[0]; - if(state.chart.chart_type === 'line') - state.peity_options.fill = NETDATA.themes.current.background; - else - state.peity_options.fill = NETDATA.colorLuminance(state.chartColors()[0], NETDATA.chartDefaults.fill_luminance); - } - - $(state.peity_instance).peity('line', state.peity_options); - return true; - }; - - NETDATA.peityChartCreate = function(state, data) { - state.peity_instance = document.createElement('div'); - state.element_chart.appendChild(state.peity_instance); - - var self = $(state.element); - state.peity_options = { - stroke: NETDATA.themes.current.foreground, - strokeWidth: self.data('peity-strokewidth') || 1, - width: state.chartWidth(), - height: state.chartHeight(), - fill: NETDATA.themes.current.foreground - }; - - NETDATA.peityChartUpdate(state, data); - return true; - }; - - // ---------------------------------------------------------------------------------------------------------------- - // sparkline - - NETDATA.sparklineInitialize = function(callback) { - if(typeof netdataNoSparklines === 'undefined' || !netdataNoSparklines) { - $.ajax({ - url: NETDATA.sparkline_js, - cache: true, - dataType: "script" - }) - .done(function() { - NETDATA.registerChartLibrary('sparkline', NETDATA.sparkline_js); - }) - .fail(function() { - NETDATA.chartLibraries.sparkline.enabled = false; - NETDATA.error(100, NETDATA.sparkline_js); - }) - .always(function() { - if(typeof callback === "function") - callback(); - }); - } - else { - NETDATA.chartLibraries.sparkline.enabled = false; - if(typeof callback === "function") - callback(); - } - }; - - NETDATA.sparklineChartUpdate = function(state, data) { - state.sparkline_options.width = state.chartWidth(); - state.sparkline_options.height = state.chartHeight(); - - $(state.element_chart).sparkline(data.result, state.sparkline_options); - return true; - }; - - NETDATA.sparklineChartCreate = function(state, data) { - var self = $(state.element); - var type = self.data('sparkline-type') || 'line'; - var lineColor = self.data('sparkline-linecolor') || state.chartColors()[0]; - var fillColor = self.data('sparkline-fillcolor') || (state.chart.chart_type === 'line')?NETDATA.themes.current.background:NETDATA.colorLuminance(lineColor, NETDATA.chartDefaults.fill_luminance); - var chartRangeMin = self.data('sparkline-chartrangemin') || undefined; - var chartRangeMax = self.data('sparkline-chartrangemax') || undefined; - var composite = self.data('sparkline-composite') || undefined; - var enableTagOptions = self.data('sparkline-enabletagoptions') || undefined; - var tagOptionPrefix = self.data('sparkline-tagoptionprefix') || undefined; - var tagValuesAttribute = self.data('sparkline-tagvaluesattribute') || undefined; - var disableHiddenCheck = self.data('sparkline-disablehiddencheck') || undefined; - var defaultPixelsPerValue = self.data('sparkline-defaultpixelspervalue') || undefined; - var spotColor = self.data('sparkline-spotcolor') || undefined; - var minSpotColor = self.data('sparkline-minspotcolor') || undefined; - var maxSpotColor = self.data('sparkline-maxspotcolor') || undefined; - var spotRadius = self.data('sparkline-spotradius') || undefined; - var valueSpots = self.data('sparkline-valuespots') || undefined; - var highlightSpotColor = self.data('sparkline-highlightspotcolor') || undefined; - var highlightLineColor = self.data('sparkline-highlightlinecolor') || undefined; - var lineWidth = self.data('sparkline-linewidth') || undefined; - var normalRangeMin = self.data('sparkline-normalrangemin') || undefined; - var normalRangeMax = self.data('sparkline-normalrangemax') || undefined; - var drawNormalOnTop = self.data('sparkline-drawnormalontop') || undefined; - var xvalues = self.data('sparkline-xvalues') || undefined; - var chartRangeClip = self.data('sparkline-chartrangeclip') || undefined; - var chartRangeMinX = self.data('sparkline-chartrangeminx') || undefined; - var chartRangeMaxX = self.data('sparkline-chartrangemaxx') || undefined; - var disableInteraction = self.data('sparkline-disableinteraction') || false; - var disableTooltips = self.data('sparkline-disabletooltips') || false; - var disableHighlight = self.data('sparkline-disablehighlight') || false; - var highlightLighten = self.data('sparkline-highlightlighten') || 1.4; - var highlightColor = self.data('sparkline-highlightcolor') || undefined; - var tooltipContainer = self.data('sparkline-tooltipcontainer') || undefined; - var tooltipClassname = self.data('sparkline-tooltipclassname') || undefined; - var tooltipFormat = self.data('sparkline-tooltipformat') || undefined; - var tooltipPrefix = self.data('sparkline-tooltipprefix') || undefined; - var tooltipSuffix = self.data('sparkline-tooltipsuffix') || ' ' + state.units; - var tooltipSkipNull = self.data('sparkline-tooltipskipnull') || true; - var tooltipValueLookups = self.data('sparkline-tooltipvaluelookups') || undefined; - var tooltipFormatFieldlist = self.data('sparkline-tooltipformatfieldlist') || undefined; - var tooltipFormatFieldlistKey = self.data('sparkline-tooltipformatfieldlistkey') || undefined; - var numberFormatter = self.data('sparkline-numberformatter') || function(n){ return n.toFixed(2); }; - var numberDigitGroupSep = self.data('sparkline-numberdigitgroupsep') || undefined; - var numberDecimalMark = self.data('sparkline-numberdecimalmark') || undefined; - var numberDigitGroupCount = self.data('sparkline-numberdigitgroupcount') || undefined; - var animatedZooms = self.data('sparkline-animatedzooms') || false; - - state.sparkline_options = { - type: type, - lineColor: lineColor, - fillColor: fillColor, - chartRangeMin: chartRangeMin, - chartRangeMax: chartRangeMax, - composite: composite, - enableTagOptions: enableTagOptions, - tagOptionPrefix: tagOptionPrefix, - tagValuesAttribute: tagValuesAttribute, - disableHiddenCheck: disableHiddenCheck, - defaultPixelsPerValue: defaultPixelsPerValue, - spotColor: spotColor, - minSpotColor: minSpotColor, - maxSpotColor: maxSpotColor, - spotRadius: spotRadius, - valueSpots: valueSpots, - highlightSpotColor: highlightSpotColor, - highlightLineColor: highlightLineColor, - lineWidth: lineWidth, - normalRangeMin: normalRangeMin, - normalRangeMax: normalRangeMax, - drawNormalOnTop: drawNormalOnTop, - xvalues: xvalues, - chartRangeClip: chartRangeClip, - chartRangeMinX: chartRangeMinX, - chartRangeMaxX: chartRangeMaxX, - disableInteraction: disableInteraction, - disableTooltips: disableTooltips, - disableHighlight: disableHighlight, - highlightLighten: highlightLighten, - highlightColor: highlightColor, - tooltipContainer: tooltipContainer, - tooltipClassname: tooltipClassname, - tooltipChartTitle: state.title, - tooltipFormat: tooltipFormat, - tooltipPrefix: tooltipPrefix, - tooltipSuffix: tooltipSuffix, - tooltipSkipNull: tooltipSkipNull, - tooltipValueLookups: tooltipValueLookups, - tooltipFormatFieldlist: tooltipFormatFieldlist, - tooltipFormatFieldlistKey: tooltipFormatFieldlistKey, - numberFormatter: numberFormatter, - numberDigitGroupSep: numberDigitGroupSep, - numberDecimalMark: numberDecimalMark, - numberDigitGroupCount: numberDigitGroupCount, - animatedZooms: animatedZooms, - width: state.chartWidth(), - height: state.chartHeight() - }; - - $(state.element_chart).sparkline(data.result, state.sparkline_options); - return true; - }; - - // ---------------------------------------------------------------------------------------------------------------- - // dygraph - - NETDATA.dygraph = { - smooth: false - }; - - NETDATA.dygraphToolboxPanAndZoom = function(state, after, before) { - if(after < state.netdata_first) - after = state.netdata_first; - - if(before > state.netdata_last) - before = state.netdata_last; - - state.setMode('zoom'); - state.globalSelectionSyncStop(); - state.globalSelectionSyncDelay(); - state.dygraph_user_action = true; - state.dygraph_force_zoom = true; - state.updateChartPanOrZoom(after, before); - NETDATA.globalPanAndZoom.setMaster(state, after, before); - }; - - NETDATA.dygraphSetSelection = function(state, t) { - if(typeof state.dygraph_instance !== 'undefined') { - var r = state.calculateRowForTime(t); - if(r !== -1) - state.dygraph_instance.setSelection(r); - else { - state.dygraph_instance.clearSelection(); - state.legendShowUndefined(); - } - } - - return true; - }; - - NETDATA.dygraphClearSelection = function(state, t) { - if(typeof state.dygraph_instance !== 'undefined') { - state.dygraph_instance.clearSelection(); - } - return true; - }; - - NETDATA.dygraphSmoothInitialize = function(callback) { - $.ajax({ - url: NETDATA.dygraph_smooth_js, - cache: true, - dataType: "script" - }) - .done(function() { - NETDATA.dygraph.smooth = true; - smoothPlotter.smoothing = 0.3; - }) - .fail(function() { - NETDATA.dygraph.smooth = false; - }) - .always(function() { - if(typeof callback === "function") - callback(); - }); - }; - - NETDATA.dygraphInitialize = function(callback) { - if(typeof netdataNoDygraphs === 'undefined' || !netdataNoDygraphs) { - $.ajax({ - url: NETDATA.dygraph_js, - cache: true, - dataType: "script" - }) - .done(function() { - NETDATA.registerChartLibrary('dygraph', NETDATA.dygraph_js); - }) - .fail(function() { - NETDATA.chartLibraries.dygraph.enabled = false; - NETDATA.error(100, NETDATA.dygraph_js); - }) - .always(function() { - if(NETDATA.chartLibraries.dygraph.enabled === true && NETDATA.options.current.smooth_plot === true) - NETDATA.dygraphSmoothInitialize(callback); - else if(typeof callback === "function") - callback(); - }); - } - else { - NETDATA.chartLibraries.dygraph.enabled = false; - if(typeof callback === "function") - callback(); - } - }; - - NETDATA.dygraphChartUpdate = function(state, data) { - var dygraph = state.dygraph_instance; - - if(typeof dygraph === 'undefined') - return NETDATA.dygraphChartCreate(state, data); - - // when the chart is not visible, and hidden - // if there is a window resize, dygraph detects - // its element size as 0x0. - // this will make it re-appear properly - - if(state.tm.last_unhidden > state.dygraph_last_rendered) - dygraph.resize(); - - var options = { - file: data.result.data, - colors: state.chartColors(), - labels: data.result.labels, - labelsDivWidth: state.chartWidth() - 70, - visibility: state.dimensions_visibility.selected2BooleanArray(state.data.dimension_names) - }; - - if(state.dygraph_force_zoom === true) { - if(NETDATA.options.debug.dygraph === true || state.debug === true) - state.log('dygraphChartUpdate() forced zoom update'); - - options.dateWindow = (state.requested_padding !== null)?[ state.view_after, state.view_before ]:null; - options.valueRange = null; - options.isZoomedIgnoreProgrammaticZoom = true; - state.dygraph_force_zoom = false; - } - else if(state.current.name !== 'auto') { - if(NETDATA.options.debug.dygraph === true || state.debug === true) - state.log('dygraphChartUpdate() loose update'); - } - else { - if(NETDATA.options.debug.dygraph === true || state.debug === true) - state.log('dygraphChartUpdate() strict update'); - - options.dateWindow = (state.requested_padding !== null)?[ state.view_after, state.view_before ]:null; - options.valueRange = null; - options.isZoomedIgnoreProgrammaticZoom = true; - } - - if(state.dygraph_smooth_eligible === true) { - if((NETDATA.options.current.smooth_plot === true && state.dygraph_options.plotter !== smoothPlotter) - || (NETDATA.options.current.smooth_plot === false && state.dygraph_options.plotter === smoothPlotter)) { - NETDATA.dygraphChartCreate(state, data); - return; - } - } - - dygraph.updateOptions(options); - - state.dygraph_last_rendered = new Date().getTime(); - return true; - }; - - NETDATA.dygraphChartCreate = function(state, data) { - if(NETDATA.options.debug.dygraph === true || state.debug === true) - state.log('dygraphChartCreate()'); - - var self = $(state.element); - - var chart_type = state.chart.chart_type; - if(chart_type === 'stacked' && data.dimensions === 1) chart_type = 'area'; - chart_type = self.data('dygraph-type') || chart_type; - - var smooth = (chart_type === 'line' && !NETDATA.chartLibraries.dygraph.isSparkline(state))?true:false; - smooth = self.data('dygraph-smooth') || smooth; - - if(NETDATA.dygraph.smooth === false) - smooth = false; - - var strokeWidth = (chart_type === 'stacked')?0.1:((smooth)?1.5:0.7) - var highlightCircleSize = (NETDATA.chartLibraries.dygraph.isSparkline(state))?3:4; - - state.dygraph_options = { - colors: self.data('dygraph-colors') || state.chartColors(), - - // leave a few pixels empty on the right of the chart - rightGap: self.data('dygraph-rightgap') || 5, - showRangeSelector: self.data('dygraph-showrangeselector') || false, - showRoller: self.data('dygraph-showroller') || false, - - title: self.data('dygraph-title') || state.title, - titleHeight: self.data('dygraph-titleheight') || 19, - - legend: self.data('dygraph-legend') || 'always', // 'onmouseover', - labels: data.result.labels, - labelsDiv: self.data('dygraph-labelsdiv') || state.element_legend_childs.hidden, - labelsDivStyles: self.data('dygraph-labelsdivstyles') || { 'fontSize':'1px' }, - labelsDivWidth: self.data('dygraph-labelsdivwidth') || state.chartWidth() - 70, - labelsSeparateLines: self.data('dygraph-labelsseparatelines') || true, - labelsShowZeroValues: self.data('dygraph-labelsshowzerovalues') || true, - labelsKMB: false, - labelsKMG2: false, - showLabelsOnHighlight: self.data('dygraph-showlabelsonhighlight') || true, - hideOverlayOnMouseOut: self.data('dygraph-hideoverlayonmouseout') || true, - - ylabel: state.units, - yLabelWidth: self.data('dygraph-ylabelwidth') || 12, - - // the function to plot the chart - plotter: null, - - // The width of the lines connecting data points. This can be used to increase the contrast or some graphs. - strokeWidth: self.data('dygraph-strokewidth') || strokeWidth, - strokePattern: self.data('dygraph-strokepattern') || undefined, - - // The size of the dot to draw on each point in pixels (see drawPoints). A dot is always drawn when a point is "isolated", - // i.e. there is a missing point on either side of it. This also controls the size of those dots. - drawPoints: self.data('dygraph-drawpoints') || false, - - // Draw points at the edges of gaps in the data. This improves visibility of small data segments or other data irregularities. - drawGapEdgePoints: self.data('dygraph-drawgapedgepoints') || true, - - connectSeparatedPoints: self.data('dygraph-connectseparatedpoints') || false, - pointSize: self.data('dygraph-pointsize') || 1, - - // enabling this makes the chart with little square lines - stepPlot: self.data('dygraph-stepplot') || false, - - // Draw a border around graph lines to make crossing lines more easily distinguishable. Useful for graphs with many lines. - strokeBorderColor: self.data('dygraph-strokebordercolor') || NETDATA.themes.current.background, - strokeBorderWidth: self.data('dygraph-strokeborderwidth') || (chart_type === 'stacked')?0.0:0.0, - - fillGraph: self.data('dygraph-fillgraph') || (chart_type === 'area' || chart_type === 'stacked')?true:false, - fillAlpha: self.data('dygraph-fillalpha') || (chart_type === 'stacked')?NETDATA.options.current.color_fill_opacity_stacked:NETDATA.options.current.color_fill_opacity_area, - stackedGraph: self.data('dygraph-stackedgraph') || (chart_type === 'stacked')?true:false, - stackedGraphNaNFill: self.data('dygraph-stackedgraphnanfill') || 'none', - - drawAxis: self.data('dygraph-drawaxis') || true, - axisLabelFontSize: self.data('dygraph-axislabelfontsize') || 10, - axisLineColor: self.data('dygraph-axislinecolor') || NETDATA.themes.current.axis, - axisLineWidth: self.data('dygraph-axislinewidth') || 0.3, - - drawGrid: self.data('dygraph-drawgrid') || true, - drawXGrid: self.data('dygraph-drawxgrid') || undefined, - drawYGrid: self.data('dygraph-drawygrid') || undefined, - gridLinePattern: self.data('dygraph-gridlinepattern') || null, - gridLineWidth: self.data('dygraph-gridlinewidth') || 0.3, - gridLineColor: self.data('dygraph-gridlinecolor') || NETDATA.themes.current.grid, - - maxNumberWidth: self.data('dygraph-maxnumberwidth') || 8, - sigFigs: self.data('dygraph-sigfigs') || null, - digitsAfterDecimal: self.data('dygraph-digitsafterdecimal') || 2, - valueFormatter: self.data('dygraph-valueformatter') || function(x){ return x.toFixed(2); }, - - highlightCircleSize: self.data('dygraph-highlightcirclesize') || highlightCircleSize, - highlightSeriesOpts: self.data('dygraph-highlightseriesopts') || null, // TOO SLOW: { strokeWidth: 1.5 }, - highlightSeriesBackgroundAlpha: self.data('dygraph-highlightseriesbackgroundalpha') || null, // TOO SLOW: (chart_type === 'stacked')?0.7:0.5, - - pointClickCallback: self.data('dygraph-pointclickcallback') || undefined, - visibility: state.dimensions_visibility.selected2BooleanArray(state.data.dimension_names), - axes: { - x: { - pixelsPerLabel: 50, - ticker: Dygraph.dateTicker, - axisLabelFormatter: function (d, gran) { - return NETDATA.zeropad(d.getHours()) + ":" + NETDATA.zeropad(d.getMinutes()) + ":" + NETDATA.zeropad(d.getSeconds()); - }, - valueFormatter: function (ms) { - var d = new Date(ms); - return d.toLocaleDateString() + ' ' + d.toLocaleTimeString(); - // return NETDATA.zeropad(d.getHours()) + ":" + NETDATA.zeropad(d.getMinutes()) + ":" + NETDATA.zeropad(d.getSeconds()); - } - }, - y: { - pixelsPerLabel: 15, - valueFormatter: function (x) { - // we format legends with the state object - // no need to do anything here - // return (Math.round(x*100) / 100).toLocaleString(); - // return state.legendFormatValue(x); - return x; - } - } - }, - legendFormatter: function(data) { - var elements = state.element_legend_childs; - - // if the hidden div is not there - // we are not managing the legend - if(elements.hidden === null) return; - - if (typeof data.x !== 'undefined') { - state.legendSetDate(data.x); - var i = data.series.length; - while(i--) { - var series = data.series[i]; - if(!series.isVisible) continue; - state.legendSetLabelValue(series.label, series.y); - } - } - - return ''; - }, - drawCallback: function(dygraph, is_initial) { - if(state.current.name !== 'auto' && state.dygraph_user_action === true) { - state.dygraph_user_action = false; - - var x_range = dygraph.xAxisRange(); - var after = Math.round(x_range[0]); - var before = Math.round(x_range[1]); - - if(NETDATA.options.debug.dygraph === true) - state.log('dygraphDrawCallback(dygraph, ' + is_initial + '): ' + (after / 1000).toString() + ' - ' + (before / 1000).toString()); - - if(before <= state.netdata_last && after >= state.netdata_first) - state.updateChartPanOrZoom(after, before); - } - }, - zoomCallback: function(minDate, maxDate, yRanges) { - if(NETDATA.options.debug.dygraph === true) - state.log('dygraphZoomCallback()'); - - state.globalSelectionSyncStop(); - state.globalSelectionSyncDelay(); - state.setMode('zoom'); - - // refresh it to the greatest possible zoom level - state.dygraph_user_action = true; - state.dygraph_force_zoom = true; - state.updateChartPanOrZoom(minDate, maxDate); - }, - highlightCallback: function(event, x, points, row, seriesName) { - if(NETDATA.options.debug.dygraph === true || state.debug === true) - state.log('dygraphHighlightCallback()'); - - state.pauseChart(); - - // there is a bug in dygraph when the chart is zoomed enough - // the time it thinks is selected is wrong - // here we calculate the time t based on the row number selected - // which is ok - var t = state.data_after + row * state.data_update_every; - // console.log('row = ' + row + ', x = ' + x + ', t = ' + t + ' ' + ((t === x)?'SAME':(Math.abs(x-t)<=state.data_update_every)?'SIMILAR':'DIFFERENT') + ', rows in db: ' + state.data_points + ' visible(x) = ' + state.timeIsVisible(x) + ' visible(t) = ' + state.timeIsVisible(t) + ' r(x) = ' + state.calculateRowForTime(x) + ' r(t) = ' + state.calculateRowForTime(t) + ' range: ' + state.data_after + ' - ' + state.data_before + ' real: ' + state.data.after + ' - ' + state.data.before + ' every: ' + state.data_update_every); - - state.globalSelectionSync(x); - - // fix legend zIndex using the internal structures of dygraph legend module - // this works, but it is a hack! - // state.dygraph_instance.plugins_[0].plugin.legend_div_.style.zIndex = 10000; - }, - unhighlightCallback: function(event) { - if(NETDATA.options.debug.dygraph === true || state.debug === true) - state.log('dygraphUnhighlightCallback()'); - - state.unpauseChart(); - state.globalSelectionSyncStop(); - }, - interactionModel : { - mousedown: function(event, dygraph, context) { - if(NETDATA.options.debug.dygraph === true || state.debug === true) - state.log('interactionModel.mousedown()'); - - state.dygraph_user_action = true; - state.globalSelectionSyncStop(); - - if(NETDATA.options.debug.dygraph === true) - state.log('dygraphMouseDown()'); - - // Right-click should not initiate a zoom. - if(event.button && event.button === 2) return; - - context.initializeMouseDown(event, dygraph, context); - - if(event.button && event.button === 1) { - if (event.altKey || event.shiftKey) { - state.setMode('pan'); - state.globalSelectionSyncDelay(); - Dygraph.startPan(event, dygraph, context); - } - else { - state.setMode('zoom'); - state.globalSelectionSyncDelay(); - Dygraph.startZoom(event, dygraph, context); - } - } - else { - if (event.altKey || event.shiftKey) { - state.setMode('zoom'); - state.globalSelectionSyncDelay(); - Dygraph.startZoom(event, dygraph, context); - } - else { - state.setMode('pan'); - state.globalSelectionSyncDelay(); - Dygraph.startPan(event, dygraph, context); - } - } - }, - mousemove: function(event, dygraph, context) { - if(NETDATA.options.debug.dygraph === true || state.debug === true) - state.log('interactionModel.mousemove()'); - - if(context.isPanning) { - state.dygraph_user_action = true; - state.globalSelectionSyncStop(); - state.globalSelectionSyncDelay(); - state.setMode('pan'); - Dygraph.movePan(event, dygraph, context); - } - else if(context.isZooming) { - state.dygraph_user_action = true; - state.globalSelectionSyncStop(); - state.globalSelectionSyncDelay(); - state.setMode('zoom'); - Dygraph.moveZoom(event, dygraph, context); - } - }, - mouseup: function(event, dygraph, context) { - if(NETDATA.options.debug.dygraph === true || state.debug === true) - state.log('interactionModel.mouseup()'); - - if (context.isPanning) { - state.dygraph_user_action = true; - state.globalSelectionSyncDelay(); - Dygraph.endPan(event, dygraph, context); - } - else if (context.isZooming) { - state.dygraph_user_action = true; - state.globalSelectionSyncDelay(); - Dygraph.endZoom(event, dygraph, context); - } - }, - click: function(event, dygraph, context) { - if(NETDATA.options.debug.dygraph === true || state.debug === true) - state.log('interactionModel.click()'); - - event.preventDefault(); - }, - dblclick: function(event, dygraph, context) { - if(NETDATA.options.debug.dygraph === true || state.debug === true) - state.log('interactionModel.dblclick()'); - NETDATA.resetAllCharts(state); - }, - mousewheel: function(event, dygraph, context) { - if(NETDATA.options.debug.dygraph === true || state.debug === true) - state.log('interactionModel.mousewheel()'); - - // Take the offset of a mouse event on the dygraph canvas and - // convert it to a pair of percentages from the bottom left. - // (Not top left, bottom is where the lower value is.) - function offsetToPercentage(g, offsetX, offsetY) { - // This is calculating the pixel offset of the leftmost date. - var xOffset = g.toDomCoords(g.xAxisRange()[0], null)[0]; - var yar0 = g.yAxisRange(0); - - // This is calculating the pixel of the higest value. (Top pixel) - var yOffset = g.toDomCoords(null, yar0[1])[1]; - - // x y w and h are relative to the corner of the drawing area, - // so that the upper corner of the drawing area is (0, 0). - var x = offsetX - xOffset; - var y = offsetY - yOffset; - - // This is computing the rightmost pixel, effectively defining the - // width. - var w = g.toDomCoords(g.xAxisRange()[1], null)[0] - xOffset; - - // This is computing the lowest pixel, effectively defining the height. - var h = g.toDomCoords(null, yar0[0])[1] - yOffset; - - // Percentage from the left. - var xPct = w === 0 ? 0 : (x / w); - // Percentage from the top. - var yPct = h === 0 ? 0 : (y / h); - - // The (1-) part below changes it from "% distance down from the top" - // to "% distance up from the bottom". - return [xPct, (1-yPct)]; - } - - // Adjusts [x, y] toward each other by zoomInPercentage% - // Split it so the left/bottom axis gets xBias/yBias of that change and - // tight/top gets (1-xBias)/(1-yBias) of that change. - // - // If a bias is missing it splits it down the middle. - function zoomRange(g, zoomInPercentage, xBias, yBias) { - xBias = xBias || 0.5; - yBias = yBias || 0.5; - - function adjustAxis(axis, zoomInPercentage, bias) { - var delta = axis[1] - axis[0]; - var increment = delta * zoomInPercentage; - var foo = [increment * bias, increment * (1-bias)]; - - return [ axis[0] + foo[0], axis[1] - foo[1] ]; - } - - var yAxes = g.yAxisRanges(); - var newYAxes = []; - for (var i = 0; i < yAxes.length; i++) { - newYAxes[i] = adjustAxis(yAxes[i], zoomInPercentage, yBias); - } - - return adjustAxis(g.xAxisRange(), zoomInPercentage, xBias); - } - - if(event.altKey || event.shiftKey) { - state.dygraph_user_action = true; - - state.globalSelectionSyncStop(); - state.globalSelectionSyncDelay(); - - // http://dygraphs.com/gallery/interaction-api.js - var normal = (event.detail) ? event.detail * -1 : event.wheelDelta / 40; - var percentage = normal / 50; - - if (!(event.offsetX && event.offsetY)){ - event.offsetX = event.layerX - event.target.offsetLeft; - event.offsetY = event.layerY - event.target.offsetTop; - } - - var percentages = offsetToPercentage(dygraph, event.offsetX, event.offsetY); - var xPct = percentages[0]; - var yPct = percentages[1]; - - var new_x_range = zoomRange(dygraph, percentage, xPct, yPct); - - var after = new_x_range[0]; - var before = new_x_range[1]; - - var first = state.netdata_first + state.data_update_every; - var last = state.netdata_last + state.data_update_every; - - if(before > last) { - after -= (before - last); - before = last; - } - if(after < first) { - after = first; - } - - state.setMode('zoom'); - if(state.updateChartPanOrZoom(after, before) === true) - dygraph.updateOptions({ dateWindow: [ after, before ] }); - - event.preventDefault(); - } - }, - touchstart: function(event, dygraph, context) { - if(NETDATA.options.debug.dygraph === true || state.debug === true) - state.log('interactionModel.touchstart()'); - - state.dygraph_user_action = true; - state.setMode('zoom'); - state.pauseChart(); - - Dygraph.defaultInteractionModel.touchstart(event, dygraph, context); - - // we overwrite the touch directions at the end, to overwrite - // the internal default of dygraphs - context.touchDirections = { x: true, y: false }; - - state.dygraph_last_touch_start = new Date().getTime(); - state.dygraph_last_touch_move = 0; - - if(typeof event.touches[0].pageX === 'number') - state.dygraph_last_touch_page_x = event.touches[0].pageX; - else - state.dygraph_last_touch_page_x = 0; - }, - touchmove: function(event, dygraph, context) { - if(NETDATA.options.debug.dygraph === true || state.debug === true) - state.log('interactionModel.touchmove()'); - - state.dygraph_user_action = true; - Dygraph.defaultInteractionModel.touchmove(event, dygraph, context); - - state.dygraph_last_touch_move = new Date().getTime(); - }, - touchend: function(event, dygraph, context) { - if(NETDATA.options.debug.dygraph === true || state.debug === true) - state.log('interactionModel.touchend()'); - - state.dygraph_user_action = true; - Dygraph.defaultInteractionModel.touchend(event, dygraph, context); - - // if it didn't move, it is a selection - if(state.dygraph_last_touch_move === 0 && state.dygraph_last_touch_page_x !== 0) { - // internal api of dygraphs - var pct = (state.dygraph_last_touch_page_x - (dygraph.plotter_.area.x + state.element.getBoundingClientRect().left)) / dygraph.plotter_.area.w; - var t = Math.round(state.data_after + (state.data_before - state.data_after) * pct); - if(NETDATA.dygraphSetSelection(state, t) === true) - state.globalSelectionSync(t); - } - - // if it was double tap within double click time, reset the charts - var now = new Date().getTime(); - if(typeof state.dygraph_last_touch_end !== 'undefined') { - if(state.dygraph_last_touch_move === 0) { - var dt = now - state.dygraph_last_touch_end; - if(dt <= NETDATA.options.current.double_click_speed) - NETDATA.resetAllCharts(state); - } - } - - // remember the timestamp of the last touch end - state.dygraph_last_touch_end = now; - } - } - }; - - if(NETDATA.chartLibraries.dygraph.isSparkline(state)) { - state.dygraph_options.drawGrid = false; - state.dygraph_options.drawAxis = false; - state.dygraph_options.title = undefined; - state.dygraph_options.units = undefined; - state.dygraph_options.ylabel = undefined; - state.dygraph_options.yLabelWidth = 0; - state.dygraph_options.labelsDivWidth = 120; - state.dygraph_options.labelsDivStyles.width = '120px'; - state.dygraph_options.labelsSeparateLines = true; - state.dygraph_options.rightGap = 0; - } - - if(smooth === true) { - state.dygraph_smooth_eligible = true; - - if(NETDATA.options.current.smooth_plot === true) - state.dygraph_options.plotter = smoothPlotter; - } - else state.dygraph_smooth_eligible = false; - - state.dygraph_instance = new Dygraph(state.element_chart, - data.result.data, state.dygraph_options); - - state.dygraph_force_zoom = false; - state.dygraph_user_action = false; - state.dygraph_last_rendered = new Date().getTime(); - return true; - }; - - // ---------------------------------------------------------------------------------------------------------------- - // morris - - NETDATA.morrisInitialize = function(callback) { - if(typeof netdataNoMorris === 'undefined' || !netdataNoMorris) { - - // morris requires raphael - if(!NETDATA.chartLibraries.raphael.initialized) { - if(NETDATA.chartLibraries.raphael.enabled) { - NETDATA.raphaelInitialize(function() { - NETDATA.morrisInitialize(callback); - }); - } - else { - NETDATA.chartLibraries.morris.enabled = false; - if(typeof callback === "function") - callback(); - } - } - else { - NETDATA._loadCSS(NETDATA.morris_css); - - $.ajax({ - url: NETDATA.morris_js, - cache: true, - dataType: "script" - }) - .done(function() { - NETDATA.registerChartLibrary('morris', NETDATA.morris_js); - }) - .fail(function() { - NETDATA.chartLibraries.morris.enabled = false; - NETDATA.error(100, NETDATA.morris_js); - }) - .always(function() { - if(typeof callback === "function") - callback(); - }); - } - } - else { - NETDATA.chartLibraries.morris.enabled = false; - if(typeof callback === "function") - callback(); - } - }; - - NETDATA.morrisChartUpdate = function(state, data) { - state.morris_instance.setData(data.result.data); - return true; - }; - - NETDATA.morrisChartCreate = function(state, data) { - - state.morris_options = { - element: state.element_chart.id, - data: data.result.data, - xkey: 'time', - ykeys: data.dimension_names, - labels: data.dimension_names, - lineWidth: 2, - pointSize: 3, - smooth: true, - hideHover: 'auto', - parseTime: true, - continuousLine: false, - behaveLikeLine: false - }; - - if(state.chart.chart_type === 'line') - state.morris_instance = new Morris.Line(state.morris_options); - - else if(state.chart.chart_type === 'area') { - state.morris_options.behaveLikeLine = true; - state.morris_instance = new Morris.Area(state.morris_options); - } - else // stacked - state.morris_instance = new Morris.Area(state.morris_options); - - return true; - }; - - // ---------------------------------------------------------------------------------------------------------------- - // raphael - - NETDATA.raphaelInitialize = function(callback) { - if(typeof netdataStopRaphael === 'undefined' || !netdataStopRaphael) { - $.ajax({ - url: NETDATA.raphael_js, - cache: true, - dataType: "script" - }) - .done(function() { - NETDATA.registerChartLibrary('raphael', NETDATA.raphael_js); - }) - .fail(function() { - NETDATA.chartLibraries.raphael.enabled = false; - NETDATA.error(100, NETDATA.raphael_js); - }) - .always(function() { - if(typeof callback === "function") - callback(); - }); - } - else { - NETDATA.chartLibraries.raphael.enabled = false; - if(typeof callback === "function") - callback(); - } - }; - - NETDATA.raphaelChartUpdate = function(state, data) { - $(state.element_chart).raphael(data.result, { - width: state.chartWidth(), - height: state.chartHeight() - }); - - return false; - }; - - NETDATA.raphaelChartCreate = function(state, data) { - $(state.element_chart).raphael(data.result, { - width: state.chartWidth(), - height: state.chartHeight() - }); - - return false; - }; - - // ---------------------------------------------------------------------------------------------------------------- - // C3 - - NETDATA.c3Initialize = function(callback) { - if(typeof netdataNoC3 === 'undefined' || !netdataNoC3) { - - // C3 requires D3 - if(!NETDATA.chartLibraries.d3.initialized) { - if(NETDATA.chartLibraries.d3.enabled) { - NETDATA.d3Initialize(function() { - NETDATA.c3Initialize(callback); - }); - } - else { - NETDATA.chartLibraries.c3.enabled = false; - if(typeof callback === "function") - callback(); - } - } - else { - NETDATA._loadCSS(NETDATA.c3_css); - - $.ajax({ - url: NETDATA.c3_js, - cache: true, - dataType: "script" - }) - .done(function() { - NETDATA.registerChartLibrary('c3', NETDATA.c3_js); - }) - .fail(function() { - NETDATA.chartLibraries.c3.enabled = false; - NETDATA.error(100, NETDATA.c3_js); - }) - .always(function() { - if(typeof callback === "function") - callback(); - }); - } - } - else { - NETDATA.chartLibraries.c3.enabled = false; - if(typeof callback === "function") - callback(); - } - }; - - NETDATA.c3ChartUpdate = function(state, data) { - state.c3_instance.destroy(); - return NETDATA.c3ChartCreate(state, data); - - //state.c3_instance.load({ - // rows: data.result, - // unload: true - //}); - - //return true; - }; - - NETDATA.c3ChartCreate = function(state, data) { - - state.element_chart.id = 'c3-' + state.uuid; - // console.log('id = ' + state.element_chart.id); - - state.c3_instance = c3.generate({ - bindto: '#' + state.element_chart.id, - size: { - width: state.chartWidth(), - height: state.chartHeight() - }, - color: { - pattern: state.chartColors() - }, - data: { - x: 'time', - rows: data.result, - type: (state.chart.chart_type === 'line')?'spline':'area-spline' - }, - axis: { - x: { - type: 'timeseries', - tick: { - format: function(x) { - return NETDATA.zeropad(x.getHours()) + ":" + NETDATA.zeropad(x.getMinutes()) + ":" + NETDATA.zeropad(x.getSeconds()); - } - } - } - }, - grid: { - x: { - show: true - }, - y: { - show: true - } - }, - point: { - show: false - }, - line: { - connectNull: false - }, - transition: { - duration: 0 - }, - interaction: { - enabled: true - } - }); - - // console.log(state.c3_instance); - - return true; - }; - - // ---------------------------------------------------------------------------------------------------------------- - // D3 - - NETDATA.d3Initialize = function(callback) { - if(typeof netdataStopD3 === 'undefined' || !netdataStopD3) { - $.ajax({ - url: NETDATA.d3_js, - cache: true, - dataType: "script" - }) - .done(function() { - NETDATA.registerChartLibrary('d3', NETDATA.d3_js); - }) - .fail(function() { - NETDATA.chartLibraries.d3.enabled = false; - NETDATA.error(100, NETDATA.d3_js); - }) - .always(function() { - if(typeof callback === "function") - callback(); - }); - } - else { - NETDATA.chartLibraries.d3.enabled = false; - if(typeof callback === "function") - callback(); - } - }; - - NETDATA.d3ChartUpdate = function(state, data) { - return false; - }; - - NETDATA.d3ChartCreate = function(state, data) { - return false; - }; - - // ---------------------------------------------------------------------------------------------------------------- - // google charts - - NETDATA.googleInitialize = function(callback) { - if(typeof netdataNoGoogleCharts === 'undefined' || !netdataNoGoogleCharts) { - $.ajax({ - url: NETDATA.google_js, - cache: true, - dataType: "script" - }) - .done(function() { - NETDATA.registerChartLibrary('google', NETDATA.google_js); - google.load('visualization', '1.1', { - 'packages': ['corechart', 'controls'], - 'callback': callback - }); - }) - .fail(function() { - NETDATA.chartLibraries.google.enabled = false; - NETDATA.error(100, NETDATA.google_js); - if(typeof callback === "function") - callback(); - }); - } - else { - NETDATA.chartLibraries.google.enabled = false; - if(typeof callback === "function") - callback(); - } - }; - - NETDATA.googleChartUpdate = function(state, data) { - var datatable = new google.visualization.DataTable(data.result); - state.google_instance.draw(datatable, state.google_options); - return true; - }; - - NETDATA.googleChartCreate = function(state, data) { - var datatable = new google.visualization.DataTable(data.result); - - state.google_options = { - colors: state.chartColors(), - - // do not set width, height - the chart resizes itself - //width: state.chartWidth(), - //height: state.chartHeight(), - lineWidth: 1, - title: state.title, - fontSize: 11, - hAxis: { - // title: "Time of Day", - // format:'HH:mm:ss', - viewWindowMode: 'maximized', - slantedText: false, - format:'HH:mm:ss', - textStyle: { - fontSize: 9 - }, - gridlines: { - color: '#EEE' - } - }, - vAxis: { - title: state.units, - viewWindowMode: 'pretty', - minValue: -0.1, - maxValue: 0.1, - direction: 1, - textStyle: { - fontSize: 9 - }, - gridlines: { - color: '#EEE' - } - }, - chartArea: { - width: '65%', - height: '80%' - }, - focusTarget: 'category', - annotation: { - '1': { - style: 'line' - } - }, - pointsVisible: 0, - titlePosition: 'out', - titleTextStyle: { - fontSize: 11 - }, - tooltip: { - isHtml: false, - ignoreBounds: true, - textStyle: { - fontSize: 9 - } - }, - curveType: 'function', - areaOpacity: 0.3, - isStacked: false - }; - - switch(state.chart.chart_type) { - case "area": - state.google_options.vAxis.viewWindowMode = 'maximized'; - state.google_options.areaOpacity = NETDATA.options.current.color_fill_opacity_area; - state.google_instance = new google.visualization.AreaChart(state.element_chart); - break; - - case "stacked": - state.google_options.isStacked = true; - state.google_options.areaOpacity = NETDATA.options.current.color_fill_opacity_stacked; - state.google_options.vAxis.viewWindowMode = 'maximized'; - state.google_options.vAxis.minValue = null; - state.google_options.vAxis.maxValue = null; - state.google_instance = new google.visualization.AreaChart(state.element_chart); - break; - - default: - case "line": - state.google_options.lineWidth = 2; - state.google_instance = new google.visualization.LineChart(state.element_chart); - break; - } - - state.google_instance.draw(datatable, state.google_options); - return true; - }; - - // ---------------------------------------------------------------------------------------------------------------- - - NETDATA.percentFromValueMax = function(value, max) { - if(value === null) value = 0; - if(max < value) max = value; - - var pcent = 0; - if(max !== 0) { - pcent = Math.round(value * 100 / max); - if(pcent === 0 && value > 0) pcent = 1; - } - - return pcent; - }; - - // ---------------------------------------------------------------------------------------------------------------- - // easy-pie-chart - - NETDATA.easypiechartInitialize = function(callback) { - if(typeof netdataNoEasyPieChart === 'undefined' || !netdataNoEasyPieChart) { - $.ajax({ - url: NETDATA.easypiechart_js, - cache: true, - dataType: "script" - }) - .done(function() { - NETDATA.registerChartLibrary('easypiechart', NETDATA.easypiechart_js); - }) - .fail(function() { - NETDATA.chartLibraries.easypiechart.enabled = false; - NETDATA.error(100, NETDATA.easypiechart_js); - }) - .always(function() { - if(typeof callback === "function") - callback(); - }) - } - else { - NETDATA.chartLibraries.easypiechart.enabled = false; - if(typeof callback === "function") - callback(); - } - }; - - NETDATA.easypiechartClearSelection = function(state) { - if(typeof state.easyPieChartEvent !== 'undefined') { - if(state.easyPieChartEvent.timer !== null) - clearTimeout(state.easyPieChartEvent.timer); - - state.easyPieChartEvent.timer = null; - } - - if(state.isAutoRefreshable() === true && state.data !== null) { - NETDATA.easypiechartChartUpdate(state, state.data); - } - else { - state.easyPieChartLabel.innerHTML = state.legendFormatValue(null); - state.easyPieChart_instance.update(0); - } - state.easyPieChart_instance.enableAnimation(); - - return true; - }; - - NETDATA.easypiechartSetSelection = function(state, t) { - if(state.timeIsVisible(t) !== true) - return NETDATA.easypiechartClearSelection(state); - - var slot = state.calculateRowForTime(t); - if(slot < 0 || slot >= state.data.result.length) - return NETDATA.easypiechartClearSelection(state); - - if(typeof state.easyPieChartEvent === 'undefined') { - state.easyPieChartEvent = { - timer: null, - value: 0, - pcent: 0 - }; - } - - var value = state.data.result[state.data.result.length - 1 - slot]; - var max = (state.easyPieChartMax === null)?state.data.max:state.easyPieChartMax; - var pcent = NETDATA.percentFromValueMax(value, max); - - state.easyPieChartEvent.value = value; - state.easyPieChartEvent.pcent = pcent; - state.easyPieChartLabel.innerHTML = state.legendFormatValue(value); - - if(state.easyPieChartEvent.timer === null) { - state.easyPieChart_instance.disableAnimation(); - - state.easyPieChartEvent.timer = setTimeout(function() { - state.easyPieChartEvent.timer = null; - state.easyPieChart_instance.update(state.easyPieChartEvent.pcent); - }, NETDATA.options.current.charts_selection_animation_delay); - } - - return true; - }; - - NETDATA.easypiechartChartUpdate = function(state, data) { - var value, max, pcent; - - if(NETDATA.globalPanAndZoom.isActive() === true || state.isAutoRefreshable() === false) { - value = null; - max = 0; - pcent = 0; - } - else { - value = data.result[0]; - max = (state.easyPieChartMax === null)?data.max:state.easyPieChartMax; - pcent = NETDATA.percentFromValueMax(value, max); - } - - state.easyPieChartLabel.innerHTML = state.legendFormatValue(value); - state.easyPieChart_instance.update(pcent); - return true; - }; - - NETDATA.easypiechartChartCreate = function(state, data) { - var self = $(state.element); - var chart = $(state.element_chart); - - var value = data.result[0]; - var max = self.data('easypiechart-max-value') || null; - var adjust = self.data('easypiechart-adjust') || null; - - if(max === null) { - max = data.max; - state.easyPieChartMax = null; - } - else - state.easyPieChartMax = max; - - var pcent = NETDATA.percentFromValueMax(value, max); - - chart.data('data-percent', pcent); - - var size; - switch(adjust) { - case 'width': size = state.chartHeight(); break; - case 'min': size = Math.min(state.chartWidth(), state.chartHeight()); break; - case 'max': size = Math.max(state.chartWidth(), state.chartHeight()); break; - case 'height': - default: size = state.chartWidth(); break; - } - state.element.style.width = size + 'px'; - state.element.style.height = size + 'px'; - - var stroke = Math.floor(size / 22); - if(stroke < 3) stroke = 2; - - var valuefontsize = Math.floor((size * 2 / 3) / 5); - var valuetop = Math.round((size - valuefontsize - (size / 40)) / 2); - state.easyPieChartLabel = document.createElement('span'); - state.easyPieChartLabel.className = 'easyPieChartLabel'; - state.easyPieChartLabel.innerHTML = state.legendFormatValue(value); - state.easyPieChartLabel.style.fontSize = valuefontsize + 'px'; - state.easyPieChartLabel.style.top = valuetop.toString() + 'px'; - state.element_chart.appendChild(state.easyPieChartLabel); - - var titlefontsize = Math.round(valuefontsize * 1.6 / 3); - var titletop = Math.round(valuetop - (titlefontsize * 2) - (size / 40)); - state.easyPieChartTitle = document.createElement('span'); - state.easyPieChartTitle.className = 'easyPieChartTitle'; - state.easyPieChartTitle.innerHTML = state.title; - state.easyPieChartTitle.style.fontSize = titlefontsize + 'px'; - state.easyPieChartTitle.style.lineHeight = titlefontsize + 'px'; - state.easyPieChartTitle.style.top = titletop.toString() + 'px'; - state.element_chart.appendChild(state.easyPieChartTitle); - - var unitfontsize = Math.round(titlefontsize * 0.9); - var unittop = Math.round(valuetop + (valuefontsize + unitfontsize) + (size / 40)); - state.easyPieChartUnits = document.createElement('span'); - state.easyPieChartUnits.className = 'easyPieChartUnits'; - state.easyPieChartUnits.innerHTML = state.units; - state.easyPieChartUnits.style.fontSize = unitfontsize + 'px'; - state.easyPieChartUnits.style.top = unittop.toString() + 'px'; - state.element_chart.appendChild(state.easyPieChartUnits); - - chart.easyPieChart({ - barColor: self.data('easypiechart-barcolor') || state.chartColors()[0], //'#ef1e25', - trackColor: self.data('easypiechart-trackcolor') || NETDATA.themes.current.easypiechart_track, - scaleColor: self.data('easypiechart-scalecolor') || NETDATA.themes.current.easypiechart_scale, - scaleLength: self.data('easypiechart-scalelength') || 5, - lineCap: self.data('easypiechart-linecap') || 'round', - lineWidth: self.data('easypiechart-linewidth') || stroke, - trackWidth: self.data('easypiechart-trackwidth') || undefined, - size: self.data('easypiechart-size') || size, - rotate: self.data('easypiechart-rotate') || 0, - animate: self.data('easypiechart-rotate') || {duration: 500, enabled: true}, - easing: self.data('easypiechart-easing') || undefined - }); - - // when we just re-create the chart - // do not animate the first update - var animate = true; - if(typeof state.easyPieChart_instance !== 'undefined') - animate = false; - - state.easyPieChart_instance = chart.data('easyPieChart'); - if(animate === false) state.easyPieChart_instance.disableAnimation(); - state.easyPieChart_instance.update(pcent); - if(animate === false) state.easyPieChart_instance.enableAnimation(); - return true; - }; - - // ---------------------------------------------------------------------------------------------------------------- - // gauge.js - - NETDATA.gaugeInitialize = function(callback) { - if(typeof netdataNoGauge === 'undefined' || !netdataNoGauge) { - $.ajax({ - url: NETDATA.gauge_js, - cache: true, - dataType: "script" - }) - .done(function() { - NETDATA.registerChartLibrary('gauge', NETDATA.gauge_js); - }) - .fail(function() { - NETDATA.chartLibraries.gauge.enabled = false; - NETDATA.error(100, NETDATA.gauge_js); - }) - .always(function() { - if(typeof callback === "function") - callback(); - }) - } - else { - NETDATA.chartLibraries.gauge.enabled = false; - if(typeof callback === "function") - callback(); - } - }; - - NETDATA.gaugeAnimation = function(state, status) { - var speed = 32; - - if(typeof status === 'boolean' && status === false) - speed = 1000000000; - else if(typeof status === 'number') - speed = status; - - state.gauge_instance.animationSpeed = speed; - state.___gaugeOld__.speed = speed; - }; - - NETDATA.gaugeSet = function(state, value, min, max) { - if(typeof value !== 'number') value = 0; - if(typeof min !== 'number') min = 0; - if(typeof max !== 'number') max = 0; - if(value > max) max = value; - if(value < min) min = value; - if(min > max) { - var t = min; - min = max; - max = t; - } - else if(min == max) - max = min + 1; - - // gauge.js has an issue if the needle - // is smaller than min or larger than max - // when we set the new values - // the needle will go crazy - - // to prevent it, we always feed it - // with a percentage, so that the needle - // is always between min and max - var pcent = (value - min) * 100 / (max - min); - - // these should never happen - if(pcent < 0) pcent = 0; - if(pcent > 100) pcent = 100; - - state.gauge_instance.set(pcent); - - state.___gaugeOld__.value = value; - state.___gaugeOld__.min = min; - state.___gaugeOld__.max = max; - }; - - NETDATA.gaugeSetLabels = function(state, value, min, max) { - if(state.___gaugeOld__.valueLabel !== value) { - state.___gaugeOld__.valueLabel = value; - state.gaugeChartLabel.innerHTML = state.legendFormatValue(value); - } - if(state.___gaugeOld__.minLabel !== min) { - state.___gaugeOld__.minLabel = min; - state.gaugeChartMin.innerHTML = state.legendFormatValue(min); - } - if(state.___gaugeOld__.maxLabel !== max) { - state.___gaugeOld__.maxLabel = max; - state.gaugeChartMax.innerHTML = state.legendFormatValue(max); - } - }; - - NETDATA.gaugeClearSelection = function(state) { - if(typeof state.gaugeEvent !== 'undefined') { - if(state.gaugeEvent.timer !== null) - clearTimeout(state.gaugeEvent.timer); - - state.gaugeEvent.timer = null; - } - - if(state.isAutoRefreshable() === true && state.data !== null) { - NETDATA.gaugeChartUpdate(state, state.data); - } - else { - NETDATA.gaugeAnimation(state, false); - NETDATA.gaugeSet(state, null, null, null); - NETDATA.gaugeSetLabels(state, null, null, null); - } - - NETDATA.gaugeAnimation(state, true); - return true; - }; - - NETDATA.gaugeSetSelection = function(state, t) { - if(state.timeIsVisible(t) !== true) - return NETDATA.gaugeClearSelection(state); - - var slot = state.calculateRowForTime(t); - if(slot < 0 || slot >= state.data.result.length) - return NETDATA.gaugeClearSelection(state); - - if(typeof state.gaugeEvent === 'undefined') { - state.gaugeEvent = { - timer: null, - value: 0, - min: 0, - max: 0 - }; - } - - var value = state.data.result[state.data.result.length - 1 - slot]; - var max = (state.gaugeMax === null)?state.data.max:state.gaugeMax; - var min = 0; - - state.gaugeEvent.value = value; - state.gaugeEvent.max = max; - state.gaugeEvent.min = min; - NETDATA.gaugeSetLabels(state, value, min, max); - - if(state.gaugeEvent.timer === null) { - NETDATA.gaugeAnimation(state, false); - - state.gaugeEvent.timer = setTimeout(function() { - state.gaugeEvent.timer = null; - NETDATA.gaugeSet(state, state.gaugeEvent.value, state.gaugeEvent.min, state.gaugeEvent.max); - }, NETDATA.options.current.charts_selection_animation_delay); - } - - return true; - }; - - NETDATA.gaugeChartUpdate = function(state, data) { - var value, min, max; - - if(NETDATA.globalPanAndZoom.isActive() === true || state.isAutoRefreshable() === false) { - value = 0; - min = 0; - max = 1; - NETDATA.gaugeSetLabels(state, null, null, null); - } - else { - value = data.result[0]; - min = 0; - max = (state.gaugeMax === null)?data.max:state.gaugeMax; - if(value > max) max = value; - NETDATA.gaugeSetLabels(state, value, min, max); - } - - NETDATA.gaugeSet(state, value, min, max); - return true; - }; - - NETDATA.gaugeChartCreate = function(state, data) { - var self = $(state.element); - // var chart = $(state.element_chart); - - var value = data.result[0]; - var max = self.data('gauge-max-value') || null; - var adjust = self.data('gauge-adjust') || null; - var pointerColor = self.data('gauge-pointer-color') || NETDATA.themes.current.gauge_pointer; - var strokeColor = self.data('gauge-stroke-color') || NETDATA.themes.current.gauge_stroke; - var startColor = self.data('gauge-start-color') || state.chartColors()[0]; - var stopColor = self.data('gauge-stop-color') || void 0; - var generateGradient = self.data('gauge-generate-gradient') || false; - - if(max === null) { - max = data.max; - state.gaugeMax = null; - } - else - state.gaugeMax = max; - - var width = state.chartWidth(), height = state.chartHeight(); //, ratio = 1.5; - //switch(adjust) { - // case 'width': width = height * ratio; break; - // case 'height': - // default: height = width / ratio; break; - //} - //state.element.style.width = width.toString() + 'px'; - //state.element.style.height = height.toString() + 'px'; - - var lum_d = 0.05; - - var options = { - lines: 12, // The number of lines to draw - angle: 0.15, // The length of each line - lineWidth: 0.44, // 0.44 The line thickness - pointer: { - length: 0.8, // 0.9 The radius of the inner circle - strokeWidth: 0.035, // The rotation offset - color: pointerColor // Fill color - }, - colorStart: startColor, // Colors - colorStop: stopColor, // just experiment with them - strokeColor: strokeColor, // to see which ones work best for you - limitMax: true, - generateGradient: (generateGradient === true)?true:false, - gradientType: 0 - }; - - if (generateGradient.constructor === Array) { - // example options: - // data-gauge-generate-gradient="[0, 50, 100]" - // data-gauge-gradient-percent-color-0="#FFFFFF" - // data-gauge-gradient-percent-color-50="#999900" - // data-gauge-gradient-percent-color-100="#000000" - - options.percentColors = new Array(); - var len = generateGradient.length; - while(len--) { - var pcent = generateGradient[len]; - var color = self.data('gauge-gradient-percent-color-' + pcent.toString()) || false; - if(color !== false) { - var a = new Array(); - a[0] = pcent / 100; - a[1] = color; - options.percentColors.unshift(a); - } - } - if(options.percentColors.length === 0) - delete options.percentColors; - } - else if(generateGradient === false && NETDATA.themes.current.gauge_gradient === true) { - options.percentColors = [ - [0.0, NETDATA.colorLuminance(startColor, (lum_d * 10) - (lum_d * 0))], - [0.1, NETDATA.colorLuminance(startColor, (lum_d * 10) - (lum_d * 1))], - [0.2, NETDATA.colorLuminance(startColor, (lum_d * 10) - (lum_d * 2))], - [0.3, NETDATA.colorLuminance(startColor, (lum_d * 10) - (lum_d * 3))], - [0.4, NETDATA.colorLuminance(startColor, (lum_d * 10) - (lum_d * 4))], - [0.5, NETDATA.colorLuminance(startColor, (lum_d * 10) - (lum_d * 5))], - [0.6, NETDATA.colorLuminance(startColor, (lum_d * 10) - (lum_d * 6))], - [0.7, NETDATA.colorLuminance(startColor, (lum_d * 10) - (lum_d * 7))], - [0.8, NETDATA.colorLuminance(startColor, (lum_d * 10) - (lum_d * 8))], - [0.9, NETDATA.colorLuminance(startColor, (lum_d * 10) - (lum_d * 9))], - [1.0, NETDATA.colorLuminance(startColor, 0.0)]]; - } - - state.gauge_canvas = document.createElement('canvas'); - state.gauge_canvas.id = 'gauge-' + state.uuid + '-canvas'; - state.gauge_canvas.className = 'gaugeChart'; - state.gauge_canvas.width = width; - state.gauge_canvas.height = height; - state.element_chart.appendChild(state.gauge_canvas); - - var valuefontsize = Math.floor(height / 6); - var valuetop = Math.round((height - valuefontsize - (height / 6)) / 2); - state.gaugeChartLabel = document.createElement('span'); - state.gaugeChartLabel.className = 'gaugeChartLabel'; - state.gaugeChartLabel.style.fontSize = valuefontsize + 'px'; - state.gaugeChartLabel.style.top = valuetop.toString() + 'px'; - state.element_chart.appendChild(state.gaugeChartLabel); - - var titlefontsize = Math.round(valuefontsize / 2); - var titletop = 0; - state.gaugeChartTitle = document.createElement('span'); - state.gaugeChartTitle.className = 'gaugeChartTitle'; - state.gaugeChartTitle.innerHTML = state.title; - state.gaugeChartTitle.style.fontSize = titlefontsize + 'px'; - state.gaugeChartTitle.style.lineHeight = titlefontsize + 'px'; - state.gaugeChartTitle.style.top = titletop.toString() + 'px'; - state.element_chart.appendChild(state.gaugeChartTitle); - - var unitfontsize = Math.round(titlefontsize * 0.9); - state.gaugeChartUnits = document.createElement('span'); - state.gaugeChartUnits.className = 'gaugeChartUnits'; - state.gaugeChartUnits.innerHTML = state.units; - state.gaugeChartUnits.style.fontSize = unitfontsize + 'px'; - state.element_chart.appendChild(state.gaugeChartUnits); - - state.gaugeChartMin = document.createElement('span'); - state.gaugeChartMin.className = 'gaugeChartMin'; - state.gaugeChartMin.style.fontSize = Math.round(valuefontsize * 0.75).toString() + 'px'; - state.element_chart.appendChild(state.gaugeChartMin); - - state.gaugeChartMax = document.createElement('span'); - state.gaugeChartMax.className = 'gaugeChartMax'; - state.gaugeChartMax.style.fontSize = Math.round(valuefontsize * 0.75).toString() + 'px'; - state.element_chart.appendChild(state.gaugeChartMax); - - // when we just re-create the chart - // do not animate the first update - var animate = true; - if(typeof state.gauge_instance !== 'undefined') - animate = false; - - state.gauge_instance = new Gauge(state.gauge_canvas).setOptions(options); // create sexy gauge! - - state.___gaugeOld__ = { - value: value, - min: 0, - max: max, - valueLabel: null, - minLabel: null, - maxLabel: null - }; - - // we will always feed a percentage - state.gauge_instance.minValue = 0; - state.gauge_instance.maxValue = 100; - - NETDATA.gaugeAnimation(state, animate); - NETDATA.gaugeSet(state, value, 0, max); - NETDATA.gaugeSetLabels(state, value, 0, max); - NETDATA.gaugeAnimation(state, true); - return true; - }; - - // ---------------------------------------------------------------------------------------------------------------- - // Charts Libraries Registration - - NETDATA.chartLibraries = { - "dygraph": { - initialize: NETDATA.dygraphInitialize, - create: NETDATA.dygraphChartCreate, - update: NETDATA.dygraphChartUpdate, - resize: function(state) { - if(typeof state.dygraph_instance.resize === 'function') - state.dygraph_instance.resize(); - }, - setSelection: NETDATA.dygraphSetSelection, - clearSelection: NETDATA.dygraphClearSelection, - toolboxPanAndZoom: NETDATA.dygraphToolboxPanAndZoom, - initialized: false, - enabled: true, - format: function(state) { return 'json'; }, - options: function(state) { return 'ms|flip'; }, - legend: function(state) { - if(this.isSparkline(state) === false) - return 'right-side'; - else - return null; - }, - autoresize: function(state) { return true; }, - max_updates_to_recreate: function(state) { return 5000; }, - track_colors: function(state) { return true; }, - pixels_per_point: function(state) { - if(this.isSparkline(state) === false) - return 3; - else - return 2; - }, - - isSparkline: function(state) { - if(typeof state.dygraph_sparkline === 'undefined') { - var t = $(state.element).data('dygraph-theme'); - if(t === 'sparkline') - state.dygraph_sparkline = true; - else - state.dygraph_sparkline = false; - } - return state.dygraph_sparkline; - } - }, - "sparkline": { - initialize: NETDATA.sparklineInitialize, - create: NETDATA.sparklineChartCreate, - update: NETDATA.sparklineChartUpdate, - resize: null, - setSelection: undefined, // function(state, t) { return true; }, - clearSelection: undefined, // function(state) { return true; }, - toolboxPanAndZoom: null, - initialized: false, - enabled: true, - format: function(state) { return 'array'; }, - options: function(state) { return 'flip|abs'; }, - legend: function(state) { return null; }, - autoresize: function(state) { return false; }, - max_updates_to_recreate: function(state) { return 5000; }, - track_colors: function(state) { return false; }, - pixels_per_point: function(state) { return 3; } - }, - "peity": { - initialize: NETDATA.peityInitialize, - create: NETDATA.peityChartCreate, - update: NETDATA.peityChartUpdate, - resize: null, - setSelection: undefined, // function(state, t) { return true; }, - clearSelection: undefined, // function(state) { return true; }, - toolboxPanAndZoom: null, - initialized: false, - enabled: true, - format: function(state) { return 'ssvcomma'; }, - options: function(state) { return 'null2zero|flip|abs'; }, - legend: function(state) { return null; }, - autoresize: function(state) { return false; }, - max_updates_to_recreate: function(state) { return 5000; }, - track_colors: function(state) { return false; }, - pixels_per_point: function(state) { return 3; } - }, - "morris": { - initialize: NETDATA.morrisInitialize, - create: NETDATA.morrisChartCreate, - update: NETDATA.morrisChartUpdate, - resize: null, - setSelection: undefined, // function(state, t) { return true; }, - clearSelection: undefined, // function(state) { return true; }, - toolboxPanAndZoom: null, - initialized: false, - enabled: true, - format: function(state) { return 'json'; }, - options: function(state) { return 'objectrows|ms'; }, - legend: function(state) { return null; }, - autoresize: function(state) { return false; }, - max_updates_to_recreate: function(state) { return 50; }, - track_colors: function(state) { return false; }, - pixels_per_point: function(state) { return 15; } - }, - "google": { - initialize: NETDATA.googleInitialize, - create: NETDATA.googleChartCreate, - update: NETDATA.googleChartUpdate, - resize: null, - setSelection: undefined, //function(state, t) { return true; }, - clearSelection: undefined, //function(state) { return true; }, - toolboxPanAndZoom: null, - initialized: false, - enabled: true, - format: function(state) { return 'datatable'; }, - options: function(state) { return ''; }, - legend: function(state) { return null; }, - autoresize: function(state) { return false; }, - max_updates_to_recreate: function(state) { return 300; }, - track_colors: function(state) { return false; }, - pixels_per_point: function(state) { return 4; } - }, - "raphael": { - initialize: NETDATA.raphaelInitialize, - create: NETDATA.raphaelChartCreate, - update: NETDATA.raphaelChartUpdate, - resize: null, - setSelection: undefined, // function(state, t) { return true; }, - clearSelection: undefined, // function(state) { return true; }, - toolboxPanAndZoom: null, - initialized: false, - enabled: true, - format: function(state) { return 'json'; }, - options: function(state) { return ''; }, - legend: function(state) { return null; }, - autoresize: function(state) { return false; }, - max_updates_to_recreate: function(state) { return 5000; }, - track_colors: function(state) { return false; }, - pixels_per_point: function(state) { return 3; } - }, - "c3": { - initialize: NETDATA.c3Initialize, - create: NETDATA.c3ChartCreate, - update: NETDATA.c3ChartUpdate, - resize: null, - setSelection: undefined, // function(state, t) { return true; }, - clearSelection: undefined, // function(state) { return true; }, - toolboxPanAndZoom: null, - initialized: false, - enabled: true, - format: function(state) { return 'csvjsonarray'; }, - options: function(state) { return 'milliseconds'; }, - legend: function(state) { return null; }, - autoresize: function(state) { return false; }, - max_updates_to_recreate: function(state) { return 5000; }, - track_colors: function(state) { return false; }, - pixels_per_point: function(state) { return 15; } - }, - "d3": { - initialize: NETDATA.d3Initialize, - create: NETDATA.d3ChartCreate, - update: NETDATA.d3ChartUpdate, - resize: null, - setSelection: undefined, // function(state, t) { return true; }, - clearSelection: undefined, // function(state) { return true; }, - toolboxPanAndZoom: null, - initialized: false, - enabled: true, - format: function(state) { return 'json'; }, - options: function(state) { return ''; }, - legend: function(state) { return null; }, - autoresize: function(state) { return false; }, - max_updates_to_recreate: function(state) { return 5000; }, - track_colors: function(state) { return false; }, - pixels_per_point: function(state) { return 3; } - }, - "easypiechart": { - initialize: NETDATA.easypiechartInitialize, - create: NETDATA.easypiechartChartCreate, - update: NETDATA.easypiechartChartUpdate, - resize: null, - setSelection: NETDATA.easypiechartSetSelection, - clearSelection: NETDATA.easypiechartClearSelection, - toolboxPanAndZoom: null, - initialized: false, - enabled: true, - format: function(state) { return 'array'; }, - options: function(state) { return 'absolute'; }, - legend: function(state) { return null; }, - autoresize: function(state) { return false; }, - max_updates_to_recreate: function(state) { return 5000; }, - track_colors: function(state) { return true; }, - pixels_per_point: function(state) { return 3; }, - aspect_ratio: 100 - }, - "gauge": { - initialize: NETDATA.gaugeInitialize, - create: NETDATA.gaugeChartCreate, - update: NETDATA.gaugeChartUpdate, - resize: null, - setSelection: NETDATA.gaugeSetSelection, - clearSelection: NETDATA.gaugeClearSelection, - toolboxPanAndZoom: null, - initialized: false, - enabled: true, - format: function(state) { return 'array'; }, - options: function(state) { return 'absolute'; }, - legend: function(state) { return null; }, - autoresize: function(state) { return false; }, - max_updates_to_recreate: function(state) { return 5000; }, - track_colors: function(state) { return true; }, - pixels_per_point: function(state) { return 3; }, - aspect_ratio: 70 - } - }; - - NETDATA.registerChartLibrary = function(library, url) { - if(NETDATA.options.debug.libraries === true) - console.log("registering chart library: " + library); - - NETDATA.chartLibraries[library].url = url; - NETDATA.chartLibraries[library].initialized = true; - NETDATA.chartLibraries[library].enabled = true; - }; - - // ---------------------------------------------------------------------------------------------------------------- - // Load required JS libraries and CSS - - NETDATA.requiredJs = [ - { - url: NETDATA.serverDefault + 'lib/bootstrap.min.js', - isAlreadyLoaded: function() { - // check if bootstrap is loaded - if(typeof $().emulateTransitionEnd == 'function') - return true; - else { - if(typeof netdataNoBootstrap !== 'undefined' && netdataNoBootstrap) - return true; - else - return false; - } - } - }, - { - url: NETDATA.serverDefault + 'lib/jquery.nanoscroller.min.js', - isAlreadyLoaded: function() { return false; } - }, - { - url: NETDATA.serverDefault + 'lib/bootstrap-toggle.min.js', - isAlreadyLoaded: function() { return false; } - } - ]; - - NETDATA.requiredCSS = [ - { - url: NETDATA.themes.current.bootstrap_css, - isAlreadyLoaded: function() { - if(typeof netdataNoBootstrap !== 'undefined' && netdataNoBootstrap) - return true; - else - return false; - } - }, - { - url: NETDATA.serverDefault + 'css/font-awesome.min.css', - isAlreadyLoaded: function() { return false; } - }, - { - url: NETDATA.themes.current.dashboard_css, - isAlreadyLoaded: function() { return false; } - }, - { - url: NETDATA.serverDefault + 'css/bootstrap-toggle.min.css', - isAlreadyLoaded: function() { return false; } - } - ]; - - NETDATA.loadRequiredJs = function(index, callback) { - if(index >= NETDATA.requiredJs.length) { - if(typeof callback === 'function') - callback(); - return; - } - - if(NETDATA.requiredJs[index].isAlreadyLoaded()) { - NETDATA.loadRequiredJs(++index, callback); - return; - } - - if(NETDATA.options.debug.main_loop === true) - console.log('loading ' + NETDATA.requiredJs[index].url); - - $.ajax({ - url: NETDATA.requiredJs[index].url, - cache: true, - dataType: "script" - }) - .success(function() { - if(NETDATA.options.debug.main_loop === true) - console.log('loaded ' + NETDATA.requiredJs[index].url); - - NETDATA.loadRequiredJs(++index, callback); - }) - .fail(function() { - alert('Cannot load required JS library: ' + NETDATA.requiredJs[index].url); - }) - }; - - NETDATA.loadRequiredCSS = function(index) { - if(index >= NETDATA.requiredCSS.length) - return; - - if(NETDATA.requiredCSS[index].isAlreadyLoaded()) { - NETDATA.loadRequiredCSS(++index); - return; - } - - if(NETDATA.options.debug.main_loop === true) - console.log('loading ' + NETDATA.requiredCSS[index].url); - - NETDATA._loadCSS(NETDATA.requiredCSS[index].url); - NETDATA.loadRequiredCSS(++index); - }; - - - // ---------------------------------------------------------------------------------------------------------------- - // Registry of netdata hosts - - NETDATA.registry = { - server: null, // the netdata registry server - person_guid: null, // the unique ID of this browser / user - machine_guid: null, // the unique ID the netdata server that served dashboard.js - hostname: null, // the hostname of the netdata server that served dashboard.js - urls: null, // the user's other URLs - urls_array: null, // the user's other URLs in an array - - parsePersonUrls: function(person_urls) { - // console.log(person_urls); - - if(person_urls) { - NETDATA.registry.urls = {}; - NETDATA.registry.urls_array = new Array(); - - var now = new Date().getTime(); - var apu = person_urls; - var i = apu.length; - while(i--) { - if(typeof NETDATA.registry.urls[apu[i][0]] === 'undefined') { - // console.log('adding: ' + apu[i][4] + ', ' + ((now - apu[i][2]) / 1000).toString()); - - var obj = { - guid: apu[i][0], - url: apu[i][1], - last_t: apu[i][2], - accesses: apu[i][3], - name: apu[i][4], - alternate_urls: new Array() - }; - - NETDATA.registry.urls[apu[i][0]] = obj; - NETDATA.registry.urls_array.push(obj); - } - else { - // console.log('appending: ' + apu[i][4] + ', ' + ((now - apu[i][2]) / 1000).toString()); - - var pu = NETDATA.registry.urls[apu[i][0]]; - if(pu.last_t < apu[i][2]) { - pu.url = apu[i][1]; - pu.last_t = apu[i][2]; - pu.name = apu[i][4]; - } - pu.accesses += apu[i][3]; - pu.alternate_urls.push(apu[i][1]); - } - } - } - - if(typeof netdataRegistryCallback === 'function') - netdataRegistryCallback(NETDATA.registry.urls_array); - }, - - init: function() { - if(typeof netdataNoRegistry !== 'undefined' && netdataNoRegistry) - return; - - NETDATA.registry.hello(NETDATA.serverDefault, function(data) { - if(data) { - NETDATA.registry.server = data.registry; - NETDATA.registry.machine_guid = data.machine_guid; - NETDATA.registry.hostname = data.hostname; - - NETDATA.registry.access(10, function (person_urls) { - NETDATA.registry.parsePersonUrls(person_urls); - - }); - } - }); - }, - - hello: function(host, callback) { - // send HELLO to a netdata server: - // 1. verifies the server is reachable - // 2. responds with the registry URL, the machine GUID of this netdata server and its hostname - $.ajax({ - url: host + '/api/v1/registry?action=hello', - async: true, - cache: false, - xhrFields: { withCredentials: true } // required for the cookie - }) - .done(function(data) { - if(typeof data.status !== 'string' || data.status !== 'ok') { - NETDATA.error(408, host + ' response: ' + JSON.stringify(data)); - data = null; - } - - if(typeof callback === 'function') - callback(data); - }) - .fail(function() { - NETDATA.error(407, host); - - if(typeof callback === 'function') - callback(null); - }); - }, - - access: function(max_redirects, callback) { - // send ACCESS to a netdata registry: - // 1. it lets it know we are accessing a netdata server (its machine GUID and its URL) - // 2. it responds with a list of netdata servers we know - // the registry identifies us using a cookie it sets the first time we access it - // the registry may respond with a redirect URL to send us to another registry - $.ajax({ - url: NETDATA.registry.server + '/api/v1/registry?action=access&machine=' + NETDATA.registry.machine_guid + '&name=' + encodeURIComponent(NETDATA.registry.hostname) + '&url=' + encodeURIComponent(NETDATA.serverDefault), // + '&visible_url=' + encodeURIComponent(document.location), - async: true, - cache: false, - xhrFields: { withCredentials: true } // required for the cookie - }) - .done(function(data) { - var redirect = null; - if(typeof data.registry === 'string') - redirect = data.registry; - - if(typeof data.status !== 'string' || data.status !== 'ok') { - NETDATA.error(409, NETDATA.registry.server + ' responded with: ' + JSON.stringify(data)); - data = null; - } - - if(data === null && redirect !== null && max_redirects > 0) { - NETDATA.registry.server = redirect; - NETDATA.registry.access(max_redirects - 1, callback); - } - else { - if(typeof data.person_guid === 'string') - NETDATA.registry.person_guid = data.person_guid; - - if(typeof callback === 'function') - callback(data.urls); - } - }) - .fail(function() { - NETDATA.error(410, NETDATA.registry.server); - - if(typeof callback === 'function') - callback(null); - }); - }, - - delete: function(delete_url, callback) { - // send DELETE to a netdata registry: - $.ajax({ - url: NETDATA.registry.server + '/api/v1/registry?action=delete&machine=' + NETDATA.registry.machine_guid + '&name=' + encodeURIComponent(NETDATA.registry.hostname) + '&url=' + encodeURIComponent(NETDATA.serverDefault) + '&delete_url=' + encodeURIComponent(delete_url), - async: true, - cache: false, - xhrFields: { withCredentials: true } // required for the cookie - }) - .done(function(data) { - if(typeof data.status !== 'string' || data.status !== 'ok') { - NETDATA.error(411, NETDATA.registry.server + ' responded with: ' + JSON.stringify(data)); - data = null; - } - - if(typeof callback === 'function') - callback(data); - }) - .fail(function() { - NETDATA.error(412, NETDATA.registry.server); - - if(typeof callback === 'function') - callback(null); - }); - }, - - switch: function(new_person_guid, callback) { - // impersonate - $.ajax({ - url: NETDATA.registry.server + '/api/v1/registry?action=switch&machine=' + NETDATA.registry.machine_guid + '&name=' + encodeURIComponent(NETDATA.registry.hostname) + '&url=' + encodeURIComponent(NETDATA.serverDefault) + '&to=' + new_person_guid, - async: true, - cache: false, - xhrFields: { withCredentials: true } // required for the cookie - }) - .done(function(data) { - if(typeof data.status !== 'string' || data.status !== 'ok') { - NETDATA.error(413, NETDATA.registry.server + ' responded with: ' + JSON.stringify(data)); - data = null; - } - - if(typeof callback === 'function') - callback(data); - }) - .fail(function() { - NETDATA.error(414, NETDATA.registry.server); - - if(typeof callback === 'function') - callback(null); - }); - } - }; - - // ---------------------------------------------------------------------------------------------------------------- - // Boot it! - - NETDATA.errorReset(); - NETDATA.loadRequiredCSS(0); - - NETDATA._loadjQuery(function() { - NETDATA.loadRequiredJs(0, function() { - if(typeof $().emulateTransitionEnd !== 'function') { - // bootstrap is not available - NETDATA.options.current.show_help = false; - } - - if(typeof netdataDontStart === 'undefined' || !netdataDontStart) { - if(NETDATA.options.debug.main_loop === true) - console.log('starting chart refresh thread'); - - NETDATA.start(); - } - }); - }); - - // window.NETDATA = NETDATA; + sync_selection_delay: 1500, // ms - when you pan or zoom a chart, wait this amount + // of time before setting up synchronized selections + // on hover. + + sync_selection: true, // enable or disable selection sync + + pan_and_zoom_delay: 50, // when panning or zooming, how ofter to update the chart + + sync_pan_and_zoom: true, // enable or disable pan and zoom sync + + pan_and_zoom_data_padding: true, // fetch more data for the master chart when panning or zooming + + update_only_visible: true, // enable or disable visibility management + + parallel_refresher: true, // enable parallel refresh of charts + + concurrent_refreshes: true, // when parallel_refresher is enabled, sync also the charts + + destroy_on_hide: false, // destroy charts when they are not visible + + show_help: netdataShowHelp, // when enabled the charts will show some help + show_help_delay_show_ms: 500, + show_help_delay_hide_ms: 0, + + eliminate_zero_dimensions: true, // do not show dimensions with just zeros + + stop_updates_when_focus_is_lost: true, // boolean - shall we stop auto-refreshes when document does not have user focus + stop_updates_while_resizing: 1000, // ms - time to stop auto-refreshes while resizing the charts + + double_click_speed: 500, // ms - time between clicks / taps to detect double click/tap + + smooth_plot: true, // enable smooth plot, where possible + + charts_selection_animation_delay: 50, // delay to animate charts when syncing selection + + color_fill_opacity_line: 1.0, + color_fill_opacity_area: 0.2, + color_fill_opacity_stacked: 0.8, + + pan_and_zoom_factor: 0.25, // the increment when panning and zooming with the toolbox + pan_and_zoom_factor_multiplier_control: 2.0, + pan_and_zoom_factor_multiplier_shift: 3.0, + pan_and_zoom_factor_multiplier_alt: 4.0, + + abort_ajax_on_scroll: false, + + setOptionCallback: function() { ; } + }, + + debug: { + show_boxes: false, + main_loop: false, + focus: false, + visibility: false, + chart_data_url: false, + chart_errors: false, // FIXME + chart_timing: false, + chart_calls: false, + libraries: false, + dygraph: false + } + }; + + NETDATA.statistics = { + refreshes_total: 0, + refreshes_active: 0, + refreshes_active_max: 0 + }; + + + // ---------------------------------------------------------------------------------------------------------------- + // local storage options + + NETDATA.localStorage = { + default: {}, + current: {}, + callback: {} // only used for resetting back to defaults + }; + + NETDATA.localStorageGet = function(key, def, callback) { + var ret = def; + + if(typeof NETDATA.localStorage.default[key.toString()] === 'undefined') { + NETDATA.localStorage.default[key.toString()] = def; + NETDATA.localStorage.callback[key.toString()] = callback; + } + + if(typeof Storage !== "undefined" && typeof localStorage === 'object') { + try { + // console.log('localStorage: loading "' + key.toString() + '"'); + ret = localStorage.getItem(key.toString()); + // console.log('netdata loaded: ' + key.toString() + ' = ' + ret.toString()); + if(ret === null || ret === 'undefined') { + // console.log('localStorage: cannot load it, saving "' + key.toString() + '" with value "' + JSON.stringify(def) + '"'); + localStorage.setItem(key.toString(), JSON.stringify(def)); + ret = def; + } + else { + // console.log('localStorage: got "' + key.toString() + '" with value "' + ret + '"'); + ret = JSON.parse(ret); + // console.log('localStorage: loaded "' + key.toString() + '" as value ' + ret + ' of type ' + typeof(ret)); + } + } + catch(error) { + console.log('localStorage: failed to read "' + key.toString() + '", using default: "' + def.toString() + '"'); + ret = def; + } + } + + if(typeof ret === 'undefined' || ret === 'undefined') { + console.log('localStorage: LOADED UNDEFINED "' + key.toString() + '" as value ' + ret + ' of type ' + typeof(ret)); + ret = def; + } + + NETDATA.localStorage.current[key.toString()] = ret; + return ret; + }; + + NETDATA.localStorageSet = function(key, value, callback) { + if(typeof value === 'undefined' || value === 'undefined') { + console.log('localStorage: ATTEMPT TO SET UNDEFINED "' + key.toString() + '" as value ' + value + ' of type ' + typeof(value)); + } + + if(typeof NETDATA.localStorage.default[key.toString()] === 'undefined') { + NETDATA.localStorage.default[key.toString()] = value; + NETDATA.localStorage.current[key.toString()] = value; + NETDATA.localStorage.callback[key.toString()] = callback; + } + + if(typeof Storage !== "undefined" && typeof localStorage === 'object') { + // console.log('localStorage: saving "' + key.toString() + '" with value "' + JSON.stringify(value) + '"'); + try { + localStorage.setItem(key.toString(), JSON.stringify(value)); + } + catch(e) { + console.log('localStorage: failed to save "' + key.toString() + '" with value: "' + value.toString() + '"'); + } + } + + NETDATA.localStorage.current[key.toString()] = value; + return value; + }; + + NETDATA.localStorageGetRecursive = function(obj, prefix, callback) { + for(var i in obj) { + if(typeof obj[i] === 'object') { + //console.log('object ' + prefix + '.' + i.toString()); + NETDATA.localStorageGetRecursive(obj[i], prefix + '.' + i.toString(), callback); + continue; + } + + obj[i] = NETDATA.localStorageGet(prefix + '.' + i.toString(), obj[i], callback); + } + }; + + NETDATA.setOption = function(key, value) { + if(key.toString() === 'setOptionCallback') { + if(typeof NETDATA.options.current.setOptionCallback === 'function') { + NETDATA.options.current[key.toString()] = value; + NETDATA.options.current.setOptionCallback(); + } + } + else if(NETDATA.options.current[key.toString()] !== value) { + var name = 'options.' + key.toString(); + + if(typeof NETDATA.localStorage.default[name.toString()] === 'undefined') + console.log('localStorage: setOption() on unsaved option: "' + name.toString() + '", value: ' + value); + + //console.log(NETDATA.localStorage); + //console.log('setOption: setting "' + key.toString() + '" to "' + value + '" of type ' + typeof(value) + ' original type ' + typeof(NETDATA.options.current[key.toString()])); + //console.log(NETDATA.options); + NETDATA.options.current[key.toString()] = NETDATA.localStorageSet(name.toString(), value, null); + + if(typeof NETDATA.options.current.setOptionCallback === 'function') + NETDATA.options.current.setOptionCallback(); + } + + return true; + }; + + NETDATA.getOption = function(key) { + return NETDATA.options.current[key.toString()]; + }; + + // read settings from local storage + NETDATA.localStorageGetRecursive(NETDATA.options.current, 'options', null); + + // always start with this option enabled. + NETDATA.setOption('stop_updates_when_focus_is_lost', true); + + NETDATA.resetOptions = function() { + for(var i in NETDATA.localStorage.default) { + var a = i.split('.'); + + if(a[0] === 'options') { + if(a[1] === 'setOptionCallback') continue; + if(typeof NETDATA.localStorage.default[i] === 'undefined') continue; + if(NETDATA.options.current[i] === NETDATA.localStorage.default[i]) continue; + + NETDATA.setOption(a[1], NETDATA.localStorage.default[i]); + } + else if(a[0] === 'chart_heights') { + if(typeof NETDATA.localStorage.callback[i] === 'function' && typeof NETDATA.localStorage.default[i] !== 'undefined') { + NETDATA.localStorage.callback[i](NETDATA.localStorage.default[i]); + } + } + } + } + + // ---------------------------------------------------------------------------------------------------------------- + + if(NETDATA.options.debug.main_loop === true) + console.log('welcome to NETDATA'); + + NETDATA.onresize = function() { + NETDATA.options.last_resized = new Date().getTime(); + NETDATA.onscroll(); + }; + + NETDATA.onscroll = function() { + // console.log('onscroll'); + + NETDATA.options.last_page_scroll = new Date().getTime(); + NETDATA.options.auto_refresher_stop_until = 0; + + if(NETDATA.options.targets === null) return; + + // when the user scrolls he sees that we have + // hidden all the not-visible charts + // using this little function we try to switch + // the charts back to visible quickly + var targets = NETDATA.options.targets; + var len = targets.length; + if(NETDATA.options.abort_ajax_on_scroll === true) { + while (len--) { + if (targets[len]._updating === true) { + if (typeof targets[len].xhr !== 'undefined') { + targets[len].xhr.abort(); + targets[len].running = false; + targets[len]._updating = false; + } + targets[len].isVisible(); + } + } + } + else { + while (len--) + targets[len].isVisible(); + } + }; + + window.onresize = NETDATA.onresize; + window.onscroll = NETDATA.onscroll; + + // ---------------------------------------------------------------------------------------------------------------- + // Error Handling + + NETDATA.errorCodes = { + 100: { message: "Cannot load chart library", alert: true }, + 101: { message: "Cannot load jQuery", alert: true }, + 402: { message: "Chart library not found", alert: false }, + 403: { message: "Chart library not enabled/is failed", alert: false }, + 404: { message: "Chart not found", alert: false }, + 405: { message: "Cannot download charts index from server", alert: true }, + 406: { message: "Invalid charts index downloaded from server", alert: true }, + 407: { message: "Cannot HELLO netdata server", alert: false }, + 408: { message: "Netdata servers sent invalid response to HELLO", alert: false }, + 409: { message: "Cannot ACCESS netdata registry", alert: false }, + 410: { message: "Netdata registry ACCESS failed", alert: false }, + 411: { message: "Netdata registry server send invalid response to DELETE ", alert: false }, + 412: { message: "Netdata registry DELETE failed", alert: false }, + 413: { message: "Netdata registry server send invalid response to SWITCH ", alert: false }, + 414: { message: "Netdata registry SWITCH failed", alert: false }, + 415: { message: "Netdata alarms download failed", alert: false }, + 416: { message: "Netdata alarms log download failed", alert: false } + }; + NETDATA.errorLast = { + code: 0, + message: "", + datetime: 0 + }; + + NETDATA.error = function(code, msg) { + NETDATA.errorLast.code = code; + NETDATA.errorLast.message = msg; + NETDATA.errorLast.datetime = new Date().getTime(); + + console.log("ERROR " + code + ": " + NETDATA.errorCodes[code].message + ": " + msg); + + var ret = true; + if(typeof netdataErrorCallback === 'function') { + ret = netdataErrorCallback('system', code, msg); + } + + if(ret && NETDATA.errorCodes[code].alert) + alert("ERROR " + code + ": " + NETDATA.errorCodes[code].message + ": " + msg); + }; + + NETDATA.errorReset = function() { + NETDATA.errorLast.code = 0; + NETDATA.errorLast.message = "You are doing fine!"; + NETDATA.errorLast.datetime = 0; + }; + + // ---------------------------------------------------------------------------------------------------------------- + // Chart Registry + + // When multiple charts need the same chart, we avoid downloading it + // multiple times (and having it in browser memory multiple time) + // by using this registry. + + // Every time we download a chart definition, we save it here with .add() + // Then we try to get it back with .get(). If that fails, we download it. + + NETDATA.chartRegistry = { + charts: {}, + + fixid: function(id) { + return id.replace(/:/g, "_").replace(/\//g, "_"); + }, + + add: function(host, id, data) { + host = this.fixid(host); + id = this.fixid(id); + + if(typeof this.charts[host] === 'undefined') + this.charts[host] = {}; + + //console.log('added ' + host + '/' + id); + this.charts[host][id] = data; + }, + + get: function(host, id) { + host = this.fixid(host); + id = this.fixid(id); + + if(typeof this.charts[host] === 'undefined') + return null; + + if(typeof this.charts[host][id] === 'undefined') + return null; + + //console.log('cached ' + host + '/' + id); + return this.charts[host][id]; + }, + + downloadAll: function(host, callback) { + while(host.slice(-1) === '/') + host = host.substring(0, host.length - 1); + + var self = this; + + $.ajax({ + url: host + '/api/v1/charts', + async: true, + cache: false, + xhrFields: { withCredentials: true } // required for the cookie + }) + .done(function(data) { + if(data !== null) { + var h = NETDATA.chartRegistry.fixid(host); + self.charts[h] = data.charts; + } + else NETDATA.error(406, host + '/api/v1/charts'); + + if(typeof callback === 'function') + callback(data); + }) + .fail(function() { + NETDATA.error(405, host + '/api/v1/charts'); + + if(typeof callback === 'function') + callback(null); + }); + } + }; + + // ---------------------------------------------------------------------------------------------------------------- + // Global Pan and Zoom on charts + + // Using this structure are synchronize all the charts, so that + // when you pan or zoom one, all others are automatically refreshed + // to the same timespan. + + NETDATA.globalPanAndZoom = { + seq: 0, // timestamp ms + // every time a chart is panned or zoomed + // we set the timestamp here + // then we use it as a sequence number + // to find if other charts are syncronized + // to this timerange + + master: null, // the master chart (state), to which all others + // are synchronized + + force_before_ms: null, // the timespan to sync all other charts + force_after_ms: null, + + callback: null, + + // set a new master + setMaster: function(state, after, before) { + if(NETDATA.options.current.sync_pan_and_zoom === false) + return; + + if(this.master !== null && this.master !== state) + this.master.resetChart(true, true); + + var now = new Date().getTime(); + this.master = state; + this.seq = now; + this.force_after_ms = after; + this.force_before_ms = before; + NETDATA.options.auto_refresher_stop_until = now + NETDATA.options.current.global_pan_sync_time; + + if(typeof this.callback === 'function') + this.callback(true, after, before); + }, + + // clear the master + clearMaster: function() { + if(this.master !== null) { + var st = this.master; + this.master = null; + st.resetChart(); + } + + this.master = null; + this.seq = 0; + this.force_after_ms = null; + this.force_before_ms = null; + NETDATA.options.auto_refresher_stop_until = 0; + + if(typeof this.callback === 'function') + this.callback(false, 0, 0); + }, + + // is the given state the master of the global + // pan and zoom sync? + isMaster: function(state) { + if(this.master === state) return true; + return false; + }, + + // are we currently have a global pan and zoom sync? + isActive: function() { + if(this.master !== null && this.force_before_ms !== null && this.force_after_ms !== null && this.seq !== 0) return true; + return false; + }, + + // check if a chart, other than the master + // needs to be refreshed, due to the global pan and zoom + shouldBeAutoRefreshed: function(state) { + if(this.master === null || this.seq === 0) + return false; + + //if(state.needsRecreation()) + // return true; + + if(state.tm.pan_and_zoom_seq === this.seq) + return false; + + return true; + } + }; + + // ---------------------------------------------------------------------------------------------------------------- + // dimensions selection + + // FIXME + // move color assignment to dimensions, here + + dimensionStatus = function(parent, label, name_div, value_div, color) { + this.enabled = false; + this.parent = parent; + this.label = label; + this.name_div = null; + this.value_div = null; + this.color = NETDATA.themes.current.foreground; + + if(parent.selected_count > parent.unselected_count) + this.selected = true; + else + this.selected = false; + + this.setOptions(name_div, value_div, color); + }; + + dimensionStatus.prototype.invalidate = function() { + this.name_div = null; + this.value_div = null; + this.enabled = false; + }; + + dimensionStatus.prototype.setOptions = function(name_div, value_div, color) { + this.color = color; + + if(this.name_div != name_div) { + this.name_div = name_div; + this.name_div.title = this.label; + this.name_div.style.color = this.color; + if(this.selected === false) + this.name_div.className = 'netdata-legend-name not-selected'; + else + this.name_div.className = 'netdata-legend-name selected'; + } + + if(this.value_div != value_div) { + this.value_div = value_div; + this.value_div.title = this.label; + this.value_div.style.color = this.color; + if(this.selected === false) + this.value_div.className = 'netdata-legend-value not-selected'; + else + this.value_div.className = 'netdata-legend-value selected'; + } + + this.enabled = true; + this.setHandler(); + }; + + dimensionStatus.prototype.setHandler = function() { + if(this.enabled === false) return; + + var ds = this; + + // this.name_div.onmousedown = this.value_div.onmousedown = function(e) { + this.name_div.onclick = this.value_div.onclick = function(e) { + e.preventDefault(); + if(ds.isSelected()) { + // this is selected + if(e.shiftKey === true || e.ctrlKey === true) { + // control or shift key is pressed -> unselect this (except is none will remain selected, in which case select all) + ds.unselect(); + + if(ds.parent.countSelected() === 0) + ds.parent.selectAll(); + } + else { + // no key is pressed -> select only this (except if it is the only selected already, in which case select all) + if(ds.parent.countSelected() === 1) { + ds.parent.selectAll(); + } + else { + ds.parent.selectNone(); + ds.select(); + } + } + } + else { + // this is not selected + if(e.shiftKey === true || e.ctrlKey === true) { + // control or shift key is pressed -> select this too + ds.select(); + } + else { + // no key is pressed -> select only this + ds.parent.selectNone(); + ds.select(); + } + } + + ds.parent.state.redrawChart(); + } + }; + + dimensionStatus.prototype.select = function() { + if(this.enabled === false) return; + + this.name_div.className = 'netdata-legend-name selected'; + this.value_div.className = 'netdata-legend-value selected'; + this.selected = true; + }; + + dimensionStatus.prototype.unselect = function() { + if(this.enabled === false) return; + + this.name_div.className = 'netdata-legend-name not-selected'; + this.value_div.className = 'netdata-legend-value hidden'; + this.selected = false; + }; + + dimensionStatus.prototype.isSelected = function() { + return(this.enabled === true && this.selected === true); + }; + + // ---------------------------------------------------------------------------------------------------------------- + + dimensionsVisibility = function(state) { + this.state = state; + this.len = 0; + this.dimensions = {}; + this.selected_count = 0; + this.unselected_count = 0; + }; + + dimensionsVisibility.prototype.dimensionAdd = function(label, name_div, value_div, color) { + if(typeof this.dimensions[label] === 'undefined') { + this.len++; + this.dimensions[label] = new dimensionStatus(this, label, name_div, value_div, color); + } + else + this.dimensions[label].setOptions(name_div, value_div, color); + + return this.dimensions[label]; + }; + + dimensionsVisibility.prototype.dimensionGet = function(label) { + return this.dimensions[label]; + }; + + dimensionsVisibility.prototype.invalidateAll = function() { + for(var d in this.dimensions) + this.dimensions[d].invalidate(); + }; + + dimensionsVisibility.prototype.selectAll = function() { + for(var d in this.dimensions) + this.dimensions[d].select(); + }; + + dimensionsVisibility.prototype.countSelected = function() { + var i = 0; + for(var d in this.dimensions) + if(this.dimensions[d].isSelected()) i++; + + return i; + }; + + dimensionsVisibility.prototype.selectNone = function() { + for(var d in this.dimensions) + this.dimensions[d].unselect(); + }; + + dimensionsVisibility.prototype.selected2BooleanArray = function(array) { + var ret = new Array(); + this.selected_count = 0; + this.unselected_count = 0; + + for(var i = 0, len = array.length; i < len ; i++) { + var ds = this.dimensions[array[i]]; + if(typeof ds === 'undefined') { + // console.log(array[i] + ' is not found'); + ret.push(false); + continue; + } + + if(ds.isSelected()) { + ret.push(true); + this.selected_count++; + } + else { + ret.push(false); + this.unselected_count++; + } + } + + if(this.selected_count === 0 && this.unselected_count !== 0) { + this.selectAll(); + return this.selected2BooleanArray(array); + } + + return ret; + }; + + + // ---------------------------------------------------------------------------------------------------------------- + // global selection sync + + NETDATA.globalSelectionSync = { + state: null, + dont_sync_before: 0, + last_t: 0, + slaves: [], + + stop: function() { + if(this.state !== null) + this.state.globalSelectionSyncStop(); + }, + + delay: function() { + if(this.state !== null) { + this.state.globalSelectionSyncDelay(); + } + } + }; + + // ---------------------------------------------------------------------------------------------------------------- + // Our state object, where all per-chart values are stored + + chartState = function(element) { + var self = $(element); + this.element = element; + + // IMPORTANT: + // all private functions should use 'that', instead of 'this' + var that = this; + + /* error() - private + * show an error instead of the chart + */ + var error = function(msg) { + var ret = true; + + if(typeof netdataErrorCallback === 'function') { + ret = netdataErrorCallback('chart', that.id, msg); + } + + if(ret) { + that.element.innerHTML = that.id + ': ' + msg; + that.enabled = false; + that.current = that.pan; + } + }; + + // GUID - a unique identifier for the chart + this.uuid = NETDATA.guid(); + + // string - the name of chart + this.id = self.data('netdata'); + + // string - the key for localStorage settings + this.settings_id = self.data('id') || null; + + // the user given dimensions of the element + this.width = self.data('width') || NETDATA.chartDefaults.width; + this.height = self.data('height') || NETDATA.chartDefaults.height; + + if(this.settings_id !== null) { + this.height = NETDATA.localStorageGet('chart_heights.' + this.settings_id, this.height, function(height) { + // this is the callback that will be called + // if and when the user resets all localStorage variables + // to their defaults + + resizeChartToHeight(height); + }); + } + + // string - the netdata server URL, without any path + this.host = self.data('host') || NETDATA.chartDefaults.host; + + // make sure the host does not end with / + // all netdata API requests use absolute paths + while(this.host.slice(-1) === '/') + this.host = this.host.substring(0, this.host.length - 1); + + // string - the grouping method requested by the user + this.method = self.data('method') || NETDATA.chartDefaults.method; + + // the time-range requested by the user + this.after = self.data('after') || NETDATA.chartDefaults.after; + this.before = self.data('before') || NETDATA.chartDefaults.before; + + // the pixels per point requested by the user + this.pixels_per_point = self.data('pixels-per-point') || 1; + this.points = self.data('points') || null; + + // the dimensions requested by the user + this.dimensions = self.data('dimensions') || null; + + // the chart library requested by the user + this.library_name = self.data('chart-library') || NETDATA.chartDefaults.library; + + // object - the chart library used + this.library = null; + + // color management + this.colors = null; + this.colors_assigned = {}; + this.colors_available = null; + + // the element already created by the user + this.element_message = null; + + // the element with the chart + this.element_chart = null; + + // the element with the legend of the chart (if created by us) + this.element_legend = null; + this.element_legend_childs = { + hidden: null, + title_date: null, + title_time: null, + title_units: null, + nano: null, + nano_options: null, + series: null + }; + + this.chart_url = null; // string - the url to download chart info + this.chart = null; // object - the chart as downloaded from the server + + this.title = self.data('title') || null; // the title of the chart + this.units = self.data('units') || null; // the units of the chart dimensions + this.append_options = self.data('append-options') || null; // the units of the chart dimensions + + this.running = false; // boolean - true when the chart is being refreshed now + this.validated = false; // boolean - has the chart been validated? + this.enabled = true; // boolean - is the chart enabled for refresh? + this.paused = false; // boolean - is the chart paused for any reason? + this.selected = false; // boolean - is the chart shown a selection? + this.debug = false; // boolean - console.log() debug info about this chart + + this.netdata_first = 0; // milliseconds - the first timestamp in netdata + this.netdata_last = 0; // milliseconds - the last timestamp in netdata + this.requested_after = null; // milliseconds - the timestamp of the request after param + this.requested_before = null; // milliseconds - the timestamp of the request before param + this.requested_padding = null; + this.view_after = 0; + this.view_before = 0; + + this.auto = { + name: 'auto', + autorefresh: true, + force_update_at: 0, // the timestamp to force the update at + force_before_ms: null, + force_after_ms: null + }; + this.pan = { + name: 'pan', + autorefresh: false, + force_update_at: 0, // the timestamp to force the update at + force_before_ms: null, + force_after_ms: null + }; + this.zoom = { + name: 'zoom', + autorefresh: false, + force_update_at: 0, // the timestamp to force the update at + force_before_ms: null, + force_after_ms: null + }; + + // this is a pointer to one of the sub-classes below + // auto, pan, zoom + this.current = this.auto; + + // check the requested library is available + // we don't initialize it here - it will be initialized when + // this chart will be first used + if(typeof NETDATA.chartLibraries[that.library_name] === 'undefined') { + NETDATA.error(402, that.library_name); + error('chart library "' + that.library_name + '" is not found'); + return; + } + else if(NETDATA.chartLibraries[that.library_name].enabled === false) { + NETDATA.error(403, that.library_name); + error('chart library "' + that.library_name + '" is not enabled'); + return; + } + else + that.library = NETDATA.chartLibraries[that.library_name]; + + // milliseconds - the time the last refresh took + this.refresh_dt_ms = 0; + + // if we need to report the rendering speed + // find the element that needs to be updated + var refresh_dt_element_name = self.data('dt-element-name') || null; // string - the element to print refresh_dt_ms + + if(refresh_dt_element_name !== null) + this.refresh_dt_element = document.getElementById(refresh_dt_element_name) || null; + else + this.refresh_dt_element = null; + + this.dimensions_visibility = new dimensionsVisibility(this); + + this._updating = false; + + // ============================================================================================================ + // PRIVATE FUNCTIONS + + var createDOM = function() { + if(that.enabled === false) return; + + if(that.element_message !== null) that.element_message.innerHTML = ''; + if(that.element_legend !== null) that.element_legend.innerHTML = ''; + if(that.element_chart !== null) that.element_chart.innerHTML = ''; + + that.element.innerHTML = ''; + + that.element_message = document.createElement('div'); + that.element_message.className = ' netdata-message hidden'; + that.element.appendChild(that.element_message); + + that.element_chart = document.createElement('div'); + that.element_chart.id = that.library_name + '-' + that.uuid + '-chart'; + that.element.appendChild(that.element_chart); + + if(that.hasLegend() === true) { + that.element.className = "netdata-container-with-legend"; + that.element_chart.className = 'netdata-chart-with-legend-right netdata-' + that.library_name + '-chart-with-legend-right'; + + that.element_legend = document.createElement('div'); + that.element_legend.className = 'netdata-chart-legend netdata-' + that.library_name + '-legend'; + that.element.appendChild(that.element_legend); + } + else { + that.element.className = "netdata-container"; + that.element_chart.className = ' netdata-chart netdata-' + that.library_name + '-chart'; + + that.element_legend = null; + } + that.element_legend_childs.series = null; + + if(typeof(that.width) === 'string') + $(that.element).css('width', that.width); + else if(typeof(that.width) === 'number') + $(that.element).css('width', that.width + 'px'); + + if(typeof(that.library.aspect_ratio) === 'undefined') { + if(typeof(that.height) === 'string') + $(that.element).css('height', that.height); + else if(typeof(that.height) === 'number') + $(that.element).css('height', that.height + 'px'); + } + else { + var w = that.element.offsetWidth; + if(w === null || w === 0) { + // the div is hidden + // this will resize the chart when next viewed + that.tm.last_resized = 0; + } + else + $(that.element).css('height', (that.element.offsetWidth * that.library.aspect_ratio / 100).toString() + 'px'); + } + + if(NETDATA.chartDefaults.min_width !== null) + $(that.element).css('min-width', NETDATA.chartDefaults.min_width); + + that.tm.last_dom_created = new Date().getTime(); + + showLoading(); + }; + + /* init() private + * initialize state variables + * destroy all (possibly) created state elements + * create the basic DOM for a chart + */ + var init = function() { + if(that.enabled === false) return; + + that.paused = false; + that.selected = false; + + that.chart_created = false; // boolean - is the library.create() been called? + that.updates_counter = 0; // numeric - the number of refreshes made so far + that.updates_since_last_unhide = 0; // numeric - the number of refreshes made since the last time the chart was unhidden + that.updates_since_last_creation = 0; // numeric - the number of refreshes made since the last time the chart was created + + that.tm = { + last_initialized: 0, // milliseconds - the timestamp it was last initialized + last_dom_created: 0, // milliseconds - the timestamp its DOM was last created + last_mode_switch: 0, // milliseconds - the timestamp it switched modes + + last_info_downloaded: 0, // milliseconds - the timestamp we downloaded the chart + last_updated: 0, // the timestamp the chart last updated with data + pan_and_zoom_seq: 0, // the sequence number of the global synchronization + // between chart. + // Used with NETDATA.globalPanAndZoom.seq + last_visible_check: 0, // the time we last checked if it is visible + last_resized: 0, // the time the chart was resized + last_hidden: 0, // the time the chart was hidden + last_unhidden: 0, // the time the chart was unhidden + last_autorefreshed: 0 // the time the chart was last refreshed + }; + + that.data = null; // the last data as downloaded from the netdata server + that.data_url = 'invalid://'; // string - the last url used to update the chart + that.data_points = 0; // number - the number of points returned from netdata + that.data_after = 0; // milliseconds - the first timestamp of the data + that.data_before = 0; // milliseconds - the last timestamp of the data + that.data_update_every = 0; // milliseconds - the frequency to update the data + + that.tm.last_initialized = new Date().getTime(); + createDOM(); + + that.setMode('auto'); + }; + + var maxMessageFontSize = function() { + // normally we want a font size, as tall as the element + var h = that.element_message.clientHeight; + + // but give it some air, 20% let's say, or 5 pixels min + var lost = Math.max(h * 0.2, 5); + h -= lost; + + // center the text, vertically + var paddingTop = (lost - 5) / 2; + + // but check the width too + // it should fit 10 characters in it + var w = that.element_message.clientWidth / 10; + if(h > w) { + paddingTop += (h - w) / 2; + h = w; + } + + // and don't make it too huge + // 5% of the screen size is good + if(h > screen.height / 20) { + paddingTop += (h - (screen.height / 20)) / 2; + h = screen.height / 20; + } + + // set it + that.element_message.style.fontSize = h.toString() + 'px'; + that.element_message.style.paddingTop = paddingTop.toString() + 'px'; + }; + + var showMessage = function(msg) { + that.element_message.className = 'netdata-message'; + that.element_message.innerHTML = msg; + that.element_message.style.fontSize = 'x-small'; + that.element_message.style.paddingTop = '0px'; + that.___messageHidden___ = undefined; + }; + + var showMessageIcon = function(icon) { + that.element_message.innerHTML = icon; + that.element_message.className = 'netdata-message icon'; + maxMessageFontSize(); + that.___messageHidden___ = undefined; + }; + + var hideMessage = function() { + if(typeof that.___messageHidden___ === 'undefined') { + that.___messageHidden___ = true; + that.element_message.className = 'netdata-message hidden'; + } + }; + + var showRendering = function() { + var icon; + if(that.chart !== null) { + if(that.chart.chart_type === 'line') + icon = '<i class="fa fa-line-chart"></i>'; + else + icon = '<i class="fa fa-area-chart"></i>'; + } + else + icon = '<i class="fa fa-area-chart"></i>'; + + showMessageIcon(icon + ' netdata'); + }; + + var showLoading = function() { + if(that.chart_created === false) { + showMessageIcon('<i class="fa fa-refresh"></i> netdata'); + return true; + } + return false; + }; + + var isHidden = function() { + if(typeof that.___chartIsHidden___ !== 'undefined') + return true; + + return false; + }; + + // hide the chart, when it is not visible - called from isVisible() + var hideChart = function() { + // hide it, if it is not already hidden + if(isHidden() === true) return; + + if(that.chart_created === true) { + if(NETDATA.options.current.destroy_on_hide === true) { + // we should destroy it + init(); + } + else { + showRendering(); + that.element_chart.style.display = 'none'; + if(that.element_legend !== null) that.element_legend.style.display = 'none'; + that.tm.last_hidden = new Date().getTime(); + + // de-allocate data + // This works, but I not sure there are no corner cases somewhere + // so it is commented - if the user has memory issues he can + // set Destroy on Hide for all charts + // that.data = null; + } + } + + that.___chartIsHidden___ = true; + }; + + // unhide the chart, when it is visible - called from isVisible() + var unhideChart = function() { + if(isHidden() === false) return; + + that.___chartIsHidden___ = undefined; + that.updates_since_last_unhide = 0; + + if(that.chart_created === false) { + // we need to re-initialize it, to show our background + // logo in bootstrap tabs, until the chart loads + init(); + } + else { + that.tm.last_unhidden = new Date().getTime(); + that.element_chart.style.display = ''; + if(that.element_legend !== null) that.element_legend.style.display = ''; + resizeChart(); + hideMessage(); + } + }; + + var canBeRendered = function() { + if(isHidden() === true || that.isVisible(true) === false) + return false; + + return true; + }; + + // https://github.com/petkaantonov/bluebird/wiki/Optimization-killers + var callChartLibraryUpdateSafely = function(data) { + var status; + + if(canBeRendered() === false) + return false; + + if(NETDATA.options.debug.chart_errors === true) + status = that.library.update(that, data); + else { + try { + status = that.library.update(that, data); + } + catch(err) { + status = false; + } + } + + if(status === false) { + error('chart failed to be updated as ' + that.library_name); + return false; + } + + return true; + }; + + // https://github.com/petkaantonov/bluebird/wiki/Optimization-killers + var callChartLibraryCreateSafely = function(data) { + var status; + + if(canBeRendered() === false) + return false; + + if(NETDATA.options.debug.chart_errors === true) + status = that.library.create(that, data); + else { + try { + status = that.library.create(that, data); + } + catch(err) { + status = false; + } + } + + if(status === false) { + error('chart failed to be created as ' + that.library_name); + return false; + } + + that.chart_created = true; + that.updates_since_last_creation = 0; + return true; + }; + + // ---------------------------------------------------------------------------------------------------------------- + // Chart Resize + + // resizeChart() - private + // to be called just before the chart library to make sure that + // a properly sized dom is available + var resizeChart = function() { + if(that.isVisible() === true && that.tm.last_resized < NETDATA.options.last_resized) { + if(that.chart_created === false) return; + + if(that.needsRecreation()) { + init(); + } + else if(typeof that.library.resize === 'function') { + that.library.resize(that); + + if(that.element_legend_childs.nano !== null && that.element_legend_childs.nano_options !== null) + $(that.element_legend_childs.nano).nanoScroller(); + + maxMessageFontSize(); + } + + that.tm.last_resized = new Date().getTime(); + } + }; + + // this is the actual chart resize algorithm + // it will: + // - resize the entire container + // - update the internal states + // - resize the chart as the div changes height + // - update the scrollbar of the legend + var resizeChartToHeight = function(h) { + // console.log(h); + that.element.style.height = h; + + if(that.settings_id !== null) + NETDATA.localStorageSet('chart_heights.' + that.settings_id, h); + + var now = new Date().getTime(); + NETDATA.options.last_page_scroll = now; + NETDATA.options.auto_refresher_stop_until = now + NETDATA.options.current.stop_updates_while_resizing; + + // force a resize + that.tm.last_resized = 0; + resizeChart(); + }; + + this.resizeHandler = function(e) { + e.preventDefault(); + + if(typeof this.event_resize === 'undefined' + || this.event_resize.chart_original_w === 'undefined' + || this.event_resize.chart_original_h === 'undefined') + this.event_resize = { + chart_original_w: this.element.clientWidth, + chart_original_h: this.element.clientHeight, + last: 0 + }; + + if(e.type === 'touchstart') { + this.event_resize.mouse_start_x = e.touches.item(0).pageX; + this.event_resize.mouse_start_y = e.touches.item(0).pageY; + } + else { + this.event_resize.mouse_start_x = e.clientX; + this.event_resize.mouse_start_y = e.clientY; + } + + this.event_resize.chart_start_w = this.element.clientWidth; + this.event_resize.chart_start_h = this.element.clientHeight; + this.event_resize.chart_last_w = this.element.clientWidth; + this.event_resize.chart_last_h = this.element.clientHeight; + + var now = new Date().getTime(); + if(now - this.event_resize.last <= NETDATA.options.current.double_click_speed) { + // double click / double tap event + + // the optimal height of the chart + // showing the entire legend + var optimal = this.event_resize.chart_last_h + + this.element_legend_childs.content.scrollHeight + - this.element_legend_childs.content.clientHeight; + + // if we are not optimal, be optimal + if(this.event_resize.chart_last_h != optimal) + resizeChartToHeight(optimal.toString() + 'px'); + + // else if we do not have the original height + // reset to the original height + else if(this.event_resize.chart_last_h != this.event_resize.chart_original_h) + resizeChartToHeight(this.event_resize.chart_original_h.toString() + 'px'); + } + else { + this.event_resize.last = now; + + // process movement event + document.onmousemove = + document.ontouchmove = + this.element_legend_childs.resize_handler.onmousemove = + this.element_legend_childs.resize_handler.ontouchmove = + function(e) { + var y = null; + + switch(e.type) { + case 'mousemove': y = e.clientY; break; + case 'touchmove': y = e.touches.item(e.touches - 1).pageY; break; + } + + if(y !== null) { + var newH = that.event_resize.chart_start_h + y - that.event_resize.mouse_start_y; + + if(newH >= 70 && newH !== that.event_resize.chart_last_h) { + resizeChartToHeight(newH.toString() + 'px'); + that.event_resize.chart_last_h = newH; + } + } + }; + + // process end event + document.onmouseup = + document.ontouchend = + this.element_legend_childs.resize_handler.onmouseup = + this.element_legend_childs.resize_handler.ontouchend = + function(e) { + // remove all the hooks + document.onmouseup = + document.onmousemove = + document.ontouchmove = + document.ontouchend = + that.element_legend_childs.resize_handler.onmousemove = + that.element_legend_childs.resize_handler.ontouchmove = + that.element_legend_childs.resize_handler.onmouseout = + that.element_legend_childs.resize_handler.onmouseup = + that.element_legend_childs.resize_handler.ontouchend = + null; + + // allow auto-refreshes + NETDATA.options.auto_refresher_stop_until = 0; + }; + } + }; + + + var noDataToShow = function() { + showMessageIcon('<i class="fa fa-warning"></i> empty'); + that.legendUpdateDOM(); + that.tm.last_autorefreshed = new Date().getTime(); + // that.data_update_every = 30 * 1000; + //that.element_chart.style.display = 'none'; + //if(that.element_legend !== null) that.element_legend.style.display = 'none'; + //that.___chartIsHidden___ = true; + }; + + // ============================================================================================================ + // PUBLIC FUNCTIONS + + this.error = function(msg) { + error(msg); + }; + + this.setMode = function(m) { + if(this.current !== null && this.current.name === m) return; + + if(m === 'auto') + this.current = this.auto; + else if(m === 'pan') + this.current = this.pan; + else if(m === 'zoom') + this.current = this.zoom; + else + this.current = this.auto; + + this.current.force_update_at = 0; + this.current.force_before_ms = null; + this.current.force_after_ms = null; + + this.tm.last_mode_switch = new Date().getTime(); + }; + + // ---------------------------------------------------------------------------------------------------------------- + // global selection sync + + // prevent to global selection sync for some time + this.globalSelectionSyncDelay = function(ms) { + if(NETDATA.options.current.sync_selection === false) + return; + + if(typeof ms === 'number') + NETDATA.globalSelectionSync.dont_sync_before = new Date().getTime() + ms; + else + NETDATA.globalSelectionSync.dont_sync_before = new Date().getTime() + NETDATA.options.current.sync_selection_delay; + }; + + // can we globally apply selection sync? + this.globalSelectionSyncAbility = function() { + if(NETDATA.options.current.sync_selection === false) + return false; + + if(NETDATA.globalSelectionSync.dont_sync_before > new Date().getTime()) + return false; + + return true; + }; + + this.globalSelectionSyncIsMaster = function() { + if(NETDATA.globalSelectionSync.state === this) + return true; + else + return false; + }; + + // this chart is the master of the global selection sync + this.globalSelectionSyncBeMaster = function() { + // am I the master? + if(this.globalSelectionSyncIsMaster()) { + if(this.debug === true) + this.log('sync: I am the master already.'); + + return; + } + + if(NETDATA.globalSelectionSync.state) { + if(this.debug === true) + this.log('sync: I am not the sync master. Resetting global sync.'); + + this.globalSelectionSyncStop(); + } + + // become the master + if(this.debug === true) + this.log('sync: becoming sync master.'); + + this.selected = true; + NETDATA.globalSelectionSync.state = this; + + // find the all slaves + var targets = NETDATA.options.targets; + var len = targets.length; + while(len--) { + st = targets[len]; + + if(st === this) { + if(this.debug === true) + st.log('sync: not adding me to sync'); + } + else if(st.globalSelectionSyncIsEligible()) { + if(this.debug === true) + st.log('sync: adding to sync as slave'); + + st.globalSelectionSyncBeSlave(); + } + } + + // this.globalSelectionSyncDelay(100); + }; + + // can the chart participate to the global selection sync as a slave? + this.globalSelectionSyncIsEligible = function() { + if(this.enabled === true + && this.library !== null + && typeof this.library.setSelection === 'function' + && this.isVisible() === true + && this.chart_created === true) + return true; + + return false; + }; + + // this chart becomes a slave of the global selection sync + this.globalSelectionSyncBeSlave = function() { + if(NETDATA.globalSelectionSync.state !== this) + NETDATA.globalSelectionSync.slaves.push(this); + }; + + // sync all the visible charts to the given time + // this is to be called from the chart libraries + this.globalSelectionSync = function(t) { + if(this.globalSelectionSyncAbility() === false) { + if(this.debug === true) + this.log('sync: cannot sync (yet?).'); + + return; + } + + if(this.globalSelectionSyncIsMaster() === false) { + if(this.debug === true) + this.log('sync: trying to be sync master.'); + + this.globalSelectionSyncBeMaster(); + + if(this.globalSelectionSyncAbility() === false) { + if(this.debug === true) + this.log('sync: cannot sync (yet?).'); + + return; + } + } + + NETDATA.globalSelectionSync.last_t = t; + $.each(NETDATA.globalSelectionSync.slaves, function(i, st) { + st.setSelection(t); + }); + }; + + // stop syncing all charts to the given time + this.globalSelectionSyncStop = function() { + if(NETDATA.globalSelectionSync.slaves.length) { + if(this.debug === true) + this.log('sync: cleaning up...'); + + $.each(NETDATA.globalSelectionSync.slaves, function(i, st) { + if(st === that) { + if(that.debug === true) + st.log('sync: not adding me to sync stop'); + } + else { + if(that.debug === true) + st.log('sync: removed slave from sync'); + + st.clearSelection(); + } + }); + + NETDATA.globalSelectionSync.last_t = 0; + NETDATA.globalSelectionSync.slaves = []; + NETDATA.globalSelectionSync.state = null; + } + + this.clearSelection(); + }; + + this.setSelection = function(t) { + if(typeof this.library.setSelection === 'function') { + if(this.library.setSelection(this, t) === true) + this.selected = true; + else + this.selected = false; + } + else this.selected = true; + + if(this.selected === true && this.debug === true) + this.log('selection set to ' + t.toString()); + + return this.selected; + }; + + this.clearSelection = function() { + if(this.selected === true) { + if(typeof this.library.clearSelection === 'function') { + if(this.library.clearSelection(this) === true) + this.selected = false; + else + this.selected = true; + } + else this.selected = false; + + if(this.selected === false && this.debug === true) + this.log('selection cleared'); + + this.legendReset(); + } + + return this.selected; + }; + + // find if a timestamp (ms) is shown in the current chart + this.timeIsVisible = function(t) { + if(t >= this.data_after && t <= this.data_before) + return true; + return false; + }; + + this.calculateRowForTime = function(t) { + if(this.timeIsVisible(t) === false) return -1; + return Math.floor((t - this.data_after) / this.data_update_every); + }; + + // ---------------------------------------------------------------------------------------------------------------- + + // console logging + this.log = function(msg) { + console.log(this.id + ' (' + this.library_name + ' ' + this.uuid + '): ' + msg); + }; + + this.pauseChart = function() { + if(this.paused === false) { + if(this.debug === true) + this.log('pauseChart()'); + + this.paused = true; + } + }; + + this.unpauseChart = function() { + if(this.paused === true) { + if(this.debug === true) + this.log('unpauseChart()'); + + this.paused = false; + } + }; + + this.resetChart = function(dont_clear_master, dont_update) { + if(this.debug === true) + this.log('resetChart(' + dont_clear_master + ', ' + dont_update + ') called'); + + if(typeof dont_clear_master === 'undefined') + dont_clear_master = false; + + if(typeof dont_update === 'undefined') + dont_update = false; + + if(dont_clear_master !== true && NETDATA.globalPanAndZoom.isMaster(this) === true) { + if(this.debug === true) + this.log('resetChart() diverting to clearMaster().'); + // this will call us back with master === true + NETDATA.globalPanAndZoom.clearMaster(); + return; + } + + this.clearSelection(); + + this.tm.pan_and_zoom_seq = 0; + + this.setMode('auto'); + this.current.force_update_at = 0; + this.current.force_before_ms = null; + this.current.force_after_ms = null; + this.tm.last_autorefreshed = 0; + this.paused = false; + this.selected = false; + this.enabled = true; + // this.debug = false; + + // do not update the chart here + // or the chart will flip-flop when it is the master + // of a selection sync and another chart becomes + // the new master + + if(dont_update !== true && this.isVisible() === true) { + this.updateChart(); + } + }; + + this.updateChartPanOrZoom = function(after, before) { + var logme = 'updateChartPanOrZoom(' + after + ', ' + before + '): '; + var ret = true; + + if(this.debug === true) + this.log(logme); + + if(before < after) { + if(this.debug === true) + this.log(logme + 'flipped parameters, rejecting it.'); + + return false; + } + + if(typeof this.fixed_min_duration === 'undefined') + this.fixed_min_duration = Math.round((this.chartWidth() / 30) * this.chart.update_every * 1000); + + var min_duration = this.fixed_min_duration; + var current_duration = Math.round(this.view_before - this.view_after); + + // round the numbers + after = Math.round(after); + before = Math.round(before); + + // align them to update_every + // stretching them further away + after -= after % this.data_update_every; + before += this.data_update_every - (before % this.data_update_every); + + // the final wanted duration + var wanted_duration = before - after; + + // to allow panning, accept just a point below our minimum + if((current_duration - this.data_update_every) < min_duration) + min_duration = current_duration - this.data_update_every; + + // we do it, but we adjust to minimum size and return false + // when the wanted size is below the current and the minimum + // and we zoom + if(wanted_duration < current_duration && wanted_duration < min_duration) { + if(this.debug === true) + this.log(logme + 'too small: min_duration: ' + (min_duration / 1000).toString() + ', wanted: ' + (wanted_duration / 1000).toString()); + + min_duration = this.fixed_min_duration; + + var dt = (min_duration - wanted_duration) / 2; + before += dt; + after -= dt; + wanted_duration = before - after; + ret = false; + } + + var tolerance = this.data_update_every * 2; + var movement = Math.abs(before - this.view_before); + + if(Math.abs(current_duration - wanted_duration) <= tolerance && movement <= tolerance && ret === true) { + if(this.debug === true) + this.log(logme + 'REJECTING UPDATE: current/min duration: ' + (current_duration / 1000).toString() + '/' + (this.fixed_min_duration / 1000).toString() + ', wanted duration: ' + (wanted_duration / 1000).toString() + ', duration diff: ' + (Math.round(Math.abs(current_duration - wanted_duration) / 1000)).toString() + ', movement: ' + (movement / 1000).toString() + ', tolerance: ' + (tolerance / 1000).toString() + ', returning: ' + false); + return false; + } + + if(this.current.name === 'auto') { + this.log(logme + 'caller called me with mode: ' + this.current.name); + this.setMode('pan'); + } + + if(this.debug === true) + this.log(logme + 'ACCEPTING UPDATE: current/min duration: ' + (current_duration / 1000).toString() + '/' + (this.fixed_min_duration / 1000).toString() + ', wanted duration: ' + (wanted_duration / 1000).toString() + ', duration diff: ' + (Math.round(Math.abs(current_duration - wanted_duration) / 1000)).toString() + ', movement: ' + (movement / 1000).toString() + ', tolerance: ' + (tolerance / 1000).toString() + ', returning: ' + ret); + + this.current.force_update_at = new Date().getTime() + NETDATA.options.current.pan_and_zoom_delay; + this.current.force_after_ms = after; + this.current.force_before_ms = before; + NETDATA.globalPanAndZoom.setMaster(this, after, before); + return ret; + }; + + this.legendFormatValue = function(value) { + if(value === null || value === 'undefined') return '-'; + if(typeof value !== 'number') return value; + + var abs = Math.abs(value); + if(abs >= 1000) return (Math.round(value)).toLocaleString(); + if(abs >= 100 ) return (Math.round(value * 10) / 10).toLocaleString(); + if(abs >= 1 ) return (Math.round(value * 100) / 100).toLocaleString(); + if(abs >= 0.1 ) return (Math.round(value * 1000) / 1000).toLocaleString(); + return (Math.round(value * 10000) / 10000).toLocaleString(); + }; + + this.legendSetLabelValue = function(label, value) { + var series = this.element_legend_childs.series[label]; + if(typeof series === 'undefined') return; + if(series.value === null && series.user === null) return; + + // if the value has not changed, skip DOM update + //if(series.last === value) return; + + var s, r; + if(typeof value === 'number') { + var v = Math.abs(value); + s = r = this.legendFormatValue(value); + + if(typeof series.last === 'number') { + if(v > series.last) s += '<i class="fa fa-angle-up" style="width: 8px; text-align: center; overflow: hidden; vertical-align: middle;"></i>'; + else if(v < series.last) s += '<i class="fa fa-angle-down" style="width: 8px; text-align: center; overflow: hidden; vertical-align: middle;"></i>'; + else s += '<i class="fa fa-angle-left" style="width: 8px; text-align: center; overflow: hidden; vertical-align: middle;"></i>'; + } + else s += '<i class="fa fa-angle-right" style="width: 8px; text-align: center; overflow: hidden; vertical-align: middle;"></i>'; + series.last = v; + } + else { + s = r = value; + series.last = value; + } + + if(series.value !== null) series.value.innerHTML = s; + if(series.user !== null) series.user.innerHTML = r; + }; + + this.legendSetDate = function(ms) { + if(typeof ms !== 'number') { + this.legendShowUndefined(); + return; + } + + var d = new Date(ms); + + if(this.element_legend_childs.title_date) + this.element_legend_childs.title_date.innerHTML = d.toLocaleDateString(); + + if(this.element_legend_childs.title_time) + this.element_legend_childs.title_time.innerHTML = d.toLocaleTimeString(); + + if(this.element_legend_childs.title_units) + this.element_legend_childs.title_units.innerHTML = this.units; + }; + + this.legendShowUndefined = function() { + if(this.element_legend_childs.title_date) + this.element_legend_childs.title_date.innerHTML = ' '; + + if(this.element_legend_childs.title_time) + this.element_legend_childs.title_time.innerHTML = this.chart.name; + + if(this.element_legend_childs.title_units) + this.element_legend_childs.title_units.innerHTML = ' '; + + if(this.data && this.element_legend_childs.series !== null) { + var labels = this.data.dimension_names; + var i = labels.length; + while(i--) { + var label = labels[i]; + + if(typeof label === 'undefined') continue; + if(typeof this.element_legend_childs.series[label] === 'undefined') continue; + this.legendSetLabelValue(label, null); + } + } + }; + + this.legendShowLatestValues = function() { + if(this.chart === null) return; + if(this.selected) return; + + if(this.data === null || this.element_legend_childs.series === null) { + this.legendShowUndefined(); + return; + } + + var show_undefined = true; + if(Math.abs(this.netdata_last - this.view_before) <= this.data_update_every) + show_undefined = false; + + if(show_undefined) { + this.legendShowUndefined(); + return; + } + + this.legendSetDate(this.view_before); + + var labels = this.data.dimension_names; + var i = labels.length; + while(i--) { + var label = labels[i]; + + if(typeof label === 'undefined') continue; + if(typeof this.element_legend_childs.series[label] === 'undefined') continue; + + if(show_undefined) + this.legendSetLabelValue(label, null); + else + this.legendSetLabelValue(label, this.data.view_latest_values[i]); + } + }; + + this.legendReset = function() { + this.legendShowLatestValues(); + }; + + // this should be called just ONCE per dimension per chart + this._chartDimensionColor = function(label) { + if(this.colors === null) this.chartColors(); + + if(typeof this.colors_assigned[label] === 'undefined') { + if(this.colors_available.length === 0) { + for(var i = 0, len = NETDATA.themes.current.colors.length; i < len ; i++) + this.colors_available.push(NETDATA.themes.current.colors[i]); + } + + this.colors_assigned[label] = this.colors_available.shift(); + + if(this.debug === true) + this.log('label "' + label + '" got color "' + this.colors_assigned[label]); + } + else { + if(this.debug === true) + this.log('label "' + label + '" already has color "' + this.colors_assigned[label] + '"'); + } + + this.colors.push(this.colors_assigned[label]); + return this.colors_assigned[label]; + }; + + this.chartColors = function() { + if(this.colors !== null) return this.colors; + + this.colors = new Array(); + this.colors_available = new Array(); + var i, len; + + var c = $(this.element).data('colors'); + // this.log('read colors: ' + c); + if(typeof c !== 'undefined' && c !== null && c.length > 0) { + if(typeof c !== 'string') { + this.log('invalid color given: ' + c + ' (give a space separated list of colors)'); + } + else { + c = c.split(' '); + var added = 0; + + while(added < 20) { + for(i = 0, len = c.length; i < len ; i++) { + added++; + this.colors_available.push(c[i]); + // this.log('adding color: ' + c[i]); + } + } + } + } + + // push all the standard colors too + for(i = 0, len = NETDATA.themes.current.colors.length; i < len ; i++) + this.colors_available.push(NETDATA.themes.current.colors[i]); + + return this.colors; + }; + + this.legendUpdateDOM = function() { + var needed = false; + + // check that the legend DOM is up to date for the downloaded dimensions + if(typeof this.element_legend_childs.series !== 'object' || this.element_legend_childs.series === null) { + // this.log('the legend does not have any series - requesting legend update'); + needed = true; + } + else if(this.data === null) { + // this.log('the chart does not have any data - requesting legend update'); + needed = true; + } + else if(typeof this.element_legend_childs.series.labels_key === 'undefined') { + needed = true; + } + else { + var labels = this.data.dimension_names.toString(); + if(labels !== this.element_legend_childs.series.labels_key) { + needed = true; + + if(this.debug === true) + this.log('NEW LABELS: "' + labels + '" NOT EQUAL OLD LABELS: "' + this.element_legend_childs.series.labels_key + '"'); + } + } + + if(needed === false) { + // make sure colors available + this.chartColors(); + + // do we have to update the current values? + // we do this, only when the visible chart is current + if(Math.abs(this.netdata_last - this.view_before) <= this.data_update_every) { + if(this.debug === true) + this.log('chart is in latest position... updating values on legend...'); + + //var labels = this.data.dimension_names; + //var i = labels.length; + //while(i--) + // this.legendSetLabelValue(labels[i], this.data.latest_values[i]); + } + return; + } + if(this.colors === null) { + // this is the first time we update the chart + // let's assign colors to all dimensions + if(this.library.track_colors() === true) + for(var dim in this.chart.dimensions) + this._chartDimensionColor(this.chart.dimensions[dim].name); + } + // we will re-generate the colors for the chart + // based on the selected dimensions + this.colors = null; + + if(this.debug === true) + this.log('updating Legend DOM'); + + // mark all dimensions as invalid + this.dimensions_visibility.invalidateAll(); + + var genLabel = function(state, parent, dim, name, count) { + var color = state._chartDimensionColor(name); + + var user_element = null; + var user_id = self.data('show-value-of-' + dim + '-at') || null; + if(user_id !== null) { + user_element = document.getElementById(user_id) || null; + if(user_element === null) + state.log('Cannot find element with id: ' + user_id); + } + + state.element_legend_childs.series[name] = { + name: document.createElement('span'), + value: document.createElement('span'), + user: user_element, + last: null + }; + + var label = state.element_legend_childs.series[name]; + + // create the dimension visibility tracking for this label + state.dimensions_visibility.dimensionAdd(name, label.name, label.value, color); + + var rgb = NETDATA.colorHex2Rgb(color); + label.name.innerHTML = '<table class="netdata-legend-name-table-' + + state.chart.chart_type + + '" style="background-color: ' + + 'rgba(' + rgb.r + ',' + rgb.g + ',' + rgb.b + ',' + NETDATA.options.current['color_fill_opacity_' + state.chart.chart_type] + ')' + + '"><tr class="netdata-legend-name-tr"><td class="netdata-legend-name-td"></td></tr></table>' + + var text = document.createTextNode(' ' + name); + label.name.appendChild(text); + + if(count > 0) + parent.appendChild(document.createElement('br')); + + parent.appendChild(label.name); + parent.appendChild(label.value); + }; + + var content = document.createElement('div'); + + if(this.hasLegend()) { + this.element_legend_childs = { + content: content, + resize_handler: document.createElement('div'), + toolbox: document.createElement('div'), + toolbox_left: document.createElement('div'), + toolbox_right: document.createElement('div'), + toolbox_reset: document.createElement('div'), + toolbox_zoomin: document.createElement('div'), + toolbox_zoomout: document.createElement('div'), + toolbox_volume: document.createElement('div'), + title_date: document.createElement('span'), + title_time: document.createElement('span'), + title_units: document.createElement('span'), + nano: document.createElement('div'), + nano_options: { + paneClass: 'netdata-legend-series-pane', + sliderClass: 'netdata-legend-series-slider', + contentClass: 'netdata-legend-series-content', + enabledClass: '__enabled', + flashedClass: '__flashed', + activeClass: '__active', + tabIndex: -1, + alwaysVisible: true, + sliderMinHeight: 10 + }, + series: {} + }; + + this.element_legend.innerHTML = ''; + + if(this.library.toolboxPanAndZoom !== null) { + + function get_pan_and_zoom_step(event) { + if (event.ctrlKey) + return NETDATA.options.current.pan_and_zoom_factor * NETDATA.options.current.pan_and_zoom_factor_multiplier_control; + + else if (event.shiftKey) + return NETDATA.options.current.pan_and_zoom_factor * NETDATA.options.current.pan_and_zoom_factor_multiplier_shift; + + else if (event.altKey) + return NETDATA.options.current.pan_and_zoom_factor * NETDATA.options.current.pan_and_zoom_factor_multiplier_alt; + + else + return NETDATA.options.current.pan_and_zoom_factor; + } + + this.element_legend_childs.toolbox.className += ' netdata-legend-toolbox'; + this.element.appendChild(this.element_legend_childs.toolbox); + + this.element_legend_childs.toolbox_left.className += ' netdata-legend-toolbox-button'; + this.element_legend_childs.toolbox_left.innerHTML = '<i class="fa fa-backward"></i>'; + this.element_legend_childs.toolbox.appendChild(this.element_legend_childs.toolbox_left); + this.element_legend_childs.toolbox_left.onclick = function(e) { + e.preventDefault(); + + var step = (that.view_before - that.view_after) * get_pan_and_zoom_step(e); + var before = that.view_before - step; + var after = that.view_after - step; + if(after >= that.netdata_first) + that.library.toolboxPanAndZoom(that, after, before); + }; + if(NETDATA.options.current.show_help === true) + $(this.element_legend_childs.toolbox_left).popover({ + container: "body", + animation: false, + html: true, + trigger: 'hover', + placement: 'bottom', + delay: { show: NETDATA.options.current.show_help_delay_show_ms, hide: NETDATA.options.current.show_help_delay_hide_ms }, + title: 'Pan Left', + content: 'Pan the chart to the left. You can also <b>drag it</b> with your mouse or your finger (on touch devices).<br/><small>Help, can be disabled from the settings.</small>' + }); + + + this.element_legend_childs.toolbox_reset.className += ' netdata-legend-toolbox-button'; + this.element_legend_childs.toolbox_reset.innerHTML = '<i class="fa fa-play"></i>'; + this.element_legend_childs.toolbox.appendChild(this.element_legend_childs.toolbox_reset); + this.element_legend_childs.toolbox_reset.onclick = function(e) { + e.preventDefault(); + NETDATA.resetAllCharts(that); + }; + if(NETDATA.options.current.show_help === true) + $(this.element_legend_childs.toolbox_reset).popover({ + container: "body", + animation: false, + html: true, + trigger: 'hover', + placement: 'bottom', + delay: { show: NETDATA.options.current.show_help_delay_show_ms, hide: NETDATA.options.current.show_help_delay_hide_ms }, + title: 'Chart Reset', + content: 'Reset all the charts to their default auto-refreshing state. You can also <b>double click</b> the chart contents with your mouse or your finger (on touch devices).<br/><small>Help, can be disabled from the settings.</small>' + }); + + this.element_legend_childs.toolbox_right.className += ' netdata-legend-toolbox-button'; + this.element_legend_childs.toolbox_right.innerHTML = '<i class="fa fa-forward"></i>'; + this.element_legend_childs.toolbox.appendChild(this.element_legend_childs.toolbox_right); + this.element_legend_childs.toolbox_right.onclick = function(e) { + e.preventDefault(); + var step = (that.view_before - that.view_after) * get_pan_and_zoom_step(e); + var before = that.view_before + step; + var after = that.view_after + step; + if(before <= that.netdata_last) + that.library.toolboxPanAndZoom(that, after, before); + }; + if(NETDATA.options.current.show_help === true) + $(this.element_legend_childs.toolbox_right).popover({ + container: "body", + animation: false, + html: true, + trigger: 'hover', + placement: 'bottom', + delay: { show: NETDATA.options.current.show_help_delay_show_ms, hide: NETDATA.options.current.show_help_delay_hide_ms }, + title: 'Pan Right', + content: 'Pan the chart to the right. You can also <b>drag it</b> with your mouse or your finger (on touch devices).<br/><small>Help, can be disabled from the settings.</small>' + }); + + + this.element_legend_childs.toolbox_zoomin.className += ' netdata-legend-toolbox-button'; + this.element_legend_childs.toolbox_zoomin.innerHTML = '<i class="fa fa-plus"></i>'; + this.element_legend_childs.toolbox.appendChild(this.element_legend_childs.toolbox_zoomin); + this.element_legend_childs.toolbox_zoomin.onclick = function(e) { + e.preventDefault(); + var dt = ((that.view_before - that.view_after) * (get_pan_and_zoom_step(e) * 0.8) / 2); + var before = that.view_before - dt; + var after = that.view_after + dt; + that.library.toolboxPanAndZoom(that, after, before); + }; + if(NETDATA.options.current.show_help === true) + $(this.element_legend_childs.toolbox_zoomin).popover({ + container: "body", + animation: false, + html: true, + trigger: 'hover', + placement: 'bottom', + delay: { show: NETDATA.options.current.show_help_delay_show_ms, hide: NETDATA.options.current.show_help_delay_hide_ms }, + title: 'Chart Zoom In', + content: 'Zoom in the chart. You can also press SHIFT and select an area of the chart to zoom in. On Chrome and Opera, you can press the SHIFT or the ALT keys and then use the mouse wheel to zoom in or out.<br/><small>Help, can be disabled from the settings.</small>' + }); + + this.element_legend_childs.toolbox_zoomout.className += ' netdata-legend-toolbox-button'; + this.element_legend_childs.toolbox_zoomout.innerHTML = '<i class="fa fa-minus"></i>'; + this.element_legend_childs.toolbox.appendChild(this.element_legend_childs.toolbox_zoomout); + this.element_legend_childs.toolbox_zoomout.onclick = function(e) { + e.preventDefault(); + var dt = (((that.view_before - that.view_after) / (1.0 - (get_pan_and_zoom_step(e) * 0.8)) - (that.view_before - that.view_after)) / 2); + var before = that.view_before + dt; + var after = that.view_after - dt; + + that.library.toolboxPanAndZoom(that, after, before); + }; + if(NETDATA.options.current.show_help === true) + $(this.element_legend_childs.toolbox_zoomout).popover({ + container: "body", + animation: false, + html: true, + trigger: 'hover', + placement: 'bottom', + delay: { show: NETDATA.options.current.show_help_delay_show_ms, hide: NETDATA.options.current.show_help_delay_hide_ms }, + title: 'Chart Zoom Out', + content: 'Zoom out the chart. On Chrome and Opera, you can also press the SHIFT or the ALT keys and then use the mouse wheel to zoom in or out.<br/><small>Help, can be disabled from the settings.</small>' + }); + + //this.element_legend_childs.toolbox_volume.className += ' netdata-legend-toolbox-button'; + //this.element_legend_childs.toolbox_volume.innerHTML = '<i class="fa fa-sort-amount-desc"></i>'; + //this.element_legend_childs.toolbox_volume.title = 'Visible Volume'; + //this.element_legend_childs.toolbox.appendChild(this.element_legend_childs.toolbox_volume); + //this.element_legend_childs.toolbox_volume.onclick = function(e) { + //e.preventDefault(); + //alert('clicked toolbox_volume on ' + that.id); + //} + } + else { + this.element_legend_childs.toolbox = null; + this.element_legend_childs.toolbox_left = null; + this.element_legend_childs.toolbox_reset = null; + this.element_legend_childs.toolbox_right = null; + this.element_legend_childs.toolbox_zoomin = null; + this.element_legend_childs.toolbox_zoomout = null; + this.element_legend_childs.toolbox_volume = null; + } + + this.element_legend_childs.resize_handler.className += " netdata-legend-resize-handler"; + this.element_legend_childs.resize_handler.innerHTML = '<i class="fa fa-chevron-up"></i><i class="fa fa-chevron-down"></i>'; + this.element.appendChild(this.element_legend_childs.resize_handler); + if(NETDATA.options.current.show_help === true) + $(this.element_legend_childs.resize_handler).popover({ + container: "body", + animation: false, + html: true, + trigger: 'hover', + placement: 'bottom', + delay: { show: NETDATA.options.current.show_help_delay_show_ms, hide: NETDATA.options.current.show_help_delay_hide_ms }, + title: 'Chart Resize', + content: 'Drag this point with your mouse or your finger (on touch devices), to resize the chart vertically. You can also <b>double click it</b> or <b>double tap it</b> to reset between 2 states: the default and the one that fits all the values.<br/><small>Help, can be disabled from the settings.</small>' + }); + + // mousedown event + this.element_legend_childs.resize_handler.onmousedown = + function(e) { + that.resizeHandler(e); + }; + + // touchstart event + this.element_legend_childs.resize_handler.addEventListener('touchstart', function(e) { + that.resizeHandler(e); + }, false); + + this.element_legend_childs.title_date.className += " netdata-legend-title-date"; + this.element_legend.appendChild(this.element_legend_childs.title_date); + + this.element_legend.appendChild(document.createElement('br')); + + this.element_legend_childs.title_time.className += " netdata-legend-title-time"; + this.element_legend.appendChild(this.element_legend_childs.title_time); + + this.element_legend.appendChild(document.createElement('br')); + + this.element_legend_childs.title_units.className += " netdata-legend-title-units"; + this.element_legend.appendChild(this.element_legend_childs.title_units); + + this.element_legend.appendChild(document.createElement('br')); + + this.element_legend_childs.nano.className = 'netdata-legend-series'; + this.element_legend.appendChild(this.element_legend_childs.nano); + + content.className = 'netdata-legend-series-content'; + this.element_legend_childs.nano.appendChild(content); + + if(NETDATA.options.current.show_help === true) + $(content).popover({ + container: "body", + animation: false, + html: true, + trigger: 'hover', + placement: 'bottom', + title: 'Chart Legend', + delay: { show: NETDATA.options.current.show_help_delay_show_ms, hide: NETDATA.options.current.show_help_delay_hide_ms }, + content: 'You can click or tap on the values or the labels to select dimentions. By pressing SHIFT or CONTROL, you can enable or disable multiple dimensions.<br/><small>Help, can be disabled from the settings.</small>' + }); + } + else { + this.element_legend_childs = { + content: content, + resize_handler: null, + toolbox: null, + toolbox_left: null, + toolbox_right: null, + toolbox_reset: null, + toolbox_zoomin: null, + toolbox_zoomout: null, + toolbox_volume: null, + title_date: null, + title_time: null, + title_units: null, + nano: null, + nano_options: null, + series: {} + }; + } + + if(this.data) { + this.element_legend_childs.series.labels_key = this.data.dimension_names.toString(); + if(this.debug === true) + this.log('labels from data: "' + this.element_legend_childs.series.labels_key + '"'); + + for(var i = 0, len = this.data.dimension_names.length; i < len ;i++) { + genLabel(this, content, this.data.dimension_ids[i], this.data.dimension_names[i], i); + } + } + else { + var tmp = new Array(); + for(var dim in this.chart.dimensions) { + tmp.push(this.chart.dimensions[dim].name); + genLabel(this, content, dim, this.chart.dimensions[dim].name, i); + } + this.element_legend_childs.series.labels_key = tmp.toString(); + if(this.debug === true) + this.log('labels from chart: "' + this.element_legend_childs.series.labels_key + '"'); + } + + // create a hidden div to be used for hidding + // the original legend of the chart library + var el = document.createElement('div'); + if(this.element_legend !== null) + this.element_legend.appendChild(el); + el.style.display = 'none'; + + this.element_legend_childs.hidden = document.createElement('div'); + el.appendChild(this.element_legend_childs.hidden); + + if(this.element_legend_childs.nano !== null && this.element_legend_childs.nano_options !== null) + $(this.element_legend_childs.nano).nanoScroller(this.element_legend_childs.nano_options); + + this.legendShowLatestValues(); + }; + + this.hasLegend = function() { + if(typeof this.___hasLegendCache___ !== 'undefined') + return this.___hasLegendCache___; + + var leg = false; + if(this.library && this.library.legend(this) === 'right-side') { + var legend = $(this.element).data('legend') || 'yes'; + if(legend === 'yes') leg = true; + } + + this.___hasLegendCache___ = leg; + return leg; + }; + + this.legendWidth = function() { + return (this.hasLegend())?140:0; + }; + + this.legendHeight = function() { + return $(this.element).height(); + }; + + this.chartWidth = function() { + return $(this.element).width() - this.legendWidth(); + }; + + this.chartHeight = function() { + return $(this.element).height(); + }; + + this.chartPixelsPerPoint = function() { + // force an options provided detail + var px = this.pixels_per_point; + + if(this.library && px < this.library.pixels_per_point(this)) + px = this.library.pixels_per_point(this); + + if(px < NETDATA.options.current.pixels_per_point) + px = NETDATA.options.current.pixels_per_point; + + return px; + }; + + this.needsRecreation = function() { + return ( + this.chart_created === true + && this.library + && this.library.autoresize() === false + && this.tm.last_resized < NETDATA.options.last_resized + ); + }; + + this.chartURL = function() { + var after, before, points_multiplier = 1; + if(NETDATA.globalPanAndZoom.isActive() && NETDATA.globalPanAndZoom.isMaster(this) === false) { + this.tm.pan_and_zoom_seq = NETDATA.globalPanAndZoom.seq; + + after = Math.round(NETDATA.globalPanAndZoom.force_after_ms / 1000); + before = Math.round(NETDATA.globalPanAndZoom.force_before_ms / 1000); + this.view_after = after * 1000; + this.view_before = before * 1000; + + this.requested_padding = null; + points_multiplier = 1; + } + else if(this.current.force_before_ms !== null && this.current.force_after_ms !== null) { + this.tm.pan_and_zoom_seq = 0; + + before = Math.round(this.current.force_before_ms / 1000); + after = Math.round(this.current.force_after_ms / 1000); + this.view_after = after * 1000; + this.view_before = before * 1000; + + if(NETDATA.options.current.pan_and_zoom_data_padding === true) { + this.requested_padding = Math.round((before - after) / 2); + after -= this.requested_padding; + before += this.requested_padding; + this.requested_padding *= 1000; + points_multiplier = 2; + } + + this.current.force_before_ms = null; + this.current.force_after_ms = null; + } + else { + this.tm.pan_and_zoom_seq = 0; + + before = this.before; + after = this.after; + this.view_after = after * 1000; + this.view_before = before * 1000; + + this.requested_padding = null; + points_multiplier = 1; + } + + this.requested_after = after * 1000; + this.requested_before = before * 1000; + + this.data_points = this.points || Math.round(this.chartWidth() / this.chartPixelsPerPoint()); + + // build the data URL + this.data_url = this.host + this.chart.data_url; + this.data_url += "&format=" + this.library.format(); + this.data_url += "&points=" + (this.data_points * points_multiplier).toString(); + this.data_url += "&group=" + this.method; + this.data_url += "&options=" + this.library.options(this); + this.data_url += '|jsonwrap'; + + if(NETDATA.options.current.eliminate_zero_dimensions === true) + this.data_url += '|nonzero'; + + if(this.append_options !== null) + this.data_url += '|' + this.append_options.toString(); + + if(after) + this.data_url += "&after=" + after.toString(); + + if(before) + this.data_url += "&before=" + before.toString(); + + if(this.dimensions) + this.data_url += "&dimensions=" + this.dimensions; + + if(NETDATA.options.debug.chart_data_url === true || this.debug === true) + this.log('chartURL(): ' + this.data_url + ' WxH:' + this.chartWidth() + 'x' + this.chartHeight() + ' points: ' + this.data_points + ' library: ' + this.library_name); + }; + + this.redrawChart = function() { + if(this.data !== null) + this.updateChartWithData(this.data); + }; + + this.updateChartWithData = function(data) { + if(this.debug === true) + this.log('updateChartWithData() called.'); + + // this may force the chart to be re-created + resizeChart(); + + this.data = data; + this.updates_counter++; + this.updates_since_last_unhide++; + this.updates_since_last_creation++; + + var started = new Date().getTime(); + + // if the result is JSON, find the latest update-every + this.data_update_every = data.view_update_every * 1000; + this.data_after = data.after * 1000; + this.data_before = data.before * 1000; + this.netdata_first = data.first_entry * 1000; + this.netdata_last = data.last_entry * 1000; + this.data_points = data.points; + data.state = this; + + if(NETDATA.options.current.pan_and_zoom_data_padding === true && this.requested_padding !== null) { + if(this.view_after < this.data_after) { + // console.log('adusting view_after from ' + this.view_after + ' to ' + this.data_after); + this.view_after = this.data_after; + } + + if(this.view_before > this.data_before) { + // console.log('adusting view_before from ' + this.view_before + ' to ' + this.data_before); + this.view_before = this.data_before; + } + } + else { + this.view_after = this.data_after; + this.view_before = this.data_before; + } + + if(this.debug === true) { + this.log('UPDATE No ' + this.updates_counter + ' COMPLETED'); + + if(this.current.force_after_ms) + this.log('STATUS: forced : ' + (this.current.force_after_ms / 1000).toString() + ' - ' + (this.current.force_before_ms / 1000).toString()); + else + this.log('STATUS: forced : unset'); + + this.log('STATUS: requested : ' + (this.requested_after / 1000).toString() + ' - ' + (this.requested_before / 1000).toString()); + this.log('STATUS: downloaded: ' + (this.data_after / 1000).toString() + ' - ' + (this.data_before / 1000).toString()); + this.log('STATUS: rendered : ' + (this.view_after / 1000).toString() + ' - ' + (this.view_before / 1000).toString()); + this.log('STATUS: points : ' + (this.data_points).toString()); + } + + if(this.data_points === 0) { + noDataToShow(); + return; + } + + if(this.updates_since_last_creation >= this.library.max_updates_to_recreate()) { + if(this.debug === true) + this.log('max updates of ' + this.updates_since_last_creation.toString() + ' reached. Forcing re-generation.'); + + this.chart_created = false; + } + + // check and update the legend + this.legendUpdateDOM(); + + if(this.chart_created === true + && typeof this.library.update === 'function') { + + if(this.debug === true) + this.log('updating chart...'); + + if(callChartLibraryUpdateSafely(data) === false) + return; + } + else { + if(this.debug === true) + this.log('creating chart...'); + + if(callChartLibraryCreateSafely(data) === false) + return; + } + hideMessage(); + this.legendShowLatestValues(); + if(this.selected === true) + NETDATA.globalSelectionSync.stop(); + + // update the performance counters + var now = new Date().getTime(); + this.tm.last_updated = now; + + // don't update last_autorefreshed if this chart is + // forced to be updated with global PanAndZoom + if(NETDATA.globalPanAndZoom.isActive()) + this.tm.last_autorefreshed = 0; + else { + if(NETDATA.options.current.parallel_refresher === true && NETDATA.options.current.concurrent_refreshes === true) + this.tm.last_autorefreshed = now - (now % this.data_update_every); + else + this.tm.last_autorefreshed = now; + } + + this.refresh_dt_ms = now - started; + NETDATA.options.auto_refresher_fast_weight += this.refresh_dt_ms; + + if(this.refresh_dt_element !== null) + this.refresh_dt_element.innerHTML = this.refresh_dt_ms.toString(); + }; + + this.updateChart = function(callback) { + if(this.debug === true) + this.log('updateChart() called.'); + + if(this._updating === true) { + if(this.debug === true) + this.log('I am already updating...'); + + if(typeof callback === 'function') callback(); + return false; + } + + // due to late initialization of charts and libraries + // we need to check this too + if(this.enabled === false) { + if(this.debug === true) + this.log('I am not enabled'); + + if(typeof callback === 'function') callback(); + return false; + } + + if(canBeRendered() === false) { + if(typeof callback === 'function') callback(); + return false; + } + + if(this.chart === null) { + this.getChart(function() { that.updateChart(callback); }); + return false; + } + + if(this.library.initialized === false) { + if(this.library.enabled === true) { + this.library.initialize(function() { that.updateChart(callback); }); + return false; + } + else { + error('chart library "' + this.library_name + '" is not available.'); + if(typeof callback === 'function') callback(); + return false; + } + } + + this.clearSelection(); + this.chartURL(); + + if(this.debug === true) + this.log('updating from ' + this.data_url); + + NETDATA.statistics.refreshes_total++; + NETDATA.statistics.refreshes_active++; + + if(NETDATA.statistics.refreshes_active > NETDATA.statistics.refreshes_active_max) + NETDATA.statistics.refreshes_active_max = NETDATA.statistics.refreshes_active; + + this._updating = true; + + this.xhr = $.ajax( { + url: this.data_url, + cache: false, + async: true, + xhrFields: { withCredentials: true } // required for the cookie + }) + .done(function(data) { + that.xhr = undefined; + + if(that.debug === true) + that.log('data received. updating chart.'); + + that.updateChartWithData(data); + }) + .fail(function(msg) { + that.xhr = undefined; + + if(msg.statusText !== 'abort') + error('data download failed for url: ' + that.data_url); + }) + .always(function() { + that.xhr = undefined; + + NETDATA.statistics.refreshes_active--; + that._updating = false; + if(typeof callback === 'function') callback(); + }); + + return true; + }; + + this.isVisible = function(nocache) { + if(typeof nocache === 'undefined') + nocache = false; + + // this.log('last_visible_check: ' + this.tm.last_visible_check + ', last_page_scroll: ' + NETDATA.options.last_page_scroll); + + // caching - we do not evaluate the charts visibility + // if the page has not been scrolled since the last check + if(nocache === false && this.tm.last_visible_check > NETDATA.options.last_page_scroll) + return this.___isVisible___; + + this.tm.last_visible_check = new Date().getTime(); + + var wh = window.innerHeight; + var x = this.element.getBoundingClientRect(); + var ret = 0; + var tolerance = 0; + + if(x.width === 0 || x.height === 0) { + hideChart(); + this.___isVisible___ = false; + return this.___isVisible___; + } + + if(x.top < 0 && -x.top > x.height) { + // the chart is entirely above + ret = -x.top - x.height; + } + else if(x.top > wh) { + // the chart is entirely below + ret = x.top - wh; + } + + if(ret > tolerance) { + // the chart is too far + + hideChart(); + this.___isVisible___ = false; + return this.___isVisible___; + } + else { + // the chart is inside or very close + + unhideChart(); + this.___isVisible___ = true; + return this.___isVisible___; + } + }; + + this.isAutoRefreshable = function() { + return (this.current.autorefresh); + }; + + this.canBeAutoRefreshed = function() { + var now = new Date().getTime(); + + if(this.running === true) { + if(this.debug === true) + this.log('I am already running'); + + return false; + } + + if(this.enabled === false) { + if(this.debug === true) + this.log('I am not enabled'); + + return false; + } + + if(this.library === null || this.library.enabled === false) { + error('charting library "' + this.library_name + '" is not available'); + if(this.debug === true) + this.log('My chart library ' + this.library_name + ' is not available'); + + return false; + } + + if(this.isVisible() === false) { + if(NETDATA.options.debug.visibility === true || this.debug === true) + this.log('I am not visible'); + + return false; + } + + if(this.current.force_update_at !== 0 && this.current.force_update_at < now) { + if(this.debug === true) + this.log('timed force update detected - allowing this update'); + + this.current.force_update_at = 0; + return true; + } + + if(this.isAutoRefreshable() === true) { + // allow the first update, even if the page is not visible + if(this.updates_counter && this.updates_since_last_unhide && NETDATA.options.page_is_visible === false) { + if(NETDATA.options.debug.focus === true || this.debug === true) + this.log('canBeAutoRefreshed(): page does not have focus'); + + return false; + } + + if(this.needsRecreation() === true) { + if(this.debug === true) + this.log('canBeAutoRefreshed(): needs re-creation.'); + + return true; + } + + // options valid only for autoRefresh() + if(NETDATA.options.auto_refresher_stop_until === 0 || NETDATA.options.auto_refresher_stop_until < now) { + if(NETDATA.globalPanAndZoom.isActive()) { + if(NETDATA.globalPanAndZoom.shouldBeAutoRefreshed(this)) { + if(this.debug === true) + this.log('canBeAutoRefreshed(): global panning: I need an update.'); + + return true; + } + else { + if(this.debug === true) + this.log('canBeAutoRefreshed(): global panning: I am already up to date.'); + + return false; + } + } + + if(this.selected === true) { + if(this.debug === true) + this.log('canBeAutoRefreshed(): I have a selection in place.'); + + return false; + } + + if(this.paused === true) { + if(this.debug === true) + this.log('canBeAutoRefreshed(): I am paused.'); + + return false; + } + + if(now - this.tm.last_autorefreshed >= this.data_update_every) { + if(this.debug === true) + this.log('canBeAutoRefreshed(): It is time to update me.'); + + return true; + } + } + } + + return false; + }; + + this.autoRefresh = function(callback) { + if(this.canBeAutoRefreshed() === true && this.running === false) { + var state = this; + + state.running = true; + state.updateChart(function() { + state.running = false; + + if(typeof callback !== 'undefined') + callback(); + }); + } + else { + if(typeof callback !== 'undefined') + callback(); + } + }; + + this._defaultsFromDownloadedChart = function(chart) { + this.chart = chart; + this.chart_url = chart.url; + this.data_update_every = chart.update_every * 1000; + this.data_points = Math.round(this.chartWidth() / this.chartPixelsPerPoint()); + this.tm.last_info_downloaded = new Date().getTime(); + + if(this.title === null) + this.title = chart.title; + + if(this.units === null) + this.units = chart.units; + }; + + // fetch the chart description from the netdata server + this.getChart = function(callback) { + this.chart = NETDATA.chartRegistry.get(this.host, this.id); + if(this.chart) { + this._defaultsFromDownloadedChart(this.chart); + if(typeof callback === 'function') callback(); + } + else { + this.chart_url = "/api/v1/chart?chart=" + this.id; + + if(this.debug === true) + this.log('downloading ' + this.chart_url); + + $.ajax( { + url: this.host + this.chart_url, + cache: false, + async: true, + xhrFields: { withCredentials: true } // required for the cookie + }) + .done(function(chart) { + chart.url = that.chart_url; + that._defaultsFromDownloadedChart(chart); + NETDATA.chartRegistry.add(that.host, that.id, chart); + }) + .fail(function() { + NETDATA.error(404, that.chart_url); + error('chart not found on url "' + that.chart_url + '"'); + }) + .always(function() { + if(typeof callback === 'function') callback(); + }); + } + }; + + // ============================================================================================================ + // INITIALIZATION + + init(); + }; + + NETDATA.resetAllCharts = function(state) { + // first clear the global selection sync + // to make sure no chart is in selected state + state.globalSelectionSyncStop(); + + // there are 2 possibilities here + // a. state is the global Pan and Zoom master + // b. state is not the global Pan and Zoom master + var master = true; + if(NETDATA.globalPanAndZoom.isMaster(state) === false) + master = false; + + // clear the global Pan and Zoom + // this will also refresh the master + // and unblock any charts currently mirroring the master + NETDATA.globalPanAndZoom.clearMaster(); + + // if we were not the master, reset our status too + // this is required because most probably the mouse + // is over this chart, blocking it from auto-refreshing + if(master === false && (state.paused === true || state.selected === true)) + state.resetChart(); + }; + + // get or create a chart state, given a DOM element + NETDATA.chartState = function(element) { + var state = $(element).data('netdata-state-object') || null; + if(state === null) { + state = new chartState(element); + $(element).data('netdata-state-object', state); + } + return state; + }; + + // ---------------------------------------------------------------------------------------------------------------- + // Library functions + + // Load a script without jquery + // This is used to load jquery - after it is loaded, we use jquery + NETDATA._loadjQuery = function(callback) { + if(typeof jQuery === 'undefined') { + if(NETDATA.options.debug.main_loop === true) + console.log('loading ' + NETDATA.jQuery); + + var script = document.createElement('script'); + script.type = 'text/javascript'; + script.async = true; + script.src = NETDATA.jQuery; + + // script.onabort = onError; + script.onerror = function(err, t) { NETDATA.error(101, NETDATA.jQuery); }; + if(typeof callback === "function") + script.onload = callback; + + var s = document.getElementsByTagName('script')[0]; + s.parentNode.insertBefore(script, s); + } + else if(typeof callback === "function") + callback(); + }; + + NETDATA._loadCSS = function(filename) { + // don't use jQuery here + // styles are loaded before jQuery + // to eliminate showing an unstyled page to the user + + var fileref = document.createElement("link"); + fileref.setAttribute("rel", "stylesheet"); + fileref.setAttribute("type", "text/css"); + fileref.setAttribute("href", filename); + + if (typeof fileref !== 'undefined') + document.getElementsByTagName("head")[0].appendChild(fileref); + }; + + NETDATA.colorHex2Rgb = function(hex) { + // Expand shorthand form (e.g. "03F") to full form (e.g. "0033FF") + var shorthandRegex = /^#?([a-f\d])([a-f\d])([a-f\d])$/i; + hex = hex.replace(shorthandRegex, function(m, r, g, b) { + return r + r + g + g + b + b; + }); + + var result = /^#?([a-f\d]{2})([a-f\d]{2})([a-f\d]{2})$/i.exec(hex); + return result ? { + r: parseInt(result[1], 16), + g: parseInt(result[2], 16), + b: parseInt(result[3], 16) + } : null; + }; + + NETDATA.colorLuminance = function(hex, lum) { + // validate hex string + hex = String(hex).replace(/[^0-9a-f]/gi, ''); + if (hex.length < 6) + hex = hex[0]+hex[0]+hex[1]+hex[1]+hex[2]+hex[2]; + + lum = lum || 0; + + // convert to decimal and change luminosity + var rgb = "#", c, i; + for (i = 0; i < 3; i++) { + c = parseInt(hex.substr(i*2,2), 16); + c = Math.round(Math.min(Math.max(0, c + (c * lum)), 255)).toString(16); + rgb += ("00"+c).substr(c.length); + } + + return rgb; + }; + + NETDATA.guid = function() { + function s4() { + return Math.floor((1 + Math.random()) * 0x10000) + .toString(16) + .substring(1); + } + + return s4() + s4() + '-' + s4() + '-' + s4() + '-' + s4() + '-' + s4() + s4() + s4(); + }; + + NETDATA.zeropad = function(x) { + if(x > -10 && x < 10) return '0' + x.toString(); + else return x.toString(); + }; + + // user function to signal us the DOM has been + // updated. + NETDATA.updatedDom = function() { + NETDATA.options.updated_dom = true; + }; + + NETDATA.ready = function(callback) { + NETDATA.options.pauseCallback = callback; + }; + + NETDATA.pause = function(callback) { + if(NETDATA.options.pause === true) + callback(); + else + NETDATA.options.pauseCallback = callback; + }; + + NETDATA.unpause = function() { + NETDATA.options.pauseCallback = null; + NETDATA.options.updated_dom = true; + NETDATA.options.pause = false; + }; + + // ---------------------------------------------------------------------------------------------------------------- + + // this is purely sequencial charts refresher + // it is meant to be autonomous + NETDATA.chartRefresherNoParallel = function(index) { + if(NETDATA.options.debug.mail_loop === true) + console.log('NETDATA.chartRefresherNoParallel(' + index + ')'); + + if(NETDATA.options.updated_dom === true) { + // the dom has been updated + // get the dom parts again + NETDATA.parseDom(NETDATA.chartRefresher); + return; + } + if(index >= NETDATA.options.targets.length) { + if(NETDATA.options.debug.main_loop === true) + console.log('waiting to restart main loop...'); + + NETDATA.options.auto_refresher_fast_weight = 0; + + setTimeout(function() { + NETDATA.chartRefresher(); + }, NETDATA.options.current.idle_between_loops); + } + else { + var state = NETDATA.options.targets[index]; + + if(NETDATA.options.auto_refresher_fast_weight < NETDATA.options.current.fast_render_timeframe) { + if(NETDATA.options.debug.main_loop === true) + console.log('fast rendering...'); + + state.autoRefresh(function() { + NETDATA.chartRefresherNoParallel(++index); + }); + } + else { + if(NETDATA.options.debug.main_loop === true) console.log('waiting for next refresh...'); + NETDATA.options.auto_refresher_fast_weight = 0; + + setTimeout(function() { + state.autoRefresh(function() { + NETDATA.chartRefresherNoParallel(++index); + }); + }, NETDATA.options.current.idle_between_charts); + } + } + }; + + // this is part of the parallel refresher + // its cause is to refresh sequencially all the charts + // that depend on chart library initialization + // it will call the parallel refresher back + // as soon as it sees a chart that its chart library + // is initialized + NETDATA.chartRefresher_uninitialized = function() { + if(NETDATA.options.updated_dom === true) { + // the dom has been updated + // get the dom parts again + NETDATA.parseDom(NETDATA.chartRefresher); + return; + } + + if(NETDATA.options.sequencial.length === 0) + NETDATA.chartRefresher(); + else { + var state = NETDATA.options.sequencial.pop(); + if(state.library.initialized === true) + NETDATA.chartRefresher(); + else + state.autoRefresh(NETDATA.chartRefresher_uninitialized); + } + }; + + NETDATA.chartRefresherWaitTime = function() { + return NETDATA.options.current.idle_parallel_loops; + }; + + // the default refresher + // it will create 2 sets of charts: + // - the ones that can be refreshed in parallel + // - the ones that depend on something else + // the first set will be executed in parallel + // the second will be given to NETDATA.chartRefresher_uninitialized() + NETDATA.chartRefresher = function() { + // console.log('auto-refresher...'); + + if(NETDATA.options.pause === true) { + // console.log('auto-refresher is paused'); + setTimeout(NETDATA.chartRefresher, + NETDATA.chartRefresherWaitTime()); + return; + } + + if(typeof NETDATA.options.pauseCallback === 'function') { + // console.log('auto-refresher is calling pauseCallback'); + NETDATA.options.pause = true; + NETDATA.options.pauseCallback(); + NETDATA.chartRefresher(); + return; + } + + if(NETDATA.options.current.parallel_refresher === false) { + // console.log('auto-refresher is calling chartRefresherNoParallel(0)'); + NETDATA.chartRefresherNoParallel(0); + return; + } + + if(NETDATA.options.updated_dom === true) { + // the dom has been updated + // get the dom parts again + // console.log('auto-refresher is calling parseDom()'); + NETDATA.parseDom(NETDATA.chartRefresher); + return; + } + + var parallel = new Array(); + var targets = NETDATA.options.targets; + var len = targets.length; + var state; + while(len--) { + state = targets[len]; + if(state.isVisible() === false || state.running === true) + continue; + + if(state.library.initialized === false) { + if(state.library.enabled === true) { + state.library.initialize(NETDATA.chartRefresher); + return; + } + else { + state.error('chart library "' + state.library_name + '" is not enabled.'); + } + } + + parallel.unshift(state); + } + + if(parallel.length > 0) { + // console.log('auto-refresher executing in parallel for ' + parallel.length.toString() + ' charts'); + // this will execute the jobs in parallel + $(parallel).each(function() { + this.autoRefresh(); + }) + } + //else { + // console.log('auto-refresher nothing to do'); + //} + + // run the next refresh iteration + setTimeout(NETDATA.chartRefresher, + NETDATA.chartRefresherWaitTime()); + }; + + NETDATA.parseDom = function(callback) { + NETDATA.options.last_page_scroll = new Date().getTime(); + NETDATA.options.updated_dom = false; + + var targets = $('div[data-netdata]'); //.filter(':visible'); + + if(NETDATA.options.debug.main_loop === true) + console.log('DOM updated - there are ' + targets.length + ' charts on page.'); + + NETDATA.options.targets = new Array(); + var len = targets.length; + while(len--) { + // the initialization will take care of sizing + // and the "loading..." message + NETDATA.options.targets.push(NETDATA.chartState(targets[len])); + } + + if(typeof callback === 'function') callback(); + }; + + // this is the main function - where everything starts + NETDATA.start = function() { + // this should be called only once + + NETDATA.options.page_is_visible = true; + + $(window).blur(function() { + if(NETDATA.options.current.stop_updates_when_focus_is_lost === true) { + NETDATA.options.page_is_visible = false; + if(NETDATA.options.debug.focus === true) + console.log('Lost Focus!'); + } + }); + + $(window).focus(function() { + if(NETDATA.options.current.stop_updates_when_focus_is_lost === true) { + NETDATA.options.page_is_visible = true; + if(NETDATA.options.debug.focus === true) + console.log('Focus restored!'); + } + }); + + if(typeof document.hasFocus === 'function' && !document.hasFocus()) { + if(NETDATA.options.current.stop_updates_when_focus_is_lost === true) { + NETDATA.options.page_is_visible = false; + if(NETDATA.options.debug.focus === true) + console.log('Document has no focus!'); + } + } + + // bootstrap tab switching + $('a[data-toggle="tab"]').on('shown.bs.tab', NETDATA.onscroll); + + // bootstrap modal switching + $('.modal').on('hidden.bs.modal', NETDATA.onscroll); + $('.modal').on('shown.bs.modal', NETDATA.onscroll); + + // bootstrap collapse switching + $('.collapse').on('hidden.bs.collapse', NETDATA.onscroll); + $('.collapse').on('shown.bs.collapse', NETDATA.onscroll); + + NETDATA.parseDom(NETDATA.chartRefresher); + + // Alarms initialization + if(netdataShowAlarms === true) + setTimeout(NETDATA.alarms.init, 1000); + + // Registry initialization + setTimeout(NETDATA.registry.init, 1500); + }; + + // ---------------------------------------------------------------------------------------------------------------- + // peity + + NETDATA.peityInitialize = function(callback) { + if(typeof netdataNoPeitys === 'undefined' || !netdataNoPeitys) { + $.ajax({ + url: NETDATA.peity_js, + cache: true, + dataType: "script", + xhrFields: { withCredentials: true } // required for the cookie + }) + .done(function() { + NETDATA.registerChartLibrary('peity', NETDATA.peity_js); + }) + .fail(function() { + NETDATA.chartLibraries.peity.enabled = false; + NETDATA.error(100, NETDATA.peity_js); + }) + .always(function() { + if(typeof callback === "function") + callback(); + }); + } + else { + NETDATA.chartLibraries.peity.enabled = false; + if(typeof callback === "function") + callback(); + } + }; + + NETDATA.peityChartUpdate = function(state, data) { + state.peity_instance.innerHTML = data.result; + + if(state.peity_options.stroke !== state.chartColors()[0]) { + state.peity_options.stroke = state.chartColors()[0]; + if(state.chart.chart_type === 'line') + state.peity_options.fill = NETDATA.themes.current.background; + else + state.peity_options.fill = NETDATA.colorLuminance(state.chartColors()[0], NETDATA.chartDefaults.fill_luminance); + } + + $(state.peity_instance).peity('line', state.peity_options); + return true; + }; + + NETDATA.peityChartCreate = function(state, data) { + state.peity_instance = document.createElement('div'); + state.element_chart.appendChild(state.peity_instance); + + var self = $(state.element); + state.peity_options = { + stroke: NETDATA.themes.current.foreground, + strokeWidth: self.data('peity-strokewidth') || 1, + width: state.chartWidth(), + height: state.chartHeight(), + fill: NETDATA.themes.current.foreground + }; + + NETDATA.peityChartUpdate(state, data); + return true; + }; + + // ---------------------------------------------------------------------------------------------------------------- + // sparkline + + NETDATA.sparklineInitialize = function(callback) { + if(typeof netdataNoSparklines === 'undefined' || !netdataNoSparklines) { + $.ajax({ + url: NETDATA.sparkline_js, + cache: true, + dataType: "script", + xhrFields: { withCredentials: true } // required for the cookie + }) + .done(function() { + NETDATA.registerChartLibrary('sparkline', NETDATA.sparkline_js); + }) + .fail(function() { + NETDATA.chartLibraries.sparkline.enabled = false; + NETDATA.error(100, NETDATA.sparkline_js); + }) + .always(function() { + if(typeof callback === "function") + callback(); + }); + } + else { + NETDATA.chartLibraries.sparkline.enabled = false; + if(typeof callback === "function") + callback(); + } + }; + + NETDATA.sparklineChartUpdate = function(state, data) { + state.sparkline_options.width = state.chartWidth(); + state.sparkline_options.height = state.chartHeight(); + + $(state.element_chart).sparkline(data.result, state.sparkline_options); + return true; + }; + + NETDATA.sparklineChartCreate = function(state, data) { + var self = $(state.element); + var type = self.data('sparkline-type') || 'line'; + var lineColor = self.data('sparkline-linecolor') || state.chartColors()[0]; + var fillColor = self.data('sparkline-fillcolor') || (state.chart.chart_type === 'line')?NETDATA.themes.current.background:NETDATA.colorLuminance(lineColor, NETDATA.chartDefaults.fill_luminance); + var chartRangeMin = self.data('sparkline-chartrangemin') || undefined; + var chartRangeMax = self.data('sparkline-chartrangemax') || undefined; + var composite = self.data('sparkline-composite') || undefined; + var enableTagOptions = self.data('sparkline-enabletagoptions') || undefined; + var tagOptionPrefix = self.data('sparkline-tagoptionprefix') || undefined; + var tagValuesAttribute = self.data('sparkline-tagvaluesattribute') || undefined; + var disableHiddenCheck = self.data('sparkline-disablehiddencheck') || undefined; + var defaultPixelsPerValue = self.data('sparkline-defaultpixelspervalue') || undefined; + var spotColor = self.data('sparkline-spotcolor') || undefined; + var minSpotColor = self.data('sparkline-minspotcolor') || undefined; + var maxSpotColor = self.data('sparkline-maxspotcolor') || undefined; + var spotRadius = self.data('sparkline-spotradius') || undefined; + var valueSpots = self.data('sparkline-valuespots') || undefined; + var highlightSpotColor = self.data('sparkline-highlightspotcolor') || undefined; + var highlightLineColor = self.data('sparkline-highlightlinecolor') || undefined; + var lineWidth = self.data('sparkline-linewidth') || undefined; + var normalRangeMin = self.data('sparkline-normalrangemin') || undefined; + var normalRangeMax = self.data('sparkline-normalrangemax') || undefined; + var drawNormalOnTop = self.data('sparkline-drawnormalontop') || undefined; + var xvalues = self.data('sparkline-xvalues') || undefined; + var chartRangeClip = self.data('sparkline-chartrangeclip') || undefined; + var chartRangeMinX = self.data('sparkline-chartrangeminx') || undefined; + var chartRangeMaxX = self.data('sparkline-chartrangemaxx') || undefined; + var disableInteraction = self.data('sparkline-disableinteraction') || false; + var disableTooltips = self.data('sparkline-disabletooltips') || false; + var disableHighlight = self.data('sparkline-disablehighlight') || false; + var highlightLighten = self.data('sparkline-highlightlighten') || 1.4; + var highlightColor = self.data('sparkline-highlightcolor') || undefined; + var tooltipContainer = self.data('sparkline-tooltipcontainer') || undefined; + var tooltipClassname = self.data('sparkline-tooltipclassname') || undefined; + var tooltipFormat = self.data('sparkline-tooltipformat') || undefined; + var tooltipPrefix = self.data('sparkline-tooltipprefix') || undefined; + var tooltipSuffix = self.data('sparkline-tooltipsuffix') || ' ' + state.units; + var tooltipSkipNull = self.data('sparkline-tooltipskipnull') || true; + var tooltipValueLookups = self.data('sparkline-tooltipvaluelookups') || undefined; + var tooltipFormatFieldlist = self.data('sparkline-tooltipformatfieldlist') || undefined; + var tooltipFormatFieldlistKey = self.data('sparkline-tooltipformatfieldlistkey') || undefined; + var numberFormatter = self.data('sparkline-numberformatter') || function(n){ return n.toFixed(2); }; + var numberDigitGroupSep = self.data('sparkline-numberdigitgroupsep') || undefined; + var numberDecimalMark = self.data('sparkline-numberdecimalmark') || undefined; + var numberDigitGroupCount = self.data('sparkline-numberdigitgroupcount') || undefined; + var animatedZooms = self.data('sparkline-animatedzooms') || false; + + if(spotColor === 'disable') spotColor=''; + if(minSpotColor === 'disable') minSpotColor=''; + if(maxSpotColor === 'disable') maxSpotColor=''; + + state.sparkline_options = { + type: type, + lineColor: lineColor, + fillColor: fillColor, + chartRangeMin: chartRangeMin, + chartRangeMax: chartRangeMax, + composite: composite, + enableTagOptions: enableTagOptions, + tagOptionPrefix: tagOptionPrefix, + tagValuesAttribute: tagValuesAttribute, + disableHiddenCheck: disableHiddenCheck, + defaultPixelsPerValue: defaultPixelsPerValue, + spotColor: spotColor, + minSpotColor: minSpotColor, + maxSpotColor: maxSpotColor, + spotRadius: spotRadius, + valueSpots: valueSpots, + highlightSpotColor: highlightSpotColor, + highlightLineColor: highlightLineColor, + lineWidth: lineWidth, + normalRangeMin: normalRangeMin, + normalRangeMax: normalRangeMax, + drawNormalOnTop: drawNormalOnTop, + xvalues: xvalues, + chartRangeClip: chartRangeClip, + chartRangeMinX: chartRangeMinX, + chartRangeMaxX: chartRangeMaxX, + disableInteraction: disableInteraction, + disableTooltips: disableTooltips, + disableHighlight: disableHighlight, + highlightLighten: highlightLighten, + highlightColor: highlightColor, + tooltipContainer: tooltipContainer, + tooltipClassname: tooltipClassname, + tooltipChartTitle: state.title, + tooltipFormat: tooltipFormat, + tooltipPrefix: tooltipPrefix, + tooltipSuffix: tooltipSuffix, + tooltipSkipNull: tooltipSkipNull, + tooltipValueLookups: tooltipValueLookups, + tooltipFormatFieldlist: tooltipFormatFieldlist, + tooltipFormatFieldlistKey: tooltipFormatFieldlistKey, + numberFormatter: numberFormatter, + numberDigitGroupSep: numberDigitGroupSep, + numberDecimalMark: numberDecimalMark, + numberDigitGroupCount: numberDigitGroupCount, + animatedZooms: animatedZooms, + width: state.chartWidth(), + height: state.chartHeight() + }; + + $(state.element_chart).sparkline(data.result, state.sparkline_options); + return true; + }; + + // ---------------------------------------------------------------------------------------------------------------- + // dygraph + + NETDATA.dygraph = { + smooth: false + }; + + NETDATA.dygraphToolboxPanAndZoom = function(state, after, before) { + if(after < state.netdata_first) + after = state.netdata_first; + + if(before > state.netdata_last) + before = state.netdata_last; + + state.setMode('zoom'); + state.globalSelectionSyncStop(); + state.globalSelectionSyncDelay(); + state.dygraph_user_action = true; + state.dygraph_force_zoom = true; + state.updateChartPanOrZoom(after, before); + NETDATA.globalPanAndZoom.setMaster(state, after, before); + }; + + NETDATA.dygraphSetSelection = function(state, t) { + if(typeof state.dygraph_instance !== 'undefined') { + var r = state.calculateRowForTime(t); + if(r !== -1) + state.dygraph_instance.setSelection(r); + else { + state.dygraph_instance.clearSelection(); + state.legendShowUndefined(); + } + } + + return true; + }; + + NETDATA.dygraphClearSelection = function(state, t) { + if(typeof state.dygraph_instance !== 'undefined') { + state.dygraph_instance.clearSelection(); + } + return true; + }; + + NETDATA.dygraphSmoothInitialize = function(callback) { + $.ajax({ + url: NETDATA.dygraph_smooth_js, + cache: true, + dataType: "script", + xhrFields: { withCredentials: true } // required for the cookie + }) + .done(function() { + NETDATA.dygraph.smooth = true; + smoothPlotter.smoothing = 0.3; + }) + .fail(function() { + NETDATA.dygraph.smooth = false; + }) + .always(function() { + if(typeof callback === "function") + callback(); + }); + }; + + NETDATA.dygraphInitialize = function(callback) { + if(typeof netdataNoDygraphs === 'undefined' || !netdataNoDygraphs) { + $.ajax({ + url: NETDATA.dygraph_js, + cache: true, + dataType: "script", + xhrFields: { withCredentials: true } // required for the cookie + }) + .done(function() { + NETDATA.registerChartLibrary('dygraph', NETDATA.dygraph_js); + }) + .fail(function() { + NETDATA.chartLibraries.dygraph.enabled = false; + NETDATA.error(100, NETDATA.dygraph_js); + }) + .always(function() { + if(NETDATA.chartLibraries.dygraph.enabled === true && NETDATA.options.current.smooth_plot === true) + NETDATA.dygraphSmoothInitialize(callback); + else if(typeof callback === "function") + callback(); + }); + } + else { + NETDATA.chartLibraries.dygraph.enabled = false; + if(typeof callback === "function") + callback(); + } + }; + + NETDATA.dygraphChartUpdate = function(state, data) { + var dygraph = state.dygraph_instance; + + if(typeof dygraph === 'undefined') + return NETDATA.dygraphChartCreate(state, data); + + // when the chart is not visible, and hidden + // if there is a window resize, dygraph detects + // its element size as 0x0. + // this will make it re-appear properly + + if(state.tm.last_unhidden > state.dygraph_last_rendered) + dygraph.resize(); + + var options = { + file: data.result.data, + colors: state.chartColors(), + labels: data.result.labels, + labelsDivWidth: state.chartWidth() - 70, + visibility: state.dimensions_visibility.selected2BooleanArray(state.data.dimension_names) + }; + + if(state.dygraph_force_zoom === true) { + if(NETDATA.options.debug.dygraph === true || state.debug === true) + state.log('dygraphChartUpdate() forced zoom update'); + + options.dateWindow = (state.requested_padding !== null)?[ state.view_after, state.view_before ]:null; + options.valueRange = state.dygraph_options.valueRange; + options.isZoomedIgnoreProgrammaticZoom = true; + state.dygraph_force_zoom = false; + } + else if(state.current.name !== 'auto') { + if(NETDATA.options.debug.dygraph === true || state.debug === true) + state.log('dygraphChartUpdate() loose update'); + + options.valueRange = state.dygraph_options.valueRange; + } + else { + if(NETDATA.options.debug.dygraph === true || state.debug === true) + state.log('dygraphChartUpdate() strict update'); + + options.dateWindow = (state.requested_padding !== null)?[ state.view_after, state.view_before ]:null; + options.valueRange = state.dygraph_options.valueRange; + options.isZoomedIgnoreProgrammaticZoom = true; + } + + if(state.dygraph_smooth_eligible === true) { + if((NETDATA.options.current.smooth_plot === true && state.dygraph_options.plotter !== smoothPlotter) + || (NETDATA.options.current.smooth_plot === false && state.dygraph_options.plotter === smoothPlotter)) { + NETDATA.dygraphChartCreate(state, data); + return; + } + } + + dygraph.updateOptions(options); + + state.dygraph_last_rendered = new Date().getTime(); + return true; + }; + + NETDATA.dygraphChartCreate = function(state, data) { + if(NETDATA.options.debug.dygraph === true || state.debug === true) + state.log('dygraphChartCreate()'); + + var self = $(state.element); + + var chart_type = state.chart.chart_type; + if(chart_type === 'stacked' && data.dimensions === 1) chart_type = 'area'; + chart_type = self.data('dygraph-type') || chart_type; + + var smooth = (chart_type === 'line' && !NETDATA.chartLibraries.dygraph.isSparkline(state))?true:false; + smooth = self.data('dygraph-smooth') || smooth; + + if(NETDATA.dygraph.smooth === false) + smooth = false; + + var strokeWidth = (chart_type === 'stacked')?0.1:((smooth)?1.5:0.7) + var highlightCircleSize = (NETDATA.chartLibraries.dygraph.isSparkline(state))?3:4; + + state.dygraph_options = { + colors: self.data('dygraph-colors') || state.chartColors(), + + // leave a few pixels empty on the right of the chart + rightGap: self.data('dygraph-rightgap') || 5, + showRangeSelector: self.data('dygraph-showrangeselector') || false, + showRoller: self.data('dygraph-showroller') || false, + + title: self.data('dygraph-title') || state.title, + titleHeight: self.data('dygraph-titleheight') || 19, + + legend: self.data('dygraph-legend') || 'always', // 'onmouseover', + labels: data.result.labels, + labelsDiv: self.data('dygraph-labelsdiv') || state.element_legend_childs.hidden, + labelsDivStyles: self.data('dygraph-labelsdivstyles') || { 'fontSize':'1px' }, + labelsDivWidth: self.data('dygraph-labelsdivwidth') || state.chartWidth() - 70, + labelsSeparateLines: self.data('dygraph-labelsseparatelines') || true, + labelsShowZeroValues: self.data('dygraph-labelsshowzerovalues') || true, + labelsKMB: false, + labelsKMG2: false, + showLabelsOnHighlight: self.data('dygraph-showlabelsonhighlight') || true, + hideOverlayOnMouseOut: self.data('dygraph-hideoverlayonmouseout') || true, + + includeZero: self.data('dygraph-includezero') || false, + xRangePad: self.data('dygraph-xrangepad') || 0, + yRangePad: self.data('dygraph-yrangepad') || 1, + + valueRange: self.data('dygraph-valuerange') || null, + + ylabel: state.units, + yLabelWidth: self.data('dygraph-ylabelwidth') || 12, + + // the function to plot the chart + plotter: null, + + // The width of the lines connecting data points. This can be used to increase the contrast or some graphs. + strokeWidth: self.data('dygraph-strokewidth') || strokeWidth, + strokePattern: self.data('dygraph-strokepattern') || undefined, + + // The size of the dot to draw on each point in pixels (see drawPoints). A dot is always drawn when a point is "isolated", + // i.e. there is a missing point on either side of it. This also controls the size of those dots. + drawPoints: self.data('dygraph-drawpoints') || false, + + // Draw points at the edges of gaps in the data. This improves visibility of small data segments or other data irregularities. + drawGapEdgePoints: self.data('dygraph-drawgapedgepoints') || true, + + connectSeparatedPoints: self.data('dygraph-connectseparatedpoints') || false, + pointSize: self.data('dygraph-pointsize') || 1, + + // enabling this makes the chart with little square lines + stepPlot: self.data('dygraph-stepplot') || false, + + // Draw a border around graph lines to make crossing lines more easily distinguishable. Useful for graphs with many lines. + strokeBorderColor: self.data('dygraph-strokebordercolor') || NETDATA.themes.current.background, + strokeBorderWidth: self.data('dygraph-strokeborderwidth') || (chart_type === 'stacked')?0.0:0.0, + + fillGraph: self.data('dygraph-fillgraph') || (chart_type === 'area' || chart_type === 'stacked')?true:false, + fillAlpha: self.data('dygraph-fillalpha') || (chart_type === 'stacked')?NETDATA.options.current.color_fill_opacity_stacked:NETDATA.options.current.color_fill_opacity_area, + stackedGraph: self.data('dygraph-stackedgraph') || (chart_type === 'stacked')?true:false, + stackedGraphNaNFill: self.data('dygraph-stackedgraphnanfill') || 'none', + + drawAxis: self.data('dygraph-drawaxis') || true, + axisLabelFontSize: self.data('dygraph-axislabelfontsize') || 10, + axisLineColor: self.data('dygraph-axislinecolor') || NETDATA.themes.current.axis, + axisLineWidth: self.data('dygraph-axislinewidth') || 0.3, + + drawGrid: self.data('dygraph-drawgrid') || true, + drawXGrid: self.data('dygraph-drawxgrid') || undefined, + drawYGrid: self.data('dygraph-drawygrid') || undefined, + gridLinePattern: self.data('dygraph-gridlinepattern') || null, + gridLineWidth: self.data('dygraph-gridlinewidth') || 0.3, + gridLineColor: self.data('dygraph-gridlinecolor') || NETDATA.themes.current.grid, + + maxNumberWidth: self.data('dygraph-maxnumberwidth') || 8, + sigFigs: self.data('dygraph-sigfigs') || null, + digitsAfterDecimal: self.data('dygraph-digitsafterdecimal') || 2, + valueFormatter: self.data('dygraph-valueformatter') || function(x){ return x.toFixed(2); }, + + highlightCircleSize: self.data('dygraph-highlightcirclesize') || highlightCircleSize, + highlightSeriesOpts: self.data('dygraph-highlightseriesopts') || null, // TOO SLOW: { strokeWidth: 1.5 }, + highlightSeriesBackgroundAlpha: self.data('dygraph-highlightseriesbackgroundalpha') || null, // TOO SLOW: (chart_type === 'stacked')?0.7:0.5, + + pointClickCallback: self.data('dygraph-pointclickcallback') || undefined, + visibility: state.dimensions_visibility.selected2BooleanArray(state.data.dimension_names), + axes: { + x: { + pixelsPerLabel: 50, + ticker: Dygraph.dateTicker, + axisLabelFormatter: function (d, gran) { + return NETDATA.zeropad(d.getHours()) + ":" + NETDATA.zeropad(d.getMinutes()) + ":" + NETDATA.zeropad(d.getSeconds()); + }, + valueFormatter: function (ms) { + var d = new Date(ms); + return d.toLocaleDateString() + ' ' + d.toLocaleTimeString(); + // return NETDATA.zeropad(d.getHours()) + ":" + NETDATA.zeropad(d.getMinutes()) + ":" + NETDATA.zeropad(d.getSeconds()); + } + }, + y: { + pixelsPerLabel: 15, + valueFormatter: function (x) { + // we format legends with the state object + // no need to do anything here + // return (Math.round(x*100) / 100).toLocaleString(); + // return state.legendFormatValue(x); + return x; + } + } + }, + legendFormatter: function(data) { + var elements = state.element_legend_childs; + + // if the hidden div is not there + // we are not managing the legend + if(elements.hidden === null) return; + + if (typeof data.x !== 'undefined') { + state.legendSetDate(data.x); + var i = data.series.length; + while(i--) { + var series = data.series[i]; + if(!series.isVisible) continue; + state.legendSetLabelValue(series.label, series.y); + } + } + + return ''; + }, + drawCallback: function(dygraph, is_initial) { + if(state.current.name !== 'auto' && state.dygraph_user_action === true) { + state.dygraph_user_action = false; + + var x_range = dygraph.xAxisRange(); + var after = Math.round(x_range[0]); + var before = Math.round(x_range[1]); + + if(NETDATA.options.debug.dygraph === true) + state.log('dygraphDrawCallback(dygraph, ' + is_initial + '): ' + (after / 1000).toString() + ' - ' + (before / 1000).toString()); + + if(before <= state.netdata_last && after >= state.netdata_first) + state.updateChartPanOrZoom(after, before); + } + }, + zoomCallback: function(minDate, maxDate, yRanges) { + if(NETDATA.options.debug.dygraph === true) + state.log('dygraphZoomCallback()'); + + state.globalSelectionSyncStop(); + state.globalSelectionSyncDelay(); + state.setMode('zoom'); + + // refresh it to the greatest possible zoom level + state.dygraph_user_action = true; + state.dygraph_force_zoom = true; + state.updateChartPanOrZoom(minDate, maxDate); + }, + highlightCallback: function(event, x, points, row, seriesName) { + if(NETDATA.options.debug.dygraph === true || state.debug === true) + state.log('dygraphHighlightCallback()'); + + state.pauseChart(); + + // there is a bug in dygraph when the chart is zoomed enough + // the time it thinks is selected is wrong + // here we calculate the time t based on the row number selected + // which is ok + var t = state.data_after + row * state.data_update_every; + // console.log('row = ' + row + ', x = ' + x + ', t = ' + t + ' ' + ((t === x)?'SAME':(Math.abs(x-t)<=state.data_update_every)?'SIMILAR':'DIFFERENT') + ', rows in db: ' + state.data_points + ' visible(x) = ' + state.timeIsVisible(x) + ' visible(t) = ' + state.timeIsVisible(t) + ' r(x) = ' + state.calculateRowForTime(x) + ' r(t) = ' + state.calculateRowForTime(t) + ' range: ' + state.data_after + ' - ' + state.data_before + ' real: ' + state.data.after + ' - ' + state.data.before + ' every: ' + state.data_update_every); + + state.globalSelectionSync(x); + + // fix legend zIndex using the internal structures of dygraph legend module + // this works, but it is a hack! + // state.dygraph_instance.plugins_[0].plugin.legend_div_.style.zIndex = 10000; + }, + unhighlightCallback: function(event) { + if(NETDATA.options.debug.dygraph === true || state.debug === true) + state.log('dygraphUnhighlightCallback()'); + + state.unpauseChart(); + state.globalSelectionSyncStop(); + }, + interactionModel : { + mousedown: function(event, dygraph, context) { + if(NETDATA.options.debug.dygraph === true || state.debug === true) + state.log('interactionModel.mousedown()'); + + state.dygraph_user_action = true; + state.globalSelectionSyncStop(); + + if(NETDATA.options.debug.dygraph === true) + state.log('dygraphMouseDown()'); + + // Right-click should not initiate a zoom. + if(event.button && event.button === 2) return; + + context.initializeMouseDown(event, dygraph, context); + + if(event.button && event.button === 1) { + if (event.altKey || event.shiftKey) { + state.setMode('pan'); + state.globalSelectionSyncDelay(); + Dygraph.startPan(event, dygraph, context); + } + else { + state.setMode('zoom'); + state.globalSelectionSyncDelay(); + Dygraph.startZoom(event, dygraph, context); + } + } + else { + if (event.altKey || event.shiftKey) { + state.setMode('zoom'); + state.globalSelectionSyncDelay(); + Dygraph.startZoom(event, dygraph, context); + } + else { + state.setMode('pan'); + state.globalSelectionSyncDelay(); + Dygraph.startPan(event, dygraph, context); + } + } + }, + mousemove: function(event, dygraph, context) { + if(NETDATA.options.debug.dygraph === true || state.debug === true) + state.log('interactionModel.mousemove()'); + + if(context.isPanning) { + state.dygraph_user_action = true; + state.globalSelectionSyncStop(); + state.globalSelectionSyncDelay(); + state.setMode('pan'); + Dygraph.movePan(event, dygraph, context); + } + else if(context.isZooming) { + state.dygraph_user_action = true; + state.globalSelectionSyncStop(); + state.globalSelectionSyncDelay(); + state.setMode('zoom'); + Dygraph.moveZoom(event, dygraph, context); + } + }, + mouseup: function(event, dygraph, context) { + if(NETDATA.options.debug.dygraph === true || state.debug === true) + state.log('interactionModel.mouseup()'); + + if (context.isPanning) { + state.dygraph_user_action = true; + state.globalSelectionSyncDelay(); + Dygraph.endPan(event, dygraph, context); + } + else if (context.isZooming) { + state.dygraph_user_action = true; + state.globalSelectionSyncDelay(); + Dygraph.endZoom(event, dygraph, context); + } + }, + click: function(event, dygraph, context) { + if(NETDATA.options.debug.dygraph === true || state.debug === true) + state.log('interactionModel.click()'); + + event.preventDefault(); + }, + dblclick: function(event, dygraph, context) { + if(NETDATA.options.debug.dygraph === true || state.debug === true) + state.log('interactionModel.dblclick()'); + NETDATA.resetAllCharts(state); + }, + mousewheel: function(event, dygraph, context) { + if(NETDATA.options.debug.dygraph === true || state.debug === true) + state.log('interactionModel.mousewheel()'); + + // Take the offset of a mouse event on the dygraph canvas and + // convert it to a pair of percentages from the bottom left. + // (Not top left, bottom is where the lower value is.) + function offsetToPercentage(g, offsetX, offsetY) { + // This is calculating the pixel offset of the leftmost date. + var xOffset = g.toDomCoords(g.xAxisRange()[0], null)[0]; + var yar0 = g.yAxisRange(0); + + // This is calculating the pixel of the higest value. (Top pixel) + var yOffset = g.toDomCoords(null, yar0[1])[1]; + + // x y w and h are relative to the corner of the drawing area, + // so that the upper corner of the drawing area is (0, 0). + var x = offsetX - xOffset; + var y = offsetY - yOffset; + + // This is computing the rightmost pixel, effectively defining the + // width. + var w = g.toDomCoords(g.xAxisRange()[1], null)[0] - xOffset; + + // This is computing the lowest pixel, effectively defining the height. + var h = g.toDomCoords(null, yar0[0])[1] - yOffset; + + // Percentage from the left. + var xPct = w === 0 ? 0 : (x / w); + // Percentage from the top. + var yPct = h === 0 ? 0 : (y / h); + + // The (1-) part below changes it from "% distance down from the top" + // to "% distance up from the bottom". + return [xPct, (1-yPct)]; + } + + // Adjusts [x, y] toward each other by zoomInPercentage% + // Split it so the left/bottom axis gets xBias/yBias of that change and + // tight/top gets (1-xBias)/(1-yBias) of that change. + // + // If a bias is missing it splits it down the middle. + function zoomRange(g, zoomInPercentage, xBias, yBias) { + xBias = xBias || 0.5; + yBias = yBias || 0.5; + + function adjustAxis(axis, zoomInPercentage, bias) { + var delta = axis[1] - axis[0]; + var increment = delta * zoomInPercentage; + var foo = [increment * bias, increment * (1-bias)]; + + return [ axis[0] + foo[0], axis[1] - foo[1] ]; + } + + var yAxes = g.yAxisRanges(); + var newYAxes = []; + for (var i = 0; i < yAxes.length; i++) { + newYAxes[i] = adjustAxis(yAxes[i], zoomInPercentage, yBias); + } + + return adjustAxis(g.xAxisRange(), zoomInPercentage, xBias); + } + + if(event.altKey || event.shiftKey) { + state.dygraph_user_action = true; + + state.globalSelectionSyncStop(); + state.globalSelectionSyncDelay(); + + // http://dygraphs.com/gallery/interaction-api.js + var normal = (event.detail) ? event.detail * -1 : event.wheelDelta / 40; + var percentage = normal / 50; + + if (!(event.offsetX && event.offsetY)){ + event.offsetX = event.layerX - event.target.offsetLeft; + event.offsetY = event.layerY - event.target.offsetTop; + } + + var percentages = offsetToPercentage(dygraph, event.offsetX, event.offsetY); + var xPct = percentages[0]; + var yPct = percentages[1]; + + var new_x_range = zoomRange(dygraph, percentage, xPct, yPct); + + var after = new_x_range[0]; + var before = new_x_range[1]; + + var first = state.netdata_first + state.data_update_every; + var last = state.netdata_last + state.data_update_every; + + if(before > last) { + after -= (before - last); + before = last; + } + if(after < first) { + after = first; + } + + state.setMode('zoom'); + if(state.updateChartPanOrZoom(after, before) === true) + dygraph.updateOptions({ dateWindow: [ after, before ] }); + + event.preventDefault(); + } + }, + touchstart: function(event, dygraph, context) { + if(NETDATA.options.debug.dygraph === true || state.debug === true) + state.log('interactionModel.touchstart()'); + + state.dygraph_user_action = true; + state.setMode('zoom'); + state.pauseChart(); + + Dygraph.defaultInteractionModel.touchstart(event, dygraph, context); + + // we overwrite the touch directions at the end, to overwrite + // the internal default of dygraphs + context.touchDirections = { x: true, y: false }; + + state.dygraph_last_touch_start = new Date().getTime(); + state.dygraph_last_touch_move = 0; + + if(typeof event.touches[0].pageX === 'number') + state.dygraph_last_touch_page_x = event.touches[0].pageX; + else + state.dygraph_last_touch_page_x = 0; + }, + touchmove: function(event, dygraph, context) { + if(NETDATA.options.debug.dygraph === true || state.debug === true) + state.log('interactionModel.touchmove()'); + + state.dygraph_user_action = true; + Dygraph.defaultInteractionModel.touchmove(event, dygraph, context); + + state.dygraph_last_touch_move = new Date().getTime(); + }, + touchend: function(event, dygraph, context) { + if(NETDATA.options.debug.dygraph === true || state.debug === true) + state.log('interactionModel.touchend()'); + + state.dygraph_user_action = true; + Dygraph.defaultInteractionModel.touchend(event, dygraph, context); + + // if it didn't move, it is a selection + if(state.dygraph_last_touch_move === 0 && state.dygraph_last_touch_page_x !== 0) { + // internal api of dygraphs + var pct = (state.dygraph_last_touch_page_x - (dygraph.plotter_.area.x + state.element.getBoundingClientRect().left)) / dygraph.plotter_.area.w; + var t = Math.round(state.data_after + (state.data_before - state.data_after) * pct); + if(NETDATA.dygraphSetSelection(state, t) === true) + state.globalSelectionSync(t); + } + + // if it was double tap within double click time, reset the charts + var now = new Date().getTime(); + if(typeof state.dygraph_last_touch_end !== 'undefined') { + if(state.dygraph_last_touch_move === 0) { + var dt = now - state.dygraph_last_touch_end; + if(dt <= NETDATA.options.current.double_click_speed) + NETDATA.resetAllCharts(state); + } + } + + // remember the timestamp of the last touch end + state.dygraph_last_touch_end = now; + } + } + }; + + if(NETDATA.chartLibraries.dygraph.isSparkline(state)) { + state.dygraph_options.drawGrid = false; + state.dygraph_options.drawAxis = false; + state.dygraph_options.title = undefined; + state.dygraph_options.units = undefined; + state.dygraph_options.ylabel = undefined; + state.dygraph_options.yLabelWidth = 0; + state.dygraph_options.labelsDivWidth = 120; + state.dygraph_options.labelsDivStyles.width = '120px'; + state.dygraph_options.labelsSeparateLines = true; + state.dygraph_options.rightGap = 0; + state.dygraph_options.yRangePad = 1; + } + + if(smooth === true) { + state.dygraph_smooth_eligible = true; + + if(NETDATA.options.current.smooth_plot === true) + state.dygraph_options.plotter = smoothPlotter; + } + else state.dygraph_smooth_eligible = false; + + state.dygraph_instance = new Dygraph(state.element_chart, + data.result.data, state.dygraph_options); + + state.dygraph_force_zoom = false; + state.dygraph_user_action = false; + state.dygraph_last_rendered = new Date().getTime(); + return true; + }; + + // ---------------------------------------------------------------------------------------------------------------- + // morris + + NETDATA.morrisInitialize = function(callback) { + if(typeof netdataNoMorris === 'undefined' || !netdataNoMorris) { + + // morris requires raphael + if(!NETDATA.chartLibraries.raphael.initialized) { + if(NETDATA.chartLibraries.raphael.enabled) { + NETDATA.raphaelInitialize(function() { + NETDATA.morrisInitialize(callback); + }); + } + else { + NETDATA.chartLibraries.morris.enabled = false; + if(typeof callback === "function") + callback(); + } + } + else { + NETDATA._loadCSS(NETDATA.morris_css); + + $.ajax({ + url: NETDATA.morris_js, + cache: true, + dataType: "script", + xhrFields: { withCredentials: true } // required for the cookie + }) + .done(function() { + NETDATA.registerChartLibrary('morris', NETDATA.morris_js); + }) + .fail(function() { + NETDATA.chartLibraries.morris.enabled = false; + NETDATA.error(100, NETDATA.morris_js); + }) + .always(function() { + if(typeof callback === "function") + callback(); + }); + } + } + else { + NETDATA.chartLibraries.morris.enabled = false; + if(typeof callback === "function") + callback(); + } + }; + + NETDATA.morrisChartUpdate = function(state, data) { + state.morris_instance.setData(data.result.data); + return true; + }; + + NETDATA.morrisChartCreate = function(state, data) { + + state.morris_options = { + element: state.element_chart.id, + data: data.result.data, + xkey: 'time', + ykeys: data.dimension_names, + labels: data.dimension_names, + lineWidth: 2, + pointSize: 3, + smooth: true, + hideHover: 'auto', + parseTime: true, + continuousLine: false, + behaveLikeLine: false + }; + + if(state.chart.chart_type === 'line') + state.morris_instance = new Morris.Line(state.morris_options); + + else if(state.chart.chart_type === 'area') { + state.morris_options.behaveLikeLine = true; + state.morris_instance = new Morris.Area(state.morris_options); + } + else // stacked + state.morris_instance = new Morris.Area(state.morris_options); + + return true; + }; + + // ---------------------------------------------------------------------------------------------------------------- + // raphael + + NETDATA.raphaelInitialize = function(callback) { + if(typeof netdataStopRaphael === 'undefined' || !netdataStopRaphael) { + $.ajax({ + url: NETDATA.raphael_js, + cache: true, + dataType: "script", + xhrFields: { withCredentials: true } // required for the cookie + }) + .done(function() { + NETDATA.registerChartLibrary('raphael', NETDATA.raphael_js); + }) + .fail(function() { + NETDATA.chartLibraries.raphael.enabled = false; + NETDATA.error(100, NETDATA.raphael_js); + }) + .always(function() { + if(typeof callback === "function") + callback(); + }); + } + else { + NETDATA.chartLibraries.raphael.enabled = false; + if(typeof callback === "function") + callback(); + } + }; + + NETDATA.raphaelChartUpdate = function(state, data) { + $(state.element_chart).raphael(data.result, { + width: state.chartWidth(), + height: state.chartHeight() + }); + + return false; + }; + + NETDATA.raphaelChartCreate = function(state, data) { + $(state.element_chart).raphael(data.result, { + width: state.chartWidth(), + height: state.chartHeight() + }); + + return false; + }; + + // ---------------------------------------------------------------------------------------------------------------- + // C3 + + NETDATA.c3Initialize = function(callback) { + if(typeof netdataNoC3 === 'undefined' || !netdataNoC3) { + + // C3 requires D3 + if(!NETDATA.chartLibraries.d3.initialized) { + if(NETDATA.chartLibraries.d3.enabled) { + NETDATA.d3Initialize(function() { + NETDATA.c3Initialize(callback); + }); + } + else { + NETDATA.chartLibraries.c3.enabled = false; + if(typeof callback === "function") + callback(); + } + } + else { + NETDATA._loadCSS(NETDATA.c3_css); + + $.ajax({ + url: NETDATA.c3_js, + cache: true, + dataType: "script", + xhrFields: { withCredentials: true } // required for the cookie + }) + .done(function() { + NETDATA.registerChartLibrary('c3', NETDATA.c3_js); + }) + .fail(function() { + NETDATA.chartLibraries.c3.enabled = false; + NETDATA.error(100, NETDATA.c3_js); + }) + .always(function() { + if(typeof callback === "function") + callback(); + }); + } + } + else { + NETDATA.chartLibraries.c3.enabled = false; + if(typeof callback === "function") + callback(); + } + }; + + NETDATA.c3ChartUpdate = function(state, data) { + state.c3_instance.destroy(); + return NETDATA.c3ChartCreate(state, data); + + //state.c3_instance.load({ + // rows: data.result, + // unload: true + //}); + + //return true; + }; + + NETDATA.c3ChartCreate = function(state, data) { + + state.element_chart.id = 'c3-' + state.uuid; + // console.log('id = ' + state.element_chart.id); + + state.c3_instance = c3.generate({ + bindto: '#' + state.element_chart.id, + size: { + width: state.chartWidth(), + height: state.chartHeight() + }, + color: { + pattern: state.chartColors() + }, + data: { + x: 'time', + rows: data.result, + type: (state.chart.chart_type === 'line')?'spline':'area-spline' + }, + axis: { + x: { + type: 'timeseries', + tick: { + format: function(x) { + return NETDATA.zeropad(x.getHours()) + ":" + NETDATA.zeropad(x.getMinutes()) + ":" + NETDATA.zeropad(x.getSeconds()); + } + } + } + }, + grid: { + x: { + show: true + }, + y: { + show: true + } + }, + point: { + show: false + }, + line: { + connectNull: false + }, + transition: { + duration: 0 + }, + interaction: { + enabled: true + } + }); + + // console.log(state.c3_instance); + + return true; + }; + + // ---------------------------------------------------------------------------------------------------------------- + // D3 + + NETDATA.d3Initialize = function(callback) { + if(typeof netdataStopD3 === 'undefined' || !netdataStopD3) { + $.ajax({ + url: NETDATA.d3_js, + cache: true, + dataType: "script", + xhrFields: { withCredentials: true } // required for the cookie + }) + .done(function() { + NETDATA.registerChartLibrary('d3', NETDATA.d3_js); + }) + .fail(function() { + NETDATA.chartLibraries.d3.enabled = false; + NETDATA.error(100, NETDATA.d3_js); + }) + .always(function() { + if(typeof callback === "function") + callback(); + }); + } + else { + NETDATA.chartLibraries.d3.enabled = false; + if(typeof callback === "function") + callback(); + } + }; + + NETDATA.d3ChartUpdate = function(state, data) { + return false; + }; + + NETDATA.d3ChartCreate = function(state, data) { + return false; + }; + + // ---------------------------------------------------------------------------------------------------------------- + // google charts + + NETDATA.googleInitialize = function(callback) { + if(typeof netdataNoGoogleCharts === 'undefined' || !netdataNoGoogleCharts) { + $.ajax({ + url: NETDATA.google_js, + cache: true, + dataType: "script", + xhrFields: { withCredentials: true } // required for the cookie + }) + .done(function() { + NETDATA.registerChartLibrary('google', NETDATA.google_js); + google.load('visualization', '1.1', { + 'packages': ['corechart', 'controls'], + 'callback': callback + }); + }) + .fail(function() { + NETDATA.chartLibraries.google.enabled = false; + NETDATA.error(100, NETDATA.google_js); + if(typeof callback === "function") + callback(); + }); + } + else { + NETDATA.chartLibraries.google.enabled = false; + if(typeof callback === "function") + callback(); + } + }; + + NETDATA.googleChartUpdate = function(state, data) { + var datatable = new google.visualization.DataTable(data.result); + state.google_instance.draw(datatable, state.google_options); + return true; + }; + + NETDATA.googleChartCreate = function(state, data) { + var datatable = new google.visualization.DataTable(data.result); + + state.google_options = { + colors: state.chartColors(), + + // do not set width, height - the chart resizes itself + //width: state.chartWidth(), + //height: state.chartHeight(), + lineWidth: 1, + title: state.title, + fontSize: 11, + hAxis: { + // title: "Time of Day", + // format:'HH:mm:ss', + viewWindowMode: 'maximized', + slantedText: false, + format:'HH:mm:ss', + textStyle: { + fontSize: 9 + }, + gridlines: { + color: '#EEE' + } + }, + vAxis: { + title: state.units, + viewWindowMode: 'pretty', + minValue: -0.1, + maxValue: 0.1, + direction: 1, + textStyle: { + fontSize: 9 + }, + gridlines: { + color: '#EEE' + } + }, + chartArea: { + width: '65%', + height: '80%' + }, + focusTarget: 'category', + annotation: { + '1': { + style: 'line' + } + }, + pointsVisible: 0, + titlePosition: 'out', + titleTextStyle: { + fontSize: 11 + }, + tooltip: { + isHtml: false, + ignoreBounds: true, + textStyle: { + fontSize: 9 + } + }, + curveType: 'function', + areaOpacity: 0.3, + isStacked: false + }; + + switch(state.chart.chart_type) { + case "area": + state.google_options.vAxis.viewWindowMode = 'maximized'; + state.google_options.areaOpacity = NETDATA.options.current.color_fill_opacity_area; + state.google_instance = new google.visualization.AreaChart(state.element_chart); + break; + + case "stacked": + state.google_options.isStacked = true; + state.google_options.areaOpacity = NETDATA.options.current.color_fill_opacity_stacked; + state.google_options.vAxis.viewWindowMode = 'maximized'; + state.google_options.vAxis.minValue = null; + state.google_options.vAxis.maxValue = null; + state.google_instance = new google.visualization.AreaChart(state.element_chart); + break; + + default: + case "line": + state.google_options.lineWidth = 2; + state.google_instance = new google.visualization.LineChart(state.element_chart); + break; + } + + state.google_instance.draw(datatable, state.google_options); + return true; + }; + + // ---------------------------------------------------------------------------------------------------------------- + + NETDATA.percentFromValueMax = function(value, max) { + if(value === null) value = 0; + if(max < value) max = value; + + var pcent = 0; + if(max !== 0) { + pcent = Math.round(value * 100 / max); + if(pcent === 0 && value > 0) pcent = 1; + } + + return pcent; + }; + + // ---------------------------------------------------------------------------------------------------------------- + // easy-pie-chart + + NETDATA.easypiechartInitialize = function(callback) { + if(typeof netdataNoEasyPieChart === 'undefined' || !netdataNoEasyPieChart) { + $.ajax({ + url: NETDATA.easypiechart_js, + cache: true, + dataType: "script", + xhrFields: { withCredentials: true } // required for the cookie + }) + .done(function() { + NETDATA.registerChartLibrary('easypiechart', NETDATA.easypiechart_js); + }) + .fail(function() { + NETDATA.chartLibraries.easypiechart.enabled = false; + NETDATA.error(100, NETDATA.easypiechart_js); + }) + .always(function() { + if(typeof callback === "function") + callback(); + }) + } + else { + NETDATA.chartLibraries.easypiechart.enabled = false; + if(typeof callback === "function") + callback(); + } + }; + + NETDATA.easypiechartClearSelection = function(state) { + if(typeof state.easyPieChartEvent !== 'undefined') { + if(state.easyPieChartEvent.timer !== null) + clearTimeout(state.easyPieChartEvent.timer); + + state.easyPieChartEvent.timer = null; + } + + if(state.isAutoRefreshable() === true && state.data !== null) { + NETDATA.easypiechartChartUpdate(state, state.data); + } + else { + state.easyPieChartLabel.innerHTML = state.legendFormatValue(null); + state.easyPieChart_instance.update(0); + } + state.easyPieChart_instance.enableAnimation(); + + return true; + }; + + NETDATA.easypiechartSetSelection = function(state, t) { + if(state.timeIsVisible(t) !== true) + return NETDATA.easypiechartClearSelection(state); + + var slot = state.calculateRowForTime(t); + if(slot < 0 || slot >= state.data.result.length) + return NETDATA.easypiechartClearSelection(state); + + if(typeof state.easyPieChartEvent === 'undefined') { + state.easyPieChartEvent = { + timer: null, + value: 0, + pcent: 0 + }; + } + + var value = state.data.result[state.data.result.length - 1 - slot]; + var max = (state.easyPieChartMax === null)?state.data.max:state.easyPieChartMax; + var pcent = NETDATA.percentFromValueMax(value, max); + + state.easyPieChartEvent.value = value; + state.easyPieChartEvent.pcent = pcent; + state.easyPieChartLabel.innerHTML = state.legendFormatValue(value); + + if(state.easyPieChartEvent.timer === null) { + state.easyPieChart_instance.disableAnimation(); + + state.easyPieChartEvent.timer = setTimeout(function() { + state.easyPieChartEvent.timer = null; + state.easyPieChart_instance.update(state.easyPieChartEvent.pcent); + }, NETDATA.options.current.charts_selection_animation_delay); + } + + return true; + }; + + NETDATA.easypiechartChartUpdate = function(state, data) { + var value, max, pcent; + + if(NETDATA.globalPanAndZoom.isActive() === true || state.isAutoRefreshable() === false) { + value = null; + max = 0; + pcent = 0; + } + else { + value = data.result[0]; + max = (state.easyPieChartMax === null)?data.max:state.easyPieChartMax; + pcent = NETDATA.percentFromValueMax(value, max); + } + + state.easyPieChartLabel.innerHTML = state.legendFormatValue(value); + state.easyPieChart_instance.update(pcent); + return true; + }; + + NETDATA.easypiechartChartCreate = function(state, data) { + var self = $(state.element); + var chart = $(state.element_chart); + + var value = data.result[0]; + var max = self.data('easypiechart-max-value') || null; + var adjust = self.data('easypiechart-adjust') || null; + + if(max === null) { + max = data.max; + state.easyPieChartMax = null; + } + else + state.easyPieChartMax = max; + + var pcent = NETDATA.percentFromValueMax(value, max); + + chart.data('data-percent', pcent); + + var size; + switch(adjust) { + case 'width': size = state.chartHeight(); break; + case 'min': size = Math.min(state.chartWidth(), state.chartHeight()); break; + case 'max': size = Math.max(state.chartWidth(), state.chartHeight()); break; + case 'height': + default: size = state.chartWidth(); break; + } + state.element.style.width = size + 'px'; + state.element.style.height = size + 'px'; + + var stroke = Math.floor(size / 22); + if(stroke < 3) stroke = 2; + + var valuefontsize = Math.floor((size * 2 / 3) / 5); + var valuetop = Math.round((size - valuefontsize - (size / 40)) / 2); + state.easyPieChartLabel = document.createElement('span'); + state.easyPieChartLabel.className = 'easyPieChartLabel'; + state.easyPieChartLabel.innerHTML = state.legendFormatValue(value); + state.easyPieChartLabel.style.fontSize = valuefontsize + 'px'; + state.easyPieChartLabel.style.top = valuetop.toString() + 'px'; + state.element_chart.appendChild(state.easyPieChartLabel); + + var titlefontsize = Math.round(valuefontsize * 1.6 / 3); + var titletop = Math.round(valuetop - (titlefontsize * 2) - (size / 40)); + state.easyPieChartTitle = document.createElement('span'); + state.easyPieChartTitle.className = 'easyPieChartTitle'; + state.easyPieChartTitle.innerHTML = state.title; + state.easyPieChartTitle.style.fontSize = titlefontsize + 'px'; + state.easyPieChartTitle.style.lineHeight = titlefontsize + 'px'; + state.easyPieChartTitle.style.top = titletop.toString() + 'px'; + state.element_chart.appendChild(state.easyPieChartTitle); + + var unitfontsize = Math.round(titlefontsize * 0.9); + var unittop = Math.round(valuetop + (valuefontsize + unitfontsize) + (size / 40)); + state.easyPieChartUnits = document.createElement('span'); + state.easyPieChartUnits.className = 'easyPieChartUnits'; + state.easyPieChartUnits.innerHTML = state.units; + state.easyPieChartUnits.style.fontSize = unitfontsize + 'px'; + state.easyPieChartUnits.style.top = unittop.toString() + 'px'; + state.element_chart.appendChild(state.easyPieChartUnits); + + chart.easyPieChart({ + barColor: self.data('easypiechart-barcolor') || state.chartColors()[0], //'#ef1e25', + trackColor: self.data('easypiechart-trackcolor') || NETDATA.themes.current.easypiechart_track, + scaleColor: self.data('easypiechart-scalecolor') || NETDATA.themes.current.easypiechart_scale, + scaleLength: self.data('easypiechart-scalelength') || 5, + lineCap: self.data('easypiechart-linecap') || 'round', + lineWidth: self.data('easypiechart-linewidth') || stroke, + trackWidth: self.data('easypiechart-trackwidth') || undefined, + size: self.data('easypiechart-size') || size, + rotate: self.data('easypiechart-rotate') || 0, + animate: self.data('easypiechart-rotate') || {duration: 500, enabled: true}, + easing: self.data('easypiechart-easing') || undefined + }); + + // when we just re-create the chart + // do not animate the first update + var animate = true; + if(typeof state.easyPieChart_instance !== 'undefined') + animate = false; + + state.easyPieChart_instance = chart.data('easyPieChart'); + if(animate === false) state.easyPieChart_instance.disableAnimation(); + state.easyPieChart_instance.update(pcent); + if(animate === false) state.easyPieChart_instance.enableAnimation(); + return true; + }; + + // ---------------------------------------------------------------------------------------------------------------- + // gauge.js + + NETDATA.gaugeInitialize = function(callback) { + if(typeof netdataNoGauge === 'undefined' || !netdataNoGauge) { + $.ajax({ + url: NETDATA.gauge_js, + cache: true, + dataType: "script", + xhrFields: { withCredentials: true } // required for the cookie + }) + .done(function() { + NETDATA.registerChartLibrary('gauge', NETDATA.gauge_js); + }) + .fail(function() { + NETDATA.chartLibraries.gauge.enabled = false; + NETDATA.error(100, NETDATA.gauge_js); + }) + .always(function() { + if(typeof callback === "function") + callback(); + }) + } + else { + NETDATA.chartLibraries.gauge.enabled = false; + if(typeof callback === "function") + callback(); + } + }; + + NETDATA.gaugeAnimation = function(state, status) { + var speed = 32; + + if(typeof status === 'boolean' && status === false) + speed = 1000000000; + else if(typeof status === 'number') + speed = status; + + state.gauge_instance.animationSpeed = speed; + state.___gaugeOld__.speed = speed; + }; + + NETDATA.gaugeSet = function(state, value, min, max) { + if(typeof value !== 'number') value = 0; + if(typeof min !== 'number') min = 0; + if(typeof max !== 'number') max = 0; + if(value > max) max = value; + if(value < min) min = value; + if(min > max) { + var t = min; + min = max; + max = t; + } + else if(min == max) + max = min + 1; + + // gauge.js has an issue if the needle + // is smaller than min or larger than max + // when we set the new values + // the needle will go crazy + + // to prevent it, we always feed it + // with a percentage, so that the needle + // is always between min and max + var pcent = (value - min) * 100 / (max - min); + + // these should never happen + if(pcent < 0) pcent = 0; + if(pcent > 100) pcent = 100; + + state.gauge_instance.set(pcent); + + state.___gaugeOld__.value = value; + state.___gaugeOld__.min = min; + state.___gaugeOld__.max = max; + }; + + NETDATA.gaugeSetLabels = function(state, value, min, max) { + if(state.___gaugeOld__.valueLabel !== value) { + state.___gaugeOld__.valueLabel = value; + state.gaugeChartLabel.innerHTML = state.legendFormatValue(value); + } + if(state.___gaugeOld__.minLabel !== min) { + state.___gaugeOld__.minLabel = min; + state.gaugeChartMin.innerHTML = state.legendFormatValue(min); + } + if(state.___gaugeOld__.maxLabel !== max) { + state.___gaugeOld__.maxLabel = max; + state.gaugeChartMax.innerHTML = state.legendFormatValue(max); + } + }; + + NETDATA.gaugeClearSelection = function(state) { + if(typeof state.gaugeEvent !== 'undefined') { + if(state.gaugeEvent.timer !== null) + clearTimeout(state.gaugeEvent.timer); + + state.gaugeEvent.timer = null; + } + + if(state.isAutoRefreshable() === true && state.data !== null) { + NETDATA.gaugeChartUpdate(state, state.data); + } + else { + NETDATA.gaugeAnimation(state, false); + NETDATA.gaugeSet(state, null, null, null); + NETDATA.gaugeSetLabels(state, null, null, null); + } + + NETDATA.gaugeAnimation(state, true); + return true; + }; + + NETDATA.gaugeSetSelection = function(state, t) { + if(state.timeIsVisible(t) !== true) + return NETDATA.gaugeClearSelection(state); + + var slot = state.calculateRowForTime(t); + if(slot < 0 || slot >= state.data.result.length) + return NETDATA.gaugeClearSelection(state); + + if(typeof state.gaugeEvent === 'undefined') { + state.gaugeEvent = { + timer: null, + value: 0, + min: 0, + max: 0 + }; + } + + var value = state.data.result[state.data.result.length - 1 - slot]; + var max = (state.gaugeMax === null)?state.data.max:state.gaugeMax; + var min = 0; + + state.gaugeEvent.value = value; + state.gaugeEvent.max = max; + state.gaugeEvent.min = min; + NETDATA.gaugeSetLabels(state, value, min, max); + + if(state.gaugeEvent.timer === null) { + NETDATA.gaugeAnimation(state, false); + + state.gaugeEvent.timer = setTimeout(function() { + state.gaugeEvent.timer = null; + NETDATA.gaugeSet(state, state.gaugeEvent.value, state.gaugeEvent.min, state.gaugeEvent.max); + }, NETDATA.options.current.charts_selection_animation_delay); + } + + return true; + }; + + NETDATA.gaugeChartUpdate = function(state, data) { + var value, min, max; + + if(NETDATA.globalPanAndZoom.isActive() === true || state.isAutoRefreshable() === false) { + value = 0; + min = 0; + max = 1; + NETDATA.gaugeSetLabels(state, null, null, null); + } + else { + value = data.result[0]; + min = 0; + max = (state.gaugeMax === null)?data.max:state.gaugeMax; + if(value > max) max = value; + NETDATA.gaugeSetLabels(state, value, min, max); + } + + NETDATA.gaugeSet(state, value, min, max); + return true; + }; + + NETDATA.gaugeChartCreate = function(state, data) { + var self = $(state.element); + // var chart = $(state.element_chart); + + var value = data.result[0]; + var max = self.data('gauge-max-value') || null; + var adjust = self.data('gauge-adjust') || null; + var pointerColor = self.data('gauge-pointer-color') || NETDATA.themes.current.gauge_pointer; + var strokeColor = self.data('gauge-stroke-color') || NETDATA.themes.current.gauge_stroke; + var startColor = self.data('gauge-start-color') || state.chartColors()[0]; + var stopColor = self.data('gauge-stop-color') || void 0; + var generateGradient = self.data('gauge-generate-gradient') || false; + + if(max === null) { + max = data.max; + state.gaugeMax = null; + } + else + state.gaugeMax = max; + + var width = state.chartWidth(), height = state.chartHeight(); //, ratio = 1.5; + //switch(adjust) { + // case 'width': width = height * ratio; break; + // case 'height': + // default: height = width / ratio; break; + //} + //state.element.style.width = width.toString() + 'px'; + //state.element.style.height = height.toString() + 'px'; + + var lum_d = 0.05; + + var options = { + lines: 12, // The number of lines to draw + angle: 0.15, // The length of each line + lineWidth: 0.44, // 0.44 The line thickness + pointer: { + length: 0.8, // 0.9 The radius of the inner circle + strokeWidth: 0.035, // The rotation offset + color: pointerColor // Fill color + }, + colorStart: startColor, // Colors + colorStop: stopColor, // just experiment with them + strokeColor: strokeColor, // to see which ones work best for you + limitMax: true, + generateGradient: (generateGradient === true)?true:false, + gradientType: 0 + }; + + if (generateGradient.constructor === Array) { + // example options: + // data-gauge-generate-gradient="[0, 50, 100]" + // data-gauge-gradient-percent-color-0="#FFFFFF" + // data-gauge-gradient-percent-color-50="#999900" + // data-gauge-gradient-percent-color-100="#000000" + + options.percentColors = new Array(); + var len = generateGradient.length; + while(len--) { + var pcent = generateGradient[len]; + var color = self.data('gauge-gradient-percent-color-' + pcent.toString()) || false; + if(color !== false) { + var a = new Array(); + a[0] = pcent / 100; + a[1] = color; + options.percentColors.unshift(a); + } + } + if(options.percentColors.length === 0) + delete options.percentColors; + } + else if(generateGradient === false && NETDATA.themes.current.gauge_gradient === true) { + options.percentColors = [ + [0.0, NETDATA.colorLuminance(startColor, (lum_d * 10) - (lum_d * 0))], + [0.1, NETDATA.colorLuminance(startColor, (lum_d * 10) - (lum_d * 1))], + [0.2, NETDATA.colorLuminance(startColor, (lum_d * 10) - (lum_d * 2))], + [0.3, NETDATA.colorLuminance(startColor, (lum_d * 10) - (lum_d * 3))], + [0.4, NETDATA.colorLuminance(startColor, (lum_d * 10) - (lum_d * 4))], + [0.5, NETDATA.colorLuminance(startColor, (lum_d * 10) - (lum_d * 5))], + [0.6, NETDATA.colorLuminance(startColor, (lum_d * 10) - (lum_d * 6))], + [0.7, NETDATA.colorLuminance(startColor, (lum_d * 10) - (lum_d * 7))], + [0.8, NETDATA.colorLuminance(startColor, (lum_d * 10) - (lum_d * 8))], + [0.9, NETDATA.colorLuminance(startColor, (lum_d * 10) - (lum_d * 9))], + [1.0, NETDATA.colorLuminance(startColor, 0.0)]]; + } + + state.gauge_canvas = document.createElement('canvas'); + state.gauge_canvas.id = 'gauge-' + state.uuid + '-canvas'; + state.gauge_canvas.className = 'gaugeChart'; + state.gauge_canvas.width = width; + state.gauge_canvas.height = height; + state.element_chart.appendChild(state.gauge_canvas); + + var valuefontsize = Math.floor(height / 6); + var valuetop = Math.round((height - valuefontsize - (height / 6)) / 2); + state.gaugeChartLabel = document.createElement('span'); + state.gaugeChartLabel.className = 'gaugeChartLabel'; + state.gaugeChartLabel.style.fontSize = valuefontsize + 'px'; + state.gaugeChartLabel.style.top = valuetop.toString() + 'px'; + state.element_chart.appendChild(state.gaugeChartLabel); + + var titlefontsize = Math.round(valuefontsize / 2); + var titletop = 0; + state.gaugeChartTitle = document.createElement('span'); + state.gaugeChartTitle.className = 'gaugeChartTitle'; + state.gaugeChartTitle.innerHTML = state.title; + state.gaugeChartTitle.style.fontSize = titlefontsize + 'px'; + state.gaugeChartTitle.style.lineHeight = titlefontsize + 'px'; + state.gaugeChartTitle.style.top = titletop.toString() + 'px'; + state.element_chart.appendChild(state.gaugeChartTitle); + + var unitfontsize = Math.round(titlefontsize * 0.9); + state.gaugeChartUnits = document.createElement('span'); + state.gaugeChartUnits.className = 'gaugeChartUnits'; + state.gaugeChartUnits.innerHTML = state.units; + state.gaugeChartUnits.style.fontSize = unitfontsize + 'px'; + state.element_chart.appendChild(state.gaugeChartUnits); + + state.gaugeChartMin = document.createElement('span'); + state.gaugeChartMin.className = 'gaugeChartMin'; + state.gaugeChartMin.style.fontSize = Math.round(valuefontsize * 0.75).toString() + 'px'; + state.element_chart.appendChild(state.gaugeChartMin); + + state.gaugeChartMax = document.createElement('span'); + state.gaugeChartMax.className = 'gaugeChartMax'; + state.gaugeChartMax.style.fontSize = Math.round(valuefontsize * 0.75).toString() + 'px'; + state.element_chart.appendChild(state.gaugeChartMax); + + // when we just re-create the chart + // do not animate the first update + var animate = true; + if(typeof state.gauge_instance !== 'undefined') + animate = false; + + state.gauge_instance = new Gauge(state.gauge_canvas).setOptions(options); // create sexy gauge! + + state.___gaugeOld__ = { + value: value, + min: 0, + max: max, + valueLabel: null, + minLabel: null, + maxLabel: null + }; + + // we will always feed a percentage + state.gauge_instance.minValue = 0; + state.gauge_instance.maxValue = 100; + + NETDATA.gaugeAnimation(state, animate); + NETDATA.gaugeSet(state, value, 0, max); + NETDATA.gaugeSetLabels(state, value, 0, max); + NETDATA.gaugeAnimation(state, true); + return true; + }; + + // ---------------------------------------------------------------------------------------------------------------- + // Charts Libraries Registration + + NETDATA.chartLibraries = { + "dygraph": { + initialize: NETDATA.dygraphInitialize, + create: NETDATA.dygraphChartCreate, + update: NETDATA.dygraphChartUpdate, + resize: function(state) { + if(typeof state.dygraph_instance.resize === 'function') + state.dygraph_instance.resize(); + }, + setSelection: NETDATA.dygraphSetSelection, + clearSelection: NETDATA.dygraphClearSelection, + toolboxPanAndZoom: NETDATA.dygraphToolboxPanAndZoom, + initialized: false, + enabled: true, + format: function(state) { return 'json'; }, + options: function(state) { return 'ms|flip'; }, + legend: function(state) { + if(this.isSparkline(state) === false) + return 'right-side'; + else + return null; + }, + autoresize: function(state) { return true; }, + max_updates_to_recreate: function(state) { return 5000; }, + track_colors: function(state) { return true; }, + pixels_per_point: function(state) { + if(this.isSparkline(state) === false) + return 3; + else + return 2; + }, + + isSparkline: function(state) { + if(typeof state.dygraph_sparkline === 'undefined') { + var t = $(state.element).data('dygraph-theme'); + if(t === 'sparkline') + state.dygraph_sparkline = true; + else + state.dygraph_sparkline = false; + } + return state.dygraph_sparkline; + } + }, + "sparkline": { + initialize: NETDATA.sparklineInitialize, + create: NETDATA.sparklineChartCreate, + update: NETDATA.sparklineChartUpdate, + resize: null, + setSelection: undefined, // function(state, t) { return true; }, + clearSelection: undefined, // function(state) { return true; }, + toolboxPanAndZoom: null, + initialized: false, + enabled: true, + format: function(state) { return 'array'; }, + options: function(state) { return 'flip|abs'; }, + legend: function(state) { return null; }, + autoresize: function(state) { return false; }, + max_updates_to_recreate: function(state) { return 5000; }, + track_colors: function(state) { return false; }, + pixels_per_point: function(state) { return 3; } + }, + "peity": { + initialize: NETDATA.peityInitialize, + create: NETDATA.peityChartCreate, + update: NETDATA.peityChartUpdate, + resize: null, + setSelection: undefined, // function(state, t) { return true; }, + clearSelection: undefined, // function(state) { return true; }, + toolboxPanAndZoom: null, + initialized: false, + enabled: true, + format: function(state) { return 'ssvcomma'; }, + options: function(state) { return 'null2zero|flip|abs'; }, + legend: function(state) { return null; }, + autoresize: function(state) { return false; }, + max_updates_to_recreate: function(state) { return 5000; }, + track_colors: function(state) { return false; }, + pixels_per_point: function(state) { return 3; } + }, + "morris": { + initialize: NETDATA.morrisInitialize, + create: NETDATA.morrisChartCreate, + update: NETDATA.morrisChartUpdate, + resize: null, + setSelection: undefined, // function(state, t) { return true; }, + clearSelection: undefined, // function(state) { return true; }, + toolboxPanAndZoom: null, + initialized: false, + enabled: true, + format: function(state) { return 'json'; }, + options: function(state) { return 'objectrows|ms'; }, + legend: function(state) { return null; }, + autoresize: function(state) { return false; }, + max_updates_to_recreate: function(state) { return 50; }, + track_colors: function(state) { return false; }, + pixels_per_point: function(state) { return 15; } + }, + "google": { + initialize: NETDATA.googleInitialize, + create: NETDATA.googleChartCreate, + update: NETDATA.googleChartUpdate, + resize: null, + setSelection: undefined, //function(state, t) { return true; }, + clearSelection: undefined, //function(state) { return true; }, + toolboxPanAndZoom: null, + initialized: false, + enabled: true, + format: function(state) { return 'datatable'; }, + options: function(state) { return ''; }, + legend: function(state) { return null; }, + autoresize: function(state) { return false; }, + max_updates_to_recreate: function(state) { return 300; }, + track_colors: function(state) { return false; }, + pixels_per_point: function(state) { return 4; } + }, + "raphael": { + initialize: NETDATA.raphaelInitialize, + create: NETDATA.raphaelChartCreate, + update: NETDATA.raphaelChartUpdate, + resize: null, + setSelection: undefined, // function(state, t) { return true; }, + clearSelection: undefined, // function(state) { return true; }, + toolboxPanAndZoom: null, + initialized: false, + enabled: true, + format: function(state) { return 'json'; }, + options: function(state) { return ''; }, + legend: function(state) { return null; }, + autoresize: function(state) { return false; }, + max_updates_to_recreate: function(state) { return 5000; }, + track_colors: function(state) { return false; }, + pixels_per_point: function(state) { return 3; } + }, + "c3": { + initialize: NETDATA.c3Initialize, + create: NETDATA.c3ChartCreate, + update: NETDATA.c3ChartUpdate, + resize: null, + setSelection: undefined, // function(state, t) { return true; }, + clearSelection: undefined, // function(state) { return true; }, + toolboxPanAndZoom: null, + initialized: false, + enabled: true, + format: function(state) { return 'csvjsonarray'; }, + options: function(state) { return 'milliseconds'; }, + legend: function(state) { return null; }, + autoresize: function(state) { return false; }, + max_updates_to_recreate: function(state) { return 5000; }, + track_colors: function(state) { return false; }, + pixels_per_point: function(state) { return 15; } + }, + "d3": { + initialize: NETDATA.d3Initialize, + create: NETDATA.d3ChartCreate, + update: NETDATA.d3ChartUpdate, + resize: null, + setSelection: undefined, // function(state, t) { return true; }, + clearSelection: undefined, // function(state) { return true; }, + toolboxPanAndZoom: null, + initialized: false, + enabled: true, + format: function(state) { return 'json'; }, + options: function(state) { return ''; }, + legend: function(state) { return null; }, + autoresize: function(state) { return false; }, + max_updates_to_recreate: function(state) { return 5000; }, + track_colors: function(state) { return false; }, + pixels_per_point: function(state) { return 3; } + }, + "easypiechart": { + initialize: NETDATA.easypiechartInitialize, + create: NETDATA.easypiechartChartCreate, + update: NETDATA.easypiechartChartUpdate, + resize: null, + setSelection: NETDATA.easypiechartSetSelection, + clearSelection: NETDATA.easypiechartClearSelection, + toolboxPanAndZoom: null, + initialized: false, + enabled: true, + format: function(state) { return 'array'; }, + options: function(state) { return 'absolute'; }, + legend: function(state) { return null; }, + autoresize: function(state) { return false; }, + max_updates_to_recreate: function(state) { return 5000; }, + track_colors: function(state) { return true; }, + pixels_per_point: function(state) { return 3; }, + aspect_ratio: 100 + }, + "gauge": { + initialize: NETDATA.gaugeInitialize, + create: NETDATA.gaugeChartCreate, + update: NETDATA.gaugeChartUpdate, + resize: null, + setSelection: NETDATA.gaugeSetSelection, + clearSelection: NETDATA.gaugeClearSelection, + toolboxPanAndZoom: null, + initialized: false, + enabled: true, + format: function(state) { return 'array'; }, + options: function(state) { return 'absolute'; }, + legend: function(state) { return null; }, + autoresize: function(state) { return false; }, + max_updates_to_recreate: function(state) { return 5000; }, + track_colors: function(state) { return true; }, + pixels_per_point: function(state) { return 3; }, + aspect_ratio: 70 + } + }; + + NETDATA.registerChartLibrary = function(library, url) { + if(NETDATA.options.debug.libraries === true) + console.log("registering chart library: " + library); + + NETDATA.chartLibraries[library].url = url; + NETDATA.chartLibraries[library].initialized = true; + NETDATA.chartLibraries[library].enabled = true; + }; + + // ---------------------------------------------------------------------------------------------------------------- + // Load required JS libraries and CSS + + NETDATA.requiredJs = [ + { + url: NETDATA.serverDefault + 'lib/bootstrap.min.js', + isAlreadyLoaded: function() { + // check if bootstrap is loaded + if(typeof $().emulateTransitionEnd == 'function') + return true; + else { + if(typeof netdataNoBootstrap !== 'undefined' && netdataNoBootstrap) + return true; + else + return false; + } + } + }, + { + url: NETDATA.serverDefault + 'lib/jquery.nanoscroller.min.js', + isAlreadyLoaded: function() { return false; } + }, + { + url: NETDATA.serverDefault + 'lib/bootstrap-toggle.min.js', + isAlreadyLoaded: function() { return false; } + } + ]; + + NETDATA.requiredCSS = [ + { + url: NETDATA.themes.current.bootstrap_css, + isAlreadyLoaded: function() { + if(typeof netdataNoBootstrap !== 'undefined' && netdataNoBootstrap) + return true; + else + return false; + } + }, + { + url: NETDATA.serverDefault + 'css/font-awesome.min.css', + isAlreadyLoaded: function() { return false; } + }, + { + url: NETDATA.themes.current.dashboard_css, + isAlreadyLoaded: function() { return false; } + }, + { + url: NETDATA.serverDefault + 'css/bootstrap-toggle.min.css', + isAlreadyLoaded: function() { return false; } + } + ]; + + NETDATA.loadRequiredJs = function(index, callback) { + if(index >= NETDATA.requiredJs.length) { + if(typeof callback === 'function') + callback(); + return; + } + + if(NETDATA.requiredJs[index].isAlreadyLoaded()) { + NETDATA.loadRequiredJs(++index, callback); + return; + } + + if(NETDATA.options.debug.main_loop === true) + console.log('loading ' + NETDATA.requiredJs[index].url); + + $.ajax({ + url: NETDATA.requiredJs[index].url, + cache: true, + dataType: "script", + xhrFields: { withCredentials: true } // required for the cookie + }) + .success(function() { + if(NETDATA.options.debug.main_loop === true) + console.log('loaded ' + NETDATA.requiredJs[index].url); + + NETDATA.loadRequiredJs(++index, callback); + }) + .fail(function() { + alert('Cannot load required JS library: ' + NETDATA.requiredJs[index].url); + }) + }; + + NETDATA.loadRequiredCSS = function(index) { + if(index >= NETDATA.requiredCSS.length) + return; + + if(NETDATA.requiredCSS[index].isAlreadyLoaded()) { + NETDATA.loadRequiredCSS(++index); + return; + } + + if(NETDATA.options.debug.main_loop === true) + console.log('loading ' + NETDATA.requiredCSS[index].url); + + NETDATA._loadCSS(NETDATA.requiredCSS[index].url); + NETDATA.loadRequiredCSS(++index); + }; + + + // ---------------------------------------------------------------------------------------------------------------- + // Registry of netdata hosts + + NETDATA.alarms = { + current: null, + callback: null, + + get: function(what, callback) { + $.ajax({ + url: NETDATA.serverDefault + '/api/v1/alarms?' + what.toString(), + async: true, + cache: false, + xhrFields: { withCredentials: true } // required for the cookie + }) + .done(function(data) { + if(typeof callback === 'function') + callback(data); + }) + .fail(function() { + NETDATA.error(415, host); + + if(typeof callback === 'function') + callback(null); + }); + }, + + update_forever: function() { + NETDATA.alarms.get('active', function(data) { + if(data !== null) { + NETDATA.alarms.current = data; + + if (typeof NETDATA.alarms.callback === 'function') { + NETDATA.alarms.callback(data); + } + } + + setTimeout(NETDATA.alarms.update_forever, 10000); + }); + }, + + get_log: function(callback) { + $.ajax({ + url: NETDATA.serverDefault + '/api/v1/alarm_log', + async: true, + cache: false, + xhrFields: { withCredentials: true } // required for the cookie + }) + .done(function(data) { + if(typeof callback === 'function') + callback(data); + }) + .fail(function() { + NETDATA.error(416, host); + + if(typeof callback === 'function') + callback(null); + }); + }, + + init: function() { + NETDATA.alarms.update_forever(); + } + }; + + // ---------------------------------------------------------------------------------------------------------------- + // Registry of netdata hosts + + NETDATA.registry = { + server: null, // the netdata registry server + person_guid: null, // the unique ID of this browser / user + machine_guid: null, // the unique ID the netdata server that served dashboard.js + hostname: null, // the hostname of the netdata server that served dashboard.js + machines: null, // the user's other URLs + machines_array: null, // the user's other URLs in an array + person_urls: null, + + parsePersonUrls: function(person_urls) { + // console.log(person_urls); + NETDATA.registry.person_urls = person_urls; + + if(person_urls) { + NETDATA.registry.machines = {}; + NETDATA.registry.machines_array = new Array(); + + var now = new Date().getTime(); + var apu = person_urls; + var i = apu.length; + while(i--) { + if(typeof NETDATA.registry.machines[apu[i][0]] === 'undefined') { + // console.log('adding: ' + apu[i][4] + ', ' + ((now - apu[i][2]) / 1000).toString()); + + var obj = { + guid: apu[i][0], + url: apu[i][1], + last_t: apu[i][2], + accesses: apu[i][3], + name: apu[i][4], + alternate_urls: new Array() + }; + obj.alternate_urls.push(apu[i][1]); + + NETDATA.registry.machines[apu[i][0]] = obj; + NETDATA.registry.machines_array.push(obj); + } + else { + // console.log('appending: ' + apu[i][4] + ', ' + ((now - apu[i][2]) / 1000).toString()); + + var pu = NETDATA.registry.machines[apu[i][0]]; + if(pu.last_t < apu[i][2]) { + pu.url = apu[i][1]; + pu.last_t = apu[i][2]; + pu.name = apu[i][4]; + } + pu.accesses += apu[i][3]; + pu.alternate_urls.push(apu[i][1]); + } + } + } + + if(typeof netdataRegistryCallback === 'function') + netdataRegistryCallback(NETDATA.registry.machines_array); + }, + + init: function() { + if(typeof netdataNoRegistry !== 'undefined' && netdataNoRegistry) + return; + + NETDATA.registry.hello(NETDATA.serverDefault, function(data) { + if(data) { + NETDATA.registry.server = data.registry; + NETDATA.registry.machine_guid = data.machine_guid; + NETDATA.registry.hostname = data.hostname; + + NETDATA.registry.access(2, function (person_urls) { + NETDATA.registry.parsePersonUrls(person_urls); + + }); + } + }); + }, + + hello: function(host, callback) { + while(host.slice(-1) === '/') + host = host.substring(0, host.length - 1); + + // send HELLO to a netdata server: + // 1. verifies the server is reachable + // 2. responds with the registry URL, the machine GUID of this netdata server and its hostname + $.ajax({ + url: host + '/api/v1/registry?action=hello', + async: true, + cache: false, + xhrFields: { withCredentials: true } // required for the cookie + }) + .done(function(data) { + if(typeof data.status !== 'string' || data.status !== 'ok') { + NETDATA.error(408, host + ' response: ' + JSON.stringify(data)); + data = null; + } + + if(typeof callback === 'function') + callback(data); + }) + .fail(function() { + NETDATA.error(407, host); + + if(typeof callback === 'function') + callback(null); + }); + }, + + access: function(max_redirects, callback) { + // send ACCESS to a netdata registry: + // 1. it lets it know we are accessing a netdata server (its machine GUID and its URL) + // 2. it responds with a list of netdata servers we know + // the registry identifies us using a cookie it sets the first time we access it + // the registry may respond with a redirect URL to send us to another registry + $.ajax({ + url: NETDATA.registry.server + '/api/v1/registry?action=access&machine=' + NETDATA.registry.machine_guid + '&name=' + encodeURIComponent(NETDATA.registry.hostname) + '&url=' + encodeURIComponent(NETDATA.serverDefault), // + '&visible_url=' + encodeURIComponent(document.location), + async: true, + cache: false, + xhrFields: { withCredentials: true } // required for the cookie + }) + .done(function(data) { + var redirect = null; + if(typeof data.registry === 'string') + redirect = data.registry; + + if(typeof data.status !== 'string' || data.status !== 'ok') { + NETDATA.error(409, NETDATA.registry.server + ' responded with: ' + JSON.stringify(data)); + data = null; + } + + if(data === null) { + if(redirect !== null && max_redirects > 0) { + NETDATA.registry.server = redirect; + NETDATA.registry.access(max_redirects - 1, callback); + } + else { + if(typeof callback === 'function') + callback(null); + } + } + else { + if(typeof data.person_guid === 'string') + NETDATA.registry.person_guid = data.person_guid; + + if(typeof callback === 'function') + callback(data.urls); + } + }) + .fail(function() { + NETDATA.error(410, NETDATA.registry.server); + + if(typeof callback === 'function') + callback(null); + }); + }, + + delete: function(delete_url, callback) { + // send DELETE to a netdata registry: + $.ajax({ + url: NETDATA.registry.server + '/api/v1/registry?action=delete&machine=' + NETDATA.registry.machine_guid + '&name=' + encodeURIComponent(NETDATA.registry.hostname) + '&url=' + encodeURIComponent(NETDATA.serverDefault) + '&delete_url=' + encodeURIComponent(delete_url), + async: true, + cache: false, + xhrFields: { withCredentials: true } // required for the cookie + }) + .done(function(data) { + if(typeof data.status !== 'string' || data.status !== 'ok') { + NETDATA.error(411, NETDATA.registry.server + ' responded with: ' + JSON.stringify(data)); + data = null; + } + + if(typeof callback === 'function') + callback(data); + }) + .fail(function() { + NETDATA.error(412, NETDATA.registry.server); + + if(typeof callback === 'function') + callback(null); + }); + }, + + switch: function(new_person_guid, callback) { + // impersonate + $.ajax({ + url: NETDATA.registry.server + '/api/v1/registry?action=switch&machine=' + NETDATA.registry.machine_guid + '&name=' + encodeURIComponent(NETDATA.registry.hostname) + '&url=' + encodeURIComponent(NETDATA.serverDefault) + '&to=' + new_person_guid, + async: true, + cache: false, + xhrFields: { withCredentials: true } // required for the cookie + }) + .done(function(data) { + if(typeof data.status !== 'string' || data.status !== 'ok') { + NETDATA.error(413, NETDATA.registry.server + ' responded with: ' + JSON.stringify(data)); + data = null; + } + + if(typeof callback === 'function') + callback(data); + }) + .fail(function() { + NETDATA.error(414, NETDATA.registry.server); + + if(typeof callback === 'function') + callback(null); + }); + } + }; + + // ---------------------------------------------------------------------------------------------------------------- + // Boot it! + + NETDATA.errorReset(); + NETDATA.loadRequiredCSS(0); + + NETDATA._loadjQuery(function() { + NETDATA.loadRequiredJs(0, function() { + if(typeof $().emulateTransitionEnd !== 'function') { + // bootstrap is not available + NETDATA.options.current.show_help = false; + } + + if(typeof netdataDontStart === 'undefined' || !netdataDontStart) { + if(NETDATA.options.debug.main_loop === true) + console.log('starting chart refresh thread'); + + NETDATA.start(); + } + }); + }); + + // window.NETDATA = NETDATA; // })(window, document); diff --git a/web/dashboard.slate.css b/web/dashboard.slate.css index 0536a3ed6..9223958ae 100644 --- a/web/dashboard.slate.css +++ b/web/dashboard.slate.css @@ -14,6 +14,19 @@ body { border-left: 2px solid #765d9c; } +.morelink { + color: #765d9c; + text-decoration: none; +} +.morelink:hover { + color: #563d7c; + text-decoration: none; +} +.morelink:focus { + color: #765d9c; + text-decoration: none; +} + .netdata-chart-alignment { margin-left: 55px; } diff --git a/web/demo.html b/web/demo.html index 4b91d8394..89bcb68c7 100644 --- a/web/demo.html +++ b/web/demo.html @@ -1,42 +1,51 @@ <!DOCTYPE html> <html lang="en"> <head> - <title>NetData Dashboard</title> + <title>NetData Dashboard</title> + <meta name="application-name" content="netdata"> - <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> - <meta charset="utf-8"> - <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1"> - <meta name="viewport" content="width=device-width, initial-scale=1"> - <meta name="apple-mobile-web-app-capable" content="yes"> - <meta name="apple-mobile-web-app-status-bar-style" content="black-translucent"> - <meta name="author" content="costa@tsaousis.gr"> + <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> + <meta charset="utf-8"> + <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1"> + <meta name="viewport" content="width=device-width, initial-scale=1"> + <meta name="apple-mobile-web-app-capable" content="yes"> + <meta name="apple-mobile-web-app-status-bar-style" content="black-translucent"> + <meta name="author" content="costa@tsaousis.gr"> - <script type="text/javascript" src="dashboard.js?v37"></script> + <meta property="og:locale" content="en_US" /> + <meta property="og:image" content="https://my-netdata.io/images/seo-performance-512.png"/> + <meta property="og:url" content="http://my-netdata.io/"/> + <meta property="og:type" content="website"/> + <meta property="og:site_name" content="netdata"/> + <meta property="og:title" content="netdata - real-time performance monitoring, done right!"/> + <meta property="og:description" content="Stunning real-time dashboards, blazingly fast and extremely interactive. Zero configuration, zero dependencies, zero maintenance." /> + + <script type="text/javascript" src="dashboard.js?v42"></script> </head> <body> <div style="width: 100%; text-align: center;"> - <div data-netdata="netdata.server_cpu" - data-dimensions="user" - data-chart-library="gauge" - data-width="150px" - data-after="-60" - data-points="60" - data-title="Yes! Realtime!" - data-units="I am alive!" - data-colors="#FF5555" - ></div> - <br/> - <div data-netdata="netdata.server_cpu" - data-dimensions="user" - data-chart-library="dygraph" - data-dygraph-theme="sparkline" - data-width="200px" - data-height="30px" - data-after="-60" - data-points="60" - data-colors="#FF5555" - ></div> + <div data-netdata="netdata.server_cpu" + data-dimensions="user" + data-chart-library="gauge" + data-width="150px" + data-after="-60" + data-points="60" + data-title="Yes! Realtime!" + data-units="I am alive!" + data-colors="#FF5555" + ></div> + <br/> + <div data-netdata="netdata.server_cpu" + data-dimensions="user" + data-chart-library="dygraph" + data-dygraph-theme="sparkline" + data-width="200px" + data-height="30px" + data-after="-60" + data-points="60" + data-colors="#FF5555" + ></div> </div> </body> </html> diff --git a/web/demo2.html b/web/demo2.html index 9530d914e..5f8136ce5 100644 --- a/web/demo2.html +++ b/web/demo2.html @@ -1,134 +1,143 @@ <!DOCTYPE html> <html lang="en"> <head> - <title>NetData Dashboard</title> + <title>NetData Dashboard</title> + <meta name="application-name" content="netdata"> - <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> - <meta charset="utf-8"> - <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1"> - <meta name="viewport" content="width=device-width, initial-scale=1"> - <meta name="apple-mobile-web-app-capable" content="yes"> - <meta name="apple-mobile-web-app-status-bar-style" content="black-translucent"> - <meta name="author" content="costa@tsaousis.gr"> + <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> + <meta charset="utf-8"> + <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1"> + <meta name="viewport" content="width=device-width, initial-scale=1"> + <meta name="apple-mobile-web-app-capable" content="yes"> + <meta name="apple-mobile-web-app-status-bar-style" content="black-translucent"> + <meta name="author" content="costa@tsaousis.gr"> - <script>var netdataTheme = 'slate';</script> - <script type="text/javascript" src="dashboard.js?v37"></script> + <meta property="og:locale" content="en_US" /> + <meta property="og:image" content="https://my-netdata.io/images/seo-performance-512.png"/> + <meta property="og:url" content="http://my-netdata.io/"/> + <meta property="og:type" content="website"/> + <meta property="og:site_name" content="netdata"/> + <meta property="og:title" content="netdata - real-time performance monitoring, done right!"/> + <meta property="og:description" content="Stunning real-time dashboards, blazingly fast and extremely interactive. Zero configuration, zero dependencies, zero maintenance." /> + + <script>var netdataTheme = 'slate';</script> + <script type="text/javascript" src="http://my-netdata.io/dashboard.js?v42"></script> </head> <body> <div class="container" style="width: 90%; padding-top: 10px; text-align: center; color: #AAA"> - <div style="font-size: 7vw;">why netdata?</div> - <br/> - <div style="font-size: 2vw; color: white;">These charts visualize the same data...</div> + <div style="font-size: 7vw;">why netdata?</div> + <br/> + <div style="font-size: 2vw; color: white;">These charts visualize the same data...</div> - <!-- Nav tabs --> - <ul class="nav nav-tabs" role="tablist"> - <li role="presentation" class="active"><a href="#gauge" aria-controls="gauge" role="tab" data-toggle="tab">Gauge.js</a></li> - <li role="presentation"><a href="#easypiechart" aria-controls="easypiechart" role="tab" data-toggle="tab">Easy Pie Chart</a></li> - </ul> + <!-- Nav tabs --> + <ul class="nav nav-tabs" role="tablist"> + <li role="presentation" class="active"><a href="#gauge" aria-controls="gauge" role="tab" data-toggle="tab">Gauge.js</a></li> + <li role="presentation"><a href="#easypiechart" aria-controls="easypiechart" role="tab" data-toggle="tab">Easy Pie Chart</a></li> + </ul> - <!-- Tab panes --> - <div class="tab-content"> - <div role="tabpanel" class="tab-pane active" id="gauge"> + <!-- Tab panes --> + <div class="tab-content"> + <div role="tabpanel" class="tab-pane active" id="gauge"> - <div style="display: inline-block; width: 35.8%"> - <div style="font-size: 1.2vw; color: #666; padding-top: 10px;"><i class="fa fa-comment"></i> I can trace an issue like this</div> - <br/> - <div data-netdata="example.random2" - data-dimensions="random" - data-chart-library="gauge" - data-gauge-max-value="32767" - data-width="100%" - data-after="-600" - data-points="600" - data-title="1/second (netdata default)" - data-units="important metric" - data-colors="#5A5" - ></div> - </div> - <div style="display: inline-block; width: 50%"> - <div style="font-size: 1.2vw; color: #666;"><i class="fa fa-comment"></i> Can you trace an issue like these?<br/> <br/></div> - <div data-netdata="example.random2" - data-dimensions="random" - data-chart-library="gauge" - data-gauge-max-value="32767" - data-width="45%" - data-after="-600" - data-points="40" - data-title="Updates Every 15 Sec" - data-units="important metric" - data-colors="#C55" - ></div> - <div data-netdata="example.random2" - data-dimensions="random" - data-chart-library="gauge" - data-gauge-max-value="32767" - data-width="45%" - data-after="-600" - data-points="2" - data-title="Updates Every 5 Mins" - data-units="important metric" - data-colors="#C55" - ></div> - </div> - </div> - <div role="tabpanel" class="tab-pane" id="easypiechart"> + <div style="display: inline-block; width: 35.8%"> + <div style="font-size: 1.2vw; color: #666; padding-top: 10px;"><i class="fa fa-comment"></i> I can trace an issue like this</div> + <br/> + <div data-netdata="example.random2" + data-dimensions="random" + data-chart-library="gauge" + data-gauge-max-value="32767" + data-width="100%" + data-after="-600" + data-points="600" + data-title="1/second (netdata default)" + data-units="important metric" + data-colors="#5A5" + ></div> + </div> + <div style="display: inline-block; width: 50%"> + <div style="font-size: 1.2vw; color: #666;"><i class="fa fa-comment"></i> Can you trace an issue like these?<br/> <br/></div> + <div data-netdata="example.random2" + data-dimensions="random" + data-chart-library="gauge" + data-gauge-max-value="32767" + data-width="45%" + data-after="-600" + data-points="60" + data-title="Updates Every 10 Sec" + data-units="important metric" + data-colors="#C55" + ></div> + <div data-netdata="example.random2" + data-dimensions="random" + data-chart-library="gauge" + data-gauge-max-value="32767" + data-width="45%" + data-after="-600" + data-points="2" + data-title="Updates Every 5 Mins" + data-units="important metric" + data-colors="#C55" + ></div> + </div> + </div> + <div role="tabpanel" class="tab-pane" id="easypiechart"> - <div style="display: inline-block; width: 25%"> - <div style="font-size: 1.2vw; color: #666; padding-top: 10px;"><i class="fa fa-comment"></i> I can trace an issue like this</div> - <br/> - <div data-netdata="example.random2" - data-dimensions="random" - data-chart-library="easypiechart" - data-easypiechart-max-value="32767" - data-width="100%" - data-after="-600" - data-points="600" - data-title="1/second (netdata default)" - data-units="important metric" - data-colors="#5A5" - ></div> - </div> - <div style="display: inline-block; width: 40%"> - <div style="font-size: 1.2vw; color: #666;"><i class="fa fa-comment"></i> Can you trace an issue like these?<br/> <br/></div> - <div data-netdata="example.random2" - data-dimensions="random" - data-chart-library="easypiechart" - data-easypiechart-max-value="32767" - data-width="45%" - data-after="-600" - data-points="40" - data-title="Updates Every 15 Sec (<a href='https://github.com/OpenTSDB/opentsdb.net/blob/gh-pages/docs/source/user_guide/utilities/tcollector.rst#collecting-lots-of-metrics-with-tcollector' target='_blank'>OpenTSDB</a>)" - data-units="important metric" - data-colors="#C55" - ></div> - <div data-netdata="example.random2" - data-dimensions="random" - data-chart-library="easypiechart" - data-easypiechart-max-value="32767" - data-width="45%" - data-after="-600" - data-points="2" - data-title="Updates Every 5 Mins (your NMS)" - data-units="important metric" - data-colors="#C55" - ></div> - </div> - </div> - </div> - <div style="font-size: 1.5vw;">Hover on the chart below, to see the selected value on the charts above!</div> - <div data-netdata="example.random2" - data-dimensions="random" - data-dygraph-theme="sparkline" - data-width="100%" - data-height="20vh" - data-after="-600" - data-points="600" - data-title="1/second (netdata default)" - data-units="something" - data-colors="#888" - ></div> + <div style="display: inline-block; width: 25%"> + <div style="font-size: 1.2vw; color: #666; padding-top: 10px;"><i class="fa fa-comment"></i> I can trace an issue like this</div> + <br/> + <div data-netdata="example.random2" + data-dimensions="random" + data-chart-library="easypiechart" + data-easypiechart-max-value="32767" + data-width="100%" + data-after="-600" + data-points="600" + data-title="1/second (netdata default)" + data-units="important metric" + data-colors="#5A5" + ></div> + </div> + <div style="display: inline-block; width: 40%"> + <div style="font-size: 1.2vw; color: #666;"><i class="fa fa-comment"></i> Can you trace an issue like these?<br/> <br/></div> + <div data-netdata="example.random2" + data-dimensions="random" + data-chart-library="easypiechart" + data-easypiechart-max-value="32767" + data-width="45%" + data-after="-600" + data-points="60" + data-title="Updates Every 10 Sec" + data-units="important metric" + data-colors="#C55" + ></div> + <div data-netdata="example.random2" + data-dimensions="random" + data-chart-library="easypiechart" + data-easypiechart-max-value="32767" + data-width="45%" + data-after="-600" + data-points="2" + data-title="Updates Every 5 Mins" + data-units="important metric" + data-colors="#C55" + ></div> + </div> + </div> + </div> + <div style="font-size: 1.5vw;">Hover on the chart below, to see the selected value on the charts above!</div> + <div data-netdata="example.random2" + data-dimensions="random" + data-dygraph-theme="sparkline" + data-width="100%" + data-height="20vh" + data-after="-600" + data-points="600" + data-title="1/second (netdata default)" + data-units="something" + data-colors="#888" + ></div> </div> </body> </html> diff --git a/web/demosites.html b/web/demosites.html index f5047f4b2..d57bb4e71 100644 --- a/web/demosites.html +++ b/web/demosites.html @@ -1,721 +1,1045 @@ <!DOCTYPE html> <html lang="en"> <head> - <title>NetData - Real-time performance monitoring, done right!</title> + <title>NetData - Real-time performance monitoring, done right!</title> + <meta name="application-name" content="netdata"> - <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> - <meta charset="utf-8"> - <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1"> - <meta name="viewport" content="width=device-width, initial-scale=1"> - <meta name="apple-mobile-web-app-capable" content="yes"> - <meta name="apple-mobile-web-app-status-bar-style" content="black-translucent"> + <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> + <meta charset="utf-8"> + <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1"> + <meta name="viewport" content="width=device-width, initial-scale=1"> + <meta name="apple-mobile-web-app-capable" content="yes"> + <meta name="apple-mobile-web-app-status-bar-style" content="black-translucent"> - <script> - // --- OPTIONS FOR THE DASHBOARD -- + <meta property="og:locale" content="en_US" /> + <meta property="og:image" content="https://my-netdata.io/images/seo-performance-512.png"/> + <meta property="og:url" content="http://my-netdata.io/"/> + <meta property="og:type" content="website"/> + <meta property="og:site_name" content="netdata"/> + <meta property="og:title" content="netdata - real-time performance monitoring, done right!"/> + <meta property="og:description" content="Stunning real-time dashboards, blazingly fast and extremely interactive. Zero configuration, zero dependencies, zero maintenance." /> - // this section has to appear before loading dashboard.js + <script> + // --- OPTIONS FOR THE DASHBOARD -- - // Select a theme. - // uncomment on of the two themes: + // this section has to appear before loading dashboard.js - // var netdataTheme = 'default'; // this is white - var netdataTheme = 'slate'; // this is dark + // Select a theme. + // uncomment on of the two themes: + // var netdataTheme = 'default'; // this is white + var netdataTheme = 'slate'; // this is dark - // Set the default netdata server. - // on charts without a 'data-host', this one will be used. - // the default is the server that dashboard.js is downloaded from. - // var netdataServer = 'http://my.server:19999/'; - </script> + // Set the default netdata server. + // on charts without a 'data-host', this one will be used. + // the default is the server that dashboard.js is downloaded from. - <!-- - --- LOAD dashboard.js --- + // var netdataServer = 'http://my.server:19999/'; + </script> - to host this HTML file on your web server, - you have to load dashboard.js from the netdata server. + <!-- + --- LOAD dashboard.js --- - So, pick one the two below - If you pick the first, set the server name/IP. + to host this HTML file on your web server, + you have to load dashboard.js from the netdata server. - The second assumes you host this file on /usr/share/netdata/web - and that you have chown it to be owned by netdata:netdata - --> - <!-- <script type="text/javascript" src="http://my.server:19999/dashboard.js"></script> --> - <script type="text/javascript" src="dashboard.js?v37"></script> + So, pick one the two below + If you pick the first, set the server name/IP. - <script> - // --- OPTIONS FOR THE CHARTS -- + The second assumes you host this file on /usr/share/netdata/web + and that you have chown it to be owned by netdata:netdata + --> + <!-- <script type="text/javascript" src="http://my.server:19999/dashboard.js"></script> --> + <script type="text/javascript" src="dashboard.js?v42"></script> - // destroy charts not shown (lowers memory on the browsers) - // set this to 'yes' to destroy, 'false' to hide the charts - NETDATA.options.current.destroy_on_hide = false; - - // set this to false, to always show all dimensions - NETDATA.options.current.eliminate_zero_dimensions = true; - - // set this to false, to lower the pressure on the browser - NETDATA.options.current.concurrent_refreshes = true; + <script> + // --- OPTIONS FOR THE CHARTS -- - // if you need to support slow mobile phones, set this to false - NETDATA.options.current.parallel_refresher = true; + // destroy charts not shown (lowers memory on the browsers) + // set this to 'true' to destroy, 'false' to hide the charts + NETDATA.options.current.destroy_on_hide = true; + + // set this to false, to always show all dimensions + NETDATA.options.current.eliminate_zero_dimensions = true; + + // set this to false, to lower the pressure on the browser + NETDATA.options.current.concurrent_refreshes = false; - // set this to false, to always update the charts, even if focus is lost - NETDATA.options.current.stop_updates_when_focus_is_lost = true; - </script> + // if you need to support slow mobile phones, set this to false + NETDATA.options.current.parallel_refresher = true; - <style> + // set this to false, to always update the charts, even if focus is lost + NETDATA.options.current.stop_updates_when_focus_is_lost = true; + </script> + + <style> body { - font-size: 1vw; + font-size: 1vw; } .mysparkline { - position: relative; - display: inline-block; - min-height: 50px; - width: 100%; - height: 8vmax; - text-align: left; + position: relative; + display: inline-block; + min-height: 50px; + width: 100%; + height: 8vmax; + text-align: left; } .mysparkline-overchart-label { - position: absolute; - display: block; - top: 0; - left: 10px; - bottom: 0; - right: 0; - font-size: 1vmax; - z-index: 1; + position: absolute; + display: block; + top: 0; + left: 10px; + bottom: 0; + right: 0; + font-size: 1vmax; + z-index: 1; } .mysparkline-overchart-value { - position: absolute; - display: block; - top: 1.1vmax; - left: 10px; - bottom: 0; - right: 0; - font-size: 5vmax; - z-index: 2; - text-shadow: #333 0px 0px 2px; + position: absolute; + display: block; + top: 1.1vmax; + left: 10px; + bottom: 0; + right: 0; + font-size: 5vmax; + z-index: 2; + text-shadow: #333 0px 0px 2px; } .myfullchart { - position: relative; - display: inline-block; - width: 100%; - height: 14vmax; - min-height: 150px; - text-align: left; + position: relative; + display: inline-block; + width: 100%; + height: 14vmax; + min-height: 150px; + text-align: left; } .mygauge-combo { - display: inline-block; + display: inline-block; } .mygauge { - position: relative; - display: block; - width: 20vw; - height: 12vw; + position: relative; + display: block; + width: 18vw; + height: 11vw; } .mygauge-button { - display: block; + display: block; } .mytitle { - padding-top: 6vw; - padding-bottom: 1vw; - text-align: center; - font-size: 2.4vw; + padding-top: 6vw; + padding-bottom: 1vw; + text-align: center; + font-size: 2.4vw; } .mysubtitle { - padding-top: 2vw; - padding-bottom: 1vw; - text-align: center; - font-size: 1.8vw; + padding-top: 2vw; + padding-bottom: 1vw; + text-align: center; + font-size: 1.8vw; } .mycontent { - text-align: center; - font-size: 1.5vw; + text-align: center; + font-size: 1.5vw; } @media only screen and (min-width : 992px) { - .container { - width: 90%; - } + .container { + width: 90%; + } } @media only screen and (max-width : 992px) { - .container { - width: 100%; - } + .container { + width: 100%; + } } - </style> + </style> </head> <body style="text-align: center;"> <div class="container"> - <div style="text-align: center; font-size: 13vw; height: 14vw;"> - <b>netdata</b> - </div> - <div style="text-align: center; font-size: 2vw; height: 2.5vw;"> - real-time performance monitoring - </div> - <div style="width:80%; text-align: right; font-size: 2.7vw;"> - <strong>scaled out</strong>! - </div> - <div class="mytitle"> - pick a <b>netdata</b> demo server - </div> - <div class="mycontent"> - these demo servers show what you will get by installing <b>netdata</b> - </div> - - <div style="width: 100%; text-align: center; padding-top: 2vw;"> - <div style="width: 100%; text-align: center;"> - - <div class="mygauge-combo"> - <div class="mygauge"> - <div data-netdata="netdata.requests" - data-host="//london.my-netdata.io" - data-title="EU - London" - data-chart-library="gauge" - data-width="100%" - data-after="-300" - data-points="300" - data-colors="#558855" - ></div> - </div> - <div class="mygauge-button"> - <br/> <br/> - <button type="button" class="btn btn-default" data-toggle="button" aria-pressed="false" autocomplete="off" onclick="window.location='//london.my-netdata.io/default.html'" style="font-size: 1.5vw;">Enter London!</button> - <div style="font-size: 1vw;"> - Donated by DigitalOcean.com - </div> - </div> - </div> - <div class="mygauge-combo"> - <div class="mygauge"> - <div data-netdata="netdata.requests" - data-host="//atlanta.my-netdata.io" - data-title="US - Atlanta" - data-chart-library="gauge" - data-width="100%" - data-after="-300" - data-points="300" - data-colors="#AA5555" - ></div> - </div> - <div class="mygauge-button"> - <br/> <br/> - <button type="button" class="btn btn-default" data-toggle="button" aria-pressed="false" autocomplete="off" onclick="window.location='//atlanta.my-netdata.io/default.html'" style="font-size: 1.5vw;">Enter Atlanta!</button> - <div style="font-size: 1vw;"> - Donated by CDN77.com - </div> - </div> - </div> - <div class="mygauge-combo"> - <div class="mygauge"> - <div data-netdata="netdata.requests" - data-host="//athens.my-netdata.io" - data-title="EU - Greece" - data-chart-library="gauge" - data-width="100%" - data-after="-300" - data-points="300" - data-colors="#5555AA" - ></div> - </div> - <div class="mygauge-button"> - <br/> <br/> - <button type="button" class="btn btn-default" data-toggle="button" aria-pressed="false" autocomplete="off" onclick="window.location='//athens.my-netdata.io/default.html'" style="font-size: 1.5vw;">Come to Greece!</button> - <div style="font-size: 0.8vw;"> - - </div> - </div> - </div> - </div> - </div> - - <div class="mytitle"> - this page is a custom <b>netdata</b> dashboard - </div> - <div class="mycontent"> - charts are coming from 3 servers, in parallel - <br/> - the servers are not aware of this multi-server dashboard, - <br/> - each server is not aware of the other 2 servers, - <br/> - but on this dashboard <b>they are one</b>! - </div> - <div style="padding-top: 1vw; width: 100%; text-align: center; font-size: 1.5vw;"> - <i class="fa fa-comment" aria-hidden="true"></i> - hover on a chart below, or drag it to show the past - <b>the others will follow</b>! - <br/> - double click on a chart to reset them all - </div> - - <div class="mytitle"> - our <code>ngnix</code> performance - </div> - <div class="mycontent"> - (we proxy netdata through nginx, on the demo sites) - </div> - - <!-- Nav tabs --> - <ul class="nav nav-tabs" role="tablist" style="padding-top: 1vw;"> - <li role="presentation" class="active"><a href="#nginx_requests" aria-controls="nginx_requests" role="tab" data-toggle="tab">Requests</a></li> - <li role="presentation"><a href="#nginx_connections" aria-controls="nginx_connections" role="tab" data-toggle="tab">Connections</a></li> - </ul> - - <!-- Tab panes --> - <div class="tab-content"> - <div role="tabpanel" class="tab-pane active" id="nginx_requests"> - <div class="mysparkline"> - <div class="mysparkline-overchart-label"> - <b>EU - London</b> web requests/s - </div> - <div class="mysparkline-overchart-value" id="nginx.requests.netdata" > - </div> - <div data-netdata="nginx.requests" - data-dimensions="requests" - data-host="//london.my-netdata.io" - data-chart-library="dygraph" - data-dygraph-theme="sparkline" - data-dygraph-type="area" - data-width="100%" - data-height="100%" - data-after="-300" - data-colors="#558855" - data-show-value-of-requests-at="nginx.requests.netdata" - ></div> - </div> - - <div class="mysparkline"> - <div class="mysparkline-overchart-label"> - <b>US - Atlanta</b> web requests/s - </div> - <div class="mysparkline-overchart-value" id="nginx.requests.netdata2" > - </div> - <div data-netdata="nginx.requests" - data-dimensions="requests" - data-host="//atlanta.my-netdata.io" - data-chart-library="dygraph" - data-dygraph-theme="sparkline" - data-dygraph-type="area" - data-width="100%" - data-height="100%" - data-after="-300" - data-colors="#AA5555" - data-show-value-of-requests-at="nginx.requests.netdata2" - ></div> - </div> - - <div class="mysparkline"> - <div class="mysparkline-overchart-label"> - <b>EU - Greece</b> web requests/s - </div> - <div class="mysparkline-overchart-value" id="nginx.requests.netdata3" > - </div> - <div data-netdata="nginx.requests" - data-dimensions="requests" - data-host="//athens.my-netdata.io" - data-chart-library="dygraph" - data-dygraph-theme="sparkline" - data-dygraph-type="area" - data-width="100%" - data-height="100%" - data-after="-300" - data-colors="#5555AA" - data-show-value-of-requests-at="nginx.requests.netdata3" - ></div> - </div> - </div> - - <div role="tabpanel" class="tab-pane" id="nginx_connections"> - <div class="mysparkline"> - <div class="mysparkline-overchart-label"> - <b>EU - London</b> active connections - </div> - <div class="mysparkline-overchart-value" id="nginx.connections.netdata1" > - </div> - <div data-netdata="nginx.connections" - data-dimensions="active" - data-host="//london.my-netdata.io" - data-chart-library="dygraph" - data-dygraph-theme="sparkline" - data-dygraph-type="area" - data-width="100%" - data-height="100%" - data-after="-300" - data-colors="#558855" - data-show-value-of-active-at="nginx.connections.netdata1" - ></div> - </div> - - <div class="mysparkline"> - <div class="mysparkline-overchart-label"> - <b>US - Atlanta</b> active connections - </div> - <div class="mysparkline-overchart-value" id="nginx.connections.netdata2" > - </div> - <div data-netdata="nginx.connections" - data-dimensions="active" - data-host="//atlanta.my-netdata.io" - data-chart-library="dygraph" - data-dygraph-theme="sparkline" - data-dygraph-type="area" - data-width="100%" - data-height="100%" - data-after="-300" - data-colors="#AA5555" - data-show-value-of-active-at="nginx.connections.netdata2" - ></div> - </div> - - <div class="mysparkline"> - <div class="mysparkline-overchart-label"> - <b>EU - Greece</b> active connections - </div> - <div class="mysparkline-overchart-value" id="nginx.connections.netdata3" > - </div> - <div data-netdata="nginx.connections" - data-dimensions="active" - data-host="//athens.my-netdata.io" - data-chart-library="dygraph" - data-dygraph-theme="sparkline" - data-dygraph-type="area" - data-width="100%" - data-height="100%" - data-after="-300" - data-colors="#5555AA" - data-show-value-of-active-at="nginx.connections.netdata3" - ></div> - </div> - </div> - </div> - - <div style="width: 100%; text-align: right; font-size: 1vw;"> - <i class="fa fa-comment" aria-hidden="true"></i> these charts are draggable and touchable, double click them to reset them - </div> - - - <div class="mytitle"> - bandwidth consumption on the demo sites - </div> - <div class="mycontent"> - Linux QoS is configured by <a href="https://github.com/firehol/netdata/wiki/You-should-install-QoS-on-all-your-servers">FireQOS</a> - </div> - - <!-- Nav tabs --> - <ul class="nav nav-tabs" role="tablist" style="padding-top: 1vw;"> - <li role="presentation" class="active"><a href="#outbout" aria-controls="outbout" role="tab" data-toggle="tab">Outbound</a></li> - <li role="presentation"><a href="#inbound" aria-controls="inbound" role="tab" data-toggle="tab">Inbound</a></li> - </ul> - - <!-- Tab panes --> - <div class="tab-content"> - <div role="tabpanel" class="tab-pane active" id="outbout"> - <div class="myfullchart"> - <div data-netdata="tc.world_out" - data-host="//london.my-netdata.io" - data-chart-library="dygraph" - data-title="EU - London, traffic we send per service" - data-width="100%" - data-height="100%" - data-after="-300" - ></div> - </div> - - <div class="myfullchart"> - <div data-netdata="tc.world_out" - data-host="//atlanta.my-netdata.io" - data-chart-library="dygraph" - data-title="US - Atlanta, traffic we send per service" - data-width="100%" - data-height="100%" - data-after="-300" - ></div> - - </div> - - <div class="myfullchart"> - <div data-netdata="tc.world_out" - data-host="//athens.my-netdata.io" - data-chart-library="dygraph" - data-title="EU - Greece, traffic we send per service" - data-width="100%" - data-height="100%" - data-after="-300" - ></div> - </div> - </div> - <div role="tabpanel" class="tab-pane" id="inbound"> - <div class="myfullchart"> - <div data-netdata="tc.world_in" - data-host="//london.my-netdata.io" - data-chart-library="dygraph" - data-title="EU - London, traffic we receive per service" - data-width="100%" - data-height="100%" - data-after="-300" - ></div> - - </div> - - <div class="myfullchart"> - <div data-netdata="tc.world_in" - data-host="//atlanta.my-netdata.io" - data-chart-library="dygraph" - data-title="US - Atlanta, traffic we receive per service" - data-width="100%" - data-height="100%" - data-after="-300" - ></div> - - </div> - - <div class="myfullchart"> - <div data-netdata="tc.world_in" - data-host="//athens.my-netdata.io" - data-chart-library="dygraph" - data-title="EU - Greece, traffic we receive per service" - data-width="100%" - data-height="100%" - data-after="-300" - ></div> - </div> - </div> - </div> - <div style="width: 100%; text-align: right; font-size: 1vw;"> - <i class="fa fa-comment" aria-hidden="true"></i> <i>these legends are interactive and the charts are resizable here ^^^</i> - </div> - - <div class="mytitle"> - DDoS protection performance on the demo sites - </div> - <div class="mycontent"> - iptables SYNPROXY configured by <a href="https://github.com/firehol/netdata/wiki/Monitoring-SYNPROXY">FireHOL</a> - </div> - - <div style="padding-top: 4vw; width: 100%; text-align: center; font-size: 1.5vw;"> - - <div class="mysparkline"> - <div class="mysparkline-overchart-label"> - <b>EU - London</b>, TCP SYN packets/s received - </div> - <div class="mysparkline-overchart-value" id="netfilter.synproxy_syn_received.netdata1" > - </div> - <div data-netdata="netfilter.synproxy_syn_received" - data-dimensions="received" - data-host="//london.my-netdata.io" - data-chart-library="dygraph" - data-dygraph-theme="sparkline" - data-dygraph-type="area" - data-width="100%" - data-height="100%" - data-after="-300" - data-colors="#558855" - data-show-value-of-received-at="netfilter.synproxy_syn_received.netdata1" - ></div> - </div> - - <div class="mysparkline"> - <div class="mysparkline-overchart-label"> - <b>US - Atlanta</b>, TCP SYN packets/s received - </div> - <div class="mysparkline-overchart-value" id="netfilter.synproxy_syn_received.netdata2" > - </div> - <div data-netdata="netfilter.synproxy_syn_received" - data-dimensions="received" - data-host="//atlanta.my-netdata.io" - data-chart-library="dygraph" - data-dygraph-theme="sparkline" - data-dygraph-type="area" - data-width="100%" - data-height="100%" - data-after="-300" - data-colors="#885555" - data-show-value-of-received-at="netfilter.synproxy_syn_received.netdata2" - ></div> - </div> - - <div class="mysparkline"> - <div class="mysparkline-overchart-label"> - <b>EU - Greece</b>, TCP SYN packets/s received - </div> - <div class="mysparkline-overchart-value" id="netfilter.synproxy_syn_received.netdata3" > - </div> - <div data-netdata="netfilter.synproxy_syn_received" - data-dimensions="received" - data-host="//athens.my-netdata.io" - data-chart-library="dygraph" - data-dygraph-theme="sparkline" - data-dygraph-type="area" - data-width="100%" - data-height="100%" - data-after="-300" - data-colors="#555588" - data-show-value-of-received-at="netfilter.synproxy_syn_received.netdata3" - ></div> - </div> - </div> - <div style="width: 100%; text-align: right; font-size: 1vw;"> - <i class="fa fa-comment" aria-hidden="true"></i> <i>did you notice the decimal numbers? - <br/>netdata interpolates collected values at second boundaries, with nanosecond detail!</i> - </div> - - - <div class="mytitle"> - CPU Utilization of the demo sites - </div> - - <div style="padding-top: 1vw;"> - <div class="myfullchart"> - <div data-netdata="system.cpu" - data-host="//london.my-netdata.io" - data-chart-library="dygraph" - data-title="EU - London, CPU Usage" - data-width="100%" - data-height="100%" - data-after="-300" - ></div> - </div> - - <div class="myfullchart"> - <div data-netdata="system.cpu" - data-host="//atlanta.my-netdata.io" - data-chart-library="dygraph" - data-title="US - Atlanta, CPU Usage" - data-width="100%" - data-height="100%" - data-after="-300" - ></div> - </div> - - <div class="myfullchart"> - <div data-netdata="system.cpu" - data-host="//athens.my-netdata.io" - data-chart-library="dygraph" - data-title="EU - Greece, CPU Usage" - data-width="100%" - data-height="100%" - data-after="-300" - ></div> - </div> - </div> - <div style="width: 100%; text-align: right; font-size: 1vw;"> - <i class="fa fa-comment" aria-hidden="true"></i> <i>what is using so much CPU? - <br/>The site <a href="//iplists.firehol.org/">iplists.firehol.org</a> is maintained by FireHOL - the CPU is used for comparing security IP Lists.</i> - </div> - - <div class="mytitle"> - CPU Usage of the netdata user - </div> - <div class="mycontent"> - netdata monitors <b>users</b>, <b>user groups</b>, <b>applications (process trees)</b> - <br/> - and <b>containers</b> (<code>lxc</code>, <code>docker</code>, etc.) - </div> - - <div style="padding-top: 4vw; width: 100%; text-align: center; font-size: 1.5vw;"> - - <div class="mysparkline"> - <div class="mysparkline-overchart-label"> - <b>EU - London</b>, CPU % of a single core - </div> - <div class="mysparkline-overchart-value" id="users.cpu.netdata1" > - </div> - <div data-netdata="users.cpu" - data-dimensions="netdata" - data-host="//london.my-netdata.io" - data-chart-library="dygraph" - data-dygraph-theme="sparkline" - data-dygraph-type="area" - data-width="100%" - data-height="100%" - data-after="-300" - data-colors="#558855" - data-show-value-of-netdata-at="users.cpu.netdata1" - ></div> - </div> - - <div class="mysparkline"> - <div class="mysparkline-overchart-label"> - <b>US - Atlanta</b>, CPU % of a single core - </div> - <div class="mysparkline-overchart-value" id="users.cpu.netdata2" > - </div> - <div data-netdata="users.cpu" - data-dimensions="netdata" - data-host="//atlanta.my-netdata.io" - data-chart-library="dygraph" - data-dygraph-theme="sparkline" - data-dygraph-type="area" - data-width="100%" - data-height="100%" - data-after="-300" - data-colors="#885555" - data-show-value-of-netdata-at="users.cpu.netdata2" - ></div> - </div> - - <div class="mysparkline"> - <div class="mysparkline-overchart-label"> - <b>EU - Greece</b>, CPU % of a single core - </div> - <div class="mysparkline-overchart-value" id="users.cpu.netdata3" > - </div> - <div data-netdata="users.cpu" - data-dimensions="netdata" - data-host="//athens.my-netdata.io" - data-chart-library="dygraph" - data-dygraph-theme="sparkline" - data-dygraph-type="area" - data-width="100%" - data-height="100%" - data-after="-300" - data-colors="#555588" - data-show-value-of-netdata-at="users.cpu.netdata3" - ></div> - </div> - </div> - <div style="width: 100%; text-align: right; font-size: 1vw;"> - <i class="fa fa-comment" aria-hidden="true"></i> <i>this utilization is about the whole netdata process tree and the percentage is of <b>a single core</b>! - <br/>including <b>BASH</b> plugins (it monitors <code>mysql</code> on the demo sites), <b>node.js</b> plugins (it monitors <code>bind9</code> on the demo sites), etc. - <br/>and including the chart refreshes for the dashboards of all viewers.</i> - </div> - - <div style="padding-top: 6vw; width: 100%; text-align: center; font-size: 2vw;"> - want to know more? - <br/> - jump to <a href="https://github.com/firehol/netdata/">the netdata page at github</a> - <br/> - it needs just 3 mins to be installed on your servers! - <br/> - - </div> + <div style="text-align: center; font-size: 13vw; height: 14vw;"> + <b>netdata</b> + </div> + <div style="text-align: center; font-size: 2vw; height: 2.5vw;"> + real-time performance monitoring + </div> + <div style="width:80%; text-align: right; font-size: 2.7vw;"> + <strong>scaled out</strong>! + </div> + <div class="mytitle"> + pick a <b>netdata</b> demo server + </div> + <div class="mycontent"> + these demo servers show what you will get by installing <b>netdata</b> + </div> + + <div style="width: 100%; text-align: center; padding-top: 2vw;"> + <div style="width: 100%; text-align: center;"> + + <div class="mygauge-combo"> + <div class="mygauge"> + <div data-netdata="netdata.requests" + data-host="//london.my-netdata.io" + data-title="EU - London" + data-chart-library="gauge" + data-width="100%" + data-after="-300" + data-points="300" + data-colors="#558855" + ></div> + </div> + <div class="mygauge-button"> + <br/> <br/> + <button type="button" class="btn btn-default" data-toggle="button" aria-pressed="false" autocomplete="off" onclick="window.location='//london.my-netdata.io/default.html'" style="font-size: 1.0vw;">Enter London!</button> + <div style="font-size: 0.8vw;"> + Donated by DigitalOcean.com + </div> + </div> + </div> + <div class="mygauge-combo"> + <div class="mygauge"> + <div data-netdata="netdata.requests" + data-host="//atlanta.my-netdata.io" + data-title="US - Atlanta" + data-chart-library="gauge" + data-width="100%" + data-after="-300" + data-points="300" + data-colors="#AA5555" + ></div> + </div> + <div class="mygauge-button"> + <br/> <br/> + <button type="button" class="btn btn-default" data-toggle="button" aria-pressed="false" autocomplete="off" onclick="window.location='//atlanta.my-netdata.io/default.html'" style="font-size: 1.0vw;">Enter Atlanta!</button> + <div style="font-size: 0.8vw;"> + Donated by CDN77.com + </div> + </div> + </div> + <div class="mygauge-combo"> + <div class="mygauge"> + <div data-netdata="netdata.requests" + data-host="//sanfrancisco.netdata.rocks" + data-title="US - California" + data-chart-library="gauge" + data-width="100%" + data-after="-300" + data-points="300" + data-colors="#5555AA" + ></div> + </div> + <div class="mygauge-button"> + <br/> <br/> + <button type="button" class="btn btn-default" data-toggle="button" aria-pressed="false" autocomplete="off" onclick="window.location='//sanfrancisco.netdata.rocks/default.html'" style="font-size: 1.0vw;">Enter California!</button> + <div style="font-size: 0.8vw;"> + Donated by DigitalOcean.com + </div> + </div> + </div> + <div class="mygauge-combo"> + <div class="mygauge"> + <div data-netdata="netdata.requests" + data-host="//toronto.netdata.rocks" + data-title="Canada" + data-chart-library="gauge" + data-width="100%" + data-after="-300" + data-points="300" + data-colors="#885588" + ></div> + </div> + <div class="mygauge-button"> + <br/> <br/> + <button type="button" class="btn btn-default" data-toggle="button" aria-pressed="false" autocomplete="off" onclick="window.location='//toronto.netdata.rocks/default.html'" style="font-size: 1.0vw;">Enter Canada!</button> + <div style="font-size: 0.8vw;"> + Donated by DigitalOcean.com + </div> + </div> + </div> + <br/> <br/> + <div class="mygauge-combo"> + <div class="mygauge"> + <div data-netdata="netdata.requests" + data-host="//frankfurt.netdata.rocks" + data-title="EU - Germany" + data-chart-library="easypiechart" + data-width="75%" + data-after="-300" + data-points="300" + data-colors="#AAAA55" + ></div> + </div> + <div class="mygauge-button"> + <br/> <br/> + <button type="button" class="btn btn-default" data-toggle="button" aria-pressed="false" autocomplete="off" onclick="window.location='//frankfurt.netdata.rocks/default.html'" style="font-size: 1.0vw;">Enter Germany!</button> + <div style="font-size: 0.8vw;"> + Donated by DigitalOcean.com + </div> + </div> + </div> + <div class="mygauge-combo"> + <div class="mygauge"> + <div data-netdata="netdata.requests" + data-host="//newyork.netdata.rocks" + data-title="US - New York" + data-chart-library="easypiechart" + data-width="75%" + data-after="-300" + data-points="300" + data-colors="#BB5533" + ></div> + </div> + <div class="mygauge-button"> + <br/> <br/> + <button type="button" class="btn btn-default" data-toggle="button" aria-pressed="false" autocomplete="off" onclick="window.location='//newyork.netdata.rocks/default.html'" style="font-size: 1.0vw;">Enter New York!</button> + <div style="font-size: 0.8vw;"> + Donated by DigitalOcean.com + </div> + </div> + </div> + <div class="mygauge-combo"> + <div class="mygauge"> + <div data-netdata="netdata.requests" + data-host="//singapore.netdata.rocks" + data-title="Signapore" + data-chart-library="easypiechart" + data-width="75%" + data-after="-300" + data-points="300" + data-colors="#5588BB" + ></div> + </div> + <div class="mygauge-button"> + <br/> <br/> + <button type="button" class="btn btn-default" data-toggle="button" aria-pressed="false" autocomplete="off" onclick="window.location='//singapore.netdata.rocks/default.html'" style="font-size: 1.0vw;">Enter Singapore!</button> + <div style="font-size: 0.8vw;"> + Donated by DigitalOcean.com + </div> + </div> + </div> + <div class="mygauge-combo"> + <div class="mygauge"> + <div data-netdata="netdata.requests" + data-host="//bangalore.netdata.rocks" + data-title="India" + data-chart-library="easypiechart" + data-width="75%" + data-after="-300" + data-points="300" + data-colors="#BB55BB" + ></div> + </div> + <div class="mygauge-button"> + <br/> <br/> + <button type="button" class="btn btn-default" data-toggle="button" aria-pressed="false" autocomplete="off" onclick="window.location='//bangalore.netdata.rocks/default.html'" style="font-size: 1.0vw;">Enter India!</button> + <div style="font-size: 0.8vw;"> + Donated by DigitalOcean.com + </div> + </div> + </div> + </div> + </div> + + <div class="mytitle"> + this page is a custom <b>netdata</b> dashboard + </div> + <div class="mycontent"> + charts are coming from 8 servers, in parallel + <br/> + the servers are not aware of this multi-server dashboard, + <br/> + each server is not aware of the other servers, + <br/> + but on this dashboard <b>they are one</b>! + </div> + <div style="padding-top: 1vw; width: 100%; text-align: center; font-size: 1.5vw;"> + <i class="fa fa-comment" aria-hidden="true"></i> + hover on a chart below, or drag it to show the past - <b>the others will follow</b>! + <br/> + double click on a chart to reset them all + </div> + + <div class="mytitle"> + our <code>nginx</code> performance + </div> + <div class="mycontent"> + (we proxy netdata through nginx, on the demo sites) + </div> + + <!-- Nav tabs --> + <ul class="nav nav-tabs" role="tablist" style="padding-top: 1vw;"> + <li role="presentation" class="active"><a href="#nginx_requests" aria-controls="nginx_requests" role="tab" data-toggle="tab">Requests</a></li> + <li role="presentation"><a href="#nginx_connections" aria-controls="nginx_connections" role="tab" data-toggle="tab">Connections</a></li> + </ul> + + <!-- Tab panes --> + <div class="tab-content"> + <div role="tabpanel" class="tab-pane active" id="nginx_requests"> + <div class="mysparkline"> + <div class="mysparkline-overchart-label"> + <b>EU - London</b> web requests/s + </div> + <div class="mysparkline-overchart-value" id="nginx_local.requests.netdata" > + </div> + <div data-netdata="nginx_local.requests" + data-dimensions="requests" + data-host="//london.my-netdata.io" + data-chart-library="dygraph" + data-dygraph-theme="sparkline" + data-dygraph-type="area" + data-width="100%" + data-height="100%" + data-after="-300" + data-colors="#558855" + data-show-value-of-requests-at="nginx_local.requests.netdata" + ></div> + </div> + + <div class="mysparkline"> + <div class="mysparkline-overchart-label"> + <b>US - Atlanta</b> web requests/s + </div> + <div class="mysparkline-overchart-value" id="nginx_local.requests.netdata2" > + </div> + <div data-netdata="nginx_local.requests" + data-dimensions="requests" + data-host="//atlanta.my-netdata.io" + data-chart-library="dygraph" + data-dygraph-theme="sparkline" + data-dygraph-type="area" + data-width="100%" + data-height="100%" + data-after="-300" + data-colors="#AA5555" + data-show-value-of-requests-at="nginx_local.requests.netdata2" + ></div> + </div> + + <div class="mysparkline"> + <div class="mysparkline-overchart-label"> + <b>US - California</b> web requests/s + </div> + <div class="mysparkline-overchart-value" id="nginx_local.requests.netdata3" > + </div> + <div data-netdata="nginx_local.requests" + data-dimensions="requests" + data-host="//sanfrancisco.netdata.rocks" + data-chart-library="dygraph" + data-dygraph-theme="sparkline" + data-dygraph-type="area" + data-width="100%" + data-height="100%" + data-after="-300" + data-colors="#5555AA" + data-show-value-of-requests-at="nginx_local.requests.netdata3" + ></div> + </div> + + <div class="mysparkline"> + <div class="mysparkline-overchart-label"> + <b>Canada</b> web requests/s + </div> + <div class="mysparkline-overchart-value" id="nginx_local.requests.netdata4" > + </div> + <div data-netdata="nginx_local.requests" + data-dimensions="requests" + data-host="//toronto.netdata.rocks" + data-chart-library="dygraph" + data-dygraph-theme="sparkline" + data-dygraph-type="area" + data-width="100%" + data-height="100%" + data-after="-300" + data-colors="#885588" + data-show-value-of-requests-at="nginx_local.requests.netdata4" + ></div> + </div> + </div> + + <div role="tabpanel" class="tab-pane" id="nginx_connections"> + <div class="mysparkline"> + <div class="mysparkline-overchart-label"> + <b>EU - London</b> active connections + </div> + <div class="mysparkline-overchart-value" id="nginx_local.connections.netdata1" > + </div> + <div data-netdata="nginx_local.connections" + data-dimensions="active" + data-host="//london.my-netdata.io" + data-chart-library="dygraph" + data-dygraph-theme="sparkline" + data-dygraph-type="area" + data-width="100%" + data-height="100%" + data-after="-300" + data-colors="#558855" + data-show-value-of-active-at="nginx_local.connections.netdata1" + ></div> + </div> + + <div class="mysparkline"> + <div class="mysparkline-overchart-label"> + <b>US - Atlanta</b> active connections + </div> + <div class="mysparkline-overchart-value" id="nginx_local.connections.netdata2" > + </div> + <div data-netdata="nginx_local.connections" + data-dimensions="active" + data-host="//atlanta.my-netdata.io" + data-chart-library="dygraph" + data-dygraph-theme="sparkline" + data-dygraph-type="area" + data-width="100%" + data-height="100%" + data-after="-300" + data-colors="#AA5555" + data-show-value-of-active-at="nginx_local.connections.netdata2" + ></div> + </div> + + <div class="mysparkline"> + <div class="mysparkline-overchart-label"> + <b>US - California</b> active connections + </div> + <div class="mysparkline-overchart-value" id="nginx_local.connections.netdata3" > + </div> + <div data-netdata="nginx_local.connections" + data-dimensions="active" + data-host="//sanfrancisco.netdata.rocks" + data-chart-library="dygraph" + data-dygraph-theme="sparkline" + data-dygraph-type="area" + data-width="100%" + data-height="100%" + data-after="-300" + data-colors="#5555AA" + data-show-value-of-active-at="nginx_local.connections.netdata3" + ></div> + </div> + + <div class="mysparkline"> + <div class="mysparkline-overchart-label"> + <b>Canada</b> active connections + </div> + <div class="mysparkline-overchart-value" id="nginx_local.connections.netdata4" > + </div> + <div data-netdata="nginx_local.connections" + data-dimensions="active" + data-host="//toronto.netdata.rocks" + data-chart-library="dygraph" + data-dygraph-theme="sparkline" + data-dygraph-type="area" + data-width="100%" + data-height="100%" + data-after="-300" + data-colors="#885588" + data-show-value-of-active-at="nginx_local.connections.netdata4" + ></div> + </div> + </div> + </div> + + <div style="width: 100%; text-align: right; font-size: 1vw;"> + <i class="fa fa-comment" aria-hidden="true"></i> these charts are draggable and touchable, double click them to reset them + </div> + + + <div class="mytitle"> + bandwidth consumption on the demo sites + </div> + <div class="mycontent"> + Linux QoS is configured by <a href="https://github.com/firehol/netdata/wiki/You-should-install-QoS-on-all-your-servers">FireQOS</a> + </div> + + <!-- Nav tabs --> + <ul class="nav nav-tabs" role="tablist" style="padding-top: 1vw;"> + <li role="presentation" class="active"><a href="#outbout" aria-controls="outbout" role="tab" data-toggle="tab">Outbound</a></li> + <li role="presentation"><a href="#inbound" aria-controls="inbound" role="tab" data-toggle="tab">Inbound</a></li> + </ul> + + <!-- Tab panes --> + <div class="tab-content"> + <div role="tabpanel" class="tab-pane active" id="outbout"> + <div class="myfullchart"> + <div data-netdata="tc.world_out" + data-host="//london.my-netdata.io" + data-chart-library="dygraph" + data-title="EU - London, traffic we send per service" + data-width="100%" + data-height="100%" + data-after="-300" + ></div> + </div> + + <div class="myfullchart"> + <div data-netdata="tc.world_out" + data-host="//atlanta.my-netdata.io" + data-chart-library="dygraph" + data-title="US - Atlanta, traffic we send per service" + data-width="100%" + data-height="100%" + data-after="-300" + ></div> + + </div> + + <div class="myfullchart"> + <div data-netdata="tc.world_out" + data-host="//sanfrancisco.netdata.rocks" + data-chart-library="dygraph" + data-title="US - California, traffic we send per service" + data-width="100%" + data-height="100%" + data-after="-300" + ></div> + </div> + + <div class="myfullchart"> + <div data-netdata="tc.world_out" + data-host="//toronto.netdata.rocks" + data-chart-library="dygraph" + data-title="Canada, traffic we send per service" + data-width="100%" + data-height="100%" + data-after="-300" + ></div> + </div> + </div> + + <div role="tabpanel" class="tab-pane" id="inbound"> + <div class="myfullchart"> + <div data-netdata="tc.world_in" + data-host="//london.my-netdata.io" + data-chart-library="dygraph" + data-title="EU - London, traffic we receive per service" + data-width="100%" + data-height="100%" + data-after="-300" + ></div> + + </div> + + <div class="myfullchart"> + <div data-netdata="tc.world_in" + data-host="//atlanta.my-netdata.io" + data-chart-library="dygraph" + data-title="US - Atlanta, traffic we receive per service" + data-width="100%" + data-height="100%" + data-after="-300" + ></div> + + </div> + + <div class="myfullchart"> + <div data-netdata="tc.world_in" + data-host="//sanfrancisco.netdata.rocks" + data-chart-library="dygraph" + data-title="US - California, traffic we receive per service" + data-width="100%" + data-height="100%" + data-after="-300" + ></div> + </div> + + <div class="myfullchart"> + <div data-netdata="tc.world_in" + data-host="//toronto.netdata.rocks" + data-chart-library="dygraph" + data-title="Canada, traffic we receive per service" + data-width="100%" + data-height="100%" + data-after="-300" + ></div> + </div> + </div> + </div> + <div style="width: 100%; text-align: right; font-size: 1vw;"> + <i class="fa fa-comment" aria-hidden="true"></i> <i>these legends are interactive and the charts are resizable here ^^^</i> + </div> + + <div class="mytitle"> + DDoS protection performance on the demo sites + </div> + <div class="mycontent"> + iptables SYNPROXY configured by <a href="https://github.com/firehol/netdata/wiki/Monitoring-SYNPROXY">FireHOL</a> + </div> + + <div style="padding-top: 4vw; width: 100%; text-align: center; font-size: 1.5vw;"> + + <div class="mysparkline"> + <div class="mysparkline-overchart-label"> + <b>EU - London</b>, TCP SYN packets/s received + </div> + <div class="mysparkline-overchart-value" id="netfilter.synproxy_syn_received.netdata1" > + </div> + <div data-netdata="netfilter.synproxy_syn_received" + data-dimensions="received" + data-host="//london.my-netdata.io" + data-chart-library="dygraph" + data-dygraph-theme="sparkline" + data-dygraph-type="area" + data-width="100%" + data-height="100%" + data-after="-300" + data-colors="#558855" + data-show-value-of-received-at="netfilter.synproxy_syn_received.netdata1" + ></div> + </div> + + <div class="mysparkline"> + <div class="mysparkline-overchart-label"> + <b>US - Atlanta</b>, TCP SYN packets/s received + </div> + <div class="mysparkline-overchart-value" id="netfilter.synproxy_syn_received.netdata2" > + </div> + <div data-netdata="netfilter.synproxy_syn_received" + data-dimensions="received" + data-host="//atlanta.my-netdata.io" + data-chart-library="dygraph" + data-dygraph-theme="sparkline" + data-dygraph-type="area" + data-width="100%" + data-height="100%" + data-after="-300" + data-colors="#885555" + data-show-value-of-received-at="netfilter.synproxy_syn_received.netdata2" + ></div> + </div> + + <div class="mysparkline"> + <div class="mysparkline-overchart-label"> + <b>US - California</b>, TCP SYN packets/s received + </div> + <div class="mysparkline-overchart-value" id="netfilter.synproxy_syn_received.netdata3" > + </div> + <div data-netdata="netfilter.synproxy_syn_received" + data-dimensions="received" + data-host="//sanfrancisco.netdata.rocks" + data-chart-library="dygraph" + data-dygraph-theme="sparkline" + data-dygraph-type="area" + data-width="100%" + data-height="100%" + data-after="-300" + data-colors="#555588" + data-show-value-of-received-at="netfilter.synproxy_syn_received.netdata3" + ></div> + </div> + + <div class="mysparkline"> + <div class="mysparkline-overchart-label"> + <b>Canada</b>, TCP SYN packets/s received + </div> + <div class="mysparkline-overchart-value" id="netfilter.synproxy_syn_received.netdata4" > + </div> + <div data-netdata="netfilter.synproxy_syn_received" + data-dimensions="received" + data-host="//toronto.netdata.rocks" + data-chart-library="dygraph" + data-dygraph-theme="sparkline" + data-dygraph-type="area" + data-width="100%" + data-height="100%" + data-after="-300" + data-colors="#885588" + data-show-value-of-received-at="netfilter.synproxy_syn_received.netdata4" + ></div> + </div> + </div> + <div style="width: 100%; text-align: right; font-size: 1vw;"> + <i class="fa fa-comment" aria-hidden="true"></i> <i>did you notice the decimal numbers? + <br/>netdata interpolates collected values at second boundaries, with nanosecond detail!</i> + </div> + + + <div class="mytitle"> + CPU Utilization of the demo sites + </div> + + <div style="padding-top: 1vw;"> + <div class="myfullchart"> + <div data-netdata="system.cpu" + data-host="//london.my-netdata.io" + data-chart-library="dygraph" + data-title="EU - London, CPU Usage" + data-width="100%" + data-height="100%" + data-after="-300" + data-dygraph-valuerange="[0, 100]" + ></div> + </div> + + <div class="myfullchart"> + <div data-netdata="system.cpu" + data-host="//atlanta.my-netdata.io" + data-chart-library="dygraph" + data-title="US - Atlanta, CPU Usage" + data-width="100%" + data-height="100%" + data-after="-300" + data-dygraph-valuerange="[0, 100]" + ></div> + </div> + + <div class="myfullchart"> + <div data-netdata="system.cpu" + data-host="//sanfrancisco.netdata.rocks" + data-chart-library="dygraph" + data-title="US - California, CPU Usage" + data-width="100%" + data-height="100%" + data-after="-300" + data-dygraph-valuerange="[0, 100]" + ></div> + </div> + + <div class="myfullchart"> + <div data-netdata="system.cpu" + data-host="//toronto.netdata.rocks" + data-chart-library="dygraph" + data-title="Canada, CPU Usage" + data-width="100%" + data-height="100%" + data-after="-300" + data-dygraph-valuerange="[0, 100]" + ></div> + </div> + </div> + <div style="width: 100%; text-align: right; font-size: 1vw;"> + <i class="fa fa-comment" aria-hidden="true"></i> <i>what is using so much CPU? + <br/>The site <a href="//iplists.firehol.org/">iplists.firehol.org</a> is maintained by FireHOL - the CPU is used for comparing security IP Lists.</i> + </div> + + <div class="mytitle"> + Netdata performance + </div> + <div class="mycontent"> + netdata monitors <b>users</b>, <b>user groups</b>, <b>applications (process trees)</b> + <br/> + <b>containers</b> (<code>lxc</code>, <code>docker</code>, etc.) and SNMP devices. + </div> + + <!-- Nav tabs --> + <ul class="nav nav-tabs" role="tablist" style="padding-top: 1vw;"> + <li role="presentation" class="active"><a href="#netdata_cpu" aria-controls="netdata_cpu" role="tab" data-toggle="tab">CPU</a></li> + <li role="presentation"><a href="#netdata_avgtime" aria-controls="netdata_avgtime" role="tab" data-toggle="tab">Average Response Time</a></li> + </ul> + + <!-- Tab panes --> + <div class="tab-content"> + <div role="tabpanel" class="tab-pane active" id="netdata_cpu"> + <div class="mysparkline"> + <div class="mysparkline-overchart-label"> + <b>EU - London</b>, CPU % of a single core + </div> + <div class="mysparkline-overchart-value" id="users.cpu.netdata1" > + </div> + <div data-netdata="apps.cpu" + data-dimensions="netdata" + data-host="//london.my-netdata.io" + data-chart-library="dygraph" + data-dygraph-theme="sparkline" + data-dygraph-type="area" + data-width="100%" + data-height="100%" + data-after="-300" + data-colors="#558855" + data-show-value-of-netdata-at="users.cpu.netdata1" + ></div> + </div> + + <div class="mysparkline"> + <div class="mysparkline-overchart-label"> + <b>US - Atlanta</b>, CPU % of a single core + </div> + <div class="mysparkline-overchart-value" id="users.cpu.netdata2" > + </div> + <div data-netdata="apps.cpu" + data-dimensions="netdata" + data-host="//atlanta.my-netdata.io" + data-chart-library="dygraph" + data-dygraph-theme="sparkline" + data-dygraph-type="area" + data-width="100%" + data-height="100%" + data-after="-300" + data-colors="#885555" + data-show-value-of-netdata-at="users.cpu.netdata2" + ></div> + </div> + + <div class="mysparkline"> + <div class="mysparkline-overchart-label"> + <b>US - California</b>, CPU % of a single core + </div> + <div class="mysparkline-overchart-value" id="users.cpu.netdata3" > + </div> + <div data-netdata="apps.cpu" + data-dimensions="netdata" + data-host="//sanfrancisco.netdata.rocks" + data-chart-library="dygraph" + data-dygraph-theme="sparkline" + data-dygraph-type="area" + data-width="100%" + data-height="100%" + data-after="-300" + data-colors="#555588" + data-show-value-of-netdata-at="users.cpu.netdata3" + ></div> + </div> + + <div class="mysparkline"> + <div class="mysparkline-overchart-label"> + <b>Toronto</b>, CPU % of a single core + </div> + <div class="mysparkline-overchart-value" id="users.cpu.netdata4" > + </div> + <div data-netdata="apps.cpu" + data-dimensions="netdata" + data-host="//toronto.netdata.rocks" + data-chart-library="dygraph" + data-dygraph-theme="sparkline" + data-dygraph-type="area" + data-width="100%" + data-height="100%" + data-after="-300" + data-colors="#885588" + data-show-value-of-netdata-at="users.cpu.netdata4" + ></div> + </div> + + <div style="width: 100%; text-align: right; font-size: 1vw;"> + <i class="fa fa-comment" aria-hidden="true"></i> <i>this utilization is about the whole netdata process tree and the percentage is of <b>a single core</b>! + <br/>including <b>BASH</b> plugins (it monitors <code>mysql</code> on the demo sites), <b>node.js</b> plugins (it monitors <code>bind9</code> on the demo sites), etc. + <br/>and including the chart refreshes for the dashboards of all viewers.</i> + </div> + </div> + + <div role="tabpanel" class="tab-pane" id="netdata_avgtime"> + <div class="mysparkline"> + <div class="mysparkline-overchart-label"> + <b>EU - London</b>, API average response time in milliseconds + </div> + <div class="mysparkline-overchart-value" id="netdata.response_time1" > + </div> + <div data-netdata="netdata.response_time" + data-host="//london.my-netdata.io" + data-chart-library="dygraph" + data-dygraph-theme="sparkline" + data-dygraph-type="area" + data-width="100%" + data-height="100%" + data-after="-300" + data-colors="#558855 #356835" + data-show-value-of-average-at="netdata.response_time1" + ></div> + </div> + + <div class="mysparkline"> + <div class="mysparkline-overchart-label"> + <b>US - Atlanta</b>, API average response time in milliseconds + </div> + <div class="mysparkline-overchart-value" id="netdata.response_time2" > + </div> + <div data-netdata="netdata.response_time" + data-host="//atlanta.my-netdata.io" + data-chart-library="dygraph" + data-dygraph-theme="sparkline" + data-dygraph-type="area" + data-width="100%" + data-height="100%" + data-after="-300" + data-colors="#885555 #683535" + data-show-value-of-average-at="netdata.response_time2" + ></div> + </div> + + <div class="mysparkline"> + <div class="mysparkline-overchart-label"> + <b>US - California</b>, API average response time in milliseconds + </div> + <div class="mysparkline-overchart-value" id="netdata.response_time3" > + </div> + <div data-netdata="netdata.response_time" + data-host="//sanfrancisco.netdata.rocks" + data-chart-library="dygraph" + data-dygraph-theme="sparkline" + data-dygraph-type="area" + data-width="100%" + data-height="100%" + data-after="-300" + data-colors="#555588 #353568" + data-show-value-of-average-at="netdata.response_time3" + ></div> + </div> + + <div class="mysparkline"> + <div class="mysparkline-overchart-label"> + <b>Canada</b>, API average response time in milliseconds + </div> + <div class="mysparkline-overchart-value" id="netdata.response_time4" > + </div> + <div data-netdata="netdata.response_time" + data-host="//toronto.netdata.rocks" + data-chart-library="dygraph" + data-dygraph-theme="sparkline" + data-dygraph-type="area" + data-width="100%" + data-height="100%" + data-after="-300" + data-colors="#885588 #683568" + data-show-value-of-average-at="netdata.response_time4" + ></div> + </div> + + <div style="width: 100%; text-align: right; font-size: 1vw;"> + <i class="fa fa-comment" aria-hidden="true"></i> <i>netdata is really <b>fast</b> (the values are milliseconds!) + <br/> + These values include everything, from the reception of the first byte to the dispatch of the last, including gzip compression. + <br/> + Values above 2-3ms are usually chart refreshes of charts with several dimensions, charts with very long durations (zoomed out), or file transfers. + </i> + </div> + </div> + </div> + + <div style="padding-top: 6vw; width: 100%; text-align: center; font-size: 2vw;"> + want to know more? + <br/> + jump to <a href="https://github.com/firehol/netdata/">the netdata page at github</a> + <br/> + it needs just 3 mins to be installed on your servers! + <br/> + + </div> </div> </body> <script> - // google analytics when this is used for the home page of the demo sites - // you don't need this if you customize this dashboard for your needs - setTimeout(function() { - (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ - (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), - m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) - })(window,document,'script','https://www.google-analytics.com/analytics.js','ga'); - - ga('create', 'UA-64295674-3', 'auto'); - ga('send', 'pageview'); - }, 2000); + // google analytics when this is used for the home page of the demo sites + // you don't need this if you customize this dashboard for your needs + setTimeout(function() { + (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ + (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), + m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) + })(window,document,'script','https://www.google-analytics.com/analytics.js','ga'); + + ga('create', 'UA-64295674-3', 'auto'); + ga('send', 'pageview'); + }, 2000); </script> </html> diff --git a/web/images/animated.gif b/web/images/animated.gif Binary files differnew file mode 100644 index 000000000..0e94a20ba --- /dev/null +++ b/web/images/animated.gif diff --git a/web/images/post.png b/web/images/post.png Binary files differnew file mode 100644 index 000000000..239af429a --- /dev/null +++ b/web/images/post.png diff --git a/web/index.html b/web/index.html index 9cc2b4bbe..3e6c22190 100644 --- a/web/index.html +++ b/web/index.html @@ -1,1607 +1,2048 @@ -<!DOCTYPE html> +<!DOCTYPE html> <html lang="en"> <head> - <title>netdata dashboard</title> - - <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> - <meta charset="utf-8"> - <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1"> - <meta name="viewport" content="width=device-width, initial-scale=1"> - <meta name="apple-mobile-web-app-capable" content="yes"> - <meta name="apple-mobile-web-app-status-bar-style" content="black-translucent"> - <meta name="author" content="costa@tsaousis.gr"> - - <link rel="shortcut icon" href="images/seo-performance-multi-size.ico"> - - <link rel="apple-touch-icon" href="images/seo-performance-72.png"> - <link rel="apple-touch-icon" sizes="72x72" href="images/seo-performance-72.png"> - <link rel="apple-touch-icon" sizes="114x114" href="images/seo-performance-114.png"> - - <link rel="icon" type="image/png" sizes="512x512" href="images/seo-performance-512.png"> - <link rel="icon" type="image/png" sizes="256x256" href="images/seo-performance-256.png"> - <link rel="icon" type="image/png" sizes="128x128" href="images/seo-performance-128.png"> - <link rel="icon" type="image/png" sizes="64x64" href="images/seo-performance-64.png"> - <link rel="icon" type="image/png" sizes="48x48" href="images/seo-performance-48.png"> - <link rel="icon" type="image/png" sizes="32x32" href="images/seo-performance-32.png"> - <link rel="icon" type="image/png" sizes="24x24" href="images/seo-performance-24.png"> - <link rel="icon" type="image/png" sizes="16x16" href="images/seo-performance-16.png"> - - <style> - - /* prevent body from hiding under the navbar */ - body { - padding-top: 50px; - } - - .modal-wide .modal-dialog { - width: 80%; - } - - /* fix # anchors scrolling under the navbar - https://github.com/twbs/bootstrap/issues/1768#issuecomment-46519033 - */ - h1 { - position: relative; - z-index: -1; - } - h2 { - position: relative; - z-index: -2; - } - h1:before, h2:before { - display: block; - content: " "; - margin-top: -70px; - height: 70px; - visibility: hidden; - } - - .p { - display: block; - margin-top: 15px; - } - - .option-row, - .option-control { - vertical-align: top; - padding: 10px; - padding-top: 30px; - padding-left: 30px; - } - - .option-info { - padding: 10px; - } - - .chart-message { - display: block; - margin-top: 10px; - } - - .container { - width: 90% !important; - } - - #masthead h1 { - /*font-size: 30px;*/ - line-height: 1; - padding-top: 30px; - } - - #masthead .well { - margin-top:4%; - } - - /* fix the navbar shifting when a modal is open */ - /* https://github.com/twbs/bootstrap/issues/14040#issuecomment-159891033 */ - body.modal-open{ - width: 100% !important; - padding-right: 0 !important; -/* overflow-y: scroll !important; */ -/* position: fixed !important;*/ - overflow: visible; - } - - /* - * Side navigation - * - * Scrollspy and affixed enhanced navigation to highlight sections and secondary - * sections of docs content. - */ - - .affix { - position: static; - top: 70px !important; - /*width: 220px;*/ - } - - .affix-top { - /*width: 220px;*/ - } - - .dashboard-sidebar { - max-height: calc(100% - 70px) !important; - overflow-y: auto; - /*width: 220px !important;*/ - } - - /* By default it's not affixed in mobile views, so undo that */ - .dashboard-sidebar.affix { - position: static; - } - - @media (min-width: 768px) { - .dashboard-sidebar { - padding-left: 20px; - } - } - - /* First level of nav */ - .dashboard-sidenav { - margin-top: 20px; - margin-bottom: 20px; - } - - /* All levels of nav */ - .dashboard-sidebar .nav > li > a { - display: block; - padding: 4px 20px; - font-size: 13px; - font-weight: 500; - color: #767676; - } - .dashboard-sidebar .nav > li > a:hover, - .dashboard-sidebar .nav > li > a:focus { - padding-left: 19px; - color: #563d7c; - text-decoration: none; - background-color: transparent; - border-left: 1px solid #563d7c; - } - .dashboard-sidebar .nav > .active > a, - .dashboard-sidebar .nav > .active:hover > a, - .dashboard-sidebar .nav > .active:focus > a { - padding-left: 18px; - font-weight: bold; - color: #563d7c; - background-color: transparent; - border-left: 2px solid #563d7c; - } - - /* Nav: second level (shown on .active) */ - .dashboard-sidebar .nav .nav { - display: none; /* Hide by default, but at >768px, show it */ - padding-bottom: 10px; - } - .dashboard-sidebar .nav .nav > li > a { - padding-top: 1px; - padding-bottom: 1px; - padding-left: 30px; - font-size: 12px; - font-weight: normal; - } - .dashboard-sidebar .nav .nav > li > a:hover, - .dashboard-sidebar .nav .nav > li > a:focus { - padding-left: 29px; - } - .dashboard-sidebar .nav .nav > .active > a, - .dashboard-sidebar .nav .nav > .active:hover > a, - .dashboard-sidebar .nav .nav > .active:focus > a { - padding-left: 28px; - font-weight: 500; - } - - .dropdown-menu { - min-width: 200px; - } - .dropdown-menu.columns-2 { - margin: 0; - padding: 0; - width: 400px; - } - .dropdown-menu li a { - padding: 5px 15px; - font-weight: 300; - } - .dropdown-menu.multi-column { - overflow-x: hidden; - } - .multi-column-dropdown { - list-style: none; - padding: 0; - } - .multi-column-dropdown li a { - display: block; - clear: both; - line-height: 1.428571429; - white-space: normal; - } - .multi-column-dropdown li a:hover { - text-decoration: none; - color: #f5f5f5; - background-color: #262626; - } - .scrollable-menu { - height: auto; - max-height: 80vh; - overflow-x: hidden; - } - - /* Back to top (hidden on mobile) */ - .back-to-top, - .dashboard-theme-toggle { - display: none; - padding: 4px 10px; - margin-top: 10px; - margin-left: 10px; - font-size: 12px; - font-weight: 500; - color: #999; - } - .back-to-top:hover, - .dashboard-theme-toggle:hover { - color: #563d7c; - text-decoration: none; - } - .dashboard-theme-toggle { - margin-top: 0; - } - - @media (min-width: 768px) { - .back-to-top, - .dashboard-theme-toggle { - display: block; - } - - /* Widen the fixed sidebar */ - .dashboard-sidebar.affix, - .dashboard-sidebar.affix-top, - .dashboard-sidebar.affix-bottom { - width: 200px !important; - } - - .dashboard-sidebar.affix { - position: fixed; /* Undo the static from mobile first approach */ - top: 20px; - } - - .dashboard-sidebar.affix-bottom { - position: absolute; /* Undo the static from mobile first approach */ - } - - .dashboard-sidebar.affix-bottom .dashboard-sidenav, - .dashboard-sidebar.affix .dashboard-sidenav { - margin-top: 0; - margin-bottom: 0; - } - } - - /* Show and affix the side nav when space allows it */ - @media (min-width: 992px) { - .dashboard-sidebar .nav > .active > ul { - display: block; - } - - /* Widen the fixed sidebar */ - .dashboard-sidebar.affix, - .dashboard-sidebar.affix-top, - .dashboard-sidebar.affix-bottom { - width: 200px !important; - } - .dashboard-sidebar.affix { - position: fixed; /* Undo the static from mobile first approach */ - top: 20px; - } - .dashboard-sidebar.affix-bottom { - position: absolute; /* Undo the static from mobile first approach */ - } - .dashboard-sidebar.affix-bottom .dashboard-sidenav, - .dashboard-sidebar.affix .dashboard-sidenav { - margin-top: 0; - margin-bottom: 0; - } - } - - @media (min-width: 1200px) { - /* Widen the fixed sidebar again */ - .dashboard-sidebar.affix-bottom, - .dashboard-sidebar.affix-top, - .dashboard-sidebar.affix { - width: 263px; - } - } - </style> - - <!-- you can set your netdata server globally, by ucommenting this --> - <!-- you can also give a different server per chart, with the attribute: data-host="http://netdata.server:19999" --> - <!-- <script> netdataServer = "http://box:19999"; </script> --> - - <!-- check which theme to use --> - <script> - function loadLocalStorage(name) { - var ret = null; - - try { - if(typeof Storage !== "undefined" && typeof localStorage === 'object') - ret = localStorage.getItem(name); - } - catch(error) { - ; - } - - if(typeof ret === 'undefined' || ret === null) - return null; - - return ret; - } - - function saveLocalStorage(name, value) { - try { - if(typeof Storage !== "undefined" && typeof localStorage === 'object') { - localStorage.setItem(name, value.toString()); - return true; - } - } - catch(error) { - ; - } - - return false; - } - - function getTheme(def) { - var ret = loadLocalStorage('netdataTheme'); - if(typeof ret === 'undefined' || ret === null || ret === 'undefined') - return def; - else - return ret; - } - - var netdataTheme = getTheme('slate'); - - function setTheme(theme) { - if(theme === netdataTheme) return false; - - return saveLocalStorage('netdataTheme', theme); - } - - var netdataRegistryCallback = function(urls_array) { - var el = ''; - var a1 = ''; - var found = 0; - - if(urls_array) { - function name_comparator_desc(a, b) { - if (a.name > b.name) return -1; - if (a.name < b.name) return 1; - return 0; - } - - var urls = urls_array.sort(name_comparator_desc); - var len = urls.length; - while(len--) { - var u = urls[len]; - - var status = "enabled"; - found++; - - if(u.guid === NETDATA.registry.machine_guid) - status = "disabled" - - el += '<li id="registry_server_' + u.guid + '" class="' + status + '"><a href="' + u.url + '">' + u.name + '</a></li>'; - a1 += '<li id="registry_action_' + u.guid + '"><a href="#" onclick="deleteRegistryModalHandler(\'' + u.guid + '\',\'' + u.name + '\',\'' + u.url + '\'); return false;"><i class="fa fa-trash-o" aria-hidden="true" style="color: #999;"></i></a></li>'; - } - } - - if(!found) { - if(urls) - el += '<li><a href="https://github.com/firehol/netdata/wiki/mynetdata-menu-item" style="color: #666;" target="_blank">your netdata server list is empty...</a></li>'; - else - el += '<li><a href="https://github.com/firehol/netdata/wiki/mynetdata-menu-item" style="color: #666;" target="_blank">failed to contact the registry...</a></li>'; - - a1 += '<li><a href="#"> </a></li>'; - - el += '<li role="separator" class="divider"></li>' + - '<li><a href="//london.netdata.rocks/default.html">EU - London (DigitalOcean.com)</a></li>' + - '<li><a href="//atlanta.netdata.rocks/default.html">US - Atlanta (CDN77.com)</a></li>' + - '<li><a href="//athens.netdata.rocks/default.html">EU - Athens</a></li>'; - a1 += '<li role="separator" class="divider"></li>' + - '<li><a href="#"> </a></li>' + - '<li><a href="#"> </a></li>'+ - '<li><a href="#"> </a></li>'; - } - - el += '<li role="separator" class="divider"></li>'; - a1 += '<li role="separator" class="divider"></li>'; - - el += '<li><a href="https://github.com/firehol/netdata/wiki/mynetdata-menu-item" style="color: #999;" target="_blank">What is this?</a></li>'; - a1 += '<li><a href="#" style="color: #999;" onclick="switchRegistryModalHandler(); return false;"><i class="fa fa-cog" aria-hidden="true" style="color: #999;"></i></a></li>' - - document.getElementById('mynetdata_servers').innerHTML = el; - document.getElementById('mynetdata_servers2').innerHTML = el; - document.getElementById('mynetdata_actions1').innerHTML = a1; - }; - - </script> - - <!-- load the dashboard manager - it will do the rest --> - <script type="text/javascript" src="dashboard.js?v37"></script> -</head> + <title>netdata dashboard</title> + <meta name="application-name" content="netdata"> + + <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> + <meta charset="utf-8"> + <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1"> + <meta name="viewport" content="width=device-width, initial-scale=1"> + <meta name="apple-mobile-web-app-capable" content="yes"> + <meta name="apple-mobile-web-app-status-bar-style" content="black-translucent"> + <meta name="author" content="costa@tsaousis.gr"> + + <link rel="shortcut icon" href="images/seo-performance-multi-size.ico"> + + <link rel="apple-touch-icon" href="images/seo-performance-72.png"> + <link rel="apple-touch-icon" sizes="72x72" href="images/seo-performance-72.png"> + <link rel="apple-touch-icon" sizes="114x114" href="images/seo-performance-114.png"> + + <link rel="icon" type="image/png" sizes="512x512" href="images/seo-performance-512.png"> + <link rel="icon" type="image/png" sizes="256x256" href="images/seo-performance-256.png"> + <link rel="icon" type="image/png" sizes="128x128" href="images/seo-performance-128.png"> + <link rel="icon" type="image/png" sizes="64x64" href="images/seo-performance-64.png"> + <link rel="icon" type="image/png" sizes="48x48" href="images/seo-performance-48.png"> + <link rel="icon" type="image/png" sizes="32x32" href="images/seo-performance-32.png"> + <link rel="icon" type="image/png" sizes="24x24" href="images/seo-performance-24.png"> + <link rel="icon" type="image/png" sizes="16x16" href="images/seo-performance-16.png"> + + <meta property="og:locale" content="en_US" /> + <meta property="og:image" content="http://my-netdata.io/images/post.png"/> + <meta property="og:url" content="http://my-netdata.io/"/> + <meta property="og:type" content="website"/> + <meta property="og:site_name" content="netdata"/> + <meta property="og:title" content="netdata - real-time performance monitoring, done right!"/> + <meta property="og:description" content="Stunning real-time dashboards, blazingly fast and extremely interactive. Zero configuration, zero dependencies, zero maintenance." /> + + <style> + + /* prevent body from hiding under the navbar */ + body { + padding-top: 50px; + } + + .loadOverlay { + position: absolute; + top: 0px; + left: 0px; + width: 100%; + height:100%; + z-index: 2000; + font-size: 10vh; + font-family: sans-serif; + padding: 40vh 0 40vh 0; + font-weight: bold; + text-align: center; + } + + .modal-wide .modal-dialog { + width: 80%; + } + + /* fix # anchors scrolling under the navbar + https://github.com/twbs/bootstrap/issues/1768#issuecomment-46519033 + */ + h1 { + position: relative; + z-index: -1; + } + h2 { + position: relative; + z-index: -2; + } + h1:before, h2:before { + display: block; + content: " "; + margin-top: -70px; + height: 70px; + visibility: hidden; + } + + .p { + display: block; + margin-top: 15px; + } + + .option-row, + .option-control { + vertical-align: top; + padding: 10px; + padding-top: 30px; + padding-left: 30px; + } + + .option-info { + padding: 10px; + } + + .chart-message { + display: block; + margin-top: 10px; + } + + .container { + width: 90% !important; + } + + #masthead h1 { + /*font-size: 30px;*/ + line-height: 1; + padding-top: 30px; + } + + #masthead .well { + margin-top:4%; + } + + /* fix the navbar shifting when a modal is open */ + /* https://github.com/twbs/bootstrap/issues/14040#issuecomment-159891033 */ + body.modal-open{ + width: 100% !important; + padding-right: 0 !important; +/* overflow-y: scroll !important; */ +/* position: fixed !important;*/ + overflow: visible; + } + + /* + * Side navigation + * + * Scrollspy and affixed enhanced navigation to highlight sections and secondary + * sections of docs content. + */ + + .affix { + position: static; + top: 70px !important; + /*width: 220px;*/ + } + + .affix-top { + /*width: 220px;*/ + } + .dashboard-sidebar { + max-height: calc(100% - 70px) !important; + overflow-y: auto; + /*width: 220px !important;*/ + } + + /* By default it's not affixed in mobile views, so undo that */ + .dashboard-sidebar.affix { + position: static; + } + + @media (min-width: 768px) { + .dashboard-sidebar { + padding-left: 20px; + } + } + + /* First level of nav */ + .dashboard-sidenav { + margin-top: 20px; + margin-bottom: 20px; + } + + /* All levels of nav */ + .dashboard-sidebar .nav > li > a { + display: block; + padding: 4px 20px; + font-size: 13px; + font-weight: 500; + color: #767676; + } + .dashboard-sidebar .nav > li > a:hover, + .dashboard-sidebar .nav > li > a:focus { + padding-left: 19px; + color: #563d7c; + text-decoration: none; + background-color: transparent; + border-left: 1px solid #563d7c; + } + .dashboard-sidebar .nav > .active > a, + .dashboard-sidebar .nav > .active:hover > a, + .dashboard-sidebar .nav > .active:focus > a { + padding-left: 18px; + font-weight: bold; + color: #563d7c; + background-color: transparent; + border-left: 2px solid #563d7c; + } + + /* Nav: second level (shown on .active) */ + .dashboard-sidebar .nav .nav { + display: none; /* Hide by default, but at >768px, show it */ + padding-bottom: 10px; + } + .dashboard-sidebar .nav .nav > li > a { + padding-top: 1px; + padding-bottom: 1px; + padding-left: 30px; + font-size: 12px; + font-weight: normal; + } + .dashboard-sidebar .nav .nav > li > a:hover, + .dashboard-sidebar .nav .nav > li > a:focus { + padding-left: 29px; + } + .dashboard-sidebar .nav .nav > .active > a, + .dashboard-sidebar .nav .nav > .active:hover > a, + .dashboard-sidebar .nav .nav > .active:focus > a { + padding-left: 28px; + font-weight: 500; + } + + .dropdown-menu { + min-width: 200px; + } + .dropdown-menu.columns-2 { + margin: 0; + padding: 0; + width: 400px; + } + .dropdown-menu li a { + padding: 5px 15px; + font-weight: 300; + } + .dropdown-menu.multi-column { + overflow-x: hidden; + } + .multi-column-dropdown { + list-style: none; + padding: 0; + } + .multi-column-dropdown li a { + display: block; + clear: both; + line-height: 1.428571429; + white-space: normal; + } + .multi-column-dropdown li a:hover { + text-decoration: none; + color: #f5f5f5; + background-color: #262626; + } + .scrollable-menu { + height: auto; + max-height: 80vh; + overflow-x: hidden; + } + + /* Back to top (hidden on mobile) */ + .back-to-top, + .dashboard-theme-toggle { + display: none; + padding: 4px 10px; + margin-top: 10px; + margin-left: 10px; + font-size: 12px; + font-weight: 500; + color: #999; + } + .back-to-top:hover, + .dashboard-theme-toggle:hover { + color: #563d7c; + text-decoration: none; + } + .dashboard-theme-toggle { + margin-top: 0; + } + + @media (min-width: 768px) { + .back-to-top, + .dashboard-theme-toggle { + display: block; + } + + /* Widen the fixed sidebar */ + .dashboard-sidebar.affix, + .dashboard-sidebar.affix-top, + .dashboard-sidebar.affix-bottom { + width: 200px !important; + } + + .dashboard-sidebar.affix { + position: fixed; /* Undo the static from mobile first approach */ + top: 20px; + } + + .dashboard-sidebar.affix-bottom { + position: absolute; /* Undo the static from mobile first approach */ + } + + .dashboard-sidebar.affix-bottom .dashboard-sidenav, + .dashboard-sidebar.affix .dashboard-sidenav { + margin-top: 0; + margin-bottom: 0; + } + } + + /* Show and affix the side nav when space allows it */ + @media (min-width: 992px) { + .dashboard-sidebar .nav > .active > ul { + display: block; + } + + /* Widen the fixed sidebar */ + .dashboard-sidebar.affix, + .dashboard-sidebar.affix-top, + .dashboard-sidebar.affix-bottom { + width: 200px !important; + } + .dashboard-sidebar.affix { + position: fixed; /* Undo the static from mobile first approach */ + top: 20px; + } + .dashboard-sidebar.affix-bottom { + position: absolute; /* Undo the static from mobile first approach */ + } + .dashboard-sidebar.affix-bottom .dashboard-sidenav, + .dashboard-sidebar.affix .dashboard-sidenav { + margin-top: 0; + margin-bottom: 0; + } + } + + @media (min-width: 1200px) { + /* Widen the fixed sidebar again */ + .dashboard-sidebar.affix-bottom, + .dashboard-sidebar.affix-top, + .dashboard-sidebar.affix { + width: 263px; + } + } + </style> + + <!-- check which theme to use --> + <script type="text/javascript"> + // -------------------------------------------------------------------- + // urlOptions + + var urlOptions = { + hash: '#', + theme: null, + help: null, + pan_and_zoom: false, + after: 0, + before: 0, + nowelcome: 0, + hasProperty: function(property) { + // console.log('checking property ' + property + ' of type ' + typeof(this[property])); + return typeof this[property] !== 'undefined'; + } + }; + + function netdataPanAndZoomCallback(status, after, before) { + urlOptions.pan_and_zoom = status; + urlOptions.after = after; + urlOptions.before = before; + netdataHashUpdate(); + } + + function netdataHashUpdate() { + history.replaceState(null, '', netdataHash()); + } + + function netdataHash() { + var hash = urlOptions.hash; + + if(urlOptions.pan_and_zoom === true) { + hash += ';after=' + urlOptions.after.toString() + + ';before=' + urlOptions.before.toString(); + } + + if(urlOptions.theme !== null) + hash += ';theme=' + urlOptions.theme.toString(); + + if(urlOptions.help !== null) + hash += ';help=' + urlOptions.help.toString(); + + return hash; + } + + function netdataHashParse() { + var variables = document.location.hash.split(';'); + var len = variables.length; + while(len--) { + if(len !== 0) { + var p = variables[len].split('='); + if(urlOptions.hasProperty(p[0]) && typeof p[1] !== 'undefined') + urlOptions[p[0]] = p[1]; + } + else { + if(variables[len].length > 0) + urlOptions.hash = variables[len]; + } + } + + if(urlOptions.before > 0 && urlOptions.after > 0) + urlOptions.pan_and_zoom = true; + + // console.log(urlOptions); + } + + netdataHashParse(); + + // -------------------------------------------------------------------- + // check options that should be processed before loading netdata.js + + function loadLocalStorage(name) { + var ret = null; + + try { + if(typeof Storage !== "undefined" && typeof localStorage === 'object') + ret = localStorage.getItem(name); + } + catch(error) { + ; + } + + if(typeof ret === 'undefined' || ret === null) + return null; + + // console.log('loaded: ' + name.toString() + ' = ' + ret.toString()); + + return ret; + } + + function saveLocalStorage(name, value) { + // console.log('saving: ' + name.toString() + ' = ' + value.toString()); + try { + if(typeof Storage !== "undefined" && typeof localStorage === 'object') { + localStorage.setItem(name, value.toString()); + return true; + } + } + catch(error) { + ; + } + + return false; + } + + function getTheme(def) { + var ret = loadLocalStorage('netdataTheme'); + if(typeof ret === 'undefined' || ret === null || ret === 'undefined') + return def; + else + return ret; + } + + function setTheme(theme) { + if(theme === netdataTheme) return false; + return saveLocalStorage('netdataTheme', theme); + } + + var netdataTheme = getTheme('slate'); + var netdataShowHelp = true; + + if(urlOptions.theme !== null) { + setTheme(urlOptions.theme); + netdataTheme = urlOptions.theme; + } + else + urlOptions.theme = netdataTheme; + + if(urlOptions.help !== null) { + saveLocalStorage('options.show_help', urlOptions.help); + netdataShowHelp = urlOptions.help; + } + else { + urlOptions.help = loadLocalStorage('options.show_help'); + } + + // -------------------------------------------------------------------- + // registry call back to render my-netdata menu + + var netdataRegistryCallback = function(machines_array) { + var el = ''; + var a1 = ''; + var found = 0; + + if(machines_array) { + function name_comparator_desc(a, b) { + if (a.name > b.name) return -1; + if (a.name < b.name) return 1; + return 0; + } + + var machines = machines_array.sort(name_comparator_desc); + var len = machines.length; + while(len--) { + var u = machines[len]; + found++; + el += '<li id="registry_server_' + u.guid + '"><a class="registry_link" href="' + u.url + '" onClick="return gotoServerModalHandler(\'' + u.guid + '\');">' + u.name + '</a></li>'; + a1 += '<li id="registry_action_' + u.guid + '"><a href="#" onclick="deleteRegistryModalHandler(\'' + u.guid + '\',\'' + u.name + '\',\'' + u.url + '\'); return false;"><i class="fa fa-trash-o" aria-hidden="true" style="color: #999;"></i></a></li>'; + } + } + + if(!found) { + if(machines) + el += '<li><a href="https://github.com/firehol/netdata/wiki/mynetdata-menu-item" style="color: #666;" target="_blank">your netdata server list is empty...</a></li>'; + else + el += '<li><a href="https://github.com/firehol/netdata/wiki/mynetdata-menu-item" style="color: #666;" target="_blank">failed to contact the registry...</a></li>'; + + a1 += '<li><a href="#" onClick="return false;"> </a></li>'; + + el += '<li role="separator" class="divider"></li>' + + '<li><a href="//london.netdata.rocks/default.html">EU - London (DigitalOcean.com)</a></li>' + + '<li><a href="//atlanta.netdata.rocks/default.html">US - Atlanta (CDN77.com)</a></li>' + + '<li><a href="//athens.netdata.rocks/default.html">EU - Athens</a></li>'; + a1 += '<li role="separator" class="divider"></li>' + + '<li><a href="#"> </a></li>' + + '<li><a href="#"> </a></li>'+ + '<li><a href="#"> </a></li>'; + } + + el += '<li role="separator" class="divider"></li>'; + a1 += '<li role="separator" class="divider"></li>'; + + el += '<li><a href="https://github.com/firehol/netdata/wiki/mynetdata-menu-item" style="color: #999;" target="_blank">What is this?</a></li>'; + a1 += '<li><a href="#" style="color: #999;" onclick="switchRegistryModalHandler(); return false;"><i class="fa fa-cog" aria-hidden="true" style="color: #999;"></i></a></li>' + + document.getElementById('mynetdata_servers').innerHTML = el; + document.getElementById('mynetdata_servers2').innerHTML = el; + document.getElementById('mynetdata_actions1').innerHTML = a1; + + gotoServerInit(); + }; + + </script> + + <!-- load the dashboard manager - it will do the rest --> + <script type="text/javascript" src="dashboard.js?v42"></script> +</head> <body data-spy="scroll" data-target="#sidebar"> - <nav class="navbar navbar-default navbar-fixed-top" role="banner"> - <div class="container"> - <nav id="mynetdata_nav" class="collapse navbar-collapse navbar-left hidden-sm hidden-xs" role="navigation" style="padding-right: 20px;"> - <ul class="nav navbar-nav"> - <li class="dropdown"> - <a href="#" class="dropdown-toggle" data-toggle="dropdown" id="current_view">my-netdata <strong class="caret"></strong></a> - <ul class="dropdown-menu scrollable-menu inpagemenu multi-column columns-2" role="menu"> - <div class="row"> - <div class="col-sm-6" style="width: 85%; padding-right: 0;"> - <ul id="mynetdata_servers" class="multi-column-dropdown"> - <li><a href="#" onclck="return false;" style="color: #999;">loading...</a></li> - </ul> - </div> - <div class="col-sm-3 hidden-xs" style="width: 15%; padding-left: 0;"> - <ul id="mynetdata_actions1" class="multi-column-dropdown"> - <li style="color: #999;"> </li> - </ul> - </div> - </div> - </ul> - </li> - </ul> - </nav> - <div class="navbar-header"> - <button class="navbar-toggle" type="button" data-toggle="collapse" data-target=".navbar-collapse"> - <span class="sr-only">Toggle navigation</span> - <span class="icon-bar"></span> - <span class="icon-bar"></span> - <span class="icon-bar"></span> - </button> - <a href="/" class="navbar-brand" id="hostname">netdata</a> - </div> - <nav class="collapse navbar-collapse navbar-right" role="navigation"> - <ul class="nav navbar-nav"> - <li class="hidden-sm"><a href="#" class="btn" data-toggle="modal" data-target="#optionsModal"><i class="fa fa-cog"></i> settings</a></li> - <li class="hidden-sm"><a href="https://github.com/firehol/netdata/wiki" class="btn" target="_blank"><i class="fa fa-github"></i> community</a></li> - <li class="hidden-sm" id="updateButton"><a href="#" class="btn" data-toggle="modal" data-target="#updateModal"><i class="fa fa-cloud-download"></i> update</a></li> - <li class="hidden-sm"><a href="#" class="btn" data-toggle="modal" data-target="#helpModal"><i class="fa fa-question-circle"></i> help</a></li> - <li class="dropdown hidden-md hidden-lg hidden-xs"> - <a href="#" class="dropdown-toggle" data-toggle="dropdown" id="current_view">Menu <strong class="caret"></strong></a> - <ul class="dropdown-menu scrollable-menu inpagemenu" role="menu"> - <li><a href="#" class="btn" data-toggle="modal" data-target="#optionsModal"><i class="fa fa-cog"></i> settings</a></li> - <li><a href="https://github.com/firehol/netdata/wiki" class="btn" target="_blank"><i class="fa fa-github"></i> community</a></li> - <li><a href="#" class="btn" data-toggle="modal" data-target="#helpModal"><i class="fa fa-question-circle"></i> help</a></li> - </ul> - </li> - <li class="dropdown hidden-sm hidden-md hidden-lg"> - <a href="#" class="dropdown-toggle" data-toggle="dropdown" id="current_view">my-netdata <strong class="caret"></strong></a> - <ul id="mynetdata_servers2" class="dropdown-menu scrollable-menu inpagemenu" role="menu"> - <li><a href="#" onclck="return false;" style="color: #999;">loading...</a></li> - </ul> - </li> - </ul> - </nav> - </nav> - </div> - </nav> - - <div id="masthead" style="display: none;"> - <div class="container"> - <div class="row"> - <div class="col-md-7"> - <h1>Netdata - <p class="lead">Real-time performance monitoring, in the greatest possible detail</p> - </h1> - </div> - <div class="col-md-5"> - <div class="well well-lg"> - <div class="row"> - <div class="col-md-6"> - <b>Drag</b> charts to pan. - <b>Shift + wheel</b> on them, to zoom in and out. - <b>Double-click</b> on them, to reset. - <b>Hover</b> on them too! - </div> - <div class="col-md-6"> - <div data-netdata="system.intr" data-chart-library="dygraph" data-dygraph-theme="sparkline" data-dygraph-type="line" data-dygraph-strokewidth="3" data-dygraph-smooth="true" data-dygraph-highlightcirclesize="6" data-after="-90" data-height="60px" data-colors="#C66"></div> - </div> - </div> - </div> - </div> - </div> - </div> - </div> - - <div class="container"> - <div class="row"> - <div class="col-md-10" role="main"> - <div id="charts_div"></div> - </div> - <div class="col-md-2" role="complementary"> - <nav class="dashboard-sidebar hidden-print hidden-xs hidden-sm" id="sidebar" role="menu"></nav> - </div> - </div> - </div> - - <div id="footer" class="container" style="display: none;"> - <div class="row"> - <div class="col-md-10" role="main"> - <div class="p"> - <big><a href="https://github.com/firehol/netdata/wiki" target="_blank">netdata</a></big><br/> - <i class="fa fa-copyright"></i> Copyright 2016, Costa Tsaousis.<br/> - Released under <a href="http://www.gnu.org/licenses/gpl-3.0.en.html" target="_blank">GPL v3 or later</a>.<br/> - </div> - <div class="p"> - <small> - <a href="https://github.com/firehol/netdata/wiki" target="_blank">netdata</a> re-distributes these software tools: - - <i class="fa fa-circle"></i> The excellent <a href="http://dygraphs.com/" target="_blank">Dygraphs.com</a> web chart library, - <i class="fa fa-copyright"></i> Copyright 2009, Dan Vanderkam, <a href="http://dygraphs.com/legal.html" target="_blank">MIT License</a> - - <i class="fa fa-circle"></i> <a href="http://omnipotent.net/jquery.sparkline/" target="_blank">jQuery Sparklines</a> web chart library, - <i class="fa fa-copyright"></i> Copyright 2009-2012, Splunk Inc., <a href="http://opensource.org/licenses/BSD-3-Clause" target="_blank">New BSD License</a> - - <i class="fa fa-circle"></i> <a href="http://benpickles.github.io/peity/" target="_blank">Peity</a> web chart library, - <i class="fa fa-copyright"></i> Copyright 2009-2015, Ben Pickles, <a href="https://github.com/benpickles/peity/blob/master/MIT-LICENCE" target="_blank">MIT License</a> - - <i class="fa fa-circle"></i> <a href="https://rendro.github.io/easy-pie-chart/" target="_blank">Easy Pie Chart</a> web chart library, - <i class="fa fa-copyright"></i> Copyright 2013, Robert Fleischmann, <a href="https://github.com/rendro/easy-pie-chart/blob/master/LICENSE" target="_blank">MIT License</a> - - <i class="fa fa-circle"></i> <a href="http://bernii.github.io/gauge.js/" target="_blank">Gauge.js</a> web chart library, - <i class="fa fa-copyright"></i> Copyright, Bernard Kobos, <a href="http://bernii.github.io/gauge.js/" target="_blank">MIT License</a> - - <i class="fa fa-circle"></i> <a href="https://jquery.org/" target="_blank">jQuery</a>, - <i class="fa fa-copyright"></i> Copyright 2015, jQuery Foundation, <a href="https://jquery.org/license/" target="_blank">MIT License</a> - - <i class="fa fa-circle"></i> <a href="http://getbootstrap.com/getting-started/" target="_blank">Bootstrap</a>, - <i class="fa fa-copyright"></i> Copyright 2015, Twitter, <a href="http://getbootstrap.com/getting-started/#license-faqs" target="_blank">MIT License</a> - - <i class="fa fa-circle"></i> <a href="http://www.bootstraptoggle.com/" target="_blank">Bootstrap Toggle</a>, - <i class="fa fa-copyright"></i> Copyright (c) 2011-2014 Min Hur, The New York Times Company, <a href="https://github.com/minhur/bootstrap-toggle/blob/master/LICENSE" target="_blank">MIT License</a> - - <i class="fa fa-circle"></i> <a href="https://jamesflorentino.github.io/nanoScrollerJS/" target="_blank">NanoScroller</a>, - <i class="fa fa-copyright"></i> Copyright 2012, James Florentino, <a href="https://github.com/jamesflorentino/nanoScrollerJS/blob/master/LICENSE" target="_blank">MIT License</a> - - <i class="fa fa-circle"></i> <a href="https://github.com/marcj/css-element-queries" target="_blank">CSS Element Queries</a>, - <i class="fa fa-copyright"></i> Copyright Marc J. Schmidt, <a href="https://github.com/marcj/css-element-queries/blob/master/LICENSE" target="_blank">MIT License</a> - - <i class="fa fa-circle"></i> <a href="https://fortawesome.github.io/Font-Awesome/" target="_blank">FontAwesome</a>, - <i class="fa fa-copyright"></i> Created by Dave Gandy, Font: <a href="http://scripts.sil.org/OFL" target="_blank">SIL OFL 1.1 License</a>, CSS: <a href="http://opensource.org/licenses/mit-license.html" target="_blank">MIT License</a> - - <i class="fa fa-circle"></i> <a href="http://www.iconsdb.com/soylent-red-icons/seo-performance-icon.html" target="_blank">IconsDB.com Icons</a>, Icons provided as CC0 1.0 Universal (CC0 1.0) Public Domain Dedication. - - <i class="fa fa-circle"></i> <a href="http://morrisjs.github.io/morris.js/" target="_blank">morris.js</a>, - <i class="fa fa-copyright"></i> Copyright 2013, Olly Smith, <a href="http://morrisjs.github.io/morris.js/" target="_blank">Simplified BSD License</a> - - <i class="fa fa-circle"></i> <a href="http://raphaeljs.com/" target="_blank">Raphaël</a>, - <i class="fa fa-copyright"></i> Copyright 2008, Dmitry Baranovskiy, <a href="http://raphaeljs.com/license.html" target="_blank">MIT License</a> - - <i class="fa fa-circle"></i> <a href="http://C3js.org/" target="_blank">C3</a>, - <i class="fa fa-copyright"></i> Copyright 2013, Masayuki Tanaka, <a href="https://github.com/masayuki0812/c3/blob/master/LICENSE" target="_blank">MIT License</a> - - <i class="fa fa-circle"></i> <a href="http://D3js.org/" target="_blank">D3</a>, - <i class="fa fa-copyright"></i> Copyright 2015, Mike Bostock, <a href="http://opensource.org/licenses/BSD-3-Clause" target="_blank">BSD License</a> - - </small> - </div> - </div> - </div> - </div> - - <div class="modal fade" id="welcomeModal" tabindex="-1" role="dialog" aria-labelledby="welcomeModalLabel"> - <div class="modal-dialog modal-lg" role="document"> - <div class="modal-content"> - <div class="modal-header"> - <button type="button" class="close" data-dismiss="modal" aria-label="Close"><span aria-hidden="true">×</span></button> - <h4 class="modal-title" id="welcomeModalLabel">Welcome!</h4> - </div> - <div class="modal-body"> - <div class="p"> - <b><a href="https://github.com/firehol/netdata/wiki" target="_blank">netdata</a></b> is the fastest way to visualize metrics. It is a resource efficient, highly optimized system for collecting and visualizing any type of real-time time series data, from CPU usage, disk activity, SQL queries, API calls, web site visitors, etc. - </div> - <div class="p"> - <b><a href="https://github.com/firehol/netdata/wiki" target="_blank">netdata</a></b> tries to visualize the truth of <b>now</b>, in its <b>greatest detail</b>, so that you can get insights of what is happening now and what just happened, on your systems and applications. - </div> - <div class="p"> - To make a chart in <b><a href="https://github.com/firehol/netdata/wiki" target="_blank">netdata</a></b>, you just need a <b>number</b>. Just a number you can read somehow. <b><a href="https://github.com/firehol/netdata/wiki" target="_blank">netdata</a></b> will turn this number to a real time, interactive, web chart. For collecting these numbers, it supports <a href="https://github.com/firehol/netdata/wiki/External-Plugins" target="_blank">external plugins</a>, even <a href="https://github.com/firehol/netdata/wiki/General-Info---charts.d" target="_blank">shell</a> or <a href="https://github.com/firehol/netdata/wiki/General-Info---charts.d" target="_blank">node.js</a> plugins. Any computer program, in any language, that can print a few lines of text on its standard output, can be a netdata data collector. - </div> - <div class="p"> - <b><a href="https://github.com/firehol/netdata/wiki" target="_blank">netdata</a></b> can embed charts everywhere, like this one <div data-netdata="system.cpu" data-dimensions="system" data-after="-120" data-width="25%" data-height="15px" data-chart-library="dygraph" data-dygraph-theme="sparkline" data-show-value-of-system-at="system.cpu.system.modal.1"></div> (my CPU system usage which is <span id="system.cpu.system.modal.1" style="display: inline-block; width: 40px; text-align: right;"></span>%), - or this one <div data-netdata="ipv4.tcppackets" data-dimensions="received" data-after="-120" data-width="25%" data-height="15px" data-chart-library="dygraph" data-dygraph-theme="sparkline" data-show-value-of-received-at="ipv4.tcppackets.received.modal.1"></div> (my IPv4 received TCP packets, which are <span id="ipv4.tcppackets.received.modal.1" style="display: inline-block; width: 60px; text-align: right;"></span>/second). - </div> - <div class="p"> - You can have <b><a href="https://github.com/firehol/netdata/wiki" target="_blank">netdata</a></b> charts on your site too. Just give it a <code>div</code> and a real time chart, zoomable and draggable will appear (try it even on these tiny ones - <b>drag</b> them to pan horizontally, <b>shift + drag</b> to zoom in, on <b>chrome shift + mouse wheel</b> to zoom in/out, <b>double click</b> on them to reset them - don't be afraid of <b><a href="https://github.com/firehol/netdata/wiki" target="_blank">netdata</a></b> performance - <a href="https://github.com/firehol/netdata/wiki/Performance" target="_blank">a raspberry pi 2 can sustain 300 charts updates per second</a>!). - </div> - <div class="p"> - For more information please refer to the <b><a href="https://github.com/firehol/netdata/wiki" target="_blank">netdata wiki</a></b>. - </div> - </div> - <div class="modal-footer"> - <button type="button" class="btn btn-default" data-dismiss="modal">Close</button> - </div> - </div> - </div> - </div> - - <div class="modal fade" id="helpModal" tabindex="-1" role="dialog" aria-labelledby="helpModalLabel"> - <div class="modal-dialog modal-lg" role="document"> - <div class="modal-content"> - <div class="modal-header"> - <button type="button" class="close" data-dismiss="modal" aria-label="Close"><span aria-hidden="true">×</span></button> - <h4 class="modal-title" id="helpModalLabel">Dashboard Help</h4> - </div> - <div class="modal-body"> - - <h4>Dygraphs (line, area and stacked area charts)</h4> - - <!-- Nav tabs --> - <ul class="nav nav-tabs" role="tablist"> - <li role="presentation" class="active"><a href="#help_mouse" aria-controls="help_mouse" role="tab" data-toggle="tab">Mouse Interface</a></li> - <li role="presentation"><a href="#help_touch" aria-controls="help_touch" role="tab" data-toggle="tab">Touch Interface</a></li> - </ul> - - <!-- Tab panes --> - <div class="tab-content"> - <div role="tabpanel" class="tab-pane active" id="help_mouse"> - <div class="p"> - <h4>Mouse Over / Hover</h4> - Mouse over on a chart to show, at its legend, the values for the timestamp under the mouse (the chart will also highlight the point at the chart). - <br/> - All the other visible charts will also show and highlight their values for the same timestamp. - </div> - <hr/> - <div class="p"> - <h4>Drag Chart Contents</h4> - Drag the contents of a chart to pan it horizontally. - <br/> - All the charts will follow soon after you let the chart alone (this little delay is by design: it speeds up your browser and lets you focus on what you are exploring). - <br/> - Once a chart is panned, auto refreshing stops for all charts. To enable it again, <b>double click</b> a panned chart. - </div> - <hr/> - <div class="p"> - <h4>Double Click</h4> - Double Click a chart to reset all the charts to their default auto-refreshing state. - </div> - <hr/> - <div class="p"> - <h4>SHIFT + Drag</h4> - While pressing the shift key, click on the contents of a chart and move the mouse to select an area, to zoom in. The other charts will follow too. Zooming is performed in two phases: - <ul> - <li>The already loaded chart contents are zoomed (low resolution)</li> - <li>New data are transferred from the netdata server, to refresh the chart with possibly more detail.</li> - </ul> - Once a chart is zoomed, auto refreshing stops for all charts. To enable it again, <b>double click</b> a zoomed chart. - </div> - <hr/> - <div class="p"> - <h4>SHIFT + Mouse Wheel <small>(does not work on firefox and IE/Edge)</small></h4> - While pressing the shift key and the mouse pointer is over the contents of a chart, scroll the mouse wheel to zoom in or out. This kind of zooming is aligned to center below the mouse pointer. The other charts will follow too. - <br/> - Once a chart is zoomed, auto refreshing stops for all charts. To enable it again, <b>double click</b> a zoomed chart. - </div> - <hr/> - <div class="p"> - <h4>Legend Operations</h4> - Click on the label or value of a dimension, will select / un-select this dimension. - <br/> - You can press any of the SHIFT or CONTROL keys and then click on legend labels or values, to select / un-select multiple dimensions. - </div> - </div> - <div role="tabpanel" class="tab-pane" id="help_touch"> - <div class="p"> - <h4>Single Tap</h4> - Single Tap on the contents of a chart to show, at its legend, the values for the timestamp tapped (the chart will also highlight the point at the chart). - <br/> - All the other visible charts will also show and highlight their values for the same timestamp. - </div> - <hr/> - <div class="p"> - <h4>Drag Chart Contents</h4> - Touch and Drag the contents of a chart to pan it horizontally. - <br/> - All the charts will follow soon after you let the chart alone (this little delay is by design: it speeds up your browser and lets you focus on what you are exploring). - <br/> - Once a chart is panned, auto refreshing stops for all charts. To enable it again, <b>double tap</b> a panned chart. - </div> - <hr/> - <div class="p"> - <h4>Double Tap</h4> - Double tap a chart to reset all the charts to their default auto-refreshing state. - </div> - <hr/> - <div class="p"> - <h4>Zoom <small>(does not work on firefox and IE/Edge)</small></h4> - With two fingers, zoom in or out. - <br/> - Once a chart is zoomed, auto refreshing stops for all charts. To enable it again, <b>double click</b> a zoomed chart. - </div> - <hr/> - <div class="p"> - <h4>Legend Operations</h4> - Tap on the label or value of a dimension, will select / un-select this dimension. - </div> - </div> - </div> - </div> - <div class="modal-footer"> - <button type="button" class="btn btn-default" data-dismiss="modal">Close</button> - </div> - </div> - </div> - </div> - - <div class="modal fade" id="optionsModal" tabindex="-1" role="dialog" aria-labelledby="optionsModalLabel"> - <div class="modal-dialog modal-lg" role="document"> - <div class="modal-content"> - <div class="modal-header"> - <button type="button" class="close" data-dismiss="modal" aria-label="Close"><span aria-hidden="true">×</span></button> - <h4 class="modal-title" id="optionsModalLabel">netdata dashboard options</h4> - </div> - <div class="modal-body"> - <center> - <small style="color: #BBBBBB;">These are browser settings. Each viewer has its own. They do not affect the operation of your netdata server. - <br/> - Settings take effect immediately and are saved permanently to browser local storage (except the refresh on focus / always option). - <br/> - To reset all options (including charts sizes) to their defaults, click <a href="#" onclick="resetDashboardOptions(); return false;">here</a>.</small> - </center> - <div style="padding: 10px;"></div> - - <!-- Nav tabs --> - <ul class="nav nav-tabs" role="tablist"> - <li role="presentation" class="active"><a href="#settings_performance" aria-controls="settings_performance" role="tab" data-toggle="tab">Performance</a></li> - <li role="presentation"><a href="#settings_sync" aria-controls="settings_sync" role="tab" data-toggle="tab">Synchronization</a></li> - <li role="presentation"><a href="#settings_visual" aria-controls="settings_visual" role="tab" data-toggle="tab">Visual</a></li> - </ul> - - <!-- Tab panes --> - <div class="tab-content"> - <div role="tabpanel" class="tab-pane active" id="settings_performance"> - <form id="optionsForm1" method="get" class="form-horizontal"> - <div class="form-group"> - <table> - <tr class="option-row"> - <td class="option-control"><input id="stop_updates_when_focus_is_lost" type="checkbox" checked data-toggle="toggle" data-offstyle="danger" data-onstyle="success" data-on="On Focus" data-off="Always" data-width="110px"></td> - <td class="option-info"><strong>When to refresh the charts?</strong><br/> - <small>When set to <b>On Focus</b>, the charts will stop being updated if the page / tab does not have the focus of the user. When set to <b>Always</b>, the charts will always be refreshed. Set it to <b>On Focus</b> it to lower the CPU requirements of the browser (and extend the battery of laptops and tablets) when this page does not have your focus. Set to <b>Always</b> to work on another window (i.e. change the settings of something) and have the charts auto-refresh in this window.</small> - </td> - </tr> - <tr class="option-row"> - <td class="option-control"> - <input id="eliminate_zero_dimensions" type="checkbox" checked data-toggle="toggle" data-on="Non Zero" data-off="All" data-width="110px"> - </td> - <td class="option-info"><strong>Which dimensions to show?</strong><br/> - <small>When set to <b>Non Zero</b>, dimensions that have all their values (within the current view) set to zero will not be transferred from the netdata server (except if all dimensions of the chart are zero, in which case this setting does nothing - all dimensions are transferred and shown). When set to <b>All</b>, all dimensions will always be shown. Set it to <b>Non Zero</b> to lower the data transferred between netdata and your browser, lower the CPU requirements of your browser (fewer lines to draw) and increase the focus on the legends (fewer entries at the legends).</small> - </td> - </tr> - <tr class="option-row"> - <td class="option-control"><input id="destroy_on_hide" type="checkbox" data-toggle="toggle" data-on="Destroy" data-off="Hide" data-width="110px"></td> - <td class="option-info"><strong>How to handle hidden charts?</strong><br/> - <small>When set to <b>Destroy</b>, charts that are not in the current viewport of the browser (are above, or below the visible area of the page), will be destroyed and re-created if and when they become visible again. When set to <b>Hide</b>, the not-visible charts will be just hidden, to simplify the DOM and speed up your browser. Set it to <b>Destroy</b>, to lower the memory requirements of your browser. Set it to <b>Hide</b> for smoother page scrolling.</small> - </td> - </tr> - </table> - </div> - </form> - </div> - <div role="tabpanel" class="tab-pane" id="settings_sync"> - <form id="optionsForm2" method="get" class="form-horizontal"> - <div class="form-group"> - <table> - <tr class="option-row"> - <td class="option-control"><input id="parallel_refresher" type="checkbox" checked data-toggle="toggle" data-on="Parallel" data-off="Sequential" data-width="110px"></td> - <td class="option-info"><strong>Which chart refresh policy to use?</strong><br/> - <small>When set to <b>parallel</b>, visible charts are refreshed in parallel (all queries are sent to netdata server in parallel) and are rendered asynchronously. When set to <b>sequential</b> charts are refreshed one after another. Set it to parallel if your browser can cope with it (most modern browsers do), set it to sequential if you work on an older/slower computer.</small> - </td> - </tr> - <tr class="option-row" id="concurrent_refreshes_row"> - <td class="option-control"></td> - <td class="option-info"> - <table> - <tr class="option-row"> - <td class="option-control"> - <input id="concurrent_refreshes" type="checkbox" checked data-toggle="toggle" data-on="Resync" data-off="Best Effort" data-width="110px"> - </td> - <td class="option-info"> - <strong>Shall we re-sync chart refreshes?</strong><br/> - <small>When set to <b>Resync</b>, the dashboard will attempt to re-synchronize all the charts so that they are refreshed concurrently. When set to <b>Best Effort</b>, each chart may be refreshed with a little time difference to the others. Normally, the dashboard starts refreshing them in parallel, but depending on the speed of your computer and the network latencies, charts start having a slight time difference. Setting this to <b>Resync</b> will attempt to re-synchronize the charts on every update. Setting it to <b>Best Effort</b> may lower the pressure on your browser and the network.</small> - </td> - </tr> - </table> - </td> - </tr> - <tr class="option-row"> - <td class="option-control"><input id="sync_selection" type="checkbox" checked data-toggle="toggle" data-on="Sync" data-off="Don't Sync" data-onstyle="success" data-offstyle="danger" data-width="110px"></td> - <td class="option-info"><strong>Sync hover selection on all charts?</strong><br/> - <small>When enabled, a selection on one chart will automatically select the same time on all other visible charts and the legends of all visible charts will be updated to show the selected values. When disabled, only the chart getting the user's attention will be selected. Enable it to get better insights of the data. Disable it if you are on a very slow computer that cannot actually do it.</small> - </td> - </tr> - <tr class="option-row"> - <td class="option-control"><input id="sync_pan_and_zoom" type="checkbox" checked data-toggle="toggle" data-on="Sync" data-off="Don't Sync" data-onstyle="success" data-offstyle="danger" data-width="110px"></td> - <td class="option-info"><strong>Sync pan and zoom on all charts?</strong><br/> - <small>When enabled, pan and zoom operations on a chart will be replicated to all charts (even the ones that are not currently visible - of course only when they will become visible). When disabled, pan and zoom operations will not be propagated to other charts.</small> - </td> - </tr> - </table> - </div> - </form> - </div> - <div role="tabpanel" class="tab-pane" id="settings_visual"> - <form id="optionsForm3" method="get" class="form-horizontal"> - <div class="form-group"> - <table> - <tr class="option-row"> - <td class="option-control"><input id="netdata_theme_control" type="checkbox" checked data-toggle="toggle" data-offstyle="danger" data-onstyle="success" data-on="Dark" data-off="White" data-width="110px"></td> - <td class="option-info"><strong>Which theme to use?</strong><br/> - <small>Netdata comes with two themes: <b>Dark</b> (the default) and <b>White</b>. - <br/> - <b>Switching this will reload the dashboard</b>. - </small> - </td> - </tr> - <tr class="option-row"> - <td class="option-control"><input id="show_help" type="checkbox" checked data-toggle="toggle" data-on="Help Me" data-off="No Help" data-width="110px"></td> - <td class="option-info"><strong>Do you need help?</strong><br/> - <small>Netdata can show some help in some areas to help you use the dashboard. If all these balloons bother you, disable them using this switch. - <br/> - <b>Switching this will reload the dashboard</b>. - </small> - </td> - </tr> - <tr class="option-row"> - <td class="option-control"><input id="pan_and_zoom_data_padding" type="checkbox" checked data-toggle="toggle" data-on="Pad" data-off="Don't Pad" data-width="110px"></td> - <td class="option-info"><strong>Enable data padding when panning and zooming?</strong><br/> - <small>When set to <b>Pad</b> the charts will be padded with more data, both before and after the visible area, thus giving the impression the whole database is loaded. This padding will happen only after the first pan or zoom operation on the chart (initially all charts have only the visible data). When set to <b>Don't Pad</b> only the visible data will be transfered from the netdata server, even after the first pan and zoom operation.</small> - </td> - </tr> - <tr class="option-row"> - <td class="option-control"><input id="smooth_plot" type="checkbox" checked data-toggle="toggle" data-on="Smooth" data-off="Rough" data-width="110px"></td> - <td class="option-info"><strong>Enable Bézier lines on charts?</strong><br/> - <small>When set to <b>Smooth</b> the charts libraries that support it, will plot smooth curves instead of simple straight lines to connect the points. - <br/> - Keep in mind <a href="http://dygraphs.com" target="_blank">dygraphs</a>, the main charting library in netdata dashboards, can only smooth line charts. It cannot smooth area or stacked charts. When set to <b>Rough</b>, this setting can lower the CPU resources consumed by your browser.</small> - </td> - </tr> - </table> - </div> - </form> - </div> - </div> - </div> - <div class="modal-footer"> - <button type="button" class="btn btn-default" data-dismiss="modal">Close</button> - </div> - </div> - </div> - </div> - - - <div class="modal fade" id="updateModal" tabindex="-1" role="dialog" aria-labelledby="updateModalLabel"> - <div class="modal-dialog" role="document"> - <div class="modal-content"> - <div class="modal-header"> - <button type="button" class="close" data-dismiss="modal" aria-label="Close"><span aria-hidden="true">×</span></button> - <h4 class="modal-title" id="updateModalLabel">Update Check</h4> - </div> - <div class="modal-body"> - Your netdata version: <b><code><span id="netdataVersion">Unknown</span></code></b> - <br/> - <div style="padding: 10px;"></div> - <div id="versionCheckLog">Not checked yet. Please press the Check Now button.</div> - </div> - <div class="modal-footer"> - <a href="#" onclick="notifyForUpdate(true);" type="button" class="btn btn-default">Check Now</a> - <button type="button" class="btn btn-default" data-dismiss="modal">Close</button> - </div> - </div> - </div> - </div> - - <div class="modal fade" id="deleteRegistryModal" tabindex="-1" role="dialog" aria-labelledby="deleteRegistryModalLabel"> - <div class="modal-dialog" role="document"> - <div class="modal-content"> - <div class="modal-header"> - <button type="button" class="close" data-dismiss="modal" aria-label="Close"><span aria-hidden="true">×</span></button> - <h4 class="modal-title" id="deleteRegistryModalLabel">Delete <span id="deleteRegistryServerName"></span>?</h4> - </div> - <div class="modal-body"> - You are about to delete, from your personal list of netdata servers, the following server: - <p style="text-align: center; padding-top: 10px; padding-bottom: 10px; line-height: 2;"> - <b><span id="deleteRegistryServerName2"></span></b> - <br/> - <b><span id="deleteRegistryServerURL"></span></b> - </p> - Are you sure you want to do this? - <br/> - <div style="padding: 10px;"></div> - <small>Keep in mind, this server will be added back if and when you visit it again.</small> - <br/> - <div id="deleteRegistryResponse" style="display: block; width: 100%; text-align: center; padding-top: 20px;"></div> - </div> - <div class="modal-footer"> - <button type="button" class="btn btn-success" data-dismiss="modal">keep it</button> - <a href="#" onclick="notifyForDeleteRegistry(true); return false;" type="button" class="btn btn-danger">delete it</a> - </div> - </div> - </div> - </div> - - <div class="modal fade" id="switchRegistryModal" tabindex="-1" role="dialog" aria-labelledby="switchRegistryModalLabel"> - <div class="modal-dialog" role="document"> - <div class="modal-content"> - <div class="modal-header"> - <button type="button" class="close" data-dismiss="modal" aria-label="Close"><span aria-hidden="true">×</span></button> - <h4 class="modal-title" id="switchRegistryModalLabel">Switch Netdata Registry Identity</h4> - </div> - <div class="modal-body"> - You can copy and paste the following ID to all your browsers (e.g. work and home). - <br/> - All the browsers with the same ID will identify <b>you</b>, so please don't share this with others. - <p style="text-align: center; padding-top: 10px; padding-bottom: 10px; line-height: 2;"> - <form action="#"> - <input type="text" class="form-control" id="switchRegistryPersonGUID" placeholder="your personal ID" maxlength="36" autocomplete="off" style="text-align: center; font-size: 1.4em;"> - </form> - </p> - Either copy this ID and paste it to another browser, or paste here the ID you have taken from another browser. - <p style="padding-top: 10px;"><small> - Keep in mind that: - <ul> - <li>when you switch ID, your previous ID will be lost forever - this is irreversible.</li> - <li>both IDs (your old and the new) must list this netdata at their personal lists.</li> - <li>both IDs have to be known by the registry: <b><span id="switchRegistryURL"></span></b>.</li> - <li>to get a new ID, just clear your browser cookies.</li> - </ul> - </small></p> - <div id="switchRegistryResponse" style="display: block; width: 100%; text-align: center; padding-top: 20px;"></div> - </div> - <div class="modal-footer"> - <button type="button" class="btn btn-success" data-dismiss="modal">cancel</button> - <a href="#" onclick="notifyForSwitchRegistry(true); return false;" type="button" class="btn btn-danger">impersonate</a> - </div> - </div> - </div> - </div> - -<script> + <div id="loadOverlay" class="loadOverlay" style="background-color: #888; color: #888;"> + netdata<br/><div style="font-size: 3vh;">Real-time performance monitoring, done right!</div> + </div> + <script type="text/javascript"> + // change the loadOverlay colors ASAP to match the theme + document.getElementById('loadOverlay').style = (urlOptions.theme === 'slate')?"background-color:#272b30; color: #373b40;":"background-color:#fff; color: #ddd;"; + </script> + <nav class="navbar navbar-default navbar-fixed-top" role="banner"> + <div class="container"> + <nav id="mynetdata_nav" class="collapse navbar-collapse navbar-left hidden-sm hidden-xs" role="navigation" style="padding-right: 20px;"> + <ul class="nav navbar-nav"> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown" id="current_view">my-netdata <strong class="caret"></strong></a> + <ul class="dropdown-menu scrollable-menu inpagemenu multi-column columns-2" role="menu"> + <div class="row"> + <div class="col-sm-6" style="width: 85%; padding-right: 0;"> + <ul id="mynetdata_servers" class="multi-column-dropdown"> + <li><a href="#" onclick="return false;" style="color: #999;">loading...</a></li> + </ul> + </div> + <div class="col-sm-3 hidden-xs" style="width: 15%; padding-left: 0;"> + <ul id="mynetdata_actions1" class="multi-column-dropdown"> + <li style="color: #999;"> </li> + </ul> + </div> + </div> + </ul> + </li> + </ul> + </nav> + <div class="navbar-header"> + <button class="navbar-toggle" type="button" data-toggle="collapse" data-target=".navbar-collapse"> + <span class="sr-only">Toggle navigation</span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + </button> + <a href="/" class="navbar-brand" id="hostname" title="reload the dashboard">netdata</a> + </div> + <nav class="collapse navbar-collapse navbar-right" role="navigation"> + <ul class="nav navbar-nav"> + <li class="hidden-sm"><a href="#" class="btn" data-toggle="modal" data-target="#alarmsModal" title="alarms"><i class="fa fa-bell"></i><span id="alarms_count_badge" class="badge"></span></a></li> + <li class="hidden-sm"><a href="#" class="btn" data-toggle="modal" data-target="#optionsModal" title="dashboard settings"><i class="fa fa-cog"></i></a></li> + <li class="hidden-sm"><a href="https://github.com/firehol/netdata/wiki" class="btn" target="_blank" title="netdata community"><i class="fa fa-github"></i></a></li> + <li class="hidden-sm" id="updateButton"><a href="#" class="btn" data-toggle="modal" data-target="#updateModal" title="check for update"><i class="fa fa-cloud-download"></i><span id="update_badge" class="badge"></span></a></li> + <li class="hidden-sm"><a href="#" class="btn" data-toggle="modal" data-target="#helpModal" title="dashboard help"><i class="fa fa-question-circle"></i></a></li> + <li class="dropdown hidden-md hidden-lg hidden-xs"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown" id="current_view">Menu <strong class="caret"></strong></a> + <ul class="dropdown-menu scrollable-menu inpagemenu" role="menu"> + <li><a href="#" class="btn" data-toggle="modal" data-target="#alarmsModal"><i class="fa fa-cog"></i> alarms</a></li> + <li><a href="#" class="btn" data-toggle="modal" data-target="#optionsModal"><i class="fa fa-cog"></i> settings</a></li> + <li><a href="https://github.com/firehol/netdata/wiki" class="btn" target="_blank"><i class="fa fa-github"></i> community</a></li> + <li><a href="#" class="btn" data-toggle="modal" data-target="#helpModal"><i class="fa fa-question-circle"></i> help</a></li> + </ul> + </li> + <li class="dropdown hidden-sm hidden-md hidden-lg"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown" id="current_view">my-netdata <strong class="caret"></strong></a> + <ul id="mynetdata_servers2" class="dropdown-menu scrollable-menu inpagemenu" role="menu"> + <li><a href="#" onclick="return false;" style="color: #999;">loading...</a></li> + </ul> + </li> + </ul> + </nav> + </nav> + </div> + </nav> + + <div id="masthead" style="display: none;"> + <div class="container"> + <div class="row"> + <div class="col-md-7"> + <h1>Netdata + <p class="lead">Real-time performance monitoring, in the greatest possible detail</p> + </h1> + </div> + <div class="col-md-5"> + <div class="well well-lg"> + <div class="row"> + <div class="col-md-6"> + <b>Drag</b> charts to pan. + <b>Shift + wheel</b> on them, to zoom in and out. + <b>Double-click</b> on them, to reset. + <b>Hover</b> on them too! + </div> + <div class="col-md-6"> + <div data-netdata="system.intr" data-chart-library="dygraph" data-dygraph-theme="sparkline" data-dygraph-type="line" data-dygraph-strokewidth="3" data-dygraph-smooth="true" data-dygraph-highlightcirclesize="6" data-after="-90" data-height="60px" data-colors="#C66"></div> + </div> + </div> + </div> + </div> + </div> + </div> + </div> + + <div class="container"> + <div class="row"> + <div class="col-md-10" role="main"> + <div id="charts_div"></div> + </div> + <div class="col-md-2" role="complementary"> + <nav class="dashboard-sidebar hidden-print hidden-xs hidden-sm" id="sidebar" role="menu"></nav> + </div> + </div> + </div> + + <div id="footer" class="container" style="display: none;"> + <div class="row"> + <div class="col-md-10" role="main"> + <div class="p"> + <big><a href="https://github.com/firehol/netdata/wiki" target="_blank">netdata</a></big><br/> + <i class="fa fa-copyright"></i> Copyright 2016, Costa Tsaousis.<br/> + Released under <a href="http://www.gnu.org/licenses/gpl-3.0.en.html" target="_blank">GPL v3 or later</a>.<br/> + </div> + <div class="p"> + <small> + <a href="https://github.com/firehol/netdata/wiki" target="_blank">netdata</a> re-distributes these software tools: + + <i class="fa fa-circle"></i> The excellent <a href="http://dygraphs.com/" target="_blank">Dygraphs.com</a> web chart library, + <i class="fa fa-copyright"></i> Copyright 2009, Dan Vanderkam, <a href="http://dygraphs.com/legal.html" target="_blank">MIT License</a> + + <i class="fa fa-circle"></i> <a href="http://omnipotent.net/jquery.sparkline/" target="_blank">jQuery Sparklines</a> web chart library, + <i class="fa fa-copyright"></i> Copyright 2009-2012, Splunk Inc., <a href="http://opensource.org/licenses/BSD-3-Clause" target="_blank">New BSD License</a> + + <i class="fa fa-circle"></i> <a href="http://benpickles.github.io/peity/" target="_blank">Peity</a> web chart library, + <i class="fa fa-copyright"></i> Copyright 2009-2015, Ben Pickles, <a href="https://github.com/benpickles/peity/blob/master/MIT-LICENCE" target="_blank">MIT License</a> + + <i class="fa fa-circle"></i> <a href="https://rendro.github.io/easy-pie-chart/" target="_blank">Easy Pie Chart</a> web chart library, + <i class="fa fa-copyright"></i> Copyright 2013, Robert Fleischmann, <a href="https://github.com/rendro/easy-pie-chart/blob/master/LICENSE" target="_blank">MIT License</a> + + <i class="fa fa-circle"></i> <a href="http://bernii.github.io/gauge.js/" target="_blank">Gauge.js</a> web chart library, + <i class="fa fa-copyright"></i> Copyright, Bernard Kobos, <a href="http://bernii.github.io/gauge.js/" target="_blank">MIT License</a> + + <i class="fa fa-circle"></i> <a href="https://jquery.org/" target="_blank">jQuery</a>, + <i class="fa fa-copyright"></i> Copyright 2015, jQuery Foundation, <a href="https://jquery.org/license/" target="_blank">MIT License</a> + + <i class="fa fa-circle"></i> <a href="http://getbootstrap.com/getting-started/" target="_blank">Bootstrap</a>, + <i class="fa fa-copyright"></i> Copyright 2015, Twitter, <a href="http://getbootstrap.com/getting-started/#license-faqs" target="_blank">MIT License</a> + + <i class="fa fa-circle"></i> <a href="http://www.bootstraptoggle.com/" target="_blank">Bootstrap Toggle</a>, + <i class="fa fa-copyright"></i> Copyright (c) 2011-2014 Min Hur, The New York Times Company, <a href="https://github.com/minhur/bootstrap-toggle/blob/master/LICENSE" target="_blank">MIT License</a> + + <i class="fa fa-circle"></i> <a href="https://jamesflorentino.github.io/nanoScrollerJS/" target="_blank">NanoScroller</a>, + <i class="fa fa-copyright"></i> Copyright 2012, James Florentino, <a href="https://github.com/jamesflorentino/nanoScrollerJS/blob/master/LICENSE" target="_blank">MIT License</a> + + <i class="fa fa-circle"></i> <a href="https://github.com/marcj/css-element-queries" target="_blank">CSS Element Queries</a>, + <i class="fa fa-copyright"></i> Copyright Marc J. Schmidt, <a href="https://github.com/marcj/css-element-queries/blob/master/LICENSE" target="_blank">MIT License</a> + + <i class="fa fa-circle"></i> <a href="https://fortawesome.github.io/Font-Awesome/" target="_blank">FontAwesome</a>, + <i class="fa fa-copyright"></i> Created by Dave Gandy, Font: <a href="http://scripts.sil.org/OFL" target="_blank">SIL OFL 1.1 License</a>, CSS: <a href="http://opensource.org/licenses/mit-license.html" target="_blank">MIT License</a> + + <i class="fa fa-circle"></i> <a href="http://www.iconsdb.com/soylent-red-icons/seo-performance-icon.html" target="_blank">IconsDB.com Icons</a>, Icons provided as CC0 1.0 Universal (CC0 1.0) Public Domain Dedication. + + <i class="fa fa-circle"></i> <a href="http://morrisjs.github.io/morris.js/" target="_blank">morris.js</a>, + <i class="fa fa-copyright"></i> Copyright 2013, Olly Smith, <a href="http://morrisjs.github.io/morris.js/" target="_blank">Simplified BSD License</a> + + <i class="fa fa-circle"></i> <a href="http://raphaeljs.com/" target="_blank">Raphaël</a>, + <i class="fa fa-copyright"></i> Copyright 2008, Dmitry Baranovskiy, <a href="http://raphaeljs.com/license.html" target="_blank">MIT License</a> + + <i class="fa fa-circle"></i> <a href="http://C3js.org/" target="_blank">C3</a>, + <i class="fa fa-copyright"></i> Copyright 2013, Masayuki Tanaka, <a href="https://github.com/masayuki0812/c3/blob/master/LICENSE" target="_blank">MIT License</a> + + <i class="fa fa-circle"></i> <a href="http://D3js.org/" target="_blank">D3</a>, + <i class="fa fa-copyright"></i> Copyright 2015, Mike Bostock, <a href="http://opensource.org/licenses/BSD-3-Clause" target="_blank">BSD License</a> + + </small> + </div> + </div> + </div> + </div> + + <div class="modal fade" id="welcomeModal" tabindex="-1" role="dialog" aria-labelledby="welcomeModalLabel"> + <div class="modal-dialog modal-lg" role="document"> + <div class="modal-content"> + <div class="modal-header"> + <button type="button" class="close" data-dismiss="modal" aria-label="Close"><span aria-hidden="true">×</span></button> + <h4 class="modal-title" id="welcomeModalLabel">Welcome!</h4> + </div> + <div class="modal-body"> + <div class="p"> + <b><a href="https://github.com/firehol/netdata/wiki" target="_blank">netdata</a></b> is the fastest way to visualize metrics. It is a resource efficient, highly optimized system for collecting and visualizing any type of real-time time series data, from CPU usage, disk activity, SQL queries, API calls, web site visitors, etc. + </div> + <div class="p"> + <b><a href="https://github.com/firehol/netdata/wiki" target="_blank">netdata</a></b> tries to visualize the truth of <b>now</b>, in its <b>greatest detail</b>, so that you can get insights of what is happening now and what just happened, on your systems and applications. + </div> + <div class="p"> + To make a chart in <b><a href="https://github.com/firehol/netdata/wiki" target="_blank">netdata</a></b>, you just need a <b>number</b>. Just a number you can read somehow. <b><a href="https://github.com/firehol/netdata/wiki" target="_blank">netdata</a></b> will turn this number to a real time, interactive, web chart. For collecting these numbers, it supports <a href="https://github.com/firehol/netdata/wiki/External-Plugins" target="_blank">external plugins</a>, even <a href="https://github.com/firehol/netdata/wiki/General-Info---charts.d" target="_blank">shell</a> or <a href="https://github.com/firehol/netdata/wiki/General-Info---charts.d" target="_blank">node.js</a> plugins. Any computer program, in any language, that can print a few lines of text on its standard output, can be a netdata data collector. + </div> + <div class="p"> + <b><a href="https://github.com/firehol/netdata/wiki" target="_blank">netdata</a></b> can embed charts everywhere, like this one <div data-netdata="system.cpu" data-dimensions="system" data-after="-120" data-width="25%" data-height="15px" data-chart-library="dygraph" data-dygraph-theme="sparkline" data-show-value-of-system-at="system.cpu.system.modal.1"></div> (my CPU system usage which is <span id="system.cpu.system.modal.1" style="display: inline-block; width: 40px; text-align: right;"></span>%), + or this one <div data-netdata="ipv4.tcppackets" data-dimensions="received" data-after="-120" data-width="25%" data-height="15px" data-chart-library="dygraph" data-dygraph-theme="sparkline" data-show-value-of-received-at="ipv4.tcppackets.received.modal.1"></div> (my IPv4 received TCP packets, which are <span id="ipv4.tcppackets.received.modal.1" style="display: inline-block; width: 60px; text-align: right;"></span>/second). + </div> + <div class="p"> + You can have <b><a href="https://github.com/firehol/netdata/wiki" target="_blank">netdata</a></b> charts on your site too. Just give it a <code>div</code> and a real time chart, zoomable and draggable will appear (try it even on these tiny ones - <b>drag</b> them to pan horizontally, <b>shift + drag</b> to zoom in, on <b>chrome shift + mouse wheel</b> to zoom in/out, <b>double click</b> on them to reset them - don't be afraid of <b><a href="https://github.com/firehol/netdata/wiki" target="_blank">netdata</a></b> performance - <a href="https://github.com/firehol/netdata/wiki/Performance" target="_blank">a raspberry pi 2 can sustain 300 charts updates per second</a>!). + </div> + <div class="p"> + For more information please refer to the <b><a href="https://github.com/firehol/netdata/wiki" target="_blank">netdata wiki</a></b>. + </div> + </div> + <div class="modal-footer"> + <button type="button" class="btn btn-default" data-dismiss="modal">Close</button> + </div> + </div> + </div> + </div> + + <div class="modal fade" id="helpModal" tabindex="-1" role="dialog" aria-labelledby="helpModalLabel"> + <div class="modal-dialog modal-lg" role="document"> + <div class="modal-content"> + <div class="modal-header"> + <button type="button" class="close" data-dismiss="modal" aria-label="Close"><span aria-hidden="true">×</span></button> + <h4 class="modal-title" id="helpModalLabel">Dashboard Help</h4> + </div> + <div class="modal-body"> + + <h4>Dygraphs (line, area and stacked area charts)</h4> + + <!-- Nav tabs --> + <ul class="nav nav-tabs" role="tablist"> + <li role="presentation" class="active"><a href="#help_mouse" aria-controls="help_mouse" role="tab" data-toggle="tab">Mouse Interface</a></li> + <li role="presentation"><a href="#help_touch" aria-controls="help_touch" role="tab" data-toggle="tab">Touch Interface</a></li> + </ul> + + <!-- Tab panes --> + <div class="tab-content"> + <div role="tabpanel" class="tab-pane active" id="help_mouse"> + <div class="p"> + <h4>Mouse Over / Hover</h4> + Mouse over on a chart to show, at its legend, the values for the timestamp under the mouse (the chart will also highlight the point at the chart). + <br/> + All the other visible charts will also show and highlight their values for the same timestamp. + </div> + <hr/> + <div class="p"> + <h4>Drag Chart Contents</h4> + Drag the contents of a chart to pan it horizontally. + <br/> + All the charts will follow soon after you let the chart alone (this little delay is by design: it speeds up your browser and lets you focus on what you are exploring). + <br/> + Once a chart is panned, auto refreshing stops for all charts. To enable it again, <b>double click</b> a panned chart. + </div> + <hr/> + <div class="p"> + <h4>Double Click</h4> + Double Click a chart to reset all the charts to their default auto-refreshing state. + </div> + <hr/> + <div class="p"> + <h4>SHIFT + Drag</h4> + While pressing the shift key, click on the contents of a chart and move the mouse to select an area, to zoom in. The other charts will follow too. Zooming is performed in two phases: + <ul> + <li>The already loaded chart contents are zoomed (low resolution)</li> + <li>New data are transferred from the netdata server, to refresh the chart with possibly more detail.</li> + </ul> + Once a chart is zoomed, auto refreshing stops for all charts. To enable it again, <b>double click</b> a zoomed chart. + </div> + <hr/> + <div class="p"> + <h4>SHIFT + Mouse Wheel <small>(does not work on firefox and IE/Edge)</small></h4> + While pressing the shift key and the mouse pointer is over the contents of a chart, scroll the mouse wheel to zoom in or out. This kind of zooming is aligned to center below the mouse pointer. The other charts will follow too. + <br/> + Once a chart is zoomed, auto refreshing stops for all charts. To enable it again, <b>double click</b> a zoomed chart. + </div> + <hr/> + <div class="p"> + <h4>Legend Operations</h4> + Click on the label or value of a dimension, will select / un-select this dimension. + <br/> + You can press any of the SHIFT or CONTROL keys and then click on legend labels or values, to select / un-select multiple dimensions. + </div> + </div> + <div role="tabpanel" class="tab-pane" id="help_touch"> + <div class="p"> + <h4>Single Tap</h4> + Single Tap on the contents of a chart to show, at its legend, the values for the timestamp tapped (the chart will also highlight the point at the chart). + <br/> + All the other visible charts will also show and highlight their values for the same timestamp. + </div> + <hr/> + <div class="p"> + <h4>Drag Chart Contents</h4> + Touch and Drag the contents of a chart to pan it horizontally. + <br/> + All the charts will follow soon after you let the chart alone (this little delay is by design: it speeds up your browser and lets you focus on what you are exploring). + <br/> + Once a chart is panned, auto refreshing stops for all charts. To enable it again, <b>double tap</b> a panned chart. + </div> + <hr/> + <div class="p"> + <h4>Double Tap</h4> + Double tap a chart to reset all the charts to their default auto-refreshing state. + </div> + <hr/> + <div class="p"> + <h4>Zoom <small>(does not work on firefox and IE/Edge)</small></h4> + With two fingers, zoom in or out. + <br/> + Once a chart is zoomed, auto refreshing stops for all charts. To enable it again, <b>double click</b> a zoomed chart. + </div> + <hr/> + <div class="p"> + <h4>Legend Operations</h4> + Tap on the label or value of a dimension, will select / un-select this dimension. + </div> + </div> + </div> + </div> + <div class="modal-footer"> + <button type="button" class="btn btn-default" data-dismiss="modal">Close</button> + </div> + </div> + </div> + </div> + + <div class="modal fade" id="alarmsModal" tabindex="-1" role="dialog" aria-labelledby="alarmsModalLabel"> + <div class="modal-dialog modal-lg" role="document"> + <div class="modal-content"> + <div class="modal-header"> + <button type="button" class="close" data-dismiss="modal" aria-label="Close"><span aria-hidden="true">×</span></button> + <h4 class="modal-title" id="alarmsModalLabel">netdata alarms</h4> + </div> + <div class="modal-body"> + <!-- Nav tabs --> + <ul class="nav nav-tabs" role="tablist"> + <li role="presentation" class="active"><a href="#alarms_active" aria-controls="alarms_active" role="tab" data-toggle="tab">Active</a></li> + <li role="presentation"><a href="#alarms_all" aria-controls="alarms_all" role="tab" data-toggle="tab">All</a></li> + <li role="presentation"><a href="#alarms_log" aria-controls="alarms_log" role="tab" data-toggle="tab">Log</a></li> + </ul> + + <!-- Tab panes --> + <div class="tab-content"> + <div role="tabpanel" class="tab-pane active" id="alarms_active"> + loading... + </div> + <div role="tabpanel" class="tab-pane" id="alarms_all"> + loading... + </div> + <div role="tabpanel" class="tab-pane" id="alarms_log"> + loading... + </div> + </div> + </div> + <div class="modal-footer"> + <!-- <a href="#" onclick="alarmsUpdateModal(); return false;" type="button" class="btn btn-default">Update</a> --> + <button type="button" class="btn btn-default" data-dismiss="modal">Close</button> + </div> + </div> + </div> + </div> + + <div class="modal fade" id="optionsModal" tabindex="-1" role="dialog" aria-labelledby="optionsModalLabel"> + <div class="modal-dialog modal-lg" role="document"> + <div class="modal-content"> + <div class="modal-header"> + <button type="button" class="close" data-dismiss="modal" aria-label="Close"><span aria-hidden="true">×</span></button> + <h4 class="modal-title" id="optionsModalLabel">netdata dashboard options</h4> + </div> + <div class="modal-body"> + <center> + <small style="color: #BBBBBB;">These are browser settings. Each viewer has its own. They do not affect the operation of your netdata server. + <br/> + Settings take effect immediately and are saved permanently to browser local storage (except the refresh on focus / always option). + <br/> + To reset all options (including charts sizes) to their defaults, click <a href="#" onclick="resetDashboardOptions(); return false;">here</a>.</small> + </center> + <div style="padding: 10px;"></div> + + <!-- Nav tabs --> + <ul class="nav nav-tabs" role="tablist"> + <li role="presentation" class="active"><a href="#settings_performance" aria-controls="settings_performance" role="tab" data-toggle="tab">Performance</a></li> + <li role="presentation"><a href="#settings_sync" aria-controls="settings_sync" role="tab" data-toggle="tab">Synchronization</a></li> + <li role="presentation"><a href="#settings_visual" aria-controls="settings_visual" role="tab" data-toggle="tab">Visual</a></li> + </ul> + + <!-- Tab panes --> + <div class="tab-content"> + <div role="tabpanel" class="tab-pane active" id="settings_performance"> + <form id="optionsForm1" method="get" class="form-horizontal"> + <div class="form-group"> + <table> + <tr class="option-row"> + <td class="option-control"><input id="stop_updates_when_focus_is_lost" type="checkbox" checked data-toggle="toggle" data-offstyle="danger" data-onstyle="success" data-on="On Focus" data-off="Always" data-width="110px"></td> + <td class="option-info"><strong>When to refresh the charts?</strong><br/> + <small>When set to <b>On Focus</b>, the charts will stop being updated if the page / tab does not have the focus of the user. When set to <b>Always</b>, the charts will always be refreshed. Set it to <b>On Focus</b> it to lower the CPU requirements of the browser (and extend the battery of laptops and tablets) when this page does not have your focus. Set to <b>Always</b> to work on another window (i.e. change the settings of something) and have the charts auto-refresh in this window.</small> + </td> + </tr> + <tr class="option-row"> + <td class="option-control"> + <input id="eliminate_zero_dimensions" type="checkbox" checked data-toggle="toggle" data-on="Non Zero" data-off="All" data-width="110px"> + </td> + <td class="option-info"><strong>Which dimensions to show?</strong><br/> + <small>When set to <b>Non Zero</b>, dimensions that have all their values (within the current view) set to zero will not be transferred from the netdata server (except if all dimensions of the chart are zero, in which case this setting does nothing - all dimensions are transferred and shown). When set to <b>All</b>, all dimensions will always be shown. Set it to <b>Non Zero</b> to lower the data transferred between netdata and your browser, lower the CPU requirements of your browser (fewer lines to draw) and increase the focus on the legends (fewer entries at the legends).</small> + </td> + </tr> + <tr class="option-row"> + <td class="option-control"><input id="destroy_on_hide" type="checkbox" data-toggle="toggle" data-on="Destroy" data-off="Hide" data-width="110px"></td> + <td class="option-info"><strong>How to handle hidden charts?</strong><br/> + <small>When set to <b>Destroy</b>, charts that are not in the current viewport of the browser (are above, or below the visible area of the page), will be destroyed and re-created if and when they become visible again. When set to <b>Hide</b>, the not-visible charts will be just hidden, to simplify the DOM and speed up your browser. Set it to <b>Destroy</b>, to lower the memory requirements of your browser. Set it to <b>Hide</b> for smoother page scrolling.</small> + </td> + </tr> + </table> + </div> + </form> + </div> + <div role="tabpanel" class="tab-pane" id="settings_sync"> + <form id="optionsForm2" method="get" class="form-horizontal"> + <div class="form-group"> + <table> + <tr class="option-row"> + <td class="option-control"><input id="parallel_refresher" type="checkbox" checked data-toggle="toggle" data-on="Parallel" data-off="Sequential" data-width="110px"></td> + <td class="option-info"><strong>Which chart refresh policy to use?</strong><br/> + <small>When set to <b>parallel</b>, visible charts are refreshed in parallel (all queries are sent to netdata server in parallel) and are rendered asynchronously. When set to <b>sequential</b> charts are refreshed one after another. Set it to parallel if your browser can cope with it (most modern browsers do), set it to sequential if you work on an older/slower computer.</small> + </td> + </tr> + <tr class="option-row" id="concurrent_refreshes_row"> + <td class="option-control"></td> + <td class="option-info"> + <table> + <tr class="option-row"> + <td class="option-control"> + <input id="concurrent_refreshes" type="checkbox" checked data-toggle="toggle" data-on="Resync" data-off="Best Effort" data-width="110px"> + </td> + <td class="option-info"> + <strong>Shall we re-sync chart refreshes?</strong><br/> + <small>When set to <b>Resync</b>, the dashboard will attempt to re-synchronize all the charts so that they are refreshed concurrently. When set to <b>Best Effort</b>, each chart may be refreshed with a little time difference to the others. Normally, the dashboard starts refreshing them in parallel, but depending on the speed of your computer and the network latencies, charts start having a slight time difference. Setting this to <b>Resync</b> will attempt to re-synchronize the charts on every update. Setting it to <b>Best Effort</b> may lower the pressure on your browser and the network.</small> + </td> + </tr> + </table> + </td> + </tr> + <tr class="option-row"> + <td class="option-control"><input id="sync_selection" type="checkbox" checked data-toggle="toggle" data-on="Sync" data-off="Don't Sync" data-onstyle="success" data-offstyle="danger" data-width="110px"></td> + <td class="option-info"><strong>Sync hover selection on all charts?</strong><br/> + <small>When enabled, a selection on one chart will automatically select the same time on all other visible charts and the legends of all visible charts will be updated to show the selected values. When disabled, only the chart getting the user's attention will be selected. Enable it to get better insights of the data. Disable it if you are on a very slow computer that cannot actually do it.</small> + </td> + </tr> + <tr class="option-row"> + <td class="option-control"><input id="sync_pan_and_zoom" type="checkbox" checked data-toggle="toggle" data-on="Sync" data-off="Don't Sync" data-onstyle="success" data-offstyle="danger" data-width="110px"></td> + <td class="option-info"><strong>Sync pan and zoom on all charts?</strong><br/> + <small>When enabled, pan and zoom operations on a chart will be replicated to all charts (even the ones that are not currently visible - of course only when they will become visible). When disabled, pan and zoom operations will not be propagated to other charts.</small> + </td> + </tr> + </table> + </div> + </form> + </div> + <div role="tabpanel" class="tab-pane" id="settings_visual"> + <form id="optionsForm3" method="get" class="form-horizontal"> + <div class="form-group"> + <table> + <tr class="option-row"> + <td class="option-control"><input id="netdata_theme_control" type="checkbox" checked data-toggle="toggle" data-offstyle="danger" data-onstyle="success" data-on="Dark" data-off="White" data-width="110px"></td> + <td class="option-info"><strong>Which theme to use?</strong><br/> + <small>Netdata comes with two themes: <b>Dark</b> (the default) and <b>White</b>. + <br/> + <b>Switching this will reload the dashboard</b>. + </small> + </td> + </tr> + <tr class="option-row"> + <td class="option-control"><input id="show_help" type="checkbox" checked data-toggle="toggle" data-on="Help Me" data-off="No Help" data-width="110px"></td> + <td class="option-info"><strong>Do you need help?</strong><br/> + <small>Netdata can show some help in some areas to help you use the dashboard. If all these balloons bother you, disable them using this switch. + <br/> + <b>Switching this will reload the dashboard</b>. + </small> + </td> + </tr> + <tr class="option-row"> + <td class="option-control"><input id="pan_and_zoom_data_padding" type="checkbox" checked data-toggle="toggle" data-on="Pad" data-off="Don't Pad" data-width="110px"></td> + <td class="option-info"><strong>Enable data padding when panning and zooming?</strong><br/> + <small>When set to <b>Pad</b> the charts will be padded with more data, both before and after the visible area, thus giving the impression the whole database is loaded. This padding will happen only after the first pan or zoom operation on the chart (initially all charts have only the visible data). When set to <b>Don't Pad</b> only the visible data will be transfered from the netdata server, even after the first pan and zoom operation.</small> + </td> + </tr> + <tr class="option-row"> + <td class="option-control"><input id="smooth_plot" type="checkbox" checked data-toggle="toggle" data-on="Smooth" data-off="Rough" data-width="110px"></td> + <td class="option-info"><strong>Enable Bézier lines on charts?</strong><br/> + <small>When set to <b>Smooth</b> the charts libraries that support it, will plot smooth curves instead of simple straight lines to connect the points. + <br/> + Keep in mind <a href="http://dygraphs.com" target="_blank">dygraphs</a>, the main charting library in netdata dashboards, can only smooth line charts. It cannot smooth area or stacked charts. When set to <b>Rough</b>, this setting can lower the CPU resources consumed by your browser.</small> + </td> + </tr> + </table> + </div> + </form> + </div> + </div> + </div> + <div class="modal-footer"> + <button type="button" class="btn btn-default" data-dismiss="modal">Close</button> + </div> + </div> + </div> + </div> + + + <div class="modal fade" id="updateModal" tabindex="-1" role="dialog" aria-labelledby="updateModalLabel"> + <div class="modal-dialog" role="document"> + <div class="modal-content"> + <div class="modal-header"> + <button type="button" class="close" data-dismiss="modal" aria-label="Close"><span aria-hidden="true">×</span></button> + <h4 class="modal-title" id="updateModalLabel">Update Check</h4> + </div> + <div class="modal-body"> + Your netdata version: <b><code><span id="netdataVersion">Unknown</span></code></b> + <br/> + <div style="padding: 10px;"></div> + <div id="versionCheckLog">Not checked yet. Please press the Check Now button.</div> + </div> + <div class="modal-footer"> + <a href="#" onclick="notifyForUpdate(true); return false;" type="button" class="btn btn-default">Check Now</a> + <button type="button" class="btn btn-default" data-dismiss="modal">Close</button> + </div> + </div> + </div> + </div> + + <div class="modal fade" id="deleteRegistryModal" tabindex="-1" role="dialog" aria-labelledby="deleteRegistryModalLabel"> + <div class="modal-dialog" role="document"> + <div class="modal-content"> + <div class="modal-header"> + <button type="button" class="close" data-dismiss="modal" aria-label="Close"><span aria-hidden="true">×</span></button> + <h4 class="modal-title" id="deleteRegistryModalLabel">Delete <span id="deleteRegistryServerName"></span>?</h4> + </div> + <div class="modal-body"> + You are about to delete, from your personal list of netdata servers, the following server: + <p style="text-align: center; padding-top: 10px; padding-bottom: 10px; line-height: 2;"> + <b><span id="deleteRegistryServerName2"></span></b> + <br/> + <b><span id="deleteRegistryServerURL"></span></b> + </p> + Are you sure you want to do this? + <br/> + <div style="padding: 10px;"></div> + <small>Keep in mind, this server will be added back if and when you visit it again.</small> + <br/> + <div id="deleteRegistryResponse" style="display: block; width: 100%; text-align: center; padding-top: 20px;"></div> + </div> + <div class="modal-footer"> + <button type="button" class="btn btn-success" data-dismiss="modal">keep it</button> + <a href="#" onclick="notifyForDeleteRegistry(true); return false;" type="button" class="btn btn-danger">delete it</a> + </div> + </div> + </div> + </div> + + <div class="modal fade" id="switchRegistryModal" tabindex="-1" role="dialog" aria-labelledby="switchRegistryModalLabel"> + <div class="modal-dialog" role="document"> + <div class="modal-content"> + <div class="modal-header"> + <button type="button" class="close" data-dismiss="modal" aria-label="Close"><span aria-hidden="true">×</span></button> + <h4 class="modal-title" id="switchRegistryModalLabel">Switch Netdata Registry Identity</h4> + </div> + <div class="modal-body"> + You can copy and paste the following ID to all your browsers (e.g. work and home). + <br/> + All the browsers with the same ID will identify <b>you</b>, so please don't share this with others. + <p style="text-align: center; padding-top: 10px; padding-bottom: 10px; line-height: 2;"> + <form action="#"> + <input type="text" class="form-control" id="switchRegistryPersonGUID" placeholder="your personal ID" maxlength="36" autocomplete="off" style="text-align: center; font-size: 1.4em;"> + </form> + </p> + Either copy this ID and paste it to another browser, or paste here the ID you have taken from another browser. + <p style="padding-top: 10px;"><small> + Keep in mind that: + <ul> + <li>when you switch ID, your previous ID will be lost forever - this is irreversible.</li> + <li>both IDs (your old and the new) must list this netdata at their personal lists.</li> + <li>both IDs have to be known by the registry: <b><span id="switchRegistryURL"></span></b>.</li> + <li>to get a new ID, just clear your browser cookies.</li> + </ul> + </small></p> + <div id="switchRegistryResponse" style="display: block; width: 100%; text-align: center; padding-top: 20px;"></div> + </div> + <div class="modal-footer"> + <button type="button" class="btn btn-success" data-dismiss="modal">cancel</button> + <a href="#" onclick="notifyForSwitchRegistry(true); return false;" type="button" class="btn btn-danger">impersonate</a> + </div> + </div> + </div> + </div> + + <div class="modal fade" id="gotoServerModal" tabindex="-1" role="dialog" aria-labelledby="gotoServerModalLabel"> + <div class="modal-dialog" role="document"> + <div class="modal-content"> + <div class="modal-header"> + <button type="button" class="close" data-dismiss="modal" aria-label="Close"><span aria-hidden="true">×</span></button> + <h4 class="modal-title" id="gotoServerModalLabel"><span id="gotoServerName"></span></h4> + </div> + <div class="modal-body"> + Checking known URLs for this server... + <div style="padding-top: 20px;"> + <table id="gotoServerList"> + </table> + </div> + <p style="padding-top: 10px;"><small> + Checks may fail if you are viewing an HTTPS page and the server to be checked is HTTP only. + </small></p> + <div id="gotoServerResponse" style="display: block; width: 100%; text-align: center; padding-top: 20px;"></div> + </div> + <div class="modal-footer"> + <button type="button" class="btn btn-default" data-dismiss="modal">Close</button> + </div> + </div> + </div> + </div> + +<script type="text/javascript"> var this_is_demo = null; function isdemo() { - if(this_is_demo !== null) return this_is_demo; - this_is_demo = false; - - try { - if(typeof document.location.hostname === 'string') { - if(document.location.hostname.endsWith('.my-netdata.io') || - document.location.hostname.endsWith('.mynetdata.io') || - document.location.hostname.endsWith('.netdata.rocks') || - document.location.hostname.endsWith('.firehol.org') || - document.location.hostname.endsWith('.netdata.online')) - this_is_demo = true; - } - } - catch(error) { - ; - } - - return this_is_demo; + if(this_is_demo !== null) return this_is_demo; + this_is_demo = false; + + try { + if(typeof document.location.hostname === 'string') { + if(document.location.hostname.endsWith('.my-netdata.io') || + document.location.hostname.endsWith('.mynetdata.io') || + document.location.hostname.endsWith('.netdata.rocks') || + document.location.hostname.endsWith('.firehol.org') || + document.location.hostname.endsWith('.netdata.online')) + this_is_demo = true; + } + } + catch(error) { + ; + } + + return this_is_demo; } if(isdemo()) { - document.getElementById('masthead').style.display = 'block'; + document.getElementById('masthead').style.display = 'block'; +} + +function netdataURL(url) { + if(typeof url === 'undefined') + url = document.location.toString(); + + if(url.indexOf('#') !== -1) + url = url.substring(0, url.indexOf('#')); + + var hash = netdataHash(); + + // console.log('netdataURL: ' + url + hash); + + return url + hash; +} + +function netdataReload(url) { + var t = netdataURL(url); + // console.log('netdataReload: ' + t); + document.location = t; + + // since we play with hash + // this is needed to reload the page + location.reload(); +} + +var gotoServerValidateRemaining = 0; +var gotoServerMiddleClick = false; +var gotoServerStop = false; +function gotoServerValidateUrl(id, guid, url) { + var penaldy = 0; + if(document.location.toString().startsWith('http://') && url.toString().startsWith('https://')) + // we penalize https only if the current url is http + // to allow the user walk through all its servers. + penaldy = 500; + + var finalURL = netdataURL(url); + + setTimeout(function() { + document.getElementById('gotoServerList').innerHTML += '<tr><td style="padding-left: 20px;"><a href="' + finalURL + '" target="_blank">' + url + '</a></td><td style="padding-left: 30px;"><code id="' + guid + '-' + id + '-status">checking...</code></td></tr>'; + + NETDATA.registry.hello(url, function(data) { + if (data) { + // console.log('OK ' + id + ' URL: ' + url); + document.getElementById(guid + '-' + id + '-status').innerHTML = "OK"; + + if(!gotoServerStop) { + gotoServerStop = true; + + if(gotoServerMiddleClick) { + window.open(finalURL, '_blank'); + gotoServerMiddleClick = false; + document.getElementById('gotoServerResponse').innerHTML = '<b>Opening new window to ' + NETDATA.registry.machines[guid].name + '<br/><a href="' + finalURL + '">' + url + '</a></b><br/>(check your pop-up blocker if it fails)'; + } + else + document.location = finalURL; + } + } + else { + document.getElementById(guid + '-' + id + '-status').innerHTML = "failed!"; + gotoServerValidateRemaining--; + if(gotoServerValidateRemaining <= 0) { + gotoServerMiddleClick = false; + document.getElementById('gotoServerResponse').innerHTML = '<b>Sorry! I cannot find any operational URL for this server</b>'; + } + } + }); + }, (id * 50) + penaldy); +} + +function gotoServerModalHandler(guid) { + // console.log('goto server: ' + guid); + + gotoServerStop = false; + var len = NETDATA.registry.machines[guid].alternate_urls.length; + + document.getElementById('gotoServerResponse').innerHTML = ''; + document.getElementById('gotoServerList').innerHTML = ''; + document.getElementById('gotoServerName').innerHTML = NETDATA.registry.machines[guid].name; + $('#gotoServerModal').modal('show'); + + gotoServerValidateRemaining = len; + while(len--) + gotoServerValidateUrl(len, guid, NETDATA.registry.machines[guid].alternate_urls[len]); + + return false; +} + +function gotoServerInit() { + $(".registry_link").on('click', function(e) { + if(e.which === 2) { + e.preventDefault(); + gotoServerMiddleClick = true; + } + else { + gotoServerMiddleClick = false; + } + + return true; + }); } function switchRegistryModalHandler() { - document.getElementById('switchRegistryPersonGUID').value = NETDATA.registry.person_guid; - document.getElementById('switchRegistryURL').innerHTML = NETDATA.registry.server; - document.getElementById('switchRegistryResponse').innerHTML = ''; - $('#switchRegistryModal').modal('show'); + document.getElementById('switchRegistryPersonGUID').value = NETDATA.registry.person_guid; + document.getElementById('switchRegistryURL').innerHTML = NETDATA.registry.server; + document.getElementById('switchRegistryResponse').innerHTML = ''; + $('#switchRegistryModal').modal('show'); } function notifyForSwitchRegistry() { - var n = document.getElementById('switchRegistryPersonGUID').value; - - if(n !== '' && n.length === 36) { - NETDATA.registry.switch(n, function(result) { - if(result !== null) { - $('#switchRegistryModal').modal('hide'); - NETDATA.registry.init(); - } - else { - document.getElementById('switchRegistryResponse').innerHTML = "<b>Sorry! The registry rejected your request.</b>"; - } - }); - } - else - document.getElementById('switchRegistryResponse').innerHTML = "<b>The ID you have entered is not a GUID.</b>"; + var n = document.getElementById('switchRegistryPersonGUID').value; + + if(n !== '' && n.length === 36) { + NETDATA.registry.switch(n, function(result) { + if(result !== null) { + $('#switchRegistryModal').modal('hide'); + NETDATA.registry.init(); + } + else { + document.getElementById('switchRegistryResponse').innerHTML = "<b>Sorry! The registry rejected your request.</b>"; + } + }); + } + else + document.getElementById('switchRegistryResponse').innerHTML = "<b>The ID you have entered is not a GUID.</b>"; } var deleteRegistryUrl = null; function deleteRegistryModalHandler(guid, name, url) { - deleteRegistryUrl = url; - document.getElementById('deleteRegistryServerName').innerHTML = name; - document.getElementById('deleteRegistryServerName2').innerHTML = name; - document.getElementById('deleteRegistryServerURL').innerHTML = url; - document.getElementById('deleteRegistryResponse').innerHTML = ''; - $('#deleteRegistryModal').modal('show'); + deleteRegistryUrl = url; + document.getElementById('deleteRegistryServerName').innerHTML = name; + document.getElementById('deleteRegistryServerName2').innerHTML = name; + document.getElementById('deleteRegistryServerURL').innerHTML = url; + document.getElementById('deleteRegistryResponse').innerHTML = ''; + $('#deleteRegistryModal').modal('show'); } function notifyForDeleteRegistry() { - if(deleteRegistryUrl) { - NETDATA.registry.delete(deleteRegistryUrl, function(result) { - if(result !== null) { - deleteRegistryUrl = null; - $('#deleteRegistryModal').modal('hide'); - NETDATA.registry.init(); - } - else { - document.getElementById('deleteRegistryResponse').innerHTML = "<b>Sorry! this command was rejected by the registry server.</b>"; - } - }); - } + if(deleteRegistryUrl) { + NETDATA.registry.delete(deleteRegistryUrl, function(result) { + if(result !== null) { + deleteRegistryUrl = null; + $('#deleteRegistryModal').modal('hide'); + NETDATA.registry.init(); + } + else { + document.getElementById('deleteRegistryResponse').innerHTML = "<b>Sorry! this command was rejected by the registry server.</b>"; + } + }); + } } var options = { - sparklines_registry: {}, - submenu_names: {}, - data: null, - hostname: 'netdata_server', // will be overwritten by the netdata server - categories: new Array(), - categories_idx: {}, - families: new Array(), - families_idx: {}, - - chartsPerRow: 0, - chartsMinWidth: 1450, - chartsHeight: 180, - sparklinesHeight: 60 + sparklines_registry: {}, + menus: {}, + submenu_names: {}, + data: null, + hostname: 'netdata_server', // will be overwritten by the netdata server + categories: new Array(), + categories_idx: {}, + families: new Array(), + families_idx: {}, + + chartsPerRow: 0, + chartsMinWidth: 1450, + chartsHeight: 180, + sparklinesHeight: 60, }; // generate a sparkline // used in the documentation function sparkline(chart, dimension, units) { - var key = chart + '.' + dimension; + var key = chart + '.' + dimension; - if(typeof units === 'undefined') - units = ''; + if(typeof units === 'undefined') + units = ''; - if(typeof options.sparklines_registry[key] === 'undefined') - options.sparklines_registry[key] = { count: 1 }; - else - options.sparklines_registry[key].count++; + if(typeof options.sparklines_registry[key] === 'undefined') + options.sparklines_registry[key] = { count: 1 }; + else + options.sparklines_registry[key].count++; - key = key + '.' + options.sparklines_registry[key].count; + key = key + '.' + options.sparklines_registry[key].count; - var h = '<div data-netdata="' + chart + '" data-after="-120" data-width="25%" data-height="15px" data-chart-library="dygraph" data-dygraph-theme="sparkline" data-dimensions="' + dimension + '" data-show-value-of-' + dimension + '-at="' + key + '"></div> (<span id="' + key + '" style="display: inline-block; min-width: 50px; text-align: right;">X</span>' + units + ')'; + var h = '<div data-netdata="' + chart + '" data-after="-120" data-width="25%" data-height="15px" data-chart-library="dygraph" data-dygraph-theme="sparkline" data-dimensions="' + dimension + '" data-show-value-of-' + dimension + '-at="' + key + '"></div> (<span id="' + key + '" style="display: inline-block; min-width: 50px; text-align: right;">X</span>' + units + ')'; - return h; + return h; } function chartsPerRow(total) { - if(options.chartsPerRow === 0) { - width = Math.floor(total / options.chartsMinWidth); - if(width === 0) width = 1; - return width; - } - else return options.chartsPerRow; + if(options.chartsPerRow === 0) { + width = Math.floor(total / options.chartsMinWidth); + if(width === 0) width = 1; + return width; + } + else return options.chartsPerRow; } function prioritySort(a, b) { - if(a.priority < b.priority) return -1; - if(a.priority > b.priority) return 1; - if(a.name < b.name) return -1; - return 1; + if(a.priority < b.priority) return -1; + if(a.priority > b.priority) return 1; + if(a.name < b.name) return -1; + return 1; } function sortObjectByPriority(object) { - var idx = {}; - var sorted = new Array(); - - for(var i in object) { - if(typeof idx[i] === 'undefined') { - idx[i] = object[i]; - sorted.push(i); - } - } - - sorted.sort(function(a, b) { - if(idx[a].priority < idx[b].priority) return -1; - if(idx[a].priority > idx[b].priority) return 1; - if(a < b) return -1; - return 1; - }); - - return sorted; + var idx = {}; + var sorted = new Array(); + + for(var i in object) { + if(typeof idx[i] === 'undefined') { + idx[i] = object[i]; + sorted.push(i); + } + } + + sorted.sort(function(a, b) { + if(idx[a].priority < idx[b].priority) return -1; + if(idx[a].priority > idx[b].priority) return 1; + if(a < b) return -1; + return 1; + }); + + return sorted; +} + + +// ---------------------------------------------------------------------------- +// scroll to a section, without changing the browser history + +function scrollToId(hash) { + if(hash && hash != '') { + var offset = $('#' + hash).offset(); + if(typeof offset !== 'undefined') + $('html, body').animate({ scrollTop: offset.top }, 0); + } + + // we must return false to prevent the default action + return false; } // ---------------------------------------------------------------------------- function gaugeChart(title, width, dimensions, colors) { - if(typeof colors === 'undefined') - colors = ''; - - if(typeof dimensions === 'undefined') - dimensions = ''; - - return '<div data-netdata="CHART_UNIQUE_ID"' - + ' data-dimensions="' + dimensions + '"' - + ' data-chart-library="gauge"' - + ' data-gauge-adjust="width"' - + ' data-title="' + title + '"' - + ' data-width="' + width + '"' - + ' data-before="0"' - + ' data-after="-CHART_DURATION"' - + ' data-points="CHART_DURATION"' - + ' data-colors="' + colors + '"' - + ' role="application"></div>'; + if(typeof colors === 'undefined') + colors = ''; + + if(typeof dimensions === 'undefined') + dimensions = ''; + + return '<div data-netdata="CHART_UNIQUE_ID"' + + ' data-dimensions="' + dimensions + '"' + + ' data-chart-library="gauge"' + + ' data-gauge-adjust="width"' + + ' data-title="' + title + '"' + + ' data-width="' + width + '"' + + ' data-before="0"' + + ' data-after="-CHART_DURATION"' + + ' data-points="CHART_DURATION"' + + ' data-colors="' + colors + '"' + + ' role="application"></div>'; } // ---------------------------------------------------------------------------- var menuData = { - 'system': { - title: 'System Overview', - info: 'Overview of the key system metrics.' - }, - - 'ap': { - title: 'Access Points', - info: undefined - }, - - 'tc': { - title: 'Quality of Service', - info: 'Netdata collects and visualizes tc class utilization using its <a href="https://github.com/firehol/netdata/blob/master/plugins.d/tc-qos-helper.sh" target="_blank">tc-helper plugin</a>. If you also use <a href="http://firehol.org/#fireqos" target="_blank">FireQOS</a> for setting up QoS, netdata automatically collects interface and class names. If your QoS configuration includes overheads calculation, the values shown here will include these overheads (the total bandwidth for the same interface as reported in the Network Interfaces section, will be lower than the total bandwidth reported here). Also, data collection may have a slight time difference compared to the interface (QoS data collection is implemented with a BASH script, so a shift in data collection of a few milliseconds should be justified).' - }, - - 'net': { - title: 'Network Interfaces', - info: 'Per network interface statistics collected from <code>/proc/net/dev</code>.' - }, - - 'ipv4': { - title: 'IPv4 Networking', - info: undefined - }, - - 'ipv6': { - title: 'IPv6 Networking', - info: undefined - }, - - 'ipvs': { - title: 'IP Virtual Server', - info: undefined - }, - - 'netfilter': { - title: 'Firewall (netfilter)', - info: undefined - }, - - 'cpu': { - title: 'CPUs', - info: undefined - }, - - 'mem': { - title: 'Memory', - info: undefined - }, - - 'disk': { - title: 'Disks', - info: undefined - }, - - 'sensors': { - title: 'Sensors', - info: undefined - }, - - 'nfsd': { - title: 'File Server (nfsd)', - info: undefined - }, - - 'apps': { - title: 'Applications', - info: 'Per application statistics are collected using netdata\'s <code>apps.plugin</code>. This plugin walks through the entire <code>/proc</code> filesystem and aggregates statistics for applications of interest, defined in <code>/etc/netdata/apps_groups.conf</code> (the default is <a href="https://github.com/firehol/netdata/blob/master/conf.d/apps_groups.conf" target="_blank">here</a>). The plugin internally builds a process tree (much like <code>ps fax</code> does), and groups processes together (evaluating both child and parent processes) so that the result is always a chart with a predefined set of dimensions (of course, only application groups found running are reported).<br/><b>IMPORTANT</b>: The values shown here are not 100% accurate. They only include values for the processes running. If an application is spawning children continuously, which are terminated in just a few milliseconds (like shell scripts do), the values reported will be inaccurate. Linux does report the values for the exited children of a process. However, these values are reported to the parent process <b>only when the child exits</b>. If these values, of the exited child processes, were also aggregated in the charts below, the charts would have been full of spikes, presenting unrealistic utilization for each process group. So, we decided to ignore these values and present only the utilization of <b>the currently running processes</b>.', - height: 1.5 - }, - - 'users': { - title: 'Users', - info: 'Per user statistics are collected using netdata\'s <code>apps.plugin</code>. This plugin walks through the entire <code>/proc</code> filesystem and aggregates statistics per user.<br/><b>IMPORTANT</b>: The values shown here are not 100% accurate. They only include values for the processes running. If an application is spawning children continuously, which are terminated in just a few milliseconds (like shell scripts do), the values reported will be inaccurate. Linux does report the values for the exited children of a process. However, these values are reported to the parent process <b>only when the child exits</b>. If these values, of the exited child processes, were also aggregated in the charts below, the charts would have been full of spikes, presenting unrealistic utilization for each process group. So, we decided to ignore these values and present only the utilization of <b>the currently running processes</b>.', - height: 1.5 - }, - - 'groups': { - title: 'User Groups', - info: 'Per user group statistics are collected using netdata\'s <code>apps.plugin</code>. This plugin walks through the entire <code>/proc</code> filesystem and aggregates statistics per user group.<br/><b>IMPORTANT</b>: The values shown here are not 100% accurate. They only include values for the processes running. If an application is spawning children continuously, which are terminated in just a few milliseconds (like shell scripts do), the values reported will be inaccurate. Linux does report the values for the exited children of a process. However, these values are reported to the parent process <b>only when the child exits</b>. If these values, of the exited child processes, were also aggregated in the charts below, the charts would have been full of spikes, presenting unrealistic utilization for each process group. So, we decided to ignore these values and present only the utilization of <b>the currently running processes</b>.', - height: 1.5 - }, - - 'netdata': { - title: 'Netdata Monitoring', - info: undefined - }, - - 'example': { - title: 'Example Charts', - info: undefined - }, - - 'cgroup': { - title: 'Container', - info: undefined - }, - - 'mysql': { - title: 'MySQL', - info: undefined - }, - - 'named': { - title: 'named', - info: undefined - }, + 'system': { + title: 'System Overview', + info: 'Overview of the key system metrics.' + }, + + 'ap': { + title: 'Access Points', + info: undefined + }, + + 'tc': { + title: 'Quality of Service', + info: 'Netdata collects and visualizes tc class utilization using its <a href="https://github.com/firehol/netdata/blob/master/plugins.d/tc-qos-helper.sh" target="_blank">tc-helper plugin</a>. If you also use <a href="http://firehol.org/#fireqos" target="_blank">FireQOS</a> for setting up QoS, netdata automatically collects interface and class names. If your QoS configuration includes overheads calculation, the values shown here will include these overheads (the total bandwidth for the same interface as reported in the Network Interfaces section, will be lower than the total bandwidth reported here). Also, data collection may have a slight time difference compared to the interface (QoS data collection is implemented with a BASH script, so a shift in data collection of a few milliseconds should be justified).' + }, + + 'net': { + title: 'Network Interfaces', + info: 'Per network interface statistics collected from <code>/proc/net/dev</code>.' + }, + + 'ipv4': { + title: 'IPv4 Networking', + info: undefined + }, + + 'ipv6': { + title: 'IPv6 Networking', + info: undefined + }, + + 'ipvs': { + title: 'IP Virtual Server', + info: undefined + }, + + 'netfilter': { + title: 'Firewall (netfilter)', + info: undefined + }, + + 'cpu': { + title: 'CPUs', + info: undefined + }, + + 'mem': { + title: 'Memory', + info: undefined + }, + + 'disk': { + title: 'Disks', + info: 'Charts with performance information for all the system disks. Special care has been given to present disk performance metrics in a way compatible with <code>iostat -x</code>. netdata by default prevents rendering performance charts for individual partitions and unmounted virtual disks. Disabled charts can still be enabled by altering the relative settings in the netdata configuration file.' + }, + + 'sensors': { + title: 'Sensors', + info: undefined + }, + + 'nfsd': { + title: 'File Server (nfsd)', + info: undefined + }, + + 'apps': { + title: 'Applications', + info: 'Per application statistics are collected using netdata\'s <code>apps.plugin</code>. This plugin walks through the entire <code>/proc</code> filesystem and aggregates statistics for applications of interest, defined in <code>/etc/netdata/apps_groups.conf</code> (the default is <a href="https://github.com/firehol/netdata/blob/master/conf.d/apps_groups.conf" target="_blank">here</a>). The plugin internally builds a process tree (much like <code>ps fax</code> does), and groups processes together (evaluating both child and parent processes) so that the result is always a chart with a predefined set of dimensions (of course, only application groups found running are reported). The reported values are compatible with <code>top</code>, although the netdata plugin counts also the resources of exited children (unlike <code>top</code> which shows only the resources of the currently running processes). So for processes like shell scripts, the reported values include the resources used by the commands these scripts run within each timeframe.', + height: 1.5 + }, + + 'users': { + title: 'Users', + info: 'Per user statistics are collected using netdata\'s <code>apps.plugin</code>. This plugin walks through the entire <code>/proc</code> filesystem and aggregates statistics per user. The reported values are compatible with <code>top</code>, although the netdata plugin counts also the resources of exited children (unlike <code>top</code> which shows only the resources of the currently running processes). So for processes like shell scripts, the reported values include the resources used by the commands these scripts run within each timeframe.', + height: 1.5 + }, + + 'groups': { + title: 'User Groups', + info: 'Per user group statistics are collected using netdata\'s <code>apps.plugin</code>. This plugin walks through the entire <code>/proc</code> filesystem and aggregates statistics per user group. The reported values are compatible with <code>top</code>, although the netdata plugin counts also the resources of exited children (unlike <code>top</code> which shows only the resources of the currently running processes). So for processes like shell scripts, the reported values include the resources used by the commands these scripts run within each timeframe.', + height: 1.5 + }, + + 'netdata': { + title: 'Netdata Monitoring', + info: undefined + }, + + 'example': { + title: 'Example Charts', + info: undefined + }, + + 'cgroup': { + title: 'Container', + info: undefined + }, + + 'cgqemu': { + title: 'VM', + info: undefined + }, + + 'memcached': { + title: 'memcached', + info: undefined + }, + + 'mysql': { + title: 'MySQL', + info: undefined + }, + + 'redis': { + title: 'Redis', + info: undefined + }, + + 'ipfs': { + title: 'IPFS', + info: undefined + }, + + 'phpfpm': { + title: 'PHP-FPM', + info: undefined, + }, + + 'nginx': { + title: 'nginx', + info: undefined, + }, + + 'apache': { + title: 'Apache', + info: undefined, + }, + + 'named': { + title: 'named', + info: undefined + } }; var submenuData = { - 'mem.ksm': { - title: 'Memory Deduper', - info: 'Kernel Same-page Merging (KSM) performance monitoring, read from several files in <code>/sys/kernel/mm/ksm/</code>. KSM is a memory-saving de-duplication feature in the Linux kernel (since version 2.6.32). The KSM daemon ksmd periodically scans those areas of user memory which have been registered with it, looking for pages of identical content which can be replaced by a single write-protected page (which is automatically copied if a process later wants to update its content). KSM was originally developed for use with KVM (where it was known as Kernel Shared Memory), to fit more virtual machines into physical memory, by sharing the data common between them. But it can be useful to any application which generates many instances of the same data.' - }, - - 'netfilter.conntrack': { - title: 'Connection Tracker', - info: 'Netfilter Connection Tracker performance monitoring, read from <code>/proc/net/stat/nf_conntrack</code>. The connection tracker keeps track of all connections of the machine, inbound and outbound. It works by keeping a database with all open connections, tracking network and address translation and connection expectations.' - }, - - 'netfilter.nfacct': { - title: 'Bandwidth Accounting', - info: 'The following information is read using the <code>nfacct.plugin</code>.' - }, - - 'netfilter.synproxy': { - title: 'DDoS Protection', - info: 'DDoS Protection performance monitoring read from <code>/proc/net/stat/synproxy</code>. <a href="https://github.com/firehol/firehol/wiki/Working-with-SYNPROXY" target="_blank">SYNPROXY</a> is a TCP SYN packets proxy. It is used to protect any TCP server (like a web server) from SYN floods and similar DDoS attacks. It is a netfilter module, in the Linux kernel (since version 3.12). It is optimized to handle millions of packets per second utilizing all CPUs available without any concurrency locking between the connections. It can be used for any kind of TCP traffic (even encrypted), since it does not interfere with the content itself.' - } + 'mem.ksm': { + title: 'Memory Deduper', + info: 'Kernel Same-page Merging (KSM) performance monitoring, read from several files in <code>/sys/kernel/mm/ksm/</code>. KSM is a memory-saving de-duplication feature in the Linux kernel (since version 2.6.32). The KSM daemon ksmd periodically scans those areas of user memory which have been registered with it, looking for pages of identical content which can be replaced by a single write-protected page (which is automatically copied if a process later wants to update its content). KSM was originally developed for use with KVM (where it was known as Kernel Shared Memory), to fit more virtual machines into physical memory, by sharing the data common between them. But it can be useful to any application which generates many instances of the same data.' + }, + + 'netfilter.conntrack': { + title: 'Connection Tracker', + info: 'Netfilter Connection Tracker performance monitoring, read from <code>/proc/net/stat/nf_conntrack</code>. The connection tracker keeps track of all connections of the machine, inbound and outbound. It works by keeping a database with all open connections, tracking network and address translation and connection expectations.' + }, + + 'netfilter.nfacct': { + title: 'Bandwidth Accounting', + info: 'The following information is read using the <code>nfacct.plugin</code>.' + }, + + 'netfilter.synproxy': { + title: 'DDoS Protection', + info: 'DDoS Protection performance monitoring read from <code>/proc/net/stat/synproxy</code>. <a href="https://github.com/firehol/firehol/wiki/Working-with-SYNPROXY" target="_blank">SYNPROXY</a> is a TCP SYN packets proxy. It is used to protect any TCP server (like a web server) from SYN floods and similar DDoS attacks. It is a netfilter module, in the Linux kernel (since version 3.12). It is optimized to handle millions of packets per second utilizing all CPUs available without any concurrency locking between the connections. It can be used for any kind of TCP traffic (even encrypted), since it does not interfere with the content itself.' + } }; +// +// chartData works on the context of a chart +// Its purpose is to set: +// +// info: the text above the charts +// heads: the representation of the chart at the top the subsection (second level menu) +// mainheads: the representation of the chart at the top of the section (first level menu) +// colors: the dimension colors of the chart (the default colors are appended) +// height: the ratio of the chart height relative to the default +// var chartData = { - 'system.cpu': { - info: 'Total CPU utilization (all cores). 100% here means there is no CPU idle time at all. You can get per core usage at the <a href="#cpu">CPUs</a> section and per application usage at the <a href="#apps">Applications Monitoring</a> section.<br/>Keep an eye on <b>iowait</b> ' + sparkline('system.cpu', 'iowait', '%') + '. If it is constantly high, your disks are a bottleneck and they slow your system down.<br/>Another important metric worth monitoring, is <b>softirq</b> ' + sparkline('system.cpu', 'softirq', '%') + '. A constantly high percentage of softirq may indicate network drivers issues.' - }, - - 'system.load': { - info: 'Current system load read from <code>/proc/loadavg</code>.', - height: 0.7 - }, - - 'system.io': { - info: 'Total Disk I/O, for all disks, read from <code>/proc/vmstat</code>. You can get detailed information about each disk at the <a href="#disk">Disks</a> section and per application Disk usage at the <a href="#apps">Applications Monitoring</a> section.' - }, - - 'system.swapio': { - info: 'Total Swap I/O, read from <code>/proc/vmstat</code>. (netdata measures both <code>in</code> and <code>out</code>. If either of them is not shown in the chart, it is because it is zero - you can change the page settings to always render all the available dimensions on all charts).' - }, - - 'system.pgfaults': { - info: 'Total page faults, read from <code>/proc/vmstat</code>. <b>Major page faults</b> indicates that the system is using its swap. You can find which applications use the swap at the <a href="#apps">Applications Monitoring</a> section.' - }, - - 'system.entropy': { - colors: '#CC22AA', - info: '<a href="https://en.wikipedia.org/wiki/Entropy_(computing)" target="_blank">Entropy</a>, read from <code>/proc/sys/kernel/random/entropy_avail</code>, is like a pool of random numbers that are mainly used in cryptography. It is advised that the pool remains always <a href="https://blog.cloudflare.com/ensuring-randomness-with-linuxs-random-number-generator/" target="_blank">above 200</a>. If the pool of entropy gets empty, you risk your security to be predictable and you should install a user-space random numbers generating daemon, like <a href="http://www.issihosts.com/haveged/" target="_blank">haveged</a>, to keep the pool in healthy levels.' - }, - - 'system.forks': { - colors: '#5555DD', - info: 'The number of new processes created per second, read from <code>/proc/stat</code>.' - }, - - 'system.intr': { - colors: '#DD5555', - info: 'Total number of CPU interrupts, read from <code>/proc/stat</code>. Check <code>system.interrupts</code> that gives more detail about each interrupt and also the <a href="#cpu">CPUs</a> section where interrupts are analyzed per CPU core.' - }, - - 'system.interrupts': { - info: 'CPU interrupts in detail, read from <code>/proc/interrupts</code>. At the <a href="#cpu">CPUs</a> section, interrupts are analyzed per CPU core.' - }, - - 'system.softirqs': { - info: 'CPU softirqs in detail, read from <code>/proc/softirqs</code>. At the <a href="#cpu">CPUs</a> section, softirqs are analyzed per CPU core.' - }, - - 'system.processes': { - info: 'System processes, read from <code>/proc/stat</code>. <b>Blocked</b> are processes that are willing to execute but they cannot, e.g. because they wait for disk activity.' - }, - - 'system.active_processes': { - info: 'All system active processes, read from <code>/proc/loadavg</code>.' - }, - - 'system.ctxt': { - info: '<a href="https://en.wikipedia.org/wiki/Context_switch" target="_blank">Context Switches</a>, read from <code>/proc/stat</code>, is the switching of the CPU from one process, task or thread to another. If there are many processes or threads willing to execute and very few CPU cores available to handle them, the system is making more context switching to balance the CPU resources among them. The whole process is computationally intensive. The more the context switches, the slower the system gets.' - }, - - 'system.idlejitter': { - colors: '#5555AA', - info: 'Idle jitter is calculated by netdata. A thread is spawned that requests to sleep for a few microseconds. When the system wakes it up, it measures how many microseconds have passed. The difference between the requested and the actual duration of the sleep, is the <b>idle jitter</b>. This number is useful in real-time environments, where CPU jitter can affect the quality of the service (like VoIP media gateways).' - }, - - 'system.ipv4': { - info: 'Total IPv4 Traffic, read from <code>/proc/net/netstat</code>.' - }, - - 'system.ipv6': { - info: 'Total IPv6 Traffic, read from <code>/proc/net/snmp6</code>.' - }, - - 'system.ram': { - info: 'System memory, read from <code>/proc/meminfo</code>.' - }, - - 'system.swap': { - info: 'System swap memory, read from <code>/proc/meminfo</code>.' - }, - - 'mem.ksm_savings': { - heads: [ - gaugeChart('Saved', '12%', 'savings', '#0099CC') - ] - }, - - 'mem.ksm_ratios': { - heads: [ - function(id) { - return '<div data-netdata="' + id + '"' - + ' data-gauge-max-value="100"' - + ' data-chart-library="gauge"' - + ' data-title="Savings"' - + ' data-units="percentage %"' - + ' data-gauge-adjust="width"' - + ' data-width="12%"' - + ' data-before="0"' - + ' data-after="-CHART_DURATION"' - + ' data-points="CHART_DURATION"' - + ' role="application"></div>'; - } - ] - }, - - 'mem.committed': { - colors: NETDATA.colors[3] - }, - - 'apps.cpu': { - height: 2.0 - }, - - 'apps.preads': { - height: 2.0 - }, - - 'apps.pwrites': { - height: 2.0 - }, - - 'users.cpu': { - height: 2.0 - }, - - 'users.preads': { - height: 2.0 - }, - - 'users.pwrites': { - height: 2.0 - }, - - 'groups.cpu': { - height: 2.0 - }, - - 'groups.preads': { - height: 2.0 - }, - - 'groups.pwrites': { - height: 2.0 - }, - - 'tc.qos': { - heads: [ - function(id) { - if(id.match(/.*-ifb$/)) - return gaugeChart('Inbound', '12%', '', '#5555AA'); - else - return gaugeChart('Outbound', '12%', '', '#AA9900'); - } - ] - }, - - 'net.net': { - heads: [ - gaugeChart('Received', '12%', 'received'), - gaugeChart('Sent', '12%', 'sent') - ] - }, - - 'disk.util': { - colors: '#FF5588', - heads: [ - gaugeChart('Utilization', '12%', '', '#FF5588') - ] - }, - - 'disk.backlog': { - colors: '#0099CC' - }, - - 'disk.io': { - heads: [ - gaugeChart('Read', '12%', 'reads'), - gaugeChart('Write', '12%', 'writes') - ] - }, - - 'netfilter.sockets': { - colors: '#88AA00', - heads: [ - gaugeChart('Active Connections', '12%', '', '#88AA00') - ] - }, - - 'netfilter.new': { - heads: [ - gaugeChart('New Connections', '12%', 'new', '#5555AA') - ] - }, - - 'disk.iotime': { - height: 0.5 - }, - 'disk.mops': { - height: 0.5 - }, - 'disk.svctm': { - height: 0.5 - }, - 'disk.avgsz': { - height: 0.5 - }, - 'disk.await': { - height: 0.5 - }, - - 'apache.connections': { - colors: NETDATA.colors[4], - mainheads: [ - gaugeChart('Connections', '12%', '', NETDATA.colors[4]) - ] - }, - - 'apache.requests': { - colors: NETDATA.colors[0], - mainheads: [ - gaugeChart('Connections', '12%', '', NETDATA.colors[0]) - ] - }, - - 'apache.net': { - colors: NETDATA.colors[3], - mainheads: [ - gaugeChart('Bandwidth', '12%', '', NETDATA.colors[3]) - ] - }, - - 'apache.workers': { - mainheads: [ - function(id) { - return '<div data-netdata="' + id + '"' - + ' data-dimensions="busy"' - + ' data-append-options="percentage"' - + ' data-gauge-max-value="100"' - + ' data-chart-library="gauge"' - + ' data-title="Workers Utilization"' - + ' data-units="percentage %"' - + ' data-gauge-adjust="width"' - + ' data-width="12%"' - + ' data-before="0"' - + ' data-after="-CHART_DURATION"' - + ' data-points="CHART_DURATION"' - + ' role="application"></div>'; - } - ] - }, - - 'apache.bytesperreq': { - colors: NETDATA.colors[3], - height: 0.5 - }, - - 'apache.reqpersec': { - colors: NETDATA.colors[4], - height: 0.5 - }, - - 'apache.bytespersec': { - colors: NETDATA.colors[6], - height: 0.5 - }, - - 'nginx.connections': { - colors: NETDATA.colors[4], - mainheads: [ - gaugeChart('Connections', '12%', '', NETDATA.colors[4]) - ] - }, - - 'nginx.requests': { - colors: NETDATA.colors[0], - mainheads: [ - gaugeChart('Requests', '12%', '', NETDATA.colors[0]) - ] - } + 'system.cpu': { + info: 'Total CPU utilization (all cores). 100% here means there is no CPU idle time at all. You can get per core usage at the <a href="#cpu">CPUs</a> section and per application usage at the <a href="#apps">Applications Monitoring</a> section.<br/>Keep an eye on <b>iowait</b> ' + sparkline('system.cpu', 'iowait', '%') + '. If it is constantly high, your disks are a bottleneck and they slow your system down.<br/>Another important metric worth monitoring, is <b>softirq</b> ' + sparkline('system.cpu', 'softirq', '%') + '. A constantly high percentage of softirq may indicate network drivers issues.' + }, + + 'system.load': { + info: 'Current system load, i.e. the number of processes using CPU or waiting for system resources (usually CPU and disk). The 3 metrics refer to 1, 5 and 15 minute averages. Linux calculates this once every 5 seconds. Netdata reads them from <code>/proc/loadavg</code>. For more information check <a href="https://en.wikipedia.org/wiki/Load_(computing)" target="_blank">this wikipedia article</a>', + height: 0.7 + }, + + 'system.io': { + info: 'Total Disk I/O, for all disks, read from <code>/proc/vmstat</code>. You can get detailed information about each disk at the <a href="#disk">Disks</a> section and per application Disk usage at the <a href="#apps">Applications Monitoring</a> section.' + }, + + 'system.swapio': { + info: 'Total Swap I/O, read from <code>/proc/vmstat</code>. (netdata measures both <code>in</code> and <code>out</code>. If either of them is not shown in the chart, it is because it is zero - you can change the page settings to always render all the available dimensions on all charts).' + }, + + 'system.pgfaults': { + info: 'Total page faults, read from <code>/proc/vmstat</code>. <b>Major page faults</b> indicates that the system is using its swap. You can find which applications use the swap at the <a href="#apps">Applications Monitoring</a> section.' + }, + + 'system.entropy': { + colors: '#CC22AA', + info: '<a href="https://en.wikipedia.org/wiki/Entropy_(computing)" target="_blank">Entropy</a>, read from <code>/proc/sys/kernel/random/entropy_avail</code>, is like a pool of random numbers that are mainly used in cryptography. It is advised that the pool remains always <a href="https://blog.cloudflare.com/ensuring-randomness-with-linuxs-random-number-generator/" target="_blank">above 200</a>. If the pool of entropy gets empty, you risk your security to be predictable and you should install a user-space random numbers generating daemon, like <a href="http://www.issihosts.com/haveged/" target="_blank">haveged</a>, to keep the pool in healthy levels.' + }, + + 'system.forks': { + colors: '#5555DD', + info: 'The number of new processes created per second, read from <code>/proc/stat</code>.' + }, + + 'system.intr': { + colors: '#DD5555', + info: 'Total number of CPU interrupts, read from <code>/proc/stat</code>. Check <code>system.interrupts</code> that gives more detail about each interrupt and also the <a href="#cpu">CPUs</a> section where interrupts are analyzed per CPU core.' + }, + + 'system.interrupts': { + info: 'CPU interrupts in detail, read from <code>/proc/interrupts</code>. At the <a href="#cpu">CPUs</a> section, interrupts are analyzed per CPU core.' + }, + + 'system.softirqs': { + info: 'CPU softirqs in detail, read from <code>/proc/softirqs</code>. At the <a href="#cpu">CPUs</a> section, softirqs are analyzed per CPU core.' + }, + + 'system.processes': { + info: 'System processes, read from <code>/proc/stat</code>. <b>Blocked</b> are processes that are willing to execute but they cannot, e.g. because they wait for disk activity.' + }, + + 'system.active_processes': { + info: 'All system active processes, read from <code>/proc/loadavg</code>.' + }, + + 'system.ctxt': { + info: '<a href="https://en.wikipedia.org/wiki/Context_switch" target="_blank">Context Switches</a>, read from <code>/proc/stat</code>, is the switching of the CPU from one process, task or thread to another. If there are many processes or threads willing to execute and very few CPU cores available to handle them, the system is making more context switching to balance the CPU resources among them. The whole process is computationally intensive. The more the context switches, the slower the system gets.' + }, + + 'system.idlejitter': { + colors: '#5555AA', + info: 'Idle jitter is calculated by netdata. A thread is spawned that requests to sleep for a few microseconds. When the system wakes it up, it measures how many microseconds have passed. The difference between the requested and the actual duration of the sleep, is the <b>idle jitter</b>. This number is useful in real-time environments, where CPU jitter can affect the quality of the service (like VoIP media gateways).' + }, + + 'system.ipv4': { + info: 'Total IPv4 Traffic, read from <code>/proc/net/netstat</code>.' + }, + + 'system.ipv6': { + info: 'Total IPv6 Traffic, read from <code>/proc/net/snmp6</code>.' + }, + + 'system.ram': { + info: 'System memory, read from <code>/proc/meminfo</code>.' + }, + + 'system.swap': { + info: 'System swap memory, read from <code>/proc/meminfo</code>.' + }, + + // ------------------------------------------------------------------------ + // MEMORY + + 'mem.ksm_savings': { + heads: [ + gaugeChart('Saved', '12%', 'savings', '#0099CC') + ] + }, + + 'mem.ksm_ratios': { + heads: [ + function(id) { + return '<div data-netdata="' + id + '"' + + ' data-gauge-max-value="100"' + + ' data-chart-library="gauge"' + + ' data-title="Savings"' + + ' data-units="percentage %"' + + ' data-gauge-adjust="width"' + + ' data-width="12%"' + + ' data-before="0"' + + ' data-after="-CHART_DURATION"' + + ' data-points="CHART_DURATION"' + + ' role="application"></div>'; + } + ] + }, + + 'mem.committed': { + colors: NETDATA.colors[3] + }, + + // ------------------------------------------------------------------------ + // APPS + + 'apps.cpu': { + height: 2.0 + }, + + 'apps.preads': { + height: 2.0 + }, + + 'apps.pwrites': { + height: 2.0 + }, + + // ------------------------------------------------------------------------ + // USERS + + 'users.cpu': { + height: 2.0 + }, + + 'users.preads': { + height: 2.0 + }, + + 'users.pwrites': { + height: 2.0 + }, + + // ------------------------------------------------------------------------ + // GROUPS + + 'groups.cpu': { + height: 2.0 + }, + + 'groups.preads': { + height: 2.0 + }, + + 'groups.pwrites': { + height: 2.0 + }, + + // ------------------------------------------------------------------------ + // NETWORK QoS + + 'tc.qos': { + heads: [ + function(id) { + if(id.match(/.*-ifb$/)) + return gaugeChart('Inbound', '12%', '', '#5555AA'); + else + return gaugeChart('Outbound', '12%', '', '#AA9900'); + } + ] + }, + + // ------------------------------------------------------------------------ + // NETWORK INTERFACES + + 'net.net': { + heads: [ + gaugeChart('Received', '12%', 'received'), + gaugeChart('Sent', '12%', 'sent') + ] + }, + + // ------------------------------------------------------------------------ + // NETFILTER + + 'netfilter.sockets': { + colors: '#88AA00', + heads: [ + gaugeChart('Active Connections', '12%', '', '#88AA00') + ] + }, + + 'netfilter.new': { + heads: [ + gaugeChart('New Connections', '12%', 'new', '#5555AA') + ] + }, + + // ------------------------------------------------------------------------ + // DISKS + + 'disk.util': { + colors: '#FF5588', + heads: [ + gaugeChart('Utilization', '12%', '', '#FF5588') + ], + info: 'Disk Utilization measures the amount of time the disk was busy with something. This is not related to its performance. 100% means that the Linux kernel always had an outstanding operation on the disk. Keep in mind that depending on the underlying technology of the disk, 100% here may or may not be an indication of congestion.' + }, + + 'disk.backlog': { + colors: '#0099CC', + info: 'Backlog is an indication of the duration of pending disk operations. On every I/O event the Linux kernel is multiplying the time spent doing I/O since the last update of this field with the number of pending operations. While not accurate, this metric can provide an indication of the expected completion time of the operations in progress.' + }, + + 'disk.io': { + heads: [ + gaugeChart('Read', '12%', 'reads'), + gaugeChart('Write', '12%', 'writes') + ], + info: 'Amount of data transferred to and from disk.' + }, + + 'disk.ops': { + info: 'Completed disk I/O operations. Keep in mind the number of operations requested might be higher, since the Linux kernel is able to merge adjacent to each other (see merged operations chart).' + }, + + 'disk.qops': { + info: 'I/O operations currently in progress. This metric is a snapshot - it is not an average over the last interval.' + }, + + 'disk.iotime': { + height: 0.5, + info: 'The sum of the duration of all completed I/O operations. This number can exceed the interval if the disk is able to execute I/O operations in parallel.' + }, + 'disk.mops': { + height: 0.5, + info: 'The number of merged disk operations. The Linux kernel is able to merge adjacent I/O operations, for example two 4KB reads can become one 8KB read before given to disk.' + }, + 'disk.svctm': { + height: 0.5, + info: 'The average service time for completed I/O operations. This metric is calculated using the total busy time of the disk and the number of completed operations. If the disk is able to execute multiple parallel operations the reporting average service time will be misleading.' + }, + 'disk.avgsz': { + height: 0.5, + info: 'The average I/O operation size.' + }, + 'disk.await': { + height: 0.5, + info: 'The average time for I/O requests issued to the device to be served. This includes the time spent by the requests in queue and the time spent servicing them.' + }, + + 'disk.space': { + info: 'Disk space utilization. reserved for root is automatically reserved by the system to prevent the root user from getting out of space.' + }, + 'disk.inodes': { + info: 'inodes (or index nodes) are filesystem objects (e.g. files and directories). On many types of file system implementations, the maximum number of inodes is fixed at filesystem creation, limiting the maximum number of files the filesystem can hold. It is possible for a device to run out of inodes. When this happens, new files cannot be created on the device, even though there may be free space available.' + }, + + 'mysql.net': { + info: 'The amount of data sent to mysql clients (<strong>out</strong>) and received from mysql clients (<strong>in</strong>).' + }, + + // ------------------------------------------------------------------------ + // MYSQL + + 'mysql.queries': { + info: 'The number of statements executed by the server.<ul>' + + '<li><strong>queries</strong> counts the statements executed within stored SQL programs.</li>' + + '<li><strong>questions</strong> counts the statements sent to the mysql server by mysql clients.</li>' + + '<li><strong>slow queries</strong> counts the number of statements that took more than <a href="http://dev.mysql.com/doc/refman/5.7/en/server-system-variables.html#sysvar_long_query_time" target="_blank">long_query_time</a> seconds to be executed.' + + ' For more information about slow queries check the mysql <a href="http://dev.mysql.com/doc/refman/5.7/en/slow-query-log.html" target="_blank">slow query log</a>.</li>' + + '</ul>' + }, + + 'mysql.handlers': { + info: 'Usage of the internal handlers of mysql. This chart provides very good insights of what the mysql server is actually doing.' + + ' (if the chart is not showing all these dimensions it is because they are zero - set <strong>Which dimensions to show?</strong> to <strong>All</strong> from the dashboard settings, to render even the zero values)<ul>' + + '<li><strong>commit</strong>, the number of internal <a href="http://dev.mysql.com/doc/refman/5.7/en/commit.html" target="_blank">COMMIT</a> statements.</li>' + + '<li><strong>delete</strong>, the number of times that rows have been deleted from tables.</li>' + + '<li><strong>prepare</strong>, a counter for the prepare phase of two-phase commit operations.</li>' + + '<li><strong>read first</strong>, the number of times the first entry in an index was read. A high value suggests that the server is doing a lot of full index scans; e.g. <strong>SELECT col1 FROM foo</strong>, with col1 indexed.</li>' + + '<li><strong>read key</strong>, the number of requests to read a row based on a key. If this value is high, it is a good indication that your tables are properly indexed for your queries.</li>' + + '<li><strong>read next</strong>, the number of requests to read the next row in key order. This value is incremented if you are querying an index column with a range constraint or if you are doing an index scan.</li>' + + '<li><strong>read prev</strong>, the number of requests to read the previous row in key order. This read method is mainly used to optimize <strong>ORDER BY ... DESC</strong>.</li>' + + '<li><strong>read rnd</strong>, the number of requests to read a row based on a fixed position. A high value indicates you are doing a lot of queries that require sorting of the result. You probably have a lot of queries that require MySQL to scan entire tables or you have joins that do not use keys properly.</li>' + + '<li><strong>read rnd next</strong>, the number of requests to read the next row in the data file. This value is high if you are doing a lot of table scans. Generally this suggests that your tables are not properly indexed or that your queries are not written to take advantage of the indexes you have.</li>' + + '<li><strong>rollback</strong>, the number of requests for a storage engine to perform a rollback operation.</li>' + + '<li><strong>savepoint</strong>, the number of requests for a storage engine to place a savepoint.</li>' + + '<li><strong>savepoint rollback</strong>, the number of requests for a storage engine to roll back to a savepoint.</li>' + + '<li><strong>update</strong>, the number of requests to update a row in a table.</li>' + + '<li><strong>write</strong>, the number of requests to insert a row in a table.</li>' + + '</ul>' + }, + + 'mysql.table_locks': { + info: 'MySQL table locks counters: <ul>' + + '<li><strong>immediate</strong>, the number of times that a request for a table lock could be granted immediately.</li>' + + '<li><strong>waited</strong>, the number of times that a request for a table lock could not be granted immediately and a wait was needed. If this is high and you have performance problems, you should first optimize your queries, and then either split your table or tables or use replication.</li>' + + '</ul>' + }, + + // ------------------------------------------------------------------------ + // APACHE + + 'apache.connections': { + colors: NETDATA.colors[4], + mainheads: [ + gaugeChart('Connections', '12%', '', NETDATA.colors[4]) + ] + }, + + 'apache.requests': { + colors: NETDATA.colors[0], + mainheads: [ + gaugeChart('Requests', '12%', '', NETDATA.colors[0]) + ] + }, + + 'apache.net': { + colors: NETDATA.colors[3], + mainheads: [ + gaugeChart('Bandwidth', '12%', '', NETDATA.colors[3]) + ] + }, + + 'apache.workers': { + mainheads: [ + function(id) { + return '<div data-netdata="' + id + '"' + + ' data-dimensions="busy"' + + ' data-append-options="percentage"' + + ' data-gauge-max-value="100"' + + ' data-chart-library="gauge"' + + ' data-title="Workers Utilization"' + + ' data-units="percentage %"' + + ' data-gauge-adjust="width"' + + ' data-width="12%"' + + ' data-before="0"' + + ' data-after="-CHART_DURATION"' + + ' data-points="CHART_DURATION"' + + ' role="application"></div>'; + } + ] + }, + + 'apache.bytesperreq': { + colors: NETDATA.colors[3], + height: 0.5 + }, + + 'apache.reqpersec': { + colors: NETDATA.colors[4], + height: 0.5 + }, + + 'apache.bytespersec': { + colors: NETDATA.colors[6], + height: 0.5 + }, + + + // ------------------------------------------------------------------------ + // NGINX + + 'nginx.connections': { + colors: NETDATA.colors[4], + mainheads: [ + gaugeChart('Connections', '12%', '', NETDATA.colors[4]) + ] + }, + + 'nginx.requests': { + colors: NETDATA.colors[0], + mainheads: [ + gaugeChart('Requests', '12%', '', NETDATA.colors[0]) + ] + }, + + // ------------------------------------------------------------------------ + // NETDATA + + 'netdata.response_time': { + info: 'The netdata API response time measures the time netdata needed to serve requests. This time includes everything, from the reception of the first byte of a request, to the dispatch of the last byte of its reply, therefore it includes all network latencies involved (i.e. a client over a slow network will influence these metrics).' + } }; function anyAttribute(obj, attr, key, def) { - if(typeof obj[key] !== 'undefined') { - if(typeof obj[key][attr] !== 'undefined') - return obj[key][attr]; - } - return def; + if(typeof obj[key] !== 'undefined') { + if(typeof obj[key][attr] !== 'undefined') + return obj[key][attr]; + } + return def; } function menuTitle(chart) { - if(typeof chart.menu_pattern !== 'undefined') { - return anyAttribute(menuData, 'title', chart.menu_pattern, chart.menu_pattern).toString() - + ': ' + chart.type.slice(-(chart.type.length - chart.menu_pattern.length - 1)).toString(); - } + if(typeof chart.menu_pattern !== 'undefined') { + return anyAttribute(menuData, 'title', chart.menu_pattern, chart.menu_pattern).toString() + + ': ' + chart.type.slice(-(chart.type.length - chart.menu_pattern.length - 1)).toString(); + } - return anyAttribute(menuData, 'title', chart.menu, chart.menu); + return anyAttribute(menuData, 'title', chart.menu, chart.menu); } function menuInfo(menu) { - return anyAttribute(menuData, 'info', menu, null); + return anyAttribute(menuData, 'info', menu, null); } function menuHeight(menu, relative) { - return anyAttribute(menuData, 'height', menu, 1.0) * relative; + return anyAttribute(menuData, 'height', menu, 1.0) * relative; } function submenuTitle(menu, submenu) { - var key = menu + '.' + submenu; - return anyAttribute(submenuData, 'title', key, submenu); + var key = menu + '.' + submenu; + return anyAttribute(submenuData, 'title', key, submenu); } function submenuInfo(menu, submenu) { - var key = menu + '.' + submenu; - return anyAttribute(submenuData, 'info', key, null); + var key = menu + '.' + submenu; + return anyAttribute(submenuData, 'info', key, null); } function submenuHeight(menu, submenu, relative) { - var key = menu + '.' + submenu; - return anyAttribute(submenuData, 'height', key, 1.0) * relative; + var key = menu + '.' + submenu; + return anyAttribute(submenuData, 'height', key, 1.0) * relative; } function chartInfo(id) { - if(typeof chartData[id] !== 'undefined' && typeof chartData[id].info !== 'undefined') - return '<div class="chart-message netdata-chart-alignment" role="document">' + chartData[id].info + '</div>'; - else - return ''; + if(typeof chartData[id] !== 'undefined' && typeof chartData[id].info !== 'undefined') + return '<div class="chart-message netdata-chart-alignment" role="document">' + chartData[id].info + '</div>'; + else + return ''; } function chartHeight(id, def) { - if(typeof chartData[id] !== 'undefined' && typeof chartData[id].height !== 'undefined') - return def * chartData[id].height; - else - return def; + if(typeof chartData[id] !== 'undefined' && typeof chartData[id].height !== 'undefined') + return def * chartData[id].height; + else + return def; } @@ -1610,657 +2051,915 @@ function chartHeight(id, def) { // enrich the data structure returned by netdata // to reflect our menu system and content function enrichChartData(chart) { - var tmp = chart.type.split('_')[0]; - - switch(tmp) { - case 'ap': - case 'net': - case 'disk': - chart.menu = tmp; - break; - - case 'mysql': - case 'named': - case 'cgroup': - chart.menu = chart.type; - chart.menu_pattern = tmp; - break; - - case 'tc': - chart.menu = tmp; - - // find a name for this device from fireqos info - // we strip '_(in|out)' or '(in|out)_' - if(typeof options.submenu_names[chart.family] === 'undefined' || options.submenu_names[chart.family] === chart.family) { - var n = chart.name.split('.')[1]; - if(n.endsWith('_in')) - options.submenu_names[chart.family] = n.slice(0, n.lastIndexOf('_in')); - else if(n.endsWith('_out')) - options.submenu_names[chart.family] = n.slice(0, n.lastIndexOf('_out')); - else if(n.startsWith('in_')) - options.submenu_names[chart.family] = n.slice(3, n.length); - else if(n.startsWith('out_')) - options.submenu_names[chart.family] = n.slice(4, n.length); - } - - // increase the priority of IFB devices - // to have inbound appear before outbound - if(chart.id.match(/.*-ifb$/)) - chart.priority--; - - break; - - default: - chart.menu = chart.type; - break; - } - - chart.submenu = chart.family; + var tmp = chart.type.split('_')[0]; + + switch(tmp) { + case 'ap': + case 'net': + case 'disk': + chart.menu = tmp; + break; + + case 'cgroup': + chart.menu = chart.type; + if(chart.id.match(/.*[\._\/-:]qemu[\._\/-:]*/) || chart.id.match(/.*[\._\/-:]kvm[\._\/-:]*/)) + chart.menu_pattern = 'cgqemu'; + else + chart.menu_pattern = 'cgroup'; + break; + + case 'apache': + case 'exim': + case 'memcached': + case 'mysql': + case 'named': + case 'nginx': + case 'phpfpm': + case 'postfix': + case 'redis': + case 'ipfs': + case 'squid': + case 'snmp': + case 'tomcat': + chart.menu = chart.type; + chart.menu_pattern = tmp; + break; + + case 'tc': + chart.menu = tmp; + + // find a name for this device from fireqos info + // we strip '_(in|out)' or '(in|out)_' + if(typeof options.submenu_names[chart.family] === 'undefined' || options.submenu_names[chart.family] === chart.family) { + var n = chart.name.split('.')[1]; + if(n.endsWith('_in')) + options.submenu_names[chart.family] = n.slice(0, n.lastIndexOf('_in')); + else if(n.endsWith('_out')) + options.submenu_names[chart.family] = n.slice(0, n.lastIndexOf('_out')); + else if(n.startsWith('in_')) + options.submenu_names[chart.family] = n.slice(3, n.length); + else if(n.startsWith('out_')) + options.submenu_names[chart.family] = n.slice(4, n.length); + } + + // increase the priority of IFB devices + // to have inbound appear before outbound + if(chart.id.match(/.*-ifb$/)) + chart.priority--; + + break; + + default: + chart.menu = chart.type; + break; + } + + chart.submenu = chart.family; } // ---------------------------------------------------------------------------- function name2id(s) { - return s - .replace(/ /g, '_') - .replace(/\(/g, '_') - .replace(/\)/g, '_') - .replace(/\./g, '_') - .replace(/\//g, '_'); + return s + .replace(/ /g, '_') + .replace(/\(/g, '_') + .replace(/\)/g, '_') + .replace(/\./g, '_') + .replace(/\//g, '_'); } function headMain(charts, duration) { - var head = ''; - - if(typeof charts['system.swap'] !== 'undefined') - head += '<div style="margin-right: 10px;" data-netdata="system.swap"' - + ' data-dimensions="free"' - + ' data-append-options="percentage"' - + ' data-chart-library="easypiechart"' - + ' data-title="Free Swap"' - + ' data-units="%"' - + ' data-easypiechart-max-value="100"' - + ' data-width="8%"' - + ' data-before="0"' - + ' data-after="-' + duration.toString() + '"' - + ' data-points="' + duration.toString() + '"' - + ' data-colors="#DD4400"' - + ' role="application"></div>'; - - if(typeof charts['system.io'] !== 'undefined') { - head += '<div style="margin-right: 10px;" data-netdata="system.io"' - + ' data-dimensions="in"' - + ' data-chart-library="easypiechart"' - + ' data-title="Disk Read"' - + ' data-units="KB / s"' - + ' data-width="10%"' - + ' data-before="0"' - + ' data-after="-' + duration.toString() + '"' - + ' data-points="' + duration.toString() + '"' - + ' role="application"></div>'; - - head += '<div style="margin-right: 10px;" data-netdata="system.io"' - + ' data-dimensions="out"' - + ' data-chart-library="easypiechart"' - + ' data-title="Disk Write"' - + ' data-units="KB / s"' - + ' data-width="10%"' - + ' data-before="0"' - + ' data-after="-' + duration.toString() + '"' - + ' data-points="' + duration.toString() + '"' - + ' role="application"></div>'; - } - - if(typeof charts['system.cpu'] !== 'undefined') - head += '<div data-netdata="system.cpu"' - + ' data-chart-library="gauge"' - + ' data-title="CPU"' - + ' data-units="%"' - + ' data-gauge-max-value="100"' - + ' data-width="18%"' - + ' data-after="-' + duration.toString() + '"' - + ' data-points="' + duration.toString() + '"' - + ' data-colors="' + NETDATA.colors[12] + '"' - + ' role="application"></div>'; - - if(typeof charts['system.ipv4'] !== 'undefined') { - head += '<div style="margin-right: 10px;" data-netdata="system.ipv4"' - + ' data-dimensions="received"' - + ' data-chart-library="easypiechart"' - + ' data-title="IPv4 Inbound"' - + ' data-units="kbps"' - + ' data-width="10%"' - + ' data-before="0"' - + ' data-after="-' + duration.toString() + '"' - + ' data-points="' + duration.toString() + '"' - + ' role="application"></div>'; - - head += '<div style="margin-right: 10px;" data-netdata="system.ipv4"' - + ' data-dimensions="sent"' - + ' data-chart-library="easypiechart"' - + ' data-title="IPv4 Outbound"' - + ' data-units="kbps"' - + ' data-width="10%"' - + ' data-before="0"' - + ' data-after="-' + duration.toString() + '"' - + ' data-points="' + duration.toString() + '"' - + ' role="application"></div>'; - } - else if(typeof charts['system.ipv6'] !== 'undefined') { - head += '<div style="margin-right: 10px;" data-netdata="system.ipv6"' - + ' data-dimensions="received"' - + ' data-chart-library="easypiechart"' - + ' data-title="IPv6 Inbound"' - + ' data-units="kbps"' - + ' data-width="10%"' - + ' data-before="0"' - + ' data-after="-' + duration.toString() + '"' - + ' data-points="' + duration.toString() + '"' - + ' role="application"></div>'; - - head += '<div style="margin-right: 10px;" data-netdata="system.ipv6"' - + ' data-dimensions="sent"' - + ' data-chart-library="easypiechart"' - + ' data-title="IPv6 Outbound"' - + ' data-units="kbps"' - + ' data-width="10%"' - + ' data-before="0"' - + ' data-after="-' + duration.toString() + '"' - + ' data-points="' + duration.toString() + '"' - + ' role="application"></div>'; - } - - if(typeof charts['system.ram'] !== 'undefined') - head += '<div style="margin-right: 10px;" data-netdata="system.ram"' - + ' data-dimensions="cached|free"' - + ' data-append-options="percentage"' - + ' data-chart-library="easypiechart"' - + ' data-title="Available RAM"' - + ' data-units="%"' - + ' data-easypiechart-max-value="100"' - + ' data-width="8%"' - + ' data-after="-' + duration.toString() + '"' - + ' data-points="' + duration.toString() + '"' - + ' data-colors="' + NETDATA.colors[7] + '"' - + ' role="application"></div>'; - - return head; + var head = ''; + + if(typeof charts['system.swap'] !== 'undefined') + head += '<div style="margin-right: 10px;" data-netdata="system.swap"' + + ' data-dimensions="free"' + + ' data-append-options="percentage"' + + ' data-chart-library="easypiechart"' + + ' data-title="Free Swap"' + + ' data-units="%"' + + ' data-easypiechart-max-value="100"' + + ' data-width="8%"' + + ' data-before="0"' + + ' data-after="-' + duration.toString() + '"' + + ' data-points="' + duration.toString() + '"' + + ' data-colors="#DD4400"' + + ' role="application"></div>'; + + if(typeof charts['system.io'] !== 'undefined') { + head += '<div style="margin-right: 10px;" data-netdata="system.io"' + + ' data-dimensions="in"' + + ' data-chart-library="easypiechart"' + + ' data-title="Disk Read"' + + ' data-width="10%"' + + ' data-before="0"' + + ' data-after="-' + duration.toString() + '"' + + ' data-points="' + duration.toString() + '"' + + ' role="application"></div>'; + + head += '<div style="margin-right: 10px;" data-netdata="system.io"' + + ' data-dimensions="out"' + + ' data-chart-library="easypiechart"' + + ' data-title="Disk Write"' + + ' data-width="10%"' + + ' data-before="0"' + + ' data-after="-' + duration.toString() + '"' + + ' data-points="' + duration.toString() + '"' + + ' role="application"></div>'; + } + + if(typeof charts['system.cpu'] !== 'undefined') + head += '<div data-netdata="system.cpu"' + + ' data-chart-library="gauge"' + + ' data-title="CPU"' + + ' data-units="%"' + + ' data-gauge-max-value="100"' + + ' data-width="18%"' + + ' data-after="-' + duration.toString() + '"' + + ' data-points="' + duration.toString() + '"' + + ' data-colors="' + NETDATA.colors[12] + '"' + + ' role="application"></div>'; + + if(typeof charts['system.ipv4'] !== 'undefined') { + head += '<div style="margin-right: 10px;" data-netdata="system.ipv4"' + + ' data-dimensions="received"' + + ' data-chart-library="easypiechart"' + + ' data-title="IPv4 Inbound"' + + ' data-width="10%"' + + ' data-before="0"' + + ' data-after="-' + duration.toString() + '"' + + ' data-points="' + duration.toString() + '"' + + ' role="application"></div>'; + + head += '<div style="margin-right: 10px;" data-netdata="system.ipv4"' + + ' data-dimensions="sent"' + + ' data-chart-library="easypiechart"' + + ' data-title="IPv4 Outbound"' + + ' data-width="10%"' + + ' data-before="0"' + + ' data-after="-' + duration.toString() + '"' + + ' data-points="' + duration.toString() + '"' + + ' role="application"></div>'; + } + else if(typeof charts['system.ipv6'] !== 'undefined') { + head += '<div style="margin-right: 10px;" data-netdata="system.ipv6"' + + ' data-dimensions="received"' + + ' data-chart-library="easypiechart"' + + ' data-title="IPv6 Inbound"' + + ' data-units="kbps"' + + ' data-width="10%"' + + ' data-before="0"' + + ' data-after="-' + duration.toString() + '"' + + ' data-points="' + duration.toString() + '"' + + ' role="application"></div>'; + + head += '<div style="margin-right: 10px;" data-netdata="system.ipv6"' + + ' data-dimensions="sent"' + + ' data-chart-library="easypiechart"' + + ' data-title="IPv6 Outbound"' + + ' data-units="kbps"' + + ' data-width="10%"' + + ' data-before="0"' + + ' data-after="-' + duration.toString() + '"' + + ' data-points="' + duration.toString() + '"' + + ' role="application"></div>'; + } + + if(typeof charts['system.ram'] !== 'undefined') + head += '<div style="margin-right: 10px;" data-netdata="system.ram"' + + ' data-dimensions="cached|free"' + + ' data-append-options="percentage"' + + ' data-chart-library="easypiechart"' + + ' data-title="Available RAM"' + + ' data-units="%"' + + ' data-easypiechart-max-value="100"' + + ' data-width="8%"' + + ' data-after="-' + duration.toString() + '"' + + ' data-points="' + duration.toString() + '"' + + ' data-colors="' + NETDATA.colors[7] + '"' + + ' role="application"></div>'; + + return head; } function generateHeadCharts(type, chart, duration) { - var head = ''; - var hcharts = anyAttribute(chartData, type, chart.context, []); - if(hcharts.length > 0) { - var hi = 0, hlen = hcharts.length; - while(hi < hlen) { - if(typeof hcharts[hi] === 'function') - head += hcharts[hi](chart.id).replace('CHART_DURATION', duration.toString()).replace('CHART_UNIQUE_ID', chart.id); - else - head += hcharts[hi].replace('CHART_DURATION', duration.toString()).replace('CHART_UNIQUE_ID', chart.id); - hi++; - } - } - return head; + var head = ''; + var hcharts = anyAttribute(chartData, type, chart.context, []); + if(hcharts.length > 0) { + var hi = 0, hlen = hcharts.length; + while(hi < hlen) { + if(typeof hcharts[hi] === 'function') + head += hcharts[hi](chart.id).replace('CHART_DURATION', duration.toString()).replace('CHART_UNIQUE_ID', chart.id); + else + head += hcharts[hi].replace('CHART_DURATION', duration.toString()).replace('CHART_UNIQUE_ID', chart.id); + hi++; + } + } + return head; } function renderPage(menus, data) { - var div = document.getElementById('charts_div'); - var pcent_width = Math.floor(100 / chartsPerRow($(div).width())); - - // find the proper duration for per-second updates - var duration = Math.round(($(div).width() * pcent_width / 100 * data.update_every / 3) / 60) * 60; - var html = ''; - var sidebar = '<ul class="nav dashboard-sidenav" data-spy="affix" id="sidebar_ul">'; - var mainhead = headMain(data.charts, duration); - - // sort the menus - var main = sortObjectByPriority(menus); - var i = 0, len = main.length; - while(i < len) { - var menu = main[i++]; - - // generate an entry at the main menu - - var menuid = name2id(menu); - sidebar += '<li class=""><a href="#' + menuid + '">' + menus[menu].title + '</a><ul class="nav">'; - html += '<div role="section"><div role="sectionhead"><h1 id="' + menuid + '" role="heading">' + menus[menu].title + '</h1></div><div id="menu_' + menuid + '" role="document">'; - - if(menus[menu].info !== null) - html += menus[menu].info; - - // console.log(' >> ' + menu + ' (' + menus[menu].priority + '): ' + menus[menu].title); - - var shtml = ''; - var mhead = '<div class="netdata-chart-row">' + mainhead; - mainhead = ''; - - // sort the submenus of this menu - var sub = sortObjectByPriority(menus[menu].submenus); - var si = 0, slen = sub.length; - while(si < slen) { - var submenu = sub[si++]; - - // generate an entry at the submenu - var submenuid = name2id(menu + '_' + submenu); - sidebar += '<li class><a href="#' + submenuid + '">' + menus[menu].submenus[submenu].title + '</a></li>'; - shtml += '<div class="netdata-group-container" id="submenu_' + submenuid + '" style="display: inline-block; width: ' + pcent_width.toString() + '%"><h2 id="' + submenuid + '" class="netdata-chart-alignment" role="heading">' + menus[menu].submenus[submenu].title + '</h2>'; - - if(menus[menu].submenus[submenu].info !== null) - shtml += '<div class="chart-message netdata-chart-alignment" role="document">' + menus[menu].submenus[submenu].info + '</div>'; - - var head = '<div class="netdata-chart-row">'; - var chtml = ''; - - // console.log(' \------- ' + submenu + ' (' + menus[menu].submenus[submenu].priority + '): ' + menus[menu].submenus[submenu].title); - - // sort the charts in this submenu of this menu - menus[menu].submenus[submenu].charts.sort(prioritySort); - var ci = 0, clen = menus[menu].submenus[submenu].charts.length; - while(ci < clen) { - var chart = menus[menu].submenus[submenu].charts[ci++]; - - // generate the submenu heading charts - mhead += generateHeadCharts('mainheads', chart, duration); - head += generateHeadCharts('heads', chart, duration); - - // generate the chart - chtml += chartInfo(chart.context) + '<div data-netdata="' + chart.id + '"' - + ' data-width="' + pcent_width.toString() + '%"' - + ' data-height="' + chartHeight(chart.context, options.chartsHeight).toString() + 'px"' - + ' data-before="0"' - + ' data-after="-' + duration.toString() + '"' - + ' data-id="' + name2id(options.hostname + '/' + chart.id) + '"' - + ' data-colors="' + anyAttribute(chartData, 'colors', chart.context, '') + '"' - + ' role="application"></div>'; - - // console.log(' \------- ' + chart.id + ' (' + chart.priority + '): ' + chart.context + ' height: ' + menus[menu].submenus[submenu].height); - } - - head += '</div>'; - shtml += head + chtml + '</div>'; - } - - mhead += '</div>'; - sidebar += '</ul></li>'; - html += mhead + shtml + '</div></div><hr role="separator"/>'; - } - - sidebar += '</ul>'; - div.innerHTML = html; - document.getElementById('sidebar').innerHTML = sidebar; - finalizePage(); + var div = document.getElementById('charts_div'); + var pcent_width = Math.floor(100 / chartsPerRow($(div).width())); + + // find the proper duration for per-second updates + var duration = Math.round(($(div).width() * pcent_width / 100 * data.update_every / 3) / 60) * 60; + var html = ''; + var sidebar = '<ul class="nav dashboard-sidenav" data-spy="affix" id="sidebar_ul">'; + var mainhead = headMain(data.charts, duration); + + // sort the menus + var main = sortObjectByPriority(menus); + var i = 0, len = main.length; + while(i < len) { + var menu = main[i++]; + + // generate an entry at the main menu + + var menuid = name2id(menu); + sidebar += '<li class=""><a href="#' + menuid + '" onClick="return scrollToId(\'' + menuid + '\');">' + menus[menu].title + '</a><ul class="nav">'; + html += '<div role="section"><div role="sectionhead"><h1 id="' + menuid + '" role="heading">' + menus[menu].title + '</h1></div><div id="menu_' + menuid + '" role="document">'; + + if(menus[menu].info !== null) + html += menus[menu].info; + + // console.log(' >> ' + menu + ' (' + menus[menu].priority + '): ' + menus[menu].title); + + var shtml = ''; + var mhead = '<div class="netdata-chart-row">' + mainhead; + mainhead = ''; + + // sort the submenus of this menu + var sub = sortObjectByPriority(menus[menu].submenus); + var si = 0, slen = sub.length; + while(si < slen) { + var submenu = sub[si++]; + + // generate an entry at the submenu + var submenuid = name2id(menu + '_' + submenu); + sidebar += '<li class><a href="#' + submenuid + '" onClick="return scrollToId(\'' + submenuid + '\');">' + menus[menu].submenus[submenu].title + '</a></li>'; + shtml += '<div class="netdata-group-container" id="submenu_' + submenuid + '" style="display: inline-block; width: ' + pcent_width.toString() + '%"><h2 id="' + submenuid + '" class="netdata-chart-alignment" role="heading">' + menus[menu].submenus[submenu].title + '</h2>'; + + if(menus[menu].submenus[submenu].info !== null) + shtml += '<div class="chart-message netdata-chart-alignment" role="document">' + menus[menu].submenus[submenu].info + '</div>'; + + var head = '<div class="netdata-chart-row">'; + var chtml = ''; + + // console.log(' \------- ' + submenu + ' (' + menus[menu].submenus[submenu].priority + '): ' + menus[menu].submenus[submenu].title); + + // sort the charts in this submenu of this menu + menus[menu].submenus[submenu].charts.sort(prioritySort); + var ci = 0, clen = menus[menu].submenus[submenu].charts.length; + while(ci < clen) { + var chart = menus[menu].submenus[submenu].charts[ci++]; + + // generate the submenu heading charts + mhead += generateHeadCharts('mainheads', chart, duration); + head += generateHeadCharts('heads', chart, duration); + + // generate the chart + chtml += chartInfo(chart.context) + '<div data-netdata="' + chart.id + '"' + + ' data-width="' + pcent_width.toString() + '%"' + + ' data-height="' + chartHeight(chart.context, options.chartsHeight).toString() + 'px"' + + ' data-before="0"' + + ' data-after="-' + duration.toString() + '"' + + ' data-id="' + name2id(options.hostname + '/' + chart.id) + '"' + + ' data-colors="' + anyAttribute(chartData, 'colors', chart.context, '') + '"' + + ' role="application"></div>'; + + // console.log(' \------- ' + chart.id + ' (' + chart.priority + '): ' + chart.context + ' height: ' + menus[menu].submenus[submenu].height); + } + + head += '</div>'; + shtml += head + chtml + '</div>'; + } + + mhead += '</div>'; + sidebar += '</ul></li>'; + html += mhead + shtml + '</div></div><hr role="separator"/>'; + } + + sidebar += '</ul>'; + div.innerHTML = html; + document.getElementById('sidebar').innerHTML = sidebar; + finalizePage(); } function renderChartsAndMenu(data) { - var menus = {}; - var charts = data.charts; - - for(var c in charts) { - enrichChartData(charts[c]); - - // create the menu - if(typeof menus[charts[c].menu] === 'undefined') { - menus[charts[c].menu] = { - priority: charts[c].priority, - submenus: {}, - title: menuTitle(charts[c]), - info: menuInfo(charts[c].menu), - height: menuHeight(charts[c].menu, options.chartsHeight) - }; - } - - if(charts[c].priority < menus[charts[c].menu].priority) - menus[charts[c].menu].priority = charts[c].priority; - - // create the submenu - if(typeof menus[charts[c].menu].submenus[charts[c].submenu] === 'undefined') { - menus[charts[c].menu].submenus[charts[c].submenu] = { - priority: charts[c].priority, - charts: new Array(), - title: null, - info: submenuInfo(charts[c].menu, charts[c].submenu), - height: submenuHeight(charts[c].menu, charts[c].submenu, menus[charts[c].menu].height) - }; - } - - if(charts[c].priority < menus[charts[c].menu].submenus[charts[c].submenu].priority) - menus[charts[c].menu].submenus[charts[c].submenu].priority = charts[c].priority; - - // index the chart in the menu/submenu - menus[charts[c].menu].submenus[charts[c].submenu].charts.push(charts[c]); - } - - // propagate the descriptive subname given to QoS - // to all the other submenus with the same name - for(var m in menus) { - for(var s in menus[m].submenus) { - // set the family using a name - if(typeof options.submenu_names[s] !== 'undefined') { - menus[m].submenus[s].title = s + ' (' + options.submenu_names[s] + ')'; - } - else { - menus[m].submenus[s].title = submenuTitle(m, s); - } - } - } - - renderPage(menus, data); + var menus = options.menus; + var charts = data.charts; + + for(var c in charts) { + enrichChartData(charts[c]); + + // create the menu + if(typeof menus[charts[c].menu] === 'undefined') { + menus[charts[c].menu] = { + priority: charts[c].priority, + submenus: {}, + title: menuTitle(charts[c]), + info: menuInfo(charts[c].menu), + height: menuHeight(charts[c].menu, options.chartsHeight) + }; + } + + if(charts[c].priority < menus[charts[c].menu].priority) + menus[charts[c].menu].priority = charts[c].priority; + + // create the submenu + if(typeof menus[charts[c].menu].submenus[charts[c].submenu] === 'undefined') { + menus[charts[c].menu].submenus[charts[c].submenu] = { + priority: charts[c].priority, + charts: new Array(), + title: null, + info: submenuInfo(charts[c].menu, charts[c].submenu), + height: submenuHeight(charts[c].menu, charts[c].submenu, menus[charts[c].menu].height) + }; + } + + if(charts[c].priority < menus[charts[c].menu].submenus[charts[c].submenu].priority) + menus[charts[c].menu].submenus[charts[c].submenu].priority = charts[c].priority; + + // index the chart in the menu/submenu + menus[charts[c].menu].submenus[charts[c].submenu].charts.push(charts[c]); + } + + // propagate the descriptive subname given to QoS + // to all the other submenus with the same name + for(var m in menus) { + for(var s in menus[m].submenus) { + // set the family using a name + if(typeof options.submenu_names[s] !== 'undefined') { + menus[m].submenus[s].title = s + ' (' + options.submenu_names[s] + ')'; + } + else { + menus[m].submenus[s].title = submenuTitle(m, s); + } + } + } + + renderPage(menus, data); +} + +function alarmsUpdateModal() { + var active = '<h3>Raised Alarms</h3><table class="table">'; + var all = '<h3>All Running Alarms</h3><table class="table">'; + var log = '<h3>Alarm Log</h3><table class="table"><tr><th>When</th><th>Chart</th><th>Alarm</th><th>Status</th></tr>'; + var footer = '<hr/>These are <a href="https://github.com/firehol/netdata/wiki/Generating-Badges" target="_blank">netdata badges</a>. You can copy and paste their URLs to embed them in any web page. Their color indicates the state of the alarm: <b>red</b> is critical, <b>orange</b> is warning, <b>bright green</b> is ok, <b>light grey</b> is undefined (i.e. no data or no status), <b>black</b> is not initialized.'; + + NETDATA.alarms.get('all', function(data) { + if(data === null) { + document.getElementById('alarms_active').innerHTML = + document.getElementById('alarms_all').innerHTML = + document.getElementById('alarms_log').innerHTML = + 'failed to load alarm data!'; + return; + } + + var now = new Date().getTime(); + var x; + var count_active = 0; + var count_all = 0; + var families = {}; + for(x in data.alarms) { + var alarm = data.alarms[x]; + var chart = options.data.charts[alarm.chart]; + var family = alarm.family; + + if(typeof chart !== 'undefined' && typeof chart.menu !== 'undefined' && typeof chart.submenu !== 'undefined') + family = chart.menu + ' - ' + chart.submenu; + + if(typeof families[family] === 'undefined') + families[family] = new Array(); + + families[family].push(alarm); + } + + for(x in families) { + var active_family_added = false; + var all_family_added = false; + + var arr = families[x]; + var c = arr.length; + while(c--) { + var alarm = arr[c]; + if(alarm.status === 'WARNING' || alarm.status === 'CRITICAL') { + if(!active_family_added) { + active_family_added = true; + active += '<tr><th class="text-center"><h4>' + x + '</h4></th><th></th></tr>'; + } + active += '<tr><td class="text-center" style="vertical-align:middle"><embed><img src="' + NETDATA.serverDefault + '/api/v1/badge.svg?chart=' + alarm.chart + '&alarm=' + alarm.name + '&refresh=auto&_=' + now.toString()+ '"></img></embed></td><td><b>' + alarm.info + '</b><table>' + ((typeof alarm.warn !== 'undefined')?('<tr><td> warning when </td><td><code>' + alarm.warn + '</code></td></tr>'):'') + ((typeof alarm.crit !== 'undefined')?('<tr><td style="text-align:right"> critical when </td><td><code>' + alarm.crit + '</code></td></tr>'):'') + '<tr><td style="text-align:right"><small> source </small></td><td><small>' + alarm.source + '</small></td></tr></table></td></tr>'; + count_active++; + } + + if(!all_family_added) { + all_family_added = true; + all += '<tr><th class="text-center"><h4>' + x + '</h4></th><th></th></tr>'; + } + all += '<tr><td class="text-center" style="vertical-align:middle"><embed><img src="' + NETDATA.serverDefault + '/api/v1/badge.svg?chart=' + alarm.chart + '&alarm=' + alarm.name + '&refresh=auto&_=' + now.toString()+ '"></img></embed></td><td><b>' + alarm.info + '</b><table>' + ((typeof alarm.warn !== 'undefined')?('<tr><td> warning when </td><td><code>' + alarm.warn + '</code></td></tr>'):'') + ((typeof alarm.crit !== 'undefined')?('<tr><td style="text-align:right"> critical when </td><td><code>' + alarm.crit + '</code></td></tr>'):'') + '<tr><td style="text-align:right"><small> source </small></td><td><small>' + alarm.source + '</small></td></tr></table></td></tr>'; + count_all++; + } + } + active += "</table>"; + all += "</table>"; + + if(!count_active) + active += "<h4>Everything is normal. No raised alarms.</h4>"; + else + active += footer; + + if(!count_all) + all += "<h4>No alarms are running in this system.</h4>"; + else + all += footer; + + document.getElementById('alarms_active').innerHTML = active; + document.getElementById('alarms_all').innerHTML = all; + + NETDATA.alarms.get_log(function(data) { + if(data === null) { + document.getElementById('alarms_log').innerHTML = + 'failed to load alarm data!'; + return; + } + + var i = 0; + var len = data.length; + if(len > 50) len = 50; + while(i < len) { + var time = new Date(data[i].when * 1000); + log += '<tr><td>' + time.toLocaleDateString() + ' ' + time.toLocaleTimeString() + '</td><td>' + data[i].chart + '</td><td>' + data[i].name + '</td><td>' + data[i].status + '</td></tr>'; + i++; + } + log += "</table>"; + + if(i == 0) + log += "<h4>No alarms have been logged in this system.</h4>"; + + document.getElementById('alarms_log').innerHTML = log; + }) + }); +} + +function alarmsCallback(data) { + var count = 0; + for(x in data.alarms) + count++; + + if(count > 0) + document.getElementById('alarms_count_badge').innerHTML = count.toString(); + else + document.getElementById('alarms_count_badge').innerHTML = ''; } function downloadAllCharts(netdata_url) { - if(typeof netdata_url === 'undefined' || netdata_url === null) - netdata_url = NETDATA.serverDefault; + if(typeof netdata_url === 'undefined' || netdata_url === null) + netdata_url = NETDATA.serverDefault; + + NETDATA.pause(function() { + $("#loadOverlay").css("display","none"); + $("#loadOverlay").css("display","none"); - NETDATA.pause(function() { + NETDATA.alarms.callback = alarmsCallback; - // download all the charts the server knows - NETDATA.chartRegistry.downloadAll(netdata_url, function(data) { + // download all the charts the server knows + NETDATA.chartRegistry.downloadAll(netdata_url, function(data) { - options.data = data; - options.hostname = data.hostname; - document.getElementById('hostname').innerHTML = options.hostname; - document.getElementById('hostname').href = NETDATA.serverDefault; - document.title = options.hostname + ' dashboard'; + options.data = data; + options.hostname = data.hostname; + document.getElementById('hostname').innerHTML = options.hostname; + document.getElementById('hostname').href = NETDATA.serverDefault; + document.title = options.hostname + ' netdata dashboard'; - renderChartsAndMenu(data); + renderChartsAndMenu(data); - // prepare our DOM -// prepareScreen(data); + // prepare our DOM +// prepareScreen(data); - // due to affix issues, prepareScreen() will unpause - // netdata as needed - }); - }); + // due to affix issues, prepareScreen() will unpause + // netdata as needed + }); + }); } // ---------------------------------------------------------------------------- function versionLog(msg) { - document.getElementById('versionCheckLog').innerHTML = msg; + document.getElementById('versionCheckLog').innerHTML = msg; } function getNetdataVersion(callback) { - versionLog('Downloading installed version info from netdata...'); - - $.ajax({ - url: 'version.txt', - async: true, - cache: false - }) - .done(function(data) { - data = data.replace(/(\r\n|\n|\r| |\t)/gm,""); - if(data.length !== 40) { - versionLog('Received version string is invalid.'); - callback(null); - } - else { - versionLog('Installed version of netdata is ' + data); - document.getElementById('netdataVersion').innerHTML = data; - callback(data); - } - }) - .fail(function() { - versionLog('Failed to download installed version info from netdata!'); - callback(null); - }); + versionLog('Downloading installed version info from netdata...'); + + $.ajax({ + url: 'version.txt', + async: true, + cache: false + }) + .done(function(data) { + data = data.replace(/(\r\n|\n|\r| |\t)/gm,""); + if(data.length !== 40) { + versionLog('Received version string is invalid.'); + callback(null); + } + else { + versionLog('Installed version of netdata is ' + data); + document.getElementById('netdataVersion').innerHTML = data; + callback(data); + } + }) + .fail(function() { + versionLog('Failed to download installed version info from netdata!'); + callback(null); + }); } function getGithubLatestCommit(callback) { - versionLog('Downloading latest version info from github...'); - - $.ajax({ - url: 'https://api.github.com/repos/firehol/netdata/commits', - async: true, - cache: false - }) - .done(function(data) { - versionLog('Latest version info from github is ' + data[0].sha); - callback(data[0].sha); - }) - .fail(function() { - versionLog('Failed to download installed version info from github!'); - callback(null); - }); + versionLog('Downloading latest version info from github...'); + + $.ajax({ + url: 'https://api.github.com/repos/firehol/netdata/commits', + async: true, + cache: false + }) + .done(function(data) { + versionLog('Latest version info from github is ' + data[0].sha); + callback(data[0].sha); + }) + .fail(function() { + versionLog('Failed to download installed version info from github!'); + callback(null); + }); } function checkForUpdate(callback) { - getNetdataVersion(function(sha1) { - if(sha1 === null) callback(null, null); + getNetdataVersion(function(sha1) { + if(sha1 === null) callback(null, null); - getGithubLatestCommit(function(sha2) { - callback(sha1, sha2); - }); - }); + getGithubLatestCommit(function(sha2) { + callback(sha1, sha2); + }); + }); - return null; + return null; } -var updateBlinkCounter = 0; function notifyForUpdate(force) { - versionLog('<p>checking for updates...</p>'); - - var now = new Date().getTime(); - - if(typeof force === 'undefined' || force !== true) { - var last = loadLocalStorage('last_update_check'); - - if(typeof last === 'string') - last = parseInt(last); - else - last = 0; - - if(now - last < 3600000 * 8) { - // no need to check it - too soon - return; - } - } - - checkForUpdate(function(sha1, sha2) { - var save = false; - - if(sha1 === null) { - save = false; - versionLog('<p><big>Failed to get your netdata version!</big></p><p>You can always get the latest version of netdata from <a href="https://github.com/firehol/netdata" target="_blank">its github page</a>.</p>'); - } - else if(sha2 === null) { - save = false; - versionLog('<p><big>Failed to get the latest version from github.</big></p><p>You can always get the latest version of netdata from <a href="https://github.com/firehol/netdata" target="_blank">its github page</a>.</p>'); - } - else if(sha1 === sha2) { - save = true; - versionLog('<p><big>You already have the latest version of netdata!</big></p><p>No update yet?<br/>Probably, we need some motivation to keep going on!</p><p>If you haven\'t already, <a href="https://github.com/firehol/netdata" target="_blank">give netdata a <b>Star</b> at its github page</a>.</p>'); - } - else { - save = true; - var compare = 'https://github.com/firehol/netdata/compare/' + sha1.toString() + '...' + sha2.toString(); - - versionLog('<p><big><strong>New version of netdata available!</strong></big></p><p>Latest version: ' + sha2.toString() + '</p><p><a href="' + compare + '" target="_blank">Click here for the changes log</a> since your installed version, and<br/><a href="https://github.com/firehol/netdata/wiki/Updating-Netdata" target="_blank">click here for directions on updating</a> your netdata installation.</p><p>We suggest to review the changes log for new features you may be interested, or important bug fixes you may need.<br/>Keeping your netdata updated, is generally a good idea.</p>'); - - function updateButtonBlink() { - updateBlinkCounter--; - if(updateBlinkCounter > 0) - $('#updateButton').fadeOut(500).fadeIn(500, updateButtonBlink); - } - - if(updateBlinkCounter === 0) { - updateBlinkCounter = 300; - updateButtonBlink(); - } - } - - if(save) - saveLocalStorage('last_update_check', now.toString()); - }); -} + versionLog('<p>checking for updates...</p>'); -// ---------------------------------------------------------------------------- + var now = new Date().getTime(); -function getUrlParameter(sParam) { - var sPageURL = decodeURIComponent(window.location.search.substring(1)), - sURLVariables = sPageURL.split('&'), - sParameterName, - i; + if(typeof force === 'undefined' || force !== true) { + var last = loadLocalStorage('last_update_check'); - for (i = 0; i < sURLVariables.length; i++) { - sParameterName = sURLVariables[i].split('='); + if(typeof last === 'string') + last = parseInt(last); + else + last = 0; - if (sParameterName[0] === sParam) { - return sParameterName[1] === undefined ? true : sParameterName[1]; + if(now - last < 3600000 * 8) { + // no need to check it - too soon + return; } } + + checkForUpdate(function(sha1, sha2) { + var save = false; + + if(sha1 === null) { + save = false; + versionLog('<p><big>Failed to get your netdata version!</big></p><p>You can always get the latest version of netdata from <a href="https://github.com/firehol/netdata" target="_blank">its github page</a>.</p>'); + } + else if(sha2 === null) { + save = false; + versionLog('<p><big>Failed to get the latest version from github.</big></p><p>You can always get the latest version of netdata from <a href="https://github.com/firehol/netdata" target="_blank">its github page</a>.</p>'); + } + else if(sha1 === sha2) { + save = true; + versionLog('<p><big>You already have the latest version of netdata!</big></p><p>No update yet?<br/>Probably, we need some motivation to keep going on!</p><p>If you haven\'t already, <a href="https://github.com/firehol/netdata" target="_blank">give netdata a <b>Star</b> at its github page</a>.</p>'); + } + else { + save = true; + var compare = 'https://github.com/firehol/netdata/compare/' + sha1.toString() + '...' + sha2.toString(); + + versionLog('<p><big><strong>New version of netdata available!</strong></big></p><p>Latest version: ' + sha2.toString() + '</p><p><a href="' + compare + '" target="_blank">Click here for the changes log</a> since your installed version, and<br/><a href="https://github.com/firehol/netdata/wiki/Updating-Netdata" target="_blank">click here for directions on updating</a> your netdata installation.</p><p>We suggest to review the changes log for new features you may be interested, or important bug fixes you may need.<br/>Keeping your netdata updated, is generally a good idea.</p>'); + + document.getElementById('update_badge').innerHTML = '!'; + } + + if(save) + saveLocalStorage('last_update_check', now.toString()); + }); } +// ---------------------------------------------------------------------------- + function finalizePage() { - // resize all charts - without starting the background thread - // this has to be done while NETDATA is paused - // if we ommit this, the affix menu will be wrong, since all - // the Dom elements are initially zero-sized - NETDATA.parseDom(); - - var before = 0, after = 0, nowelcome = 0; - after = getUrlParameter('force_after_ms'); - before = getUrlParameter('force_before_ms'); - nowelcome = (getUrlParameter('nowelcome') === true)?true:false; - - if(before > 0 && after > 0) { - nowelcome = true; - NETDATA.globalPanAndZoom.setMaster(NETDATA.options.targets[0], after, before); - } - - // let it run (update the charts) - NETDATA.unpause(); - - // check if we have to jump to a specific section - var hash = location.hash.replace('#',''); - if(hash != '') { - // Clear the hash in the URL - // location.hash = ''; // delete front "//" if you want to change the address bar - var offset = $(location.hash).offset(); - if(typeof offset !== 'undefined') - $('html, body').animate({ scrollTop: offset.top }, 0); - } - - /* activate bootstrap sidebar (affix) */ - $('#sidebar').affix({ - offset: { - top: (isdemo())?150:0, - bottom: 0 - } - }); - - /* fix scrolling of very long affix lists - http://stackoverflow.com/questions/21691585/bootstrap-3-1-0-affix-too-long - */ - $('#sidebar').on('affixed.bs.affix', function() { - $(this).removeAttr('style'); - }); - - /* activate bootstrap scrollspy (needed for sidebar) */ - $(document.body).scrollspy({ - target: '#sidebar', - offset: $(window).height() / 3 // controls the diff of the <hX> element to the top, to select it - }); - - document.getElementById('footer').style.display = 'block'; - - - var update_options_modal = function() { - // console.log('update_options_modal'); - - var sync_option = function(option) { - var self = $('#' + option); - - if(self.prop('checked') !== NETDATA.getOption(option)) { - // console.log('switching ' + option.toString()); - self.bootstrapToggle(NETDATA.getOption(option)?'on':'off'); - } - } - - var theme_sync_option = function(option) { - var self = $('#' + option); - - self.bootstrapToggle(netdataTheme === 'slate'?'on':'off'); - } - - sync_option('eliminate_zero_dimensions'); - sync_option('destroy_on_hide'); - sync_option('parallel_refresher'); - sync_option('concurrent_refreshes'); - sync_option('sync_selection'); - sync_option('sync_pan_and_zoom'); - sync_option('stop_updates_when_focus_is_lost'); - sync_option('smooth_plot'); - sync_option('pan_and_zoom_data_padding'); - sync_option('show_help'); - theme_sync_option('netdata_theme_control'); - - if(NETDATA.getOption('parallel_refresher') === false) { - $('#concurrent_refreshes_row').hide(); - } - else { - $('#concurrent_refreshes_row').show(); - } - }; - NETDATA.setOption('setOptionCallback', update_options_modal); - - // handle options changes - $('#eliminate_zero_dimensions').change(function() { NETDATA.setOption('eliminate_zero_dimensions', $(this).prop('checked')); }); - $('#destroy_on_hide').change(function() { NETDATA.setOption('destroy_on_hide', $(this).prop('checked')); }); - $('#parallel_refresher').change(function() { NETDATA.setOption('parallel_refresher', $(this).prop('checked')); }); - $('#concurrent_refreshes').change(function() { NETDATA.setOption('concurrent_refreshes', $(this).prop('checked')); }); - $('#sync_selection').change(function() { NETDATA.setOption('sync_selection', $(this).prop('checked')); }); - $('#sync_pan_and_zoom').change(function() { NETDATA.setOption('sync_pan_and_zoom', $(this).prop('checked')); }); - $('#stop_updates_when_focus_is_lost').change(function() { NETDATA.setOption('stop_updates_when_focus_is_lost', $(this).prop('checked')); }); - $('#smooth_plot').change(function() { NETDATA.setOption('smooth_plot', $(this).prop('checked')); }); - $('#pan_and_zoom_data_padding').change(function() { NETDATA.setOption('pan_and_zoom_data_padding', $(this).prop('checked')); }); - $('#show_help').change(function() { NETDATA.setOption('show_help', $(this).prop('checked')); location.reload(); }); - - // this has to be the last - // it reloads the page - $('#netdata_theme_control').change(function() { - if(setTheme($(this).prop('checked')?'slate':'white')) - location.reload(); - }); - - $('#updateModal').on('shown.bs.modal', function() { - notifyForUpdate(true); - }); - - $('#deleteRegistryModal').on('hidden.bs.modal', function() { - deleteRegistryGuid = null; - }); - - if(isdemo()) { - if(!nowelcome) { - setTimeout(function() { - $('#welcomeModal').modal(); - }, 1000); - } - - // google analytics when this is used for the home page of the demo sites - // this does not run on user's installations - setTimeout(function() { - (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ - (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), - m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) - })(window,document,'script','https://www.google-analytics.com/analytics.js','ga'); - - ga('create', 'UA-64295674-3', 'auto'); - ga('send', 'pageview'); - }, 2000); - } - else notifyForUpdate(); + // resize all charts - without starting the background thread + // this has to be done while NETDATA is paused + // if we ommit this, the affix menu will be wrong, since all + // the Dom elements are initially zero-sized + NETDATA.parseDom(); + + if(urlOptions.pan_and_zoom === true) { + urlOptions.nowelcome = true; + NETDATA.globalPanAndZoom.setMaster(NETDATA.options.targets[0], urlOptions.after, urlOptions.before); + } + + // ------------------------------------------------------------------------ + // https://github.com/viralpatel/jquery.shorten/blob/master/src/jquery.shorten.js + $.fn.shorten = function(settings) { + "use strict"; + + var config = { + showChars: 750, + minHideChars: 10, + ellipsesText: "...", + moreText: '<i class="fa fa-expand" aria-hidden="true"></i> show more information', + lessText: '<i class="fa fa-compress" aria-hidden="true"></i> show less information', + onLess: function() { NETDATA.onscroll(); }, + onMore: function() { NETDATA.onscroll(); }, + errMsg: null, + force: false + }; + + if (settings) { + $.extend(config, settings); + } + + if ($(this).data('jquery.shorten') && !config.force) { + return false; + } + $(this).data('jquery.shorten', true); + + $(document).off("click", '.morelink'); + + $(document).on({ + click: function() { + + var $this = $(this); + if ($this.hasClass('less')) { + $this.removeClass('less'); + $this.html(config.moreText); + $this.parent().prev().animate({'height':'0'+'%'}, 0, function () { $this.parent().prev().prev().show(); }).hide(0, function() { + config.onLess(); + }); + + } else { + $this.addClass('less'); + $this.html(config.lessText); + $this.parent().prev().animate({'height':'100'+'%'}, 0, function () { $this.parent().prev().prev().hide(); }).show(0, function() { + config.onMore(); + }); + } + return false; + } + }, '.morelink'); + + return this.each(function() { + var $this = $(this); + + var content = $this.html(); + var contentlen = $this.text().length; + if (contentlen > config.showChars + config.minHideChars) { + var c = content.substr(0, config.showChars); + if (c.indexOf('<') >= 0) // If there's HTML don't want to cut it + { + var inTag = false; // I'm in a tag? + var bag = ''; // Put the characters to be shown here + var countChars = 0; // Current bag size + var openTags = []; // Stack for opened tags, so I can close them later + var tagName = null; + + for (var i = 0, r = 0; r <= config.showChars; i++) { + if (content[i] == '<' && !inTag) { + inTag = true; + + // This could be "tag" or "/tag" + tagName = content.substring(i + 1, content.indexOf('>', i)); + + // If its a closing tag + if (tagName[0] == '/') { + + + if (tagName != '/' + openTags[0]) { + config.errMsg = 'ERROR en HTML: the top of the stack should be the tag that closes'; + } else { + openTags.shift(); // Pops the last tag from the open tag stack (the tag is closed in the retult HTML!) + } + + } else { + // There are some nasty tags that don't have a close tag like <br/> + if (tagName.toLowerCase() != 'br') { + openTags.unshift(tagName); // Add to start the name of the tag that opens + } + } + } + if (inTag && content[i] == '>') { + inTag = false; + } + + if (inTag) { bag += content.charAt(i); } // Add tag name chars to the result + else { + r++; + if (countChars <= config.showChars) { + bag += content.charAt(i); // Fix to ie 7 not allowing you to reference string characters using the [] + countChars++; + } else // Now I have the characters needed + { + if (openTags.length > 0) // I have unclosed tags + { + //console.log('They were open tags'); + //console.log(openTags); + for (j = 0; j < openTags.length; j++) { + //console.log('Cierro tag ' + openTags[j]); + bag += '</' + openTags[j] + '>'; // Close all tags that were opened + + // You could shift the tag from the stack to check if you end with an empty stack, that means you have closed all open tags + } + break; + } + } + } + } + c = $('<div/>').html(bag + '<span class="ellip">' + config.ellipsesText + '</span>').html(); + }else{ + c+=config.ellipsesText; + } + + var html = '<div class="shortcontent">' + c + + '</div><div class="allcontent">' + content + + '</div><span><a href="javascript://nop/" class="morelink">' + config.moreText + '</a></span>'; + + $this.html(html); + $this.find(".allcontent").hide(); // Hide all text + $('.shortcontent p:last', $this).css('margin-bottom', 0); //Remove bottom margin on last paragraph as it's likely shortened + } + }); + + }; + $(".chart-message").shorten(); + // ------------------------------------------------------------------------ + + // callback for us to track PanAndZoom operations + NETDATA.globalPanAndZoom.callback = netdataPanAndZoomCallback; + + // let it run (update the charts) + NETDATA.unpause(); + + // check if we have to jump to a specific section + scrollToId(urlOptions.hash.replace('#','')); + + /* activate bootstrap sidebar (affix) */ + $('#sidebar').affix({ + offset: { + top: (isdemo())?150:0, + bottom: 0 + } + }); + + /* fix scrolling of very long affix lists + http://stackoverflow.com/questions/21691585/bootstrap-3-1-0-affix-too-long + */ + $('#sidebar').on('affixed.bs.affix', function() { + $(this).removeAttr('style'); + }); + + /* activate bootstrap scrollspy (needed for sidebar) */ + $(document.body).scrollspy({ + target: '#sidebar', + offset: $(window).height() / 3 // controls the diff of the <hX> element to the top, to select it + }); + + // change the URL based on the current position of the screen + $('#sidebar').on('activate.bs.scrollspy', function (e) { + var el = $(e.target); + if(el.find('ul').size() == 0) { + var hash = el.find('a').attr('href'); + if(typeof hash === 'string' && hash.substring(0, 1) == '#') { + urlOptions.hash = hash; + // console.log(urlOptions.hash); + netdataHashUpdate(); + } + } + }); + + document.getElementById('footer').style.display = 'block'; + + var update_options_modal = function() { + // console.log('update_options_modal'); + + var sync_option = function(option) { + var self = $('#' + option); + + if(self.prop('checked') !== NETDATA.getOption(option)) { + // console.log('switching ' + option.toString()); + self.bootstrapToggle(NETDATA.getOption(option)?'on':'off'); + } + } + + var theme_sync_option = function(option) { + var self = $('#' + option); + + self.bootstrapToggle(netdataTheme === 'slate'?'on':'off'); + } + + sync_option('eliminate_zero_dimensions'); + sync_option('destroy_on_hide'); + sync_option('parallel_refresher'); + sync_option('concurrent_refreshes'); + sync_option('sync_selection'); + sync_option('sync_pan_and_zoom'); + sync_option('stop_updates_when_focus_is_lost'); + sync_option('smooth_plot'); + sync_option('pan_and_zoom_data_padding'); + sync_option('show_help'); + theme_sync_option('netdata_theme_control'); + + if(NETDATA.getOption('parallel_refresher') === false) { + $('#concurrent_refreshes_row').hide(); + } + else { + $('#concurrent_refreshes_row').show(); + } + }; + NETDATA.setOption('setOptionCallback', update_options_modal); + + // handle options changes + $('#eliminate_zero_dimensions').change(function() { NETDATA.setOption('eliminate_zero_dimensions', $(this).prop('checked')); }); + $('#destroy_on_hide').change(function() { NETDATA.setOption('destroy_on_hide', $(this).prop('checked')); }); + $('#parallel_refresher').change(function() { NETDATA.setOption('parallel_refresher', $(this).prop('checked')); }); + $('#concurrent_refreshes').change(function() { NETDATA.setOption('concurrent_refreshes', $(this).prop('checked')); }); + $('#sync_selection').change(function() { NETDATA.setOption('sync_selection', $(this).prop('checked')); }); + $('#sync_pan_and_zoom').change(function() { NETDATA.setOption('sync_pan_and_zoom', $(this).prop('checked')); }); + $('#stop_updates_when_focus_is_lost').change(function() { NETDATA.setOption('stop_updates_when_focus_is_lost', $(this).prop('checked')); }); + $('#smooth_plot').change(function() { NETDATA.setOption('smooth_plot', $(this).prop('checked')); }); + $('#pan_and_zoom_data_padding').change(function() { NETDATA.setOption('pan_and_zoom_data_padding', $(this).prop('checked')); }); + $('#show_help').change(function() { + urlOptions.help = $(this).prop('checked'); + NETDATA.setOption('show_help', urlOptions.help); + netdataReload(); + }); + + // this has to be the last + // it reloads the page + $('#netdata_theme_control').change(function() { + urlOptions.theme = $(this).prop('checked')?'slate':'white'; + if(setTheme(urlOptions.theme)) + netdataReload(); + }); + + $('#updateModal').on('shown.bs.modal', function() { + notifyForUpdate(true); + }); + + $('#alarmsModal').on('shown.bs.modal', function() { + alarmsUpdateModal(); + }); + + $('#alarmsModal').on('hidden.bs.modal', function() { + document.getElementById('alarms_active').innerHTML = + document.getElementById('alarms_all').innerHTML = + document.getElementById('alarms_log').innerHTML = + 'loading...'; + }); + + $('#deleteRegistryModal').on('hidden.bs.modal', function() { + deleteRegistryGuid = null; + }); + + if(isdemo()) { + if(!urlOptions.nowelcome) { + setTimeout(function() { + $('#welcomeModal').modal(); + }, 1000); + } + + // google analytics when this is used for the home page of the demo sites + // this does not run on user's installations + setTimeout(function() { + (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ + (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), + m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) + })(window,document,'script','https://www.google-analytics.com/analytics.js','ga'); + + ga('create', 'UA-64295674-3', 'auto'); + ga('send', 'pageview'); + }, 2000); + } + else notifyForUpdate(); } function resetDashboardOptions() { - var help = NETDATA.options.current.show_help; + var help = NETDATA.options.current.show_help; - NETDATA.resetOptions(); - if(setTheme('slate')) - location.reload(); + NETDATA.resetOptions(); + if(setTheme('slate')) + netdataReload(); - if(help !== NETDATA.options.current.show_help) - location.reload(); + if(help !== NETDATA.options.current.show_help) + netdataReload(); } downloadAllCharts(); diff --git a/web/netdata-swagger.json b/web/netdata-swagger.json index ba674d505..a514d7929 100644 --- a/web/netdata-swagger.json +++ b/web/netdata-swagger.json @@ -3,7 +3,7 @@ "info": { "title": "NetData API", "description": "Real time data collection and graphs...", - "version": "1.0.0" + "version": "1.2.1_master" }, "host": "netdata.firehol.org", "schemes": [ @@ -226,6 +226,171 @@ } } } + }, + "/badge.svg": { + "get": { + "summary": "Generate a SVG image for a chart (or dimension)", + "description": "Successful responses are SVG images\n", + "parameters": [ + { + "name": "chart", + "in": "query", + "description": "The id of the chart as returned by the /charts call.", + "required": true, + "type": "string", + "format": "as returned by /charts", + "allowEmptyValue": false, + "default": "system.cpu" + }, + { + "name": "dimension", + "in": "query", + "description": "zero, one or more dimension ids, as returned by the /chart call.", + "required": false, + "type": "array", + "items": { + "type": "string", + "collectionFormat": "pipes", + "format": "as returned by /charts" + }, + "allowEmptyValue": false + }, + { + "name": "after", + "in": "query", + "description": "This parameter can either be an absolute timestamp specifying the starting point of the data to be returned, or a relative number of seconds, to the last collected timestamp. Netdata will assume it is a relative number if it is smaller than the duration of the round robin database for this chart. So, if the round robin database is 3600 seconds, any value from -3600 to 3600 will trigger relative arithmetics. Netdata will adapt this parameter to the boundaries of the round robin database.", + "required": true, + "type": "number", + "format": "integer", + "allowEmptyValue": false, + "default": -600 + }, + { + "name": "before", + "in": "query", + "description": "This parameter can either be an absolute timestamp specifying the ending point of the data to be returned, or a relative number of seconds, to the last collected timestamp. Netdata will assume it is a relative number if it is smaller than the duration of the round robin database for this chart. So, if the round robin database is 3600 seconds, any value from -3600 to 3600 will trigger relative arithmetics. Netdata will adapt this parameter to the boundaries of the round robin database.", + "required": false, + "type": "number", + "format": "integer", + "default": 0 + }, + { + "name": "points", + "in": "query", + "description": "The number of points to be returned. If not given, or it is <= 0, or it is bigger than the points stored in the round robin database for this chart for the given duration, all the available collected values for the given duration are returned.", + "required": true, + "type": "number", + "format": "integer", + "allowEmptyValue": false, + "default": 20 + }, + { + "name": "group", + "in": "query", + "description": "The grouping method. If multiple collected values are to be grouped in order to return fewer points, this parameters defines the method of grouping. Two methods are supported, \"max\" and \"average\". \"max\" is actually calculated on the absolute value collected (so it works for both positive and negative dimesions to return the most extreme value in either direction).", + "required": true, + "type": "string", + "enum": [ + "max", + "average" + ], + "default": "average", + "allowEmptyValue": false + }, + { + "name": "options", + "in": "query", + "description": "Options that affect data generation.", + "required": false, + "type": "array", + "items": { + "type": "string", + "enum": [ + "nonzero", + "flip", + "abs", + "absolute", + "absolute-sum", + "null2zero", + "percentage" + ], + "collectionFormat": "pipes" + }, + "default": [ + "absolute" + ], + "allowEmptyValue": true + }, + { + "name": "label", + "in": "query", + "description": "a text to be used as the label", + "required": false, + "type": "string", + "format": "any text", + "allowEmptyValue": true + }, + { + "name": "units", + "in": "query", + "description": "a text to be used as the units", + "required": false, + "type": "string", + "format": "any text", + "allowEmptyValue": true + }, + { + "name": "label_color", + "in": "query", + "description": "a color to be used for the background of the label", + "required": false, + "type": "string", + "format": "any text", + "allowEmptyValue": true + }, + { + "name": "value_color", + "in": "query", + "description": "a color to be used for the background of the label. You can set multiple using a pipe with a condition each, like this: color<value|color>value|color:null The following operators are supported: >, <, >=, <=, =, :null (to check if no value exists).", + "required": false, + "type": "string", + "format": "any text", + "allowEmptyValue": true + }, + { + "name": "multiply", + "in": "query", + "description": "multiply the value with this number for rendering it at the image (integer value required)", + "required": false, + "type": "number", + "format": "integer", + "allowEmptyValue": true + }, + { + "name": "divide", + "in": "query", + "description": "divide the value with this number for rendering it at the image (integer value required)", + "required": false, + "type": "number", + "format": "integer", + "allowEmptyValue": true + } + ], + "responses": { + "200": { + "description": "The call was successful. The response should be an SVG image." + }, + "400": { + "description": "Bad request - the body will include a message stating what is wrong." + }, + "404": { + "description": "No chart with the given id is found." + }, + "500": { + "description": "Internal server error. This usually means the server is out of memory." + } + } + } } }, "definitions": { @@ -316,8 +481,8 @@ "description": "The update frequency of this chart, in seconds. One value every this amount of time is kept in the round robin database." }, "dimensions": { - "description": "An object containing all the chart dimensions available for the chart. This is used as an indexed array. The key of the object the id of the dimension.", "type": "object", + "description": "An object containing all the chart dimensions available for the chart. This is used as an indexed array. The key of the object the id of the dimension.", "properties": { "key": { "$ref": "#/definitions/dimension" diff --git a/web/netdata-swagger.yaml b/web/netdata-swagger.yaml index 7dfef0aa8..c1049b12d 100644 --- a/web/netdata-swagger.yaml +++ b/web/netdata-swagger.yaml @@ -2,7 +2,7 @@ swagger: '2.0' info: title: NetData API description: 'Real time data collection and graphs...' - version: 1.0.0 + version: 1.2.1_master host: netdata.firehol.org schemes: - http @@ -66,7 +66,7 @@ paths: allowEmptyValue: false - name: after in: query - description: 'This parameter can either be an absolute timestamp specifying the starting point of the data to be returned, or a relative number of seconds, to the last collected timestamp. Netdata will assume it is a relative number if it is smaller than the duration of the round robin database for this chart. So, if the round robin database is 3600 seconds, any value from -3600 to 3600 will trigger relative arithmetics. Netdata will adapt this parameter to the boundaries of the round robin database.' + description: 'This parameter can either be an absolute timestamp specifying the starting point of the data to be returned, or a relative number of seconds (relative to parameter: before). Netdata will assume it is a relative number if it is smaller than the duration of the round robin database for this chart. So, if the round robin database is 3600 seconds, any value from -3600 to 3600 will trigger relative arithmetics. Netdata will adapt this parameter to the boundaries of the round robin database.' required: true type: number format: integer @@ -89,10 +89,10 @@ paths: default: 20 - name: group in: query - description: 'The grouping method. If multiple collected values are to be grouped in order to return fewer points, this parameters defines the method of grouping. Two methods are supported, "max" and "average". "max" is actually calculated on the absolute value collected (so it works for both positive and negative dimesions to return the most extreme value in either direction).' + description: 'The grouping method. If multiple collected values are to be grouped in order to return fewer points, this parameters defines the method of grouping. methods supported "min", "max", "average", "sum", "incremental-sum". "max" is actually calculated on the absolute value collected (so it works for both positive and negative dimesions to return the most extreme value in either direction).' required: true type: string - enum: [ 'max', 'average' ] + enum: [ 'min', 'max', 'average', 'sum', 'incremental-sum' ] default: 'average' allowEmptyValue: false - name: format @@ -110,7 +110,7 @@ paths: type: array items: type: string - enum: [ 'nonzero', 'flip', 'jsonwrap', 'min2max', 'seconds', 'milliseconds', 'abs', 'absolute', 'absolute-sum', 'null2zero', 'objectrows', 'google_json', 'percentage' ] + enum: [ 'nonzero', 'flip', 'jsonwrap', 'min2max', 'seconds', 'milliseconds', 'abs', 'absolute', 'absolute-sum', 'null2zero', 'objectrows', 'google_json', 'percentage', 'unaligned' ] collectionFormat: pipes default: [seconds, jsonwrap] allowEmptyValue: false @@ -143,6 +143,122 @@ paths: description: 'No chart with the given id is found.' '500': description: 'Internal server error. This usually means the server is out of memory.' + /badge.svg: + get: + summary: 'Generate a SVG image for a chart (or dimension)' + description: | + Successful responses are SVG images + parameters: + - name: chart + in: query + description: 'The id of the chart as returned by the /charts call.' + required: true + type: string + format: 'as returned by /charts' + allowEmptyValue: false + default: system.cpu + - name: alarm + in: query + description: 'the name of an alarm linked to the chart' + required: false + type: string + format: 'any text' + allowEmptyValue: true + - name: dimension + in: query + description: 'zero, one or more dimension ids, as returned by the /chart call.' + required: false + type: array + items: + type: string + collectionFormat: pipes + format: 'as returned by /charts' + allowEmptyValue: false + - name: after + in: query + description: 'This parameter can either be an absolute timestamp specifying the starting point of the data to be returned, or a relative number of seconds, to the last collected timestamp. Netdata will assume it is a relative number if it is smaller than the duration of the round robin database for this chart. So, if the round robin database is 3600 seconds, any value from -3600 to 3600 will trigger relative arithmetics. Netdata will adapt this parameter to the boundaries of the round robin database.' + required: true + type: number + format: integer + allowEmptyValue: false + default: -600 + - name: before + in: query + description: 'This parameter can either be an absolute timestamp specifying the ending point of the data to be returned, or a relative number of seconds, to the last collected timestamp. Netdata will assume it is a relative number if it is smaller than the duration of the round robin database for this chart. So, if the round robin database is 3600 seconds, any value from -3600 to 3600 will trigger relative arithmetics. Netdata will adapt this parameter to the boundaries of the round robin database.' + required: false + type: number + format: integer + default: 0 + - name: group + in: query + description: 'The grouping method. If multiple collected values are to be grouped in order to return fewer points, this parameters defines the method of grouping. methods are supported "min", "max", "average", "sum", "incremental-sum". "max" is actually calculated on the absolute value collected (so it works for both positive and negative dimesions to return the most extreme value in either direction).' + required: true + type: string + enum: [ 'min', 'max', 'average', 'sum', 'incremental-sum' ] + default: 'average' + allowEmptyValue: false + - name: options + in: query + description: 'Options that affect data generation.' + required: false + type: array + items: + type: string + enum: [ 'abs', 'absolute', 'absolute-sum', 'null2zero', 'percentage', 'unaligned' ] + collectionFormat: pipes + default: ['absolute'] + allowEmptyValue: true + - name: label + in: query + description: 'a text to be used as the label' + required: false + type: string + format: 'any text' + allowEmptyValue: true + - name: units + in: query + description: 'a text to be used as the units' + required: false + type: string + format: 'any text' + allowEmptyValue: true + - name: label_color + in: query + description: 'a color to be used for the background of the label' + required: false + type: string + format: 'any text' + allowEmptyValue: true + - name: value_color + in: query + description: 'a color to be used for the background of the label. You can set multiple using a pipe with a condition each, like this: color<value|color>value|color:null The following operators are supported: >, <, >=, <=, =, :null (to check if no value exists).' + required: false + type: string + format: 'any text' + allowEmptyValue: true + - name: multiply + in: query + description: 'multiply the value with this number for rendering it at the image (integer value required)' + required: false + type: number + format: integer + allowEmptyValue: true + - name: divide + in: query + description: 'divide the value with this number for rendering it at the image (integer value required)' + required: false + type: number + format: integer + allowEmptyValue: true + responses: + '200': + description: 'The call was successful. The response should be an SVG image.' + '400': + description: 'Bad request - the body will include a message stating what is wrong.' + '404': + description: 'No chart with the given id is found.' + '500': + description: 'Internal server error. This usually means the server is out of memory.' definitions: chart_summary: type: object diff --git a/web/registry.html b/web/registry.html new file mode 100644 index 000000000..74c676228 --- /dev/null +++ b/web/registry.html @@ -0,0 +1,203 @@ +<!DOCTYPE html> +<html lang="en"> +<head> + <title>NetData Registry Dashboard</title> + <meta name="application-name" content="netdata"> + + <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> + <meta charset="utf-8"> + <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1"> + <meta name="viewport" content="width=device-width, initial-scale=1"> + <meta name="apple-mobile-web-app-capable" content="yes"> + <meta name="apple-mobile-web-app-status-bar-style" content="black-translucent"> + + <meta property="og:locale" content="en_US" /> + <meta property="og:image" content="https://my-netdata.io/images/seo-performance-512.png"/> + <meta property="og:url" content="http://my-netdata.io/"/> + <meta property="og:type" content="website"/> + <meta property="og:site_name" content="netdata"/> + <meta property="og:title" content="netdata - real-time performance monitoring, done right!"/> + <meta property="og:description" content="Stunning real-time dashboards, blazingly fast and extremely interactive. Zero configuration, zero dependencies, zero maintenance." /> + + <script> + // this section has to appear before loading dashboard.js + + // Select a theme. + // uncomment on of the two themes: + + // var netdataTheme = 'default'; // this is white + var netdataTheme = 'slate'; // this is dark + + + // Set the default netdata server. + // on charts without a 'data-host', this one will be used. + // the default is the server that dashboard.js is downloaded from. + + // var netdataServer = 'http://my.server:19999/'; + + function registryGotoServer(guid) { + console.log('goto server: ' + guid); + } + + function registryAddServer(u) { + return '<div id="registry_server_' + u.guid + '" class="registry-server-container" onClick="registryGotoServer(\'' + u.guid + '\'); return false;">' + + '<div class="registry-server-name">' + u.name + '</div>' + + '<div data-netdata="system.cpu"' + + ' data-host="' + u.url + '"' + + ' data-chart-library="sparkline"' + + ' data-sparkline-chartrangemin="0"' + + ' data-sparkline-chartrangemax="100"' + + ' data-sparkline-chartrangeclip="true"' + + ' data-sparkline-disabletooltips="true"' + + ' data-sparkline-disableinteraction="true"' + + ' data-sparkline-disablehighlight="true"' + + ' data-sparkline-linecolor="#444"' + + ' data-sparkline-spotcolor="disable"' + + ' data-sparkline-minspotcolor="disable"' + + ' data-sparkline-maxspotcolor="disable"' + + ' data-width="100%"' + + ' data-height="20px"' + + ' data-after="-200"' + + '></div>' + + '</div>'; + } + + var netdataRegistryCallback = function(machines_array) { + var el = ''; + var a1 = ''; + var found = 0; + + if(machines_array) { + function name_comparator_desc(a, b) { + if (a.name > b.name) return -1; + if (a.name < b.name) return 1; + return 0; + } + + var machines = machines_array.sort(name_comparator_desc); + var len = machines.length; + while(len--) { + var u = machines[len]; + + var status = "enabled"; + found++; + + if(u.guid === NETDATA.registry.machine_guid) + status = "disabled" + + el += registryAddServer(u); + a1 += '<li id="registry_action_' + u.guid + '"><a href="#" onclick="deleteRegistryModalHandler(\'' + u.guid + '\',\'' + u.name + '\',\'' + u.url + '\'); return false;"><i class="fa fa-trash-o" aria-hidden="true" style="color: #999;"></i></a></li>'; + } + } + + if(!found) { + if(machines) + el += '<li><a href="https://github.com/firehol/netdata/wiki/mynetdata-menu-item" style="color: #666;" target="_blank">your netdata server list is empty...</a></li>'; + else + el += '<li><a href="https://github.com/firehol/netdata/wiki/mynetdata-menu-item" style="color: #666;" target="_blank">failed to contact the registry...</a></li>'; + + a1 += '<li><a href="#"> </a></li>'; + + el += '<li role="separator" class="divider"></li>' + + '<li><a href="//london.netdata.rocks/default.html">EU - London (DigitalOcean.com)</a></li>' + + '<li><a href="//atlanta.netdata.rocks/default.html">US - Atlanta (CDN77.com)</a></li>' + + '<li><a href="//athens.netdata.rocks/default.html">EU - Athens</a></li>'; + a1 += '<li role="separator" class="divider"></li>' + + '<li><a href="#"> </a></li>' + + '<li><a href="#"> </a></li>'+ + '<li><a href="#"> </a></li>'; + } + + el += '<li role="separator" class="divider"></li>'; + a1 += '<li role="separator" class="divider"></li>'; + + el += '<li><a href="https://github.com/firehol/netdata/wiki/mynetdata-menu-item" style="color: #999;" target="_blank">What is this?</a></li>'; + a1 += '<li><a href="#" style="color: #999;" onclick="switchRegistryModalHandler(); return false;"><i class="fa fa-cog" aria-hidden="true" style="color: #999;"></i></a></li>' + + document.getElementById('mynetdata_servers').innerHTML = el; + //document.getElementById('mynetdata_servers2').innerHTML = el; + //document.getElementById('mynetdata_actions1').innerHTML = a1; + NETDATA.updatedDom(); + }; + </script> + + <!-- + Load dashboard.js + + to host this HTML file on your web server, + you have to load dashboard.js from the netdata server. + + So, pick one the two below + If you pick the first, set the server name/IP. + + The second assumes you host this file on /usr/share/netdata/web + and that you have chown it to be owned by netdata:netdata + --> + <!-- <script type="text/javascript" src="http://my.server:19999/dashboard.js"></script> --> + <script type="text/javascript" src="dashboard.js?v42"></script> + + <script> + // Set options for TV operation + // This has to be done, after dashboard.js is loaded + + // destroy charts not shown (lowers memory on the browser) + NETDATA.options.current.destroy_on_hide = true; + + // set this to false, to always show all dimensions + NETDATA.options.current.eliminate_zero_dimensions = true; + + // lower the pressure on this browser + NETDATA.options.current.concurrent_refreshes = true; + + // if the tv browser is too slow (a pi?) + // set this to false + NETDATA.options.current.parallel_refresher = true; + + // always update the charts, even if focus is lost + // NETDATA.options.current.stop_updates_when_focus_is_lost = false; + </script> + + <style> + .registry-container { + margin-left: 50px; + max-width: 400px; + min-width: 200px; + border: 1px solid #555; + max-height: 80vh; + height: auto; + overflow-x: hidden; + /* background-color: #3a3f44; */ + text-align: center; + } + .registry-server-container { + width: 100%; + height: auto; + overflow-x: hidden; + font-size: 10px; + padding-left: 10px; + padding-right: 10px; + } + .registry-server-container:hover { + color: #f5f5f5; + background-color: #262626; + } + .registry-server-name { + display: block; + text-decoration: none !important; + clear: both; + font-size: 16px; + font-weight: bold; + white-space: normal; + padding-top: 10px; + } + </style> +</head> +<body> + +<div id="mynetdata_servers" class="registry-container" onscroll="NETDATA.onscroll();"> + + Loading.... + +</div> <!-- registry-container --> +</body> +</html> diff --git a/web/tv.html b/web/tv.html index 58f08eb39..53c47a453 100644 --- a/web/tv.html +++ b/web/tv.html @@ -1,244 +1,253 @@ <!DOCTYPE html> <html lang="en"> <head> - <title>NetData TV Dashboard</title> - - <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> - <meta charset="utf-8"> - <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1"> - <meta name="viewport" content="width=device-width, initial-scale=1"> - <meta name="apple-mobile-web-app-capable" content="yes"> - <meta name="apple-mobile-web-app-status-bar-style" content="black-translucent"> - - <script> - // this section has to appear before loading dashboard.js - - // Select a theme. - // uncomment on of the two themes: - - // var netdataTheme = 'default'; // this is white - var netdataTheme = 'slate'; // this is dark - - - // Set the default netdata server. - // on charts without a 'data-host', this one will be used. - // the default is the server that dashboard.js is downloaded from. - - // var netdataServer = 'http://my.server:19999/'; - </script> - - <!-- - Load dashboard.js - - to host this HTML file on your web server, - you have to load dashboard.js from the netdata server. - - So, pick one the two below - If you pick the first, set the server name/IP. - - The second assumes you host this file on /usr/share/netdata/web - and that you have chown it to be owned by netdata:netdata - --> - <!-- <script type="text/javascript" src="http://my.server:19999/dashboard.js"></script> --> - <script type="text/javascript" src="dashboard.js?v37"></script> - - <script> - // Set options for TV operation - // This has to be done, after dashboard.js is loaded - - // destroy charts not shown (lowers memory on the browser) - NETDATA.options.current.destroy_on_hide = true; - - // set this to false, to always show all dimensions - NETDATA.options.current.eliminate_zero_dimensions = true; - - // lower the pressure on this browser - NETDATA.options.current.concurrent_refreshes = false; - - // if the tv browser is too slow (a pi?) - // set this to false - NETDATA.options.current.parallel_refresher = true; - - // always update the charts, even if focus is lost - // NETDATA.options.current.stop_updates_when_focus_is_lost = false; - - // Since you may render charts from many servers and any of them may - // become offline for some time, the charts will break. - // This will reload the page every RELOAD_EVERY minutes - - var RELOAD_EVERY = 5; - setTimeout(function(){ - location.reload(); - }, RELOAD_EVERY * 60 * 1000); - - </script> + <title>NetData TV Dashboard</title> + <meta name="application-name" content="netdata"> + + <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> + <meta charset="utf-8"> + <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1"> + <meta name="viewport" content="width=device-width, initial-scale=1"> + <meta name="apple-mobile-web-app-capable" content="yes"> + <meta name="apple-mobile-web-app-status-bar-style" content="black-translucent"> + + <meta property="og:locale" content="en_US" /> + <meta property="og:image" content="https://my-netdata.io/images/seo-performance-512.png"/> + <meta property="og:url" content="http://my-netdata.io/"/> + <meta property="og:type" content="website"/> + <meta property="og:site_name" content="netdata"/> + <meta property="og:title" content="netdata - real-time performance monitoring, done right!"/> + <meta property="og:description" content="Stunning real-time dashboards, blazingly fast and extremely interactive. Zero configuration, zero dependencies, zero maintenance." /> + + <script> + // this section has to appear before loading dashboard.js + + // Select a theme. + // uncomment on of the two themes: + + // var netdataTheme = 'default'; // this is white + var netdataTheme = 'slate'; // this is dark + + + // Set the default netdata server. + // on charts without a 'data-host', this one will be used. + // the default is the server that dashboard.js is downloaded from. + + // var netdataServer = 'http://my.server:19999/'; + </script> + + <!-- + Load dashboard.js + + to host this HTML file on your web server, + you have to load dashboard.js from the netdata server. + + So, pick one the two below + If you pick the first, set the server name/IP. + + The second assumes you host this file on /usr/share/netdata/web + and that you have chown it to be owned by netdata:netdata + --> + <!-- <script type="text/javascript" src="http://my.server:19999/dashboard.js"></script> --> + <script type="text/javascript" src="dashboard.js?v42"></script> + + <script> + // Set options for TV operation + // This has to be done, after dashboard.js is loaded + + // destroy charts not shown (lowers memory on the browser) + NETDATA.options.current.destroy_on_hide = true; + + // set this to false, to always show all dimensions + NETDATA.options.current.eliminate_zero_dimensions = true; + + // lower the pressure on this browser + NETDATA.options.current.concurrent_refreshes = false; + + // if the tv browser is too slow (a pi?) + // set this to false + NETDATA.options.current.parallel_refresher = true; + + // always update the charts, even if focus is lost + // NETDATA.options.current.stop_updates_when_focus_is_lost = false; + + // Since you may render charts from many servers and any of them may + // become offline for some time, the charts will break. + // This will reload the page every RELOAD_EVERY minutes + + var RELOAD_EVERY = 5; + setTimeout(function(){ + location.reload(); + }, RELOAD_EVERY * 60 * 1000); + + </script> </head> <body> <div style="width: 100%; text-align: center; display: inline-block;"> - <div style="width: 100%; height: 24vh; text-align: center; display: inline-block;"> - <div style="width: 100%; height: 15px; text-align: center; display: inline-block;"> - <b>CPU On both servers</b> - </div> - <div style="width: 100%; height: calc(100% - 15px); text-align: center; display: inline-block;"> - <br/> - <div data-netdata="system.cpu" - data-host="http://netdata.firehol.org" - data-title="CPU usage of netdata.firehol.org" - data-chart-library="dygraph" - data-width="49%" - data-height="100%" - data-after="-300" - ></div> - <div data-netdata="system.cpu" - data-title="CPU usage of your netdata server" - data-chart-library="dygraph" - data-width="49%" - data-height="100%" - data-after="-300" - ></div> - </div> - </div> - - - <div style="width: 100%; height: 24vh; text-align: center; display: inline-block;"> - <div style="width: 100%; height: 15px; text-align: center; display: inline-block;"> - <b>Disk I/O on both servers</b> - </div> - <div style="width: 100%; height: calc(100% - 15px); text-align: center; display: inline-block;"> - <div data-netdata="system.io" - data-host="http://netdata.firehol.org" - data-title="I/O on netdata.firehol.org" - data-chart-library="dygraph" - data-width="49%" - data-height="100%" - data-after="-300" - ></div> - <div data-netdata="system.io" - data-title="I/O on your netdata server" - data-chart-library="dygraph" - data-width="49%" - data-height="100%" - data-after="-300" - ></div> - </div> - </div> - - - <div style="width: 100%; height: 24vh; text-align: center; display: inline-block;"> - <div style="width: 100%; height: 15px; text-align: center; display: inline-block;"> - <b>IPv4 traffic on both servers</b> - </div> - <div style="width: 100%; height: calc(100% - 15px); text-align: center; display: inline-block;"> - <div data-netdata="system.ipv4" - data-host="http://netdata.firehol.org" - data-title="IPv4 traffic on netdata.firehol.org" - data-chart-library="dygraph" - data-width="49%" - data-height="100%" - data-after="-300" - ></div> - <div data-netdata="system.ipv4" - data-title="IPv4 traffic on your netdata server" - data-chart-library="dygraph" - data-width="49%" - data-height="100%" - data-after="-300" - ></div> - </div> - </div> - - <div style="width: 100%; height: 23vh; text-align: center; display: inline-block;"> - <div style="width: 100%; height: 15px; text-align: center; display: inline-block;"> - <b>Netdata statistics on both servers</b> - </div> - <div style="width: 100%; max-height: calc(100% - 15px); text-align: center; display: inline-block;"> - <div style="width: 49%; height:100%; align: center; display: inline-block;"> - netdata.firehol.org - <br/> - <div data-netdata="netdata.requests" - data-host="http://netdata.firehol.org" - data-title="Chart Refreshes/s" - data-chart-library="gauge" - data-width="20%" - data-height="100%" - data-after="-300" - data-points="300" - ></div> - <div data-netdata="netdata.clients" - data-host="http://netdata.firehol.org" - data-title="Sockets" - data-chart-library="gauge" - data-width="20%" - data-height="100%" - data-after="-300" - data-points="300" - data-colors="#AA5500" - ></div> - <div data-netdata="netdata.net" - data-dimensions="in" - data-host="http://netdata.firehol.org" - data-title="Requests Traffic" - data-chart-library="easypiechart" - data-width="15%" - data-height="100%" - data-after="-300" - data-points="300" - ></div> - <div data-netdata="netdata.net" - data-dimensions="out" - data-host="http://netdata.firehol.org" - data-title="Chart Data Traffic" - data-chart-library="easypiechart" - data-width="15%" - data-height="100%" - data-after="-300" - data-points="300" - ></div> - </div> - <div style="width: 49%; height:100%; align: center; display: inline-block;"> - your netdata server - <br/> - <div data-netdata="netdata.requests" - data-title="Chart Refreshes/s" - data-chart-library="gauge" - data-width="20%" - data-height="100%" - data-after="-300" - data-points="300" - ></div> - <div data-netdata="netdata.clients" - data-title="Sockets" - data-chart-library="gauge" - data-width="20%" - data-height="100%" - data-after="-300" - data-points="300" - data-colors="#AA5500" - ></div> - <div data-netdata="netdata.net" - data-dimensions="in" - data-title="Requests Traffic" - data-chart-library="easypiechart" - data-width="15%" - data-height="100%" - data-after="-300" - data-points="300" - ></div> - <div data-netdata="netdata.net" - data-dimensions="out" - data-title="Chart Data Traffic" - data-chart-library="easypiechart" - data-width="15%" - data-height="100%" - data-after="-300" - data-points="300" - ></div> - </div> - </div> - </div> + <div style="width: 100%; height: 24vh; text-align: center; display: inline-block;"> + <div style="width: 100%; height: 15px; text-align: center; display: inline-block;"> + <b>CPU On both servers</b> + </div> + <div style="width: 100%; height: calc(100% - 15px); text-align: center; display: inline-block;"> + <br/> + <div data-netdata="system.cpu" + data-host="http://netdata.firehol.org" + data-title="CPU usage of netdata.firehol.org" + data-chart-library="dygraph" + data-width="49%" + data-height="100%" + data-after="-300" + ></div> + <div data-netdata="system.cpu" + data-title="CPU usage of your netdata server" + data-chart-library="dygraph" + data-width="49%" + data-height="100%" + data-after="-300" + ></div> + </div> + </div> + + + <div style="width: 100%; height: 24vh; text-align: center; display: inline-block;"> + <div style="width: 100%; height: 15px; text-align: center; display: inline-block;"> + <b>Disk I/O on both servers</b> + </div> + <div style="width: 100%; height: calc(100% - 15px); text-align: center; display: inline-block;"> + <div data-netdata="system.io" + data-host="http://netdata.firehol.org" + data-title="I/O on netdata.firehol.org" + data-chart-library="dygraph" + data-width="49%" + data-height="100%" + data-after="-300" + ></div> + <div data-netdata="system.io" + data-title="I/O on your netdata server" + data-chart-library="dygraph" + data-width="49%" + data-height="100%" + data-after="-300" + ></div> + </div> + </div> + + + <div style="width: 100%; height: 24vh; text-align: center; display: inline-block;"> + <div style="width: 100%; height: 15px; text-align: center; display: inline-block;"> + <b>IPv4 traffic on both servers</b> + </div> + <div style="width: 100%; height: calc(100% - 15px); text-align: center; display: inline-block;"> + <div data-netdata="system.ipv4" + data-host="http://netdata.firehol.org" + data-title="IPv4 traffic on netdata.firehol.org" + data-chart-library="dygraph" + data-width="49%" + data-height="100%" + data-after="-300" + ></div> + <div data-netdata="system.ipv4" + data-title="IPv4 traffic on your netdata server" + data-chart-library="dygraph" + data-width="49%" + data-height="100%" + data-after="-300" + ></div> + </div> + </div> + + <div style="width: 100%; height: 23vh; text-align: center; display: inline-block;"> + <div style="width: 100%; height: 15px; text-align: center; display: inline-block;"> + <b>Netdata statistics on both servers</b> + </div> + <div style="width: 100%; max-height: calc(100% - 15px); text-align: center; display: inline-block;"> + <div style="width: 49%; height:100%; align: center; display: inline-block;"> + netdata.firehol.org + <br/> + <div data-netdata="netdata.requests" + data-host="http://netdata.firehol.org" + data-title="Chart Refreshes/s" + data-chart-library="gauge" + data-width="20%" + data-height="100%" + data-after="-300" + data-points="300" + ></div> + <div data-netdata="netdata.clients" + data-host="http://netdata.firehol.org" + data-title="Sockets" + data-chart-library="gauge" + data-width="20%" + data-height="100%" + data-after="-300" + data-points="300" + data-colors="#AA5500" + ></div> + <div data-netdata="netdata.net" + data-dimensions="in" + data-host="http://netdata.firehol.org" + data-title="Requests Traffic" + data-chart-library="easypiechart" + data-width="15%" + data-height="100%" + data-after="-300" + data-points="300" + ></div> + <div data-netdata="netdata.net" + data-dimensions="out" + data-host="http://netdata.firehol.org" + data-title="Chart Data Traffic" + data-chart-library="easypiechart" + data-width="15%" + data-height="100%" + data-after="-300" + data-points="300" + ></div> + </div> + <div style="width: 49%; height:100%; align: center; display: inline-block;"> + your netdata server + <br/> + <div data-netdata="netdata.requests" + data-title="Chart Refreshes/s" + data-chart-library="gauge" + data-width="20%" + data-height="100%" + data-after="-300" + data-points="300" + ></div> + <div data-netdata="netdata.clients" + data-title="Sockets" + data-chart-library="gauge" + data-width="20%" + data-height="100%" + data-after="-300" + data-points="300" + data-colors="#AA5500" + ></div> + <div data-netdata="netdata.net" + data-dimensions="in" + data-title="Requests Traffic" + data-chart-library="easypiechart" + data-width="15%" + data-height="100%" + data-after="-300" + data-points="300" + ></div> + <div data-netdata="netdata.net" + data-dimensions="out" + data-title="Chart Data Traffic" + data-chart-library="easypiechart" + data-width="15%" + data-height="100%" + data-after="-300" + data-points="300" + ></div> + </div> + </div> + </div> </div> </body> </html> diff --git a/web/version.txt b/web/version.txt index afce0e539..b44550c7a 100644 --- a/web/version.txt +++ b/web/version.txt @@ -1 +1 @@ -bb4aa949f5ac825253d8adc6070661299abc1c3b +b4591e87bd5bf5164eb55c90474bbb9f38f2dad4 |
