diff options
Diffstat (limited to '')
94 files changed, 14365 insertions, 0 deletions
diff --git a/web/api/Makefile.am b/web/api/Makefile.am new file mode 100644 index 0000000..7255ac8 --- /dev/null +++ b/web/api/Makefile.am @@ -0,0 +1,21 @@ +# SPDX-License-Identifier: GPL-3.0-or-later + +AUTOMAKE_OPTIONS = subdir-objects +MAINTAINERCLEANFILES = $(srcdir)/Makefile.in + +SUBDIRS = \ + badges \ + queries \ + exporters \ + formatters \ + health \ + $(NULL) + +dist_noinst_DATA = \ + README.md \ + $(NULL) + +dist_web_DATA = \ + netdata-swagger.yaml \ + netdata-swagger.json \ + $(NULL) diff --git a/web/api/README.md b/web/api/README.md new file mode 100644 index 0000000..1cc3439 --- /dev/null +++ b/web/api/README.md @@ -0,0 +1,14 @@ +<!-- +title: "API" +custom_edit_url: https://github.com/netdata/netdata/edit/master/web/api/README.md +--> + +# API + +## Netdata REST API + +The complete documentation of the Netdata API is available at the **[Swagger Editor](https://editor.swagger.io/?url=https://raw.githubusercontent.com/netdata/netdata/master/web/api/netdata-swagger.yaml)**. + +If your prefer it over the Swagger Editor, you can also use **[Swagger UI](https://registry.my-netdata.io/swagger/#!/default/get_data)**. This however does not provide all the information available. + +[](<>) diff --git a/web/api/badges/Makefile.am b/web/api/badges/Makefile.am new file mode 100644 index 0000000..161784b --- /dev/null +++ b/web/api/badges/Makefile.am @@ -0,0 +1,8 @@ +# SPDX-License-Identifier: GPL-3.0-or-later + +AUTOMAKE_OPTIONS = subdir-objects +MAINTAINERCLEANFILES = $(srcdir)/Makefile.in + +dist_noinst_DATA = \ + README.md \ + $(NULL) diff --git a/web/api/badges/README.md b/web/api/badges/README.md new file mode 100644 index 0000000..b5fc534 --- /dev/null +++ b/web/api/badges/README.md @@ -0,0 +1,363 @@ +<!-- +title: "Netdata badges" +custom_edit_url: https://github.com/netdata/netdata/edit/master/web/api/badges/README.md +--> + +# Netdata badges + +**Badges are cool!** + +Netdata can generate badges for any chart and any dimension at any time-frame. Badges come in `SVG` and can be added to any web page using an `<IMG>` HTML tag. + +**Netdata badges are powerful**! + +Given that Netdata collects from **1.000** to **5.000** metrics per server (depending on the number of network interfaces, disks, cpu cores, applications running, users logged in, containers running, etc) and that Netdata already has data reduction/aggregation functions embedded, the badges can be quite powerful. + +For each metric/dimension and for arbitrary time-frames badges can show **min**, **max** or **average** value, but also **sum** or **incremental-sum** to have their **volume**. + +For example, there is [a chart in Netdata that shows the current requests/s of nginx](http://london.my-netdata.io/#nginx_local_nginx). Using this chart alone we can show the following badges (we could add more time-frames, like **today**, **yesterday**, etc): + +<a href="https://registry.my-netdata.io/#nginx_local_nginx"><img src="https://registry.my-netdata.io/api/v1/badge.svg?chart=nginx_local.connections&dimensions=active&value_color=grey:null%7Cblue&label=nginx%20active%20connections%20now&units=null&precision=0"/></a> <a href="https://registry.my-netdata.io/#nginx_local_nginx"><img src="https://registry.my-netdata.io/api/v1/badge.svg?chart=nginx_local.connections&dimensions=active&after=-3600&value_color=orange&label=last%20hour%20average&units=null&options=unaligned&precision=0"/></a> <a href="https://registry.my-netdata.io/#nginx_local_nginx"><img src="https://registry.my-netdata.io/api/v1/badge.svg?chart=nginx_local.connections&dimensions=active&group=max&after=-3600&value_color=red&label=last%20hour%20max&units=null&options=unaligned&precision=0"/></a> + +Similarly, there is [a chart that shows outbound bandwidth per class](http://london.my-netdata.io/#tc_eth0), using QoS data. So it shows `kilobits/s` per class. Using this chart we can show: + +<a href="https://registry.my-netdata.io/#tc_eth0"><img src="https://registry.my-netdata.io/api/v1/badge.svg?chart=tc.world_out&dimensions=web_server&value_color=green&label=web%20server%20sends%20now&units=kbps"/></a> <a href="https://registry.my-netdata.io/#tc_eth0"><img src="https://registry.my-netdata.io/api/v1/badge.svg?chart=tc.world_out&dimensions=web_server&after=-86400&options=unaligned&group=sum÷=8388608&value_color=blue&label=web%20server%20sent%20today&units=GB"/></a> + +The right one is a **volume** calculation. Netdata calculated the total of the last 86.400 seconds (a day) which gives `kilobits`, then divided it by 8 to make it KB, then by 1024 to make it MB and then by 1024 to make it GB. Calculations like this are quite accurate, since for every value collected, every second, Netdata interpolates it to second boundary using microsecond calculations. + +Let's see a few more badge examples (they come from the [Netdata registry](/registry/README.md)): + +- **cpu usage of user `root`** (you can pick any user; 100% = 1 core). This will be `green <10%`, `yellow <20%`, `orange <50%`, `blue <100%` (1 core), `red` otherwise (you define thresholds and colors on the URL). + + <a href="https://registry.my-netdata.io/#apps_cpu"><img src="https://registry.my-netdata.io/api/v1/badge.svg?chart=users.cpu&dimensions=root&value_color=grey:null%7Cgreen%3C10%7Cyellow%3C20%7Corange%3C50%7Cblue%3C100%7Cred&label=root%20user%20cpu%20now&units=%25"></img></a> <a href="https://registry.my-netdata.io/#apps_cpu"><img src="https://registry.my-netdata.io/api/v1/badge.svg?chart=users.cpu&dimensions=root&after=-3600&value_color=grey:null%7Cgreen%3C10%7Cyellow%3C20%7Corange%3C50%7Cblue%3C100%7Cred&label=root%20user%20average%20cpu%20last%20hour&units=%25"></img></a> + +- **mysql queries per second** + + <a href="https://registry.my-netdata.io/#mysql_local"><img src="https://registry.my-netdata.io/api/v1/badge.svg?chart=mysql_local.queries&dimensions=questions&label=mysql%20queries%20now&value_color=red&units=%5Cs"></img></a> <a href="https://registry.my-netdata.io/#mysql_local"><img src="https://registry.my-netdata.io/api/v1/badge.svg?chart=mysql_local.queries&dimensions=questions&after=-3600&options=unaligned&group=sum&label=mysql%20queries%20this%20hour&value_color=green&units=null"></img></a> <a href="https://registry.my-netdata.io/#mysql_local"><img src="https://registry.my-netdata.io/api/v1/badge.svg?chart=mysql_local.queries&dimensions=questions&after=-86400&options=unaligned&group=sum&label=mysql%20queries%20today&value_color=blue&units=null"></img></a> + + niche ones: **mysql SELECT statements with JOIN, which did full table scans**: + + <a href="https://registry.my-netdata.io/#mysql_local_issues"><img src="https://registry.my-netdata.io/api/v1/badge.svg?chart=mysql_local.join_issues&dimensions=scan&after=-3600&label=full%20table%20scans%20the%20last%20hour&value_color=orange&group=sum&units=null"></img></a> + +--- + +> So, every single line on the charts of a [Netdata dashboard](http://london.my-netdata.io/), can become a badge and this badge can calculate **average**, **min**, **max**, or **volume** for any time-frame! And you can also vary the badge color using conditions on the calculated value. + +--- + +## How to create badges + +The basic URL is `http://your.netdata:19999/api/v1/badge.svg?option1&option2&option3&...`. + +Here is what you can put for `options` (these are standard Netdata API options): + +- `chart=CHART.NAME` + + The chart to get the values from. + + **This is the only parameter required** and with just this parameter, Netdata will return the sum of the latest values of all chart dimensions. + + Example: + +```html + <a href="#"> + <img src="https://registry.my-netdata.io/api/v1/badge.svg?chart=system.cpu"></img> + </a> +``` + + Which produces this: + + <a href="#"> + <img src="https://registry.my-netdata.io/api/v1/badge.svg?chart=system.cpu"></img> + </a> + +- `alarm=NAME` + + Render the current value and status of an alarm linked to the chart. This option can be ignored if the badge to be generated is not related to an alarm. + + The current value of the alarm will be rendered. The color of the badge will indicate the status of the alarm. + + For alarm badges, **both `chart` and `alarm` parameters are required**. + +- `dimensions=DIMENSION1|DIMENSION2|...` + + The dimensions of the chart to use. If you don't set any dimension, all will be used. When multiple dimensions are used, Netdata will sum their values. You can append `options=absolute` if you want this sum to convert all values to positive before adding them. + + Pipes in HTML have to escaped with `%7C`. + + Example: + +```html + <a href="#"> + <img src="https://registry.my-netdata.io/api/v1/badge.svg?chart=system.cpu&dimensions=system%7Cnice"></img> + </a> +``` + + Which produces this: + + <a href="#"> + <img src="https://registry.my-netdata.io/api/v1/badge.svg?chart=system.cpu&dimensions=system%7Cnice"></img> + </a> + +- `before=SECONDS` and `after=SECONDS` + + The timeframe. These can be absolute unix timestamps, or relative to now, number of seconds. By default `before=0` and `after=-1` (1 second in the past). + + To get the last minute set `after=-60`. This will give the average of the last complete minute (XX:XX:00 - XX:XX:59). + + To get the max of the last hour set `after=-3600&group=max`. This will give the maximum value of the last complete hour (XX:00:00 - XX:59:59) + + Example: + +```html + <a href="#"> + <img src="https://registry.my-netdata.io/api/v1/badge.svg?chart=system.cpu&after=-60"></img> + </a> +``` + + Which produces the average of last complete minute (XX:XX:00 - XX:XX:59): + + <a href="#"> + <img src="https://registry.my-netdata.io/api/v1/badge.svg?chart=system.cpu&after=-60"></img> + </a> + + While this is the previous minute (one minute before the last one, again aligned XX:XX:00 - XX:XX:59): + +```html + <a href="#"> + <img src="https://registry.my-netdata.io/api/v1/badge.svg?chart=system.cpu&before=-60&after=-60"></img> + </a> +``` + + It produces this: + + <a href="#"> + <img src="https://registry.my-netdata.io/api/v1/badge.svg?chart=system.cpu&before=-60&after=-60"></img> + </a> + +- `group=min` or `group=max` or `group=average` (the default) or `group=sum` or `group=incremental-sum` + + If Netdata will have to reduce (aggregate) the data to calculate the value, which aggregation method to use. + + - `max` will find the max value for the timeframe. This works on both positive and negative dimensions. It will find the most extreme value. + + - `min` will find the min value for the timeframe. This works on both positive and negative dimensions. It will find the number closest to zero. + + - `average` will calculate the average value for the timeframe. + + - `sum` will sum all the values for the timeframe. This is nice for finding the volume of dimensions for a timeframe. So if you have a dimension that reports `X per second`, you can find the volume of the dimension in a timeframe, by adding its values in that timeframe. + + - `incremental-sum` will sum the difference of each value to its next. Let's assume you have a dimension that does not measure the rate of something, but the absolute value of it. So it has values like this "1, 5, 3, 7, 4". `incremental-sum` will calculate the difference of adjacent values. In this example, they will be `(5 - 1) + (3 - 5) + (7 - 3) + (4 - 7) = 3` (which is equal to the last value minus the first = 4 - 1). + +- `options=opt1|opt2|opt3|...` + + These fine tune various options of the API. Here is what you can use for badges (the API has more option, but only these are useful for badges): + + - `percentage`, instead of returning the value, calculate the percentage of the sum of the selected dimensions, versus the sum of all the dimensions of the chart. This also sets the units to `%`. + + - `absolute` or `abs`, turn all values positive and then sum them. + + - `display_absolute` or `display-absolute`, to use the signed value during color calculation, but display the absolute value on the badge. + + - `min2max`, when multiple dimensions are given, do not sum them, but take their `max - min`. + + - `unaligned`, when data are reduced / aggregated (e.g. the request is about the average of the last minute, or hour), Netdata by default aligns them so that the charts will have a constant shape (so average per minute returns always XX:XX:00 - XX:XX:59). Setting the `unaligned` option, Netdata will aggregate data without any alignment, so if the request is for 60 seconds, it will aggregate the latest 60 seconds of collected data. + +These are options dedicated to badges: + +- `label=TEXT` + + The label of the badge. + +- `units=TEXT` + + The units of the badge. If you want to put a `/`, please put a `\`. This is because Netdata allows badges parameters to be given as path in URL, instead of query string. You can also use `null` or `empty` to show it without any units. + + The units `seconds`, `minutes` and `hours` trigger special formatting. The value has to be in this unit, and Netdata will automatically change it to show a more pretty duration. + +- `multiply=NUMBER` + + Multiply the value with this number. The default is `1`. + +- `divide=NUMBER` + + Divide the value with this number. The default is `1`. + +- Color customization parameters + + The following parameters specify colors of each individual part of the badge. Each parameter is documented in detail + below. + + | Area of badge | Background color parameter | Text color parameter | + | ---: | :------------------------: | :------------------: | + | Label (left) part | `label_color` | `text_color_lbl` | + | Value (right) part | `value_color` | `text_color_val` | + + - `label_color=COLOR` + + The color of the label (the left part). You can use any HTML color in `RGB` or `RRGGBB` hex notation (without + the `#` character at the beginning). Additionally, you can use one of the following predefined colors (and you + can use them by their name): + + - `green` + - `brightgreen` + - `yellow` + - `yellowgreen` + - `orange` + - `red` + - `blue` + - `grey` + - `gray` + - `lightgrey` + - `lightgray` + + These colors are taken from <https://github.com/badges/shields>, which makes them compatible with standard + badges. + + - `value_color=COLOR:null|COLOR<VALUE|COLOR>VALUE|COLOR>=VALUE|COLOR<=VALUE|...` + + You can add a pipe delimited list of conditions to pick the value color. The first matching (left to right) will + be used. + + Example: `value_color=grey:null|green<10|yellow<100|orange<1000|blue<10000|red` + + The above will set `grey` if no value exists (not collected within the `gap when lost iterations above` in + `netdata.conf` for the chart), `green` if the value is less than 10, `yellow` if the value is less than 100, and + so on. Netdata will use `red` if no other conditions match. Only integers are supported as values. + + The supported operators are `<`, `>`, `<=`, `>=`, `=` (or `:`), and `!=` (or `<>`). + + You can also use the same syntax as the `label_color` parameter to define each of these colors. You can + reference a predefined color by name or `RGB`/`RRGGBB` hex notation. + + - `text_color_lbl=RGB` or `text_color_lbl=RRGGBB` or `text_color_lbl=color_by_name` + + This value specifies the font color for the font of left/label side of the badge. The syntax is the same as the + `label_color` parameter. If not given, or given with an empty value, Netdata will use the default color. + + - `text_color_val=RGB` or `text_color_val=RRGGBB` or `text_color_lbl=color_by_name` + + This value specifies the font color for the font of right/value side of the badge. The syntax is the same as the + `label_color` parameter. If not given, or given with an empty value, Netdata will use the default color. + +- `precision=NUMBER` + + The number of decimal digits of the value. By default Netdata will add: + + - no decimal digits for values > 1000 + - 1 decimal digit for values > 100 + - 2 decimal digits for values > 1 + - 3 decimal digits for values > 0.1 + - 4 decimal digits for values \<= 0.1 + + Using the `precision=NUMBER` you can set your preference per badge. + +- `scale=XXX` + + This option scales the svg image. It accepts values above or equal to 100 (100% is the default scale). For example, lets get a few different sizes: + + <img src="https://registry.my-netdata.io/api/v1/badge.svg?chart=system.cpu&after=-60&scale=100"></img> original<br/> + <img src="https://registry.my-netdata.io/api/v1/badge.svg?chart=system.cpu&after=-60&scale=125"></img> `scale=125`<br/> + <img src="https://registry.my-netdata.io/api/v1/badge.svg?chart=system.cpu&after=-60&scale=150"></img> `scale=150`<br/> + <img src="https://registry.my-netdata.io/api/v1/badge.svg?chart=system.cpu&after=-60&scale=175"></img> `scale=175`<br/> + <img src="https://registry.my-netdata.io/api/v1/badge.svg?chart=system.cpu&after=-60&scale=200"></img> `scale=200` + +- `fixed_width_lbl=NUMBER` and `fixed_width_val=NUMBER` + + This parameter overrides auto-sizing of badges and displays them at fixed widths. `fixed_width_lbl` determines the size of the label's left side (label/name). `fixed_width_val` determines the size of the the label's right side (value). You must set both parameters together, or they will be ignored. + + You should set the label/value widths wide enough to provide space for all the possible values/contents of the badge you're requesting. In case the text cannot fit the space given it will be clipped. + + The `scale` parameter still applies on the values you give to `fixed_width_lbl` and `fixed_width_val`. + +- `refresh=auto` or `refresh=SECONDS` + + This option enables auto-refreshing of images. Netdata will send the HTTP header `Refresh: SECONDS` to the web browser, thus requesting automatic refresh of the images at regular intervals. + + `auto` will calculate the proper `SECONDS` to avoid unnecessary refreshes. If `SECONDS` is zero, this feature is disabled (it is also disabled by default). + + Auto-refreshing like this, works only if you access the badge directly. So, you may have to put it an `embed` or `iframe` for it to be auto-refreshed. Use something like this: + +```html +<embed src="BADGE_URL" type="image/svg+xml" height="20" /> +``` + + Another way is to use javascript to auto-refresh them. You can auto-refresh all the Netdata badges on a page using javascript. You have to add a class to all the Netdata badges, like this `<img class="netdata-badge" src="..."/>`. Then add this javascript code to your page (it requires jquery): + +```html +<script> + var NETDATA_BADGES_AUTOREFRESH_SECONDS = 5; + function refreshNetdataBadges() { + var now = new Date().getTime().toString(); + $('.netdata-badge').each(function() { + this.src = this.src.replace(/\&_=\d*/, '') + '&_=' + now; + }); + setTimeout(refreshNetdataBadges, NETDATA_BADGES_AUTOREFRESH_SECONDS * 1000); + } + setTimeout(refreshNetdataBadges, NETDATA_BADGES_AUTOREFRESH_SECONDS * 1000); +</script> +``` + +A more advanced badges refresh method is to include `http://your.netdata.ip:19999/refresh-badges.js` in your page. + +--- + +## Escaping URLs + +Keep in mind that if you add badge URLs to your HTML pages you have to escape the special characters: + +|character|name|escape sequence| +|:-------:|:--:|:-------------:| +|``|space (in labels and units)|`%20`| +|`#`|hash (for colors)|`%23`| +|`%`|percent (in units)|`%25`| +|`<`|less than|`%3C`| +|`>`|greater than|`%3E`| +|`\`|backslash (when you need a `/`)|`%5C`| +|`\|`|pipe (delimiting parameters)|`%7C`| + +## FAQ + +#### Is it fast? + +On modern hardware, Netdata can generate about **2.000 badges per second per core**, before noticing any delays. It generates a badge in about half a millisecond! + +Of course these timing are for badges that use recent data. If you need badges that do calculations over long durations (a day, or more), timing will differ. Netdata logs its timings at its `access.log`, so take a look there before adding a heavy badge on a busy web site. Of course, you can cache such badges or have a cron job get them from Netdata and save them at your web server at regular intervals. + +#### Embedding badges in github + +You have 2 options a) SVG images with markdown and b) SVG images with HTML (directly in .md files). + +For example, this is the cpu badge shown above: + +- Markdown example: + +```md +[](https://registry.my-netdata.io/#apps_cpu) +``` + +- HTML example: + +```html +<a href="https://registry.my-netdata.io/#apps_cpu"> + <img src="https://registry.my-netdata.io/api/v1/badge.svg?chart=users.cpu&dimensions=root&value_color=grey:null%7Cgreen%3C10%7Cyellow%3C20%7Corange%3C50%7Cblue%3C100%7Cred&label=root%20user%20cpu%20now&units=%25"></img> +</a> +``` + +Both produce this: + +<a href="https://registry.my-netdata.io/#apps_cpu"> + <img src="https://registry.my-netdata.io/api/v1/badge.svg?chart=users.cpu&dimensions=root&value_color=grey:null%7Cgreen%3C10%7Cyellow%3C20%7Corange%3C50%7Cblue%3C100%7Cred&label=root%20user%20cpu%20now&units=%25"></img> +</a> + +#### auto-refreshing badges in github + +Unfortunately it cannot be done. Github fetches all the images using a proxy and rewrites all the URLs to be served by the proxy. + +You can refresh them from your browser console though. Press F12 to open the web browser console (switch to the console too), paste the following and press enter. They will refresh: + +```js +var len = document.images.length; while(len--) { document.images[len].src = document.images[len].src.replace(/\?cacheBuster=\d*/, "") + "?cacheBuster=" + new Date().getTime().toString(); }; +``` + +[](<>) diff --git a/web/api/badges/web_buffer_svg.c b/web/api/badges/web_buffer_svg.c new file mode 100644 index 0000000..b5a1e03 --- /dev/null +++ b/web/api/badges/web_buffer_svg.c @@ -0,0 +1,1140 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#include "web_buffer_svg.h" + +#define BADGE_HORIZONTAL_PADDING 4 +#define VERDANA_KERNING 0.2 +#define VERDANA_PADDING 1.0 + +/* + * verdana11_widths[] has been generated with this method: + * https://github.com/badges/shields/blob/master/measure-text.js +*/ + +static double verdana11_widths[128] = { + [0] = 0.0, + [1] = 0.0, + [2] = 0.0, + [3] = 0.0, + [4] = 0.0, + [5] = 0.0, + [6] = 0.0, + [7] = 0.0, + [8] = 0.0, + [9] = 0.0, + [10] = 0.0, + [11] = 0.0, + [12] = 0.0, + [13] = 0.0, + [14] = 0.0, + [15] = 0.0, + [16] = 0.0, + [17] = 0.0, + [18] = 0.0, + [19] = 0.0, + [20] = 0.0, + [21] = 0.0, + [22] = 0.0, + [23] = 0.0, + [24] = 0.0, + [25] = 0.0, + [26] = 0.0, + [27] = 0.0, + [28] = 0.0, + [29] = 0.0, + [30] = 0.0, + [31] = 0.0, + [32] = 3.8671874999999996, // + [33] = 4.3291015625, // ! + [34] = 5.048828125, // " + [35] = 9.001953125, // # + [36] = 6.9931640625, // $ + [37] = 11.837890625, // % + [38] = 7.992187499999999, // & + [39] = 2.9541015625, // ' + [40] = 4.9951171875, // ( + [41] = 4.9951171875, // ) + [42] = 6.9931640625, // * + [43] = 9.001953125, // + + [44] = 4.00146484375, // , + [45] = 4.9951171875, // - + [46] = 4.00146484375, // . + [47] = 4.9951171875, // / + [48] = 6.9931640625, // 0 + [49] = 6.9931640625, // 1 + [50] = 6.9931640625, // 2 + [51] = 6.9931640625, // 3 + [52] = 6.9931640625, // 4 + [53] = 6.9931640625, // 5 + [54] = 6.9931640625, // 6 + [55] = 6.9931640625, // 7 + [56] = 6.9931640625, // 8 + [57] = 6.9931640625, // 9 + [58] = 4.9951171875, // : + [59] = 4.9951171875, // ; + [60] = 9.001953125, // < + [61] = 9.001953125, // = + [62] = 9.001953125, // > + [63] = 5.99951171875, // ? + [64] = 11.0, // @ + [65] = 7.51953125, // A + [66] = 7.541015625, // B + [67] = 7.680664062499999, // C + [68] = 8.4755859375, // D + [69] = 6.95556640625, // E + [70] = 6.32177734375, // F + [71] = 8.529296875, // G + [72] = 8.26611328125, // H + [73] = 4.6298828125, // I + [74] = 5.00048828125, // J + [75] = 7.62158203125, // K + [76] = 6.123046875, // L + [77] = 9.2705078125, // M + [78] = 8.228515625, // N + [79] = 8.658203125, // O + [80] = 6.63330078125, // P + [81] = 8.658203125, // Q + [82] = 7.6484375, // R + [83] = 7.51953125, // S + [84] = 6.7783203125, // T + [85] = 8.05126953125, // U + [86] = 7.51953125, // V + [87] = 10.87646484375, // W + [88] = 7.53564453125, // X + [89] = 6.767578125, // Y + [90] = 7.53564453125, // Z + [91] = 4.9951171875, // [ + [92] = 4.9951171875, // backslash + [93] = 4.9951171875, // ] + [94] = 9.001953125, // ^ + [95] = 6.9931640625, // _ + [96] = 6.9931640625, // ` + [97] = 6.6064453125, // a + [98] = 6.853515625, // b + [99] = 5.73095703125, // c + [100] = 6.853515625, // d + [101] = 6.552734375, // e + [102] = 3.8671874999999996, // f + [103] = 6.853515625, // g + [104] = 6.9609375, // h + [105] = 3.0185546875, // i + [106] = 3.78662109375, // j + [107] = 6.509765625, // k + [108] = 3.0185546875, // l + [109] = 10.69921875, // m + [110] = 6.9609375, // n + [111] = 6.67626953125, // o + [112] = 6.853515625, // p + [113] = 6.853515625, // q + [114] = 4.6943359375, // r + [115] = 5.73095703125, // s + [116] = 4.33447265625, // t + [117] = 6.9609375, // u + [118] = 6.509765625, // v + [119] = 9.001953125, // w + [120] = 6.509765625, // x + [121] = 6.509765625, // y + [122] = 5.779296875, // z + [123] = 6.982421875, // { + [124] = 4.9951171875, // | + [125] = 6.982421875, // } + [126] = 9.001953125, // ~ + [127] = 0.0 +}; + +// find the width of the string using the verdana 11points font +static inline double verdana11_width(const char *s, float em_size) { + double w = 0.0; + + while(*s) { + // if UTF8 multibyte char found and guess it's width equal 1em + // as label width will be updated with JavaScript this is not so important + + // TODO: maybe move UTF8 functions from url.c to separate util in libnetdata + // then use url_utf8_get_byte_length etc. + if(IS_UTF8_STARTBYTE(*s)) { + s++; + while(IS_UTF8_BYTE(*s) && !IS_UTF8_STARTBYTE(*s)){ + s++; + } + w += em_size; + } + else { + if(likely(!(*s & 0x80))){ // Byte 1XXX XXXX is not valid in UTF8 + double t = verdana11_widths[(unsigned char)*s]; + if(t != 0.0) + w += t + VERDANA_KERNING; + } + s++; + } + } + + w -= VERDANA_KERNING; + w += VERDANA_PADDING; + return w; +} + +static inline size_t escape_xmlz(char *dst, const char *src, size_t len) { + size_t i = len; + + // required escapes from + // https://github.com/badges/shields/blob/master/badge.js + while(*src && i) { + switch(*src) { + case '\\': + *dst++ = '/'; + src++; + i--; + break; + + case '&': + if(i > 5) { + strcpy(dst, "&"); + i -= 5; + dst += 5; + src++; + } + else goto cleanup; + break; + + case '<': + if(i > 4) { + strcpy(dst, "<"); + i -= 4; + dst += 4; + src++; + } + else goto cleanup; + break; + + case '>': + if(i > 4) { + strcpy(dst, ">"); + i -= 4; + dst += 4; + src++; + } + else goto cleanup; + break; + + case '"': + if(i > 6) { + strcpy(dst, """); + i -= 6; + dst += 6; + src++; + } + else goto cleanup; + break; + + case '\'': + if(i > 6) { + strcpy(dst, "'"); + i -= 6; + dst += 6; + src++; + } + else goto cleanup; + break; + + default: + i--; + *dst++ = *src++; + break; + } + } + +cleanup: + *dst = '\0'; + return len - i; +} + +static inline char *format_value_with_precision_and_unit(char *value_string, size_t value_string_len, calculated_number value, const char *units, int precision) { + if(unlikely(isnan(value) || isinf(value))) + value = 0.0; + + char *separator = ""; + if(unlikely(isalnum(*units))) + separator = " "; + + if(precision < 0) { + int len, lstop = 0, trim_zeros = 1; + + calculated_number abs = value; + if(isless(value, 0)) { + lstop = 1; + abs = calculated_number_fabs(value); + } + + if(isgreaterequal(abs, 1000)) { + len = snprintfz(value_string, value_string_len, "%0.0" LONG_DOUBLE_MODIFIER, (LONG_DOUBLE) value); + trim_zeros = 0; + } + else if(isgreaterequal(abs, 10)) len = snprintfz(value_string, value_string_len, "%0.1" LONG_DOUBLE_MODIFIER, (LONG_DOUBLE) value); + else if(isgreaterequal(abs, 1)) len = snprintfz(value_string, value_string_len, "%0.2" LONG_DOUBLE_MODIFIER, (LONG_DOUBLE) value); + else if(isgreaterequal(abs, 0.1)) len = snprintfz(value_string, value_string_len, "%0.2" LONG_DOUBLE_MODIFIER, (LONG_DOUBLE) value); + else if(isgreaterequal(abs, 0.01)) len = snprintfz(value_string, value_string_len, "%0.4" LONG_DOUBLE_MODIFIER, (LONG_DOUBLE) value); + else if(isgreaterequal(abs, 0.001)) len = snprintfz(value_string, value_string_len, "%0.5" LONG_DOUBLE_MODIFIER, (LONG_DOUBLE) value); + else if(isgreaterequal(abs, 0.0001)) len = snprintfz(value_string, value_string_len, "%0.6" LONG_DOUBLE_MODIFIER, (LONG_DOUBLE) value); + else len = snprintfz(value_string, value_string_len, "%0.7" LONG_DOUBLE_MODIFIER, (LONG_DOUBLE) value); + + if(unlikely(trim_zeros)) { + int l; + // remove trailing zeros from the decimal part + for(l = len - 1; l > lstop; l--) { + if(likely(value_string[l] == '0')) { + value_string[l] = '\0'; + len--; + } + + else if(unlikely(value_string[l] == '.')) { + value_string[l] = '\0'; + len--; + break; + } + + else + break; + } + } + + if(unlikely(len <= 0)) len = 1; + snprintfz(&value_string[len], value_string_len - len, "%s%s", separator, units); + } + else { + if(precision > 50) precision = 50; + snprintfz(value_string, value_string_len, "%0.*" LONG_DOUBLE_MODIFIER "%s%s", precision, (LONG_DOUBLE) value, separator, units); + } + + return value_string; +} + +typedef enum badge_units_format { + UNITS_FORMAT_NONE, + UNITS_FORMAT_SECONDS, + UNITS_FORMAT_SECONDS_AGO, + UNITS_FORMAT_MINUTES, + UNITS_FORMAT_MINUTES_AGO, + UNITS_FORMAT_HOURS, + UNITS_FORMAT_HOURS_AGO, + UNITS_FORMAT_ONOFF, + UNITS_FORMAT_UPDOWN, + UNITS_FORMAT_OKERROR, + UNITS_FORMAT_OKFAILED, + UNITS_FORMAT_EMPTY, + UNITS_FORMAT_PERCENT +} UNITS_FORMAT; + + +static struct units_formatter { + const char *units; + uint32_t hash; + UNITS_FORMAT format; +} badge_units_formatters[] = { + { "seconds", 0, UNITS_FORMAT_SECONDS }, + { "seconds ago", 0, UNITS_FORMAT_SECONDS_AGO }, + { "minutes", 0, UNITS_FORMAT_MINUTES }, + { "minutes ago", 0, UNITS_FORMAT_MINUTES_AGO }, + { "hours", 0, UNITS_FORMAT_HOURS }, + { "hours ago", 0, UNITS_FORMAT_HOURS_AGO }, + { "on/off", 0, UNITS_FORMAT_ONOFF }, + { "on-off", 0, UNITS_FORMAT_ONOFF }, + { "onoff", 0, UNITS_FORMAT_ONOFF }, + { "up/down", 0, UNITS_FORMAT_UPDOWN }, + { "up-down", 0, UNITS_FORMAT_UPDOWN }, + { "updown", 0, UNITS_FORMAT_UPDOWN }, + { "ok/error", 0, UNITS_FORMAT_OKERROR }, + { "ok-error", 0, UNITS_FORMAT_OKERROR }, + { "okerror", 0, UNITS_FORMAT_OKERROR }, + { "ok/failed", 0, UNITS_FORMAT_OKFAILED }, + { "ok-failed", 0, UNITS_FORMAT_OKFAILED }, + { "okfailed", 0, UNITS_FORMAT_OKFAILED }, + { "empty", 0, UNITS_FORMAT_EMPTY }, + { "null", 0, UNITS_FORMAT_EMPTY }, + { "percentage", 0, UNITS_FORMAT_PERCENT }, + { "percent", 0, UNITS_FORMAT_PERCENT }, + { "pcent", 0, UNITS_FORMAT_PERCENT }, + + // terminator + { NULL, 0, UNITS_FORMAT_NONE } +}; + +inline char *format_value_and_unit(char *value_string, size_t value_string_len, calculated_number value, const char *units, int precision) { + static int max = -1; + int i; + + if(unlikely(max == -1)) { + for(i = 0; badge_units_formatters[i].units; i++) + badge_units_formatters[i].hash = simple_hash(badge_units_formatters[i].units); + + max = i; + } + + if(unlikely(!units)) units = ""; + uint32_t hash_units = simple_hash(units); + + UNITS_FORMAT format = UNITS_FORMAT_NONE; + for(i = 0; i < max; i++) { + struct units_formatter *ptr = &badge_units_formatters[i]; + + if(hash_units == ptr->hash && !strcmp(units, ptr->units)) { + format = ptr->format; + break; + } + } + + if(unlikely(format == UNITS_FORMAT_SECONDS || format == UNITS_FORMAT_SECONDS_AGO)) { + if(value == 0.0) { + snprintfz(value_string, value_string_len, "%s", "now"); + return value_string; + } + else if(isnan(value) || isinf(value)) { + snprintfz(value_string, value_string_len, "%s", "undefined"); + return value_string; + } + + const char *suffix = (format == UNITS_FORMAT_SECONDS_AGO)?" ago":""; + + size_t s = (size_t)value; + size_t d = s / 86400; + s = s % 86400; + + size_t h = s / 3600; + s = s % 3600; + + size_t m = s / 60; + s = s % 60; + + if(d) + snprintfz(value_string, value_string_len, "%zu %s %02zu:%02zu:%02zu%s", d, (d == 1)?"day":"days", h, m, s, suffix); + else + snprintfz(value_string, value_string_len, "%02zu:%02zu:%02zu%s", h, m, s, suffix); + + return value_string; + } + + else if(unlikely(format == UNITS_FORMAT_MINUTES || format == UNITS_FORMAT_MINUTES_AGO)) { + if(value == 0.0) { + snprintfz(value_string, value_string_len, "%s", "now"); + return value_string; + } + else if(isnan(value) || isinf(value)) { + snprintfz(value_string, value_string_len, "%s", "undefined"); + return value_string; + } + + const char *suffix = (format == UNITS_FORMAT_MINUTES_AGO)?" ago":""; + + size_t m = (size_t)value; + size_t d = m / (60 * 24); + m = m % (60 * 24); + + size_t h = m / 60; + m = m % 60; + + if(d) + snprintfz(value_string, value_string_len, "%zud %02zuh %02zum%s", d, h, m, suffix); + else + snprintfz(value_string, value_string_len, "%zuh %zum%s", h, m, suffix); + + return value_string; + } + + else if(unlikely(format == UNITS_FORMAT_HOURS || format == UNITS_FORMAT_HOURS_AGO)) { + if(value == 0.0) { + snprintfz(value_string, value_string_len, "%s", "now"); + return value_string; + } + else if(isnan(value) || isinf(value)) { + snprintfz(value_string, value_string_len, "%s", "undefined"); + return value_string; + } + + const char *suffix = (format == UNITS_FORMAT_HOURS_AGO)?" ago":""; + + size_t h = (size_t)value; + size_t d = h / 24; + h = h % 24; + + if(d) + snprintfz(value_string, value_string_len, "%zud %zuh%s", d, h, suffix); + else + snprintfz(value_string, value_string_len, "%zuh%s", h, suffix); + + return value_string; + } + + else if(unlikely(format == UNITS_FORMAT_ONOFF)) { + snprintfz(value_string, value_string_len, "%s", (value != 0.0)?"on":"off"); + return value_string; + } + + else if(unlikely(format == UNITS_FORMAT_UPDOWN)) { + snprintfz(value_string, value_string_len, "%s", (value != 0.0)?"up":"down"); + return value_string; + } + + else if(unlikely(format == UNITS_FORMAT_OKERROR)) { + snprintfz(value_string, value_string_len, "%s", (value != 0.0)?"ok":"error"); + return value_string; + } + + else if(unlikely(format == UNITS_FORMAT_OKFAILED)) { + snprintfz(value_string, value_string_len, "%s", (value != 0.0)?"ok":"failed"); + return value_string; + } + + else if(unlikely(format == UNITS_FORMAT_EMPTY)) + units = ""; + + else if(unlikely(format == UNITS_FORMAT_PERCENT)) + units = "%"; + + if(unlikely(isnan(value) || isinf(value))) { + strcpy(value_string, "-"); + return value_string; + } + + return format_value_with_precision_and_unit(value_string, value_string_len, value, units, precision); +} + +static struct badge_color { + const char *name; + uint32_t hash; + const char *color; +} badge_colors[] = { + + // colors from: + // https://github.com/badges/shields/blob/master/colorscheme.json + + { "brightgreen", 0, "4c1" }, + { "green", 0, "97CA00" }, + { "yellow", 0, "dfb317" }, + { "yellowgreen", 0, "a4a61d" }, + { "orange", 0, "fe7d37" }, + { "red", 0, "e05d44" }, + { "blue", 0, "007ec6" }, + { "grey", 0, "555" }, + { "gray", 0, "555" }, + { "lightgrey", 0, "9f9f9f" }, + { "lightgray", 0, "9f9f9f" }, + + // terminator + { NULL, 0, NULL } +}; + +static inline const char *color_map(const char *color, const char *def) { + static int max = -1; + int i; + + if(unlikely(max == -1)) { + for(i = 0; badge_colors[i].name ;i++) + badge_colors[i].hash = simple_hash(badge_colors[i].name); + + max = i; + } + + uint32_t hash = simple_hash(color); + + for(i = 0; i < max; i++) { + struct badge_color *ptr = &badge_colors[i]; + + if(hash == ptr->hash && !strcmp(color, ptr->name)) + return ptr->color; + } + + return def; +} + +typedef enum color_comparison { + COLOR_COMPARE_EQUAL, + COLOR_COMPARE_NOTEQUAL, + COLOR_COMPARE_LESS, + COLOR_COMPARE_LESSEQUAL, + COLOR_COMPARE_GREATER, + COLOR_COMPARE_GREATEREQUAL, +} BADGE_COLOR_COMPARISON; + +static inline void calc_colorz(const char *color, char *final, size_t len, calculated_number value) { + if(isnan(value) || isinf(value)) + value = NAN; + + char color_buffer[256 + 1] = ""; + char value_buffer[256 + 1] = ""; + BADGE_COLOR_COMPARISON comparison = COLOR_COMPARE_GREATER; + + // example input: + // color<max|color>min|color:null... + + const char *c = color; + while(*c) { + char *dc = color_buffer, *dv = NULL; + size_t ci = 0, vi = 0; + + const char *t = c; + + while(*t && *t != '|') { + switch(*t) { + case '!': + if(t[1] == '=') t++; + comparison = COLOR_COMPARE_NOTEQUAL; + dv = value_buffer; + break; + + case '=': + case ':': + comparison = COLOR_COMPARE_EQUAL; + dv = value_buffer; + break; + + case '}': + case ')': + case '>': + if(t[1] == '=') { + comparison = COLOR_COMPARE_GREATEREQUAL; + t++; + } + else + comparison = COLOR_COMPARE_GREATER; + dv = value_buffer; + break; + + case '{': + case '(': + case '<': + if(t[1] == '=') { + comparison = COLOR_COMPARE_LESSEQUAL; + t++; + } + else if(t[1] == '>' || t[1] == ')' || t[1] == '}') { + comparison = COLOR_COMPARE_NOTEQUAL; + t++; + } + else + comparison = COLOR_COMPARE_LESS; + dv = value_buffer; + break; + + default: + if(dv) { + if(vi < 256) { + vi++; + *dv++ = *t; + } + } + else { + if(ci < 256) { + ci++; + *dc++ = *t; + } + } + break; + } + + t++; + } + + // prepare for next iteration + if(*t == '|') t++; + c = t; + + // do the math + *dc = '\0'; + if(dv) { + *dv = '\0'; + calculated_number v; + + if(!*value_buffer || !strcmp(value_buffer, "null")) { + v = NAN; + } + else { + v = str2l(value_buffer); + if(isnan(v) || isinf(v)) + v = NAN; + } + + if(unlikely(isnan(value) || isnan(v))) { + if(isnan(value) && isnan(v)) + break; + } + else { + if (unlikely(comparison == COLOR_COMPARE_LESS && isless(value, v))) break; + else if (unlikely(comparison == COLOR_COMPARE_LESSEQUAL && islessequal(value, v))) break; + else if (unlikely(comparison == COLOR_COMPARE_GREATER && isgreater(value, v))) break; + else if (unlikely(comparison == COLOR_COMPARE_GREATEREQUAL && isgreaterequal(value, v))) break; + else if (unlikely(comparison == COLOR_COMPARE_EQUAL && !islessgreater(value, v))) break; + else if (unlikely(comparison == COLOR_COMPARE_NOTEQUAL && islessgreater(value, v))) break; + } + } + else + break; + } + + const char *b; + if(color_buffer[0]) + b = color_buffer; + else + b = color; + + strncpyz(final, b, len); +} + +// value + units +#define VALUE_STRING_SIZE 100 + +// label +#define LABEL_STRING_SIZE 200 + +// colors +#define COLOR_STRING_SIZE 100 + +static inline int allowed_hexa_char(char x) { + return ( (x >= '0' && x <= '9') || + (x >= 'a' && x <= 'f') || + (x >= 'A' && x <= 'F') + ); +} + +static int html_color_check(const char *str) { + int i = 0; + while(str[i]) { + if(!allowed_hexa_char(str[i])) + return 0; + if(unlikely(i >= 6)) + return 0; + i++; + } + // want to allow either RGB or RRGGBB + return ( i == 6 || i == 3 ); +} + +// Will parse color arg as #RRGGBB or #RGB or one of the colors +// from color_map hash table +// if parsing fails (argument error) it will return default color +// given as default parameter (def) +// in any case it will return either color in "RRGGBB" or "RGB" format as string +// or whatever is given as def (without checking - caller responsible to give sensible +// safely escaped default) as default if it fails +// in any case this function must always return something we can put directly in XML +// so no escaping is necessary anymore (with excpetion of default where caller is responsible) +// to give sensible default +#define BADGE_SVG_COLOR_ARG_MAXLEN 20 + +static const char *parse_color_argument(const char *arg, const char *def) +{ + if( !arg ) + return def; + size_t len = strnlen(arg, BADGE_SVG_COLOR_ARG_MAXLEN); + if( len < 2 || len >= BADGE_SVG_COLOR_ARG_MAXLEN ) + return def; + if( html_color_check(arg) ) + return arg; + return color_map(arg, def); +} + +void buffer_svg(BUFFER *wb, const char *label, calculated_number value, const char *units, const char *label_color, const char *value_color, int precision, int scale, uint32_t options, int fixed_width_lbl, int fixed_width_val, const char* text_color_lbl, const char* text_color_val) { + char value_color_buffer[COLOR_STRING_SIZE + 1] + , value_string[VALUE_STRING_SIZE + 1] + , label_escaped[LABEL_STRING_SIZE + 1] + , value_escaped[VALUE_STRING_SIZE + 1]; + + const char *label_color_parsed; + const char *value_color_parsed; + + double label_width = (double)fixed_width_lbl, value_width = (double)fixed_width_val, total_width; + double height = 20.0, font_size = 11.0, text_offset = 5.8, round_corner = 3.0; + + if(scale < 100) scale = 100; + + if(unlikely(!value_color || !*value_color)) + value_color = (isnan(value) || isinf(value))?"999":"4c1"; + + calc_colorz(value_color, value_color_buffer, COLOR_STRING_SIZE, value); + format_value_and_unit(value_string, VALUE_STRING_SIZE, (options & RRDR_OPTION_DISPLAY_ABS)?calculated_number_fabs(value):value, units, precision); + + if(fixed_width_lbl <= 0 || fixed_width_val <= 0) { + label_width = verdana11_width(label, font_size) + (BADGE_HORIZONTAL_PADDING * 2); + value_width = verdana11_width(value_string, font_size) + (BADGE_HORIZONTAL_PADDING * 2); + } + total_width = label_width + value_width; + + escape_xmlz(label_escaped, label, LABEL_STRING_SIZE); + escape_xmlz(value_escaped, value_string, VALUE_STRING_SIZE); + + label_color_parsed = parse_color_argument(label_color, "555"); + value_color_parsed = parse_color_argument(value_color_buffer, "555"); + + wb->contenttype = CT_IMAGE_SVG_XML; + + total_width = total_width * scale / 100.0; + height = height * scale / 100.0; + font_size = font_size * scale / 100.0; + text_offset = text_offset * scale / 100.0; + label_width = label_width * scale / 100.0; + value_width = value_width * scale / 100.0; + round_corner = round_corner * scale / 100.0; + + // svg template from: + // https://raw.githubusercontent.com/badges/shields/master/templates/flat-template.svg + buffer_sprintf(wb, + "<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"%0.2f\" height=\"%0.2f\">" + "<linearGradient id=\"smooth\" x2=\"0\" y2=\"100%%\">" + "<stop offset=\"0\" stop-color=\"#bbb\" stop-opacity=\".1\"/>" + "<stop offset=\"1\" stop-opacity=\".1\"/>" + "</linearGradient>" + "<mask id=\"round\">" + "<rect class=\"bdge-ttl-width\" width=\"%0.2f\" height=\"%0.2f\" rx=\"%0.2f\" fill=\"#fff\"/>" + "</mask>" + "<g mask=\"url(#round)\">" + "<rect class=\"bdge-rect-lbl\" width=\"%0.2f\" height=\"%0.2f\" fill=\"#%s\"/>", + total_width, height, + total_width, height, round_corner, + label_width, height, label_color_parsed); //<rect class="bdge-rect-lbl" + + if(fixed_width_lbl > 0 && fixed_width_val > 0) { + buffer_sprintf(wb, + "<clipPath id=\"lbl-rect\">" + "<rect class=\"bdge-rect-lbl\" width=\"%0.2f\" height=\"%0.2f\"/>" + "</clipPath>", + label_width, height); //<clipPath id="lbl-rect"> <rect class="bdge-rect-lbl" + } + + buffer_sprintf(wb, + "<rect class=\"bdge-rect-val\" x=\"%0.2f\" width=\"%0.2f\" height=\"%0.2f\" fill=\"#%s\"/>", + label_width, value_width, height, value_color_parsed); + + if(fixed_width_lbl > 0 && fixed_width_val > 0) { + buffer_sprintf(wb, + "<clipPath id=\"val-rect\">" + "<rect class=\"bdge-rect-val\" x=\"%0.2f\" width=\"%0.2f\" height=\"%0.2f\"/>" + "</clipPath>", + label_width, value_width, height); + } + + buffer_sprintf(wb, + "<rect class=\"bdge-ttl-width\" width=\"%0.2f\" height=\"%0.2f\" fill=\"url(#smooth)\"/>" + "</g>" + "<g text-anchor=\"middle\" font-family=\"DejaVu Sans,Verdana,Geneva,sans-serif\" font-size=\"%0.2f\">" + "<text class=\"bdge-lbl-lbl\" x=\"%0.2f\" y=\"%0.0f\" fill=\"#010101\" fill-opacity=\".3\" clip-path=\"url(#lbl-rect)\">%s</text>" + "<text class=\"bdge-lbl-lbl\" x=\"%0.2f\" y=\"%0.0f\" fill=\"#%s\" clip-path=\"url(#lbl-rect)\">%s</text>" + "<text class=\"bdge-lbl-val\" x=\"%0.2f\" y=\"%0.0f\" fill=\"#010101\" fill-opacity=\".3\" clip-path=\"url(#val-rect)\">%s</text>" + "<text class=\"bdge-lbl-val\" x=\"%0.2f\" y=\"%0.0f\" fill=\"#%s\" clip-path=\"url(#val-rect)\">%s</text>" + "</g>", + total_width, height, + font_size, + label_width / 2, ceil(height - text_offset), label_escaped, + label_width / 2, ceil(height - text_offset - 1.0), parse_color_argument(text_color_lbl, "fff"), label_escaped, + label_width + value_width / 2 -1, ceil(height - text_offset), value_escaped, + label_width + value_width / 2 -1, ceil(height - text_offset - 1.0), parse_color_argument(text_color_val, "fff"), value_escaped); + + if(fixed_width_lbl <= 0 || fixed_width_val <= 0){ + buffer_sprintf(wb, + "<script type=\"text/javascript\">" + "var bdg_horiz_padding = %d;" + "function netdata_bdge_each(list, attr, value){" + "Array.prototype.forEach.call(list, function(el){" + "el.setAttribute(attr, value);" + "});" + "};" + "var this_svg = document.currentScript.closest(\"svg\");" + "var elem_lbl = this_svg.getElementsByClassName(\"bdge-lbl-lbl\");" + "var elem_val = this_svg.getElementsByClassName(\"bdge-lbl-val\");" + "var lbl_size = elem_lbl[0].getBBox();" + "var val_size = elem_val[0].getBBox();" + "var width_total = lbl_size.width + bdg_horiz_padding*2;" + "this_svg.getElementsByClassName(\"bdge-rect-lbl\")[0].setAttribute(\"width\", width_total);" + "netdata_bdge_each(elem_lbl, \"x\", (lbl_size.width / 2) + bdg_horiz_padding);" + "netdata_bdge_each(elem_val, \"x\", width_total + (val_size.width / 2) + bdg_horiz_padding);" + "var val_rect = this_svg.getElementsByClassName(\"bdge-rect-val\")[0];" + "val_rect.setAttribute(\"width\", val_size.width + bdg_horiz_padding*2);" + "val_rect.setAttribute(\"x\", width_total);" + "width_total += val_size.width + bdg_horiz_padding*2;" + "var width_update_elems = this_svg.getElementsByClassName(\"bdge-ttl-width\");" + "netdata_bdge_each(width_update_elems, \"width\", width_total);" + "this_svg.setAttribute(\"width\", width_total);" + "</script>", + BADGE_HORIZONTAL_PADDING); + } + buffer_sprintf(wb, "</svg>"); +} + +#define BADGE_URL_ARG_LBL_COLOR "text_color_lbl" +#define BADGE_URL_ARG_VAL_COLOR "text_color_val" + +int web_client_api_request_v1_badge(RRDHOST *host, struct web_client *w, char *url) { + int ret = HTTP_RESP_BAD_REQUEST; + buffer_flush(w->response.data); + + BUFFER *dimensions = NULL; + + const char *chart = NULL + , *before_str = NULL + , *after_str = NULL + , *points_str = NULL + , *multiply_str = NULL + , *divide_str = NULL + , *label = NULL + , *units = NULL + , *label_color = NULL + , *value_color = NULL + , *refresh_str = NULL + , *precision_str = NULL + , *scale_str = NULL + , *alarm = NULL + , *fixed_width_lbl_str = NULL + , *fixed_width_val_str = NULL + , *text_color_lbl_str = NULL + , *text_color_val_str = NULL; + + int group = RRDR_GROUPING_AVERAGE; + uint32_t options = 0x00000000; + + while(url) { + char *value = mystrsep(&url, "&"); + if(!value || !*value) continue; + + char *name = mystrsep(&value, "="); + if(!name || !*name) continue; + if(!value || !*value) continue; + + debug(D_WEB_CLIENT, "%llu: API v1 badge.svg query param '%s' with value '%s'", w->id, name, value); + + // name and value are now the parameters + // they are not null and not empty + + if(!strcmp(name, "chart")) chart = value; + else if(!strcmp(name, "dimension") || !strcmp(name, "dim") || !strcmp(name, "dimensions") || !strcmp(name, "dims")) { + if(!dimensions) + dimensions = buffer_create(100); + + buffer_strcat(dimensions, "|"); + buffer_strcat(dimensions, value); + } + else if(!strcmp(name, "after")) after_str = value; + else if(!strcmp(name, "before")) before_str = value; + else if(!strcmp(name, "points")) points_str = value; + else if(!strcmp(name, "group")) { + group = web_client_api_request_v1_data_group(value, RRDR_GROUPING_AVERAGE); + } + else if(!strcmp(name, "options")) { + options |= web_client_api_request_v1_data_options(value); + } + else if(!strcmp(name, "label")) label = value; + else if(!strcmp(name, "units")) units = value; + else if(!strcmp(name, "label_color")) label_color = value; + else if(!strcmp(name, "value_color")) value_color = value; + else if(!strcmp(name, "multiply")) multiply_str = value; + else if(!strcmp(name, "divide")) divide_str = value; + else if(!strcmp(name, "refresh")) refresh_str = value; + else if(!strcmp(name, "precision")) precision_str = value; + else if(!strcmp(name, "scale")) scale_str = value; + else if(!strcmp(name, "fixed_width_lbl")) fixed_width_lbl_str = value; + else if(!strcmp(name, "fixed_width_val")) fixed_width_val_str = value; + else if(!strcmp(name, "alarm")) alarm = value; + else if(!strcmp(name, BADGE_URL_ARG_LBL_COLOR)) text_color_lbl_str = value; + else if(!strcmp(name, BADGE_URL_ARG_VAL_COLOR)) text_color_val_str = value; + } + + int fixed_width_lbl = -1; + int fixed_width_val = -1; + + if(fixed_width_lbl_str && *fixed_width_lbl_str + && fixed_width_val_str && *fixed_width_val_str) { + fixed_width_lbl = str2i(fixed_width_lbl_str); + fixed_width_val = str2i(fixed_width_val_str); + } + + if(!chart || !*chart) { + buffer_no_cacheable(w->response.data); + buffer_sprintf(w->response.data, "No chart id is given at the request."); + goto cleanup; + } + + int scale = (scale_str && *scale_str)?str2i(scale_str):100; + + RRDSET *st = rrdset_find(host, chart); + if(!st) st = rrdset_find_byname(host, chart); + if(!st) { + buffer_no_cacheable(w->response.data); + buffer_svg(w->response.data, "chart not found", NAN, "", NULL, NULL, -1, scale, 0, -1, -1, NULL, NULL); + ret = HTTP_RESP_OK; + goto cleanup; + } + st->last_accessed_time = now_realtime_sec(); + + RRDCALC *rc = NULL; + if(alarm) { + rc = rrdcalc_find(st, alarm); + if (!rc) { + buffer_no_cacheable(w->response.data); + buffer_svg(w->response.data, "alarm not found", NAN, "", NULL, NULL, -1, scale, 0, -1, -1, NULL, NULL); + ret = HTTP_RESP_OK; + goto cleanup; + } + } + + long long multiply = (multiply_str && *multiply_str )?str2l(multiply_str):1; + long long divide = (divide_str && *divide_str )?str2l(divide_str):1; + long long before = (before_str && *before_str )?str2l(before_str):0; + long long after = (after_str && *after_str )?str2l(after_str):-st->update_every; + int points = (points_str && *points_str )?str2i(points_str):1; + int precision = (precision_str && *precision_str)?str2i(precision_str):-1; + + if(!multiply) multiply = 1; + if(!divide) divide = 1; + + int refresh = 0; + if(refresh_str && *refresh_str) { + if(!strcmp(refresh_str, "auto")) { + if(rc) refresh = rc->update_every; + else if(options & RRDR_OPTION_NOT_ALIGNED) + refresh = st->update_every; + else { + refresh = (int)(before - after); + if(refresh < 0) refresh = -refresh; + } + } + else { + refresh = str2i(refresh_str); + if(refresh < 0) refresh = -refresh; + } + } + + if(!label) { + if(alarm) { + char *s = (char *)alarm; + while(*s) { + if(*s == '_') *s = ' '; + s++; + } + label = alarm; + } + else if(dimensions) { + const char *dim = buffer_tostring(dimensions); + if(*dim == '|') dim++; + label = dim; + } + else + label = st->name; + } + if(!units) { + if(alarm) { + if(rc->units) + units = rc->units; + else + units = ""; + } + else if(options & RRDR_OPTION_PERCENTAGE) + units = "%"; + else + units = st->units; + } + + debug(D_WEB_CLIENT, "%llu: API command 'badge.svg' for chart '%s', alarm '%s', dimensions '%s', after '%lld', before '%lld', points '%d', group '%d', options '0x%08x'" + , w->id + , chart + , alarm?alarm:"" + , (dimensions)?buffer_tostring(dimensions):"" + , after + , before + , points + , group + , options + ); + + if(rc) { + if (refresh > 0) { + buffer_sprintf(w->response.header, "Refresh: %d\r\n", refresh); + w->response.data->expires = now_realtime_sec() + refresh; + } + else buffer_no_cacheable(w->response.data); + + if(!value_color) { + switch(rc->status) { + case RRDCALC_STATUS_CRITICAL: + value_color = "red"; + break; + + case RRDCALC_STATUS_WARNING: + value_color = "orange"; + break; + + case RRDCALC_STATUS_CLEAR: + value_color = "brightgreen"; + break; + + case RRDCALC_STATUS_UNDEFINED: + value_color = "lightgrey"; + break; + + case RRDCALC_STATUS_UNINITIALIZED: + value_color = "#000"; + break; + + default: + value_color = "grey"; + break; + } + } + + buffer_svg(w->response.data, + label, + (isnan(rc->value)||isinf(rc->value)) ? rc->value : rc->value * multiply / divide, + units, + label_color, + value_color, + precision, + scale, + options, + fixed_width_lbl, + fixed_width_val, + text_color_lbl_str, + text_color_val_str + ); + ret = HTTP_RESP_OK; + } + else { + time_t latest_timestamp = 0; + int value_is_null = 1; + calculated_number n = NAN; + ret = HTTP_RESP_INTERNAL_SERVER_ERROR; + + // if the collected value is too old, don't calculate its value + if (rrdset_last_entry_t(st) >= (now_realtime_sec() - (st->update_every * st->gap_when_lost_iterations_above))) + ret = rrdset2value_api_v1(st, w->response.data, &n, (dimensions) ? buffer_tostring(dimensions) : NULL + , points, after, before, group, 0, options, NULL, &latest_timestamp, &value_is_null); + + // if the value cannot be calculated, show empty badge + if (ret != HTTP_RESP_OK) { + buffer_no_cacheable(w->response.data); + value_is_null = 1; + n = 0; + ret = HTTP_RESP_OK; + } + else if (refresh > 0) { + buffer_sprintf(w->response.header, "Refresh: %d\r\n", refresh); + w->response.data->expires = now_realtime_sec() + refresh; + } + else buffer_no_cacheable(w->response.data); + + // render the badge + buffer_svg(w->response.data, + label, + (value_is_null)?NAN:(n * multiply / divide), + units, + label_color, + value_color, + precision, + scale, + options, + fixed_width_lbl, + fixed_width_val, + text_color_lbl_str, + text_color_val_str + ); + } + + cleanup: + buffer_free(dimensions); + return ret; +} diff --git a/web/api/badges/web_buffer_svg.h b/web/api/badges/web_buffer_svg.h new file mode 100644 index 0000000..1cf69e2 --- /dev/null +++ b/web/api/badges/web_buffer_svg.h @@ -0,0 +1,16 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#ifndef NETDATA_WEB_BUFFER_SVG_H +#define NETDATA_WEB_BUFFER_SVG_H 1 + +#include "libnetdata/libnetdata.h" +#include "web/server/web_client.h" + +extern void buffer_svg(BUFFER *wb, const char *label, calculated_number value, const char *units, const char *label_color, const char *value_color, int precision, int scale, uint32_t options, int fixed_width_lbl, int fixed_width_val, const char* text_color_lbl, const char* text_color_val); +extern char *format_value_and_unit(char *value_string, size_t value_string_len, calculated_number value, const char *units, int precision); + +extern int web_client_api_request_v1_badge(struct rrdhost *host, struct web_client *w, char *url); + +#include "web/api/web_api_v1.h" + +#endif /* NETDATA_WEB_BUFFER_SVG_H */ diff --git a/web/api/exporters/Makefile.am b/web/api/exporters/Makefile.am new file mode 100644 index 0000000..06fda51 --- /dev/null +++ b/web/api/exporters/Makefile.am @@ -0,0 +1,13 @@ +# SPDX-License-Identifier: GPL-3.0-or-later + +AUTOMAKE_OPTIONS = subdir-objects +MAINTAINERCLEANFILES = $(srcdir)/Makefile.in + +SUBDIRS = \ + shell \ + prometheus \ + $(NULL) + +dist_noinst_DATA = \ + README.md \ + $(NULL) diff --git a/web/api/exporters/README.md b/web/api/exporters/README.md new file mode 100644 index 0000000..4019647 --- /dev/null +++ b/web/api/exporters/README.md @@ -0,0 +1,10 @@ +<!-- +title: "Exporters" +custom_edit_url: https://github.com/netdata/netdata/edit/master/web/api/exporters/README.md +--> + +# Exporters + +TBD + +[](<>) diff --git a/web/api/exporters/allmetrics.c b/web/api/exporters/allmetrics.c new file mode 100644 index 0000000..d10de3d --- /dev/null +++ b/web/api/exporters/allmetrics.c @@ -0,0 +1,127 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#include "allmetrics.h" + +struct prometheus_output_options { + char *name; + PROMETHEUS_OUTPUT_OPTIONS flag; +} prometheus_output_flags_root[] = { + { "help", PROMETHEUS_OUTPUT_HELP }, + { "types", PROMETHEUS_OUTPUT_TYPES }, + { "names", PROMETHEUS_OUTPUT_NAMES }, + { "timestamps", PROMETHEUS_OUTPUT_TIMESTAMPS }, + { "variables", PROMETHEUS_OUTPUT_VARIABLES }, + { "oldunits", PROMETHEUS_OUTPUT_OLDUNITS }, + { "hideunits", PROMETHEUS_OUTPUT_HIDEUNITS }, + // terminator + { NULL, PROMETHEUS_OUTPUT_NONE }, +}; + +inline int web_client_api_request_v1_allmetrics(RRDHOST *host, struct web_client *w, char *url) { + int format = ALLMETRICS_SHELL; + const char *prometheus_server = w->client_ip; + + uint32_t prometheus_exporting_options; + if (prometheus_exporter_instance) + prometheus_exporting_options = prometheus_exporter_instance->config.options; + else + prometheus_exporting_options = global_backend_options; + + PROMETHEUS_OUTPUT_OPTIONS prometheus_output_options = + PROMETHEUS_OUTPUT_TIMESTAMPS | + ((prometheus_exporting_options & BACKEND_OPTION_SEND_NAMES) ? PROMETHEUS_OUTPUT_NAMES : 0); + + const char *prometheus_prefix; + if (prometheus_exporter_instance) + prometheus_prefix = prometheus_exporter_instance->config.prefix; + else + prometheus_prefix = global_backend_prefix; + + while(url) { + char *value = mystrsep(&url, "&"); + if (!value || !*value) continue; + + char *name = mystrsep(&value, "="); + if(!name || !*name) continue; + if(!value || !*value) continue; + + if(!strcmp(name, "format")) { + if(!strcmp(value, ALLMETRICS_FORMAT_SHELL)) + format = ALLMETRICS_SHELL; + else if(!strcmp(value, ALLMETRICS_FORMAT_PROMETHEUS)) + format = ALLMETRICS_PROMETHEUS; + else if(!strcmp(value, ALLMETRICS_FORMAT_PROMETHEUS_ALL_HOSTS)) + format = ALLMETRICS_PROMETHEUS_ALL_HOSTS; + else if(!strcmp(value, ALLMETRICS_FORMAT_JSON)) + format = ALLMETRICS_JSON; + else + format = 0; + } + else if(!strcmp(name, "server")) { + prometheus_server = value; + } + else if(!strcmp(name, "prefix")) { + prometheus_prefix = value; + } + else if(!strcmp(name, "data") || !strcmp(name, "source") || !strcmp(name, "data source") || !strcmp(name, "data-source") || !strcmp(name, "data_source") || !strcmp(name, "datasource")) { + prometheus_exporting_options = backend_parse_data_source(value, prometheus_exporting_options); + } + else { + int i; + for(i = 0; prometheus_output_flags_root[i].name ; i++) { + if(!strcmp(name, prometheus_output_flags_root[i].name)) { + if(!strcmp(value, "yes") || !strcmp(value, "1") || !strcmp(value, "true")) + prometheus_output_options |= prometheus_output_flags_root[i].flag; + else + prometheus_output_options &= ~prometheus_output_flags_root[i].flag; + + break; + } + } + } + } + + buffer_flush(w->response.data); + buffer_no_cacheable(w->response.data); + + switch(format) { + case ALLMETRICS_JSON: + w->response.data->contenttype = CT_APPLICATION_JSON; + rrd_stats_api_v1_charts_allmetrics_json(host, w->response.data); + return HTTP_RESP_OK; + + case ALLMETRICS_SHELL: + w->response.data->contenttype = CT_TEXT_PLAIN; + rrd_stats_api_v1_charts_allmetrics_shell(host, w->response.data); + return HTTP_RESP_OK; + + case ALLMETRICS_PROMETHEUS: + w->response.data->contenttype = CT_PROMETHEUS; + rrd_stats_api_v1_charts_allmetrics_prometheus_single_host( + host + , w->response.data + , prometheus_server + , prometheus_prefix + , prometheus_exporting_options + , prometheus_output_options + ); + return HTTP_RESP_OK; + + case ALLMETRICS_PROMETHEUS_ALL_HOSTS: + w->response.data->contenttype = CT_PROMETHEUS; + rrd_stats_api_v1_charts_allmetrics_prometheus_all_hosts( + host + , w->response.data + , prometheus_server + , prometheus_prefix + , prometheus_exporting_options + , prometheus_output_options + ); + return HTTP_RESP_OK; + + default: + w->response.data->contenttype = CT_TEXT_PLAIN; + buffer_strcat(w->response.data, "Which format? '" ALLMETRICS_FORMAT_SHELL "', '" ALLMETRICS_FORMAT_PROMETHEUS "', '" ALLMETRICS_FORMAT_PROMETHEUS_ALL_HOSTS "' and '" ALLMETRICS_FORMAT_JSON "' are currently supported."); + return HTTP_RESP_BAD_REQUEST; + } +} diff --git a/web/api/exporters/allmetrics.h b/web/api/exporters/allmetrics.h new file mode 100644 index 0000000..f076ff0 --- /dev/null +++ b/web/api/exporters/allmetrics.h @@ -0,0 +1,12 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#ifndef NETDATA_API_ALLMETRICS_H +#define NETDATA_API_ALLMETRICS_H + +#include "web/api/formatters/rrd2json.h" +#include "shell/allmetrics_shell.h" +#include "web/server/web_client.h" + +extern int web_client_api_request_v1_allmetrics(RRDHOST *host, struct web_client *w, char *url); + +#endif //NETDATA_API_ALLMETRICS_H diff --git a/web/api/exporters/prometheus/Makefile.am b/web/api/exporters/prometheus/Makefile.am new file mode 100644 index 0000000..161784b --- /dev/null +++ b/web/api/exporters/prometheus/Makefile.am @@ -0,0 +1,8 @@ +# SPDX-License-Identifier: GPL-3.0-or-later + +AUTOMAKE_OPTIONS = subdir-objects +MAINTAINERCLEANFILES = $(srcdir)/Makefile.in + +dist_noinst_DATA = \ + README.md \ + $(NULL) diff --git a/web/api/exporters/prometheus/README.md b/web/api/exporters/prometheus/README.md new file mode 100644 index 0000000..d26c6e4 --- /dev/null +++ b/web/api/exporters/prometheus/README.md @@ -0,0 +1,10 @@ +<!-- +title: "prometheus exporter" +custom_edit_url: https://github.com/netdata/netdata/edit/master/web/api/exporters/prometheus/README.md +--> + +# prometheus exporter + +The prometheus exporter for Netdata is located at the [backends section for prometheus](/backends/prometheus/README.md). + +[](<>) diff --git a/web/api/exporters/shell/Makefile.am b/web/api/exporters/shell/Makefile.am new file mode 100644 index 0000000..161784b --- /dev/null +++ b/web/api/exporters/shell/Makefile.am @@ -0,0 +1,8 @@ +# SPDX-License-Identifier: GPL-3.0-or-later + +AUTOMAKE_OPTIONS = subdir-objects +MAINTAINERCLEANFILES = $(srcdir)/Makefile.in + +dist_noinst_DATA = \ + README.md \ + $(NULL) diff --git a/web/api/exporters/shell/README.md b/web/api/exporters/shell/README.md new file mode 100644 index 0000000..b919045 --- /dev/null +++ b/web/api/exporters/shell/README.md @@ -0,0 +1,71 @@ +<!-- +title: "shell exporter" +custom_edit_url: https://github.com/netdata/netdata/edit/master/web/api/exporters/shell/README.md +--> + +# shell exporter + +Shell scripts can now query Netdata: + +```sh +eval "$(curl -s 'http://localhost:19999/api/v1/allmetrics')" +``` + +after this command, all the Netdata metrics are exposed to shell. Check: + +```sh +# source the metrics +eval "$(curl -s 'http://localhost:19999/api/v1/allmetrics')" + +# let's see if there are variables exposed by Netdata for system.cpu +set | grep "^NETDATA_SYSTEM_CPU" + +NETDATA_SYSTEM_CPU_GUEST=0 +NETDATA_SYSTEM_CPU_GUEST_NICE=0 +NETDATA_SYSTEM_CPU_IDLE=95 +NETDATA_SYSTEM_CPU_IOWAIT=0 +NETDATA_SYSTEM_CPU_IRQ=0 +NETDATA_SYSTEM_CPU_NICE=0 +NETDATA_SYSTEM_CPU_SOFTIRQ=0 +NETDATA_SYSTEM_CPU_STEAL=0 +NETDATA_SYSTEM_CPU_SYSTEM=1 +NETDATA_SYSTEM_CPU_USER=4 +NETDATA_SYSTEM_CPU_VISIBLETOTAL=5 + +# let's see the total cpu utilization of the system +echo ${NETDATA_SYSTEM_CPU_VISIBLETOTAL} +5 + +# what about alarms? +set | grep "^NETDATA_ALARM_SYSTEM_SWAP_" +NETDATA_ALARM_SYSTEM_SWAP_RAM_IN_SWAP_STATUS=CRITICAL +NETDATA_ALARM_SYSTEM_SWAP_RAM_IN_SWAP_VALUE=53 +NETDATA_ALARM_SYSTEM_SWAP_USED_SWAP_STATUS=CLEAR +NETDATA_ALARM_SYSTEM_SWAP_USED_SWAP_VALUE=51 + +# let's get the current status of the alarm 'ram in swap' +echo ${NETDATA_ALARM_SYSTEM_SWAP_RAM_IN_SWAP_STATUS} +CRITICAL + +# is it fast? +time curl -s 'http://localhost:19999/api/v1/allmetrics' >/dev/null + +real 0m0,070s +user 0m0,000s +sys 0m0,007s + +# it is... +# 0.07 seconds for curl to be loaded, connect to Netdata and fetch the response back... +``` + +The `_VISIBLETOTAL` variable sums up all the dimensions of each chart. + +The format of the variables is: + +```sh +NETDATA_${chart_id^^}_${dimension_id^^}="${value}" +``` + +The value is rounded to the closest integer, since shell script cannot process decimal numbers. + +[](<>) diff --git a/web/api/exporters/shell/allmetrics_shell.c b/web/api/exporters/shell/allmetrics_shell.c new file mode 100644 index 0000000..daa0049 --- /dev/null +++ b/web/api/exporters/shell/allmetrics_shell.c @@ -0,0 +1,161 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#include "allmetrics_shell.h" + +// ---------------------------------------------------------------------------- +// BASH +// /api/v1/allmetrics?format=bash + +static inline size_t shell_name_copy(char *d, const char *s, size_t usable) { + size_t n; + + for(n = 0; *s && n < usable ; d++, s++, n++) { + register char c = *s; + + if(unlikely(!isalnum(c))) *d = '_'; + else *d = (char)toupper(c); + } + *d = '\0'; + + return n; +} + +#define SHELL_ELEMENT_MAX 100 + +void rrd_stats_api_v1_charts_allmetrics_shell(RRDHOST *host, BUFFER *wb) { + rrdhost_rdlock(host); + + // for each chart + RRDSET *st; + rrdset_foreach_read(st, host) { + calculated_number total = 0.0; + char chart[SHELL_ELEMENT_MAX + 1]; + shell_name_copy(chart, st->name?st->name:st->id, SHELL_ELEMENT_MAX); + + buffer_sprintf(wb, "\n# chart: %s (name: %s)\n", st->id, st->name); + if(rrdset_is_available_for_viewers(st)) { + rrdset_rdlock(st); + + // for each dimension + RRDDIM *rd; + rrddim_foreach_read(rd, st) { + if(rd->collections_counter && !rrddim_flag_check(rd, RRDDIM_FLAG_OBSOLETE)) { + char dimension[SHELL_ELEMENT_MAX + 1]; + shell_name_copy(dimension, rd->name?rd->name:rd->id, SHELL_ELEMENT_MAX); + + calculated_number n = rd->last_stored_value; + + if(isnan(n) || isinf(n)) + buffer_sprintf(wb, "NETDATA_%s_%s=\"\" # %s\n", chart, dimension, st->units); + else { + if(rd->multiplier < 0 || rd->divisor < 0) n = -n; + n = calculated_number_round(n); + if(!rrddim_flag_check(rd, RRDDIM_FLAG_HIDDEN)) total += n; + buffer_sprintf(wb, "NETDATA_%s_%s=\"" CALCULATED_NUMBER_FORMAT_ZERO "\" # %s\n", chart, dimension, n, st->units); + } + } + } + + total = calculated_number_round(total); + buffer_sprintf(wb, "NETDATA_%s_VISIBLETOTAL=\"" CALCULATED_NUMBER_FORMAT_ZERO "\" # %s\n", chart, total, st->units); + rrdset_unlock(st); + } + } + + buffer_strcat(wb, "\n# NETDATA ALARMS RUNNING\n"); + + RRDCALC *rc; + for(rc = host->alarms; rc ;rc = rc->next) { + if(!rc->rrdset) continue; + + char chart[SHELL_ELEMENT_MAX + 1]; + shell_name_copy(chart, rc->rrdset->name?rc->rrdset->name:rc->rrdset->id, SHELL_ELEMENT_MAX); + + char alarm[SHELL_ELEMENT_MAX + 1]; + shell_name_copy(alarm, rc->name, SHELL_ELEMENT_MAX); + + calculated_number n = rc->value; + + if(isnan(n) || isinf(n)) + buffer_sprintf(wb, "NETDATA_ALARM_%s_%s_VALUE=\"\" # %s\n", chart, alarm, rc->units); + else { + n = calculated_number_round(n); + buffer_sprintf(wb, "NETDATA_ALARM_%s_%s_VALUE=\"" CALCULATED_NUMBER_FORMAT_ZERO "\" # %s\n", chart, alarm, n, rc->units); + } + + buffer_sprintf(wb, "NETDATA_ALARM_%s_%s_STATUS=\"%s\"\n", chart, alarm, rrdcalc_status2string(rc->status)); + } + + rrdhost_unlock(host); +} + +// ---------------------------------------------------------------------------- + +void rrd_stats_api_v1_charts_allmetrics_json(RRDHOST *host, BUFFER *wb) { + rrdhost_rdlock(host); + + buffer_strcat(wb, "{"); + + size_t chart_counter = 0; + size_t dimension_counter = 0; + + // for each chart + RRDSET *st; + rrdset_foreach_read(st, host) { + if(rrdset_is_available_for_viewers(st)) { + rrdset_rdlock(st); + + buffer_sprintf(wb, "%s\n" + "\t\"%s\": {\n" + "\t\t\"name\":\"%s\",\n" + "\t\t\"family\":\"%s\",\n" + "\t\t\"context\":\"%s\",\n" + "\t\t\"units\":\"%s\",\n" + "\t\t\"last_updated\": %ld,\n" + "\t\t\"dimensions\": {" + , chart_counter?",":"" + , st->id + , st->name + , st->family + , st->context + , st->units + , rrdset_last_entry_t_nolock(st) + ); + + chart_counter++; + dimension_counter = 0; + + // for each dimension + RRDDIM *rd; + rrddim_foreach_read(rd, st) { + if(rd->collections_counter && !rrddim_flag_check(rd, RRDDIM_FLAG_OBSOLETE)) { + + buffer_sprintf(wb, "%s\n" + "\t\t\t\"%s\": {\n" + "\t\t\t\t\"name\": \"%s\",\n" + "\t\t\t\t\"value\": " + , dimension_counter?",":"" + , rd->id + , rd->name + ); + + if(isnan(rd->last_stored_value)) + buffer_strcat(wb, "null"); + else + buffer_sprintf(wb, CALCULATED_NUMBER_FORMAT, rd->last_stored_value); + + buffer_strcat(wb, "\n\t\t\t}"); + + dimension_counter++; + } + } + + buffer_strcat(wb, "\n\t\t}\n\t}"); + rrdset_unlock(st); + } + } + + buffer_strcat(wb, "\n}"); + rrdhost_unlock(host); +} + diff --git a/web/api/exporters/shell/allmetrics_shell.h b/web/api/exporters/shell/allmetrics_shell.h new file mode 100644 index 0000000..1d7611a --- /dev/null +++ b/web/api/exporters/shell/allmetrics_shell.h @@ -0,0 +1,21 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#ifndef NETDATA_API_ALLMETRICS_SHELL_H +#define NETDATA_API_ALLMETRICS_SHELL_H + +#include "../allmetrics.h" + +#define ALLMETRICS_FORMAT_SHELL "shell" +#define ALLMETRICS_FORMAT_PROMETHEUS "prometheus" +#define ALLMETRICS_FORMAT_PROMETHEUS_ALL_HOSTS "prometheus_all_hosts" +#define ALLMETRICS_FORMAT_JSON "json" + +#define ALLMETRICS_SHELL 1 +#define ALLMETRICS_PROMETHEUS 2 +#define ALLMETRICS_JSON 3 +#define ALLMETRICS_PROMETHEUS_ALL_HOSTS 4 + +extern void rrd_stats_api_v1_charts_allmetrics_json(RRDHOST *host, BUFFER *wb); +extern void rrd_stats_api_v1_charts_allmetrics_shell(RRDHOST *host, BUFFER *wb); + +#endif //NETDATA_API_ALLMETRICS_SHELL_H diff --git a/web/api/formatters/Makefile.am b/web/api/formatters/Makefile.am new file mode 100644 index 0000000..11f239c --- /dev/null +++ b/web/api/formatters/Makefile.am @@ -0,0 +1,15 @@ +# SPDX-License-Identifier: GPL-3.0-or-later + +AUTOMAKE_OPTIONS = subdir-objects +MAINTAINERCLEANFILES = $(srcdir)/Makefile.in + +SUBDIRS = \ + csv \ + json \ + ssv \ + value \ + $(NULL) + +dist_noinst_DATA = \ + README.md \ + $(NULL) diff --git a/web/api/formatters/README.md b/web/api/formatters/README.md new file mode 100644 index 0000000..1fd2b30 --- /dev/null +++ b/web/api/formatters/README.md @@ -0,0 +1,78 @@ +<!-- +title: "Query formatting" +custom_edit_url: https://github.com/netdata/netdata/edit/master/web/api/formatters/README.md +--> + +# Query formatting + +API data queries need to be formatted before returned to the caller. +Using API parameters, the caller may define the format he/she wishes to get back. + +The following formats are supported: + +| format|module|content type|description| +|:----:|:----:|:----------:|:----------| +| `array`|[ssv](/web/api/formatters/ssv/README.md)|application/json|a JSON array| +| `csv`|[csv](/web/api/formatters/csv/README.md)|text/plain|a text table, comma separated, with a header line (dimension names) and `\r\n` at the end of the lines| +| `csvjsonarray`|[csv](/web/api/formatters/csv/README.md)|application/json|a JSON array, with each row as another array (the first row has the dimension names)| +| `datasource`|[json](/web/api/formatters/json/README.md)|application/json|a Google Visualization Provider `datasource` javascript callback| +| `datatable`|[json](/web/api/formatters/json/README.md)|application/json|a Google `datatable`| +| `html`|[csv](/web/api/formatters/csv/README.md)|text/html|an html table| +| `json`|[json](/web/api/formatters/json/README.md)|application/json|a JSON object| +| `jsonp`|[json](/web/api/formatters/json/README.md)|application/json|a JSONP javascript callback| +| `markdown`|[csv](/web/api/formatters/csv/README.md)|text/plain|a markdown table| +| `ssv`|[ssv](/web/api/formatters/ssv/README.md)|text/plain|a space separated list of values| +| `ssvcomma`|[ssv](/web/api/formatters/ssv/README.md)|text/plain|a comma separated list of values| +| `tsv`|[csv](/web/api/formatters/csv/README.md)|text/plain|a TAB delimited `csv` (MS Excel flavor)| + +For examples of each format, check the relative module documentation. + +## Metadata with the `jsonwrap` option + +All data queries can be encapsulated to JSON object having metadata about the query and the results. + +This is done by adding the `options=jsonwrap` to the API URL (if there are other `options` append +`,jsonwrap` to the existing ones). + +This is such an object: + +```bash +# curl -Ss 'https://registry.my-netdata.io/api/v1/data?chart=system.cpu&after=-3600&points=6&group=average&format=csv&options=nonzero,jsonwrap' +{ + "api": 1, + "id": "system.cpu", + "name": "system.cpu", + "view_update_every": 600, + "update_every": 1, + "first_entry": 1540387074, + "last_entry": 1540647070, + "before": 1540647000, + "after": 1540644000, + "dimension_names": ["steal", "softirq", "user", "system", "iowait"], + "dimension_ids": ["steal", "softirq", "user", "system", "iowait"], + "latest_values": [0, 0.2493766, 1.745636, 0.4987531, 0], + "view_latest_values": [0.0158314, 0.0516506, 0.866549, 0.7196127, 0.0050002], + "dimensions": 5, + "points": 6, + "format": "csv", + "result": "time,steal,softirq,user,system,iowait\n2018-10-27 13:30:00,0.0158314,0.0516506,0.866549,0.7196127,0.0050002\n2018-10-27 13:20:00,0.0149856,0.0529183,0.8673155,0.7121144,0.0049979\n2018-10-27 13:10:00,0.0137501,0.053315,0.8578097,0.7197613,0.0054209\n2018-10-27 13:00:00,0.0154252,0.0554688,0.899432,0.7200638,0.0067252\n2018-10-27 12:50:00,0.0145866,0.0495922,0.8404341,0.7011141,0.0041688\n2018-10-27 12:40:00,0.0162366,0.0595954,0.8827475,0.7020573,0.0041636\n", + "min": 0, + "max": 0 +} +``` + +## Downloading data query result files + +Following the [Google Visualization Provider guidelines](https://developers.google.com/chart/interactive/docs/dev/implementing_data_source), +Netdata supports parsing `tqx` options. + +Using these options, any Netdata data query can instruct the web browser to download +the result and save it under a given filename. + +For example, to download a CSV file with CPU utilization of the last hour, +[click here](https://registry.my-netdata.io/api/v1/data?chart=system.cpu&after=-3600&format=csv&options=nonzero&tqx=outFileName:system+cpu+utilization+of+the+last_hour.csv). + +This is done by appending `&tqx=outFileName:FILENAME` to any data query. +The output will be in the format given with `&format=`. + +[](<>) diff --git a/web/api/formatters/charts2json.c b/web/api/formatters/charts2json.c new file mode 100644 index 0000000..856ffb5 --- /dev/null +++ b/web/api/formatters/charts2json.c @@ -0,0 +1,192 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#include "charts2json.h" + +// generate JSON for the /api/v1/charts API call + +const char* get_release_channel() { + static int use_stable = -1; + + if (use_stable == -1) { + char filename[FILENAME_MAX + 1]; + snprintfz(filename, FILENAME_MAX, "%s/.environment", netdata_configured_user_config_dir); + procfile *ff = procfile_open(filename, "=", PROCFILE_FLAG_DEFAULT); + if(!ff) { + use_stable=1; + } else { + procfile_set_quotes(ff, "'\""); + ff = procfile_readall(ff); + if(!ff) { + use_stable=1; + } else { + unsigned int i; + for(i = 0; i < procfile_lines(ff); i++) { + if (!procfile_linewords(ff, i)) continue; + + if (!strcmp(procfile_lineword(ff, i, 0), "RELEASE_CHANNEL") && !strcmp(procfile_lineword(ff, i, 1), "stable")) { + use_stable = 1; + break; + } + } + procfile_close(ff); + if (use_stable == -1) use_stable = 0; + } + } + } + return (use_stable)?"stable":"nightly"; +} + +void charts2json(RRDHOST *host, BUFFER *wb, int skip_volatile, int show_archived) { + static char *custom_dashboard_info_js_filename = NULL; + size_t c, dimensions = 0, memory = 0, alarms = 0; + RRDSET *st; + + time_t now = now_realtime_sec(); + + if(unlikely(!custom_dashboard_info_js_filename)) + custom_dashboard_info_js_filename = config_get(CONFIG_SECTION_WEB, "custom dashboard_info.js", ""); + + buffer_sprintf(wb, "{\n" + "\t\"hostname\": \"%s\"" + ",\n\t\"version\": \"%s\"" + ",\n\t\"release_channel\": \"%s\"" + ",\n\t\"os\": \"%s\"" + ",\n\t\"timezone\": \"%s\"" + ",\n\t\"update_every\": %d" + ",\n\t\"history\": %ld" + ",\n\t\"memory_mode\": \"%s\"" + ",\n\t\"custom_info\": \"%s\"" + ",\n\t\"charts\": {" + , host->hostname + , host->program_version + , get_release_channel() + , host->os + , host->timezone + , host->rrd_update_every + , host->rrd_history_entries + , rrd_memory_mode_name(host->rrd_memory_mode) + , custom_dashboard_info_js_filename + ); + + c = 0; + rrdhost_rdlock(host); + rrdset_foreach_read(st, host) { + if ((!show_archived && rrdset_is_available_for_viewers(st)) || (show_archived && rrdset_is_archived(st))) { + if(c) buffer_strcat(wb, ","); + buffer_strcat(wb, "\n\t\t\""); + buffer_strcat(wb, st->id); + buffer_strcat(wb, "\": "); + rrdset2json(st, wb, &dimensions, &memory, skip_volatile); + + c++; + st->last_accessed_time = now; + } + } + + RRDCALC *rc; + for(rc = host->alarms; rc ; rc = rc->next) { + if(rc->rrdset) + alarms++; + } + rrdhost_unlock(host); + + buffer_sprintf(wb + , "\n\t}" + ",\n\t\"charts_count\": %zu" + ",\n\t\"dimensions_count\": %zu" + ",\n\t\"alarms_count\": %zu" + ",\n\t\"rrd_memory_bytes\": %zu" + ",\n\t\"hosts_count\": %zu" + ",\n\t\"hosts\": [" + , c + , dimensions + , alarms + , memory + , rrd_hosts_available + ); + + if(unlikely(rrd_hosts_available > 1)) { + rrd_rdlock(); + + size_t found = 0; + RRDHOST *h; + rrdhost_foreach_read(h) { + if(!rrdhost_should_be_removed(h, host, now) && !rrdhost_flag_check(h, RRDHOST_FLAG_ARCHIVED)) { + buffer_sprintf(wb + , "%s\n\t\t{" + "\n\t\t\t\"hostname\": \"%s\"" + "\n\t\t}" + , (found > 0) ? "," : "" + , h->hostname + ); + + found++; + } + } + + rrd_unlock(); + } + else { + buffer_sprintf(wb + , "\n\t\t{" + "\n\t\t\t\"hostname\": \"%s\"" + "\n\t\t}" + , host->hostname + ); + } + + buffer_sprintf(wb, "\n\t]\n}\n"); +} + +// generate collectors list for the api/v1/info call + +struct collector { + char *plugin; + char *module; +}; + +struct array_printer { + int c; + BUFFER *wb; +}; + +int print_collector(void *entry, void *data) { + struct array_printer *ap = (struct array_printer *)data; + BUFFER *wb = ap->wb; + struct collector *col=(struct collector *) entry; + if(ap->c) buffer_strcat(wb, ","); + buffer_strcat(wb, "\n\t\t{\n\t\t\t\"plugin\": \""); + buffer_strcat(wb, col->plugin); + buffer_strcat(wb, "\",\n\t\t\t\"module\": \""); + buffer_strcat(wb, col->module); + buffer_strcat(wb, "\"\n\t\t}"); + (ap->c)++; + return 0; +} + +void chartcollectors2json(RRDHOST *host, BUFFER *wb) { + DICTIONARY *dict = dictionary_create(DICTIONARY_FLAG_SINGLE_THREADED); + RRDSET *st; + char name[500]; + + time_t now = now_realtime_sec(); + rrdhost_rdlock(host); + rrdset_foreach_read(st, host) { + if (rrdset_is_available_for_viewers(st)) { + struct collector col = { + .plugin = st->plugin_name ? st->plugin_name : "", + .module = st->module_name ? st->module_name : "" + }; + sprintf(name, "%s:%s", col.plugin, col.module); + dictionary_set(dict, name, &col, sizeof(struct collector)); + st->last_accessed_time = now; + } + } + rrdhost_unlock(host); + struct array_printer ap = { + .c = 0, + .wb = wb + }; + dictionary_get_all(dict, print_collector, &ap); + dictionary_destroy(dict); +} diff --git a/web/api/formatters/charts2json.h b/web/api/formatters/charts2json.h new file mode 100644 index 0000000..2d8cce3 --- /dev/null +++ b/web/api/formatters/charts2json.h @@ -0,0 +1,12 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#ifndef NETDATA_API_FORMATTER_CHARTS2JSON_H +#define NETDATA_API_FORMATTER_CHARTS2JSON_H + +#include "rrd2json.h" + +extern void charts2json(RRDHOST *host, BUFFER *wb, int skip_volatile, int show_archived); +extern void chartcollectors2json(RRDHOST *host, BUFFER *wb); +extern const char* get_release_channel(); + +#endif //NETDATA_API_FORMATTER_CHARTS2JSON_H diff --git a/web/api/formatters/csv/Makefile.am b/web/api/formatters/csv/Makefile.am new file mode 100644 index 0000000..161784b --- /dev/null +++ b/web/api/formatters/csv/Makefile.am @@ -0,0 +1,8 @@ +# SPDX-License-Identifier: GPL-3.0-or-later + +AUTOMAKE_OPTIONS = subdir-objects +MAINTAINERCLEANFILES = $(srcdir)/Makefile.in + +dist_noinst_DATA = \ + README.md \ + $(NULL) diff --git a/web/api/formatters/csv/README.md b/web/api/formatters/csv/README.md new file mode 100644 index 0000000..2a859e2 --- /dev/null +++ b/web/api/formatters/csv/README.md @@ -0,0 +1,144 @@ +<!-- +title: "CSV formatter" +custom_edit_url: https://github.com/netdata/netdata/edit/master/web/api/formatters/csv/README.md +--> + +# CSV formatter + +The CSV formatter presents [results of database queries](/web/api/queries/README.md) in the following formats: + +| format|content type|description| +| :----:|:----------:|:----------| +| `csv`|text/plain|a text table, comma separated, with a header line (dimension names) and `\r\n` at the end of the lines| +| `csvjsonarray`|application/json|a JSON array, with each row as another array (the first row has the dimension names)| +| `tsv`|text/plain|like `csv` but TAB is used instead of comma to separate values (MS Excel flavor)| +| `html`|text/html|an html table| +| `markdown`|text/plain|markdown table| + +In all formats the date and time is the first column. + +The CSV formatter respects the following API `&options=`: + +| option|supported|description| +|:----:|:-------:|:----------| +| `nonzero`|yes|to return only the dimensions that have at least a non-zero value| +| `flip`|yes|to return the rows older to newer (the default is newer to older)| +| `seconds`|yes|to return the date and time in unix timestamp| +| `ms`|yes|to return the date and time in unit timestamp as milliseconds| +| `percent`|yes|to replace all values with their percentage over the row total| +| `abs`|yes|to turn all values positive| +| `null2zero`|yes|to replace gaps with zeros (the default prints the string `null`| + +## Examples + +Get the system total bandwidth for all physical network interfaces, over the last hour, +in 6 rows (one for every 10 minutes), in `csv` format: + +Netdata always returns bandwidth in `kilobits`. + +```bash +# curl -Ss 'https://registry.my-netdata.io/api/v1/data?chart=system.net&format=csv&after=-3600&group=sum&points=6&options=abs' +time,received,sent +2018-10-26 23:50:00,90214.67847,215137.79762 +2018-10-26 23:40:00,90126.32286,238587.57522 +2018-10-26 23:30:00,86061.22688,213389.23526 +2018-10-26 23:20:00,85590.75164,206129.01608 +2018-10-26 23:10:00,83163.30691,194311.77384 +2018-10-26 23:00:00,85167.29657,197538.07773 +``` + +--- + +Get the max RAM used by the SQL server and any cron jobs, over the last hour, in 2 rows (one for every 30 +minutes), in `tsv` format, and format the date and time as unix timestamp: + +Netdata always returns memory in `MB`. + +```bash +# curl -Ss 'https://registry.my-netdata.io/api/v1/data?chart=apps.mem&format=tsv&after=-3600&group=max&points=2&options=nonzero,seconds&dimensions=sql,cron' +time sql cron +1540598400 61.95703 0.25 +1540596600 61.95703 0.25 +``` + +--- + +Get an HTML table of the last 4 values (4 seconds) of system CPU utilization: + +Netdata always returns CPU utilization as `%`. + +```bash +# curl -Ss 'https://registry.my-netdata.io/api/v1/data?chart=system.cpu&format=html&after=-4&options=nonzero' +<html> +<center> +<table border="0" cellpadding="5" cellspacing="5"> +<tr><td>time</td><td>softirq</td><td>user</td><td>system</td></tr> +<tr><td>2018-10-27 00:16:07</td><td>0.25</td><td>1</td><td>0.75</td></tr> +<tr><td>2018-10-27 00:16:06</td><td>0</td><td>1.0025063</td><td>0.5012531</td></tr> +<tr><td>2018-10-27 00:16:05</td><td>0</td><td>1</td><td>0.75</td></tr> +<tr><td>2018-10-27 00:16:04</td><td>0</td><td>1.0025063</td><td>0.7518797</td></tr> +</table> +</center> +</html> +``` + +This is how it looks when rendered by a web browser: + + + +--- + +Get a JSON array with the average bandwidth rate of the mysql server, over the last hour, in 6 values +(one every 10 minutes), and return the date and time in milliseconds: + +Netdata always returns bandwidth rates in `kilobits/s`. + +```bash +# curl -Ss 'https://registry.my-netdata.io/api/v1/data?chart=mysql_local.net&format=csvjsonarray&after=-3600&points=6&group=average&options=abs,ms' +[ +["time","in","out"], +[1540599600000,0.7499986,120.2810185], +[1540599000000,0.7500019,120.2815509], +[1540598400000,0.7499999,120.2812319], +[1540597800000,0.7500044,120.2819634], +[1540597200000,0.7499968,120.2807337], +[1540596600000,0.7499988,120.2810527] +] +``` + +--- + +Get the number of processes started per minute, for the last 10 minutes, in `markdown` format: + +```bash +# curl -Ss 'https://registry.my-netdata.io/api/v1/data?chart=system.forks&format=markdown&after=-600&points=10&group=sum' +time | started +:---: |:---: +2018-10-27 03:52:00| 245.1706149 +2018-10-27 03:51:00| 152.6654636 +2018-10-27 03:50:00| 163.1755789 +2018-10-27 03:49:00| 176.1574766 +2018-10-27 03:48:00| 178.0137076 +2018-10-27 03:47:00| 183.8306543 +2018-10-27 03:46:00| 264.1635621 +2018-10-27 03:45:00| 205.001551 +2018-10-27 03:44:00| 7026.9852167 +2018-10-27 03:43:00| 205.9904794 +``` + +And this is how it looks when formatted: + +| time | started | +|:--:|:-----:| +| 2018-10-27 03:52:00 | 245.1706149 | +| 2018-10-27 03:51:00 | 152.6654636 | +| 2018-10-27 03:50:00 | 163.1755789 | +| 2018-10-27 03:49:00 | 176.1574766 | +| 2018-10-27 03:48:00 | 178.0137076 | +| 2018-10-27 03:47:00 | 183.8306543 | +| 2018-10-27 03:46:00 | 264.1635621 | +| 2018-10-27 03:45:00 | 205.001551 | +| 2018-10-27 03:44:00 | 7026.9852167 | +| 2018-10-27 03:43:00 | 205.9904794 | + +[](<>) diff --git a/web/api/formatters/csv/csv.c b/web/api/formatters/csv/csv.c new file mode 100644 index 0000000..da0a6b5 --- /dev/null +++ b/web/api/formatters/csv/csv.c @@ -0,0 +1,143 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#include "libnetdata/libnetdata.h" +#include "csv.h" + +void rrdr2csv(RRDR *r, BUFFER *wb, uint32_t format, RRDR_OPTIONS options, const char *startline, const char *separator, const char *endline, const char *betweenlines, RRDDIM *temp_rd) { + rrdset_check_rdlock(r->st); + + //info("RRD2CSV(): %s: BEGIN", r->st->id); + long c, i; + RRDDIM *d; + + // print the csv header + for(c = 0, i = 0, d = temp_rd?temp_rd:r->st->dimensions; d && c < r->d ;c++, d = d->next) { + if(unlikely(r->od[c] & RRDR_DIMENSION_HIDDEN)) continue; + if(unlikely((options & RRDR_OPTION_NONZERO) && !(r->od[c] & RRDR_DIMENSION_NONZERO))) continue; + + if(!i) { + buffer_strcat(wb, startline); + if(options & RRDR_OPTION_LABEL_QUOTES) buffer_strcat(wb, "\""); + buffer_strcat(wb, "time"); + if(options & RRDR_OPTION_LABEL_QUOTES) buffer_strcat(wb, "\""); + } + buffer_strcat(wb, separator); + if(options & RRDR_OPTION_LABEL_QUOTES) buffer_strcat(wb, "\""); + buffer_strcat(wb, d->name); + if(options & RRDR_OPTION_LABEL_QUOTES) buffer_strcat(wb, "\""); + i++; + } + buffer_strcat(wb, endline); + + if(format == DATASOURCE_CSV_MARKDOWN) { + // print the --- line after header + for(c = 0, i = 0, d = temp_rd?temp_rd:r->st->dimensions; d && c < r->d ;c++, d = d->next) { + if(unlikely(r->od[c] & RRDR_DIMENSION_HIDDEN)) continue; + if(unlikely((options & RRDR_OPTION_NONZERO) && !(r->od[c] & RRDR_DIMENSION_NONZERO))) continue; + + if(!i) { + buffer_strcat(wb, startline); + if(options & RRDR_OPTION_LABEL_QUOTES) buffer_strcat(wb, "\""); + buffer_strcat(wb, ":---:"); + if(options & RRDR_OPTION_LABEL_QUOTES) buffer_strcat(wb, "\""); + } + buffer_strcat(wb, separator); + if(options & RRDR_OPTION_LABEL_QUOTES) buffer_strcat(wb, "\""); + buffer_strcat(wb, ":---:"); + if(options & RRDR_OPTION_LABEL_QUOTES) buffer_strcat(wb, "\""); + i++; + } + buffer_strcat(wb, endline); + } + + if(!i) { + // no dimensions present + return; + } + + long start = 0, end = rrdr_rows(r), step = 1; + if(!(options & RRDR_OPTION_REVERSED)) { + start = rrdr_rows(r) - 1; + end = -1; + step = -1; + } + + // for each line in the array + calculated_number total = 1; + for(i = start; i != end ;i += step) { + calculated_number *cn = &r->v[ i * r->d ]; + RRDR_VALUE_FLAGS *co = &r->o[ i * r->d ]; + + buffer_strcat(wb, betweenlines); + buffer_strcat(wb, startline); + + time_t now = r->t[i]; + + if((options & RRDR_OPTION_SECONDS) || (options & RRDR_OPTION_MILLISECONDS)) { + // print the timestamp of the line + buffer_rrd_value(wb, (calculated_number)now); + // in ms + if(options & RRDR_OPTION_MILLISECONDS) buffer_strcat(wb, "000"); + } + else { + // generate the local date time + struct tm tmbuf, *tm = localtime_r(&now, &tmbuf); + if(!tm) { error("localtime() failed."); continue; } + buffer_date(wb, tm->tm_year + 1900, tm->tm_mon + 1, tm->tm_mday, tm->tm_hour, tm->tm_min, tm->tm_sec); + } + + int set_min_max = 0; + if(unlikely(options & RRDR_OPTION_PERCENTAGE)) { + total = 0; + for(c = 0, d = temp_rd?temp_rd:r->st->dimensions; d && c < r->d ;c++, d = d->next) { + calculated_number n = cn[c]; + + if(likely((options & RRDR_OPTION_ABSOLUTE) && n < 0)) + n = -n; + + total += n; + } + // prevent a division by zero + if(total == 0) total = 1; + set_min_max = 1; + } + + // for each dimension + for(c = 0, d = temp_rd?temp_rd:r->st->dimensions; d && c < r->d ;c++, d = d->next) { + if(unlikely(r->od[c] & RRDR_DIMENSION_HIDDEN)) continue; + if(unlikely((options & RRDR_OPTION_NONZERO) && !(r->od[c] & RRDR_DIMENSION_NONZERO))) continue; + + buffer_strcat(wb, separator); + + calculated_number n = cn[c]; + + if(co[c] & RRDR_VALUE_EMPTY) { + if(options & RRDR_OPTION_NULL2ZERO) + buffer_strcat(wb, "0"); + else + buffer_strcat(wb, "null"); + } + else { + if(unlikely((options & RRDR_OPTION_ABSOLUTE) && n < 0)) + n = -n; + + if(unlikely(options & RRDR_OPTION_PERCENTAGE)) { + n = n * 100 / total; + + if(unlikely(set_min_max)) { + r->min = r->max = n; + set_min_max = 0; + } + + if(n < r->min) r->min = n; + if(n > r->max) r->max = n; + } + + buffer_rrd_value(wb, n); + } + } + + buffer_strcat(wb, endline); + } + //info("RRD2CSV(): %s: END", r->st->id); +} diff --git a/web/api/formatters/csv/csv.h b/web/api/formatters/csv/csv.h new file mode 100644 index 0000000..cf6020d --- /dev/null +++ b/web/api/formatters/csv/csv.h @@ -0,0 +1,12 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#ifndef NETDATA_API_FORMATTER_CSV_H +#define NETDATA_API_FORMATTER_CSV_H + +#include "web/api/queries/rrdr.h" + +extern void rrdr2csv(RRDR *r, BUFFER *wb, uint32_t format, RRDR_OPTIONS options, const char *startline, const char *separator, const char *endline, const char *betweenlines, RRDDIM *temp_rd); + +#include "../rrd2json.h" + +#endif //NETDATA_API_FORMATTER_CSV_H diff --git a/web/api/formatters/json/Makefile.am b/web/api/formatters/json/Makefile.am new file mode 100644 index 0000000..161784b --- /dev/null +++ b/web/api/formatters/json/Makefile.am @@ -0,0 +1,8 @@ +# SPDX-License-Identifier: GPL-3.0-or-later + +AUTOMAKE_OPTIONS = subdir-objects +MAINTAINERCLEANFILES = $(srcdir)/Makefile.in + +dist_noinst_DATA = \ + README.md \ + $(NULL) diff --git a/web/api/formatters/json/README.md b/web/api/formatters/json/README.md new file mode 100644 index 0000000..685a3f2 --- /dev/null +++ b/web/api/formatters/json/README.md @@ -0,0 +1,156 @@ +<!-- +title: "JSON formatter" +custom_edit_url: https://github.com/netdata/netdata/edit/master/web/api/formatters/json/README.md +--> + +# JSON formatter + +The CSV formatter presents [results of database queries](/web/api/queries/README.md) in the following formats: + +| format | content type | description| +|:----:|:----------:|:----------| +| `json` | application/json | return the query result as a json object| +| `jsonp` | application/json | return the query result as a JSONP javascript callback| +| `datatable` | application/json | return the query result as a Google `datatable`| +| `datasource` | application/json | return the query result as a Google Visualization Provider `datasource` javascript callback| + +The CSV formatter respects the following API `&options=`: + +| option | supported | description| +|:----:|:-------:|:----------| +| `google_json` | yes | enable the Google flavor of JSON (using double quotes for strings and `Date()` function for dates| +| `objectrows` | yes | return each row as an object, instead of an array| +| `nonzero` | yes | to return only the dimensions that have at least a non-zero value| +| `flip` | yes | to return the rows older to newer (the default is newer to older)| +| `seconds` | yes | to return the date and time in unix timestamp| +| `ms` | yes | to return the date and time in unit timestamp as milliseconds| +| `percent` | yes | to replace all values with their percentage over the row total| +| `abs` | yes | to turn all values positive| +| `null2zero` | yes | to replace gaps with zeros (the default prints the string `null`| + +## Examples + +To show the differences between each format, in the following examples we query the same +chart (having just one dimension called `active`), changing only the query `format` and its `options`. + +> Using `format=json` and `options=` + +```bash +# curl -Ss 'https://registry.my-netdata.io/api/v1/data?chart=nginx_local.connections&after=-3600&points=6&group=average&format=json&options=' +{ + "labels": ["time", "active"], + "data": + [ + [ 1540644600, 224.2516667], + [ 1540644000, 229.29], + [ 1540643400, 222.41], + [ 1540642800, 226.6816667], + [ 1540642200, 246.4083333], + [ 1540641600, 241.0966667] + ] +} +``` + +> Using `format=json` and `options=objectrows` + +```bash +# curl -Ss 'https://registry.my-netdata.io/api/v1/data?chart=nginx_local.connections&after=-3600&points=6&group=average&format=json&options=objectrows' +{ + "labels": ["time", "active"], + "data": + [ + { "time": 1540644600, "active": 224.2516667}, + { "time": 1540644000, "active": 229.29}, + { "time": 1540643400, "active": 222.41}, + { "time": 1540642800, "active": 226.6816667}, + { "time": 1540642200, "active": 246.4083333}, + { "time": 1540641600, "active": 241.0966667} + ] +} +``` + +> Using `format=json` and `options=objectrows,google_json` + +```bash +# curl -Ss 'https://registry.my-netdata.io/api/v1/data?chart=nginx_local.connections&after=-3600&points=6&group=average&formatjson&options=objectrows,google_json' +{ + "labels": ["time", "active"], + "data": + [ + { "time": new Date(2018,9,27,12,50,0), "active": 224.2516667}, + { "time": new Date(2018,9,27,12,40,0), "active": 229.29}, + { "time": new Date(2018,9,27,12,30,0), "active": 222.41}, + { "time": new Date(2018,9,27,12,20,0), "active": 226.6816667}, + { "time": new Date(2018,9,27,12,10,0), "active": 246.4083333}, + { "time": new Date(2018,9,27,12,0,0), "active": 241.0966667} + ] +} +``` + +> Using `format=jsonp` and `options=` + +```bash +curl -Ss 'https://registry.my-netdata.io/api/v1/data?chart=nginx_local.connections&after=-3600&points=6&group=average&formjsonp&options=' +callback({ + "labels": ["time", "active"], + "data": + [ + [ 1540645200, 235.885], + [ 1540644600, 224.2516667], + [ 1540644000, 229.29], + [ 1540643400, 222.41], + [ 1540642800, 226.6816667], + [ 1540642200, 246.4083333] + ] +}); +``` + +> Using `format=datatable` and `options=` + +```bash +curl -Ss 'https://registry.my-netdata.io/api/v1/data?chart=nginx_local.connections&after=-3600&points=6&group=average&formdatatable&options=' +{ + "cols": + [ + {"id":"","label":"time","pattern":"","type":"datetime"}, + {"id":"","label":"","pattern":"","type":"string","p":{"role":"annotation"}}, + {"id":"","label":"","pattern":"","type":"string","p":{"role":"annotationText"}}, + {"id":"","label":"active","pattern":"","type":"number"} + ], + "rows": + [ + {"c":[{"v":"Date(2018,9,27,13,0,0)"},{"v":null},{"v":null},{"v":235.885}]}, + {"c":[{"v":"Date(2018,9,27,12,50,0)"},{"v":null},{"v":null},{"v":224.2516667}]}, + {"c":[{"v":"Date(2018,9,27,12,40,0)"},{"v":null},{"v":null},{"v":229.29}]}, + {"c":[{"v":"Date(2018,9,27,12,30,0)"},{"v":null},{"v":null},{"v":222.41}]}, + {"c":[{"v":"Date(2018,9,27,12,20,0)"},{"v":null},{"v":null},{"v":226.6816667}]}, + {"c":[{"v":"Date(2018,9,27,12,10,0)"},{"v":null},{"v":null},{"v":246.4083333}]} + ] +} +``` + +> Using `format=datasource` and `options=` + +```bash +curl -Ss 'https://registry.my-netdata.io/api/v1/data?chart=nginx_local.connections&after=-3600&points=6&group=average&format=datasource&options=' +google.visualization.Query.setResponse({version:'0.6',reqId:'0',status:'ok',sig:'1540645368',table:{ + "cols": + [ + {"id":"","label":"time","pattern":"","type":"datetime"}, + {"id":"","label":"","pattern":"","type":"string","p":{"role":"annotation"}}, + {"id":"","label":"","pattern":"","type":"string","p":{"role":"annotationText"}}, + {"id":"","label":"active","pattern":"","type":"number"} + ], + "rows": + [ + {"c":[{"v":"Date(2018,9,27,13,0,0)"},{"v":null},{"v":null},{"v":235.885}]}, + {"c":[{"v":"Date(2018,9,27,12,50,0)"},{"v":null},{"v":null},{"v":224.2516667}]}, + {"c":[{"v":"Date(2018,9,27,12,40,0)"},{"v":null},{"v":null},{"v":229.29}]}, + {"c":[{"v":"Date(2018,9,27,12,30,0)"},{"v":null},{"v":null},{"v":222.41}]}, + {"c":[{"v":"Date(2018,9,27,12,20,0)"},{"v":null},{"v":null},{"v":226.6816667}]}, + {"c":[{"v":"Date(2018,9,27,12,10,0)"},{"v":null},{"v":null},{"v":246.4083333}]} + ] +}}); +``` + +[](<>) diff --git a/web/api/formatters/json/json.c b/web/api/formatters/json/json.c new file mode 100644 index 0000000..f28eb57 --- /dev/null +++ b/web/api/formatters/json/json.c @@ -0,0 +1,248 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#include "json.h" + +#define JSON_DATES_JS 1 +#define JSON_DATES_TIMESTAMP 2 + +void rrdr2json(RRDR *r, BUFFER *wb, RRDR_OPTIONS options, int datatable, RRDDIM *temp_rd) { + rrdset_check_rdlock(r->st); + + //info("RRD2JSON(): %s: BEGIN", r->st->id); + int row_annotations = 0, dates, dates_with_new = 0; + char kq[2] = "", // key quote + sq[2] = "", // string quote + pre_label[101] = "", // before each label + post_label[101] = "", // after each label + pre_date[101] = "", // the beginning of line, to the date + post_date[101] = "", // closing the date + pre_value[101] = "", // before each value + post_value[101] = "", // after each value + post_line[101] = "", // at the end of each row + normal_annotation[201] = "", // default row annotation + overflow_annotation[201] = "", // overflow row annotation + data_begin[101] = "", // between labels and values + finish[101] = ""; // at the end of everything + + if(datatable) { + dates = JSON_DATES_JS; + if( options & RRDR_OPTION_GOOGLE_JSON ) { + kq[0] = '\0'; + sq[0] = '\''; + } + else { + kq[0] = '"'; + sq[0] = '"'; + } + row_annotations = 1; + snprintfz(pre_date, 100, " {%sc%s:[{%sv%s:%s", kq, kq, kq, kq, sq); + snprintfz(post_date, 100, "%s}", sq); + snprintfz(pre_label, 100, ",\n {%sid%s:%s%s,%slabel%s:%s", kq, kq, sq, sq, kq, kq, sq); + snprintfz(post_label, 100, "%s,%spattern%s:%s%s,%stype%s:%snumber%s}", sq, kq, kq, sq, sq, kq, kq, sq, sq); + snprintfz(pre_value, 100, ",{%sv%s:", kq, kq); + strcpy(post_value, "}"); + strcpy(post_line, "]}"); + snprintfz(data_begin, 100, "\n ],\n %srows%s:\n [\n", kq, kq); + strcpy(finish, "\n ]\n}"); + + snprintfz(overflow_annotation, 200, ",{%sv%s:%sRESET OR OVERFLOW%s},{%sv%s:%sThe counters have been wrapped.%s}", kq, kq, sq, sq, kq, kq, sq, sq); + snprintfz(normal_annotation, 200, ",{%sv%s:null},{%sv%s:null}", kq, kq, kq, kq); + + buffer_sprintf(wb, "{\n %scols%s:\n [\n", kq, kq); + buffer_sprintf(wb, " {%sid%s:%s%s,%slabel%s:%stime%s,%spattern%s:%s%s,%stype%s:%sdatetime%s},\n", kq, kq, sq, sq, kq, kq, sq, sq, kq, kq, sq, sq, kq, kq, sq, sq); + buffer_sprintf(wb, " {%sid%s:%s%s,%slabel%s:%s%s,%spattern%s:%s%s,%stype%s:%sstring%s,%sp%s:{%srole%s:%sannotation%s}},\n", kq, kq, sq, sq, kq, kq, sq, sq, kq, kq, sq, sq, kq, kq, sq, sq, kq, kq, kq, kq, sq, sq); + buffer_sprintf(wb, " {%sid%s:%s%s,%slabel%s:%s%s,%spattern%s:%s%s,%stype%s:%sstring%s,%sp%s:{%srole%s:%sannotationText%s}}", kq, kq, sq, sq, kq, kq, sq, sq, kq, kq, sq, sq, kq, kq, sq, sq, kq, kq, kq, kq, sq, sq); + + // remove the valueobjects flag + // google wants its own keys + if(options & RRDR_OPTION_OBJECTSROWS) + options &= ~RRDR_OPTION_OBJECTSROWS; + } + else { + kq[0] = '"'; + sq[0] = '"'; + if(options & RRDR_OPTION_GOOGLE_JSON) { + dates = JSON_DATES_JS; + dates_with_new = 1; + } + else { + dates = JSON_DATES_TIMESTAMP; + dates_with_new = 0; + } + if( options & RRDR_OPTION_OBJECTSROWS ) + strcpy(pre_date, " { "); + else + strcpy(pre_date, " [ "); + strcpy(pre_label, ", \""); + strcpy(post_label, "\""); + strcpy(pre_value, ", "); + if( options & RRDR_OPTION_OBJECTSROWS ) + strcpy(post_line, "}"); + else + strcpy(post_line, "]"); + snprintfz(data_begin, 100, "],\n %sdata%s:\n [\n", kq, kq); + strcpy(finish, "\n ]\n}"); + + buffer_sprintf(wb, "{\n %slabels%s: [", kq, kq); + buffer_sprintf(wb, "%stime%s", sq, sq); + } + + // ------------------------------------------------------------------------- + // print the JSON header + + long c, i; + RRDDIM *rd; + + // print the header lines + for(c = 0, i = 0, rd = temp_rd?temp_rd:r->st->dimensions; rd && c < r->d ;c++, rd = rd->next) { + if(unlikely(r->od[c] & RRDR_DIMENSION_HIDDEN)) continue; + if(unlikely((options & RRDR_OPTION_NONZERO) && !(r->od[c] & RRDR_DIMENSION_NONZERO))) continue; + + buffer_strcat(wb, pre_label); + buffer_strcat(wb, rd->name); +// buffer_strcat(wb, "."); +// buffer_strcat(wb, rd->rrdset->name); + buffer_strcat(wb, post_label); + i++; + } + if(!i) { + buffer_strcat(wb, pre_label); + buffer_strcat(wb, "no data"); + buffer_strcat(wb, post_label); + } + + // print the begin of row data + buffer_strcat(wb, data_begin); + + // if all dimensions are hidden, print a null + if(!i) { + buffer_strcat(wb, finish); + return; + } + + long start = 0, end = rrdr_rows(r), step = 1; + if(!(options & RRDR_OPTION_REVERSED)) { + start = rrdr_rows(r) - 1; + end = -1; + step = -1; + } + + // for each line in the array + calculated_number total = 1; + for(i = start; i != end ;i += step) { + calculated_number *cn = &r->v[ i * r->d ]; + RRDR_VALUE_FLAGS *co = &r->o[ i * r->d ]; + + time_t now = r->t[i]; + + if(dates == JSON_DATES_JS) { + // generate the local date time + struct tm tmbuf, *tm = localtime_r(&now, &tmbuf); + if(!tm) { error("localtime_r() failed."); continue; } + + if(likely(i != start)) buffer_strcat(wb, ",\n"); + buffer_strcat(wb, pre_date); + + if( options & RRDR_OPTION_OBJECTSROWS ) + buffer_sprintf(wb, "%stime%s: ", kq, kq); + + if(dates_with_new) + buffer_strcat(wb, "new "); + + buffer_jsdate(wb, tm->tm_year + 1900, tm->tm_mon, tm->tm_mday, tm->tm_hour, tm->tm_min, tm->tm_sec); + + buffer_strcat(wb, post_date); + + if(row_annotations) { + // google supports one annotation per row + int annotation_found = 0; + for(c = 0, rd = temp_rd?temp_rd:r->st->dimensions; rd ;c++, rd = rd->next) { + if(unlikely(!(r->od[c] & RRDR_DIMENSION_SELECTED))) continue; + + if(co[c] & RRDR_VALUE_RESET) { + buffer_strcat(wb, overflow_annotation); + annotation_found = 1; + break; + } + } + if(!annotation_found) + buffer_strcat(wb, normal_annotation); + } + } + else { + // print the timestamp of the line + if(likely(i != start)) buffer_strcat(wb, ",\n"); + buffer_strcat(wb, pre_date); + + if( options & RRDR_OPTION_OBJECTSROWS ) + buffer_sprintf(wb, "%stime%s: ", kq, kq); + + buffer_rrd_value(wb, (calculated_number)r->t[i]); + // in ms + if(options & RRDR_OPTION_MILLISECONDS) buffer_strcat(wb, "000"); + + buffer_strcat(wb, post_date); + } + + int set_min_max = 0; + if(unlikely(options & RRDR_OPTION_PERCENTAGE)) { + total = 0; + for(c = 0, rd = temp_rd?temp_rd:r->st->dimensions; rd && c < r->d ;c++, rd = rd->next) { + calculated_number n = cn[c]; + + if(likely((options & RRDR_OPTION_ABSOLUTE) && n < 0)) + n = -n; + + total += n; + } + // prevent a division by zero + if(total == 0) total = 1; + set_min_max = 1; + } + + // for each dimension + for(c = 0, rd = temp_rd?temp_rd:r->st->dimensions; rd && c < r->d ;c++, rd = rd->next) { + if(unlikely(r->od[c] & RRDR_DIMENSION_HIDDEN)) continue; + if(unlikely((options & RRDR_OPTION_NONZERO) && !(r->od[c] & RRDR_DIMENSION_NONZERO))) continue; + + calculated_number n = cn[c]; + + buffer_strcat(wb, pre_value); + + if( options & RRDR_OPTION_OBJECTSROWS ) + buffer_sprintf(wb, "%s%s%s: ", kq, rd->name, kq); + + if(co[c] & RRDR_VALUE_EMPTY) { + if(options & RRDR_OPTION_NULL2ZERO) + buffer_strcat(wb, "0"); + else + buffer_strcat(wb, "null"); + } + else { + if(unlikely((options & RRDR_OPTION_ABSOLUTE) && n < 0)) + n = -n; + + if(unlikely(options & RRDR_OPTION_PERCENTAGE)) { + n = n * 100 / total; + + if(unlikely(set_min_max)) { + r->min = r->max = n; + set_min_max = 0; + } + + if(n < r->min) r->min = n; + if(n > r->max) r->max = n; + } + + buffer_rrd_value(wb, n); + } + + buffer_strcat(wb, post_value); + } + + buffer_strcat(wb, post_line); + } + + buffer_strcat(wb, finish); + //info("RRD2JSON(): %s: END", r->st->id); +} diff --git a/web/api/formatters/json/json.h b/web/api/formatters/json/json.h new file mode 100644 index 0000000..6d73a3f --- /dev/null +++ b/web/api/formatters/json/json.h @@ -0,0 +1,10 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#ifndef NETDATA_API_FORMATTER_JSON_H +#define NETDATA_API_FORMATTER_JSON_H + +#include "../rrd2json.h" + +extern void rrdr2json(RRDR *r, BUFFER *wb, RRDR_OPTIONS options, int datatable, RRDDIM *temp_rd); + +#endif //NETDATA_API_FORMATTER_JSON_H diff --git a/web/api/formatters/json_wrapper.c b/web/api/formatters/json_wrapper.c new file mode 100644 index 0000000..cf4f109 --- /dev/null +++ b/web/api/formatters/json_wrapper.c @@ -0,0 +1,295 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#include "json_wrapper.h" + +void rrdr_json_wrapper_begin(RRDR *r, BUFFER *wb, uint32_t format, RRDR_OPTIONS options, int string_value, RRDDIM *temp_rd, char *chart_label_key) { + rrdset_check_rdlock(r->st); + + long rows = rrdr_rows(r); + long c, i; + RRDDIM *rd; + + //info("JSONWRAPPER(): %s: BEGIN", r->st->id); + char kq[2] = "", // key quote + sq[2] = ""; // string quote + + if( options & RRDR_OPTION_GOOGLE_JSON ) { + kq[0] = '\0'; + sq[0] = '\''; + } + else { + kq[0] = '"'; + sq[0] = '"'; + } + + rrdset_rdlock(r->st); + buffer_sprintf(wb, "{\n" + " %sapi%s: 1,\n" + " %sid%s: %s%s%s,\n" + " %sname%s: %s%s%s,\n" + " %sview_update_every%s: %d,\n" + " %supdate_every%s: %d,\n" + " %sfirst_entry%s: %u,\n" + " %slast_entry%s: %u,\n" + " %sbefore%s: %u,\n" + " %safter%s: %u,\n" + " %sdimension_names%s: [" + , kq, kq + , kq, kq, sq, temp_rd?r->st->context:r->st->id, sq + , kq, kq, sq, temp_rd?r->st->context:r->st->name, sq + , kq, kq, r->update_every + , kq, kq, r->st->update_every + , kq, kq, (uint32_t)rrdset_first_entry_t_nolock(r->st) + , kq, kq, (uint32_t)rrdset_last_entry_t_nolock(r->st) + , kq, kq, (uint32_t)r->before + , kq, kq, (uint32_t)r->after + , kq, kq); + rrdset_unlock(r->st); + + for(c = 0, i = 0, rd = temp_rd?temp_rd:r->st->dimensions; rd && c < r->d ;c++, rd = rd->next) { + if(unlikely(r->od[c] & RRDR_DIMENSION_HIDDEN)) continue; + if(unlikely((options & RRDR_OPTION_NONZERO) && !(r->od[c] & RRDR_DIMENSION_NONZERO))) continue; + + if(i) buffer_strcat(wb, ", "); + buffer_strcat(wb, sq); + buffer_strcat(wb, rd->name); + buffer_strcat(wb, sq); + i++; + } + if(!i) { +#ifdef NETDATA_INTERNAL_CHECKS + error("RRDR is empty for %s (RRDR has %d dimensions, options is 0x%08x)", r->st->id, r->d, options); +#endif + rows = 0; + buffer_strcat(wb, sq); + buffer_strcat(wb, "no data"); + buffer_strcat(wb, sq); + } + + buffer_sprintf(wb, "],\n" + " %sdimension_ids%s: [" + , kq, kq); + + for(c = 0, i = 0, rd = temp_rd?temp_rd:r->st->dimensions; rd && c < r->d ;c++, rd = rd->next) { + if(unlikely(r->od[c] & RRDR_DIMENSION_HIDDEN)) continue; + if(unlikely((options & RRDR_OPTION_NONZERO) && !(r->od[c] & RRDR_DIMENSION_NONZERO))) continue; + + if(i) buffer_strcat(wb, ", "); + buffer_strcat(wb, sq); + buffer_strcat(wb, rd->id); + buffer_strcat(wb, sq); + i++; + } + if(!i) { + rows = 0; + buffer_strcat(wb, sq); + buffer_strcat(wb, "no data"); + buffer_strcat(wb, sq); + } + buffer_strcat(wb, "],\n"); + + // Composite charts + if (temp_rd) { + buffer_sprintf( + wb, + " %schart_ids%s: [", + kq, kq); + + for (c = 0, i = 0, rd = temp_rd ; rd && c < r->d; c++, rd = rd->next) { + if (unlikely(r->od[c] & RRDR_DIMENSION_HIDDEN)) + continue; + if (unlikely((options & RRDR_OPTION_NONZERO) && !(r->od[c] & RRDR_DIMENSION_NONZERO))) + continue; + + if (i) + buffer_strcat(wb, ", "); + buffer_strcat(wb, sq); + buffer_strcat(wb, rd->rrdset->name); + buffer_strcat(wb, sq); + i++; + } + if (!i) { + rows = 0; + buffer_strcat(wb, sq); + buffer_strcat(wb, "no data"); + buffer_strcat(wb, sq); + } + buffer_strcat(wb, "],\n"); + if (chart_label_key) { + buffer_sprintf(wb, " %schart_labels%s: { ", kq, kq); + + SIMPLE_PATTERN *pattern = simple_pattern_create(chart_label_key, ",|\t\r\n\f\v", SIMPLE_PATTERN_EXACT); + SIMPLE_PATTERN *original_pattern = pattern; + char *label_key = NULL; + int keys = 0; + while (pattern && (label_key = simple_pattern_iterate(&pattern))) { + uint32_t key_hash = simple_hash(label_key); + struct label *current_label; + + if (keys) + buffer_strcat(wb, ", "); + buffer_sprintf(wb, "%s%s%s : [", kq, label_key, kq); + keys++; + + for (c = 0, i = 0, rd = temp_rd; rd && c < r->d; c++, rd = rd->next) { + if (unlikely(r->od[c] & RRDR_DIMENSION_HIDDEN)) + continue; + if (unlikely((options & RRDR_OPTION_NONZERO) && !(r->od[c] & RRDR_DIMENSION_NONZERO))) + continue; + if (i) + buffer_strcat(wb, ", "); + + current_label = rrdset_lookup_label_key(rd->rrdset, label_key, key_hash); + if (current_label) { + buffer_strcat(wb, sq); + buffer_strcat(wb, current_label->value); + buffer_strcat(wb, sq); + } else + buffer_strcat(wb, "null"); + i++; + } + if (!i) { + rows = 0; + buffer_strcat(wb, sq); + buffer_strcat(wb, "no data"); + buffer_strcat(wb, sq); + } + buffer_strcat(wb, "]"); + } + buffer_strcat(wb, "},\n"); + simple_pattern_free(original_pattern); + } + } + + buffer_sprintf(wb, " %slatest_values%s: [" + , kq, kq); + + for(c = 0, i = 0, rd = temp_rd?temp_rd:r->st->dimensions; rd && c < r->d ;c++, rd = rd->next) { + if(unlikely(r->od[c] & RRDR_DIMENSION_HIDDEN)) continue; + if(unlikely((options & RRDR_OPTION_NONZERO) && !(r->od[c] & RRDR_DIMENSION_NONZERO))) continue; + + if(i) buffer_strcat(wb, ", "); + i++; + + calculated_number value = rd->last_stored_value; + if (NAN == value) + buffer_strcat(wb, "null"); + else + buffer_rrd_value(wb, value); + /* + storage_number n = rd->values[rrdset_last_slot(r->st)]; + + if(!does_storage_number_exist(n)) + buffer_strcat(wb, "null"); + else + buffer_rrd_value(wb, unpack_storage_number(n)); + */ + } + if(!i) { + rows = 0; + buffer_strcat(wb, "null"); + } + + buffer_sprintf(wb, "],\n" + " %sview_latest_values%s: [" + , kq, kq); + + i = 0; + if(rows) { + calculated_number total = 1; + + if(unlikely(options & RRDR_OPTION_PERCENTAGE)) { + total = 0; + for(c = 0, rd = temp_rd?temp_rd:r->st->dimensions; rd && c < r->d ;c++, rd = rd->next) { + calculated_number *cn = &r->v[ (rrdr_rows(r) - 1) * r->d ]; + calculated_number n = cn[c]; + + if(likely((options & RRDR_OPTION_ABSOLUTE) && n < 0)) + n = -n; + + total += n; + } + // prevent a division by zero + if(total == 0) total = 1; + } + + for(c = 0, i = 0, rd = temp_rd?temp_rd:r->st->dimensions; rd && c < r->d ;c++, rd = rd->next) { + if(unlikely(r->od[c] & RRDR_DIMENSION_HIDDEN)) continue; + if(unlikely((options & RRDR_OPTION_NONZERO) && !(r->od[c] & RRDR_DIMENSION_NONZERO))) continue; + + if(i) buffer_strcat(wb, ", "); + i++; + + calculated_number *cn = &r->v[ (rrdr_rows(r) - 1) * r->d ]; + RRDR_VALUE_FLAGS *co = &r->o[ (rrdr_rows(r) - 1) * r->d ]; + calculated_number n = cn[c]; + + if(co[c] & RRDR_VALUE_EMPTY) { + if(options & RRDR_OPTION_NULL2ZERO) + buffer_strcat(wb, "0"); + else + buffer_strcat(wb, "null"); + } + else { + if(unlikely((options & RRDR_OPTION_ABSOLUTE) && n < 0)) + n = -n; + + if(unlikely(options & RRDR_OPTION_PERCENTAGE)) + n = n * 100 / total; + + buffer_rrd_value(wb, n); + } + } + } + if(!i) { + rows = 0; + buffer_strcat(wb, "null"); + } + + buffer_sprintf(wb, "],\n" + " %sdimensions%s: %ld,\n" + " %spoints%s: %ld,\n" + " %sformat%s: %s" + , kq, kq, i + , kq, kq, rows + , kq, kq, sq + ); + + rrdr_buffer_print_format(wb, format); + + if((options & RRDR_OPTION_CUSTOM_VARS) && (options & RRDR_OPTION_JSON_WRAP)) { + buffer_sprintf(wb, "%s,\n %schart_variables%s: ", sq, kq, kq); + health_api_v1_chart_custom_variables2json(r->st, wb); + } + else + buffer_sprintf(wb, "%s", sq); + + buffer_sprintf(wb, ",\n %sresult%s: ", kq, kq); + + if(string_value) buffer_strcat(wb, sq); + //info("JSONWRAPPER(): %s: END", r->st->id); +} + +void rrdr_json_wrapper_end(RRDR *r, BUFFER *wb, uint32_t format, uint32_t options, int string_value) { + (void)format; + + char kq[2] = "", // key quote + sq[2] = ""; // string quote + + if( options & RRDR_OPTION_GOOGLE_JSON ) { + kq[0] = '\0'; + sq[0] = '\''; + } + else { + kq[0] = '"'; + sq[0] = '"'; + } + + if(string_value) buffer_strcat(wb, sq); + + buffer_sprintf(wb, ",\n %smin%s: ", kq, kq); + buffer_rrd_value(wb, r->min); + buffer_sprintf(wb, ",\n %smax%s: ", kq, kq); + buffer_rrd_value(wb, r->max); + buffer_strcat(wb, "\n}\n"); +} diff --git a/web/api/formatters/json_wrapper.h b/web/api/formatters/json_wrapper.h new file mode 100644 index 0000000..d48d5d1 --- /dev/null +++ b/web/api/formatters/json_wrapper.h @@ -0,0 +1,11 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#ifndef NETDATA_API_FORMATTER_JSON_WRAPPER_H +#define NETDATA_API_FORMATTER_JSON_WRAPPER_H + +#include "rrd2json.h" + +extern void rrdr_json_wrapper_begin(RRDR *r, BUFFER *wb, uint32_t format, RRDR_OPTIONS options, int string_value, RRDDIM *temp_rd, char *chart_key); +extern void rrdr_json_wrapper_end(RRDR *r, BUFFER *wb, uint32_t format, uint32_t options, int string_value); + +#endif //NETDATA_API_FORMATTER_JSON_WRAPPER_H diff --git a/web/api/formatters/rrd2json.c b/web/api/formatters/rrd2json.c new file mode 100644 index 0000000..d8e2480 --- /dev/null +++ b/web/api/formatters/rrd2json.c @@ -0,0 +1,401 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#include "web/api/web_api_v1.h" + +static inline void free_temp_rrddim(RRDDIM *temp_rd) +{ + if (unlikely(!temp_rd)) + return; + + RRDDIM *t; + while (temp_rd) { + t = temp_rd->next; + freez((char *)temp_rd->id); + freez((char *)temp_rd->name); +#ifdef ENABLE_DBENGINE + if (temp_rd->rrd_memory_mode == RRD_MEMORY_MODE_DBENGINE) + freez(temp_rd->state->metric_uuid); +#endif + freez(temp_rd->state); + freez(temp_rd); + temp_rd = t; + } +} + +void free_context_param_list(struct context_param **param_list) +{ + if (unlikely(!param_list || !*param_list)) + return; + + free_temp_rrddim(((*param_list)->rd)); + freez((*param_list)); + *param_list = NULL; +} + +void rebuild_context_param_list(struct context_param *context_param_list, time_t after_requested) +{ + RRDDIM *temp_rd = context_param_list->rd; + RRDDIM *new_rd_list = NULL, *t; + while (temp_rd) { + t = temp_rd->next; + if (rrdset_last_entry_t(temp_rd->rrdset) >= after_requested) { + temp_rd->next = new_rd_list; + new_rd_list = temp_rd; + } else { + freez((char *)temp_rd->id); + freez((char *)temp_rd->name); +#ifdef ENABLE_DBENGINE + if (temp_rd->rrd_memory_mode == RRD_MEMORY_MODE_DBENGINE) + freez(temp_rd->state->metric_uuid); +#endif + freez(temp_rd->state); + freez(temp_rd); + } + temp_rd = t; + } + context_param_list->rd = new_rd_list; +}; + +void build_context_param_list(struct context_param **param_list, RRDSET *st) +{ + if (unlikely(!param_list || !st)) + return; + + if (unlikely(!(*param_list))) { + *param_list = mallocz(sizeof(struct context_param)); + (*param_list)->first_entry_t = LONG_MAX; + (*param_list)->last_entry_t = 0; + (*param_list)->rd = NULL; + } + + RRDDIM *rd1; + st->last_accessed_time = now_realtime_sec(); + rrdset_rdlock(st); + + (*param_list)->first_entry_t = MIN((*param_list)->first_entry_t, rrdset_first_entry_t_nolock(st)); + (*param_list)->last_entry_t = MAX((*param_list)->last_entry_t, rrdset_last_entry_t_nolock(st)); + + rrddim_foreach_read(rd1, st) { + RRDDIM *rd = mallocz(rd1->memsize); + memcpy(rd, rd1, rd1->memsize); + rd->id = strdupz(rd1->id); + rd->name = strdupz(rd1->name); + rd->state = mallocz(sizeof(*rd->state)); + memcpy(rd->state, rd1->state, sizeof(*rd->state)); + memcpy(&rd->state->collect_ops, &rd1->state->collect_ops, sizeof(struct rrddim_collect_ops)); + memcpy(&rd->state->query_ops, &rd1->state->query_ops, sizeof(struct rrddim_query_ops)); +#ifdef ENABLE_DBENGINE + if (rd->rrd_memory_mode == RRD_MEMORY_MODE_DBENGINE) { + rd->state->metric_uuid = mallocz(sizeof(uuid_t)); + uuid_copy(*rd->state->metric_uuid, *rd1->state->metric_uuid); + } +#endif + rd->next = (*param_list)->rd; + (*param_list)->rd = rd; + } + + rrdset_unlock(st); +} + +void rrd_stats_api_v1_chart(RRDSET *st, BUFFER *wb) { + rrdset2json(st, wb, NULL, NULL, 0); +} + +void rrdr_buffer_print_format(BUFFER *wb, uint32_t format) { + switch(format) { + case DATASOURCE_JSON: + buffer_strcat(wb, DATASOURCE_FORMAT_JSON); + break; + + case DATASOURCE_DATATABLE_JSON: + buffer_strcat(wb, DATASOURCE_FORMAT_DATATABLE_JSON); + break; + + case DATASOURCE_DATATABLE_JSONP: + buffer_strcat(wb, DATASOURCE_FORMAT_DATATABLE_JSONP); + break; + + case DATASOURCE_JSONP: + buffer_strcat(wb, DATASOURCE_FORMAT_JSONP); + break; + + case DATASOURCE_SSV: + buffer_strcat(wb, DATASOURCE_FORMAT_SSV); + break; + + case DATASOURCE_CSV: + buffer_strcat(wb, DATASOURCE_FORMAT_CSV); + break; + + case DATASOURCE_TSV: + buffer_strcat(wb, DATASOURCE_FORMAT_TSV); + break; + + case DATASOURCE_HTML: + buffer_strcat(wb, DATASOURCE_FORMAT_HTML); + break; + + case DATASOURCE_JS_ARRAY: + buffer_strcat(wb, DATASOURCE_FORMAT_JS_ARRAY); + break; + + case DATASOURCE_SSV_COMMA: + buffer_strcat(wb, DATASOURCE_FORMAT_SSV_COMMA); + break; + + default: + buffer_strcat(wb, "unknown"); + break; + } +} + +int rrdset2value_api_v1( + RRDSET *st + , BUFFER *wb + , calculated_number *n + , const char *dimensions + , long points + , long long after + , long long before + , int group_method + , long group_time + , uint32_t options + , time_t *db_after + , time_t *db_before + , int *value_is_null +) { + + RRDR *r = rrd2rrdr(st, points, after, before, group_method, group_time, options, dimensions, NULL); + + if(!r) { + if(value_is_null) *value_is_null = 1; + return HTTP_RESP_INTERNAL_SERVER_ERROR; + } + + if(rrdr_rows(r) == 0) { + rrdr_free(r); + + if(db_after) *db_after = 0; + if(db_before) *db_before = 0; + if(value_is_null) *value_is_null = 1; + + return HTTP_RESP_BAD_REQUEST; + } + + if(wb) { + if (r->result_options & RRDR_RESULT_OPTION_RELATIVE) + buffer_no_cacheable(wb); + else if (r->result_options & RRDR_RESULT_OPTION_ABSOLUTE) + buffer_cacheable(wb); + } + + if(db_after) *db_after = r->after; + if(db_before) *db_before = r->before; + + long i = (!(options & RRDR_OPTION_REVERSED))?rrdr_rows(r) - 1:0; + *n = rrdr2value(r, i, options, value_is_null); + + rrdr_free(r); + return HTTP_RESP_OK; +} + +int rrdset2anything_api_v1( + RRDSET *st + , BUFFER *wb + , BUFFER *dimensions + , uint32_t format + , long points + , long long after + , long long before + , int group_method + , long group_time + , uint32_t options + , time_t *latest_timestamp + , struct context_param *context_param_list + , char *chart_label_key +) { + time_t last_accessed_time = now_realtime_sec(); + st->last_accessed_time = last_accessed_time; + + + RRDR *r = rrd2rrdr(st, points, after, before, group_method, group_time, options, dimensions?buffer_tostring(dimensions):NULL, context_param_list); + if(!r) { + buffer_strcat(wb, "Cannot generate output with these parameters on this chart."); + return HTTP_RESP_INTERNAL_SERVER_ERROR; + } + + RRDDIM *temp_rd = context_param_list ? context_param_list->rd : NULL; + + if(r->result_options & RRDR_RESULT_OPTION_RELATIVE) + buffer_no_cacheable(wb); + else if(r->result_options & RRDR_RESULT_OPTION_ABSOLUTE) + buffer_cacheable(wb); + + if(latest_timestamp && rrdr_rows(r) > 0) + *latest_timestamp = r->before; + + switch(format) { + case DATASOURCE_SSV: + if(options & RRDR_OPTION_JSON_WRAP) { + wb->contenttype = CT_APPLICATION_JSON; + rrdr_json_wrapper_begin(r, wb, format, options, 1, temp_rd, chart_label_key); + rrdr2ssv(r, wb, options, "", " ", ""); + rrdr_json_wrapper_end(r, wb, format, options, 1); + } + else { + wb->contenttype = CT_TEXT_PLAIN; + rrdr2ssv(r, wb, options, "", " ", ""); + } + break; + + case DATASOURCE_SSV_COMMA: + if(options & RRDR_OPTION_JSON_WRAP) { + wb->contenttype = CT_APPLICATION_JSON; + rrdr_json_wrapper_begin(r, wb, format, options, 1, temp_rd, chart_label_key); + rrdr2ssv(r, wb, options, "", ",", ""); + rrdr_json_wrapper_end(r, wb, format, options, 1); + } + else { + wb->contenttype = CT_TEXT_PLAIN; + rrdr2ssv(r, wb, options, "", ",", ""); + } + break; + + case DATASOURCE_JS_ARRAY: + if(options & RRDR_OPTION_JSON_WRAP) { + wb->contenttype = CT_APPLICATION_JSON; + rrdr_json_wrapper_begin(r, wb, format, options, 0, temp_rd, chart_label_key); + rrdr2ssv(r, wb, options, "[", ",", "]"); + rrdr_json_wrapper_end(r, wb, format, options, 0); + } + else { + wb->contenttype = CT_APPLICATION_JSON; + rrdr2ssv(r, wb, options, "[", ",", "]"); + } + break; + + case DATASOURCE_CSV: + if(options & RRDR_OPTION_JSON_WRAP) { + wb->contenttype = CT_APPLICATION_JSON; + rrdr_json_wrapper_begin(r, wb, format, options, 1, temp_rd, chart_label_key); + rrdr2csv(r, wb, format, options, "", ",", "\\n", "", temp_rd); + rrdr_json_wrapper_end(r, wb, format, options, 1); + } + else { + wb->contenttype = CT_TEXT_PLAIN; + rrdr2csv(r, wb, format, options, "", ",", "\r\n", "", temp_rd); + } + break; + + case DATASOURCE_CSV_MARKDOWN: + if(options & RRDR_OPTION_JSON_WRAP) { + wb->contenttype = CT_APPLICATION_JSON; + rrdr_json_wrapper_begin(r, wb, format, options, 1, temp_rd, chart_label_key); + rrdr2csv(r, wb, format, options, "", "|", "\\n", "", temp_rd); + rrdr_json_wrapper_end(r, wb, format, options, 1); + } + else { + wb->contenttype = CT_TEXT_PLAIN; + rrdr2csv(r, wb, format, options, "", "|", "\r\n", "", temp_rd); + } + break; + + case DATASOURCE_CSV_JSON_ARRAY: + wb->contenttype = CT_APPLICATION_JSON; + if(options & RRDR_OPTION_JSON_WRAP) { + rrdr_json_wrapper_begin(r, wb, format, options, 0, temp_rd, chart_label_key); + buffer_strcat(wb, "[\n"); + rrdr2csv(r, wb, format, options + RRDR_OPTION_LABEL_QUOTES, "[", ",", "]", ",\n", temp_rd); + buffer_strcat(wb, "\n]"); + rrdr_json_wrapper_end(r, wb, format, options, 0); + } + else { + wb->contenttype = CT_APPLICATION_JSON; + buffer_strcat(wb, "[\n"); + rrdr2csv(r, wb, format, options + RRDR_OPTION_LABEL_QUOTES, "[", ",", "]", ",\n", temp_rd); + buffer_strcat(wb, "\n]"); + } + break; + + case DATASOURCE_TSV: + if(options & RRDR_OPTION_JSON_WRAP) { + wb->contenttype = CT_APPLICATION_JSON; + rrdr_json_wrapper_begin(r, wb, format, options, 1, temp_rd, chart_label_key); + rrdr2csv(r, wb, format, options, "", "\t", "\\n", "", temp_rd); + rrdr_json_wrapper_end(r, wb, format, options, 1); + } + else { + wb->contenttype = CT_TEXT_PLAIN; + rrdr2csv(r, wb, format, options, "", "\t", "\r\n", "", temp_rd); + } + break; + + case DATASOURCE_HTML: + if(options & RRDR_OPTION_JSON_WRAP) { + wb->contenttype = CT_APPLICATION_JSON; + rrdr_json_wrapper_begin(r, wb, format, options, 1, temp_rd, chart_label_key); + buffer_strcat(wb, "<html>\\n<center>\\n<table border=\\\"0\\\" cellpadding=\\\"5\\\" cellspacing=\\\"5\\\">\\n"); + rrdr2csv(r, wb, format, options, "<tr><td>", "</td><td>", "</td></tr>\\n", "", temp_rd); + buffer_strcat(wb, "</table>\\n</center>\\n</html>\\n"); + rrdr_json_wrapper_end(r, wb, format, options, 1); + } + else { + wb->contenttype = CT_TEXT_HTML; + buffer_strcat(wb, "<html>\n<center>\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"5\">\n"); + rrdr2csv(r, wb, format, options, "<tr><td>", "</td><td>", "</td></tr>\n", "", temp_rd); + buffer_strcat(wb, "</table>\n</center>\n</html>\n"); + } + break; + + case DATASOURCE_DATATABLE_JSONP: + wb->contenttype = CT_APPLICATION_X_JAVASCRIPT; + + if(options & RRDR_OPTION_JSON_WRAP) + rrdr_json_wrapper_begin(r, wb, format, options, 0, temp_rd, chart_label_key); + + rrdr2json(r, wb, options, 1, temp_rd); + + if(options & RRDR_OPTION_JSON_WRAP) + rrdr_json_wrapper_end(r, wb, format, options, 0); + break; + + case DATASOURCE_DATATABLE_JSON: + wb->contenttype = CT_APPLICATION_JSON; + + if(options & RRDR_OPTION_JSON_WRAP) + rrdr_json_wrapper_begin(r, wb, format, options, 0, temp_rd, chart_label_key); + + rrdr2json(r, wb, options, 1, temp_rd); + + if(options & RRDR_OPTION_JSON_WRAP) + rrdr_json_wrapper_end(r, wb, format, options, 0); + break; + + case DATASOURCE_JSONP: + wb->contenttype = CT_APPLICATION_X_JAVASCRIPT; + if(options & RRDR_OPTION_JSON_WRAP) + rrdr_json_wrapper_begin(r, wb, format, options, 0, temp_rd, chart_label_key); + + rrdr2json(r, wb, options, 0, temp_rd); + + if(options & RRDR_OPTION_JSON_WRAP) + rrdr_json_wrapper_end(r, wb, format, options, 0); + break; + + case DATASOURCE_JSON: + default: + wb->contenttype = CT_APPLICATION_JSON; + + if(options & RRDR_OPTION_JSON_WRAP) + rrdr_json_wrapper_begin(r, wb, format, options, 0, temp_rd, chart_label_key); + + rrdr2json(r, wb, options, 0, temp_rd); + + if(options & RRDR_OPTION_JSON_WRAP) + rrdr_json_wrapper_end(r, wb, format, options, 0); + break; + } + + rrdr_free(r); + return HTTP_RESP_OK; +} diff --git a/web/api/formatters/rrd2json.h b/web/api/formatters/rrd2json.h new file mode 100644 index 0000000..3dc5989 --- /dev/null +++ b/web/api/formatters/rrd2json.h @@ -0,0 +1,92 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#ifndef NETDATA_RRD2JSON_H +#define NETDATA_RRD2JSON_H 1 + +#include "web/api/web_api_v1.h" +#include "web/api/exporters/allmetrics.h" +#include "web/api/queries/rrdr.h" + +#include "web/api/formatters/csv/csv.h" +#include "web/api/formatters/ssv/ssv.h" +#include "web/api/formatters/json/json.h" +#include "web/api/formatters/value/value.h" + +#include "web/api/formatters/rrdset2json.h" +#include "web/api/formatters/charts2json.h" +#include "web/api/formatters/json_wrapper.h" + +#include "web/server/web_client.h" + +#define HOSTNAME_MAX 1024 + +#define API_RELATIVE_TIME_MAX (3 * 365 * 86400) + +// type of JSON generations +#define DATASOURCE_INVALID (-1) +#define DATASOURCE_JSON 0 +#define DATASOURCE_DATATABLE_JSON 1 +#define DATASOURCE_DATATABLE_JSONP 2 +#define DATASOURCE_SSV 3 +#define DATASOURCE_CSV 4 +#define DATASOURCE_JSONP 5 +#define DATASOURCE_TSV 6 +#define DATASOURCE_HTML 7 +#define DATASOURCE_JS_ARRAY 8 +#define DATASOURCE_SSV_COMMA 9 +#define DATASOURCE_CSV_JSON_ARRAY 10 +#define DATASOURCE_CSV_MARKDOWN 11 + +#define DATASOURCE_FORMAT_JSON "json" +#define DATASOURCE_FORMAT_DATATABLE_JSON "datatable" +#define DATASOURCE_FORMAT_DATATABLE_JSONP "datasource" +#define DATASOURCE_FORMAT_JSONP "jsonp" +#define DATASOURCE_FORMAT_SSV "ssv" +#define DATASOURCE_FORMAT_CSV "csv" +#define DATASOURCE_FORMAT_TSV "tsv" +#define DATASOURCE_FORMAT_HTML "html" +#define DATASOURCE_FORMAT_JS_ARRAY "array" +#define DATASOURCE_FORMAT_SSV_COMMA "ssvcomma" +#define DATASOURCE_FORMAT_CSV_JSON_ARRAY "csvjsonarray" +#define DATASOURCE_FORMAT_CSV_MARKDOWN "markdown" + +extern void rrd_stats_api_v1_chart(RRDSET *st, BUFFER *wb); +extern void rrdr_buffer_print_format(BUFFER *wb, uint32_t format); + +extern int rrdset2anything_api_v1( + RRDSET *st + , BUFFER *wb + , BUFFER *dimensions + , uint32_t format + , long points + , long long after + , long long before + , int group_method + , long group_time + , uint32_t options + , time_t *latest_timestamp + , struct context_param *context_param_list + , char *chart_label_key +); + +extern int rrdset2value_api_v1( + RRDSET *st + , BUFFER *wb + , calculated_number *n + , const char *dimensions + , long points + , long long after + , long long before + , int group_method + , long group_time + , uint32_t options + , time_t *db_after + , time_t *db_before + , int *value_is_null +); + +extern void build_context_param_list(struct context_param **param_list, RRDSET *st); +extern void rebuild_context_param_list(struct context_param *context_param_list, time_t after_requested); +extern void free_context_param_list(struct context_param **param_list); + +#endif /* NETDATA_RRD2JSON_H */ diff --git a/web/api/formatters/rrdset2json.c b/web/api/formatters/rrdset2json.c new file mode 100644 index 0000000..5482881 --- /dev/null +++ b/web/api/formatters/rrdset2json.c @@ -0,0 +1,161 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#include "rrdset2json.h" + +void chart_labels2json(RRDSET *st, BUFFER *wb, size_t indentation) +{ + char tabs[11]; + struct label_index *labels = &st->state->labels; + + if (indentation > 10) + indentation = 10; + + tabs[0] = '\0'; + while (indentation) { + strcat(tabs, "\t"); + indentation--; + } + + int count = 0; + netdata_rwlock_rdlock(&labels->labels_rwlock); + for (struct label *label = labels->head; label; label = label->next) { + if(count > 0) buffer_strcat(wb, ",\n"); + buffer_strcat(wb, tabs); + + char value[CONFIG_MAX_VALUE * 2 + 1]; + sanitize_json_string(value, label->value, CONFIG_MAX_VALUE * 2); + buffer_sprintf(wb, "\"%s\": \"%s\"", label->key, value); + + count++; + } + buffer_strcat(wb, "\n"); + netdata_rwlock_unlock(&labels->labels_rwlock); +} + +// generate JSON for the /api/v1/chart API call + +void rrdset2json(RRDSET *st, BUFFER *wb, size_t *dimensions_count, size_t *memory_used, int skip_volatile) { + rrdset_rdlock(st); + + time_t first_entry_t = rrdset_first_entry_t_nolock(st); + time_t last_entry_t = rrdset_last_entry_t_nolock(st); + + buffer_sprintf(wb, + "\t\t{\n" + "\t\t\t\"id\": \"%s\",\n" + "\t\t\t\"name\": \"%s\",\n" + "\t\t\t\"type\": \"%s\",\n" + "\t\t\t\"family\": \"%s\",\n" + "\t\t\t\"context\": \"%s\",\n" + "\t\t\t\"title\": \"%s (%s)\",\n" + "\t\t\t\"priority\": %ld,\n" + "\t\t\t\"plugin\": \"%s\",\n" + "\t\t\t\"module\": \"%s\",\n" + "\t\t\t\"enabled\": %s,\n" + "\t\t\t\"units\": \"%s\",\n" + "\t\t\t\"data_url\": \"/api/v1/data?chart=%s\",\n" + "\t\t\t\"chart_type\": \"%s\",\n" + , st->id + , st->name + , st->type + , st->family + , st->context + , st->title, st->name + , st->priority + , st->plugin_name?st->plugin_name:"" + , st->module_name?st->module_name:"" + , rrdset_flag_check(st, RRDSET_FLAG_ENABLED)?"true":"false" + , st->units + , st->name + , rrdset_type_name(st->chart_type) + ); + + if (likely(!skip_volatile)) + buffer_sprintf(wb, + "\t\t\t\"duration\": %ld,\n" + , last_entry_t - first_entry_t + st->update_every//st->entries * st->update_every + ); + + buffer_sprintf(wb, + "\t\t\t\"first_entry\": %ld,\n" + , first_entry_t //rrdset_first_entry_t(st) + ); + + if (likely(!skip_volatile)) + buffer_sprintf(wb, + "\t\t\t\"last_entry\": %ld,\n" + , last_entry_t//rrdset_last_entry_t(st) + ); + + buffer_sprintf(wb, + "\t\t\t\"update_every\": %d,\n" + "\t\t\t\"dimensions\": {\n" + , st->update_every + ); + + unsigned long memory = st->memsize; + + size_t dimensions = 0; + RRDDIM *rd; + rrddim_foreach_read(rd, st) { + if(rrddim_flag_check(rd, RRDDIM_FLAG_HIDDEN) || rrddim_flag_check(rd, RRDDIM_FLAG_OBSOLETE)) continue; + + memory += rd->memsize; + + if (dimensions) + buffer_strcat(wb, ",\n\t\t\t\t\""); + else + buffer_strcat(wb, "\t\t\t\t\""); + buffer_strcat_jsonescape(wb, rd->id); + buffer_strcat(wb, "\": { \"name\": \""); + buffer_strcat_jsonescape(wb, rd->name); + buffer_strcat(wb, "\" }"); + + dimensions++; + } + + if(dimensions_count) *dimensions_count += dimensions; + if(memory_used) *memory_used += memory; + + buffer_sprintf(wb, "\n\t\t\t},\n\t\t\t\"chart_variables\": "); + health_api_v1_chart_custom_variables2json(st, wb); + + buffer_strcat(wb, ",\n\t\t\t\"green\": "); + buffer_rrd_value(wb, st->green); + buffer_strcat(wb, ",\n\t\t\t\"red\": "); + buffer_rrd_value(wb, st->red); + + if (likely(!skip_volatile)) { + buffer_strcat(wb, ",\n\t\t\t\"alarms\": {\n"); + size_t alarms = 0; + RRDCALC *rc; + for (rc = st->alarms; rc; rc = rc->rrdset_next) { + buffer_sprintf( + wb, + "%s" + "\t\t\t\t\"%s\": {\n" + "\t\t\t\t\t\"id\": %u,\n" + "\t\t\t\t\t\"status\": \"%s\",\n" + "\t\t\t\t\t\"units\": \"%s\",\n" + "\t\t\t\t\t\"update_every\": %d\n" + "\t\t\t\t}", + (alarms) ? ",\n" : "", rc->name, rc->id, rrdcalc_status2string(rc->status), rc->units, + rc->update_every); + + alarms++; + } + buffer_sprintf(wb, + "\n\t\t\t}" + ); + } + buffer_strcat(wb, ",\n\t\t\t\"chart_labels\": {\n"); + chart_labels2json(st, wb, 2); + buffer_strcat(wb, "\t\t\t}\n"); + + + buffer_sprintf(wb, + "\n\t\t}" + ); + + rrdset_unlock(st); +} diff --git a/web/api/formatters/rrdset2json.h b/web/api/formatters/rrdset2json.h new file mode 100644 index 0000000..697c846 --- /dev/null +++ b/web/api/formatters/rrdset2json.h @@ -0,0 +1,10 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#ifndef NETDATA_API_FORMATTER_RRDSET2JSON_H +#define NETDATA_API_FORMATTER_RRDSET2JSON_H + +#include "rrd2json.h" + +extern void rrdset2json(RRDSET *st, BUFFER *wb, size_t *dimensions_count, size_t *memory_used, int skip_volatile); + +#endif //NETDATA_API_FORMATTER_RRDSET2JSON_H diff --git a/web/api/formatters/ssv/Makefile.am b/web/api/formatters/ssv/Makefile.am new file mode 100644 index 0000000..161784b --- /dev/null +++ b/web/api/formatters/ssv/Makefile.am @@ -0,0 +1,8 @@ +# SPDX-License-Identifier: GPL-3.0-or-later + +AUTOMAKE_OPTIONS = subdir-objects +MAINTAINERCLEANFILES = $(srcdir)/Makefile.in + +dist_noinst_DATA = \ + README.md \ + $(NULL) diff --git a/web/api/formatters/ssv/README.md b/web/api/formatters/ssv/README.md new file mode 100644 index 0000000..d439949 --- /dev/null +++ b/web/api/formatters/ssv/README.md @@ -0,0 +1,59 @@ +<!-- +title: "SSV formatter" +custom_edit_url: https://github.com/netdata/netdata/edit/master/web/api/formatters/ssv/README.md +--> + +# SSV formatter + +The SSV formatter sums all dimensions in [results of database queries](/web/api/queries/README.md) +to a single value and returns a list of such values showing how it changes through time. + +It supports the following formats: + +| format | content type | description | +|:----:|:----------:|:----------| +| `ssv` | text/plain | a space separated list of values | +| `ssvcomma` | text/plain | a comma separated list of values | +| `array` | application/json | a JSON array | + +The SSV formatter respects the following API `&options=`: + +| option | supported | description | +| :----:|:-------:|:----------| +| `nonzero` | yes | to return only the dimensions that have at least a non-zero value | +| `flip` | yes | to return the numbers older to newer (the default is newer to older) | +| `percent` | yes | to replace all values with their percentage over the row total | +| `abs` | yes | to turn all values positive, before using them | +| `min2max` | yes | to return the delta from the minimum value to the maximum value (across dimensions) | + +## Examples + +Get the average system CPU utilization of the last hour, in 6 values (one every 10 minutes): + +```bash +# curl -Ss 'https://registry.my-netdata.io/api/v1/data?chart=system.cpu&format=ssv&after=-3600&points=6&group=average' +1.741352 1.6800467 1.769411 1.6761112 1.629862 1.6807968 +``` + +--- + +Get the total mysql bandwidth (in + out) for the last hour, in 6 values (one every 10 minutes): + +Netdata returns bandwidth in `kilobits`. + +```bash +# curl -Ss 'https://registry.my-netdata.io/api/v1/data?chart=mysql_local.net&format=ssvcomma&after=-3600&points=6&group=sum&options=abs' +72618.7936215,72618.778889,72618.788084,72618.9195918,72618.7760612,72618.6712421 +``` + +--- + +Get the web server max connections for the last hour, in 12 values (one every 5 minutes) +in a JSON array: + +```bash +# curl -Ss 'https://registry.my-netdata.io/api/v1/data?chart=nginx_local.connections&format=array&after=-3600&points=12&group=max' +[278,258,268,239,259,260,243,266,278,318,264,258] +``` + +[](<>) diff --git a/web/api/formatters/ssv/ssv.c b/web/api/formatters/ssv/ssv.c new file mode 100644 index 0000000..eeba028 --- /dev/null +++ b/web/api/formatters/ssv/ssv.c @@ -0,0 +1,45 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#include "ssv.h" + +void rrdr2ssv(RRDR *r, BUFFER *wb, RRDR_OPTIONS options, const char *prefix, const char *separator, const char *suffix) { + //info("RRD2SSV(): %s: BEGIN", r->st->id); + long i; + + buffer_strcat(wb, prefix); + long start = 0, end = rrdr_rows(r), step = 1; + if(!(options & RRDR_OPTION_REVERSED)) { + start = rrdr_rows(r) - 1; + end = -1; + step = -1; + } + + // for each line in the array + for(i = start; i != end ;i += step) { + int all_values_are_null = 0; + calculated_number v = rrdr2value(r, i, options, &all_values_are_null); + + if(likely(i != start)) { + if(r->min > v) r->min = v; + if(r->max < v) r->max = v; + } + else { + r->min = v; + r->max = v; + } + + if(likely(i != start)) + buffer_strcat(wb, separator); + + if(all_values_are_null) { + if(options & RRDR_OPTION_NULL2ZERO) + buffer_strcat(wb, "0"); + else + buffer_strcat(wb, "null"); + } + else + buffer_rrd_value(wb, v); + } + buffer_strcat(wb, suffix); + //info("RRD2SSV(): %s: END", r->st->id); +} diff --git a/web/api/formatters/ssv/ssv.h b/web/api/formatters/ssv/ssv.h new file mode 100644 index 0000000..6963dcf --- /dev/null +++ b/web/api/formatters/ssv/ssv.h @@ -0,0 +1,10 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#ifndef NETDATA_API_FORMATTER_SSV_H +#define NETDATA_API_FORMATTER_SSV_H + +#include "../rrd2json.h" + +extern void rrdr2ssv(RRDR *r, BUFFER *wb, RRDR_OPTIONS options, const char *prefix, const char *separator, const char *suffix); + +#endif //NETDATA_API_FORMATTER_SSV_H diff --git a/web/api/formatters/value/Makefile.am b/web/api/formatters/value/Makefile.am new file mode 100644 index 0000000..161784b --- /dev/null +++ b/web/api/formatters/value/Makefile.am @@ -0,0 +1,8 @@ +# SPDX-License-Identifier: GPL-3.0-or-later + +AUTOMAKE_OPTIONS = subdir-objects +MAINTAINERCLEANFILES = $(srcdir)/Makefile.in + +dist_noinst_DATA = \ + README.md \ + $(NULL) diff --git a/web/api/formatters/value/README.md b/web/api/formatters/value/README.md new file mode 100644 index 0000000..21c9370 --- /dev/null +++ b/web/api/formatters/value/README.md @@ -0,0 +1,24 @@ +<!-- +title: "Value formatter" +custom_edit_url: https://github.com/netdata/netdata/edit/master/web/api/formatters/value/README.md +--> + +# Value formatter + +The Value formatter presents [results of database queries](/web/api/queries/README.md) as a single value. + +To calculate the single value to be returned, it sums the values of all dimensions. + +The Value formatter respects the following API `&options=`: + +| option | supported | description | +|:----: |:-------: |:---------- | +| `percent` | yes | to replace all values with their percentage over the row total| +| `abs` | yes | to turn all values positive, before using them | +| `min2max` | yes | to return the delta from the minimum value to the maximum value (across dimensions)| + +The Value formatter is not exposed by the API by itself. +Instead it is used by the [`ssv`](/web/api/formatters/ssv/README.md) formatter +and [health monitoring queries](/health/README.md). + +[](<>) diff --git a/web/api/formatters/value/value.c b/web/api/formatters/value/value.c new file mode 100644 index 0000000..aea6c16 --- /dev/null +++ b/web/api/formatters/value/value.c @@ -0,0 +1,94 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#include "value.h" + + +inline calculated_number rrdr2value(RRDR *r, long i, RRDR_OPTIONS options, int *all_values_are_null) { + rrdset_check_rdlock(r->st); + + long c; + RRDDIM *d; + + calculated_number *cn = &r->v[ i * r->d ]; + RRDR_VALUE_FLAGS *co = &r->o[ i * r->d ]; + + calculated_number sum = 0, min = 0, max = 0, v; + int all_null = 1, init = 1; + + calculated_number total = 1; + int set_min_max = 0; + if(unlikely(options & RRDR_OPTION_PERCENTAGE)) { + total = 0; + for(c = 0, d = r->st->dimensions; d && c < r->d ;c++, d = d->next) { + calculated_number n = cn[c]; + + if(likely((options & RRDR_OPTION_ABSOLUTE) && n < 0)) + n = -n; + + total += n; + } + // prevent a division by zero + if(total == 0) total = 1; + set_min_max = 1; + } + + // for each dimension + for(c = 0, d = r->st->dimensions; d && c < r->d ;c++, d = d->next) { + if(unlikely(r->od[c] & RRDR_DIMENSION_HIDDEN)) continue; + if(unlikely((options & RRDR_OPTION_NONZERO) && !(r->od[c] & RRDR_DIMENSION_NONZERO))) continue; + + calculated_number n = cn[c]; + + if(likely((options & RRDR_OPTION_ABSOLUTE) && n < 0)) + n = -n; + + if(unlikely(options & RRDR_OPTION_PERCENTAGE)) { + n = n * 100 / total; + + if(unlikely(set_min_max)) { + r->min = r->max = n; + set_min_max = 0; + } + + if(n < r->min) r->min = n; + if(n > r->max) r->max = n; + } + + if(unlikely(init)) { + if(n > 0) { + min = 0; + max = n; + } + else { + min = n; + max = 0; + } + init = 0; + } + + if(likely(!(co[c] & RRDR_VALUE_EMPTY))) { + all_null = 0; + sum += n; + } + + if(n < min) min = n; + if(n > max) max = n; + } + + if(unlikely(all_null)) { + if(likely(all_values_are_null)) + *all_values_are_null = 1; + return 0; + } + else { + if(likely(all_values_are_null)) + *all_values_are_null = 0; + } + + if(options & RRDR_OPTION_MIN2MAX) + v = max - min; + else + v = sum; + + return v; +} diff --git a/web/api/formatters/value/value.h b/web/api/formatters/value/value.h new file mode 100644 index 0000000..d9e981f --- /dev/null +++ b/web/api/formatters/value/value.h @@ -0,0 +1,10 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#ifndef NETDATA_API_FORMATTER_VALUE_H +#define NETDATA_API_FORMATTER_VALUE_H + +#include "../rrd2json.h" + +extern calculated_number rrdr2value(RRDR *r, long i, RRDR_OPTIONS options, int *all_values_are_null); + +#endif //NETDATA_API_FORMATTER_VALUE_H diff --git a/web/api/health/Makefile.am b/web/api/health/Makefile.am new file mode 100644 index 0000000..161784b --- /dev/null +++ b/web/api/health/Makefile.am @@ -0,0 +1,8 @@ +# SPDX-License-Identifier: GPL-3.0-or-later + +AUTOMAKE_OPTIONS = subdir-objects +MAINTAINERCLEANFILES = $(srcdir)/Makefile.in + +dist_noinst_DATA = \ + README.md \ + $(NULL) diff --git a/web/api/health/README.md b/web/api/health/README.md new file mode 100644 index 0000000..b6e8b5c --- /dev/null +++ b/web/api/health/README.md @@ -0,0 +1,225 @@ +<!-- +title: "Health API Calls" +date: 2020-04-27 +custom_edit_url: https://github.com/netdata/netdata/edit/master/web/api/health/README.md +--> + +# Health API Calls + +## Health Read API + +### Enabled Alarms + +Netdata enables alarms on demand, i.e. when the chart they should be linked to starts collecting data. So, although many +more alarms are configured, only the useful ones are enabled. + +To get the list of all enabled alarms, open your browser and navigate to `http://NODE:19999/api/v1/alarms?all`, +replacing `NODE` with the IP address or hostname for your Agent dashboard. + +### Raised Alarms + +This API call will return the alarms currently in WARNING or CRITICAL state. + +`http://NODE:19999/api/v1/alarms` + +### Event Log + +The size of the alarm log is configured in `netdata.conf`. There are 2 settings: the rotation of the alarm log file and the in memory size of the alarm log. + +``` +[health] + in memory max health log entries = 1000 + rotate log every lines = 2000 +``` + +The API call retrieves all entries of the alarm log: + +`http://NODE:19999/api/v1/alarm_log` + +### Alarm Log Incremental Updates + +`http://NODE:19999/api/v1/alarm_log?after=UNIQUEID` + +The above returns all the events in the alarm log that occurred after UNIQUEID (you poll it once without `after=`, remember the last UNIQUEID of the returned set, which you give back to get incrementally the next events). + +### Alarm badges + +The following will return an SVG badge of the alarm named `NAME`, attached to the chart named `CHART`. + +`http://NODE:19999/api/v1/badge.svg?alarm=NAME&chart=CHART` + +## Health Management API + +Netdata v1.12 and beyond provides a command API to control health checks and notifications at runtime. The feature is especially useful for maintenance periods, during which you receive meaningless alarms. +From Netdata v1.16.0 and beyond, the configuration controlled via the API commands is [persisted across Netdata restarts](#persistence). + +Specifically, the API allows you to: + +- Disable health checks completely. Alarm conditions will not be evaluated at all and no entries will be added to the alarm log. +- Silence alarm notifications. Alarm conditions will be evaluated, the alarms will appear in the log and the Netdata UI will show the alarms as active, but no notifications will be sent. +- Disable or Silence specific alarms that match selectors on alarm/template name, chart, context, host and family. + +The API is available by default, but it is protected by an `api authorization token` that is stored in the file you will see in the following entry of `http://NODE:19999/netdata.conf`: + +``` +[registry] + # netdata management api key file = /var/lib/netdata/netdata.api.key +``` + +You can access the API via GET requests, by adding the bearer token to an `Authorization` http header, like this: + +``` +curl "http://NODE:19999/api/v1/manage/health?cmd=RESET" -H "X-Auth-Token: Mytoken" +``` + +By default access to the health management API is only allowed from `localhost`. Accessing the API from anything else will return a 403 error with the message `You are not allowed to access this resource.`. You can change permissions by editing the `allow management from` variable in `netdata.conf` within the [web] section. See [web server access lists](/web/server/README.md#access-lists) for more information. + +The command `RESET` just returns Netdata to the default operation, with all health checks and notifications enabled. +If you've configured and entered your token correctly, you should see the plain text response `All health checks and notifications are enabled`. + +### Disable or silence all alarms + +If all you need is temporarily disable all health checks, then you issue the following before your maintenance period starts: + +```sh +curl "http://NODE:19999/api/v1/manage/health?cmd=DISABLE ALL" -H "X-Auth-Token: Mytoken" +``` + +The effect of disabling health checks is that the alarm criteria are not evaluated at all and nothing is written in the alarm log. +If you want the health checks to be running but to not receive any notifications during your maintenance period, you can instead use this: + +```sh +curl "http://NODE:19999/api/v1/manage/health?cmd=SILENCE ALL" -H "X-Auth-Token: Mytoken" +``` + +Alarms may then still be raised and logged in Netdata, so you'll be able to see them via the UI. + +Regardless of the option you choose, at the end of your maintenance period you revert to the normal state via the RESET command. + +```sh + curl "http://NODE:19999/api/v1/manage/health?cmd=RESET" -H "X-Auth-Token: Mytoken" +``` + +### Disable or silence specific alarms + +If you do not wish to disable/silence all alarms, then the `DISABLE ALL` and `SILENCE ALL` commands can't be used. +Instead, the following commands expect that one or more alarm selectors will be added, so that only alarms that match the selectors are disabled or silenced. + +- `DISABLE` : Set the mode to disable health checks. +- `SILENCE` : Set the mode to silence notifications. + +You will normally put one of these commands in the same request with your first alarm selector, but it's possible to issue them separately as well. +You will get a warning in the response, if a selector was added without a SILENCE/DISABLE command, or vice versa. + +Each request can specify a single alarm `selector`, with one or more `selection criteria`. +A single alarm will match a `selector` if all selection criteria match the alarm. +You can add as many selectors as you like. +In essence, the rule is: IF (alarm matches all the criteria in selector1 OR all the criteria in selector2 OR ...) THEN apply the DISABLE or SILENCE command. + +To clear all selectors and reset the mode to default, use the `RESET` command. + +The following example silences notifications for all the alarms with context=load: + +``` +curl "http://NODE:19999/api/v1/manage/health?cmd=SILENCE&context=load" -H "X-Auth-Token: Mytoken" +``` + +#### Selection criteria + +The `selection criteria` are key/value pairs, in the format `key : value`, where value is a Netdata [simple pattern](/libnetdata/simple_pattern/README.md). This means that you can create very powerful selectors (you will rarely need more than one or two). + +The accepted keys for the `selection criteria` are the following: + +- `alarm` : The expression provided will match both `alarm` and `template` names. +- `chart` : Chart ids/names, as shown on the dashboard. These will match the `on` entry of a configured `alarm`. +- `context` : Chart context, as shown on the dashboard. These will match the `on` entry of a configured `template`. +- `hosts` : The hostnames that will need to match. +- `families` : The alarm families. + +You can add any of the selection criteria you need on the request, to ensure that only the alarms you are interested in are matched and disabled/silenced. e.g. there is no reason to add `hosts: *`, if you want the criteria to be applied to alarms for all hosts. + +Example 1: Disable all health checks for context = `random` + +``` +http://NODE:19999/api/v1/manage/health?cmd=DISABLE&context=random +``` + +Example 2: Silence all alarms and templates with name starting with `out_of` on host `myhost` + +``` +http://NODE:19999/api/v1/manage/health?cmd=SILENCE&alarm=out_of*&hosts=myhost +``` + +Example 2.2: Add one more selector, to also silence alarms for cpu1 and cpu2 + +``` +http://NODE:19999/api/v1/manage/health?families=cpu1 cpu2 +``` + +### List silencers + +The command `LIST` was added in Netdata v1.16.0 and returns a JSON with the current status of the silencers. + +``` + curl "http://NODE:19999/api/v1/manage/health?cmd=LIST" -H "X-Auth-Token: Mytoken" +``` + +As an example, the following response shows that we have two silencers configured, one for an alarm called `samplealarm` and one for alarms with context `random` on host `myhost` + +``` +json +{ + "all": false, + "type": "SILENCE", + "silencers": [ + { + "alarm": "samplealarm" + }, + { + "context": "random", + "hosts": "myhost" + } + ] +} +``` + +The response below shows that we have disabled all health checks. + +``` +json +{ + "all": true, + "type": "DISABLE", + "silencers": [] +} +``` + +### Responses + +- "Auth Error" : Token authentication failed +- "All alarm notifications are silenced" : Successful response to cmd=SILENCE ALL +- "All health checks are disabled" : Successful response to cmd=DISABLE ALL +- "All health checks and notifications are enabled" : Successful response to cmd=RESET +- "Health checks disabled for alarms matching the selectors" : Added to the response for a cmd=DISABLE +- "Alarm notifications silenced for alarms matching the selectors" : Added to the response for a cmd=SILENCE +- "Alarm selector added" : Added to the response when a new selector is added +- "Invalid key. Ignoring it." : Wrong name of a parameter. Added to the response and ignored. +- "WARNING: Added alarm selector to silence/disable alarms without a SILENCE or DISABLE command." : Added to the response if a selector is added without a selector-specific command. +- "WARNING: SILENCE or DISABLE command is ineffective without defining any alarm selectors." : Added to the response if a selector-specific command is issued without a selector. + +### Persistence + +From Netdata v1.16.0 and beyond, the silencers configuration is persisted to disk and loaded when Netdata starts. +The JSON string returned by the [LIST command](#list-silencers) is automatically saved to the `silencers file`, every time a command alters the silencers configuration. +The file's location is configurable in `netdata.conf`. The default is shown below: + +``` +[health] + # silencers file = /var/lib/netdata/health.silencers.json +``` + +### Further reading + +The test script under [tests/health_mgmtapi](/tests/health_mgmtapi/README.md) contains a series of tests that you can either run or read through to understand the various calls and responses better. + +[](<>) diff --git a/web/api/health/health_cmdapi.c b/web/api/health/health_cmdapi.c new file mode 100644 index 0000000..4dd85e6 --- /dev/null +++ b/web/api/health/health_cmdapi.c @@ -0,0 +1,205 @@ +// +// Created by Christopher on 11/12/18. +// + +#include "health_cmdapi.h" + +/** + * Free Silencers + * + * Clean the silencer structure + * + * @param t is the structure that will be cleaned. + */ +void free_silencers(SILENCER *t) { + if (!t) return; + if (t->next) free_silencers(t->next); + debug(D_HEALTH, "HEALTH command API: Freeing silencer %s:%s:%s:%s:%s", t->alarms, + t->charts, t->contexts, t->hosts, t->families); + simple_pattern_free(t->alarms_pattern); + simple_pattern_free(t->charts_pattern); + simple_pattern_free(t->contexts_pattern); + simple_pattern_free(t->hosts_pattern); + simple_pattern_free(t->families_pattern); + freez(t->alarms); + freez(t->charts); + freez(t->contexts); + freez(t->hosts); + freez(t->families); + freez(t); + return; +} + +/** + * Silencers to JSON Entry + * + * Fill the buffer with the other values given. + * + * @param wb a pointer to the output buffer + * @param var the json variable + * @param val the json value + * @param hasprev has it a previous value? + * + * @return + */ +int health_silencers2json_entry(BUFFER *wb, char* var, char* val, int hasprev) { + if (val) { + buffer_sprintf(wb, "%s\n\t\t\t\"%s\": \"%s\"", (hasprev)?",":"", var, val); + return 1; + } else { + return hasprev; + } +} + +/** + * Silencer to JSON + * + * Write the silencer values using JSON format inside a buffer. + * + * @param wb is the buffer to write the silencers. + */ +void health_silencers2json(BUFFER *wb) { + buffer_sprintf(wb, "{\n\t\"all\": %s," + "\n\t\"type\": \"%s\"," + "\n\t\"silencers\": [", + (silencers->all_alarms)?"true":"false", + (silencers->stype == STYPE_NONE)?"None":((silencers->stype == STYPE_DISABLE_ALARMS)?"DISABLE":"SILENCE")); + + SILENCER *silencer; + int i = 0, j = 0; + for(silencer = silencers->silencers; silencer ; silencer = silencer->next) { + if(likely(i)) buffer_strcat(wb, ","); + buffer_strcat(wb, "\n\t\t{"); + j=health_silencers2json_entry(wb, HEALTH_ALARM_KEY, silencer->alarms, j); + j=health_silencers2json_entry(wb, HEALTH_CHART_KEY, silencer->charts, j); + j=health_silencers2json_entry(wb, HEALTH_CONTEXT_KEY, silencer->contexts, j); + j=health_silencers2json_entry(wb, HEALTH_HOST_KEY, silencer->hosts, j); + health_silencers2json_entry(wb, HEALTH_FAMILIES_KEY, silencer->families, j); + j=0; + buffer_strcat(wb, "\n\t\t}"); + i++; + } + if(likely(i)) buffer_strcat(wb, "\n\t"); + buffer_strcat(wb, "]\n}\n"); +} + +/** + * Silencer to FILE + * + * Write the sliencer buffer to a file. + * @param wb + */ +void health_silencers2file(BUFFER *wb) { + if (wb->len == 0) return; + + FILE *fd = fopen(silencers_filename, "wb"); + if(fd) { + size_t written = (size_t)fprintf(fd, "%s", wb->buffer) ; + if (written == wb->len ) { + info("Silencer changes written to %s", silencers_filename); + } + fclose(fd); + return; + } + error("Silencer changes could not be written to %s. Error %s", silencers_filename, strerror(errno)); +} + +/** + * Request V1 MGMT Health + * + * Function called by api to management the health. + * + * @param host main structure with client information! + * @param w is the structure with all information of the client request. + * @param url is the url that netdata is working + * + * @return It returns 200 on success and another code otherwise. + */ +int web_client_api_request_v1_mgmt_health(RRDHOST *host, struct web_client *w, char *url) { + int ret; + (void) host; + + BUFFER *wb = w->response.data; + buffer_flush(wb); + wb->contenttype = CT_TEXT_PLAIN; + + buffer_flush(w->response.data); + + //Local instance of the silencer + SILENCER *silencer = NULL; + int config_changed = 1; + + if (!w->auth_bearer_token) { + buffer_strcat(wb, HEALTH_CMDAPI_MSG_AUTHERROR); + ret = HTTP_RESP_FORBIDDEN; + } else { + debug(D_HEALTH, "HEALTH command API: Comparing secret '%s' to '%s'", w->auth_bearer_token, api_secret); + if (strcmp(w->auth_bearer_token, api_secret)) { + buffer_strcat(wb, HEALTH_CMDAPI_MSG_AUTHERROR); + ret = HTTP_RESP_FORBIDDEN; + } else { + while (url) { + char *value = mystrsep(&url, "&"); + if (!value || !*value) continue; + + char *key = mystrsep(&value, "="); + if (!key || !*key) continue; + if (!value || !*value) continue; + + debug(D_WEB_CLIENT, "%llu: API v1 health query param '%s' with value '%s'", w->id, key, value); + + // name and value are now the parameters + if (!strcmp(key, "cmd")) { + if (!strcmp(value, HEALTH_CMDAPI_CMD_SILENCEALL)) { + silencers->all_alarms = 1; + silencers->stype = STYPE_SILENCE_NOTIFICATIONS; + buffer_strcat(wb, HEALTH_CMDAPI_MSG_SILENCEALL); + } else if (!strcmp(value, HEALTH_CMDAPI_CMD_DISABLEALL)) { + silencers->all_alarms = 1; + silencers->stype = STYPE_DISABLE_ALARMS; + buffer_strcat(wb, HEALTH_CMDAPI_MSG_DISABLEALL); + } else if (!strcmp(value, HEALTH_CMDAPI_CMD_SILENCE)) { + silencers->stype = STYPE_SILENCE_NOTIFICATIONS; + buffer_strcat(wb, HEALTH_CMDAPI_MSG_SILENCE); + } else if (!strcmp(value, HEALTH_CMDAPI_CMD_DISABLE)) { + silencers->stype = STYPE_DISABLE_ALARMS; + buffer_strcat(wb, HEALTH_CMDAPI_MSG_DISABLE); + } else if (!strcmp(value, HEALTH_CMDAPI_CMD_RESET)) { + silencers->all_alarms = 0; + silencers->stype = STYPE_NONE; + free_silencers(silencers->silencers); + silencers->silencers = NULL; + buffer_strcat(wb, HEALTH_CMDAPI_MSG_RESET); + } else if (!strcmp(value, HEALTH_CMDAPI_CMD_LIST)) { + w->response.data->contenttype = CT_APPLICATION_JSON; + health_silencers2json(wb); + config_changed=0; + } + } else { + silencer = health_silencers_addparam(silencer, key, value); + } + } + + if (likely(silencer)) { + health_silencers_add(silencer); + buffer_strcat(wb, HEALTH_CMDAPI_MSG_ADDED); + if (silencers->stype == STYPE_NONE) { + buffer_strcat(wb, HEALTH_CMDAPI_MSG_STYPEWARNING); + } + } + if (unlikely(silencers->stype != STYPE_NONE && !silencers->all_alarms && !silencers->silencers)) { + buffer_strcat(wb, HEALTH_CMDAPI_MSG_NOSELECTORWARNING); + } + ret = HTTP_RESP_OK; + } + } + w->response.data = wb; + buffer_no_cacheable(w->response.data); + if (ret == HTTP_RESP_OK && config_changed) { + BUFFER *jsonb = buffer_create(200); + health_silencers2json(jsonb); + health_silencers2file(jsonb); + } + + return ret; +} diff --git a/web/api/health/health_cmdapi.h b/web/api/health/health_cmdapi.h new file mode 100644 index 0000000..d8ec6aa --- /dev/null +++ b/web/api/health/health_cmdapi.h @@ -0,0 +1,31 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#ifndef NETDATA_WEB_HEALTH_SVG_H +#define NETDATA_WEB_HEALTH_SVG_H 1 + +#include "libnetdata/libnetdata.h" +#include "web/server/web_client.h" +#include "health/health.h" + +#define HEALTH_CMDAPI_CMD_SILENCEALL "SILENCE ALL" +#define HEALTH_CMDAPI_CMD_DISABLEALL "DISABLE ALL" +#define HEALTH_CMDAPI_CMD_SILENCE "SILENCE" +#define HEALTH_CMDAPI_CMD_DISABLE "DISABLE" +#define HEALTH_CMDAPI_CMD_RESET "RESET" +#define HEALTH_CMDAPI_CMD_LIST "LIST" + +#define HEALTH_CMDAPI_MSG_AUTHERROR "Auth Error\n" +#define HEALTH_CMDAPI_MSG_SILENCEALL "All alarm notifications are silenced\n" +#define HEALTH_CMDAPI_MSG_DISABLEALL "All health checks are disabled\n" +#define HEALTH_CMDAPI_MSG_RESET "All health checks and notifications are enabled\n" +#define HEALTH_CMDAPI_MSG_DISABLE "Health checks disabled for alarms matching the selectors\n" +#define HEALTH_CMDAPI_MSG_SILENCE "Alarm notifications silenced for alarms matching the selectors\n" +#define HEALTH_CMDAPI_MSG_ADDED "Alarm selector added\n" +#define HEALTH_CMDAPI_MSG_STYPEWARNING "WARNING: Added alarm selector to silence/disable alarms without a SILENCE or DISABLE command.\n" +#define HEALTH_CMDAPI_MSG_NOSELECTORWARNING "WARNING: SILENCE or DISABLE command is ineffective without defining any alarm selectors.\n" + +extern int web_client_api_request_v1_mgmt_health(RRDHOST *host, struct web_client *w, char *url); + +#include "web/api/web_api_v1.h" + +#endif /* NETDATA_WEB_HEALTH_SVG_H */ diff --git a/web/api/netdata-swagger.json b/web/api/netdata-swagger.json new file mode 100644 index 0000000..ed2555f --- /dev/null +++ b/web/api/netdata-swagger.json @@ -0,0 +1,2065 @@ +{ + "openapi": "3.0.0", + "info": { + "title": "NetData API", + "description": "Real-time performance and health monitoring.", + "version": "1.11.1_rolling" + }, + "paths": { + "/info": { + "get": { + "summary": "Get netdata basic information", + "description": "The info endpoint returns basic information about netdata. It provides:\n* netdata version\n* netdata unique id\n* list of hosts mirrored (includes itself)\n* Operating System, Virtualization, K8s nodes and Container technology information\n* List of active collector plugins and modules\n* number of alarms in the host\n * number of alarms in normal state\n * number of alarms in warning state\n * number of alarms in critical state\n", + "responses": { + "200": { + "description": "netdata basic information.", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/info" + } + } + } + }, + "503": { + "description": "netdata daemon not ready (used for health checks)." + } + } + } + }, + "/charts": { + "get": { + "summary": "Get a list of all charts available at the server", + "description": "The charts endpoint returns a summary about all charts stored in the netdata server.", + "responses": { + "200": { + "description": "An array of charts.", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/chart_summary" + } + } + } + } + } + } + }, + "/chart": { + "get": { + "summary": "Get info about a specific chart", + "description": "The Chart endpoint returns detailed information about a chart.", + "parameters": [ + { + "name": "chart", + "in": "query", + "description": "The id of the chart as returned by the /charts call.", + "required": true, + "schema": { + "type": "string", + "format": "as returned by /charts", + "default": "system.cpu" + } + } + ], + "responses": { + "200": { + "description": "A javascript object with detailed information about the chart.", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/chart" + } + } + } + }, + "400": { + "description": "No chart id was supplied in the request." + }, + "404": { + "description": "No chart with the given id is found." + } + } + } + }, + "/alarm_variables": { + "get": { + "summary": "List variables available to configure alarms for a chart", + "description": "Returns the basic information of a chart and all the variables that can be used in alarm and template health configurations for the particular chart or family.", + "parameters": [ + { + "name": "chart", + "in": "query", + "description": "The id of the chart as returned by the /charts call.", + "required": true, + "schema": { + "type": "string", + "format": "as returned by /charts", + "default": "system.cpu" + } + } + ], + "responses": { + "200": { + "description": "A javascript object with information about the chart and the available variables.", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/alarm_variables" + } + } + } + }, + "400": { + "description": "Bad request - the body will include a message stating what is wrong." + }, + "404": { + "description": "No chart with the given id is found." + }, + "500": { + "description": "Internal server error. This usually means the server is out of memory." + } + } + } + }, + "/data": { + "get": { + "summary": "Get collected data for a specific chart", + "description": "The data endpoint returns data stored in the round robin database of a chart.", + "parameters": [ + { + "name": "chart", + "in": "query", + "description": "The id of the chart as returned by the /charts call. Note chart or context must be specified", + "required": false, + "allowEmptyValue": false, + "schema": { + "type": "string", + "format": "as returned by /charts", + "default": "system.cpu" + } + }, + { + "name": "context", + "in": "query", + "description": "The context of the chart as returned by the /charts call. Note chart or context must be specified", + "required": false, + "allowEmptyValue": false, + "schema": { + "type": "string", + "format": "as returned by /charts" + } + }, + { + "name": "dimension", + "in": "query", + "description": "Zero, one or more dimension ids or names, as returned by the /chart call, separated with comma or pipe. Netdata simple patterns are supported.", + "required": false, + "allowEmptyValue": false, + "schema": { + "type": "array", + "items": { + "type": "string", + "format": "as returned by /charts" + } + } + }, + { + "name": "after", + "in": "query", + "description": "This parameter can either be an absolute timestamp specifying the starting point of the data to be returned, or a relative number of seconds (negative, relative to parameter: before). Netdata will assume it is a relative number if it is less that 3 years (in seconds). If not specified the default is -600 seconds. Netdata will adapt this parameter to the boundaries of the round robin database unless the allow_past option is specified.", + "required": true, + "allowEmptyValue": false, + "schema": { + "type": "number", + "format": "integer", + "default": -600 + } + }, + { + "name": "before", + "in": "query", + "description": "This parameter can either be an absolute timestamp specifying the ending point of the data to be returned, or a relative number of seconds (negative), relative to the last collected timestamp. Netdata will assume it is a relative number if it is less than 3 years (in seconds). Netdata will adapt this parameter to the boundaries of the round robin database. The default is zero (i.e. the timestamp of the last value collected).", + "required": false, + "schema": { + "type": "number", + "format": "integer", + "default": 0 + } + }, + { + "name": "points", + "in": "query", + "description": "The number of points to be returned. If not given, or it is <= 0, or it is bigger than the points stored in the round robin database for this chart for the given duration, all the available collected values for the given duration will be returned.", + "required": true, + "allowEmptyValue": false, + "schema": { + "type": "number", + "format": "integer", + "default": 20 + } + }, + { + "name": "group", + "in": "query", + "description": "The grouping method. If multiple collected values are to be grouped in order to return fewer points, this parameters defines the method of grouping. methods supported \"min\", \"max\", \"average\", \"sum\", \"incremental-sum\". \"max\" is actually calculated on the absolute value collected (so it works for both positive and negative dimesions to return the most extreme value in either direction).", + "required": true, + "allowEmptyValue": false, + "schema": { + "type": "string", + "enum": [ + "min", + "max", + "average", + "median", + "stddev", + "sum", + "incremental-sum" + ], + "default": "average" + } + }, + { + "name": "gtime", + "in": "query", + "description": "The grouping number of seconds. This is used in conjunction with group=average to change the units of metrics (ie when the data is per-second, setting gtime=60 will turn them to per-minute).", + "required": false, + "allowEmptyValue": false, + "schema": { + "type": "number", + "format": "integer", + "default": 0 + } + }, + { + "name": "format", + "in": "query", + "description": "The format of the data to be returned.", + "required": true, + "allowEmptyValue": false, + "schema": { + "type": "string", + "enum": [ + "json", + "jsonp", + "csv", + "tsv", + "tsv-excel", + "ssv", + "ssvcomma", + "datatable", + "datasource", + "html", + "markdown", + "array", + "csvjsonarray" + ], + "default": "json" + } + }, + { + "name": "options", + "in": "query", + "description": "Options that affect data generation.", + "required": false, + "allowEmptyValue": false, + "schema": { + "type": "array", + "items": { + "type": "string", + "enum": [ + "nonzero", + "flip", + "jsonwrap", + "min2max", + "seconds", + "milliseconds", + "abs", + "absolute", + "absolute-sum", + "null2zero", + "objectrows", + "google_json", + "percentage", + "unaligned", + "match-ids", + "match-names", + "showcustomvars", + "allow_past" + ] + }, + "default": [ + "seconds", + "jsonwrap" + ] + } + }, + { + "name": "callback", + "in": "query", + "description": "For JSONP responses, the callback function name.", + "required": false, + "allowEmptyValue": true, + "schema": { + "type": "string" + } + }, + { + "name": "filename", + "in": "query", + "description": "Add Content-Disposition: attachment; filename= header to the response, that will instruct the browser to save the response with the given filename.", + "required": false, + "allowEmptyValue": true, + "schema": { + "type": "string" + } + }, + { + "name": "tqx", + "in": "query", + "description": "[Google Visualization API](https://developers.google.com/chart/interactive/docs/dev/implementing_data_source?hl=en) formatted parameter.", + "required": false, + "allowEmptyValue": true, + "schema": { + "type": "string" + } + } + ], + "responses": { + "200": { + "description": "The call was successful. The response includes the data in the format requested. Swagger2.0 does not process the discriminator field to show polymorphism. The response will be one of the sub-types of the data-schema according to the chosen format, e.g. json -> data_json.", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/data" + } + } + } + }, + "400": { + "description": "Bad request - the body will include a message stating what is wrong." + }, + "404": { + "description": "Chart or context is not found. The supplied chart or context will be reported." + }, + "500": { + "description": "Internal server error. This usually means the server is out of memory." + } + } + } + }, + "/badge.svg": { + "get": { + "summary": "Generate a badge in form of SVG image for a chart (or dimension)", + "description": "Successful responses are SVG images.", + "parameters": [ + { + "name": "chart", + "in": "query", + "description": "The id of the chart as returned by the /charts call.", + "required": true, + "allowEmptyValue": false, + "schema": { + "type": "string", + "format": "as returned by /charts", + "default": "system.cpu" + } + }, + { + "name": "alarm", + "in": "query", + "description": "The name of an alarm linked to the chart.", + "required": false, + "allowEmptyValue": true, + "schema": { + "type": "string", + "format": "any text" + } + }, + { + "name": "dimension", + "in": "query", + "description": "Zero, one or more dimension ids, as returned by the /chart call.", + "required": false, + "allowEmptyValue": false, + "schema": { + "type": "array", + "items": { + "type": "string", + "format": "as returned by /charts" + } + } + }, + { + "name": "after", + "in": "query", + "description": "This parameter can either be an absolute timestamp specifying the starting point of the data to be returned, or a relative number of seconds, to the last collected timestamp. Netdata will assume it is a relative number if it is smaller than the duration of the round robin database for this chart. So, if the round robin database is 3600 seconds, any value from -3600 to 3600 will trigger relative arithmetics. Netdata will adapt this parameter to the boundaries of the round robin database.", + "required": true, + "allowEmptyValue": false, + "schema": { + "type": "number", + "format": "integer", + "default": -600 + } + }, + { + "name": "before", + "in": "query", + "description": "This parameter can either be an absolute timestamp specifying the ending point of the data to be returned, or a relative number of seconds, to the last collected timestamp. Netdata will assume it is a relative number if it is smaller than the duration of the round robin database for this chart. So, if the round robin database is 3600 seconds, any value from -3600 to 3600 will trigger relative arithmetics. Netdata will adapt this parameter to the boundaries of the round robin database.", + "required": false, + "schema": { + "type": "number", + "format": "integer", + "default": 0 + } + }, + { + "name": "group", + "in": "query", + "description": "The grouping method. If multiple collected values are to be grouped in order to return fewer points, this parameters defines the method of grouping. methods are supported \"min\", \"max\", \"average\", \"sum\", \"incremental-sum\". \"max\" is actually calculated on the absolute value collected (so it works for both positive and negative dimesions to return the most extreme value in either direction).", + "required": true, + "allowEmptyValue": false, + "schema": { + "type": "string", + "enum": [ + "min", + "max", + "average", + "median", + "stddev", + "sum", + "incremental-sum" + ], + "default": "average" + } + }, + { + "name": "options", + "in": "query", + "description": "Options that affect data generation.", + "required": false, + "allowEmptyValue": true, + "schema": { + "type": "array", + "items": { + "type": "string", + "enum": [ + "abs", + "absolute", + "display-absolute", + "absolute-sum", + "null2zero", + "percentage", + "unaligned" + ] + }, + "default": [ + "absolute" + ] + } + }, + { + "name": "label", + "in": "query", + "description": "A text to be used as the label.", + "required": false, + "allowEmptyValue": true, + "schema": { + "type": "string", + "format": "any text" + } + }, + { + "name": "units", + "in": "query", + "description": "A text to be used as the units.", + "required": false, + "allowEmptyValue": true, + "schema": { + "type": "string", + "format": "any text" + } + }, + { + "name": "label_color", + "in": "query", + "description": "A color to be used for the background of the label side(left side) of the badge. One of predefined colors or specific color in hex `RGB` or `RRGGBB` format (without preceding `#` character). If value wrong or not given default color will be used.", + "required": false, + "allowEmptyValue": true, + "schema": { + "oneOf": [ + { + "type": "string", + "enum": [ + "green", + "brightgreen", + "yellow", + "yellowgreen", + "orange", + "red", + "blue", + "grey", + "gray", + "lightgrey", + "lightgray" + ] + }, + { + "type": "string", + "format": "^([0-9a-fA-F]{3}|[0-9a-fA-F]{6})$" + } + ] + } + }, + { + "name": "value_color", + "in": "query", + "description": "A color to be used for the background of the value *(right)* part of badge. You can set multiple using a pipe with a condition each, like this: `color<value|color:null` The following operators are supported: >, <, >=, <=, =, :null (to check if no value exists). Each color can be specified in same manner as for `label_color` parameter. Currently only integers are suported as values.", + "required": false, + "allowEmptyValue": true, + "schema": { + "type": "string", + "format": "any text" + } + }, + { + "name": "text_color_lbl", + "in": "query", + "description": "Font color for label *(left)* part of the badge. One of predefined colors or as HTML hexadecimal color without preceeding `#` character. Formats allowed `RGB` or `RRGGBB`. If no or wrong value given default color will be used.", + "required": false, + "allowEmptyValue": true, + "schema": { + "oneOf": [ + { + "type": "string", + "enum": [ + "green", + "brightgreen", + "yellow", + "yellowgreen", + "orange", + "red", + "blue", + "grey", + "gray", + "lightgrey", + "lightgray" + ] + }, + { + "type": "string", + "format": "^([0-9a-fA-F]{3}|[0-9a-fA-F]{6})$" + } + ] + } + }, + { + "name": "text_color_val", + "in": "query", + "description": "Font color for value *(right)* part of the badge. One of predefined colors or as HTML hexadecimal color without preceeding `#` character. Formats allowed `RGB` or `RRGGBB`. If no or wrong value given default color will be used.", + "required": false, + "allowEmptyValue": true, + "schema": { + "oneOf": [ + { + "type": "string", + "enum": [ + "green", + "brightgreen", + "yellow", + "yellowgreen", + "orange", + "red", + "blue", + "grey", + "gray", + "lightgrey", + "lightgray" + ] + }, + { + "type": "string", + "format": "^([0-9a-fA-F]{3}|[0-9a-fA-F]{6})$" + } + ] + } + }, + { + "name": "multiply", + "in": "query", + "description": "Multiply the value with this number for rendering it at the image (integer value required).", + "required": false, + "allowEmptyValue": true, + "schema": { + "type": "number", + "format": "integer" + } + }, + { + "name": "divide", + "in": "query", + "description": "Divide the value with this number for rendering it at the image (integer value required).", + "required": false, + "allowEmptyValue": true, + "schema": { + "type": "number", + "format": "integer" + } + }, + { + "name": "scale", + "in": "query", + "description": "Set the scale of the badge (greater or equal to 100).", + "required": false, + "allowEmptyValue": true, + "schema": { + "type": "number", + "format": "integer" + } + }, + { + "name": "fixed_width_lbl", + "in": "query", + "description": "This parameter overrides auto-sizing of badge and creates it with fixed width. This parameter determines the size of the label's left side *(label/name)*. You must set this parameter together with `fixed_width_val` otherwise it will be ignored. You should set the label/value widths wide enough to provide space for all the possible values/contents of the badge you're requesting. In case the text cannot fit the space given it will be clipped. The `scale` parameter still applies on the values you give to `fixed_width_lbl` and `fixed_width_val`.", + "required": false, + "allowEmptyValue": false, + "schema": { + "type": "number", + "format": "integer" + } + }, + { + "name": "fixed_width_val", + "in": "query", + "description": "This parameter overrides auto-sizing of badge and creates it with fixed width. This parameter determines the size of the label's right side *(value)*. You must set this parameter together with `fixed_width_lbl` otherwise it will be ignored. You should set the label/value widths wide enough to provide space for all the possible values/contents of the badge you're requesting. In case the text cannot fit the space given it will be clipped. The `scale` parameter still applies on the values you give to `fixed_width_lbl` and `fixed_width_val`.", + "required": false, + "allowEmptyValue": false, + "schema": { + "type": "number", + "format": "integer" + } + } + ], + "responses": { + "200": { + "description": "The call was successful. The response should be an SVG image." + }, + "400": { + "description": "Bad request - the body will include a message stating what is wrong." + }, + "404": { + "description": "No chart with the given id is found." + }, + "500": { + "description": "Internal server error. This usually means the server is out of memory." + } + } + } + }, + "/allmetrics": { + "get": { + "summary": "Get a value of all the metrics maintained by netdata", + "description": "The allmetrics endpoint returns the latest value of all charts and dimensions stored in the netdata server.", + "parameters": [ + { + "name": "format", + "in": "query", + "description": "The format of the response to be returned.", + "required": true, + "schema": { + "type": "string", + "enum": [ + "shell", + "prometheus", + "prometheus_all_hosts", + "json" + ], + "default": "shell" + } + }, + { + "name": "variables", + "in": "query", + "description": "When enabled, netdata will expose various system configuration metrics.", + "required": false, + "schema": { + "type": "string", + "enum": [ + "yes", + "no" + ], + "default": "no" + } + }, + { + "name": "help", + "in": "query", + "description": "Enable or disable HELP lines in prometheus output.", + "required": false, + "schema": { + "type": "string", + "enum": [ + "yes", + "no" + ], + "default": "no" + } + }, + { + "name": "types", + "in": "query", + "description": "Enable or disable TYPE lines in prometheus output.", + "required": false, + "schema": { + "type": "string", + "enum": [ + "yes", + "no" + ], + "default": "no" + } + }, + { + "name": "timestamps", + "in": "query", + "description": "Enable or disable timestamps in prometheus output.", + "required": false, + "schema": { + "type": "string", + "enum": [ + "yes", + "no" + ], + "default": "yes" + } + }, + { + "name": "names", + "in": "query", + "description": "When enabled netdata will report dimension names. When disabled netdata will report dimension IDs. The default is controlled in netdata.conf.", + "required": false, + "schema": { + "type": "string", + "enum": [ + "yes", + "no" + ], + "default": "yes" + } + }, + { + "name": "oldunits", + "in": "query", + "description": "When enabled, netdata will show metric names for the default source=average as they appeared before 1.12, by using the legacy unit naming conventions.", + "required": false, + "schema": { + "type": "string", + "enum": [ + "yes", + "no" + ], + "default": "yes" + } + }, + { + "name": "hideunits", + "in": "query", + "description": "When enabled, netdata will not include the units in the metric names, for the default source=average.", + "required": false, + "schema": { + "type": "string", + "enum": [ + "yes", + "no" + ], + "default": "yes" + } + }, + { + "name": "server", + "in": "query", + "description": "Set a distinct name of the client querying prometheus metrics. Netdata will use the client IP if this is not set.", + "required": false, + "schema": { + "type": "string", + "format": "any text" + } + }, + { + "name": "prefix", + "in": "query", + "description": "Prefix all prometheus metrics with this string.", + "required": false, + "schema": { + "type": "string", + "format": "any text" + } + }, + { + "name": "data", + "in": "query", + "description": "Select the prometheus response data source. There is a setting in netdata.conf for the default.", + "required": false, + "schema": { + "type": "string", + "enum": [ + "as-collected", + "average", + "sum" + ], + "default": "average" + } + } + ], + "responses": { + "200": { + "description": "All the metrics returned in the format requested." + }, + "400": { + "description": "The format requested is not supported." + } + } + } + }, + "/alarms": { + "get": { + "summary": "Get a list of active or raised alarms on the server", + "description": "The alarms endpoint returns the list of all raised or enabled alarms on the netdata server. Called without any parameters, the raised alarms in state WARNING or CRITICAL are returned. By passing \"?all\", all the enabled alarms are returned.", + "parameters": [ + { + "name": "all", + "in": "query", + "description": "If passed, all enabled alarms are returned.", + "required": false, + "allowEmptyValue": true, + "schema": { + "type": "boolean" + } + }, + { + "name": "active", + "in": "query", + "description": "If passed, the raised alarms in state WARNING or CRITICAL are returned.", + "required": false, + "allowEmptyValue": true, + "schema": { + "type": "boolean" + } + } + ], + "responses": { + "200": { + "description": "An object containing general info and a linked list of alarms.", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/alarms" + } + } + } + } + } + } + }, + "/alarms_values": { + "get": { + "summary": "Get a list of active or raised alarms on the server", + "description": "The alarms_values endpoint returns the list of all raised or enabled alarms on the netdata server. Called without any parameters, the raised alarms in state WARNING or CRITICAL are returned. By passing \"?all\", all the enabled alarms are returned. This option output differs from `/alarms` in the number of variables delivered. This endpoint gives to user `id`, `value` and alarm `status`.", + "parameters": [ + { + "name": "all", + "in": "query", + "description": "If passed, all enabled alarms are returned.", + "required": false, + "allowEmptyValue": true, + "schema": { + "type": "boolean" + } + }, + { + "name": "active", + "in": "query", + "description": "If passed, the raised alarms in state WARNING or CRITICAL are returned.", + "required": false, + "allowEmptyValue": true, + "schema": { + "type": "boolean" + } + } + ], + "responses": { + "200": { + "description": "An object containing general info and a linked list of alarms.", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/alarms_values" + } + } + } + } + } + } + }, + "/alarm_log": { + "get": { + "summary": "Retrieves the entries of the alarm log", + "description": "Returns an array of alarm_log entries, with historical information on raised and cleared alarms.", + "parameters": [ + { + "name": "after", + "in": "query", + "description": "Passing the parameter after=UNIQUEID returns all the events in the alarm log that occurred after UNIQUEID. An automated series of calls would call the interface once without after=, store the last UNIQUEID of the returned set, and give it back to get incrementally the next events.", + "required": false, + "schema": { + "type": "integer" + } + } + ], + "responses": { + "200": { + "description": "An array of alarm log entries.", + "content": { + "application/json": { + "schema": { + "type": "array", + "items": { + "$ref": "#/components/schemas/alarm_log_entry" + } + } + } + } + } + } + } + }, + "/alarm_count": { + "get": { + "summary": "Get an overall status of the chart", + "description": "Checks multiple charts with the same context and counts number of alarms with given status.", + "parameters": [ + { + "in": "query", + "name": "context", + "description": "Specify context which should be checked.", + "required": false, + "allowEmptyValue": true, + "schema": { + "type": "array", + "items": { + "type": "string" + }, + "default": [ + "system.cpu" + ] + } + }, + { + "in": "query", + "name": "status", + "description": "Specify alarm status to count.", + "required": false, + "allowEmptyValue": true, + "schema": { + "type": "string", + "enum": [ + "REMOVED", + "UNDEFINED", + "UNINITIALIZED", + "CLEAR", + "RAISED", + "WARNING", + "CRITICAL" + ], + "default": "RAISED" + } + } + ], + "responses": { + "200": { + "description": "An object containing a count of alarms with given status for given contexts.", + "content": { + "application/json": { + "schema": { + "type": "array", + "items": { + "type": "number" + } + } + } + } + }, + "500": { + "description": "Internal server error. This usually means the server is out of memory." + } + } + } + }, + "/manage/health": { + "get": { + "summary": "Accesses the health management API to control health checks and notifications at runtime.", + "description": "Available from Netdata v1.12 and above, protected via bearer authorization. Especially useful for maintenance periods, the API allows you to disable health checks completely, silence alarm notifications, or Disable/Silence specific alarms that match selectors on alarm/template name, chart, context, host and family. For the simple disable/silence all scenaria, only the cmd parameter is required. The other parameters are used to define alarm selectors. For more information and examples, refer to the netdata documentation.", + "parameters": [ + { + "name": "cmd", + "in": "query", + "description": "DISABLE ALL: No alarm criteria are evaluated, nothing is written in the alarm log. SILENCE ALL: No notifications are sent. RESET: Return to the default state. DISABLE/SILENCE: Set the mode to be used for the alarms matching the criteria of the alarm selectors. LIST: Show active configuration.", + "required": false, + "schema": { + "type": "string", + "enum": [ + "DISABLE ALL", + "SILENCE ALL", + "DISABLE", + "SILENCE", + "RESET", + "LIST" + ] + } + }, + { + "name": "alarm", + "in": "query", + "description": "The expression provided will match both `alarm` and `template` names.", + "schema": { + "type": "string" + } + }, + { + "name": "chart", + "in": "query", + "description": "Chart ids/names, as shown on the dashboard. These will match the `on` entry of a configured `alarm`.", + "schema": { + "type": "string" + } + }, + { + "name": "context", + "in": "query", + "description": "Chart context, as shown on the dashboard. These will match the `on` entry of a configured `template`.", + "schema": { + "type": "string" + } + }, + { + "name": "hosts", + "in": "query", + "description": "The hostnames that will need to match.", + "schema": { + "type": "string" + } + }, + { + "name": "families", + "in": "query", + "description": "The alarm families.", + "schema": { + "type": "string" + } + } + ], + "responses": { + "200": { + "description": "A plain text response based on the result of the command." + }, + "403": { + "description": "Bearer authentication error." + } + } + } + } + }, + "servers": [ + { + "url": "https://registry.my-netdata.io/api/v1" + }, + { + "url": "http://registry.my-netdata.io/api/v1" + } + ], + "components": { + "schemas": { + "info": { + "type": "object", + "properties": { + "version": { + "type": "string", + "description": "netdata version of the server.", + "example": "1.11.1_rolling" + }, + "uid": { + "type": "string", + "description": "netdata unique id of the server.", + "example": "24e9fe3c-f2ac-11e8-bafc-0242ac110002" + }, + "mirrored_hosts": { + "type": "array", + "description": "List of hosts mirrored of the server (include itself).", + "items": { + "type": "string" + }, + "example": [ + "host1.example.com", + "host2.example.com" + ] + }, + "mirrored_hosts_status": { + "type": "array", + "description": "List of details of hosts mirrored to this served (including self). Indexes correspond to indexes in \"mirrored_hosts\".", + "items": { + "type": "object", + "description": "Host data", + "properties": { + "guid": { + "type": "string", + "format": "uuid", + "nullable": false, + "description": "Host unique GUID from `netdata.public.unique.id`.", + "example": "245e4bff-3b34-47c1-a6e5-5c535a9abfb2" + }, + "reachable": { + "type": "boolean", + "nullable": false, + "description": "Current state of streaming. Always true for localhost/self." + }, + "claim_id": { + "type": "string", + "format": "uuid", + "nullable": true, + "description": "Cloud GUID/identifier in case the host is claimed. If child status unknown or unclaimed this field is set to `null`", + "example": "c3b2a66a-3052-498c-ac52-7fe9e8cccb0c" + } + } + } + }, + "os_name": { + "type": "string", + "description": "Operating System Name.", + "example": "Manjaro Linux" + }, + "os_id": { + "type": "string", + "description": "Operating System ID.", + "example": "manjaro" + }, + "os_id_like": { + "type": "string", + "description": "Known OS similar to this OS.", + "example": "arch" + }, + "os_version": { + "type": "string", + "description": "Operating System Version.", + "example": "18.0.4" + }, + "os_version_id": { + "type": "string", + "description": "Operating System Version ID.", + "example": "unknown" + }, + "os_detection": { + "type": "string", + "description": "OS parameters detection method.", + "example": "Mixed" + }, + "kernel_name": { + "type": "string", + "description": "Kernel Name.", + "example": "Linux" + }, + "kernel_version": { + "type": "string", + "description": "Kernel Version.", + "example": "4.19.32-1-MANJARO" + }, + "is_k8s_node": { + "type": "boolean", + "description": "Netdata is running on a K8s node.", + "example": false + }, + "architecture": { + "type": "string", + "description": "Kernel architecture.", + "example": "x86_64" + }, + "virtualization": { + "type": "string", + "description": "Virtualization Type.", + "example": "kvm" + }, + "virt_detection": { + "type": "string", + "description": "Virtualization detection method.", + "example": "systemd-detect-virt" + }, + "container": { + "type": "string", + "description": "Container technology.", + "example": "docker" + }, + "container_detection": { + "type": "string", + "description": "Container technology detection method.", + "example": "dockerenv" + }, + "labels": { + "type": "object", + "description": "List of host labels.", + "properties": { + "app": { + "type": "string", + "description": "Host label.", + "example": "netdata" + } + } + }, + "collectors": { + "type": "array", + "items": { + "type": "object", + "description": "Array of collector plugins and modules.", + "properties": { + "plugin": { + "type": "string", + "description": "Collector plugin.", + "example": "python.d.plugin" + }, + "module": { + "type": "string", + "description": "Module of the collector plugin.", + "example": "dockerd" + } + } + } + }, + "alarms": { + "type": "object", + "description": "Number of alarms in the server.", + "properties": { + "normal": { + "type": "integer", + "description": "Number of alarms in normal state." + }, + "warning": { + "type": "integer", + "description": "Number of alarms in warning state." + }, + "critical": { + "type": "integer", + "description": "Number of alarms in critical state." + } + } + } + } + }, + "chart_summary": { + "type": "object", + "properties": { + "hostname": { + "type": "string", + "description": "The hostname of the netdata server." + }, + "version": { + "type": "string", + "description": "netdata version of the server." + }, + "release_channel": { + "type": "string", + "description": "The release channel of the build on the server.", + "example": "nightly" + }, + "timezone": { + "type": "string", + "description": "The current timezone on the server." + }, + "os": { + "type": "string", + "description": "The netdata server host operating system.", + "enum": [ + "macos", + "linux", + "freebsd" + ] + }, + "history": { + "type": "number", + "description": "The duration, in seconds, of the round robin database maintained by netdata." + }, + "memory_mode": { + "type": "string", + "description": "The name of the database memory mode on the server." + }, + "update_every": { + "type": "number", + "description": "The default update frequency of the netdata server. All charts have an update frequency equal or bigger than this." + }, + "charts": { + "type": "object", + "description": "An object containing all the chart objects available at the netdata server. This is used as an indexed array. The key of each chart object is the id of the chart.", + "additionalProperties": { + "$ref": "#/components/schemas/chart" + } + }, + "charts_count": { + "type": "number", + "description": "The number of charts." + }, + "dimensions_count": { + "type": "number", + "description": "The total number of dimensions." + }, + "alarms_count": { + "type": "number", + "description": "The number of alarms." + }, + "rrd_memory_bytes": { + "type": "number", + "description": "The size of the round robin database in bytes." + } + } + }, + "chart": { + "type": "object", + "properties": { + "id": { + "type": "string", + "description": "The unique id of the chart." + }, + "name": { + "type": "string", + "description": "The name of the chart." + }, + "type": { + "type": "string", + "description": "The type of the chart. Types are not handled by netdata. You can use this field for anything you like." + }, + "family": { + "type": "string", + "description": "The family of the chart. Families are not handled by netdata. You can use this field for anything you like." + }, + "title": { + "type": "string", + "description": "The title of the chart." + }, + "priority": { + "type": "number", + "description": "The relative priority of the chart. NetData does not care about priorities. This is just an indication of importance for the chart viewers to sort charts of higher priority (lower number) closer to the top. Priority sorting should only be used among charts of the same type or family." + }, + "enabled": { + "type": "boolean", + "description": "True when the chart is enabled. Disabled charts do not currently collect values, but they may have historical values available." + }, + "units": { + "type": "string", + "description": "The unit of measurement for the values of all dimensions of the chart." + }, + "data_url": { + "type": "string", + "description": "The absolute path to get data values for this chart. You are expected to use this path as the base when constructing the URL to fetch data values for this chart." + }, + "chart_type": { + "type": "string", + "description": "The chart type.", + "enum": [ + "line", + "area", + "stacked" + ] + }, + "duration": { + "type": "number", + "description": "The duration, in seconds, of the round robin database maintained by netdata." + }, + "first_entry": { + "type": "number", + "description": "The UNIX timestamp of the first entry (the oldest) in the round robin database." + }, + "last_entry": { + "type": "number", + "description": "The UNIX timestamp of the latest entry in the round robin database." + }, + "update_every": { + "type": "number", + "description": "The update frequency of this chart, in seconds. One value every this amount of time is kept in the round robin database." + }, + "dimensions": { + "type": "object", + "description": "An object containing all the chart dimensions available for the chart. This is used as an indexed array. For each pair in the dictionary: the key is the id of the dimension and the value is a dictionary containing the name.", + "additionalProperties": { + "type": "object", + "properties": { + "name": { + "type": "string", + "description": "The name of the dimension" + } + } + } + }, + "chart_variables": { + "type": "object", + "additionalProperties": { + "$ref": "#/components/schemas/chart_variables" + } + }, + "green": { + "type": "number", + "nullable": true, + "description": "Chart health green threshold." + }, + "red": { + "type": "number", + "nullable": true, + "description": "Chart health red threshold." + } + } + }, + "alarm_variables": { + "type": "object", + "properties": { + "chart": { + "type": "string", + "description": "The unique id of the chart." + }, + "chart_name": { + "type": "string", + "description": "The name of the chart." + }, + "cnart_context": { + "type": "string", + "description": "The context of the chart. It is shared across multiple monitored software or hardware instances and used in alarm templates." + }, + "family": { + "type": "string", + "description": "The family of the chart." + }, + "host": { + "type": "string", + "description": "The host containing the chart." + }, + "chart_variables": { + "type": "object", + "additionalProperties": { + "$ref": "#/components/schemas/chart_variables" + } + }, + "family_variables": { + "type": "object", + "properties": { + "varname1": { + "type": "number", + "format": "float" + }, + "varname2": { + "type": "number", + "format": "float" + } + } + }, + "host_variables": { + "type": "object", + "properties": { + "varname1": { + "type": "number", + "format": "float" + }, + "varname2": { + "type": "number", + "format": "float" + } + } + } + } + }, + "chart_variables": { + "type": "object", + "properties": { + "varname1": { + "type": "number", + "format": "float" + }, + "varname2": { + "type": "number", + "format": "float" + } + } + }, + "data": { + "type": "object", + "discriminator": { + "propertyName": "format" + }, + "description": "Response will contain the appropriate subtype, e.g. data_json depending on the requested format.", + "properties": { + "api": { + "type": "number", + "description": "The API version this conforms to, currently 1." + }, + "id": { + "type": "string", + "description": "The unique id of the chart." + }, + "name": { + "type": "string", + "description": "The name of the chart." + }, + "update_every": { + "type": "number", + "description": "The update frequency of this chart, in seconds. One value every this amount of time is kept in the round robin database (indepedently of the current view)." + }, + "view_update_every": { + "type": "number", + "description": "The current view appropriate update frequency of this chart, in seconds. There is no point to request chart refreshes, using the same settings, more frequently than this." + }, + "first_entry": { + "type": "number", + "description": "The UNIX timestamp of the first entry (the oldest) in the round robin database (indepedently of the current view)." + }, + "last_entry": { + "type": "number", + "description": "The UNIX timestamp of the latest entry in the round robin database (indepedently of the current view)." + }, + "after": { + "type": "number", + "description": "The UNIX timestamp of the first entry (the oldest) returned in this response." + }, + "before": { + "type": "number", + "description": "The UNIX timestamp of the latest entry returned in this response." + }, + "min": { + "type": "number", + "description": "The minimum value returned in the current view. This can be used to size the y-series of the chart." + }, + "max": { + "type": "number", + "description": "The maximum value returned in the current view. This can be used to size the y-series of the chart." + }, + "dimension_names": { + "description": "The dimension names of the chart as returned in the current view.", + "type": "array", + "items": { + "type": "string" + } + }, + "dimension_ids": { + "description": "The dimension IDs of the chart as returned in the current view.", + "type": "array", + "items": { + "type": "string" + } + }, + "latest_values": { + "description": "The latest values collected for the chart (indepedently of the current view).", + "type": "array", + "items": { + "type": "string" + } + }, + "view_latest_values": { + "description": "The latest values returned with this response.", + "type": "array", + "items": { + "type": "string" + } + }, + "dimensions": { + "type": "number", + "description": "The number of dimensions returned." + }, + "points": { + "type": "number", + "description": "The number of rows / points returned." + }, + "format": { + "type": "string", + "description": "The format of the result returned." + }, + "chart_variables": { + "type": "object", + "additionalProperties": { + "$ref": "#/components/schemas/chart_variables" + } + } + } + }, + "data_json": { + "description": "Data response in json format.", + "allOf": [ + { + "$ref": "#/components/schemas/data" + }, + { + "properties": { + "result": { + "type": "object", + "properties": { + "labels": { + "description": "The dimensions retrieved from the chart.", + "type": "array", + "items": { + "type": "string" + } + }, + "data": { + "description": "The data requested, one element per sample with each element containing the values of the dimensions described in the labels value.", + "type": "array", + "items": { + "type": "number" + } + } + }, + "description": "The result requested, in the format requested." + } + } + } + ] + }, + "data_flat": { + "description": "Data response in csv / tsv / tsv-excel / ssv / ssv-comma / markdown / html formats.", + "allOf": [ + { + "$ref": "#/components/schemas/data" + }, + { + "properties": { + "result": { + "type": "string" + } + } + } + ] + }, + "data_array": { + "description": "Data response in array format.", + "allOf": [ + { + "$ref": "#/components/schemas/data" + }, + { + "properties": { + "result": { + "type": "array", + "items": { + "type": "number" + } + } + } + } + ] + }, + "data_csvjsonarray": { + "description": "Data response in csvjsonarray format.", + "allOf": [ + { + "$ref": "#/components/schemas/data" + }, + { + "properties": { + "result": { + "description": "The first inner array contains strings showing the labels of each column, each subsequent array contains the values for each point in time.", + "type": "array", + "items": { + "type": "array", + "items": {} + } + } + } + } + ] + }, + "data_datatable": { + "description": "Data response in datatable / datasource formats (suitable for Google Charts).", + "allOf": [ + { + "$ref": "#/components/schemas/data" + }, + { + "properties": { + "result": { + "type": "object", + "properties": { + "cols": { + "type": "array", + "items": { + "type": "object", + "properties": { + "id": { + "description": "Always empty - for future use." + }, + "label": { + "description": "The dimension returned from the chart." + }, + "pattern": { + "description": "Always empty - for future use." + }, + "type": { + "description": "The type of data in the column / chart-dimension." + }, + "p": { + "description": "Contains any annotations for the column." + } + }, + "required": [ + "id", + "label", + "pattern", + "type" + ] + } + }, + "rows": { + "type": "array", + "items": { + "type": "object", + "properties": { + "c": { + "type": "array", + "items": { + "properties": { + "v": { + "description": "Each value in the row is represented by an object named `c` with five v fields: data, null, null, 0, the value. This format is fixed by the Google Charts API." + } + } + } + } + } + } + } + } + } + } + } + ] + }, + "alarms": { + "type": "object", + "properties": { + "hostname": { + "type": "string" + }, + "latest_alarm_log_unique_id": { + "type": "integer", + "format": "int32" + }, + "status": { + "type": "boolean" + }, + "now": { + "type": "integer", + "format": "int32" + }, + "alarms": { + "type": "object", + "properties": { + "chart-name.alarm-name": { + "type": "object", + "properties": { + "id": { + "type": "integer", + "format": "int32" + }, + "name": { + "type": "string", + "description": "Full alarm name." + }, + "chart": { + "type": "string" + }, + "family": { + "type": "string" + }, + "active": { + "type": "boolean", + "description": "Will be false only if the alarm is disabled in the configuration." + }, + "disabled": { + "type": "boolean", + "description": "Whether the health check for this alarm has been disabled via a health command API DISABLE command." + }, + "silenced": { + "type": "boolean", + "description": "Whether notifications for this alarm have been silenced via a health command API SILENCE command." + }, + "exec": { + "type": "string" + }, + "recipient": { + "type": "string" + }, + "source": { + "type": "string" + }, + "units": { + "type": "string" + }, + "info": { + "type": "string" + }, + "status": { + "type": "string" + }, + "last_status_change": { + "type": "integer", + "format": "int32" + }, + "last_updated": { + "type": "integer", + "format": "int32" + }, + "next_update": { + "type": "integer", + "format": "int32" + }, + "update_every": { + "type": "integer", + "format": "int32" + }, + "delay_up_duration": { + "type": "integer", + "format": "int32" + }, + "delay_down_duration": { + "type": "integer", + "format": "int32" + }, + "delay_max_duration": { + "type": "integer", + "format": "int32" + }, + "delay_multiplier": { + "type": "integer", + "format": "int32" + }, + "delay": { + "type": "integer", + "format": "int32" + }, + "delay_up_to_timestamp": { + "type": "integer", + "format": "int32" + }, + "value_string": { + "type": "string" + }, + "no_clear_notification": { + "type": "boolean" + }, + "lookup_dimensions": { + "type": "string" + }, + "db_after": { + "type": "integer", + "format": "int32" + }, + "db_before": { + "type": "integer", + "format": "int32" + }, + "lookup_method": { + "type": "string" + }, + "lookup_after": { + "type": "integer", + "format": "int32" + }, + "lookup_before": { + "type": "integer", + "format": "int32" + }, + "lookup_options": { + "type": "string" + }, + "calc": { + "type": "string" + }, + "calc_parsed": { + "type": "string" + }, + "warn": { + "type": "string" + }, + "warn_parsed": { + "type": "string" + }, + "crit": { + "type": "string" + }, + "crit_parsed": { + "type": "string" + }, + "warn_repeat_every": { + "type": "integer", + "format": "int32" + }, + "crit_repeat_every": { + "type": "integer", + "format": "int32" + }, + "green": { + "type": "string", + "format": "nullable" + }, + "red": { + "type": "string", + "format": "nullable" + }, + "value": { + "type": "number" + } + } + } + } + } + } + }, + "alarm_log_entry": { + "type": "object", + "properties": { + "hostname": { + "type": "string" + }, + "unique_id": { + "type": "integer", + "format": "int32" + }, + "alarm_id": { + "type": "integer", + "format": "int32" + }, + "alarm_event_id": { + "type": "integer", + "format": "int32" + }, + "name": { + "type": "string" + }, + "chart": { + "type": "string" + }, + "family": { + "type": "string" + }, + "processed": { + "type": "boolean" + }, + "updated": { + "type": "boolean" + }, + "exec_run": { + "type": "integer", + "format": "int32" + }, + "exec_failed": { + "type": "boolean" + }, + "exec": { + "type": "string" + }, + "recipient": { + "type": "string" + }, + "exec_code": { + "type": "integer", + "format": "int32" + }, + "source": { + "type": "string" + }, + "units": { + "type": "string" + }, + "when": { + "type": "integer", + "format": "int32" + }, + "duration": { + "type": "integer", + "format": "int32" + }, + "non_clear_duration": { + "type": "integer", + "format": "int32" + }, + "status": { + "type": "string" + }, + "old_status": { + "type": "string" + }, + "delay": { + "type": "integer", + "format": "int32" + }, + "delay_up_to_timestamp": { + "type": "integer", + "format": "int32" + }, + "updated_by_id": { + "type": "integer", + "format": "int32" + }, + "updates_id": { + "type": "integer", + "format": "int32" + }, + "value_string": { + "type": "string" + }, + "old_value_string": { + "type": "string" + }, + "silenced": { + "type": "string" + }, + "info": { + "type": "string" + }, + "value": { + "type": "number", + "nullable": true + }, + "old_value": { + "type": "number", + "nullable": true + } + } + }, + "alarms_values": { + "type": "object", + "properties": { + "hostname": { + "type": "string" + }, + "alarms": { + "type": "object", + "description": "HashMap with keys being alarm names", + "additionalProperties": { + "type": "object", + "properties": { + "id": { + "type": "integer" + }, + "value": { + "type": "integer" + }, + "status": { + "type": "string", + "enum": [ + "REMOVED", + "UNDEFINED", + "UNINITIALIZED", + "CLEAR", + "RAISED", + "WARNING", + "CRITICAL", + "UNKNOWN" + ] + } + } + } + } + } + } + } + } +}
\ No newline at end of file diff --git a/web/api/netdata-swagger.yaml b/web/api/netdata-swagger.yaml new file mode 100644 index 0000000..7482742 --- /dev/null +++ b/web/api/netdata-swagger.yaml @@ -0,0 +1,1611 @@ +openapi: 3.0.0 +info: + title: NetData API + description: Real-time performance and health monitoring. + version: 1.11.1_rolling +paths: + /info: + get: + summary: Get netdata basic information + description: | + The info endpoint returns basic information about netdata. It provides: + * netdata version + * netdata unique id + * list of hosts mirrored (includes itself) + * Operating System, Virtualization, K8s nodes and Container technology information + * List of active collector plugins and modules + * number of alarms in the host + * number of alarms in normal state + * number of alarms in warning state + * number of alarms in critical state + responses: + "200": + description: netdata basic information. + content: + application/json: + schema: + $ref: "#/components/schemas/info" + "503": + description: netdata daemon not ready (used for health checks). + /charts: + get: + summary: Get a list of all charts available at the server + description: The charts endpoint returns a summary about all charts stored in the + netdata server. + responses: + "200": + description: An array of charts. + content: + application/json: + schema: + $ref: "#/components/schemas/chart_summary" + /chart: + get: + summary: Get info about a specific chart + description: The Chart endpoint returns detailed information about a chart. + parameters: + - name: chart + in: query + description: The id of the chart as returned by the /charts call. + required: true + schema: + type: string + format: as returned by /charts + default: system.cpu + responses: + "200": + description: A javascript object with detailed information about the chart. + content: + application/json: + schema: + $ref: "#/components/schemas/chart" + "400": + description: No chart id was supplied in the request. + "404": + description: No chart with the given id is found. + /alarm_variables: + get: + summary: List variables available to configure alarms for a chart + description: Returns the basic information of a chart and all the variables that can + be used in alarm and template health configurations for the particular + chart or family. + parameters: + - name: chart + in: query + description: The id of the chart as returned by the /charts call. + required: true + schema: + type: string + format: as returned by /charts + default: system.cpu + responses: + "200": + description: A javascript object with information about the chart and the + available variables. + content: + application/json: + schema: + $ref: "#/components/schemas/alarm_variables" + "400": + description: Bad request - the body will include a message stating what is wrong. + "404": + description: No chart with the given id is found. + "500": + description: Internal server error. This usually means the server is out of + memory. + /data: + get: + summary: Get collected data for a specific chart + description: The data endpoint returns data stored in the round robin database of a + chart. + parameters: + - name: chart + in: query + description: The id of the chart as returned by the /charts call. Note chart or context must be specified + required: false + allowEmptyValue: false + schema: + type: string + format: as returned by /charts + default: system.cpu + - name: context + in: query + description: The context of the chart as returned by the /charts call. Note chart or context must be specified + required: false + allowEmptyValue: false + schema: + type: string + format: as returned by /charts + - name: dimension + in: query + description: Zero, one or more dimension ids or names, as returned by the /chart + call, separated with comma or pipe. Netdata simple patterns are + supported. + required: false + allowEmptyValue: false + schema: + type: array + items: + type: string + format: as returned by /charts + - name: after + in: query + description: "This parameter can either be an absolute timestamp specifying the + starting point of the data to be returned, or a relative number of + seconds (negative, relative to parameter: before). Netdata will + assume it is a relative number if it is less that 3 years (in seconds). + If not specified the default is -600 seconds. Netdata will adapt this + parameter to the boundaries of the round robin database unless the allow_past + option is specified." + required: true + allowEmptyValue: false + schema: + type: number + format: integer + default: -600 + - name: before + in: query + description: This parameter can either be an absolute timestamp specifying the + ending point of the data to be returned, or a relative number of + seconds (negative), relative to the last collected timestamp. + Netdata will assume it is a relative number if it is less than 3 + years (in seconds). Netdata will adapt this parameter to the + boundaries of the round robin database. The default is zero (i.e. + the timestamp of the last value collected). + required: false + schema: + type: number + format: integer + default: 0 + - name: points + in: query + description: The number of points to be returned. If not given, or it is <= 0, or + it is bigger than the points stored in the round robin database for + this chart for the given duration, all the available collected + values for the given duration will be returned. + required: true + allowEmptyValue: false + schema: + type: number + format: integer + default: 20 + - name: group + in: query + description: The grouping method. If multiple collected values are to be grouped + in order to return fewer points, this parameters defines the method + of grouping. methods supported "min", "max", "average", "sum", + "incremental-sum". "max" is actually calculated on the absolute + value collected (so it works for both positive and negative + dimesions to return the most extreme value in either direction). + required: true + allowEmptyValue: false + schema: + type: string + enum: + - min + - max + - average + - median + - stddev + - sum + - incremental-sum + default: average + - name: gtime + in: query + description: The grouping number of seconds. This is used in conjunction with + group=average to change the units of metrics (ie when the data is + per-second, setting gtime=60 will turn them to per-minute). + required: false + allowEmptyValue: false + schema: + type: number + format: integer + default: 0 + - name: format + in: query + description: The format of the data to be returned. + required: true + allowEmptyValue: false + schema: + type: string + enum: + - json + - jsonp + - csv + - tsv + - tsv-excel + - ssv + - ssvcomma + - datatable + - datasource + - html + - markdown + - array + - csvjsonarray + default: json + - name: options + in: query + description: Options that affect data generation. + required: false + allowEmptyValue: false + schema: + type: array + items: + type: string + enum: + - nonzero + - flip + - jsonwrap + - min2max + - seconds + - milliseconds + - abs + - absolute + - absolute-sum + - null2zero + - objectrows + - google_json + - percentage + - unaligned + - match-ids + - match-names + - showcustomvars + - allow_past + default: + - seconds + - jsonwrap + - name: callback + in: query + description: For JSONP responses, the callback function name. + required: false + allowEmptyValue: true + schema: + type: string + - name: filename + in: query + description: "Add Content-Disposition: attachment; filename= header to + the response, that will instruct the browser to save the response + with the given filename." + required: false + allowEmptyValue: true + schema: + type: string + - name: tqx + in: query + description: "[Google Visualization + API](https://developers.google.com/chart/interactive/docs/dev/imple\ + menting_data_source?hl=en) formatted parameter." + required: false + allowEmptyValue: true + schema: + type: string + responses: + "200": + description: The call was successful. The response includes the data in the + format requested. Swagger2.0 does not process the discriminator + field to show polymorphism. The response will be one of the + sub-types of the data-schema according to the chosen format, e.g. + json -> data_json. + content: + application/json: + schema: + $ref: "#/components/schemas/data" + "400": + description: Bad request - the body will include a message stating what is wrong. + "404": + description: Chart or context is not found. The supplied chart or context will be reported. + "500": + description: Internal server error. This usually means the server is out of + memory. + /badge.svg: + get: + summary: Generate a badge in form of SVG image for a chart (or dimension) + description: Successful responses are SVG images. + parameters: + - name: chart + in: query + description: The id of the chart as returned by the /charts call. + required: true + allowEmptyValue: false + schema: + type: string + format: as returned by /charts + default: system.cpu + - name: alarm + in: query + description: The name of an alarm linked to the chart. + required: false + allowEmptyValue: true + schema: + type: string + format: any text + - name: dimension + in: query + description: Zero, one or more dimension ids, as returned by the /chart call. + required: false + allowEmptyValue: false + schema: + type: array + items: + type: string + format: as returned by /charts + - name: after + in: query + description: This parameter can either be an absolute timestamp specifying the + starting point of the data to be returned, or a relative number of + seconds, to the last collected timestamp. Netdata will assume it is + a relative number if it is smaller than the duration of the round + robin database for this chart. So, if the round robin database is + 3600 seconds, any value from -3600 to 3600 will trigger relative + arithmetics. Netdata will adapt this parameter to the boundaries of + the round robin database. + required: true + allowEmptyValue: false + schema: + type: number + format: integer + default: -600 + - name: before + in: query + description: This parameter can either be an absolute timestamp specifying the + ending point of the data to be returned, or a relative number of + seconds, to the last collected timestamp. Netdata will assume it is + a relative number if it is smaller than the duration of the round + robin database for this chart. So, if the round robin database is + 3600 seconds, any value from -3600 to 3600 will trigger relative + arithmetics. Netdata will adapt this parameter to the boundaries of + the round robin database. + required: false + schema: + type: number + format: integer + default: 0 + - name: group + in: query + description: The grouping method. If multiple collected values are to be grouped + in order to return fewer points, this parameters defines the method + of grouping. methods are supported "min", "max", "average", "sum", + "incremental-sum". "max" is actually calculated on the absolute + value collected (so it works for both positive and negative + dimesions to return the most extreme value in either direction). + required: true + allowEmptyValue: false + schema: + type: string + enum: + - min + - max + - average + - median + - stddev + - sum + - incremental-sum + default: average + - name: options + in: query + description: Options that affect data generation. + required: false + allowEmptyValue: true + schema: + type: array + items: + type: string + enum: + - abs + - absolute + - display-absolute + - absolute-sum + - null2zero + - percentage + - unaligned + default: + - absolute + - name: label + in: query + description: A text to be used as the label. + required: false + allowEmptyValue: true + schema: + type: string + format: any text + - name: units + in: query + description: A text to be used as the units. + required: false + allowEmptyValue: true + schema: + type: string + format: any text + - name: label_color + in: query + description: A color to be used for the background of the label side(left side) of the badge. One of predefined colors or specific color in hex `RGB` or `RRGGBB` format (without preceding `#` character). If value wrong or not given default color will be used. + required: false + allowEmptyValue: true + schema: + oneOf: + - type: string + enum: + - green + - brightgreen + - yellow + - yellowgreen + - orange + - red + - blue + - grey + - gray + - lightgrey + - lightgray + - type: string + format: ^([0-9a-fA-F]{3}|[0-9a-fA-F]{6})$ + - name: value_color + in: query + description: "A color to be used for the background of the value *(right)* part of badge. You can set + multiple using a pipe with a condition each, like this: + `color<value|color:null` The following operators are + supported: >, <, >=, <=, =, :null (to check if no value exists). Each color can be specified in same manner as for `label_color` parameter. + Currently only integers are suported as values." + required: false + allowEmptyValue: true + schema: + type: string + format: any text + - name: text_color_lbl + in: query + description: Font color for label *(left)* part of the badge. One of predefined colors or as HTML hexadecimal color without preceeding `#` character. Formats allowed `RGB` or `RRGGBB`. If no or wrong value given default color will be used. + required: false + allowEmptyValue: true + schema: + oneOf: + - type: string + enum: + - green + - brightgreen + - yellow + - yellowgreen + - orange + - red + - blue + - grey + - gray + - lightgrey + - lightgray + - type: string + format: ^([0-9a-fA-F]{3}|[0-9a-fA-F]{6})$ + - name: text_color_val + in: query + description: Font color for value *(right)* part of the badge. One of predefined colors or as HTML hexadecimal color without preceeding `#` character. Formats allowed `RGB` or `RRGGBB`. If no or wrong value given default color will be used. + required: false + allowEmptyValue: true + schema: + oneOf: + - type: string + enum: + - green + - brightgreen + - yellow + - yellowgreen + - orange + - red + - blue + - grey + - gray + - lightgrey + - lightgray + - type: string + format: ^([0-9a-fA-F]{3}|[0-9a-fA-F]{6})$ + - name: multiply + in: query + description: Multiply the value with this number for rendering it at the image + (integer value required). + required: false + allowEmptyValue: true + schema: + type: number + format: integer + - name: divide + in: query + description: Divide the value with this number for rendering it at the image + (integer value required). + required: false + allowEmptyValue: true + schema: + type: number + format: integer + - name: scale + in: query + description: Set the scale of the badge (greater or equal to 100). + required: false + allowEmptyValue: true + schema: + type: number + format: integer + - name: fixed_width_lbl + in: query + description: This parameter overrides auto-sizing of badge and creates it with fixed width. This parameter determines the size of the label's left side *(label/name)*. You must set this parameter together with `fixed_width_val` otherwise it will be ignored. You should set the label/value widths wide enough to provide space for all the possible values/contents of the badge you're requesting. In case the text cannot fit the space given it will be clipped. The `scale` parameter still applies on the values you give to `fixed_width_lbl` and `fixed_width_val`. + required: false + allowEmptyValue: false + schema: + type: number + format: integer + - name: fixed_width_val + in: query + description: This parameter overrides auto-sizing of badge and creates it with fixed width. This parameter determines the size of the label's right side *(value)*. You must set this parameter together with `fixed_width_lbl` otherwise it will be ignored. You should set the label/value widths wide enough to provide space for all the possible values/contents of the badge you're requesting. In case the text cannot fit the space given it will be clipped. The `scale` parameter still applies on the values you give to `fixed_width_lbl` and `fixed_width_val`. + required: false + allowEmptyValue: false + schema: + type: number + format: integer + responses: + "200": + description: The call was successful. The response should be an SVG image. + "400": + description: Bad request - the body will include a message stating what is wrong. + "404": + description: No chart with the given id is found. + "500": + description: Internal server error. This usually means the server is out of + memory. + /allmetrics: + get: + summary: Get a value of all the metrics maintained by netdata + description: The allmetrics endpoint returns the latest value of all charts and + dimensions stored in the netdata server. + parameters: + - name: format + in: query + description: The format of the response to be returned. + required: true + schema: + type: string + enum: + - shell + - prometheus + - prometheus_all_hosts + - json + default: shell + - name: variables + in: query + description: When enabled, netdata will expose various system + configuration metrics. + required: false + schema: + type: string + enum: + - yes + - no + default: no + - name: help + in: query + description: Enable or disable HELP lines in prometheus output. + required: false + schema: + type: string + enum: + - yes + - no + default: no + - name: types + in: query + description: Enable or disable TYPE lines in prometheus output. + required: false + schema: + type: string + enum: + - yes + - no + default: no + - name: timestamps + in: query + description: Enable or disable timestamps in prometheus output. + required: false + schema: + type: string + enum: + - yes + - no + default: yes + - name: names + in: query + description: When enabled netdata will report dimension names. When disabled + netdata will report dimension IDs. The default is controlled in + netdata.conf. + required: false + schema: + type: string + enum: + - yes + - no + default: yes + - name: oldunits + in: query + description: When enabled, netdata will show metric names for the default + source=average as they appeared before 1.12, by using the legacy + unit naming conventions. + required: false + schema: + type: string + enum: + - yes + - no + default: yes + - name: hideunits + in: query + description: When enabled, netdata will not include the units in the metric + names, for the default source=average. + required: false + schema: + type: string + enum: + - yes + - no + default: yes + - name: server + in: query + description: Set a distinct name of the client querying prometheus metrics. + Netdata will use the client IP if this is not set. + required: false + schema: + type: string + format: any text + - name: prefix + in: query + description: Prefix all prometheus metrics with this string. + required: false + schema: + type: string + format: any text + - name: data + in: query + description: Select the prometheus response data source. There is a setting in + netdata.conf for the default. + required: false + schema: + type: string + enum: + - as-collected + - average + - sum + default: average + responses: + "200": + description: All the metrics returned in the format requested. + "400": + description: The format requested is not supported. + /alarms: + get: + summary: Get a list of active or raised alarms on the server + description: The alarms endpoint returns the list of all raised or enabled alarms on + the netdata server. Called without any parameters, the raised alarms in + state WARNING or CRITICAL are returned. By passing "?all", all the + enabled alarms are returned. + parameters: + - name: all + in: query + description: If passed, all enabled alarms are returned. + required: false + allowEmptyValue: true + schema: + type: boolean + - name: active + in: query + description: If passed, the raised alarms in state WARNING or CRITICAL are returned. + required: false + allowEmptyValue: true + schema: + type: boolean + responses: + "200": + description: An object containing general info and a linked list of alarms. + content: + application/json: + schema: + $ref: "#/components/schemas/alarms" + /alarms_values: + get: + summary: Get a list of active or raised alarms on the server + description: The alarms_values endpoint returns the list of all raised or enabled alarms on + the netdata server. Called without any parameters, the raised alarms in + state WARNING or CRITICAL are returned. By passing "?all", all the + enabled alarms are returned. + This option output differs from `/alarms` in the number of variables delivered. This endpoint gives + to user `id`, `value` and alarm `status`. + parameters: + - name: all + in: query + description: If passed, all enabled alarms are returned. + required: false + allowEmptyValue: true + schema: + type: boolean + - name: active + in: query + description: If passed, the raised alarms in state WARNING or CRITICAL are returned. + required: false + allowEmptyValue: true + schema: + type: boolean + responses: + "200": + description: An object containing general info and a linked list of alarms. + content: + application/json: + schema: + $ref: "#/components/schemas/alarms_values" + /alarm_log: + get: + summary: Retrieves the entries of the alarm log + description: Returns an array of alarm_log entries, with historical information on + raised and cleared alarms. + parameters: + - name: after + in: query + description: Passing the parameter after=UNIQUEID returns all the events in the + alarm log that occurred after UNIQUEID. An automated series of calls + would call the interface once without after=, store the last + UNIQUEID of the returned set, and give it back to get incrementally + the next events. + required: false + schema: + type: integer + responses: + "200": + description: An array of alarm log entries. + content: + application/json: + schema: + type: array + items: + $ref: "#/components/schemas/alarm_log_entry" + /alarm_count: + get: + summary: Get an overall status of the chart + description: Checks multiple charts with the same context and counts number of alarms + with given status. + parameters: + - in: query + name: context + description: Specify context which should be checked. + required: false + allowEmptyValue: true + schema: + type: array + items: + type: string + default: + - system.cpu + - in: query + name: status + description: Specify alarm status to count. + required: false + allowEmptyValue: true + schema: + type: string + enum: + - REMOVED + - UNDEFINED + - UNINITIALIZED + - CLEAR + - RAISED + - WARNING + - CRITICAL + default: RAISED + responses: + "200": + description: An object containing a count of alarms with given status for given + contexts. + content: + application/json: + schema: + type: array + items: + type: number + "500": + description: Internal server error. This usually means the server is out of + memory. + /manage/health: + get: + summary: Accesses the health management API to control health checks and + notifications at runtime. + description: Available from Netdata v1.12 and above, protected via bearer + authorization. Especially useful for maintenance periods, the API allows + you to disable health checks completely, silence alarm notifications, or + Disable/Silence specific alarms that match selectors on alarm/template + name, chart, context, host and family. For the simple disable/silence + all scenaria, only the cmd parameter is required. The other parameters + are used to define alarm selectors. For more information and examples, + refer to the netdata documentation. + parameters: + - name: cmd + in: query + description: "DISABLE ALL: No alarm criteria are evaluated, nothing is written in + the alarm log. SILENCE ALL: No notifications are sent. RESET: Return + to the default state. DISABLE/SILENCE: Set the mode to be used for + the alarms matching the criteria of the alarm selectors. LIST: Show + active configuration." + required: false + schema: + type: string + enum: + - DISABLE ALL + - SILENCE ALL + - DISABLE + - SILENCE + - RESET + - LIST + - name: alarm + in: query + description: The expression provided will match both `alarm` and `template` names. + schema: + type: string + - name: chart + in: query + description: Chart ids/names, as shown on the dashboard. These will match the + `on` entry of a configured `alarm`. + schema: + type: string + - name: context + in: query + description: Chart context, as shown on the dashboard. These will match the `on` + entry of a configured `template`. + schema: + type: string + - name: hosts + in: query + description: The hostnames that will need to match. + schema: + type: string + - name: families + in: query + description: The alarm families. + schema: + type: string + responses: + "200": + description: A plain text response based on the result of the command. + "403": + description: Bearer authentication error. +servers: + - url: https://registry.my-netdata.io/api/v1 + - url: http://registry.my-netdata.io/api/v1 +components: + schemas: + info: + type: object + properties: + version: + type: string + description: netdata version of the server. + example: 1.11.1_rolling + uid: + type: string + description: netdata unique id of the server. + example: 24e9fe3c-f2ac-11e8-bafc-0242ac110002 + mirrored_hosts: + type: array + description: List of hosts mirrored of the server (include itself). + items: + type: string + example: + - host1.example.com + - host2.example.com + mirrored_hosts_status: + type: array + description: >- + List of details of hosts mirrored to this served (including self). + Indexes correspond to indexes in "mirrored_hosts". + items: + type: object + description: Host data + properties: + guid: + type: string + format: uuid + nullable: false + description: Host unique GUID from `netdata.public.unique.id`. + example: 245e4bff-3b34-47c1-a6e5-5c535a9abfb2 + reachable: + type: boolean + nullable: false + description: Current state of streaming. Always true for localhost/self. + claim_id: + type: string + format: uuid + nullable: true + description: >- + Cloud GUID/identifier in case the host is claimed. + If child status unknown or unclaimed this field is set to `null` + example: c3b2a66a-3052-498c-ac52-7fe9e8cccb0c + os_name: + type: string + description: Operating System Name. + example: Manjaro Linux + os_id: + type: string + description: Operating System ID. + example: manjaro + os_id_like: + type: string + description: Known OS similar to this OS. + example: arch + os_version: + type: string + description: Operating System Version. + example: 18.0.4 + os_version_id: + type: string + description: Operating System Version ID. + example: unknown + os_detection: + type: string + description: OS parameters detection method. + example: Mixed + kernel_name: + type: string + description: Kernel Name. + example: Linux + kernel_version: + type: string + description: Kernel Version. + example: 4.19.32-1-MANJARO + is_k8s_node: + type: boolean + description: Netdata is running on a K8s node. + example: false + architecture: + type: string + description: Kernel architecture. + example: x86_64 + virtualization: + type: string + description: Virtualization Type. + example: kvm + virt_detection: + type: string + description: Virtualization detection method. + example: systemd-detect-virt + container: + type: string + description: Container technology. + example: docker + container_detection: + type: string + description: Container technology detection method. + example: dockerenv + labels: + type: object + description: List of host labels. + properties: + app: + type: string + description: Host label. + example: netdata + collectors: + type: array + items: + type: object + description: Array of collector plugins and modules. + properties: + plugin: + type: string + description: Collector plugin. + example: python.d.plugin + module: + type: string + description: Module of the collector plugin. + example: dockerd + alarms: + type: object + description: Number of alarms in the server. + properties: + normal: + type: integer + description: Number of alarms in normal state. + warning: + type: integer + description: Number of alarms in warning state. + critical: + type: integer + description: Number of alarms in critical state. + chart_summary: + type: object + properties: + hostname: + type: string + description: The hostname of the netdata server. + version: + type: string + description: netdata version of the server. + release_channel: + type: string + description: The release channel of the build on the server. + example: nightly + timezone: + type: string + description: The current timezone on the server. + os: + type: string + description: The netdata server host operating system. + enum: + - macos + - linux + - freebsd + history: + type: number + description: The duration, in seconds, of the round robin database maintained by + netdata. + memory_mode: + type: string + description: The name of the database memory mode on the server. + update_every: + type: number + description: The default update frequency of the netdata server. All charts have + an update frequency equal or bigger than this. + charts: + type: object + description: An object containing all the chart objects available at the netdata + server. This is used as an indexed array. The key of each chart + object is the id of the chart. + additionalProperties: + $ref: "#/components/schemas/chart" + charts_count: + type: number + description: The number of charts. + dimensions_count: + type: number + description: The total number of dimensions. + alarms_count: + type: number + description: The number of alarms. + rrd_memory_bytes: + type: number + description: The size of the round robin database in bytes. + chart: + type: object + properties: + id: + type: string + description: The unique id of the chart. + name: + type: string + description: The name of the chart. + type: + type: string + description: The type of the chart. Types are not handled by netdata. You can use + this field for anything you like. + family: + type: string + description: The family of the chart. Families are not handled by netdata. You + can use this field for anything you like. + title: + type: string + description: The title of the chart. + priority: + type: number + description: The relative priority of the chart. NetData does not care about + priorities. This is just an indication of importance for the chart + viewers to sort charts of higher priority (lower number) closer to + the top. Priority sorting should only be used among charts of the + same type or family. + enabled: + type: boolean + description: True when the chart is enabled. Disabled charts do not currently + collect values, but they may have historical values available. + units: + type: string + description: The unit of measurement for the values of all dimensions of the + chart. + data_url: + type: string + description: The absolute path to get data values for this chart. You are + expected to use this path as the base when constructing the URL to + fetch data values for this chart. + chart_type: + type: string + description: The chart type. + enum: + - line + - area + - stacked + duration: + type: number + description: The duration, in seconds, of the round robin database maintained by + netdata. + first_entry: + type: number + description: The UNIX timestamp of the first entry (the oldest) in the round + robin database. + last_entry: + type: number + description: The UNIX timestamp of the latest entry in the round robin database. + update_every: + type: number + description: The update frequency of this chart, in seconds. One value every this + amount of time is kept in the round robin database. + dimensions: + type: object + description: "An object containing all the chart dimensions available for the + chart. This is used as an indexed array. For each pair in the + dictionary: the key is the id of the dimension and the value is a + dictionary containing the name." + additionalProperties: + type: object + properties: + name: + type: string + description: The name of the dimension + chart_variables: + type: object + additionalProperties: + $ref: "#/components/schemas/chart_variables" + green: + type: number + nullable: true + description: Chart health green threshold. + red: + type: number + nullable: true + description: Chart health red threshold. + alarm_variables: + type: object + properties: + chart: + type: string + description: The unique id of the chart. + chart_name: + type: string + description: The name of the chart. + cnart_context: + type: string + description: The context of the chart. It is shared across multiple monitored + software or hardware instances and used in alarm templates. + family: + type: string + description: The family of the chart. + host: + type: string + description: The host containing the chart. + chart_variables: + type: object + additionalProperties: + $ref: "#/components/schemas/chart_variables" + family_variables: + type: object + properties: + varname1: + type: number + format: float + varname2: + type: number + format: float + host_variables: + type: object + properties: + varname1: + type: number + format: float + varname2: + type: number + format: float + chart_variables: + type: object + properties: + varname1: + type: number + format: float + varname2: + type: number + format: float + data: + type: object + discriminator: + propertyName: format + description: Response will contain the appropriate subtype, e.g. data_json depending + on the requested format. + properties: + api: + type: number + description: The API version this conforms to, currently 1. + id: + type: string + description: The unique id of the chart. + name: + type: string + description: The name of the chart. + update_every: + type: number + description: The update frequency of this chart, in seconds. One value every this + amount of time is kept in the round robin database (indepedently of + the current view). + view_update_every: + type: number + description: The current view appropriate update frequency of this chart, in + seconds. There is no point to request chart refreshes, using the + same settings, more frequently than this. + first_entry: + type: number + description: The UNIX timestamp of the first entry (the oldest) in the round + robin database (indepedently of the current view). + last_entry: + type: number + description: The UNIX timestamp of the latest entry in the round robin database + (indepedently of the current view). + after: + type: number + description: The UNIX timestamp of the first entry (the oldest) returned in this + response. + before: + type: number + description: The UNIX timestamp of the latest entry returned in this response. + min: + type: number + description: The minimum value returned in the current view. This can be used to + size the y-series of the chart. + max: + type: number + description: The maximum value returned in the current view. This can be used to + size the y-series of the chart. + dimension_names: + description: The dimension names of the chart as returned in the current view. + type: array + items: + type: string + dimension_ids: + description: The dimension IDs of the chart as returned in the current view. + type: array + items: + type: string + latest_values: + description: The latest values collected for the chart (indepedently of the + current view). + type: array + items: + type: string + view_latest_values: + description: The latest values returned with this response. + type: array + items: + type: string + dimensions: + type: number + description: The number of dimensions returned. + points: + type: number + description: The number of rows / points returned. + format: + type: string + description: The format of the result returned. + chart_variables: + type: object + additionalProperties: + $ref: "#/components/schemas/chart_variables" + data_json: + description: Data response in json format. + allOf: + - $ref: "#/components/schemas/data" + - properties: + result: + type: object + properties: + labels: + description: The dimensions retrieved from the chart. + type: array + items: + type: string + data: + description: The data requested, one element per sample with each element + containing the values of the dimensions described in the + labels value. + type: array + items: + type: number + description: The result requested, in the format requested. + data_flat: + description: Data response in csv / tsv / tsv-excel / ssv / ssv-comma / markdown / + html formats. + allOf: + - $ref: "#/components/schemas/data" + - properties: + result: + type: string + data_array: + description: Data response in array format. + allOf: + - $ref: "#/components/schemas/data" + - properties: + result: + type: array + items: + type: number + data_csvjsonarray: + description: Data response in csvjsonarray format. + allOf: + - $ref: "#/components/schemas/data" + - properties: + result: + description: The first inner array contains strings showing the labels of + each column, each subsequent array contains the values for each + point in time. + type: array + items: + type: array + items: {} + data_datatable: + description: Data response in datatable / datasource formats (suitable for Google + Charts). + allOf: + - $ref: "#/components/schemas/data" + - properties: + result: + type: object + properties: + cols: + type: array + items: + type: object + properties: + id: + description: Always empty - for future use. + label: + description: The dimension returned from the chart. + pattern: + description: Always empty - for future use. + type: + description: The type of data in the column / chart-dimension. + p: + description: Contains any annotations for the column. + required: + - id + - label + - pattern + - type + rows: + type: array + items: + type: object + properties: + c: + type: array + items: + properties: + v: + description: "Each value in the row is represented by an + object named `c` with five v fields: data, null, + null, 0, the value. This format is fixed by the + Google Charts API." + alarms: + type: object + properties: + hostname: + type: string + latest_alarm_log_unique_id: + type: integer + format: int32 + status: + type: boolean + now: + type: integer + format: int32 + alarms: + type: object + properties: + chart-name.alarm-name: + type: object + properties: + id: + type: integer + format: int32 + name: + type: string + description: Full alarm name. + chart: + type: string + family: + type: string + active: + type: boolean + description: Will be false only if the alarm is disabled in the + configuration. + disabled: + type: boolean + description: Whether the health check for this alarm has been disabled + via a health command API DISABLE command. + silenced: + type: boolean + description: Whether notifications for this alarm have been silenced via + a health command API SILENCE command. + exec: + type: string + recipient: + type: string + source: + type: string + units: + type: string + info: + type: string + status: + type: string + last_status_change: + type: integer + format: int32 + last_updated: + type: integer + format: int32 + next_update: + type: integer + format: int32 + update_every: + type: integer + format: int32 + delay_up_duration: + type: integer + format: int32 + delay_down_duration: + type: integer + format: int32 + delay_max_duration: + type: integer + format: int32 + delay_multiplier: + type: integer + format: int32 + delay: + type: integer + format: int32 + delay_up_to_timestamp: + type: integer + format: int32 + value_string: + type: string + no_clear_notification: + type: boolean + lookup_dimensions: + type: string + db_after: + type: integer + format: int32 + db_before: + type: integer + format: int32 + lookup_method: + type: string + lookup_after: + type: integer + format: int32 + lookup_before: + type: integer + format: int32 + lookup_options: + type: string + calc: + type: string + calc_parsed: + type: string + warn: + type: string + warn_parsed: + type: string + crit: + type: string + crit_parsed: + type: string + warn_repeat_every: + type: integer + format: int32 + crit_repeat_every: + type: integer + format: int32 + green: + type: string + format: nullable + red: + type: string + format: nullable + value: + type: number + alarm_log_entry: + type: object + properties: + hostname: + type: string + unique_id: + type: integer + format: int32 + alarm_id: + type: integer + format: int32 + alarm_event_id: + type: integer + format: int32 + name: + type: string + chart: + type: string + family: + type: string + processed: + type: boolean + updated: + type: boolean + exec_run: + type: integer + format: int32 + exec_failed: + type: boolean + exec: + type: string + recipient: + type: string + exec_code: + type: integer + format: int32 + source: + type: string + units: + type: string + when: + type: integer + format: int32 + duration: + type: integer + format: int32 + non_clear_duration: + type: integer + format: int32 + status: + type: string + old_status: + type: string + delay: + type: integer + format: int32 + delay_up_to_timestamp: + type: integer + format: int32 + updated_by_id: + type: integer + format: int32 + updates_id: + type: integer + format: int32 + value_string: + type: string + old_value_string: + type: string + silenced: + type: string + info: + type: string + value: + type: number + nullable: true + old_value: + type: number + nullable: true + alarms_values: + type: object + properties: + hostname: + type: string + alarms: + type: object + description: HashMap with keys being alarm names + additionalProperties: + type: object + properties: + id: + type: integer + value: + type: integer + status: + type: string + enum: + - REMOVED + - UNDEFINED + - UNINITIALIZED + - CLEAR + - RAISED + - WARNING + - CRITICAL + - UNKNOWN diff --git a/web/api/queries/Makefile.am b/web/api/queries/Makefile.am new file mode 100644 index 0000000..34bfdb8 --- /dev/null +++ b/web/api/queries/Makefile.am @@ -0,0 +1,20 @@ +# SPDX-License-Identifier: GPL-3.0-or-later + +AUTOMAKE_OPTIONS = subdir-objects +MAINTAINERCLEANFILES = $(srcdir)/Makefile.in + +SUBDIRS = \ + average \ + des \ + incremental_sum \ + max \ + min \ + sum \ + median \ + ses \ + stddev \ + $(NULL) + +dist_noinst_DATA = \ + README.md \ + $(NULL) diff --git a/web/api/queries/README.md b/web/api/queries/README.md new file mode 100644 index 0000000..31ec496 --- /dev/null +++ b/web/api/queries/README.md @@ -0,0 +1,176 @@ +<!-- +title: "Database Queries" +custom_edit_url: https://github.com/netdata/netdata/edit/master/web/api/queries/README.md +--> + +# Database Queries + +Netdata database can be queried with `/api/v1/data` and `/api/v1/badge.svg` REST API methods. + +Every data query accepts the following parameters: + +|name|required|description| +|:--:|:------:|:----------| +|`chart`|yes|The chart to be queried.| +|`points`|no|The number of points to be returned. Netdata can reduce number of points by applying query grouping methods. If not given, the result will have the same granularity as the database (although this relates to `gtime`).| +|`before`|no|The absolute timestamp or the relative (to now) time the query should finish evaluating data. If not given, it defaults to the timestamp of the latest point in the database.| +|`after`|no|The absolute timestamp or the relative (to `before`) time the query should start evaluating data. if not given, it defaults to the timestamp of the oldest point in the database.| +|`group`|no|The grouping method to use when reducing the points the database has. If not given, it defaults to `average`.| +|`gtime`|no|A resampling period to change the units of the metrics (i.e. setting this to `60` will convert `per second` metrics to `per minute`. If not given it defaults to granularity of the database.| +|`options`|no|A bitmap of options that can affect the operation of the query. Only 2 options are used by the query engine: `unaligned` and `percentage`. All the other options are used by the output formatters. The default is to return aligned data.| +|`dimensions`|no|A simple pattern to filter the dimensions to be queried. The default is to return all the dimensions of the chart.| + +## Operation + +The query engine works as follows (in this order): + +#### Time-frame + +`after` and `before` define a time-frame, accepting: + +- **absolute timestamps** (unix timestamps, i.e. seconds since epoch). + +- **relative timestamps**: + + `before` is relative to now and `after` is relative to `before`. + + Example: `before=-60&after=-60` evaluates to the time-frame from -120 up to -60 seconds in + the past, relative to the latest entry of the database of the chart. + +The engine verifies that the time-frame requested is available at the database: + +- If the requested time-frame overlaps with the database, the excess requested + will be truncated. + +- If the requested time-frame does not overlap with the database, the engine will + return an empty data set. + +At the end of this operation, `after` and `before` are absolute timestamps. + +#### Data grouping + +Database points grouping is applied when the caller requests a time-frame to be +expressed with fewer points, compared to what is available at the database. + +There are 2 uses that enable this feature: + +- The caller requests a specific number of `points` to be returned. + + For example, for a time-frame of 10 minutes, the database has 600 points (1/sec), + while the caller requested these 10 minutes to be expressed in 200 points. + + This feature is used by Netdata dashboards when you zoom-out the charts. + The dashboard is requesting the number of points the user's screen has. + This saves bandwidth and speeds up the browser (fewer points to evaluate for drawing the charts). +- The caller requests a **re-sampling** of the database, by setting `gtime` to any value + above the granularity of the chart. + + For example, the chart's units is `requests/sec` and caller wants `requests/min`. + +Using `points` and `gtime` the query engine tries to find a best fit for **database-points** +vs **result-points** (we call this ratio `group points`). It always tries to keep `group points` +an integer. Keep in mind the query engine may shift `after` if required. See also the [example](#example). + +#### Time-frame Alignment + +Alignment is a very important aspect of Netdata queries. Without it, the animated +charts on the dashboards would constantly [change shape](#example) during incremental updates. + +To provide consistent grouping through time, the query engine (by default) aligns +`after` and `before` to be a multiple of `group points`. + +For example, if `group points` is 60 and alignment is enabled, the engine will return +each point with durations XX:XX:00 - XX:XX:59, matching whole minutes. + +To disable alignment, pass `&options=unaligned` to the query. + +#### Query Execution + +To execute the query, the engine evaluates all dimensions of the chart, one after another. + +The engine does not evaluate dimensions that do not match the [simple pattern](/libnetdata/simple_pattern/README.md) +given at the `dimensions` parameter, except when `options=percentage` is given (this option +requires all the dimensions to be evaluated to find the percentage of each dimension vs to chart +total). + +For each dimension, it starts evaluating values starting at `after` (not inclusive) towards +`before` (inclusive). + +For each value it calls the **grouping method** given with the `&group=` query parameter +(the default is `average`). + +## Grouping methods + +The following grouping methods are supported. These are given all the values in the time-frame +and they group the values every `group points`. + +-  finds the minimum value +-  finds the maximum value +-  finds the average value +-  adds all the values and returns the sum +-  sorts the values and returns the value in the middle of the list +-  finds the standard deviation of the values +-  finds the relative standard deviation (coefficient of variation) of the values +-  finds the exponential weighted moving average of the values +-  applies Holt-Winters double exponential smoothing +-  finds the difference of the last vs the first value + +The examples shown above, are live information from the `successful` web requests of the global Netdata registry. + +## Further processing + +The result of the query engine is always a structure that has dimensions and values +for each dimension. + +Formatting modules are then used to convert this result in many different formats and return it +to the caller. + +## Performance + +The query engine is highly optimized for speed. Most of its modules implement "online" +versions of the algorithms, requiring just one pass on the database values to produce +the result. + +## Example + +When Netdata is reducing metrics, it tries to return always the same boundaries. So, if we want 10s averages, it will always return points starting at a `unix timestamp % 10 = 0`. + +Let's see why this is needed, by looking at the error case. + +Assume we have 5 points: + +|time|value| +|:--:|:---:| +|00:01|1| +|00:02|2| +|00:03|3| +|00:04|4| +|00:05|5| + +At 00:04 you ask for 2 points for 4 seconds in the past. So `group = 2`. Netdata would return: + +|point|time|value| +|:---:|:--:|:---:| +|1|00:01 - 00:02|1.5| +|2|00:03 - 00:04|3.5| + +A second later the chart is to be refreshed, and makes again the same request at 00:05. These are the points that would have been returned: + +|point|time|value| +|:---:|:--:|:---:| +|1|00:02 - 00:03|2.5| +|2|00:04 - 00:05|4.5| + +**Wait a moment!** The chart was shifted just one point and it changed value! Point 2 was 3.5 and when shifted to point 1 is 2.5! If you see this in a chart, it's a mess. The charts change shape constantly. + +For this reason, Netdata always aligns the data it returns to the `group`. + +When you request `points=1`, Netdata understands that you need 1 point for the whole database, so `group = 3600`. Then it tries to find the starting point which would be `timestamp % 3600 = 0` Within a database of 3600 seconds, there is one such point for sure. Then it tries to find the average of 3600 points. But, most probably it will not find 3600 of them (for just 1 out of 3600 seconds this query will return something). + +So, the proper way to query the database is to also set at least `after`. The following call will returns 1 point for the last complete 10-second duration (it starts at `timestamp % 10 = 0`): + +<http://netdata.firehol.org/api/v1/data?chart=system.cpu&points=1&after=-10&options=seconds> + +When you keep calling this URL, you will see that it returns one new value every 10 seconds, and the timestamp always ends with zero. Similarly, if you say `points=1&after=-5` it will always return timestamps ending with 0 or 5. + +[](<>) diff --git a/web/api/queries/average/Makefile.am b/web/api/queries/average/Makefile.am new file mode 100644 index 0000000..161784b --- /dev/null +++ b/web/api/queries/average/Makefile.am @@ -0,0 +1,8 @@ +# SPDX-License-Identifier: GPL-3.0-or-later + +AUTOMAKE_OPTIONS = subdir-objects +MAINTAINERCLEANFILES = $(srcdir)/Makefile.in + +dist_noinst_DATA = \ + README.md \ + $(NULL) diff --git a/web/api/queries/average/README.md b/web/api/queries/average/README.md new file mode 100644 index 0000000..f32a675 --- /dev/null +++ b/web/api/queries/average/README.md @@ -0,0 +1,46 @@ +<!-- +title: "Average or Mean" +custom_edit_url: https://github.com/netdata/netdata/edit/master/web/api/queries/average/README.md +--> + +# Average or Mean + +> This query is available as `average` and `mean`. + +An average is a single number taken as representative of a list of numbers. + +It is calculated as: + +``` +average = sum(numbers) / count(numbers) +``` + +## how to use + +Use it in alarms like this: + +``` + alarm: my_alarm + on: my_chart +lookup: average -1m unaligned of my_dimension + warn: $this > 1000 +``` + +`average` does not change the units. For example, if the chart units is `requests/sec`, the result +will be again expressed in the same units. + +It can also be used in APIs and badges as `&group=average` in the URL. + +## Examples + +Examining last 1 minute `successful` web server responses: + +-  +-  +-  + +## References + +- <https://en.wikipedia.org/wiki/Average>. + +[](<>) diff --git a/web/api/queries/average/average.c b/web/api/queries/average/average.c new file mode 100644 index 0000000..2c64358 --- /dev/null +++ b/web/api/queries/average/average.c @@ -0,0 +1,62 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#include "average.h" + +// ---------------------------------------------------------------------------- +// average + +struct grouping_average { + calculated_number sum; + size_t count; +}; + +void *grouping_create_average(RRDR *r) { + (void)r; + return callocz(1, sizeof(struct grouping_average)); +} + +// resets when switches dimensions +// so, clear everything to restart +void grouping_reset_average(RRDR *r) { + struct grouping_average *g = (struct grouping_average *)r->internal.grouping_data; + g->sum = 0; + g->count = 0; +} + +void grouping_free_average(RRDR *r) { + freez(r->internal.grouping_data); + r->internal.grouping_data = NULL; +} + +void grouping_add_average(RRDR *r, calculated_number value) { + if(!isnan(value)) { + struct grouping_average *g = (struct grouping_average *)r->internal.grouping_data; + g->sum += value; + g->count++; + } +} + +calculated_number grouping_flush_average(RRDR *r, RRDR_VALUE_FLAGS *rrdr_value_options_ptr) { + struct grouping_average *g = (struct grouping_average *)r->internal.grouping_data; + + calculated_number value; + + if(unlikely(!g->count)) { + value = 0.0; + *rrdr_value_options_ptr |= RRDR_VALUE_EMPTY; + } + else { + if(unlikely(r->internal.resampling_group != 1)) { + if (unlikely(r->result_options & RRDR_RESULT_OPTION_VARIABLE_STEP)) + value = g->sum / g->count / r->internal.resampling_divisor; + else + value = g->sum / r->internal.resampling_divisor; + } else + value = g->sum / g->count; + } + + g->sum = 0.0; + g->count = 0; + + return value; +} diff --git a/web/api/queries/average/average.h b/web/api/queries/average/average.h new file mode 100644 index 0000000..9fb7de2 --- /dev/null +++ b/web/api/queries/average/average.h @@ -0,0 +1,15 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#ifndef NETDATA_API_QUERY_AVERAGE_H +#define NETDATA_API_QUERY_AVERAGE_H + +#include "../query.h" +#include "../rrdr.h" + +extern void *grouping_create_average(RRDR *r); +extern void grouping_reset_average(RRDR *r); +extern void grouping_free_average(RRDR *r); +extern void grouping_add_average(RRDR *r, calculated_number value); +extern calculated_number grouping_flush_average(RRDR *r, RRDR_VALUE_FLAGS *rrdr_value_options_ptr); + +#endif //NETDATA_API_QUERY_AVERAGE_H diff --git a/web/api/queries/des/Makefile.am b/web/api/queries/des/Makefile.am new file mode 100644 index 0000000..161784b --- /dev/null +++ b/web/api/queries/des/Makefile.am @@ -0,0 +1,8 @@ +# SPDX-License-Identifier: GPL-3.0-or-later + +AUTOMAKE_OPTIONS = subdir-objects +MAINTAINERCLEANFILES = $(srcdir)/Makefile.in + +dist_noinst_DATA = \ + README.md \ + $(NULL) diff --git a/web/api/queries/des/README.md b/web/api/queries/des/README.md new file mode 100644 index 0000000..5505de5 --- /dev/null +++ b/web/api/queries/des/README.md @@ -0,0 +1,73 @@ +<!-- +title: "double exponential smoothing" +custom_edit_url: https://github.com/netdata/netdata/edit/master/web/api/queries/des/README.md +--> + +# double exponential smoothing + +Exponential smoothing is one of many window functions commonly applied to smooth data in signal +processing, acting as low-pass filters to remove high frequency noise. + +Simple exponential smoothing does not do well when there is a trend in the data. +In such situations, several methods were devised under the name "double exponential smoothing" +or "second-order exponential smoothing.", which is the recursive application of an exponential +filter twice, thus being termed "double exponential smoothing". + +In simple terms, this is like an average value, but more recent values are given more weight +and the trend of the values influences significantly the result. + +> **IMPORTANT** +> +> It is common for `des` to provide "average" values that far beyond the minimum or the maximum +> values found in the time-series. +> `des` estimates these values because of it takes into account the trend. + +This module implements the "Holt-Winters double exponential smoothing". + +Netdata automatically adjusts the weight (`alpha`) and the trend (`beta`) based on the number +of values processed, using the formula: + +``` +window = max(number of values, 15) +alpha = 2 / (window + 1) +beta = 2 / (window + 1) +``` + +You can change the fixed value `15` by setting in `netdata.conf`: + +``` +[web] + des max window = 15 +``` + +## how to use + +Use it in alarms like this: + +``` + alarm: my_alarm + on: my_chart +lookup: des -1m unaligned of my_dimension + warn: $this > 1000 +``` + +`des` does not change the units. For example, if the chart units is `requests/sec`, the result +will be again expressed in the same units. + +It can also be used in APIs and badges as `&group=des` in the URL. + +## Examples + +Examining last 1 minute `successful` web server responses: + +-  +-  +-  +-  +-  + +## References + +- <https://en.wikipedia.org/wiki/Exponential_smoothing>. + +[](<>) diff --git a/web/api/queries/des/des.c b/web/api/queries/des/des.c new file mode 100644 index 0000000..c6236f3 --- /dev/null +++ b/web/api/queries/des/des.c @@ -0,0 +1,139 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#include <web/api/queries/rrdr.h> +#include "des.h" + + +// ---------------------------------------------------------------------------- +// single exponential smoothing + +struct grouping_des { + calculated_number alpha; + calculated_number alpha_other; + calculated_number beta; + calculated_number beta_other; + + calculated_number level; + calculated_number trend; + + size_t count; +}; + +static size_t max_window_size = 15; + +void grouping_init_des(void) { + long long ret = config_get_number(CONFIG_SECTION_WEB, "des max window", (long long)max_window_size); + if(ret <= 1) { + config_set_number(CONFIG_SECTION_WEB, "des max window", (long long)max_window_size); + } + else { + max_window_size = (size_t) ret; + } +} + +static inline calculated_number window(RRDR *r, struct grouping_des *g) { + (void)g; + + calculated_number points; + if(r->group == 1) { + // provide a running DES + points = r->internal.points_wanted; + } + else { + // provide a SES with flush points + points = r->group; + } + + // https://en.wikipedia.org/wiki/Moving_average#Exponential_moving_average + // A commonly used value for alpha is 2 / (N + 1) + return (points > max_window_size) ? max_window_size : points; +} + +static inline void set_alpha(RRDR *r, struct grouping_des *g) { + // https://en.wikipedia.org/wiki/Moving_average#Exponential_moving_average + // A commonly used value for alpha is 2 / (N + 1) + + g->alpha = 2.0 / (window(r, g) + 1.0); + g->alpha_other = 1.0 - g->alpha; + + //info("alpha for chart '%s' is " CALCULATED_NUMBER_FORMAT, r->st->name, g->alpha); +} + +static inline void set_beta(RRDR *r, struct grouping_des *g) { + // https://en.wikipedia.org/wiki/Moving_average#Exponential_moving_average + // A commonly used value for alpha is 2 / (N + 1) + + g->beta = 2.0 / (window(r, g) + 1.0); + g->beta_other = 1.0 - g->beta; + + //info("beta for chart '%s' is " CALCULATED_NUMBER_FORMAT, r->st->name, g->beta); +} + +void *grouping_create_des(RRDR *r) { + struct grouping_des *g = (struct grouping_des *)malloc(sizeof(struct grouping_des)); + set_alpha(r, g); + set_beta(r, g); + g->level = 0.0; + g->trend = 0.0; + g->count = 0; + return g; +} + +// resets when switches dimensions +// so, clear everything to restart +void grouping_reset_des(RRDR *r) { + struct grouping_des *g = (struct grouping_des *)r->internal.grouping_data; + g->level = 0.0; + g->trend = 0.0; + g->count = 0; + + // fprintf(stderr, "\nDES: "); + +} + +void grouping_free_des(RRDR *r) { + freez(r->internal.grouping_data); + r->internal.grouping_data = NULL; +} + +void grouping_add_des(RRDR *r, calculated_number value) { + struct grouping_des *g = (struct grouping_des *)r->internal.grouping_data; + + if(calculated_number_isnumber(value)) { + if(likely(g->count > 0)) { + // we have at least a number so far + + if(unlikely(g->count == 1)) { + // the second value we got + g->trend = value - g->trend; + g->level = value; + } + + // for the values, except the first + calculated_number last_level = g->level; + g->level = (g->alpha * value) + (g->alpha_other * (g->level + g->trend)); + g->trend = (g->beta * (g->level - last_level)) + (g->beta_other * g->trend); + } + else { + // the first value we got + g->level = g->trend = value; + } + + g->count++; + } + + //fprintf(stderr, "value: " CALCULATED_NUMBER_FORMAT ", level: " CALCULATED_NUMBER_FORMAT ", trend: " CALCULATED_NUMBER_FORMAT "\n", value, g->level, g->trend); +} + +calculated_number grouping_flush_des(RRDR *r, RRDR_VALUE_FLAGS *rrdr_value_options_ptr) { + struct grouping_des *g = (struct grouping_des *)r->internal.grouping_data; + + if(unlikely(!g->count || !calculated_number_isnumber(g->level))) { + *rrdr_value_options_ptr |= RRDR_VALUE_EMPTY; + return 0.0; + } + + //fprintf(stderr, " RESULT for %zu values = " CALCULATED_NUMBER_FORMAT " \n", g->count, g->level); + + return g->level; +} diff --git a/web/api/queries/des/des.h b/web/api/queries/des/des.h new file mode 100644 index 0000000..360513e --- /dev/null +++ b/web/api/queries/des/des.h @@ -0,0 +1,17 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#ifndef NETDATA_API_QUERIES_DES_H +#define NETDATA_API_QUERIES_DES_H + +#include "../query.h" +#include "../rrdr.h" + +extern void grouping_init_des(void); + +extern void *grouping_create_des(RRDR *r); +extern void grouping_reset_des(RRDR *r); +extern void grouping_free_des(RRDR *r); +extern void grouping_add_des(RRDR *r, calculated_number value); +extern calculated_number grouping_flush_des(RRDR *r, RRDR_VALUE_FLAGS *rrdr_value_options_ptr); + +#endif //NETDATA_API_QUERIES_DES_H diff --git a/web/api/queries/incremental_sum/Makefile.am b/web/api/queries/incremental_sum/Makefile.am new file mode 100644 index 0000000..161784b --- /dev/null +++ b/web/api/queries/incremental_sum/Makefile.am @@ -0,0 +1,8 @@ +# SPDX-License-Identifier: GPL-3.0-or-later + +AUTOMAKE_OPTIONS = subdir-objects +MAINTAINERCLEANFILES = $(srcdir)/Makefile.in + +dist_noinst_DATA = \ + README.md \ + $(NULL) diff --git a/web/api/queries/incremental_sum/README.md b/web/api/queries/incremental_sum/README.md new file mode 100644 index 0000000..e5f3dfc --- /dev/null +++ b/web/api/queries/incremental_sum/README.md @@ -0,0 +1,41 @@ +<!-- +title: "Incremental Sum (`incremental_sum`)" +custom_edit_url: https://github.com/netdata/netdata/edit/master/web/api/queries/incremental_sum/README.md +--> + +# Incremental Sum (`incremental_sum`) + +This modules finds the incremental sum of a period, which `last value - first value`. + +The result may be positive (rising) or negative (falling) depending on the first and last values. + +## how to use + +Use it in alarms like this: + +``` + alarm: my_alarm + on: my_chart +lookup: incremental_sum -1m unaligned of my_dimension + warn: $this > 1000 +``` + +`incremental_sum` does not change the units. For example, if the chart units is `requests/sec`, the result +will be again expressed in the same units. + +It can also be used in APIs and badges as `&group=incremental_sum` in the URL. + +## Examples + +Examining last 1 minute `successful` web server responses: + +-  +-  +-  +-  + +## References + +- none + +[](<>) diff --git a/web/api/queries/incremental_sum/incremental_sum.c b/web/api/queries/incremental_sum/incremental_sum.c new file mode 100644 index 0000000..131d85d --- /dev/null +++ b/web/api/queries/incremental_sum/incremental_sum.c @@ -0,0 +1,69 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#include "incremental_sum.h" + +// ---------------------------------------------------------------------------- +// incremental sum + +struct grouping_incremental_sum { + calculated_number first; + calculated_number last; + size_t count; +}; + +void *grouping_create_incremental_sum(RRDR *r) { + (void)r; + return callocz(1, sizeof(struct grouping_incremental_sum)); +} + +// resets when switches dimensions +// so, clear everything to restart +void grouping_reset_incremental_sum(RRDR *r) { + struct grouping_incremental_sum *g = (struct grouping_incremental_sum *)r->internal.grouping_data; + g->first = 0; + g->last = 0; + g->count = 0; +} + +void grouping_free_incremental_sum(RRDR *r) { + freez(r->internal.grouping_data); + r->internal.grouping_data = NULL; +} + +void grouping_add_incremental_sum(RRDR *r, calculated_number value) { + if(!isnan(value)) { + struct grouping_incremental_sum *g = (struct grouping_incremental_sum *)r->internal.grouping_data; + + if(unlikely(!g->count)) { + g->first = value; + g->count++; + } + else { + g->last = value; + g->count++; + } + } +} + +calculated_number grouping_flush_incremental_sum(RRDR *r, RRDR_VALUE_FLAGS *rrdr_value_options_ptr) { + struct grouping_incremental_sum *g = (struct grouping_incremental_sum *)r->internal.grouping_data; + + calculated_number value; + + if(unlikely(!g->count)) { + value = 0.0; + *rrdr_value_options_ptr |= RRDR_VALUE_EMPTY; + } + else if(unlikely(g->count == 1)) { + value = 0.0; + } + else { + value = g->last - g->first; + } + + g->first = 0.0; + g->last = 0.0; + g->count = 0; + + return value; +} diff --git a/web/api/queries/incremental_sum/incremental_sum.h b/web/api/queries/incremental_sum/incremental_sum.h new file mode 100644 index 0000000..990a2ac --- /dev/null +++ b/web/api/queries/incremental_sum/incremental_sum.h @@ -0,0 +1,15 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#ifndef NETDATA_API_QUERY_INCREMENTAL_SUM_H +#define NETDATA_API_QUERY_INCREMENTAL_SUM_H + +#include "../query.h" +#include "../rrdr.h" + +extern void *grouping_create_incremental_sum(RRDR *r); +extern void grouping_reset_incremental_sum(RRDR *r); +extern void grouping_free_incremental_sum(RRDR *r); +extern void grouping_add_incremental_sum(RRDR *r, calculated_number value); +extern calculated_number grouping_flush_incremental_sum(RRDR *r, RRDR_VALUE_FLAGS *rrdr_value_options_ptr); + +#endif //NETDATA_API_QUERY_INCREMENTAL_SUM_H diff --git a/web/api/queries/max/Makefile.am b/web/api/queries/max/Makefile.am new file mode 100644 index 0000000..161784b --- /dev/null +++ b/web/api/queries/max/Makefile.am @@ -0,0 +1,8 @@ +# SPDX-License-Identifier: GPL-3.0-or-later + +AUTOMAKE_OPTIONS = subdir-objects +MAINTAINERCLEANFILES = $(srcdir)/Makefile.in + +dist_noinst_DATA = \ + README.md \ + $(NULL) diff --git a/web/api/queries/max/README.md b/web/api/queries/max/README.md new file mode 100644 index 0000000..32b1d43 --- /dev/null +++ b/web/api/queries/max/README.md @@ -0,0 +1,38 @@ +<!-- +title: "Max" +custom_edit_url: https://github.com/netdata/netdata/edit/master/web/api/queries/max/README.md +--> + +# Max + +This module finds the max value in the time-frame given. + +## how to use + +Use it in alarms like this: + +``` + alarm: my_alarm + on: my_chart +lookup: max -1m unaligned of my_dimension + warn: $this > 1000 +``` + +`max` does not change the units. For example, if the chart units is `requests/sec`, the result +will be again expressed in the same units. + +It can also be used in APIs and badges as `&group=max` in the URL. + +## Examples + +Examining last 1 minute `successful` web server responses: + +-  +-  +-  + +## References + +- <https://en.wikipedia.org/wiki/Sample_maximum_and_minimum>. + +[](<>) diff --git a/web/api/queries/max/max.c b/web/api/queries/max/max.c new file mode 100644 index 0000000..a4be36a --- /dev/null +++ b/web/api/queries/max/max.c @@ -0,0 +1,60 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#include "max.h" + +// ---------------------------------------------------------------------------- +// max + +struct grouping_max { + calculated_number max; + size_t count; +}; + +void *grouping_create_max(RRDR *r) { + (void)r; + return callocz(1, sizeof(struct grouping_max)); +} + +// resets when switches dimensions +// so, clear everything to restart +void grouping_reset_max(RRDR *r) { + struct grouping_max *g = (struct grouping_max *)r->internal.grouping_data; + g->max = 0; + g->count = 0; +} + +void grouping_free_max(RRDR *r) { + freez(r->internal.grouping_data); + r->internal.grouping_data = NULL; +} + +void grouping_add_max(RRDR *r, calculated_number value) { + if(!isnan(value)) { + struct grouping_max *g = (struct grouping_max *)r->internal.grouping_data; + + if(!g->count || calculated_number_fabs(value) > calculated_number_fabs(g->max)) { + g->max = value; + g->count++; + } + } +} + +calculated_number grouping_flush_max(RRDR *r, RRDR_VALUE_FLAGS *rrdr_value_options_ptr) { + struct grouping_max *g = (struct grouping_max *)r->internal.grouping_data; + + calculated_number value; + + if(unlikely(!g->count)) { + value = 0.0; + *rrdr_value_options_ptr |= RRDR_VALUE_EMPTY; + } + else { + value = g->max; + } + + g->max = 0.0; + g->count = 0; + + return value; +} + diff --git a/web/api/queries/max/max.h b/web/api/queries/max/max.h new file mode 100644 index 0000000..d839fe3 --- /dev/null +++ b/web/api/queries/max/max.h @@ -0,0 +1,15 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#ifndef NETDATA_API_QUERY_MAX_H +#define NETDATA_API_QUERY_MAX_H + +#include "../query.h" +#include "../rrdr.h" + +extern void *grouping_create_max(RRDR *r); +extern void grouping_reset_max(RRDR *r); +extern void grouping_free_max(RRDR *r); +extern void grouping_add_max(RRDR *r, calculated_number value); +extern calculated_number grouping_flush_max(RRDR *r, RRDR_VALUE_FLAGS *rrdr_value_options_ptr); + +#endif //NETDATA_API_QUERY_MAX_H diff --git a/web/api/queries/median/Makefile.am b/web/api/queries/median/Makefile.am new file mode 100644 index 0000000..161784b --- /dev/null +++ b/web/api/queries/median/Makefile.am @@ -0,0 +1,8 @@ +# SPDX-License-Identifier: GPL-3.0-or-later + +AUTOMAKE_OPTIONS = subdir-objects +MAINTAINERCLEANFILES = $(srcdir)/Makefile.in + +dist_noinst_DATA = \ + README.md \ + $(NULL) diff --git a/web/api/queries/median/README.md b/web/api/queries/median/README.md new file mode 100644 index 0000000..25ce8b8 --- /dev/null +++ b/web/api/queries/median/README.md @@ -0,0 +1,45 @@ +<!-- +title: "Median" +description: "Use median in API queries and health entities to find the 'middle' value from a sample, eliminating any unwanted spikes in the returned metrics." +custom_edit_url: https://github.com/netdata/netdata/edit/master/web/api/queries/median/README.md +--> + +# Median + +The median is the value separating the higher half from the lower half of a data sample +(a population or a probability distribution). For a data set, it may be thought of as the +"middle" value. + +`median` is not an accurate average. However, it eliminates all spikes, by sorting +all the values in a period, and selecting the value in the middle of the sorted array. + +## how to use + +Use it in alarms like this: + +``` + alarm: my_alarm + on: my_chart +lookup: median -1m unaligned of my_dimension + warn: $this > 1000 +``` + +`median` does not change the units. For example, if the chart units is `requests/sec`, the result +will be again expressed in the same units. + +It can also be used in APIs and badges as `&group=median` in the URL. + +## Examples + +Examining last 1 minute `successful` web server responses: + +-  +-  +-  +-  + +## References + +- <https://en.wikipedia.org/wiki/Median>. + +[](<>) diff --git a/web/api/queries/median/median.c b/web/api/queries/median/median.c new file mode 100644 index 0000000..31916c5 --- /dev/null +++ b/web/api/queries/median/median.c @@ -0,0 +1,79 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#include "median.h" + + +// ---------------------------------------------------------------------------- +// median + +struct grouping_median { + size_t series_size; + size_t next_pos; + + LONG_DOUBLE series[]; +}; + +void *grouping_create_median(RRDR *r) { + long entries = r->group; + if(entries < 0) entries = 0; + + struct grouping_median *g = (struct grouping_median *)callocz(1, sizeof(struct grouping_median) + entries * sizeof(LONG_DOUBLE)); + g->series_size = (size_t)entries; + + return g; +} + +// resets when switches dimensions +// so, clear everything to restart +void grouping_reset_median(RRDR *r) { + struct grouping_median *g = (struct grouping_median *)r->internal.grouping_data; + g->next_pos = 0; +} + +void grouping_free_median(RRDR *r) { + freez(r->internal.grouping_data); + r->internal.grouping_data = NULL; +} + +void grouping_add_median(RRDR *r, calculated_number value) { + struct grouping_median *g = (struct grouping_median *)r->internal.grouping_data; + + if(unlikely(g->next_pos >= g->series_size)) { + error("INTERNAL ERROR: median buffer overflow on chart '%s' - next_pos = %zu, series_size = %zu, r->group = %ld.", r->st->name, g->next_pos, g->series_size, r->group); + } + else { + if(calculated_number_isnumber(value)) + g->series[g->next_pos++] = (LONG_DOUBLE)value; + } +} + +calculated_number grouping_flush_median(RRDR *r, RRDR_VALUE_FLAGS *rrdr_value_options_ptr) { + struct grouping_median *g = (struct grouping_median *)r->internal.grouping_data; + + calculated_number value; + + if(unlikely(!g->next_pos)) { + value = 0.0; + *rrdr_value_options_ptr |= RRDR_VALUE_EMPTY; + } + else { + if(g->next_pos > 1) { + sort_series(g->series, g->next_pos); + value = (calculated_number)median_on_sorted_series(g->series, g->next_pos); + } + else + value = (calculated_number)g->series[0]; + + if(!calculated_number_isnumber(value)) { + value = 0.0; + *rrdr_value_options_ptr |= RRDR_VALUE_EMPTY; + } + + //log_series_to_stderr(g->series, g->next_pos, value, "median"); + } + + g->next_pos = 0; + + return value; +} + diff --git a/web/api/queries/median/median.h b/web/api/queries/median/median.h new file mode 100644 index 0000000..dd2c1ff --- /dev/null +++ b/web/api/queries/median/median.h @@ -0,0 +1,15 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#ifndef NETDATA_API_QUERIES_MEDIAN_H +#define NETDATA_API_QUERIES_MEDIAN_H + +#include "../query.h" +#include "../rrdr.h" + +extern void *grouping_create_median(RRDR *r); +extern void grouping_reset_median(RRDR *r); +extern void grouping_free_median(RRDR *r); +extern void grouping_add_median(RRDR *r, calculated_number value); +extern calculated_number grouping_flush_median(RRDR *r, RRDR_VALUE_FLAGS *rrdr_value_options_ptr); + +#endif //NETDATA_API_QUERIES_MEDIAN_H diff --git a/web/api/queries/min/Makefile.am b/web/api/queries/min/Makefile.am new file mode 100644 index 0000000..161784b --- /dev/null +++ b/web/api/queries/min/Makefile.am @@ -0,0 +1,8 @@ +# SPDX-License-Identifier: GPL-3.0-or-later + +AUTOMAKE_OPTIONS = subdir-objects +MAINTAINERCLEANFILES = $(srcdir)/Makefile.in + +dist_noinst_DATA = \ + README.md \ + $(NULL) diff --git a/web/api/queries/min/README.md b/web/api/queries/min/README.md new file mode 100644 index 0000000..69ef4ea --- /dev/null +++ b/web/api/queries/min/README.md @@ -0,0 +1,38 @@ +<!-- +title: "Min" +custom_edit_url: https://github.com/netdata/netdata/edit/master/web/api/queries/min/README.md +--> + +# Min + +This module finds the min value in the time-frame given. + +## how to use + +Use it in alarms like this: + +``` + alarm: my_alarm + on: my_chart +lookup: min -1m unaligned of my_dimension + warn: $this > 1000 +``` + +`min` does not change the units. For example, if the chart units is `requests/sec`, the result +will be again expressed in the same units. + +It can also be used in APIs and badges as `&group=min` in the URL. + +## Examples + +Examining last 1 minute `successful` web server responses: + +-  +-  +-  + +## References + +- <https://en.wikipedia.org/wiki/Sample_maximum_and_minimum>. + +[](<>) diff --git a/web/api/queries/min/min.c b/web/api/queries/min/min.c new file mode 100644 index 0000000..9bd7460 --- /dev/null +++ b/web/api/queries/min/min.c @@ -0,0 +1,60 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#include "min.h" + +// ---------------------------------------------------------------------------- +// min + +struct grouping_min { + calculated_number min; + size_t count; +}; + +void *grouping_create_min(RRDR *r) { + (void)r; + return callocz(1, sizeof(struct grouping_min)); +} + +// resets when switches dimensions +// so, clear everything to restart +void grouping_reset_min(RRDR *r) { + struct grouping_min *g = (struct grouping_min *)r->internal.grouping_data; + g->min = 0; + g->count = 0; +} + +void grouping_free_min(RRDR *r) { + freez(r->internal.grouping_data); + r->internal.grouping_data = NULL; +} + +void grouping_add_min(RRDR *r, calculated_number value) { + if(!isnan(value)) { + struct grouping_min *g = (struct grouping_min *)r->internal.grouping_data; + + if(!g->count || calculated_number_fabs(value) < calculated_number_fabs(g->min)) { + g->min = value; + g->count++; + } + } +} + +calculated_number grouping_flush_min(RRDR *r, RRDR_VALUE_FLAGS *rrdr_value_options_ptr) { + struct grouping_min *g = (struct grouping_min *)r->internal.grouping_data; + + calculated_number value; + + if(unlikely(!g->count)) { + value = 0.0; + *rrdr_value_options_ptr |= RRDR_VALUE_EMPTY; + } + else { + value = g->min; + } + + g->min = 0.0; + g->count = 0; + + return value; +} + diff --git a/web/api/queries/min/min.h b/web/api/queries/min/min.h new file mode 100644 index 0000000..7470360 --- /dev/null +++ b/web/api/queries/min/min.h @@ -0,0 +1,15 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#ifndef NETDATA_API_QUERY_MIN_H +#define NETDATA_API_QUERY_MIN_H + +#include "../query.h" +#include "../rrdr.h" + +extern void *grouping_create_min(RRDR *r); +extern void grouping_reset_min(RRDR *r); +extern void grouping_free_min(RRDR *r); +extern void grouping_add_min(RRDR *r, calculated_number value); +extern calculated_number grouping_flush_min(RRDR *r, RRDR_VALUE_FLAGS *rrdr_value_options_ptr); + +#endif //NETDATA_API_QUERY_MIN_H diff --git a/web/api/queries/query.c b/web/api/queries/query.c new file mode 100644 index 0000000..663e4bd --- /dev/null +++ b/web/api/queries/query.c @@ -0,0 +1,1636 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#include "query.h" +#include "web/api/formatters/rrd2json.h" +#include "rrdr.h" + +#include "average/average.h" +#include "incremental_sum/incremental_sum.h" +#include "max/max.h" +#include "median/median.h" +#include "min/min.h" +#include "sum/sum.h" +#include "stddev/stddev.h" +#include "ses/ses.h" +#include "des/des.h" + +// ---------------------------------------------------------------------------- + +static struct { + const char *name; + uint32_t hash; + RRDR_GROUPING value; + + // One time initialization for the module. + // This is called once, when netdata starts. + void (*init)(void); + + // Allocate all required structures for a query. + // This is called once for each netdata query. + void *(*create)(struct rrdresult *r); + + // Cleanup collected values, but don't destroy the structures. + // This is called when the query engine switches dimensions, + // as part of the same query (so same chart, switching metric). + void (*reset)(struct rrdresult *r); + + // Free all resources allocated for the query. + void (*free)(struct rrdresult *r); + + // Add a single value into the calculation. + // The module may decide to cache it, or use it in the fly. + void (*add)(struct rrdresult *r, calculated_number value); + + // Generate a single result for the values added so far. + // More values and points may be requested later. + // It is up to the module to reset its internal structures + // when flushing it (so for a few modules it may be better to + // continue after a flush as if nothing changed, for others a + // cleanup of the internal structures may be required). + calculated_number (*flush)(struct rrdresult *r, RRDR_VALUE_FLAGS *rrdr_value_options_ptr); +} api_v1_data_groups[] = { + {.name = "average", + .hash = 0, + .value = RRDR_GROUPING_AVERAGE, + .init = NULL, + .create= grouping_create_average, + .reset = grouping_reset_average, + .free = grouping_free_average, + .add = grouping_add_average, + .flush = grouping_flush_average + }, + {.name = "mean", // alias on 'average' + .hash = 0, + .value = RRDR_GROUPING_AVERAGE, + .init = NULL, + .create= grouping_create_average, + .reset = grouping_reset_average, + .free = grouping_free_average, + .add = grouping_add_average, + .flush = grouping_flush_average + }, + {.name = "incremental_sum", + .hash = 0, + .value = RRDR_GROUPING_INCREMENTAL_SUM, + .init = NULL, + .create= grouping_create_incremental_sum, + .reset = grouping_reset_incremental_sum, + .free = grouping_free_incremental_sum, + .add = grouping_add_incremental_sum, + .flush = grouping_flush_incremental_sum + }, + {.name = "incremental-sum", + .hash = 0, + .value = RRDR_GROUPING_INCREMENTAL_SUM, + .init = NULL, + .create= grouping_create_incremental_sum, + .reset = grouping_reset_incremental_sum, + .free = grouping_free_incremental_sum, + .add = grouping_add_incremental_sum, + .flush = grouping_flush_incremental_sum + }, + {.name = "median", + .hash = 0, + .value = RRDR_GROUPING_MEDIAN, + .init = NULL, + .create= grouping_create_median, + .reset = grouping_reset_median, + .free = grouping_free_median, + .add = grouping_add_median, + .flush = grouping_flush_median + }, + {.name = "min", + .hash = 0, + .value = RRDR_GROUPING_MIN, + .init = NULL, + .create= grouping_create_min, + .reset = grouping_reset_min, + .free = grouping_free_min, + .add = grouping_add_min, + .flush = grouping_flush_min + }, + {.name = "max", + .hash = 0, + .value = RRDR_GROUPING_MAX, + .init = NULL, + .create= grouping_create_max, + .reset = grouping_reset_max, + .free = grouping_free_max, + .add = grouping_add_max, + .flush = grouping_flush_max + }, + {.name = "sum", + .hash = 0, + .value = RRDR_GROUPING_SUM, + .init = NULL, + .create= grouping_create_sum, + .reset = grouping_reset_sum, + .free = grouping_free_sum, + .add = grouping_add_sum, + .flush = grouping_flush_sum + }, + + // standard deviation + {.name = "stddev", + .hash = 0, + .value = RRDR_GROUPING_STDDEV, + .init = NULL, + .create= grouping_create_stddev, + .reset = grouping_reset_stddev, + .free = grouping_free_stddev, + .add = grouping_add_stddev, + .flush = grouping_flush_stddev + }, + {.name = "cv", // coefficient variation is calculated by stddev + .hash = 0, + .value = RRDR_GROUPING_CV, + .init = NULL, + .create= grouping_create_stddev, // not an error, stddev calculates this too + .reset = grouping_reset_stddev, // not an error, stddev calculates this too + .free = grouping_free_stddev, // not an error, stddev calculates this too + .add = grouping_add_stddev, // not an error, stddev calculates this too + .flush = grouping_flush_coefficient_of_variation + }, + {.name = "rsd", // alias of 'cv' + .hash = 0, + .value = RRDR_GROUPING_CV, + .init = NULL, + .create= grouping_create_stddev, // not an error, stddev calculates this too + .reset = grouping_reset_stddev, // not an error, stddev calculates this too + .free = grouping_free_stddev, // not an error, stddev calculates this too + .add = grouping_add_stddev, // not an error, stddev calculates this too + .flush = grouping_flush_coefficient_of_variation + }, + + /* + {.name = "mean", // same as average, no need to define it again + .hash = 0, + .value = RRDR_GROUPING_MEAN, + .setup = NULL, + .create= grouping_create_stddev, + .reset = grouping_reset_stddev, + .free = grouping_free_stddev, + .add = grouping_add_stddev, + .flush = grouping_flush_mean + }, + */ + + /* + {.name = "variance", // meaningless to offer + .hash = 0, + .value = RRDR_GROUPING_VARIANCE, + .setup = NULL, + .create= grouping_create_stddev, + .reset = grouping_reset_stddev, + .free = grouping_free_stddev, + .add = grouping_add_stddev, + .flush = grouping_flush_variance + }, + */ + + // single exponential smoothing + {.name = "ses", + .hash = 0, + .value = RRDR_GROUPING_SES, + .init = grouping_init_ses, + .create= grouping_create_ses, + .reset = grouping_reset_ses, + .free = grouping_free_ses, + .add = grouping_add_ses, + .flush = grouping_flush_ses + }, + {.name = "ema", // alias for 'ses' + .hash = 0, + .value = RRDR_GROUPING_SES, + .init = NULL, + .create= grouping_create_ses, + .reset = grouping_reset_ses, + .free = grouping_free_ses, + .add = grouping_add_ses, + .flush = grouping_flush_ses + }, + {.name = "ewma", // alias for ses + .hash = 0, + .value = RRDR_GROUPING_SES, + .init = NULL, + .create= grouping_create_ses, + .reset = grouping_reset_ses, + .free = grouping_free_ses, + .add = grouping_add_ses, + .flush = grouping_flush_ses + }, + + // double exponential smoothing + {.name = "des", + .hash = 0, + .value = RRDR_GROUPING_DES, + .init = grouping_init_des, + .create= grouping_create_des, + .reset = grouping_reset_des, + .free = grouping_free_des, + .add = grouping_add_des, + .flush = grouping_flush_des + }, + + // terminator + {.name = NULL, + .hash = 0, + .value = RRDR_GROUPING_UNDEFINED, + .init = NULL, + .create= grouping_create_average, + .reset = grouping_reset_average, + .free = grouping_free_average, + .add = grouping_add_average, + .flush = grouping_flush_average + } +}; + +void web_client_api_v1_init_grouping(void) { + int i; + + for(i = 0; api_v1_data_groups[i].name ; i++) { + api_v1_data_groups[i].hash = simple_hash(api_v1_data_groups[i].name); + + if(api_v1_data_groups[i].init) + api_v1_data_groups[i].init(); + } +} + +const char *group_method2string(RRDR_GROUPING group) { + int i; + + for(i = 0; api_v1_data_groups[i].name ; i++) { + if(api_v1_data_groups[i].value == group) { + return api_v1_data_groups[i].name; + } + } + + return "unknown-group-method"; +} + +RRDR_GROUPING web_client_api_request_v1_data_group(const char *name, RRDR_GROUPING def) { + int i; + + uint32_t hash = simple_hash(name); + for(i = 0; api_v1_data_groups[i].name ; i++) + if(unlikely(hash == api_v1_data_groups[i].hash && !strcmp(name, api_v1_data_groups[i].name))) + return api_v1_data_groups[i].value; + + return def; +} + +// ---------------------------------------------------------------------------- + +static void rrdr_disable_not_selected_dimensions(RRDR *r, RRDR_OPTIONS options, const char *dims, RRDDIM *temp_rd) { + rrdset_check_rdlock(r->st); + + if(unlikely(!dims || !*dims || (dims[0] == '*' && dims[1] == '\0'))) return; + + int match_ids = 0, match_names = 0; + + if(unlikely(options & RRDR_OPTION_MATCH_IDS)) + match_ids = 1; + if(unlikely(options & RRDR_OPTION_MATCH_NAMES)) + match_names = 1; + + if(likely(!match_ids && !match_names)) + match_ids = match_names = 1; + + SIMPLE_PATTERN *pattern = simple_pattern_create(dims, ",|\t\r\n\f\v", SIMPLE_PATTERN_EXACT); + + RRDDIM *d; + long c, dims_selected = 0, dims_not_hidden_not_zero = 0; + for(c = 0, d = temp_rd?temp_rd:r->st->dimensions; d ;c++, d = d->next) { + if( (match_ids && simple_pattern_matches(pattern, d->id)) + || (match_names && simple_pattern_matches(pattern, d->name)) + ) { + r->od[c] |= RRDR_DIMENSION_SELECTED; + if(unlikely(r->od[c] & RRDR_DIMENSION_HIDDEN)) r->od[c] &= ~RRDR_DIMENSION_HIDDEN; + dims_selected++; + + // since the user needs this dimension + // make it appear as NONZERO, to return it + // even if the dimension has only zeros + // unless option non_zero is set + if(unlikely(!(options & RRDR_OPTION_NONZERO))) + r->od[c] |= RRDR_DIMENSION_NONZERO; + + // count the visible dimensions + if(likely(r->od[c] & RRDR_DIMENSION_NONZERO)) + dims_not_hidden_not_zero++; + } + else { + r->od[c] |= RRDR_DIMENSION_HIDDEN; + if(unlikely(r->od[c] & RRDR_DIMENSION_SELECTED)) r->od[c] &= ~RRDR_DIMENSION_SELECTED; + } + } + simple_pattern_free(pattern); + + // check if all dimensions are hidden + if(unlikely(!dims_not_hidden_not_zero && dims_selected)) { + // there are a few selected dimensions + // but they are all zero + // enable the selected ones + // to avoid returning an empty chart + for(c = 0, d = temp_rd?temp_rd:r->st->dimensions; d ;c++, d = d->next) + if(unlikely(r->od[c] & RRDR_DIMENSION_SELECTED)) + r->od[c] |= RRDR_DIMENSION_NONZERO; + } +} + +// ---------------------------------------------------------------------------- +// helpers to find our way in RRDR + +static inline RRDR_VALUE_FLAGS *rrdr_line_options(RRDR *r, long rrdr_line) { + return &r->o[ rrdr_line * r->d ]; +} + +static inline calculated_number *rrdr_line_values(RRDR *r, long rrdr_line) { + return &r->v[ rrdr_line * r->d ]; +} + +static inline long rrdr_line_init(RRDR *r, time_t t, long rrdr_line) { + rrdr_line++; + + #ifdef NETDATA_INTERNAL_CHECKS + + if(unlikely(rrdr_line >= r->n)) + error("INTERNAL ERROR: requested to step above RRDR size for chart '%s'", r->st->name); + + if(unlikely(r->t[rrdr_line] != 0 && r->t[rrdr_line] != t)) + error("INTERNAL ERROR: overwriting the timestamp of RRDR line %zu from %zu to %zu, of chart '%s'", (size_t)rrdr_line, (size_t)r->t[rrdr_line], (size_t)t, r->st->name); + + #endif + + // save the time + r->t[rrdr_line] = t; + + return rrdr_line; +} + +static inline void rrdr_done(RRDR *r, long rrdr_line) { + r->rows = rrdr_line + 1; +} + + +// ---------------------------------------------------------------------------- +// fill RRDR for a single dimension + +static inline void do_dimension_variablestep( + RRDR *r + , long points_wanted + , RRDDIM *rd + , long dim_id_in_rrdr + , time_t after_wanted + , time_t before_wanted +){ +// RRDSET *st = r->st; + + time_t + now = after_wanted, + dt = r->update_every, + max_date = 0, + min_date = 0; + + long +// group_size = r->group, + points_added = 0, + values_in_group = 0, + values_in_group_non_zero = 0, + rrdr_line = -1; + + RRDR_VALUE_FLAGS + group_value_flags = RRDR_VALUE_NOTHING; + + struct rrddim_query_handle handle; + + calculated_number min = r->min, max = r->max; + size_t db_points_read = 0; + time_t db_now = now; + storage_number n_curr, n_prev = SN_EMPTY_SLOT; + calculated_number value; + + for(rd->state->query_ops.init(rd, &handle, now, before_wanted) ; points_added < points_wanted ; now += dt) { + // make sure we return data in the proper time range + if (unlikely(now > before_wanted)) { +#ifdef NETDATA_INTERNAL_CHECKS + r->internal.log = "stopped, because attempted to access the db after 'wanted before'"; +#endif + break; + } + if (unlikely(now < after_wanted)) { +#ifdef NETDATA_INTERNAL_CHECKS + r->internal.log = "skipped, because attempted to access the db before 'wanted after'"; +#endif + continue; + } + + while (now >= db_now && (!rd->state->query_ops.is_finished(&handle) || + does_storage_number_exist(n_prev))) { + value = NAN; + if (does_storage_number_exist(n_prev)) { + // use the previously read database value + n_curr = n_prev; + } else { + // read the value from the database + n_curr = rd->state->query_ops.next_metric(&handle, &db_now); + } + n_prev = SN_EMPTY_SLOT; + // db_now has a different value than above + if (likely(now >= db_now)) { + if (likely(does_storage_number_exist(n_curr))) { + value = unpack_storage_number(n_curr); + if (likely(value != 0.0)) + values_in_group_non_zero++; + + if (unlikely(did_storage_number_reset(n_curr))) + group_value_flags |= RRDR_VALUE_RESET; + } + } else { + // We must postpone processing the value and fill the result with gaps instead + if (likely(does_storage_number_exist(n_curr))) { + n_prev = n_curr; + } + } + // add this value to grouping + r->internal.grouping_add(r, value); + values_in_group++; + db_points_read++; + } + + if (0 == values_in_group) { + // add NAN to grouping + r->internal.grouping_add(r, NAN); + } + + rrdr_line = rrdr_line_init(r, now, rrdr_line); + + if(unlikely(!min_date)) min_date = now; + max_date = now; + + // find the place to store our values + RRDR_VALUE_FLAGS *rrdr_value_options_ptr = &r->o[rrdr_line * r->d + dim_id_in_rrdr]; + + // update the dimension options + if(likely(values_in_group_non_zero)) + r->od[dim_id_in_rrdr] |= RRDR_DIMENSION_NONZERO; + + // store the specific point options + *rrdr_value_options_ptr = group_value_flags; + + // store the value + value = r->internal.grouping_flush(r, rrdr_value_options_ptr); + r->v[rrdr_line * r->d + dim_id_in_rrdr] = value; + + if(likely(points_added || dim_id_in_rrdr)) { + // find the min/max across all dimensions + + if(unlikely(value < min)) min = value; + if(unlikely(value > max)) max = value; + + } + else { + // runs only when dim_id_in_rrdr == 0 && points_added == 0 + // so, on the first point added for the query. + min = max = value; + } + + points_added++; + values_in_group = 0; + group_value_flags = RRDR_VALUE_NOTHING; + values_in_group_non_zero = 0; + } + rd->state->query_ops.finalize(&handle); + + r->internal.db_points_read += db_points_read; + r->internal.result_points_generated += points_added; + + r->min = min; + r->max = max; + r->before = max_date; + r->after = min_date - (r->group - 1) * dt; + rrdr_done(r, rrdr_line); + + #ifdef NETDATA_INTERNAL_CHECKS + if(unlikely(r->rows != points_added)) + error("INTERNAL ERROR: %s.%s added %zu rows, but RRDR says I added %zu.", r->st->name, rd->name, (size_t)points_added, (size_t)r->rows); + #endif +} + +static inline void do_dimension_fixedstep( + RRDR *r + , long points_wanted + , RRDDIM *rd + , long dim_id_in_rrdr + , time_t after_wanted + , time_t before_wanted +){ + RRDSET *st = r->st; + + time_t + now = after_wanted, + dt = r->update_every / r->group, /* usually is st->update_every */ + max_date = 0, + min_date = 0; + + long + group_size = r->group, + points_added = 0, + values_in_group = 0, + values_in_group_non_zero = 0, + rrdr_line = -1; + + RRDR_VALUE_FLAGS + group_value_flags = RRDR_VALUE_NOTHING; + + struct rrddim_query_handle handle; + + calculated_number min = r->min, max = r->max; + size_t db_points_read = 0; + time_t db_now = now; + + for(rd->state->query_ops.init(rd, &handle, now, before_wanted) ; points_added < points_wanted ; now += dt) { + // make sure we return data in the proper time range + if(unlikely(now > before_wanted)) { +#ifdef NETDATA_INTERNAL_CHECKS + r->internal.log = "stopped, because attempted to access the db after 'wanted before'"; +#endif + break; + } + if(unlikely(now < after_wanted)) { +#ifdef NETDATA_INTERNAL_CHECKS + r->internal.log = "skipped, because attempted to access the db before 'wanted after'"; +#endif + continue; + } + // read the value from the database + //storage_number n = rd->values[slot]; +#ifdef NETDATA_INTERNAL_CHECKS + if ((rd->rrd_memory_mode != RRD_MEMORY_MODE_DBENGINE) && + (rrdset_time2slot(st, now) != (long unsigned)handle.slotted.slot)) { + error("INTERNAL CHECK: Unaligned query for %s, database slot: %lu, expected slot: %lu", rd->id, (long unsigned)handle.slotted.slot, rrdset_time2slot(st, now)); + } +#endif + db_now = now; // this is needed to set db_now in case the next_metric implementation does not set it + storage_number n = rd->state->query_ops.next_metric(&handle, &db_now); + if(unlikely(db_now > before_wanted)) { +#ifdef NETDATA_INTERNAL_CHECKS + r->internal.log = "stopped, because attempted to access the db after 'wanted before'"; +#endif + break; + } + for ( ; now <= db_now ; now += dt) { + calculated_number value = NAN; + if(likely(now >= db_now && does_storage_number_exist(n))) { +#if defined(NETDATA_INTERNAL_CHECKS) && defined(ENABLE_DBENGINE) + if ((rd->rrd_memory_mode == RRD_MEMORY_MODE_DBENGINE) && (now != handle.rrdeng.now)) { + error("INTERNAL CHECK: Unaligned query for %s, database time: %ld, expected time: %ld", rd->id, (long)handle.rrdeng.now, (long)now); + } +#endif + value = unpack_storage_number(n); + if(likely(value != 0.0)) + values_in_group_non_zero++; + + if(unlikely(did_storage_number_reset(n))) + group_value_flags |= RRDR_VALUE_RESET; + + } + + // add this value for grouping + r->internal.grouping_add(r, value); + values_in_group++; + db_points_read++; + + if(unlikely(values_in_group == group_size)) { + rrdr_line = rrdr_line_init(r, now, rrdr_line); + + if(unlikely(!min_date)) min_date = now; + max_date = now; + + // find the place to store our values + RRDR_VALUE_FLAGS *rrdr_value_options_ptr = &r->o[rrdr_line * r->d + dim_id_in_rrdr]; + + // update the dimension options + if(likely(values_in_group_non_zero)) + r->od[dim_id_in_rrdr] |= RRDR_DIMENSION_NONZERO; + + // store the specific point options + *rrdr_value_options_ptr = group_value_flags; + + // store the value + calculated_number value = r->internal.grouping_flush(r, rrdr_value_options_ptr); + r->v[rrdr_line * r->d + dim_id_in_rrdr] = value; + + if(likely(points_added || dim_id_in_rrdr)) { + // find the min/max across all dimensions + + if(unlikely(value < min)) min = value; + if(unlikely(value > max)) max = value; + + } + else { + // runs only when dim_id_in_rrdr == 0 && points_added == 0 + // so, on the first point added for the query. + min = max = value; + } + + points_added++; + values_in_group = 0; + group_value_flags = RRDR_VALUE_NOTHING; + values_in_group_non_zero = 0; + } + } + now = db_now; + } + rd->state->query_ops.finalize(&handle); + + r->internal.db_points_read += db_points_read; + r->internal.result_points_generated += points_added; + + r->min = min; + r->max = max; + r->before = max_date; + r->after = min_date - (r->group - 1) * dt; + rrdr_done(r, rrdr_line); + +#ifdef NETDATA_INTERNAL_CHECKS + if(unlikely(r->rows != points_added)) + error("INTERNAL ERROR: %s.%s added %zu rows, but RRDR says I added %zu.", r->st->name, rd->name, (size_t)points_added, (size_t)r->rows); +#endif +} + +// ---------------------------------------------------------------------------- +// fill RRDR for the whole chart + +#ifdef NETDATA_INTERNAL_CHECKS +static void rrd2rrdr_log_request_response_metdata(RRDR *r + , RRDR_GROUPING group_method + , int aligned + , long group + , long resampling_time + , long resampling_group + , time_t after_wanted + , time_t after_requested + , time_t before_wanted + , time_t before_requested + , long points_requested + , long points_wanted + //, size_t after_slot + //, size_t before_slot + , const char *msg + ) { + netdata_rwlock_rdlock(&r->st->rrdset_rwlock); + info("INTERNAL ERROR: rrd2rrdr() on %s update every %d with %s grouping %s (group: %ld, resampling_time: %ld, resampling_group: %ld), " + "after (got: %zu, want: %zu, req: %zu, db: %zu), " + "before (got: %zu, want: %zu, req: %zu, db: %zu), " + "duration (got: %zu, want: %zu, req: %zu, db: %zu), " + //"slot (after: %zu, before: %zu, delta: %zu), " + "points (got: %ld, want: %ld, req: %ld, db: %ld), " + "%s" + , r->st->name + , r->st->update_every + + // grouping + , (aligned) ? "aligned" : "unaligned" + , group_method2string(group_method) + , group + , resampling_time + , resampling_group + + // after + , (size_t)r->after + , (size_t)after_wanted + , (size_t)after_requested + , (size_t)rrdset_first_entry_t_nolock(r->st) + + // before + , (size_t)r->before + , (size_t)before_wanted + , (size_t)before_requested + , (size_t)rrdset_last_entry_t_nolock(r->st) + + // duration + , (size_t)(r->before - r->after + r->st->update_every) + , (size_t)(before_wanted - after_wanted + r->st->update_every) + , (size_t)(before_requested - after_requested) + , (size_t)((rrdset_last_entry_t_nolock(r->st) - rrdset_first_entry_t_nolock(r->st)) + r->st->update_every) + + // slot + /* + , after_slot + , before_slot + , (after_slot > before_slot) ? (r->st->entries - after_slot + before_slot) : (before_slot - after_slot) + */ + + // points + , r->rows + , points_wanted + , points_requested + , r->st->entries + + // message + , msg + ); + netdata_rwlock_unlock(&r->st->rrdset_rwlock); +} +#endif // NETDATA_INTERNAL_CHECKS + +// Returns 1 if an absolute period was requested or 0 if it was a relative period +static int rrdr_convert_before_after_to_absolute( + long long *after_requestedp + , long long *before_requestedp + , int update_every + , time_t first_entry_t + , time_t last_entry_t + , RRDR_OPTIONS options +) { + int absolute_period_requested = -1; + long long after_requested, before_requested; + + before_requested = *before_requestedp; + after_requested = *after_requestedp; + + if(before_requested == 0 && after_requested == 0) { + // dump the all the data + before_requested = last_entry_t; + after_requested = first_entry_t; + absolute_period_requested = 0; + } + + // allow relative for before (smaller than API_RELATIVE_TIME_MAX) + if(abs(before_requested) <= API_RELATIVE_TIME_MAX) { + if(abs(before_requested) % update_every) { + // make sure it is multiple of st->update_every + if(before_requested < 0) before_requested = before_requested - update_every - + before_requested % update_every; + else before_requested = before_requested + update_every - before_requested % update_every; + } + if(before_requested > 0) before_requested = first_entry_t + before_requested; + else before_requested = last_entry_t + before_requested; //last_entry_t is not really now_t + //TODO: fix before_requested to be relative to now_t + absolute_period_requested = 0; + } + + // allow relative for after (smaller than API_RELATIVE_TIME_MAX) + if(abs(after_requested) <= API_RELATIVE_TIME_MAX) { + if(after_requested == 0) after_requested = -update_every; + if(abs(after_requested) % update_every) { + // make sure it is multiple of st->update_every + if(after_requested < 0) after_requested = after_requested - update_every - after_requested % update_every; + else after_requested = after_requested + update_every - after_requested % update_every; + } + after_requested = before_requested + after_requested; + absolute_period_requested = 0; + } + + if(absolute_period_requested == -1) + absolute_period_requested = 1; + + // make sure they are within our timeframe + if(before_requested > last_entry_t) before_requested = last_entry_t; + if(before_requested < first_entry_t && !(options & RRDR_OPTION_ALLOW_PAST)) + before_requested = first_entry_t; + + if(after_requested > last_entry_t) after_requested = last_entry_t; + if(after_requested < first_entry_t && !(options & RRDR_OPTION_ALLOW_PAST)) + after_requested = first_entry_t; + + // check if they are reversed + if(after_requested > before_requested) { + time_t tmp = before_requested; + before_requested = after_requested; + after_requested = tmp; + } + + *before_requestedp = before_requested; + *after_requestedp = after_requested; + + return absolute_period_requested; +} + +static RRDR *rrd2rrdr_fixedstep( + RRDSET *st + , long points_requested + , long long after_requested + , long long before_requested + , RRDR_GROUPING group_method + , long resampling_time_requested + , RRDR_OPTIONS options + , const char *dimensions + , int update_every + , time_t first_entry_t + , time_t last_entry_t + , int absolute_period_requested + , struct context_param *context_param_list +) { + int aligned = !(options & RRDR_OPTION_NOT_ALIGNED); + + // the duration of the chart + time_t duration = before_requested - after_requested; + long available_points = duration / update_every; + + RRDDIM *temp_rd = context_param_list ? context_param_list->rd : NULL; + + if(duration <= 0 || available_points <= 0) + return rrdr_create(st, 1, context_param_list); + + // check the number of wanted points in the result + if(unlikely(points_requested < 0)) points_requested = -points_requested; + if(unlikely(points_requested > available_points)) points_requested = available_points; + if(unlikely(points_requested == 0)) points_requested = available_points; + + // calculate the desired grouping of source data points + long group = available_points / points_requested; + if(unlikely(group <= 0)) group = 1; + if(unlikely(available_points % points_requested > points_requested / 2)) group++; // rounding to the closest integer + + // resampling_time_requested enforces a certain grouping multiple + calculated_number resampling_divisor = 1.0; + long resampling_group = 1; + if(unlikely(resampling_time_requested > update_every)) { + if (unlikely(resampling_time_requested > duration)) { + // group_time is above the available duration + + #ifdef NETDATA_INTERNAL_CHECKS + info("INTERNAL CHECK: %s: requested gtime %ld secs, is greater than the desired duration %ld secs", st->id, resampling_time_requested, duration); + #endif + + after_requested = before_requested - resampling_time_requested; + duration = before_requested - after_requested; + available_points = duration / update_every; + group = available_points / points_requested; + } + + // if the duration is not aligned to resampling time + // extend the duration to the past, to avoid a gap at the chart + // only when the missing duration is above 1/10th of a point + if(duration % resampling_time_requested) { + time_t delta = duration % resampling_time_requested; + if(delta > resampling_time_requested / 10) { + after_requested -= resampling_time_requested - delta; + duration = before_requested - after_requested; + available_points = duration / update_every; + group = available_points / points_requested; + } + } + + // the points we should group to satisfy gtime + resampling_group = resampling_time_requested / update_every; + if(unlikely(resampling_time_requested % update_every)) { + #ifdef NETDATA_INTERNAL_CHECKS + info("INTERNAL CHECK: %s: requested gtime %ld secs, is not a multiple of the chart's data collection frequency %d secs", st->id, resampling_time_requested, update_every); + #endif + + resampling_group++; + } + + // adapt group according to resampling_group + if(unlikely(group < resampling_group)) group = resampling_group; // do not allow grouping below the desired one + if(unlikely(group % resampling_group)) group += resampling_group - (group % resampling_group); // make sure group is multiple of resampling_group + + //resampling_divisor = group / resampling_group; + resampling_divisor = (calculated_number)(group * update_every) / (calculated_number)resampling_time_requested; + } + + // now that we have group, + // align the requested timeframe to fit it. + + if(aligned) { + // alignement has been requested, so align the values + before_requested -= before_requested % (group * update_every); + after_requested -= after_requested % (group * update_every); + } + + // we align the request on requested_before + time_t before_wanted = before_requested; + if(likely(before_wanted > last_entry_t)) { + #ifdef NETDATA_INTERNAL_CHECKS + error("INTERNAL ERROR: rrd2rrdr() on %s, before_wanted is after db max", st->name); + #endif + + before_wanted = last_entry_t - (last_entry_t % ( ((aligned)?group:1) * update_every )); + } + //size_t before_slot = rrdset_time2slot(st, before_wanted); + + // we need to estimate the number of points, for having + // an integer number of values per point + long points_wanted = (before_wanted - after_requested) / (update_every * group); + + time_t after_wanted = before_wanted - (points_wanted * group * update_every) + update_every; + if(unlikely(after_wanted < first_entry_t)) { + // hm... we go to the past, calculate again points_wanted using all the db from before_wanted to the beginning + points_wanted = (before_wanted - first_entry_t) / group; + + // recalculate after wanted with the new number of points + after_wanted = before_wanted - (points_wanted * group * update_every) + update_every; + + if(unlikely(after_wanted < first_entry_t)) { + #ifdef NETDATA_INTERNAL_CHECKS + error("INTERNAL ERROR: rrd2rrdr() on %s, after_wanted is before db min", st->name); + #endif + + after_wanted = first_entry_t - (first_entry_t % ( ((aligned)?group:1) * update_every )) + ( ((aligned)?group:1) * update_every ); + } + } + //size_t after_slot = rrdset_time2slot(st, after_wanted); + + // check if they are reversed + if(unlikely(after_wanted > before_wanted)) { + #ifdef NETDATA_INTERNAL_CHECKS + error("INTERNAL ERROR: rrd2rrdr() on %s, reversed wanted after/before", st->name); + #endif + time_t tmp = before_wanted; + before_wanted = after_wanted; + after_wanted = tmp; + } + + // recalculate points_wanted using the final time-frame + points_wanted = (before_wanted - after_wanted) / update_every / group + 1; + if(unlikely(points_wanted < 0)) { + #ifdef NETDATA_INTERNAL_CHECKS + error("INTERNAL ERROR: rrd2rrdr() on %s, points_wanted is %ld", st->name, points_wanted); + #endif + points_wanted = 0; + } + +#ifdef NETDATA_INTERNAL_CHECKS + duration = before_wanted - after_wanted; + + if(after_wanted < first_entry_t) + error("INTERNAL CHECK: after_wanted %u is too small, minimum %u", (uint32_t)after_wanted, (uint32_t)first_entry_t); + + if(after_wanted > last_entry_t) + error("INTERNAL CHECK: after_wanted %u is too big, maximum %u", (uint32_t)after_wanted, (uint32_t)last_entry_t); + + if(before_wanted < first_entry_t) + error("INTERNAL CHECK: before_wanted %u is too small, minimum %u", (uint32_t)before_wanted, (uint32_t)first_entry_t); + + if(before_wanted > last_entry_t) + error("INTERNAL CHECK: before_wanted %u is too big, maximum %u", (uint32_t)before_wanted, (uint32_t)last_entry_t); + +/* + if(before_slot >= (size_t)st->entries) + error("INTERNAL CHECK: before_slot is invalid %zu, expected 0 to %ld", before_slot, st->entries - 1); + + if(after_slot >= (size_t)st->entries) + error("INTERNAL CHECK: after_slot is invalid %zu, expected 0 to %ld", after_slot, st->entries - 1); +*/ + + if(points_wanted > (before_wanted - after_wanted) / group / update_every + 1) + error("INTERNAL CHECK: points_wanted %ld is more than points %ld", points_wanted, (before_wanted - after_wanted) / group / update_every + 1); + + if(group < resampling_group) + error("INTERNAL CHECK: group %ld is less than the desired group points %ld", group, resampling_group); + + if(group > resampling_group && group % resampling_group) + error("INTERNAL CHECK: group %ld is not a multiple of the desired group points %ld", group, resampling_group); +#endif + + // ------------------------------------------------------------------------- + // initialize our result set + // this also locks the chart for us + + RRDR *r = rrdr_create(st, points_wanted, context_param_list); + if(unlikely(!r)) { + #ifdef NETDATA_INTERNAL_CHECKS + error("INTERNAL CHECK: Cannot create RRDR for %s, after=%u, before=%u, duration=%u, points=%ld", st->id, (uint32_t)after_wanted, (uint32_t)before_wanted, (uint32_t)duration, points_wanted); + #endif + return NULL; + } + + if(unlikely(!r->d || !points_wanted)) { + #ifdef NETDATA_INTERNAL_CHECKS + error("INTERNAL CHECK: Returning empty RRDR (no dimensions in RRDSET) for %s, after=%u, before=%u, duration=%zu, points=%ld", st->id, (uint32_t)after_wanted, (uint32_t)before_wanted, (size_t)duration, points_wanted); + #endif + return r; + } + + if(unlikely(absolute_period_requested == 1)) + r->result_options |= RRDR_RESULT_OPTION_ABSOLUTE; + else + r->result_options |= RRDR_RESULT_OPTION_RELATIVE; + + // find how many dimensions we have + long dimensions_count = r->d; + + // ------------------------------------------------------------------------- + // initialize RRDR + + r->group = group; + r->update_every = (int)group * update_every; + r->before = before_wanted; + r->after = after_wanted; + r->internal.points_wanted = points_wanted; + r->internal.resampling_group = resampling_group; + r->internal.resampling_divisor = resampling_divisor; + + + // ------------------------------------------------------------------------- + // assign the processor functions + + { + int i, found = 0; + for(i = 0; !found && api_v1_data_groups[i].name ;i++) { + if(api_v1_data_groups[i].value == group_method) { + r->internal.grouping_create= api_v1_data_groups[i].create; + r->internal.grouping_reset = api_v1_data_groups[i].reset; + r->internal.grouping_free = api_v1_data_groups[i].free; + r->internal.grouping_add = api_v1_data_groups[i].add; + r->internal.grouping_flush = api_v1_data_groups[i].flush; + found = 1; + } + } + if(!found) { + errno = 0; + #ifdef NETDATA_INTERNAL_CHECKS + error("INTERNAL ERROR: grouping method %u not found for chart '%s'. Using 'average'", (unsigned int)group_method, r->st->name); + #endif + r->internal.grouping_create= grouping_create_average; + r->internal.grouping_reset = grouping_reset_average; + r->internal.grouping_free = grouping_free_average; + r->internal.grouping_add = grouping_add_average; + r->internal.grouping_flush = grouping_flush_average; + } + } + + // allocate any memory required by the grouping method + r->internal.grouping_data = r->internal.grouping_create(r); + + + // ------------------------------------------------------------------------- + // disable the not-wanted dimensions + + rrdset_check_rdlock(st); + + if(dimensions) + rrdr_disable_not_selected_dimensions(r, options, dimensions, temp_rd); + + + // ------------------------------------------------------------------------- + // do the work for each dimension + + time_t max_after = 0, min_before = 0; + long max_rows = 0; + + RRDDIM *rd; + long c, dimensions_used = 0, dimensions_nonzero = 0; + for(rd = temp_rd?temp_rd:st->dimensions, c = 0 ; rd && c < dimensions_count ; rd = rd->next, c++) { + + // if we need a percentage, we need to calculate all dimensions + if(unlikely(!(options & RRDR_OPTION_PERCENTAGE) && (r->od[c] & RRDR_DIMENSION_HIDDEN))) { + if(unlikely(r->od[c] & RRDR_DIMENSION_SELECTED)) r->od[c] &= ~RRDR_DIMENSION_SELECTED; + continue; + } + r->od[c] |= RRDR_DIMENSION_SELECTED; + + // reset the grouping for the new dimension + r->internal.grouping_reset(r); + + do_dimension_fixedstep( + r + , points_wanted + , rd + , c + , after_wanted + , before_wanted + ); + + if(r->od[c] & RRDR_DIMENSION_NONZERO) + dimensions_nonzero++; + + // verify all dimensions are aligned + if(unlikely(!dimensions_used)) { + min_before = r->before; + max_after = r->after; + max_rows = r->rows; + } + else { + if(r->after != max_after) { + #ifdef NETDATA_INTERNAL_CHECKS + error("INTERNAL ERROR: 'after' mismatch between dimensions for chart '%s': max is %zu, dimension '%s' has %zu", + st->name, (size_t)max_after, rd->name, (size_t)r->after); + #endif + r->after = (r->after > max_after) ? r->after : max_after; + } + + if(r->before != min_before) { + #ifdef NETDATA_INTERNAL_CHECKS + error("INTERNAL ERROR: 'before' mismatch between dimensions for chart '%s': max is %zu, dimension '%s' has %zu", + st->name, (size_t)min_before, rd->name, (size_t)r->before); + #endif + r->before = (r->before < min_before) ? r->before : min_before; + } + + if(r->rows != max_rows) { + #ifdef NETDATA_INTERNAL_CHECKS + error("INTERNAL ERROR: 'rows' mismatch between dimensions for chart '%s': max is %zu, dimension '%s' has %zu", + st->name, (size_t)max_rows, rd->name, (size_t)r->rows); + #endif + r->rows = (r->rows > max_rows) ? r->rows : max_rows; + } + } + + dimensions_used++; + } + + #ifdef NETDATA_INTERNAL_CHECKS + if (dimensions_used) { + if(r->internal.log) + rrd2rrdr_log_request_response_metdata(r, group_method, aligned, group, resampling_time_requested, resampling_group, after_wanted, after_requested, before_wanted, before_requested, points_requested, points_wanted, /*after_slot, before_slot,*/ r->internal.log); + + if(r->rows != points_wanted) + rrd2rrdr_log_request_response_metdata(r, group_method, aligned, group, resampling_time_requested, resampling_group, after_wanted, after_requested, before_wanted, before_requested, points_requested, points_wanted, /*after_slot, before_slot,*/ "got 'points' is not wanted 'points'"); + + if(aligned && (r->before % group) != 0) + rrd2rrdr_log_request_response_metdata(r, group_method, aligned, group, resampling_time_requested, resampling_group, after_wanted, after_requested, before_wanted, before_requested, points_requested, points_wanted, /*after_slot, before_slot,*/ "'before' is not aligned but alignment is required"); + + // 'after' should not be aligned, since we start inside the first group + //if(aligned && (r->after % group) != 0) + // rrd2rrdr_log_request_response_metdata(r, group_method, aligned, group, resampling_time_requested, resampling_group, after_wanted, after_requested, before_wanted, before_requested, points_requested, points_wanted, after_slot, before_slot, "'after' is not aligned but alignment is required"); + + if(r->before != before_requested) + rrd2rrdr_log_request_response_metdata(r, group_method, aligned, group, resampling_time_requested, resampling_group, after_wanted, after_requested, before_wanted, before_requested, points_requested, points_wanted, /*after_slot, before_slot,*/ "chart is not aligned to requested 'before'"); + + if(r->before != before_wanted) + rrd2rrdr_log_request_response_metdata(r, group_method, aligned, group, resampling_time_requested, resampling_group, after_wanted, after_requested, before_wanted, before_requested, points_requested, points_wanted, /*after_slot, before_slot,*/ "got 'before' is not wanted 'before'"); + + // reported 'after' varies, depending on group + if(r->after != after_wanted) + rrd2rrdr_log_request_response_metdata(r, group_method, aligned, group, resampling_time_requested, resampling_group, after_wanted, after_requested, before_wanted, before_requested, points_requested, points_wanted, /*after_slot, before_slot,*/ "got 'after' is not wanted 'after'"); + } + #endif + + // free all resources used by the grouping method + r->internal.grouping_free(r); + + // when all the dimensions are zero, we should return all of them + if(unlikely(options & RRDR_OPTION_NONZERO && !dimensions_nonzero)) { + // all the dimensions are zero + // mark them as NONZERO to send them all + for(rd = temp_rd?temp_rd:st->dimensions, c = 0 ; rd && c < dimensions_count ; rd = rd->next, c++) { + if(unlikely(r->od[c] & RRDR_DIMENSION_HIDDEN)) continue; + r->od[c] |= RRDR_DIMENSION_NONZERO; + } + } + + rrdr_query_completed(r->internal.db_points_read, r->internal.result_points_generated); + return r; +} + +#ifdef ENABLE_DBENGINE +static RRDR *rrd2rrdr_variablestep( + RRDSET *st + , long points_requested + , long long after_requested + , long long before_requested + , RRDR_GROUPING group_method + , long resampling_time_requested + , RRDR_OPTIONS options + , const char *dimensions + , int update_every + , time_t first_entry_t + , time_t last_entry_t + , int absolute_period_requested + , struct rrdeng_region_info *region_info_array + , struct context_param *context_param_list +) { + int aligned = !(options & RRDR_OPTION_NOT_ALIGNED); + + // the duration of the chart + time_t duration = before_requested - after_requested; + long available_points = duration / update_every; + + RRDDIM *temp_rd = context_param_list ? context_param_list->rd : NULL; + + if(duration <= 0 || available_points <= 0) { + freez(region_info_array); + return rrdr_create(st, 1, context_param_list); + } + + // check the number of wanted points in the result + if(unlikely(points_requested < 0)) points_requested = -points_requested; + if(unlikely(points_requested > available_points)) points_requested = available_points; + if(unlikely(points_requested == 0)) points_requested = available_points; + + // calculate the desired grouping of source data points + long group = available_points / points_requested; + if(unlikely(group <= 0)) group = 1; + if(unlikely(available_points % points_requested > points_requested / 2)) group++; // rounding to the closest integer + + // resampling_time_requested enforces a certain grouping multiple + calculated_number resampling_divisor = 1.0; + long resampling_group = 1; + if(unlikely(resampling_time_requested > update_every)) { + if (unlikely(resampling_time_requested > duration)) { + // group_time is above the available duration + + #ifdef NETDATA_INTERNAL_CHECKS + info("INTERNAL CHECK: %s: requested gtime %ld secs, is greater than the desired duration %ld secs", st->id, resampling_time_requested, duration); + #endif + + after_requested = before_requested - resampling_time_requested; + duration = before_requested - after_requested; + available_points = duration / update_every; + group = available_points / points_requested; + } + + // if the duration is not aligned to resampling time + // extend the duration to the past, to avoid a gap at the chart + // only when the missing duration is above 1/10th of a point + if(duration % resampling_time_requested) { + time_t delta = duration % resampling_time_requested; + if(delta > resampling_time_requested / 10) { + after_requested -= resampling_time_requested - delta; + duration = before_requested - after_requested; + available_points = duration / update_every; + group = available_points / points_requested; + } + } + + // the points we should group to satisfy gtime + resampling_group = resampling_time_requested / update_every; + if(unlikely(resampling_time_requested % update_every)) { + #ifdef NETDATA_INTERNAL_CHECKS + info("INTERNAL CHECK: %s: requested gtime %ld secs, is not a multiple of the chart's data collection frequency %d secs", st->id, resampling_time_requested, update_every); + #endif + + resampling_group++; + } + + // adapt group according to resampling_group + if(unlikely(group < resampling_group)) group = resampling_group; // do not allow grouping below the desired one + if(unlikely(group % resampling_group)) group += resampling_group - (group % resampling_group); // make sure group is multiple of resampling_group + + //resampling_divisor = group / resampling_group; + resampling_divisor = (calculated_number)(group * update_every) / (calculated_number)resampling_time_requested; + } + + // now that we have group, + // align the requested timeframe to fit it. + + if(aligned) { + // alignement has been requested, so align the values + before_requested -= before_requested % (group * update_every); + after_requested -= after_requested % (group * update_every); + } + + // we align the request on requested_before + time_t before_wanted = before_requested; + if(likely(before_wanted > last_entry_t)) { + #ifdef NETDATA_INTERNAL_CHECKS + error("INTERNAL ERROR: rrd2rrdr() on %s, before_wanted is after db max", st->name); + #endif + + before_wanted = last_entry_t - (last_entry_t % ( ((aligned)?group:1) * update_every )); + } + //size_t before_slot = rrdset_time2slot(st, before_wanted); + + // we need to estimate the number of points, for having + // an integer number of values per point + long points_wanted = (before_wanted - after_requested) / (update_every * group); + + time_t after_wanted = before_wanted - (points_wanted * group * update_every) + update_every; + if(unlikely(after_wanted < first_entry_t)) { + // hm... we go to the past, calculate again points_wanted using all the db from before_wanted to the beginning + points_wanted = (before_wanted - first_entry_t) / group; + + // recalculate after wanted with the new number of points + after_wanted = before_wanted - (points_wanted * group * update_every) + update_every; + + if(unlikely(after_wanted < first_entry_t)) { + #ifdef NETDATA_INTERNAL_CHECKS + error("INTERNAL ERROR: rrd2rrdr() on %s, after_wanted is before db min", st->name); + #endif + + after_wanted = first_entry_t - (first_entry_t % ( ((aligned)?group:1) * update_every )) + ( ((aligned)?group:1) * update_every ); + } + } + //size_t after_slot = rrdset_time2slot(st, after_wanted); + + // check if they are reversed + if(unlikely(after_wanted > before_wanted)) { + #ifdef NETDATA_INTERNAL_CHECKS + error("INTERNAL ERROR: rrd2rrdr() on %s, reversed wanted after/before", st->name); + #endif + time_t tmp = before_wanted; + before_wanted = after_wanted; + after_wanted = tmp; + } + + // recalculate points_wanted using the final time-frame + points_wanted = (before_wanted - after_wanted) / update_every / group + 1; + if(unlikely(points_wanted < 0)) { + #ifdef NETDATA_INTERNAL_CHECKS + error("INTERNAL ERROR: rrd2rrdr() on %s, points_wanted is %ld", st->name, points_wanted); + #endif + points_wanted = 0; + } + +#ifdef NETDATA_INTERNAL_CHECKS + duration = before_wanted - after_wanted; + + if(after_wanted < first_entry_t) + error("INTERNAL CHECK: after_wanted %u is too small, minimum %u", (uint32_t)after_wanted, (uint32_t)first_entry_t); + + if(after_wanted > last_entry_t) + error("INTERNAL CHECK: after_wanted %u is too big, maximum %u", (uint32_t)after_wanted, (uint32_t)last_entry_t); + + if(before_wanted < first_entry_t) + error("INTERNAL CHECK: before_wanted %u is too small, minimum %u", (uint32_t)before_wanted, (uint32_t)first_entry_t); + + if(before_wanted > last_entry_t) + error("INTERNAL CHECK: before_wanted %u is too big, maximum %u", (uint32_t)before_wanted, (uint32_t)last_entry_t); + +/* + if(before_slot >= (size_t)st->entries) + error("INTERNAL CHECK: before_slot is invalid %zu, expected 0 to %ld", before_slot, st->entries - 1); + + if(after_slot >= (size_t)st->entries) + error("INTERNAL CHECK: after_slot is invalid %zu, expected 0 to %ld", after_slot, st->entries - 1); +*/ + + if(points_wanted > (before_wanted - after_wanted) / group / update_every + 1) + error("INTERNAL CHECK: points_wanted %ld is more than points %ld", points_wanted, (before_wanted - after_wanted) / group / update_every + 1); + + if(group < resampling_group) + error("INTERNAL CHECK: group %ld is less than the desired group points %ld", group, resampling_group); + + if(group > resampling_group && group % resampling_group) + error("INTERNAL CHECK: group %ld is not a multiple of the desired group points %ld", group, resampling_group); +#endif + + // ------------------------------------------------------------------------- + // initialize our result set + // this also locks the chart for us + + RRDR *r = rrdr_create(st, points_wanted, context_param_list); + if(unlikely(!r)) { + #ifdef NETDATA_INTERNAL_CHECKS + error("INTERNAL CHECK: Cannot create RRDR for %s, after=%u, before=%u, duration=%u, points=%ld", st->id, (uint32_t)after_wanted, (uint32_t)before_wanted, (uint32_t)duration, points_wanted); + #endif + freez(region_info_array); + return NULL; + } + + if(unlikely(!r->d || !points_wanted)) { + #ifdef NETDATA_INTERNAL_CHECKS + error("INTERNAL CHECK: Returning empty RRDR (no dimensions in RRDSET) for %s, after=%u, before=%u, duration=%zu, points=%ld", st->id, (uint32_t)after_wanted, (uint32_t)before_wanted, (size_t)duration, points_wanted); + #endif + freez(region_info_array); + return r; + } + + r->result_options |= RRDR_RESULT_OPTION_VARIABLE_STEP; + if(unlikely(absolute_period_requested == 1)) + r->result_options |= RRDR_RESULT_OPTION_ABSOLUTE; + else + r->result_options |= RRDR_RESULT_OPTION_RELATIVE; + + // find how many dimensions we have + long dimensions_count = r->d; + + // ------------------------------------------------------------------------- + // initialize RRDR + + r->group = group; + r->update_every = (int)group * update_every; + r->before = before_wanted; + r->after = after_wanted; + r->internal.points_wanted = points_wanted; + r->internal.resampling_group = resampling_group; + r->internal.resampling_divisor = resampling_divisor; + + + // ------------------------------------------------------------------------- + // assign the processor functions + + { + int i, found = 0; + for(i = 0; !found && api_v1_data_groups[i].name ;i++) { + if(api_v1_data_groups[i].value == group_method) { + r->internal.grouping_create= api_v1_data_groups[i].create; + r->internal.grouping_reset = api_v1_data_groups[i].reset; + r->internal.grouping_free = api_v1_data_groups[i].free; + r->internal.grouping_add = api_v1_data_groups[i].add; + r->internal.grouping_flush = api_v1_data_groups[i].flush; + found = 1; + } + } + if(!found) { + errno = 0; + #ifdef NETDATA_INTERNAL_CHECKS + error("INTERNAL ERROR: grouping method %u not found for chart '%s'. Using 'average'", (unsigned int)group_method, r->st->name); + #endif + r->internal.grouping_create= grouping_create_average; + r->internal.grouping_reset = grouping_reset_average; + r->internal.grouping_free = grouping_free_average; + r->internal.grouping_add = grouping_add_average; + r->internal.grouping_flush = grouping_flush_average; + } + } + + // allocate any memory required by the grouping method + r->internal.grouping_data = r->internal.grouping_create(r); + + + // ------------------------------------------------------------------------- + // disable the not-wanted dimensions + + rrdset_check_rdlock(st); + + if(dimensions) + rrdr_disable_not_selected_dimensions(r, options, dimensions, temp_rd); + + + // ------------------------------------------------------------------------- + // do the work for each dimension + + time_t max_after = 0, min_before = 0; + long max_rows = 0; + + RRDDIM *rd; + long c, dimensions_used = 0, dimensions_nonzero = 0; + for(rd = temp_rd?temp_rd:st->dimensions, c = 0 ; rd && c < dimensions_count ; rd = rd->next, c++) { + + // if we need a percentage, we need to calculate all dimensions + if(unlikely(!(options & RRDR_OPTION_PERCENTAGE) && (r->od[c] & RRDR_DIMENSION_HIDDEN))) { + if(unlikely(r->od[c] & RRDR_DIMENSION_SELECTED)) r->od[c] &= ~RRDR_DIMENSION_SELECTED; + continue; + } + r->od[c] |= RRDR_DIMENSION_SELECTED; + + // reset the grouping for the new dimension + r->internal.grouping_reset(r); + + do_dimension_variablestep( + r + , points_wanted + , rd + , c + , after_wanted + , before_wanted + ); + + if(r->od[c] & RRDR_DIMENSION_NONZERO) + dimensions_nonzero++; + + // verify all dimensions are aligned + if(unlikely(!dimensions_used)) { + min_before = r->before; + max_after = r->after; + max_rows = r->rows; + } + else { + if(r->after != max_after) { + #ifdef NETDATA_INTERNAL_CHECKS + error("INTERNAL ERROR: 'after' mismatch between dimensions for chart '%s': max is %zu, dimension '%s' has %zu", + st->name, (size_t)max_after, rd->name, (size_t)r->after); + #endif + r->after = (r->after > max_after) ? r->after : max_after; + } + + if(r->before != min_before) { + #ifdef NETDATA_INTERNAL_CHECKS + error("INTERNAL ERROR: 'before' mismatch between dimensions for chart '%s': max is %zu, dimension '%s' has %zu", + st->name, (size_t)min_before, rd->name, (size_t)r->before); + #endif + r->before = (r->before < min_before) ? r->before : min_before; + } + + if(r->rows != max_rows) { + #ifdef NETDATA_INTERNAL_CHECKS + error("INTERNAL ERROR: 'rows' mismatch between dimensions for chart '%s': max is %zu, dimension '%s' has %zu", + st->name, (size_t)max_rows, rd->name, (size_t)r->rows); + #endif + r->rows = (r->rows > max_rows) ? r->rows : max_rows; + } + } + + dimensions_used++; + } + + #ifdef NETDATA_INTERNAL_CHECKS + + if (dimensions_used) { + if(r->internal.log) + rrd2rrdr_log_request_response_metdata(r, group_method, aligned, group, resampling_time_requested, resampling_group, after_wanted, after_requested, before_wanted, before_requested, points_requested, points_wanted, /*after_slot, before_slot,*/ r->internal.log); + + if(r->rows != points_wanted) + rrd2rrdr_log_request_response_metdata(r, group_method, aligned, group, resampling_time_requested, resampling_group, after_wanted, after_requested, before_wanted, before_requested, points_requested, points_wanted, /*after_slot, before_slot,*/ "got 'points' is not wanted 'points'"); + + if(aligned && (r->before % group) != 0) + rrd2rrdr_log_request_response_metdata(r, group_method, aligned, group, resampling_time_requested, resampling_group, after_wanted, after_requested, before_wanted, before_requested, points_requested, points_wanted, /*after_slot, before_slot,*/ "'before' is not aligned but alignment is required"); + + // 'after' should not be aligned, since we start inside the first group + //if(aligned && (r->after % group) != 0) + // rrd2rrdr_log_request_response_metdata(r, group_method, aligned, group, resampling_time_requested, resampling_group, after_wanted, after_requested, before_wanted, before_requested, points_requested, points_wanted, after_slot, before_slot, "'after' is not aligned but alignment is required"); + + if(r->before != before_requested) + rrd2rrdr_log_request_response_metdata(r, group_method, aligned, group, resampling_time_requested, resampling_group, after_wanted, after_requested, before_wanted, before_requested, points_requested, points_wanted, /*after_slot, before_slot,*/ "chart is not aligned to requested 'before'"); + + if(r->before != before_wanted) + rrd2rrdr_log_request_response_metdata(r, group_method, aligned, group, resampling_time_requested, resampling_group, after_wanted, after_requested, before_wanted, before_requested, points_requested, points_wanted, /*after_slot, before_slot,*/ "got 'before' is not wanted 'before'"); + + // reported 'after' varies, depending on group + if(r->after != after_wanted) + rrd2rrdr_log_request_response_metdata(r, group_method, aligned, group, resampling_time_requested, resampling_group, after_wanted, after_requested, before_wanted, before_requested, points_requested, points_wanted, /*after_slot, before_slot,*/ "got 'after' is not wanted 'after'"); + } + #endif + + // free all resources used by the grouping method + r->internal.grouping_free(r); + + // when all the dimensions are zero, we should return all of them + if(unlikely(options & RRDR_OPTION_NONZERO && !dimensions_nonzero)) { + // all the dimensions are zero + // mark them as NONZERO to send them all + for(rd = temp_rd?temp_rd:st->dimensions, c = 0 ; rd && c < dimensions_count ; rd = rd->next, c++) { + if(unlikely(r->od[c] & RRDR_DIMENSION_HIDDEN)) continue; + r->od[c] |= RRDR_DIMENSION_NONZERO; + } + } + + rrdr_query_completed(r->internal.db_points_read, r->internal.result_points_generated); + freez(region_info_array); + return r; +} +#endif //#ifdef ENABLE_DBENGINE + +RRDR *rrd2rrdr( + RRDSET *st + , long points_requested + , long long after_requested + , long long before_requested + , RRDR_GROUPING group_method + , long resampling_time_requested + , RRDR_OPTIONS options + , const char *dimensions + , struct context_param *context_param_list +) +{ + int rrd_update_every; + int absolute_period_requested; + + time_t first_entry_t; + time_t last_entry_t; + if (context_param_list) { + first_entry_t = context_param_list->first_entry_t; + last_entry_t = context_param_list->last_entry_t; + } else { + rrdset_rdlock(st); + first_entry_t = rrdset_first_entry_t_nolock(st); + last_entry_t = rrdset_last_entry_t_nolock(st); + rrdset_unlock(st); + } + + rrd_update_every = st->update_every; + absolute_period_requested = rrdr_convert_before_after_to_absolute(&after_requested, &before_requested, + rrd_update_every, first_entry_t, + last_entry_t, options); + if (options & RRDR_OPTION_ALLOW_PAST) + if (first_entry_t > after_requested) + first_entry_t = after_requested; + + if (context_param_list) + rebuild_context_param_list(context_param_list, after_requested); + +#ifdef ENABLE_DBENGINE + if (st->rrd_memory_mode == RRD_MEMORY_MODE_DBENGINE) { + struct rrdeng_region_info *region_info_array; + unsigned regions, max_interval; + + /* This call takes the chart read-lock */ + regions = rrdeng_variable_step_boundaries(st, after_requested, before_requested, + ®ion_info_array, &max_interval, context_param_list); + if (1 == regions) { + if (region_info_array) { + if (rrd_update_every != region_info_array[0].update_every) { + rrd_update_every = region_info_array[0].update_every; + /* recalculate query alignment */ + absolute_period_requested = + rrdr_convert_before_after_to_absolute(&after_requested, &before_requested, rrd_update_every, + first_entry_t, last_entry_t, options); + } + freez(region_info_array); + } + return rrd2rrdr_fixedstep(st, points_requested, after_requested, before_requested, group_method, + resampling_time_requested, options, dimensions, rrd_update_every, + first_entry_t, last_entry_t, absolute_period_requested, context_param_list); + } else { + if (rrd_update_every != (uint16_t)max_interval) { + rrd_update_every = (uint16_t) max_interval; + /* recalculate query alignment */ + absolute_period_requested = rrdr_convert_before_after_to_absolute(&after_requested, &before_requested, + rrd_update_every, first_entry_t, + last_entry_t, options); + } + return rrd2rrdr_variablestep(st, points_requested, after_requested, before_requested, group_method, + resampling_time_requested, options, dimensions, rrd_update_every, + first_entry_t, last_entry_t, absolute_period_requested, region_info_array, context_param_list); + } + } +#endif + return rrd2rrdr_fixedstep(st, points_requested, after_requested, before_requested, group_method, + resampling_time_requested, options, dimensions, + rrd_update_every, first_entry_t, last_entry_t, absolute_period_requested, context_param_list); +}
\ No newline at end of file diff --git a/web/api/queries/query.h b/web/api/queries/query.h new file mode 100644 index 0000000..6b8a51c --- /dev/null +++ b/web/api/queries/query.h @@ -0,0 +1,24 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#ifndef NETDATA_API_DATA_QUERY_H +#define NETDATA_API_DATA_QUERY_H + +typedef enum rrdr_grouping { + RRDR_GROUPING_UNDEFINED = 0, + RRDR_GROUPING_AVERAGE, + RRDR_GROUPING_MIN, + RRDR_GROUPING_MAX, + RRDR_GROUPING_SUM, + RRDR_GROUPING_INCREMENTAL_SUM, + RRDR_GROUPING_MEDIAN, + RRDR_GROUPING_STDDEV, + RRDR_GROUPING_CV, + RRDR_GROUPING_SES, + RRDR_GROUPING_DES, +} RRDR_GROUPING; + +extern const char *group_method2string(RRDR_GROUPING group); +extern void web_client_api_v1_init_grouping(void); +extern RRDR_GROUPING web_client_api_request_v1_data_group(const char *name, RRDR_GROUPING def); + +#endif //NETDATA_API_DATA_QUERY_H diff --git a/web/api/queries/rrdr.c b/web/api/queries/rrdr.c new file mode 100644 index 0000000..ef237fa --- /dev/null +++ b/web/api/queries/rrdr.c @@ -0,0 +1,144 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#include "rrdr.h" + +/* +static void rrdr_dump(RRDR *r) +{ + long c, i; + RRDDIM *d; + + fprintf(stderr, "\nCHART %s (%s)\n", r->st->id, r->st->name); + + for(c = 0, d = r->st->dimensions; d ;c++, d = d->next) { + fprintf(stderr, "DIMENSION %s (%s), %s%s%s%s\n" + , d->id + , d->name + , (r->od[c] & RRDR_EMPTY)?"EMPTY ":"" + , (r->od[c] & RRDR_RESET)?"RESET ":"" + , (r->od[c] & RRDR_DIMENSION_HIDDEN)?"HIDDEN ":"" + , (r->od[c] & RRDR_DIMENSION_NONZERO)?"NONZERO ":"" + ); + } + + if(r->rows <= 0) { + fprintf(stderr, "RRDR does not have any values in it.\n"); + return; + } + + fprintf(stderr, "RRDR includes %d values in it:\n", r->rows); + + // for each line in the array + for(i = 0; i < r->rows ;i++) { + calculated_number *cn = &r->v[ i * r->d ]; + RRDR_DIMENSION_FLAGS *co = &r->o[ i * r->d ]; + + // print the id and the timestamp of the line + fprintf(stderr, "%ld %ld ", i + 1, r->t[i]); + + // for each dimension + for(c = 0, d = r->st->dimensions; d ;c++, d = d->next) { + if(unlikely(r->od[c] & RRDR_DIMENSION_HIDDEN)) continue; + if(unlikely(!(r->od[c] & RRDR_DIMENSION_NONZERO))) continue; + + if(co[c] & RRDR_EMPTY) + fprintf(stderr, "null "); + else + fprintf(stderr, CALCULATED_NUMBER_FORMAT " %s%s%s%s " + , cn[c] + , (co[c] & RRDR_EMPTY)?"E":" " + , (co[c] & RRDR_RESET)?"R":" " + , (co[c] & RRDR_DIMENSION_HIDDEN)?"H":" " + , (co[c] & RRDR_DIMENSION_NONZERO)?"N":" " + ); + } + + fprintf(stderr, "\n"); + } +} +*/ + + + + +inline static void rrdr_lock_rrdset(RRDR *r) { + if(unlikely(!r)) { + error("NULL value given!"); + return; + } + + rrdset_rdlock(r->st); + r->has_st_lock = 1; +} + +inline static void rrdr_unlock_rrdset(RRDR *r) { + if(unlikely(!r)) { + error("NULL value given!"); + return; + } + + if(likely(r->has_st_lock)) { + rrdset_unlock(r->st); + r->has_st_lock = 0; + } +} + +inline void rrdr_free(RRDR *r) +{ + if(unlikely(!r)) { + error("NULL value given!"); + return; + } + + rrdr_unlock_rrdset(r); + freez(r->t); + freez(r->v); + freez(r->o); + freez(r->od); + freez(r); +} + +RRDR *rrdr_create(struct rrdset *st, long n, struct context_param *context_param_list) +{ + if(unlikely(!st)) { + error("NULL value given!"); + return NULL; + } + + RRDR *r = callocz(1, sizeof(RRDR)); + r->st = st; + + rrdr_lock_rrdset(r); + + RRDDIM *temp_rd = context_param_list ? context_param_list->rd : NULL; + RRDDIM *rd; + if (temp_rd) { + RRDDIM *t = temp_rd; + while (t) { + r->d++; + t = t->next; + } + } else + rrddim_foreach_read(rd, st) r->d++; + + r->n = n; + + r->t = callocz((size_t)n, sizeof(time_t)); + r->v = mallocz(n * r->d * sizeof(calculated_number)); + r->o = mallocz(n * r->d * sizeof(RRDR_VALUE_FLAGS)); + r->od = mallocz(r->d * sizeof(RRDR_DIMENSION_FLAGS)); + + // set the hidden flag on hidden dimensions + int c; + for (c = 0, rd = temp_rd ? temp_rd : st->dimensions; rd; c++, rd = rd->next) { + if (unlikely(rrddim_flag_check(rd, RRDDIM_FLAG_HIDDEN))) + r->od[c] = RRDR_DIMENSION_HIDDEN; + else + r->od[c] = RRDR_DIMENSION_DEFAULT; + } + + r->group = 1; + r->update_every = 1; + + return r; +} diff --git a/web/api/queries/rrdr.h b/web/api/queries/rrdr.h new file mode 100644 index 0000000..4d349c3 --- /dev/null +++ b/web/api/queries/rrdr.h @@ -0,0 +1,115 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#ifndef NETDATA_QUERIES_RRDR_H +#define NETDATA_QUERIES_RRDR_H + +#include "libnetdata/libnetdata.h" + +typedef enum rrdr_options { + RRDR_OPTION_NONZERO = 0x00000001, // don't output dimensions with just zero values + RRDR_OPTION_REVERSED = 0x00000002, // output the rows in reverse order (oldest to newest) + RRDR_OPTION_ABSOLUTE = 0x00000004, // values positive, for DATASOURCE_SSV before summing + RRDR_OPTION_MIN2MAX = 0x00000008, // when adding dimensions, use max - min, instead of sum + RRDR_OPTION_SECONDS = 0x00000010, // output seconds, instead of dates + RRDR_OPTION_MILLISECONDS = 0x00000020, // output milliseconds, instead of dates + RRDR_OPTION_NULL2ZERO = 0x00000040, // do not show nulls, convert them to zeros + RRDR_OPTION_OBJECTSROWS = 0x00000080, // each row of values should be an object, not an array + RRDR_OPTION_GOOGLE_JSON = 0x00000100, // comply with google JSON/JSONP specs + RRDR_OPTION_JSON_WRAP = 0x00000200, // wrap the response in a JSON header with info about the result + RRDR_OPTION_LABEL_QUOTES = 0x00000400, // in CSV output, wrap header labels in double quotes + RRDR_OPTION_PERCENTAGE = 0x00000800, // give values as percentage of total + RRDR_OPTION_NOT_ALIGNED = 0x00001000, // do not align charts for persistent timeframes + RRDR_OPTION_DISPLAY_ABS = 0x00002000, // for badges, display the absolute value, but calculate colors with sign + RRDR_OPTION_MATCH_IDS = 0x00004000, // when filtering dimensions, match only IDs + RRDR_OPTION_MATCH_NAMES = 0x00008000, // when filtering dimensions, match only names + RRDR_OPTION_CUSTOM_VARS = 0x00010000, // when wraping response in a JSON, return custom variables in response + RRDR_OPTION_ALLOW_PAST = 0x00020000, // The after parameter can extend in the past before the first entry +} RRDR_OPTIONS; + +typedef enum rrdr_value_flag { + RRDR_VALUE_NOTHING = 0x00, // no flag set (a good default) + RRDR_VALUE_EMPTY = 0x01, // the database value is empty + RRDR_VALUE_RESET = 0x02, // the database value is marked as reset (overflown) +} RRDR_VALUE_FLAGS; + +typedef enum rrdr_dimension_flag { + RRDR_DIMENSION_DEFAULT = 0x00, + RRDR_DIMENSION_HIDDEN = 0x04, // the dimension is hidden (not to be presented to callers) + RRDR_DIMENSION_NONZERO = 0x08, // the dimension is non zero (contains non-zero values) + RRDR_DIMENSION_SELECTED = 0x10, // the dimension is selected for evaluation in this RRDR +} RRDR_DIMENSION_FLAGS; + +// RRDR result options +typedef enum rrdr_result_flags { + RRDR_RESULT_OPTION_ABSOLUTE = 0x00000001, // the query uses absolute time-frames + // (can be cached by browsers and proxies) + RRDR_RESULT_OPTION_RELATIVE = 0x00000002, // the query uses relative time-frames + // (should not to be cached by browsers and proxies) + RRDR_RESULT_OPTION_VARIABLE_STEP = 0x00000004, // the query uses variable-step time-frames +} RRDR_RESULT_FLAGS; + +typedef struct rrdresult { + struct rrdset *st; // the chart this result refers to + + RRDR_RESULT_FLAGS result_options; // RRDR_RESULT_OPTION_* + + int d; // the number of dimensions + long n; // the number of values in the arrays + long rows; // the number of rows used + + RRDR_DIMENSION_FLAGS *od; // the options for the dimensions + + time_t *t; // array of n timestamps + calculated_number *v; // array n x d values + RRDR_VALUE_FLAGS *o; // array n x d options for each value returned + + long group; // how many collected values were grouped for each row + int update_every; // what is the suggested update frequency in seconds + + calculated_number min; + calculated_number max; + + time_t before; + time_t after; + + int has_st_lock; // if st is read locked by us + + // internal rrd2rrdr() members below this point + struct { + long points_wanted; + long resampling_group; + calculated_number resampling_divisor; + + void *(*grouping_create)(struct rrdresult *r); + void (*grouping_reset)(struct rrdresult *r); + void (*grouping_free)(struct rrdresult *r); + void (*grouping_add)(struct rrdresult *r, calculated_number value); + calculated_number (*grouping_flush)(struct rrdresult *r, RRDR_VALUE_FLAGS *rrdr_value_options_ptr); + void *grouping_data; + + #ifdef NETDATA_INTERNAL_CHECKS + const char *log; + #endif + + size_t db_points_read; + size_t result_points_generated; + } internal; +} RRDR; + +#define rrdr_rows(r) ((r)->rows) + +#include "../../../database/rrd.h" +extern void rrdr_free(RRDR *r); +extern RRDR *rrdr_create(struct rrdset *st, long n, struct context_param *context_param_list); + +#include "../web_api_v1.h" +#include "web/api/queries/query.h" + +extern RRDR *rrd2rrdr( + RRDSET *st, long points_requested, long long after_requested, long long before_requested, + RRDR_GROUPING group_method, long resampling_time_requested, RRDR_OPTIONS options, const char *dimensions, + struct context_param *context_param_list); + +#include "query.h" + +#endif //NETDATA_QUERIES_RRDR_H diff --git a/web/api/queries/ses/Makefile.am b/web/api/queries/ses/Makefile.am new file mode 100644 index 0000000..161784b --- /dev/null +++ b/web/api/queries/ses/Makefile.am @@ -0,0 +1,8 @@ +# SPDX-License-Identifier: GPL-3.0-or-later + +AUTOMAKE_OPTIONS = subdir-objects +MAINTAINERCLEANFILES = $(srcdir)/Makefile.in + +dist_noinst_DATA = \ + README.md \ + $(NULL) diff --git a/web/api/queries/ses/README.md b/web/api/queries/ses/README.md new file mode 100644 index 0000000..c279701 --- /dev/null +++ b/web/api/queries/ses/README.md @@ -0,0 +1,61 @@ +<!-- +title: "Single (or Simple) Exponential Smoothing (`ses`)" +custom_edit_url: https://github.com/netdata/netdata/edit/master/web/api/queries/ses/README.md +--> + +# Single (or Simple) Exponential Smoothing (`ses`) + +> This query is also available as `ema` and `ewma`. + +An exponential moving average (`ema`), also known as an exponentially weighted moving average (`ewma`) +is a first-order infinite impulse response filter that applies weighting factors which decrease +exponentially. The weighting for each older datum decreases exponentially, never reaching zero. + +In simple terms, this is like an average value, but more recent values are given more weight. + +Netdata automatically adjusts the weight (`alpha`) based on the number of values processed, +using the formula: + +``` +window = max(number of values, 15) +alpha = 2 / (window + 1) +``` + +You can change the fixed value `15` by setting in `netdata.conf`: + +``` +[web] + ses max window = 15 +``` + +## how to use + +Use it in alarms like this: + +``` + alarm: my_alarm + on: my_chart +lookup: ses -1m unaligned of my_dimension + warn: $this > 1000 +``` + +`ses` does not change the units. For example, if the chart units is `requests/sec`, the exponential +moving average will be again expressed in the same units. + +It can also be used in APIs and badges as `&group=ses` in the URL. + +## Examples + +Examining last 1 minute `successful` web server responses: + +-  +-  +-  +-  + +## References + +- <https://en.wikipedia.org/wiki/Moving_average#exponential-moving-average> +- <https://en.wikipedia.org/wiki/Exponential_smoothing>. + +[](<>) diff --git a/web/api/queries/ses/ses.c b/web/api/queries/ses/ses.c new file mode 100644 index 0000000..772505f --- /dev/null +++ b/web/api/queries/ses/ses.c @@ -0,0 +1,92 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#include "ses.h" + + +// ---------------------------------------------------------------------------- +// single exponential smoothing + +struct grouping_ses { + calculated_number alpha; + calculated_number alpha_other; + calculated_number level; + size_t count; +}; + +static size_t max_window_size = 15; + +void grouping_init_ses(void) { + long long ret = config_get_number(CONFIG_SECTION_WEB, "ses max window", (long long)max_window_size); + if(ret <= 1) { + config_set_number(CONFIG_SECTION_WEB, "ses max window", (long long)max_window_size); + } + else { + max_window_size = (size_t) ret; + } +} + +static inline calculated_number window(RRDR *r, struct grouping_ses *g) { + (void)g; + + calculated_number points; + if(r->group == 1) { + // provide a running DES + points = r->internal.points_wanted; + } + else { + // provide a SES with flush points + points = r->group; + } + + return (points > max_window_size) ? max_window_size : points; +} + +static inline void set_alpha(RRDR *r, struct grouping_ses *g) { + // https://en.wikipedia.org/wiki/Moving_average#Exponential_moving_average + // A commonly used value for alpha is 2 / (N + 1) + g->alpha = 2.0 / (window(r, g) + 1.0); + g->alpha_other = 1.0 - g->alpha; +} + +void *grouping_create_ses(RRDR *r) { + struct grouping_ses *g = (struct grouping_ses *)callocz(1, sizeof(struct grouping_ses)); + set_alpha(r, g); + g->level = 0.0; + return g; +} + +// resets when switches dimensions +// so, clear everything to restart +void grouping_reset_ses(RRDR *r) { + struct grouping_ses *g = (struct grouping_ses *)r->internal.grouping_data; + g->level = 0.0; + g->count = 0; +} + +void grouping_free_ses(RRDR *r) { + freez(r->internal.grouping_data); + r->internal.grouping_data = NULL; +} + +void grouping_add_ses(RRDR *r, calculated_number value) { + struct grouping_ses *g = (struct grouping_ses *)r->internal.grouping_data; + + if(calculated_number_isnumber(value)) { + if(unlikely(!g->count)) + g->level = value; + + g->level = g->alpha * value + g->alpha_other * g->level; + g->count++; + } +} + +calculated_number grouping_flush_ses(RRDR *r, RRDR_VALUE_FLAGS *rrdr_value_options_ptr) { + struct grouping_ses *g = (struct grouping_ses *)r->internal.grouping_data; + + if(unlikely(!g->count || !calculated_number_isnumber(g->level))) { + *rrdr_value_options_ptr |= RRDR_VALUE_EMPTY; + return 0.0; + } + + return g->level; +} diff --git a/web/api/queries/ses/ses.h b/web/api/queries/ses/ses.h new file mode 100644 index 0000000..603fdb5 --- /dev/null +++ b/web/api/queries/ses/ses.h @@ -0,0 +1,17 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#ifndef NETDATA_API_QUERIES_SES_H +#define NETDATA_API_QUERIES_SES_H + +#include "../query.h" +#include "../rrdr.h" + +extern void grouping_init_ses(void); + +extern void *grouping_create_ses(RRDR *r); +extern void grouping_reset_ses(RRDR *r); +extern void grouping_free_ses(RRDR *r); +extern void grouping_add_ses(RRDR *r, calculated_number value); +extern calculated_number grouping_flush_ses(RRDR *r, RRDR_VALUE_FLAGS *rrdr_value_options_ptr); + +#endif //NETDATA_API_QUERIES_SES_H diff --git a/web/api/queries/stddev/Makefile.am b/web/api/queries/stddev/Makefile.am new file mode 100644 index 0000000..161784b --- /dev/null +++ b/web/api/queries/stddev/Makefile.am @@ -0,0 +1,8 @@ +# SPDX-License-Identifier: GPL-3.0-or-later + +AUTOMAKE_OPTIONS = subdir-objects +MAINTAINERCLEANFILES = $(srcdir)/Makefile.in + +dist_noinst_DATA = \ + README.md \ + $(NULL) diff --git a/web/api/queries/stddev/README.md b/web/api/queries/stddev/README.md new file mode 100644 index 0000000..7cd7d62 --- /dev/null +++ b/web/api/queries/stddev/README.md @@ -0,0 +1,93 @@ +<!-- +title: "standard deviation (`stddev`)" +custom_edit_url: https://github.com/netdata/netdata/edit/master/web/api/queries/stddev/README.md +--> + +# standard deviation (`stddev`) + +The standard deviation is a measure that is used to quantify the amount of variation or dispersion +of a set of data values. + +A low standard deviation indicates that the data points tend to be close to the mean (also called the +expected value) of the set, while a high standard deviation indicates that the data points are spread +out over a wider range of values. + +## how to use + +Use it in alarms like this: + +``` + alarm: my_alarm + on: my_chart +lookup: stddev -1m unaligned of my_dimension + warn: $this > 1000 +``` + +`stdev` does not change the units. For example, if the chart units is `requests/sec`, the standard +deviation will be again expressed in the same units. + +It can also be used in APIs and badges as `&group=stddev` in the URL. + +## Examples + +Examining last 1 minute `successful` web server responses: + +-  +-  +-  +-  + +## References + +Check <https://en.wikipedia.org/wiki/Standard_deviation>. + +--- + +# Coefficient of variation (`cv`) + +> This query is also available as `rsd`. + +The coefficient of variation (`cv`), also known as relative standard deviation (`rsd`), +is a standardized measure of dispersion of a probability distribution or frequency distribution. + +It is defined as the ratio of the **standard deviation** to the **mean**. + +In simple terms, it gives the percentage of change. So, if the average value of a metric is 1000 +and its standard deviation is 100 (meaning that it variates from 900 to 1100), then `cv` is 10%. + +This is an easy way to check the % variation, without using absolute values. + +For example, you may trigger an alarm if your web server requests/sec `cv` is above 20 (`%`) +over the last minute. So if your web server was serving 1000 reqs/sec over the last minute, +it will trigger the alarm if had spikes below 800/sec or above 1200/sec. + +## how to use + +Use it in alarms like this: + +``` + alarm: my_alarm + on: my_chart +lookup: cv -1m unaligned of my_dimension + units: % + warn: $this > 20 +``` + +The units reported by `cv` is always `%`. + +It can also be used in APIs and badges as `&group=cv` in the URL. + +## Examples + +Examining last 1 minute `successful` web server responses: + +-  +-  +-  +-  + +## References + +Check <https://en.wikipedia.org/wiki/Coefficient_of_variation>. + +[](<>) diff --git a/web/api/queries/stddev/stddev.c b/web/api/queries/stddev/stddev.c new file mode 100644 index 0000000..1625844 --- /dev/null +++ b/web/api/queries/stddev/stddev.c @@ -0,0 +1,176 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#include "stddev.h" + + +// ---------------------------------------------------------------------------- +// stddev + +// this implementation comes from: +// https://www.johndcook.com/blog/standard_deviation/ + +struct grouping_stddev { + long count; + calculated_number m_oldM, m_newM, m_oldS, m_newS; +}; + +void *grouping_create_stddev(RRDR *r) { + UNUSED (r); + return callocz(1, sizeof(struct grouping_stddev)); +} + +// resets when switches dimensions +// so, clear everything to restart +void grouping_reset_stddev(RRDR *r) { + struct grouping_stddev *g = (struct grouping_stddev *)r->internal.grouping_data; + g->count = 0; +} + +void grouping_free_stddev(RRDR *r) { + freez(r->internal.grouping_data); + r->internal.grouping_data = NULL; +} + +void grouping_add_stddev(RRDR *r, calculated_number value) { + struct grouping_stddev *g = (struct grouping_stddev *)r->internal.grouping_data; + + if(calculated_number_isnumber(value)) { + g->count++; + + // See Knuth TAOCP vol 2, 3rd edition, page 232 + if (g->count == 1) { + g->m_oldM = g->m_newM = value; + g->m_oldS = 0.0; + } + else { + g->m_newM = g->m_oldM + (value - g->m_oldM) / g->count; + g->m_newS = g->m_oldS + (value - g->m_oldM) * (value - g->m_newM); + + // set up for next iteration + g->m_oldM = g->m_newM; + g->m_oldS = g->m_newS; + } + } +} + +static inline calculated_number mean(struct grouping_stddev *g) { + return (g->count > 0) ? g->m_newM : 0.0; +} + +static inline calculated_number variance(struct grouping_stddev *g) { + return ( (g->count > 1) ? g->m_newS/(g->count - 1) : 0.0 ); +} +static inline calculated_number stddev(struct grouping_stddev *g) { + return sqrtl(variance(g)); +} + +calculated_number grouping_flush_stddev(RRDR *r, RRDR_VALUE_FLAGS *rrdr_value_options_ptr) { + struct grouping_stddev *g = (struct grouping_stddev *)r->internal.grouping_data; + + calculated_number value; + + if(likely(g->count > 1)) { + value = stddev(g); + + if(!calculated_number_isnumber(value)) { + value = 0.0; + *rrdr_value_options_ptr |= RRDR_VALUE_EMPTY; + } + } + else if(g->count == 1) { + value = 0.0; + } + else { + value = 0.0; + *rrdr_value_options_ptr |= RRDR_VALUE_EMPTY; + } + + grouping_reset_stddev(r); + + return value; +} + +// https://en.wikipedia.org/wiki/Coefficient_of_variation +calculated_number grouping_flush_coefficient_of_variation(RRDR *r, RRDR_VALUE_FLAGS *rrdr_value_options_ptr) { + struct grouping_stddev *g = (struct grouping_stddev *)r->internal.grouping_data; + + calculated_number value; + + if(likely(g->count > 1)) { + calculated_number m = mean(g); + value = 100.0 * stddev(g) / ((m < 0)? -m : m); + + if(unlikely(!calculated_number_isnumber(value))) { + value = 0.0; + *rrdr_value_options_ptr |= RRDR_VALUE_EMPTY; + } + } + else if(g->count == 1) { + // one value collected + value = 0.0; + } + else { + // no values collected + value = 0.0; + *rrdr_value_options_ptr |= RRDR_VALUE_EMPTY; + } + + grouping_reset_stddev(r); + + return value; +} + + +/* + * Mean = average + * +calculated_number grouping_flush_mean(RRDR *r, RRDR_VALUE_FLAGS *rrdr_value_options_ptr) { + struct grouping_stddev *g = (struct grouping_stddev *)r->internal.grouping_data; + + calculated_number value; + + if(unlikely(!g->count)) { + value = 0.0; + *rrdr_value_options_ptr |= RRDR_VALUE_EMPTY; + } + else { + value = mean(g); + + if(!isnormal(value)) { + value = 0.0; + *rrdr_value_options_ptr |= RRDR_VALUE_EMPTY; + } + } + + grouping_reset_stddev(r); + + return value; +} + */ + +/* + * It is not advised to use this version of variance directly + * +calculated_number grouping_flush_variance(RRDR *r, RRDR_VALUE_FLAGS *rrdr_value_options_ptr) { + struct grouping_stddev *g = (struct grouping_stddev *)r->internal.grouping_data; + + calculated_number value; + + if(unlikely(!g->count)) { + value = 0.0; + *rrdr_value_options_ptr |= RRDR_VALUE_EMPTY; + } + else { + value = variance(g); + + if(!isnormal(value)) { + value = 0.0; + *rrdr_value_options_ptr |= RRDR_VALUE_EMPTY; + } + } + + grouping_reset_stddev(r); + + return value; +} +*/
\ No newline at end of file diff --git a/web/api/queries/stddev/stddev.h b/web/api/queries/stddev/stddev.h new file mode 100644 index 0000000..7a46975 --- /dev/null +++ b/web/api/queries/stddev/stddev.h @@ -0,0 +1,18 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#ifndef NETDATA_API_QUERIES_STDDEV_H +#define NETDATA_API_QUERIES_STDDEV_H + +#include "../query.h" +#include "../rrdr.h" + +extern void *grouping_create_stddev(RRDR *r); +extern void grouping_reset_stddev(RRDR *r); +extern void grouping_free_stddev(RRDR *r); +extern void grouping_add_stddev(RRDR *r, calculated_number value); +extern calculated_number grouping_flush_stddev(RRDR *r, RRDR_VALUE_FLAGS *rrdr_value_options_ptr); +extern calculated_number grouping_flush_coefficient_of_variation(RRDR *r, RRDR_VALUE_FLAGS *rrdr_value_options_ptr); +// extern calculated_number grouping_flush_mean(RRDR *r, RRDR_VALUE_FLAGS *rrdr_value_options_ptr); +// extern calculated_number grouping_flush_variance(RRDR *r, RRDR_VALUE_FLAGS *rrdr_value_options_ptr); + +#endif //NETDATA_API_QUERIES_STDDEV_H diff --git a/web/api/queries/sum/Makefile.am b/web/api/queries/sum/Makefile.am new file mode 100644 index 0000000..161784b --- /dev/null +++ b/web/api/queries/sum/Makefile.am @@ -0,0 +1,8 @@ +# SPDX-License-Identifier: GPL-3.0-or-later + +AUTOMAKE_OPTIONS = subdir-objects +MAINTAINERCLEANFILES = $(srcdir)/Makefile.in + +dist_noinst_DATA = \ + README.md \ + $(NULL) diff --git a/web/api/queries/sum/README.md b/web/api/queries/sum/README.md new file mode 100644 index 0000000..aeace0a --- /dev/null +++ b/web/api/queries/sum/README.md @@ -0,0 +1,41 @@ +<!-- +title: "Sum" +custom_edit_url: https://github.com/netdata/netdata/edit/master/web/api/queries/sum/README.md +--> + +# Sum + +This module sums all the values in the time-frame requested. + +You can use `sum` to find the volume of something over a period. + +## how to use + +Use it in alarms like this: + +``` + alarm: my_alarm + on: my_chart +lookup: sum -1m unaligned of my_dimension + warn: $this > 1000 +``` + +`sum` does not change the units. For example, if the chart units is `requests/sec`, the result +will be again expressed in the same units. + +It can also be used in APIs and badges as `&group=sum` in the URL. + +## Examples + +Examining last 1 minute `successful` web server responses: + +-  +-  +-  +-  + +## References + +- <https://en.wikipedia.org/wiki/Summation>. + +[](<>) diff --git a/web/api/queries/sum/sum.c b/web/api/queries/sum/sum.c new file mode 100644 index 0000000..0da9937 --- /dev/null +++ b/web/api/queries/sum/sum.c @@ -0,0 +1,58 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#include "sum.h" + +// ---------------------------------------------------------------------------- +// sum + +struct grouping_sum { + calculated_number sum; + size_t count; +}; + +void *grouping_create_sum(RRDR *r) { + (void)r; + return callocz(1, sizeof(struct grouping_sum)); +} + +// resets when switches dimensions +// so, clear everything to restart +void grouping_reset_sum(RRDR *r) { + struct grouping_sum *g = (struct grouping_sum *)r->internal.grouping_data; + g->sum = 0; + g->count = 0; +} + +void grouping_free_sum(RRDR *r) { + freez(r->internal.grouping_data); + r->internal.grouping_data = NULL; +} + +void grouping_add_sum(RRDR *r, calculated_number value) { + if(!isnan(value)) { + struct grouping_sum *g = (struct grouping_sum *)r->internal.grouping_data; + g->sum += value; + g->count++; + } +} + +calculated_number grouping_flush_sum(RRDR *r, RRDR_VALUE_FLAGS *rrdr_value_options_ptr) { + struct grouping_sum *g = (struct grouping_sum *)r->internal.grouping_data; + + calculated_number value; + + if(unlikely(!g->count)) { + value = 0.0; + *rrdr_value_options_ptr |= RRDR_VALUE_EMPTY; + } + else { + value = g->sum; + } + + g->sum = 0.0; + g->count = 0; + + return value; +} + + diff --git a/web/api/queries/sum/sum.h b/web/api/queries/sum/sum.h new file mode 100644 index 0000000..9dc8d20 --- /dev/null +++ b/web/api/queries/sum/sum.h @@ -0,0 +1,15 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#ifndef NETDATA_API_QUERY_SUM_H +#define NETDATA_API_QUERY_SUM_H + +#include "../query.h" +#include "../rrdr.h" + +extern void *grouping_create_sum(RRDR *r); +extern void grouping_reset_sum(RRDR *r); +extern void grouping_free_sum(RRDR *r); +extern void grouping_add_sum(RRDR *r, calculated_number value); +extern calculated_number grouping_flush_sum(RRDR *r, RRDR_VALUE_FLAGS *rrdr_value_options_ptr); + +#endif //NETDATA_API_QUERY_SUM_H diff --git a/web/api/tests/valid_urls.c b/web/api/tests/valid_urls.c new file mode 100644 index 0000000..d8c261c --- /dev/null +++ b/web/api/tests/valid_urls.c @@ -0,0 +1,790 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#include "../../../libnetdata/libnetdata.h" +#include "../../../libnetdata/required_dummies.h" +#include "../../../database/rrd.h" +#include "../../../web/server/web_client.h" +#include <setjmp.h> +#include <cmocka.h> +#include <stdbool.h> + +RRDHOST *sql_create_host_by_uuid(char *hostname) +{ + (void) hostname; + return NULL; +} + +RRDHOST *__wrap_sql_create_host_by_uuid(char *hostname) +{ + (void) hostname; + return NULL; +} + +void repr(char *result, int result_size, char const *buf, int size) +{ + int n; + char *end = result + result_size - 1; + unsigned char const *ubuf = (unsigned char const *)buf; + while (size && result_size > 0) { + if (*ubuf <= 0x20 || *ubuf >= 0x80) { + n = snprintf(result, result_size, "\\%02X", *ubuf); + } else { + *result = *ubuf; + n = 1; + } + result += n; + result_size -= n; + ubuf++; + size--; + } + if (result_size > 0) + *(result++) = 0; + else + *end = 0; +} + +// ---------------------------------- Mocking accesses from web_client ------------------------------------------------ + +ssize_t send(int sockfd, const void *buf, size_t len, int flags) +{ + info("Mocking send: %zu bytes\n", len); + (void)sockfd; + (void)buf; + (void)flags; + return len; +} + +RRDHOST *__wrap_rrdhost_find_by_hostname(const char *hostname, uint32_t hash) +{ + (void)hostname; + (void)hash; + return NULL; +} + +/* Note: we've got some intricate code inside the global statistics module, might be useful to pull it inside the + test set instead of mocking it. */ +void __wrap_finished_web_request_statistics( + uint64_t dt, uint64_t bytes_received, uint64_t bytes_sent, uint64_t content_size, uint64_t compressed_content_size) +{ + (void)dt; + (void)bytes_received; + (void)bytes_sent; + (void)content_size; + (void)compressed_content_size; +} + +char *__wrap_config_get(struct config *root, const char *section, const char *name, const char *default_value) +{ + if (!strcmp(section, CONFIG_SECTION_WEB) && !strcmp(name, "web files owner")) + return "netdata"; + (void)root; + (void)default_value; + return "UNKNOWN FIX ME"; +} + +int __wrap_web_client_api_request_v1(RRDHOST *host, struct web_client *w, char *url) +{ + char url_repr[160]; + repr(url_repr, sizeof(url_repr), url, strlen(url)); + printf("web_client_api_request_v1(url=\"%s\")\n", url_repr); + check_expected_ptr(host); + check_expected_ptr(w); + check_expected_ptr(url_repr); + return HTTP_RESP_OK; +} + +int __wrap_mysendfile(struct web_client *w, char *filename) +{ + (void)w; + printf("mysendfile(filename=\"%s\"\n", filename); + check_expected_ptr(filename); + return HTTP_RESP_OK; +} + +int __wrap_rrdpush_receiver_thread_spawn(RRDHOST *host, struct web_client *w, char *url) +{ + (void)host; + (void)w; + (void)url; + return 0; +} + +RRDHOST *__wrap_rrdhost_find_by_guid(const char *guid, uint32_t hash) +{ + (void)guid; + (void)hash; + printf("FIXME: rrdset_find_guid\n"); + return NULL; +} + +RRDSET *__wrap_rrdset_find_byname(RRDHOST *host, const char *name) +{ + (void)host; + (void)name; + printf("FIXME: rrdset_find_byname\n"); + return NULL; +} + +RRDSET *__wrap_rrdset_find(RRDHOST *host, const char *id) +{ + (void)host; + (void)id; + printf("FIXME: rrdset_find\n"); + return NULL; +} + +// -------------------------------- Mocking the log - dump straight through -------------------------------------------- + +void __wrap_debug_int(const char *file, const char *function, const unsigned long line, const char *fmt, ...) +{ + (void)file; + (void)function; + (void)line; + va_list args; + va_start(args, fmt); + printf(" DEBUG: "); + printf(fmt, args); + printf("\n"); + va_end(args); +} + +void __wrap_info_int(const char *file, const char *function, const unsigned long line, const char *fmt, ...) +{ + (void)file; + (void)function; + (void)line; + va_list args; + va_start(args, fmt); + printf(" INFO: "); + printf(fmt, args); + printf("\n"); + va_end(args); +} + +void __wrap_error_int( + const char *prefix, const char *file, const char *function, const unsigned long line, const char *fmt, ...) +{ + (void)prefix; + (void)file; + (void)function; + (void)line; + va_list args; + va_start(args, fmt); + printf(" ERROR: "); + printf(fmt, args); + printf("\n"); + va_end(args); +} + +void __wrap_fatal_int(const char *file, const char *function, const unsigned long line, const char *fmt, ...) +{ + (void)file; + (void)function; + (void)line; + va_list args; + va_start(args, fmt); + printf("FATAL: "); + printf(fmt, args); + printf("\n"); + va_end(args); + fail(); +} + +WEB_SERVER_MODE web_server_mode = WEB_SERVER_MODE_STATIC_THREADED; +char *netdata_configured_web_dir = "UNKNOWN FIXME"; +RRDHOST *localhost = NULL; + +struct config netdata_config = { .first_section = NULL, + .last_section = NULL, + .mutex = NETDATA_MUTEX_INITIALIZER, + .index = { .avl_tree = { .root = NULL, .compar = appconfig_section_compare }, + .rwlock = AVL_LOCK_INITIALIZER } }; + +/* Note: this is not a CMocka group_test_setup/teardown pair. This is performed per-test. +*/ +static struct web_client *setup_fresh_web_client() +{ + struct web_client *w = (struct web_client *)malloc(sizeof(struct web_client)); + memset(w, 0, sizeof(struct web_client)); + w->response.data = buffer_create(NETDATA_WEB_RESPONSE_INITIAL_SIZE); + w->response.header = buffer_create(NETDATA_WEB_RESPONSE_HEADER_SIZE); + w->response.header_output = buffer_create(NETDATA_WEB_RESPONSE_HEADER_SIZE); + strcpy(w->origin, "*"); // Simulate web_client_create_on_fd() + w->cookie1[0] = 0; // Simulate web_client_create_on_fd() + w->cookie2[0] = 0; // Simulate web_client_create_on_fd() + w->acl = 0x1f; // Everything on + return w; +} + +static void destroy_web_client(struct web_client *w) +{ + buffer_free(w->response.data); + buffer_free(w->response.header); + buffer_free(w->response.header_output); + free(w); +} + +//////////////////////////// Test cases /////////////////////////////////////////////////////////////////////////////// + +static void only_root(void **state) +{ + (void)state; + + if (localhost != NULL) + free(localhost); + localhost = malloc(sizeof(RRDHOST)); + + struct web_client *w = setup_fresh_web_client(); + buffer_strcat(w->response.data, "GET / HTTP/1.1\r\n\r\n"); + + char debug[4096]; + repr(debug, sizeof(debug), w->response.data->buffer, w->response.data->len); + printf("-> \"%s\"\n", debug); + + //char expected_url_repr[4096]; + //repr(expected_url_repr, sizeof(expected_url_repr), def->url_out_repr, strlen(def->url_out_repr)); + + expect_string(__wrap_mysendfile, filename, "/"); + + web_client_process_request(w); + + //assert_string_equal(w->decoded_query_string, def->query_out); + destroy_web_client(w); + free(localhost); + localhost = NULL; +} + +static void two_slashes(void **state) +{ + (void)state; + + if (localhost != NULL) + free(localhost); + localhost = malloc(sizeof(RRDHOST)); + + struct web_client *w = setup_fresh_web_client(); + buffer_strcat(w->response.data, "GET // HTTP/1.1\r\n\r\n"); + + char debug[4096]; + repr(debug, sizeof(debug), w->response.data->buffer, w->response.data->len); + printf("-> \"%s\"\n", debug); + + //char expected_url_repr[4096]; + //repr(expected_url_repr, sizeof(expected_url_repr), def->url_out_repr, strlen(def->url_out_repr)); + + expect_string(__wrap_mysendfile, filename, "//"); + + web_client_process_request(w); + + //assert_string_equal(w->decoded_query_string, def->query_out); + destroy_web_client(w); + free(localhost); + localhost = NULL; +} + +static void absolute_url(void **state) +{ + (void)state; + + if (localhost != NULL) + free(localhost); + localhost = malloc(sizeof(RRDHOST)); + + struct web_client *w = setup_fresh_web_client(); + buffer_strcat(w->response.data, "GET http://localhost:19999/api/v1/info HTTP/1.1\r\n\r\n"); + + char debug[4096]; + repr(debug, sizeof(debug), w->response.data->buffer, w->response.data->len); + printf("-> \"%s\"\n", debug); + + //char expected_url_repr[4096]; + //repr(expected_url_repr, sizeof(expected_url_repr), def->url_out_repr, strlen(def->url_out_repr)); + + expect_value(__wrap_web_client_api_request_v1, host, localhost); + expect_value(__wrap_web_client_api_request_v1, w, w); + expect_string(__wrap_web_client_api_request_v1, url_repr, "info"); + + web_client_process_request(w); + + assert_string_equal(w->decoded_query_string, "?blah"); + destroy_web_client(w); + free(localhost); + localhost = NULL; +} + +static void valid_url(void **state) +{ + (void)state; + + if (localhost != NULL) + free(localhost); + localhost = malloc(sizeof(RRDHOST)); + + struct web_client *w = setup_fresh_web_client(); + buffer_strcat(w->response.data, "GET /api/v1/info?blah HTTP/1.1\r\n\r\n"); + + char debug[4096]; + repr(debug, sizeof(debug), w->response.data->buffer, w->response.data->len); + printf("-> \"%s\"\n", debug); + + //char expected_url_repr[4096]; + //repr(expected_url_repr, sizeof(expected_url_repr), def->url_out_repr, strlen(def->url_out_repr)); + + expect_value(__wrap_web_client_api_request_v1, host, localhost); + expect_value(__wrap_web_client_api_request_v1, w, w); + expect_string(__wrap_web_client_api_request_v1, url_repr, "info"); + + web_client_process_request(w); + + assert_string_equal(w->decoded_query_string, "?blah"); + destroy_web_client(w); + free(localhost); + localhost = NULL; +} + +/* RFC2616, section 4.1: + + In the interest of robustness, servers SHOULD ignore any empty + line(s) received where a Request-Line is expected. In other words, if + the server is reading the protocol stream at the beginning of a + message and receives a CRLF first, it should ignore the CRLF. +*/ +static void leading_blanks(void **state) +{ + (void)state; + + if (localhost != NULL) + free(localhost); + localhost = malloc(sizeof(RRDHOST)); + + struct web_client *w = setup_fresh_web_client(); + buffer_strcat(w->response.data, "\r\n\r\nGET /api/v1/info?blah HTTP/1.1\r\n\r\n"); + + char debug[4096]; + repr(debug, sizeof(debug), w->response.data->buffer, w->response.data->len); + printf("-> \"%s\"\n", debug); + + //char expected_url_repr[4096]; + //repr(expected_url_repr, sizeof(expected_url_repr), def->url_out_repr, strlen(def->url_out_repr)); + + expect_value(__wrap_web_client_api_request_v1, host, localhost); + expect_value(__wrap_web_client_api_request_v1, w, w); + expect_string(__wrap_web_client_api_request_v1, url_repr, "info"); + + web_client_process_request(w); + + assert_string_equal(w->decoded_query_string, "?blah"); + destroy_web_client(w); + free(localhost); + localhost = NULL; +} + +static void empty_url(void **state) +{ + (void)state; + + if (localhost != NULL) + free(localhost); + localhost = malloc(sizeof(RRDHOST)); + + struct web_client *w = setup_fresh_web_client(); + buffer_strcat(w->response.data, "GET HTTP/1.1\r\n\r\n"); + + char debug[4096]; + repr(debug, sizeof(debug), w->response.data->buffer, w->response.data->len); + printf("-> \"%s\"\n", debug); + + //char expected_url_repr[4096]; + //repr(expected_url_repr, sizeof(expected_url_repr), def->url_out_repr, strlen(def->url_out_repr)); + + expect_value(__wrap_web_client_api_request_v1, host, localhost); + expect_value(__wrap_web_client_api_request_v1, w, w); + expect_string(__wrap_web_client_api_request_v1, url_repr, "info"); + + web_client_process_request(w); + + assert_string_equal(w->decoded_query_string, "?blah"); + destroy_web_client(w); + free(localhost); + localhost = NULL; +} + +/* If the %-escape is being performed at the correct time then the url should not be treated as a query, but instead + as a path "/api/v1/info?blah?" which should despatch into the API with the given values. +*/ +static void not_a_query(void **state) +{ + (void)state; + localhost = malloc(sizeof(RRDHOST)); + + struct web_client *w = setup_fresh_web_client(); + buffer_strcat(w->response.data, "GET /api/v1/info%3fblah%3f HTTP/1.1\r\n\r\n"); + + char debug[160]; + repr(debug, sizeof(debug), w->response.data->buffer, w->response.data->len); + printf("->%s\n", debug); + + char expected_url_repr[160]; + repr(expected_url_repr, sizeof(expected_url_repr), "info?blah?", 10); + + expect_value(__wrap_web_client_api_request_v1, host, localhost); + expect_value(__wrap_web_client_api_request_v1, w, w); + expect_string(__wrap_web_client_api_request_v1, url_repr, expected_url_repr); + + web_client_process_request(w); + + assert_string_equal(w->decoded_query_string, ""); + destroy_web_client(w); + free(localhost); +} + +static void cr_in_url(void **state) +{ + (void)state; + localhost = malloc(sizeof(RRDHOST)); + + struct web_client *w = setup_fresh_web_client(); + buffer_strcat(w->response.data, "GET /api/v1/inf\ro\t?blah HTTP/1.1\r\n\r\n"); + + char debug[160]; + repr(debug, sizeof(debug), w->response.data->buffer, w->response.data->len); + printf("->%s\n", debug); + + char expected_url_repr[160]; + repr(expected_url_repr, sizeof(expected_url_repr), "inf\no\t", 6); + + web_client_process_request(w); + + assert_int_equal(w->response.code, HTTP_RESP_BAD_REQUEST); + + destroy_web_client(w); + free(localhost); +} +static void newline_in_url(void **state) +{ + (void)state; + localhost = malloc(sizeof(RRDHOST)); + + struct web_client *w = setup_fresh_web_client(); + buffer_strcat(w->response.data, "GET /api/v1/inf\no\t?blah HTTP/1.1\r\n\r\n"); + + char debug[160]; + repr(debug, sizeof(debug), w->response.data->buffer, w->response.data->len); + printf("->%s\n", debug); + + char expected_url_repr[160]; + repr(expected_url_repr, sizeof(expected_url_repr), "inf\no\t", 6); + + web_client_process_request(w); + + assert_int_equal(w->response.code, HTTP_RESP_BAD_REQUEST); + + destroy_web_client(w); + free(localhost); +} + +static void bad_version(void **state) +{ + (void)state; + localhost = malloc(sizeof(RRDHOST)); + + struct web_client *w = setup_fresh_web_client(); + buffer_strcat(w->response.data, "GET /api/v1/info?blah HTTP/1.2\r\n\r\n"); + + char debug[160]; + repr(debug, sizeof(debug), w->response.data->buffer, w->response.data->len); + printf("->%s\n", debug); + + char expected_url_repr[160]; + repr(expected_url_repr, sizeof(expected_url_repr), "inf\no\t", 6); + + web_client_process_request(w); + + assert_int_equal(w->response.code, HTTP_RESP_BAD_REQUEST); + + destroy_web_client(w); + free(localhost); +} + +static void pathless_query(void **state) +{ + (void)state; + localhost = malloc(sizeof(RRDHOST)); + + struct web_client *w = setup_fresh_web_client(); + buffer_strcat(w->response.data, "GET ?blah HTTP/1.1\r\n\r\n"); + + char debug[160]; + repr(debug, sizeof(debug), w->response.data->buffer, w->response.data->len); + printf("->%s\n", debug); + + char expected_url_repr[160]; + repr(expected_url_repr, sizeof(expected_url_repr), "inf\no\t", 6); + + web_client_process_request(w); + + assert_int_equal(w->response.code, HTTP_RESP_BAD_REQUEST); + + destroy_web_client(w); + free(localhost); +} + +static void pathless_fragment(void **state) +{ + (void)state; + localhost = malloc(sizeof(RRDHOST)); + + struct web_client *w = setup_fresh_web_client(); + buffer_strcat(w->response.data, "GET #blah HTTP/1.1\r\n\r\n"); + + char debug[160]; + repr(debug, sizeof(debug), w->response.data->buffer, w->response.data->len); + printf("->%s\n", debug); + + char expected_url_repr[160]; + repr(expected_url_repr, sizeof(expected_url_repr), "inf\no\t", 6); + + web_client_process_request(w); + + assert_int_equal(w->response.code, HTTP_RESP_BAD_REQUEST); + + destroy_web_client(w); + free(localhost); +} + +static void short_percent(void **state) +{ + (void)state; + localhost = malloc(sizeof(RRDHOST)); + + struct web_client *w = setup_fresh_web_client(); + buffer_strcat(w->response.data, "GET % HTTP/1.1\r\n\r\n"); + + char debug[160]; + repr(debug, sizeof(debug), w->response.data->buffer, w->response.data->len); + printf("->%s\n", debug); + + char expected_url_repr[160]; + repr(expected_url_repr, sizeof(expected_url_repr), "inf\no\t", 6); + + web_client_process_request(w); + + assert_int_equal(w->response.code, HTTP_RESP_BAD_REQUEST); + + destroy_web_client(w); + free(localhost); +} + +static void short_percent2(void **state) +{ + (void)state; + localhost = malloc(sizeof(RRDHOST)); + + struct web_client *w = setup_fresh_web_client(); + buffer_strcat(w->response.data, "GET %0 HTTP/1.1\r\n\r\n"); + + char debug[160]; + repr(debug, sizeof(debug), w->response.data->buffer, w->response.data->len); + printf("->%s\n", debug); + + char expected_url_repr[160]; + repr(expected_url_repr, sizeof(expected_url_repr), "inf\no\t", 6); + + web_client_process_request(w); + + assert_int_equal(w->response.code, HTTP_RESP_BAD_REQUEST); + + destroy_web_client(w); + free(localhost); +} + +static void short_percent3(void **state) +{ + (void)state; + localhost = malloc(sizeof(RRDHOST)); + + struct web_client *w = setup_fresh_web_client(); + buffer_strcat(w->response.data, "GET %"); + + char debug[160]; + repr(debug, sizeof(debug), w->response.data->buffer, w->response.data->len); + printf("->%s\n", debug); + + char expected_url_repr[160]; + repr(expected_url_repr, sizeof(expected_url_repr), "inf\no\t", 6); + + web_client_process_request(w); + + assert_int_equal(w->response.code, HTTP_RESP_BAD_REQUEST); + + destroy_web_client(w); + free(localhost); +} + +static void percent_nulls(void **state) +{ + (void)state; + localhost = malloc(sizeof(RRDHOST)); + + struct web_client *w = setup_fresh_web_client(); + buffer_strcat(w->response.data, "GET %00%00%00%00%00%00 HTTP/1.1\r\n"); + + char debug[160]; + repr(debug, sizeof(debug), w->response.data->buffer, w->response.data->len); + printf("->%s\n", debug); + + char expected_url_repr[160]; + repr(expected_url_repr, sizeof(expected_url_repr), "inf\no\t", 6); + + web_client_process_request(w); + + assert_int_equal(w->response.code, HTTP_RESP_BAD_REQUEST); + + destroy_web_client(w); + free(localhost); +} + +static void percent_invalid(void **state) +{ + (void)state; + localhost = malloc(sizeof(RRDHOST)); + + struct web_client *w = setup_fresh_web_client(); + buffer_strcat(w->response.data, "GET /%x%x%x%x%x%x HTTP/1.1\r\n"); + + char debug[160]; + repr(debug, sizeof(debug), w->response.data->buffer, w->response.data->len); + printf("->%s\n", debug); + + char expected_url_repr[160]; + repr(expected_url_repr, sizeof(expected_url_repr), "inf\no\t", 6); + + web_client_process_request(w); + + assert_int_equal(w->response.code, HTTP_RESP_BAD_REQUEST); + + destroy_web_client(w); + free(localhost); +} + +static void space_in_url(void **state) +{ + (void)state; + localhost = malloc(sizeof(RRDHOST)); + + struct web_client *w = setup_fresh_web_client(); + buffer_strcat(w->response.data, "GET / / HTTP/1.1\r\n\r\n"); + + char debug[160]; + repr(debug, sizeof(debug), w->response.data->buffer, w->response.data->len); + printf("->%s\n", debug); + + char expected_url_repr[160]; + repr(expected_url_repr, sizeof(expected_url_repr), "inf\no\t", 6); + + web_client_process_request(w); + + assert_int_equal(w->response.code, HTTP_RESP_BAD_REQUEST); + + destroy_web_client(w); + free(localhost); +} + +static void random_sploit1(void **state) +{ + (void)state; + localhost = malloc(sizeof(RRDHOST)); + + struct web_client *w = setup_fresh_web_client(); + // FIXME: Encoding probably needs to go through printf + buffer_need_bytes(w->response.data, 55); + memcpy( + w->response.data->buffer, + "GET \x03\x00\x00/*\xE0\x00\x00\x00\x00\x00Cookie: mstshash=Administr HTTP/1.1\r\n\r\n", 54); + w->response.data->len = 54; + + char debug[160]; + repr(debug, sizeof(debug), w->response.data->buffer, w->response.data->len); + printf("->%s\n", debug); + + char expected_url_repr[160]; + repr(expected_url_repr, sizeof(expected_url_repr), "inf\no\t", 6); + + web_client_process_request(w); + + assert_int_equal(w->response.code, HTTP_RESP_BAD_REQUEST); + + destroy_web_client(w); + free(localhost); +} + +static void null_in_url(void **state) +{ + (void)state; + localhost = malloc(sizeof(RRDHOST)); + + struct web_client *w = setup_fresh_web_client(); + buffer_strcat(w->response.data, "GET / / HTTP/1.1\r\n\r\n"); + w->response.data->buffer[5] = 0; + + char debug[160]; + repr(debug, sizeof(debug), w->response.data->buffer, w->response.data->len); + printf("->%s\n", debug); + + char expected_url_repr[160]; + repr(expected_url_repr, sizeof(expected_url_repr), "inf\no\t", 6); + + web_client_process_request(w); + + assert_int_equal(w->response.code, HTTP_RESP_BAD_REQUEST); + + destroy_web_client(w); + free(localhost); +} +static void many_ands(void **state) +{ + (void)state; + localhost = malloc(sizeof(RRDHOST)); + + struct web_client *w = setup_fresh_web_client(); + buffer_strcat(w->response.data, "GET foo?"); + for (size_t i = 0; i < 600; i++) + buffer_strcat(w->response.data, "&"); + buffer_strcat(w->response.data, " HTTP/1.1\r\n\r\n"); + + char debug[2048]; + repr(debug, sizeof(debug), w->response.data->buffer, w->response.data->len); + printf("->%s\n", debug); + + char expected_url_repr[160]; + repr(expected_url_repr, sizeof(expected_url_repr), "inf\no\t", 6); + + web_client_process_request(w); + + assert_int_equal(w->response.code, HTTP_RESP_BAD_REQUEST); + + destroy_web_client(w); + free(localhost); +} + +int main(void) +{ + debug_flags = 0xffffffffffff; + int fails = 0; + + struct CMUnitTest static_tests[] = { + cmocka_unit_test(only_root), cmocka_unit_test(two_slashes), cmocka_unit_test(valid_url), + cmocka_unit_test(leading_blanks), cmocka_unit_test(empty_url), cmocka_unit_test(newline_in_url), + cmocka_unit_test(not_a_query), cmocka_unit_test(cr_in_url), cmocka_unit_test(pathless_query), + cmocka_unit_test(pathless_fragment), cmocka_unit_test(short_percent), cmocka_unit_test(short_percent2), + cmocka_unit_test(short_percent3), cmocka_unit_test(percent_nulls), cmocka_unit_test(percent_invalid), + cmocka_unit_test(space_in_url), cmocka_unit_test(random_sploit1), cmocka_unit_test(null_in_url), + cmocka_unit_test(absolute_url), + // cmocka_unit_test(many_ands), CMocka cannot recover after this crash + cmocka_unit_test(bad_version) + }; + (void)many_ands; + + fails += cmocka_run_group_tests_name("static_tests", static_tests, NULL, NULL); + return fails; +} diff --git a/web/api/tests/web_api.c b/web/api/tests/web_api.c new file mode 100644 index 0000000..3cc0a79 --- /dev/null +++ b/web/api/tests/web_api.c @@ -0,0 +1,474 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#include "../../../libnetdata/libnetdata.h" +#include "../../../libnetdata/required_dummies.h" +#include "../../../database/rrd.h" +#include "../../../web/server/web_client.h" +#include <setjmp.h> +#include <cmocka.h> +#include <stdbool.h> + +RRDHOST *sql_create_host_by_uuid(char *hostname) +{ + (void) hostname; + return NULL; +} + +RRDHOST *__wrap_sql_create_host_by_uuid(char *hostname) +{ + (void) hostname; + return NULL; +} + +void repr(char *result, int result_size, char const *buf, int size) +{ + int n; + char *end = result + result_size - 1; + unsigned char const *ubuf = (unsigned char const *)buf; + while (size && result_size > 0) { + if (*ubuf <= 0x20 || *ubuf >= 0x80) { + n = snprintf(result, result_size, "\\%02X", *ubuf); + } else { + *result = *ubuf; + n = 1; + } + result += n; + result_size -= n; + ubuf++; + size--; + } + if (result_size > 0) + *(result++) = 0; + else + *end = 0; +} + +// ---------------------------------- Mocking accesses from web_client ------------------------------------------------ + +ssize_t send(int sockfd, const void *buf, size_t len, int flags) +{ + info("Mocking send: %zu bytes\n", len); + (void)sockfd; + (void)buf; + (void)flags; + return len; +} + +RRDHOST *__wrap_rrdhost_find_by_hostname(const char *hostname, uint32_t hash) +{ + (void)hostname; + (void)hash; + return NULL; +} + +/* Note: we've got some intricate code inside the global statistics module, might be useful to pull it inside the + test set instead of mocking it. */ +void __wrap_finished_web_request_statistics( + uint64_t dt, uint64_t bytes_received, uint64_t bytes_sent, uint64_t content_size, uint64_t compressed_content_size) +{ + (void)dt; + (void)bytes_received; + (void)bytes_sent; + (void)content_size; + (void)compressed_content_size; +} + +char *__wrap_config_get(struct config *root, const char *section, const char *name, const char *default_value) +{ + if (!strcmp(section, CONFIG_SECTION_WEB) && !strcmp(name, "web files owner")) + return "netdata"; + (void)root; + (void)default_value; + return "UNKNOWN FIX ME"; +} + +int __wrap_web_client_api_request_v1(RRDHOST *host, struct web_client *w, char *url) +{ + char url_repr[160]; + repr(url_repr, sizeof(url_repr), url, strlen(url)); + info("web_client_api_request_v1(url=\"%s\")\n", url_repr); + check_expected_ptr(host); + check_expected_ptr(w); + check_expected_ptr(url_repr); + return HTTP_RESP_OK; +} + +int __wrap_rrdpush_receiver_thread_spawn(RRDHOST *host, struct web_client *w, char *url) +{ + (void)host; + (void)w; + (void)url; + return 0; +} + +RRDHOST *__wrap_rrdhost_find_by_guid(const char *guid, uint32_t hash) +{ + (void)guid; + (void)hash; + printf("FIXME: rrdset_find_guid\n"); + return NULL; +} + +RRDSET *__wrap_rrdset_find_byname(RRDHOST *host, const char *name) +{ + (void)host; + (void)name; + printf("FIXME: rrdset_find_byname\n"); + return NULL; +} + +RRDSET *__wrap_rrdset_find(RRDHOST *host, const char *id) +{ + (void)host; + (void)id; + printf("FIXME: rrdset_find\n"); + return NULL; +} + +// -------------------------------- Mocking the log - capture per-test ------------------------------------------------ + +char log_buffer[10240] = { 0 }; +void __wrap_debug_int(const char *file, const char *function, const unsigned long line, const char *fmt, ...) +{ + (void)file; + (void)function; + (void)line; + va_list args; + va_start(args, fmt); + size_t cur = strlen(log_buffer); + snprintf(log_buffer + cur, sizeof(log_buffer) - cur, " DEBUG: "); + cur = strlen(log_buffer); + vsnprintf(log_buffer + cur, sizeof(log_buffer) - cur, fmt, args); + cur = strlen(log_buffer); + snprintf(log_buffer + cur, sizeof(log_buffer) - cur, "\n"); + va_end(args); +} + +void __wrap_info_int(const char *file, const char *function, const unsigned long line, const char *fmt, ...) +{ + (void)file; + (void)function; + (void)line; + va_list args; + va_start(args, fmt); + size_t cur = strlen(log_buffer); + snprintf(log_buffer + cur, sizeof(log_buffer) - cur, " INFO: "); + cur = strlen(log_buffer); + vsnprintf(log_buffer + cur, sizeof(log_buffer) - cur, fmt, args); + cur = strlen(log_buffer); + snprintf(log_buffer + cur, sizeof(log_buffer) - cur, "\n"); + va_end(args); +} + +void __wrap_error_int( + const char *prefix, const char *file, const char *function, const unsigned long line, const char *fmt, ...) +{ + (void)prefix; + (void)file; + (void)function; + (void)line; + va_list args; + va_start(args, fmt); + size_t cur = strlen(log_buffer); + snprintf(log_buffer + cur, sizeof(log_buffer) - cur, " ERROR: "); + cur = strlen(log_buffer); + vsnprintf(log_buffer + cur, sizeof(log_buffer) - cur, fmt, args); + cur = strlen(log_buffer); + snprintf(log_buffer + cur, sizeof(log_buffer) - cur, "\n"); + va_end(args); +} + +void __wrap_fatal_int(const char *file, const char *function, const unsigned long line, const char *fmt, ...) +{ + (void)file; + (void)function; + (void)line; + va_list args; + va_start(args, fmt); + printf("FATAL: "); + vprintf(fmt, args); + printf("\n"); + va_end(args); + fail(); +} + +WEB_SERVER_MODE web_server_mode = WEB_SERVER_MODE_STATIC_THREADED; +char *netdata_configured_web_dir = "UNKNOWN FIXME"; +RRDHOST *localhost = NULL; + +struct config netdata_config = { .first_section = NULL, + .last_section = NULL, + .mutex = NETDATA_MUTEX_INITIALIZER, + .index = { .avl_tree = { .root = NULL, .compar = appconfig_section_compare }, + .rwlock = AVL_LOCK_INITIALIZER } }; + +const char *http_headers[] = { "Host: 254.254.0.1", + "User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_" // No , + "0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.70 Safari/537.36", + "Connection: keep-alive", + "X-Forwarded-For: 1.254.1.251", + "Cookie: _ga=GA1.1.1227576758.1571113676; _gid=GA1.2.1222321739.1573628979", + "X-Requested-With: XMLHttpRequest", + "Accept-Encoding: gzip, deflate", + "Cache-Control: no-cache, no-store" }; +#define MAX_HEADERS (sizeof(http_headers) / (sizeof(const char *))) + +static void build_request(struct web_buffer *wb, const char *url, bool use_cr, size_t num_headers) +{ + buffer_reset(wb); + buffer_strcat(wb, "GET "); + buffer_strcat(wb, url); + buffer_strcat(wb, " HTTP/1.1"); + if (use_cr) + buffer_strcat(wb, "\r"); + buffer_strcat(wb, "\n"); + for (size_t i = 0; i < num_headers && i < MAX_HEADERS; i++) { + buffer_strcat(wb, http_headers[i]); + if (use_cr) + buffer_strcat(wb, "\r"); + buffer_strcat(wb, "\n"); + } + if (use_cr) + buffer_strcat(wb, "\r"); + buffer_strcat(wb, "\n"); +} + +/* Note: this is not a CMocka group_test_setup/teardown pair. This is performed per-test. +*/ +static struct web_client *setup_fresh_web_client() +{ + struct web_client *w = (struct web_client *)malloc(sizeof(struct web_client)); + memset(w, 0, sizeof(struct web_client)); + w->response.data = buffer_create(NETDATA_WEB_RESPONSE_INITIAL_SIZE); + w->response.data->date = 0; // Valgrind uninitialised value + w->response.data->expires = 0; // Valgrind uninitialised value + w->response.data->options = 0; // Valgrind uninitialised value + w->response.header = buffer_create(NETDATA_WEB_RESPONSE_HEADER_SIZE); + w->response.header_output = buffer_create(NETDATA_WEB_RESPONSE_HEADER_SIZE); + strcpy(w->origin, "*"); // Simulate web_client_create_on_fd() + w->cookie1[0] = 0; // Simulate web_client_create_on_fd() + w->cookie2[0] = 0; // Simulate web_client_create_on_fd() + w->acl = 0x1f; // Everything on + return w; +} + +static void destroy_web_client(struct web_client *w) +{ + buffer_free(w->response.data); + buffer_free(w->response.header); + buffer_free(w->response.header_output); + free(w); +} + +// ---------------------------------- Parameterized test-families ----------------------------------------------------- +// There is no way to pass a parameter block into the setup fixture, we would have to patch CMocka and maintain it +// locally. (The void **current_state in _run_group_tests would be set from a parameter). This is unfortunate as a +// parameteric unit-tester needs to be to pass parameters to the fixtures. We are faking this by calculating the +// space of tests in the launcher, passing an array of identical unit-tests to CMocka and then counting through the +// parameters in the shared state passed between tests. To initialise this counter structure we use this global to +// pass from the launcher (test-builder) to the setup-fixture. + +void *shared_test_state = NULL; + +// -------------------------------- Test family for /api/v1/info ------------------------------------------------------ + +struct test_def { + size_t num_headers; // Index coordinate + size_t prefix_len; // Index coordinate + char name[80]; + size_t full_len; + struct web_client *instance; // Used within this single test + bool completed, use_cr; + struct test_def *next, *prev; +}; + +static void api_info(void **state) +{ + (void)state; + struct test_def *def = (struct test_def *)shared_test_state; + shared_test_state = def->next; + + if (def->prev != NULL && !def->prev->completed && strlen(log_buffer) > 0) { + printf("Log of failing case %s:\n", def->prev->name); + puts(log_buffer); + } + log_buffer[0] = 0; + if (localhost != NULL) + free(localhost); + localhost = calloc(1,sizeof(RRDHOST)); + + def->instance = setup_fresh_web_client(); + build_request(def->instance->response.data, "/api/v1/info", def->use_cr, def->num_headers); + def->instance->response.data->len = def->prefix_len; + + char buffer_repr[1024]; + repr(buffer_repr, sizeof(buffer_repr), def->instance->response.data->buffer,def->prefix_len); + info("Buffer contains: %s [first %zu]", buffer_repr,def->prefix_len); + if (def->prefix_len == def->full_len) { + expect_value(__wrap_web_client_api_request_v1, host, localhost); + expect_value(__wrap_web_client_api_request_v1, w, def->instance); + expect_string(__wrap_web_client_api_request_v1, url_repr, "info"); + } + + web_client_process_request(def->instance); + + if (def->prefix_len == def->full_len) + assert_int_equal(def->instance->flags & WEB_CLIENT_FLAG_WAIT_RECEIVE, 0); + else + assert_int_equal(def->instance->flags & WEB_CLIENT_FLAG_WAIT_RECEIVE, WEB_CLIENT_FLAG_WAIT_RECEIVE); + assert_int_equal(def->instance->mode, WEB_CLIENT_MODE_NORMAL); + def->completed = true; + log_buffer[0] = 0; +} + +static int api_info_launcher() +{ + size_t num_tests = 0; + struct web_client *template = setup_fresh_web_client(); + struct test_def *current, *head = NULL; + struct test_def *prev = NULL; + + for (size_t i = 0; i < MAX_HEADERS; i++) { + build_request(template->response.data, "/api/v1/info", true, i); + for (size_t j = 0; j <= template->response.data->len; j++) { + if (j == 0 && i > 0) + continue; // All zero-length prefixes are identical, skip after first time + current = malloc(sizeof(struct test_def)); + if (prev != NULL) + prev->next = current; + else + head = current; + current->prev = prev; + prev = current; + + current->num_headers = i; + current->prefix_len = j; + current->full_len = template->response.data->len; + current->instance = NULL; + current->next = NULL; + current->use_cr = true; + current->completed = false; + sprintf( + current->name, "/api/v1/info@%zu,%zu/%zu+%d", current->num_headers, current->prefix_len, + current->full_len,true); + num_tests++; + } + } + for (size_t i = 0; i < MAX_HEADERS; i++) { + build_request(template->response.data, "/api/v1/info", false, i); + for (size_t j = 0; j <= template->response.data->len; j++) { + if (j == 0 && i > 0) + continue; // All zero-length prefixes are identical, skip after first time + current = malloc(sizeof(struct test_def)); + if (prev != NULL) + prev->next = current; + else + head = current; + current->prev = prev; + prev = current; + + current->num_headers = i; + current->prefix_len = j; + current->full_len = template->response.data->len; + current->instance = NULL; + current->next = NULL; + current->use_cr = false; + current->completed = false; + sprintf( + current->name, "/api/v1/info@%zu,%zu/%zu+%d", current->num_headers, current->prefix_len, + current->full_len,false); + num_tests++; + } + } + + struct CMUnitTest *tests = calloc(num_tests, sizeof(struct CMUnitTest)); + current = head; + for (size_t i = 0; i < num_tests; i++) { + tests[i].name = current->name; + tests[i].test_func = api_info; + tests[i].setup_func = NULL; + tests[i].teardown_func = NULL; + tests[i].initial_state = NULL; + current = current->next; + } + + printf("Setup %zu tests in %p\n", num_tests, head); + shared_test_state = head; + int fails = _cmocka_run_group_tests("web_api", tests, num_tests, NULL, NULL); + free(tests); + destroy_web_client(template); + current = head; + while (current != NULL) { + struct test_def *c = current; + current = current->next; + if (c->instance != NULL) // Clean up resources from tests that failed + destroy_web_client(c->instance); + free(c); + } + if (localhost!=NULL) + free(localhost); + return fails; +} + +/* Raw notes for the cases that we did not use in the unit testing suite. + Leaving them here instead of deleting them in-case we expand the suite during the + work on the URL parser. + + Any ' ' in the URI -> invalid response (Description in 5.1 of RFC2616) + Characters that can't be in paths #;? + "GET /apb/../api/v1/info" HTTP/1.1\r\n" + + https://github.com/uriparser/uriparser/blob/uriparser-0.9.3/test/FourSuite.cpp + Not clear why some of these are illegal -> reserved chars? + + ASSERT_TRUE(testBadUri("beepbeep\x07\x07", 8)); + ASSERT_TRUE(testBadUri("\n", 0)); + ASSERT_TRUE(testBadUri("::", 0)); // not OK, per Roy Fielding on the W3C uri list on 2004-04-01 + + // the following test cases are from a Perl script by David A. Wheeler + // at http://www.dwheeler.com/secure-programs/url.pl + ASSERT_TRUE(testBadUri("http://www yahoo.com", 10)); + ASSERT_TRUE(testBadUri("http://www.yahoo.com/hello world/", 26)); + ASSERT_TRUE(testBadUri("http://www.yahoo.com/yelp.html#\"", 31)); + + // the following test cases are from a Haskell program by Graham Klyne + // at http://www.ninebynine.org/Software/HaskellUtils/Network/URITest.hs + ASSERT_TRUE(testBadUri("[2010:836B:4179::836B:4179]", 0)); + ASSERT_TRUE(testBadUri(" ", 0)); + ASSERT_TRUE(testBadUri("%", 1)); + ASSERT_TRUE(testBadUri("A%Z", 2)); + ASSERT_TRUE(testBadUri("%ZZ", 1)); + ASSERT_TRUE(testBadUri("%AZ", 2)); + ASSERT_TRUE(testBadUri("A C", 1)); + ASSERT_TRUE(testBadUri("A\\'C", 1)); // r"A\'C" + ASSERT_TRUE(testBadUri("A`C", 1)); + ASSERT_TRUE(testBadUri("A<C", 1)); + ASSERT_TRUE(testBadUri("A>C", 1)); + ASSERT_TRUE(testBadUri("A^C", 1)); + ASSERT_TRUE(testBadUri("A\\\\C", 1)); // r'A\\C' + ASSERT_TRUE(testBadUri("A{C", 1)); + ASSERT_TRUE(testBadUri("A|C", 1)); + ASSERT_TRUE(testBadUri("A}C", 1)); + ASSERT_TRUE(testBadUri("A[C", 1)); + ASSERT_TRUE(testBadUri("A]C", 1)); + ASSERT_TRUE(testBadUri("A[**]C", 1)); + ASSERT_TRUE(testBadUri("http://[xyz]/", 8)); + ASSERT_TRUE(testBadUri("http://]/", 7)); + ASSERT_TRUE(testBadUri("http://example.org/[2010:836B:4179::836B:4179]", 19)); + ASSERT_TRUE(testBadUri("http://example.org/abc#[2010:836B:4179::836B:4179]", 23)); + ASSERT_TRUE(testBadUri("http://example.org/xxx/[qwerty]#a[b]", 23)); + + // from a post to the W3C uri list on 2004-02-17 + // breaks at 22 instead of 17 because everything up to that point is a valid userinfo + ASSERT_TRUE(testBadUri("http://w3c.org:80path1/path2", 22)); + +*/ + +int main(void) +{ + debug_flags = 0xffffffffffff; + int fails = 0; + fails += api_info_launcher(); + + return fails; +} diff --git a/web/api/web_api_v1.c b/web/api/web_api_v1.c new file mode 100644 index 0000000..73ac15d --- /dev/null +++ b/web/api/web_api_v1.c @@ -0,0 +1,1063 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#include "web_api_v1.h" + +char *api_secret; + +static struct { + const char *name; + uint32_t hash; + RRDR_OPTIONS value; +} api_v1_data_options[] = { + { "nonzero" , 0 , RRDR_OPTION_NONZERO} + , {"flip" , 0 , RRDR_OPTION_REVERSED} + , {"reversed" , 0 , RRDR_OPTION_REVERSED} + , {"reverse" , 0 , RRDR_OPTION_REVERSED} + , {"jsonwrap" , 0 , RRDR_OPTION_JSON_WRAP} + , {"min2max" , 0 , RRDR_OPTION_MIN2MAX} + , {"ms" , 0 , RRDR_OPTION_MILLISECONDS} + , {"milliseconds" , 0 , RRDR_OPTION_MILLISECONDS} + , {"abs" , 0 , RRDR_OPTION_ABSOLUTE} + , {"absolute" , 0 , RRDR_OPTION_ABSOLUTE} + , {"absolute_sum" , 0 , RRDR_OPTION_ABSOLUTE} + , {"absolute-sum" , 0 , RRDR_OPTION_ABSOLUTE} + , {"display_absolute", 0 , RRDR_OPTION_DISPLAY_ABS} + , {"display-absolute", 0 , RRDR_OPTION_DISPLAY_ABS} + , {"seconds" , 0 , RRDR_OPTION_SECONDS} + , {"null2zero" , 0 , RRDR_OPTION_NULL2ZERO} + , {"objectrows" , 0 , RRDR_OPTION_OBJECTSROWS} + , {"google_json" , 0 , RRDR_OPTION_GOOGLE_JSON} + , {"google-json" , 0 , RRDR_OPTION_GOOGLE_JSON} + , {"percentage" , 0 , RRDR_OPTION_PERCENTAGE} + , {"unaligned" , 0 , RRDR_OPTION_NOT_ALIGNED} + , {"match_ids" , 0 , RRDR_OPTION_MATCH_IDS} + , {"match-ids" , 0 , RRDR_OPTION_MATCH_IDS} + , {"match_names" , 0 , RRDR_OPTION_MATCH_NAMES} + , {"match-names" , 0 , RRDR_OPTION_MATCH_NAMES} + , {"showcustomvars" , 0 , RRDR_OPTION_CUSTOM_VARS} + , {"allow_past" , 0 , RRDR_OPTION_ALLOW_PAST} + , { NULL, 0, 0} +}; + +static struct { + const char *name; + uint32_t hash; + uint32_t value; +} api_v1_data_formats[] = { + { DATASOURCE_FORMAT_DATATABLE_JSON , 0 , DATASOURCE_DATATABLE_JSON} + , {DATASOURCE_FORMAT_DATATABLE_JSONP, 0 , DATASOURCE_DATATABLE_JSONP} + , {DATASOURCE_FORMAT_JSON , 0 , DATASOURCE_JSON} + , {DATASOURCE_FORMAT_JSONP , 0 , DATASOURCE_JSONP} + , {DATASOURCE_FORMAT_SSV , 0 , DATASOURCE_SSV} + , {DATASOURCE_FORMAT_CSV , 0 , DATASOURCE_CSV} + , {DATASOURCE_FORMAT_TSV , 0 , DATASOURCE_TSV} + , {"tsv-excel" , 0 , DATASOURCE_TSV} + , {DATASOURCE_FORMAT_HTML , 0 , DATASOURCE_HTML} + , {DATASOURCE_FORMAT_JS_ARRAY , 0 , DATASOURCE_JS_ARRAY} + , {DATASOURCE_FORMAT_SSV_COMMA , 0 , DATASOURCE_SSV_COMMA} + , {DATASOURCE_FORMAT_CSV_JSON_ARRAY , 0 , DATASOURCE_CSV_JSON_ARRAY} + , {DATASOURCE_FORMAT_CSV_MARKDOWN , 0 , DATASOURCE_CSV_MARKDOWN} + , { NULL, 0, 0} +}; + +static struct { + const char *name; + uint32_t hash; + uint32_t value; +} api_v1_data_google_formats[] = { + // this is not error - when google requests json, it expects javascript + // https://developers.google.com/chart/interactive/docs/dev/implementing_data_source#responseformat + { "json" , 0 , DATASOURCE_DATATABLE_JSONP} + , {"html" , 0 , DATASOURCE_HTML} + , {"csv" , 0 , DATASOURCE_CSV} + , {"tsv-excel", 0 , DATASOURCE_TSV} + , { NULL, 0, 0} +}; + +void web_client_api_v1_init(void) { + int i; + + for(i = 0; api_v1_data_options[i].name ; i++) + api_v1_data_options[i].hash = simple_hash(api_v1_data_options[i].name); + + for(i = 0; api_v1_data_formats[i].name ; i++) + api_v1_data_formats[i].hash = simple_hash(api_v1_data_formats[i].name); + + for(i = 0; api_v1_data_google_formats[i].name ; i++) + api_v1_data_google_formats[i].hash = simple_hash(api_v1_data_google_formats[i].name); + + web_client_api_v1_init_grouping(); + + uuid_t uuid; + + // generate + uuid_generate(uuid); + + // unparse (to string) + char uuid_str[37]; + uuid_unparse_lower(uuid, uuid_str); +} + +char *get_mgmt_api_key(void) { + char filename[FILENAME_MAX + 1]; + snprintfz(filename, FILENAME_MAX, "%s/netdata.api.key", netdata_configured_varlib_dir); + char *api_key_filename=config_get(CONFIG_SECTION_REGISTRY, "netdata management api key file", filename); + static char guid[GUID_LEN + 1] = ""; + + if(likely(guid[0])) + return guid; + + // read it from disk + int fd = open(api_key_filename, O_RDONLY); + if(fd != -1) { + char buf[GUID_LEN + 1]; + if(read(fd, buf, GUID_LEN) != GUID_LEN) + error("Failed to read management API key from '%s'", api_key_filename); + else { + buf[GUID_LEN] = '\0'; + if(regenerate_guid(buf, guid) == -1) { + error("Failed to validate management API key '%s' from '%s'.", + buf, api_key_filename); + + guid[0] = '\0'; + } + } + close(fd); + } + + // generate a new one? + if(!guid[0]) { + uuid_t uuid; + + uuid_generate_time(uuid); + uuid_unparse_lower(uuid, guid); + guid[GUID_LEN] = '\0'; + + // save it + fd = open(api_key_filename, O_WRONLY|O_CREAT|O_TRUNC, 444); + if(fd == -1) + fatal("Cannot create unique management API key file '%s'. Please fix this.", api_key_filename); + + if(write(fd, guid, GUID_LEN) != GUID_LEN) + fatal("Cannot write the unique management API key file '%s'. Please fix this.", api_key_filename); + + close(fd); + } + + return guid; +} + +void web_client_api_v1_management_init(void) { + api_secret = get_mgmt_api_key(); +} + +inline uint32_t web_client_api_request_v1_data_options(char *o) { + uint32_t ret = 0x00000000; + char *tok; + + while(o && *o && (tok = mystrsep(&o, ", |"))) { + if(!*tok) continue; + + uint32_t hash = simple_hash(tok); + int i; + for(i = 0; api_v1_data_options[i].name ; i++) { + if (unlikely(hash == api_v1_data_options[i].hash && !strcmp(tok, api_v1_data_options[i].name))) { + ret |= api_v1_data_options[i].value; + break; + } + } + } + + return ret; +} + +inline uint32_t web_client_api_request_v1_data_format(char *name) { + uint32_t hash = simple_hash(name); + int i; + + for(i = 0; api_v1_data_formats[i].name ; i++) { + if (unlikely(hash == api_v1_data_formats[i].hash && !strcmp(name, api_v1_data_formats[i].name))) { + return api_v1_data_formats[i].value; + } + } + + return DATASOURCE_JSON; +} + +inline uint32_t web_client_api_request_v1_data_google_format(char *name) { + uint32_t hash = simple_hash(name); + int i; + + for(i = 0; api_v1_data_google_formats[i].name ; i++) { + if (unlikely(hash == api_v1_data_google_formats[i].hash && !strcmp(name, api_v1_data_google_formats[i].name))) { + return api_v1_data_google_formats[i].value; + } + } + + return DATASOURCE_JSON; +} + +int web_client_api_request_v1_alarms_select (char *url) { + int all = 0; + while(url) { + char *value = mystrsep(&url, "&"); + if (!value || !*value) continue; + + if(!strcmp(value, "all")) all = 1; + else if(!strcmp(value, "active")) all = 0; + } + + return all; +} + +inline int web_client_api_request_v1_alarms(RRDHOST *host, struct web_client *w, char *url) { + int all = web_client_api_request_v1_alarms_select(url); + + buffer_flush(w->response.data); + w->response.data->contenttype = CT_APPLICATION_JSON; + health_alarms2json(host, w->response.data, all); + buffer_no_cacheable(w->response.data); + return HTTP_RESP_OK; +} + +inline int web_client_api_request_v1_alarms_values(RRDHOST *host, struct web_client *w, char *url) { + int all = web_client_api_request_v1_alarms_select(url); + + buffer_flush(w->response.data); + w->response.data->contenttype = CT_APPLICATION_JSON; + health_alarms_values2json(host, w->response.data, all); + buffer_no_cacheable(w->response.data); + return HTTP_RESP_OK; +} + +inline int web_client_api_request_v1_alarm_count(RRDHOST *host, struct web_client *w, char *url) { + RRDCALC_STATUS status = RRDCALC_STATUS_RAISED; + BUFFER *contexts = NULL; + + buffer_flush(w->response.data); + buffer_sprintf(w->response.data, "["); + + while(url) { + char *value = mystrsep(&url, "&"); + if(!value || !*value) continue; + + char *name = mystrsep(&value, "="); + if(!name || !*name) continue; + if(!value || !*value) continue; + + debug(D_WEB_CLIENT, "%llu: API v1 alarm_count query param '%s' with value '%s'", w->id, name, value); + + char* p = value; + if(!strcmp(name, "status")) { + while ((*p = toupper(*p))) p++; + if (!strcmp("CRITICAL", value)) status = RRDCALC_STATUS_CRITICAL; + else if (!strcmp("WARNING", value)) status = RRDCALC_STATUS_WARNING; + else if (!strcmp("UNINITIALIZED", value)) status = RRDCALC_STATUS_UNINITIALIZED; + else if (!strcmp("UNDEFINED", value)) status = RRDCALC_STATUS_UNDEFINED; + else if (!strcmp("REMOVED", value)) status = RRDCALC_STATUS_REMOVED; + else if (!strcmp("CLEAR", value)) status = RRDCALC_STATUS_CLEAR; + } + else if(!strcmp(name, "context") || !strcmp(name, "ctx")) { + if(!contexts) contexts = buffer_create(255); + buffer_strcat(contexts, "|"); + buffer_strcat(contexts, value); + } + } + + health_aggregate_alarms(host, w->response.data, contexts, status); + + buffer_sprintf(w->response.data, "]\n"); + w->response.data->contenttype = CT_APPLICATION_JSON; + buffer_no_cacheable(w->response.data); + + buffer_free(contexts); + return 200; +} + +inline int web_client_api_request_v1_alarm_log(RRDHOST *host, struct web_client *w, char *url) { + uint32_t after = 0; + + while(url) { + char *value = mystrsep(&url, "&"); + if (!value || !*value) continue; + + char *name = mystrsep(&value, "="); + if(!name || !*name) continue; + if(!value || !*value) continue; + + if(!strcmp(name, "after")) after = (uint32_t)strtoul(value, NULL, 0); + } + + buffer_flush(w->response.data); + w->response.data->contenttype = CT_APPLICATION_JSON; + health_alarm_log2json(host, w->response.data, after); + return HTTP_RESP_OK; +} + +inline int web_client_api_request_single_chart(RRDHOST *host, struct web_client *w, char *url, void callback(RRDSET *st, BUFFER *buf)) { + int ret = HTTP_RESP_BAD_REQUEST; + char *chart = NULL; + + buffer_flush(w->response.data); + + while(url) { + char *value = mystrsep(&url, "&"); + if(!value || !*value) continue; + + char *name = mystrsep(&value, "="); + if(!name || !*name) continue; + if(!value || !*value) continue; + + // name and value are now the parameters + // they are not null and not empty + + if(!strcmp(name, "chart")) chart = value; + //else { + /// buffer_sprintf(w->response.data, "Unknown parameter '%s' in request.", name); + // goto cleanup; + //} + } + + if(!chart || !*chart) { + buffer_sprintf(w->response.data, "No chart id is given at the request."); + goto cleanup; + } + + RRDSET *st = rrdset_find(host, chart); + if(!st) st = rrdset_find_byname(host, chart); + if(!st) { + buffer_strcat(w->response.data, "Chart is not found: "); + buffer_strcat_htmlescape(w->response.data, chart); + ret = HTTP_RESP_NOT_FOUND; + goto cleanup; + } + + w->response.data->contenttype = CT_APPLICATION_JSON; + st->last_accessed_time = now_realtime_sec(); + callback(st, w->response.data); + return HTTP_RESP_OK; + + cleanup: + return ret; +} + +inline int web_client_api_request_v1_alarm_variables(RRDHOST *host, struct web_client *w, char *url) { + return web_client_api_request_single_chart(host, w, url, health_api_v1_chart_variables2json); +} + +inline int web_client_api_request_v1_charts(RRDHOST *host, struct web_client *w, char *url) { + (void)url; + + buffer_flush(w->response.data); + w->response.data->contenttype = CT_APPLICATION_JSON; + charts2json(host, w->response.data, 0, 0); + return HTTP_RESP_OK; +} + +inline int web_client_api_request_v1_archivedcharts(RRDHOST *host __maybe_unused, struct web_client *w, char *url) { + (void)url; + + buffer_flush(w->response.data); + w->response.data->contenttype = CT_APPLICATION_JSON; +#ifdef ENABLE_DBENGINE + if (host->rrd_memory_mode == RRD_MEMORY_MODE_DBENGINE) + sql_rrdset2json(host, w->response.data); +#endif + return HTTP_RESP_OK; +} + +inline int web_client_api_request_v1_chart(RRDHOST *host, struct web_client *w, char *url) { + return web_client_api_request_single_chart(host, w, url, rrd_stats_api_v1_chart); +} + +void fix_google_param(char *s) { + if(unlikely(!s)) return; + + for( ; *s ;s++) { + if(!isalnum(*s) && *s != '.' && *s != '_' && *s != '-') + *s = '_'; + } +} + +// returns the HTTP code +inline int web_client_api_request_v1_data(RRDHOST *host, struct web_client *w, char *url) { + debug(D_WEB_CLIENT, "%llu: API v1 data with URL '%s'", w->id, url); + + int ret = HTTP_RESP_BAD_REQUEST; + BUFFER *dimensions = NULL; + + buffer_flush(w->response.data); + + char *google_version = "0.6", + *google_reqId = "0", + *google_sig = "0", + *google_out = "json", + *responseHandler = NULL, + *outFileName = NULL; + + time_t last_timestamp_in_data = 0, google_timestamp = 0; + + char *chart = NULL + , *before_str = NULL + , *after_str = NULL + , *group_time_str = NULL + , *points_str = NULL + , *context = NULL + , *chart_label_key = NULL; + + int group = RRDR_GROUPING_AVERAGE; + uint32_t format = DATASOURCE_JSON; + uint32_t options = 0x00000000; + + while(url) { + char *value = mystrsep(&url, "&"); + if(!value || !*value) continue; + + char *name = mystrsep(&value, "="); + if(!name || !*name) continue; + if(!value || !*value) continue; + + debug(D_WEB_CLIENT, "%llu: API v1 data query param '%s' with value '%s'", w->id, name, value); + + // name and value are now the parameters + // they are not null and not empty + + if(!strcmp(name, "context")) context = value; + else if(!strcmp(name, "chart_label_key")) chart_label_key = value; + else if(!strcmp(name, "chart")) chart = value; + else if(!strcmp(name, "dimension") || !strcmp(name, "dim") || !strcmp(name, "dimensions") || !strcmp(name, "dims")) { + if(!dimensions) dimensions = buffer_create(100); + buffer_strcat(dimensions, "|"); + buffer_strcat(dimensions, value); + } + else if(!strcmp(name, "after")) after_str = value; + else if(!strcmp(name, "before")) before_str = value; + else if(!strcmp(name, "points")) points_str = value; + else if(!strcmp(name, "gtime")) group_time_str = value; + else if(!strcmp(name, "group")) { + group = web_client_api_request_v1_data_group(value, RRDR_GROUPING_AVERAGE); + } + else if(!strcmp(name, "format")) { + format = web_client_api_request_v1_data_format(value); + } + else if(!strcmp(name, "options")) { + options |= web_client_api_request_v1_data_options(value); + } + else if(!strcmp(name, "callback")) { + responseHandler = value; + } + else if(!strcmp(name, "filename")) { + outFileName = value; + } + else if(!strcmp(name, "tqx")) { + // parse Google Visualization API options + // https://developers.google.com/chart/interactive/docs/dev/implementing_data_source + char *tqx_name, *tqx_value; + + while(value) { + tqx_value = mystrsep(&value, ";"); + if(!tqx_value || !*tqx_value) continue; + + tqx_name = mystrsep(&tqx_value, ":"); + if(!tqx_name || !*tqx_name) continue; + if(!tqx_value || !*tqx_value) continue; + + if(!strcmp(tqx_name, "version")) + google_version = tqx_value; + else if(!strcmp(tqx_name, "reqId")) + google_reqId = tqx_value; + else if(!strcmp(tqx_name, "sig")) { + google_sig = tqx_value; + google_timestamp = strtoul(google_sig, NULL, 0); + } + else if(!strcmp(tqx_name, "out")) { + google_out = tqx_value; + format = web_client_api_request_v1_data_google_format(google_out); + } + else if(!strcmp(tqx_name, "responseHandler")) + responseHandler = tqx_value; + else if(!strcmp(tqx_name, "outFileName")) + outFileName = tqx_value; + } + } + } + + // validate the google parameters given + fix_google_param(google_out); + fix_google_param(google_sig); + fix_google_param(google_reqId); + fix_google_param(google_version); + fix_google_param(responseHandler); + fix_google_param(outFileName); + + RRDSET *st = NULL; + + if((!chart || !*chart) && (!context)) { + buffer_sprintf(w->response.data, "No chart id is given at the request."); + goto cleanup; + } + + struct context_param *context_param_list = NULL; + if (context && !chart) { + RRDSET *st1; + uint32_t context_hash = simple_hash(context); + + rrdhost_rdlock(host); + rrdset_foreach_read(st1, host) { + if (st1->hash_context == context_hash && !strcmp(st1->context, context) && + (!chart_label_key || rrdset_contains_label_keylist(st1, chart_label_key))) + build_context_param_list(&context_param_list, st1); + } + rrdhost_unlock(host); + if (likely(context_param_list && context_param_list->rd)) // Just set the first one + st = context_param_list->rd->rrdset; + } + else { + st = rrdset_find(host, chart); + if (!st) + st = rrdset_find_byname(host, chart); + if (likely(st)) + st->last_accessed_time = now_realtime_sec(); + } + + if (!st && !context_param_list) { + if (context && !chart) { + if (!chart_label_key) { + buffer_strcat(w->response.data, "Context is not found: "); + buffer_strcat_htmlescape(w->response.data, context); + } else { + buffer_strcat(w->response.data, "Context: "); + buffer_strcat_htmlescape(w->response.data, context); + buffer_strcat(w->response.data, " or chart label key: "); + buffer_strcat_htmlescape(w->response.data, chart_label_key); + buffer_strcat(w->response.data, " not found"); + } + } + else { + buffer_strcat(w->response.data, "Chart is not found: "); + buffer_strcat_htmlescape(w->response.data, chart); + } + ret = HTTP_RESP_NOT_FOUND; + goto cleanup; + } + + long long before = (before_str && *before_str)?str2l(before_str):0; + long long after = (after_str && *after_str) ?str2l(after_str):-600; + int points = (points_str && *points_str)?str2i(points_str):0; + long group_time = (group_time_str && *group_time_str)?str2l(group_time_str):0; + + debug(D_WEB_CLIENT, "%llu: API command 'data' for chart '%s', dimensions '%s', after '%lld', before '%lld', points '%d', group '%d', format '%u', options '0x%08x'" + , w->id + , chart + , (dimensions)?buffer_tostring(dimensions):"" + , after + , before + , points + , group + , format + , options + ); + + if(outFileName && *outFileName) { + buffer_sprintf(w->response.header, "Content-Disposition: attachment; filename=\"%s\"\r\n", outFileName); + debug(D_WEB_CLIENT, "%llu: generating outfilename header: '%s'", w->id, outFileName); + } + + if(format == DATASOURCE_DATATABLE_JSONP) { + if(responseHandler == NULL) + responseHandler = "google.visualization.Query.setResponse"; + + debug(D_WEB_CLIENT_ACCESS, "%llu: GOOGLE JSON/JSONP: version = '%s', reqId = '%s', sig = '%s', out = '%s', responseHandler = '%s', outFileName = '%s'", + w->id, google_version, google_reqId, google_sig, google_out, responseHandler, outFileName + ); + + buffer_sprintf(w->response.data, + "%s({version:'%s',reqId:'%s',status:'ok',sig:'%ld',table:", + responseHandler, google_version, google_reqId, st->last_updated.tv_sec); + } + else if(format == DATASOURCE_JSONP) { + if(responseHandler == NULL) + responseHandler = "callback"; + + buffer_strcat(w->response.data, responseHandler); + buffer_strcat(w->response.data, "("); + } + + ret = rrdset2anything_api_v1(st, w->response.data, dimensions, format, points, after, before, group, group_time + , options, &last_timestamp_in_data, context_param_list, chart_label_key); + + free_context_param_list(&context_param_list); + + if(format == DATASOURCE_DATATABLE_JSONP) { + if(google_timestamp < last_timestamp_in_data) + buffer_strcat(w->response.data, "});"); + + else { + // the client already has the latest data + buffer_flush(w->response.data); + buffer_sprintf(w->response.data, + "%s({version:'%s',reqId:'%s',status:'error',errors:[{reason:'not_modified',message:'Data not modified'}]});", + responseHandler, google_version, google_reqId); + } + } + else if(format == DATASOURCE_JSONP) + buffer_strcat(w->response.data, ");"); + + cleanup: + buffer_free(dimensions); + return ret; +} + +// Pings a netdata server: +// /api/v1/registry?action=hello +// +// Access to a netdata registry: +// /api/v1/registry?action=access&machine=${machine_guid}&name=${hostname}&url=${url} +// +// Delete from a netdata registry: +// /api/v1/registry?action=delete&machine=${machine_guid}&name=${hostname}&url=${url}&delete_url=${delete_url} +// +// Search for the URLs of a machine: +// /api/v1/registry?action=search&machine=${machine_guid}&name=${hostname}&url=${url}&for=${machine_guid} +// +// Impersonate: +// /api/v1/registry?action=switch&machine=${machine_guid}&name=${hostname}&url=${url}&to=${new_person_guid} +inline int web_client_api_request_v1_registry(RRDHOST *host, struct web_client *w, char *url) { + static uint32_t hash_action = 0, hash_access = 0, hash_hello = 0, hash_delete = 0, hash_search = 0, + hash_switch = 0, hash_machine = 0, hash_url = 0, hash_name = 0, hash_delete_url = 0, hash_for = 0, + hash_to = 0 /*, hash_redirects = 0 */; + + if(unlikely(!hash_action)) { + hash_action = simple_hash("action"); + hash_access = simple_hash("access"); + hash_hello = simple_hash("hello"); + hash_delete = simple_hash("delete"); + hash_search = simple_hash("search"); + hash_switch = simple_hash("switch"); + hash_machine = simple_hash("machine"); + hash_url = simple_hash("url"); + hash_name = simple_hash("name"); + hash_delete_url = simple_hash("delete_url"); + hash_for = simple_hash("for"); + hash_to = simple_hash("to"); +/* + hash_redirects = simple_hash("redirects"); +*/ + } + + char person_guid[GUID_LEN + 1] = ""; + + debug(D_WEB_CLIENT, "%llu: API v1 registry with URL '%s'", w->id, url); + + // TODO + // The browser may send multiple cookies with our id + + char *cookie = strstr(w->response.data->buffer, NETDATA_REGISTRY_COOKIE_NAME "="); + if(cookie) + strncpyz(person_guid, &cookie[sizeof(NETDATA_REGISTRY_COOKIE_NAME)], 36); + + char action = '\0'; + char *machine_guid = NULL, + *machine_url = NULL, + *url_name = NULL, + *search_machine_guid = NULL, + *delete_url = NULL, + *to_person_guid = NULL; +/* + int redirects = 0; +*/ + + // Don't cache registry responses + buffer_no_cacheable(w->response.data); + + while(url) { + char *value = mystrsep(&url, "&"); + if (!value || !*value) continue; + + char *name = mystrsep(&value, "="); + if (!name || !*name) continue; + if (!value || !*value) continue; + + debug(D_WEB_CLIENT, "%llu: API v1 registry query param '%s' with value '%s'", w->id, name, value); + + uint32_t hash = simple_hash(name); + + if(hash == hash_action && !strcmp(name, "action")) { + uint32_t vhash = simple_hash(value); + + if(vhash == hash_access && !strcmp(value, "access")) action = 'A'; + else if(vhash == hash_hello && !strcmp(value, "hello")) action = 'H'; + else if(vhash == hash_delete && !strcmp(value, "delete")) action = 'D'; + else if(vhash == hash_search && !strcmp(value, "search")) action = 'S'; + else if(vhash == hash_switch && !strcmp(value, "switch")) action = 'W'; +#ifdef NETDATA_INTERNAL_CHECKS + else error("unknown registry action '%s'", value); +#endif /* NETDATA_INTERNAL_CHECKS */ + } +/* + else if(hash == hash_redirects && !strcmp(name, "redirects")) + redirects = atoi(value); +*/ + else if(hash == hash_machine && !strcmp(name, "machine")) + machine_guid = value; + + else if(hash == hash_url && !strcmp(name, "url")) + machine_url = value; + + else if(action == 'A') { + if(hash == hash_name && !strcmp(name, "name")) + url_name = value; + } + else if(action == 'D') { + if(hash == hash_delete_url && !strcmp(name, "delete_url")) + delete_url = value; + } + else if(action == 'S') { + if(hash == hash_for && !strcmp(name, "for")) + search_machine_guid = value; + } + else if(action == 'W') { + if(hash == hash_to && !strcmp(name, "to")) + to_person_guid = value; + } +#ifdef NETDATA_INTERNAL_CHECKS + else error("unused registry URL parameter '%s' with value '%s'", name, value); +#endif /* NETDATA_INTERNAL_CHECKS */ + } + + if(unlikely(respect_web_browser_do_not_track_policy && web_client_has_donottrack(w))) { + buffer_flush(w->response.data); + buffer_sprintf(w->response.data, "Your web browser is sending 'DNT: 1' (Do Not Track). The registry requires persistent cookies on your browser to work."); + return HTTP_RESP_BAD_REQUEST; + } + + if(unlikely(action == 'H')) { + // HELLO request, dashboard ACL + if(unlikely(!web_client_can_access_dashboard(w))) + return web_client_permission_denied(w); + } + else { + // everything else, registry ACL + if(unlikely(!web_client_can_access_registry(w))) + return web_client_permission_denied(w); + } + + switch(action) { + case 'A': + if(unlikely(!machine_guid || !machine_url || !url_name)) { + error("Invalid registry request - access requires these parameters: machine ('%s'), url ('%s'), name ('%s')", machine_guid ? machine_guid : "UNSET", machine_url ? machine_url : "UNSET", url_name ? url_name : "UNSET"); + buffer_flush(w->response.data); + buffer_strcat(w->response.data, "Invalid registry Access request."); + return HTTP_RESP_BAD_REQUEST; + } + + web_client_enable_tracking_required(w); + return registry_request_access_json(host, w, person_guid, machine_guid, machine_url, url_name, now_realtime_sec()); + + case 'D': + if(unlikely(!machine_guid || !machine_url || !delete_url)) { + error("Invalid registry request - delete requires these parameters: machine ('%s'), url ('%s'), delete_url ('%s')", machine_guid?machine_guid:"UNSET", machine_url?machine_url:"UNSET", delete_url?delete_url:"UNSET"); + buffer_flush(w->response.data); + buffer_strcat(w->response.data, "Invalid registry Delete request."); + return HTTP_RESP_BAD_REQUEST; + } + + web_client_enable_tracking_required(w); + return registry_request_delete_json(host, w, person_guid, machine_guid, machine_url, delete_url, now_realtime_sec()); + + case 'S': + if(unlikely(!machine_guid || !machine_url || !search_machine_guid)) { + error("Invalid registry request - search requires these parameters: machine ('%s'), url ('%s'), for ('%s')", machine_guid?machine_guid:"UNSET", machine_url?machine_url:"UNSET", search_machine_guid?search_machine_guid:"UNSET"); + buffer_flush(w->response.data); + buffer_strcat(w->response.data, "Invalid registry Search request."); + return HTTP_RESP_BAD_REQUEST; + } + + web_client_enable_tracking_required(w); + return registry_request_search_json(host, w, person_guid, machine_guid, machine_url, search_machine_guid, now_realtime_sec()); + + case 'W': + if(unlikely(!machine_guid || !machine_url || !to_person_guid)) { + error("Invalid registry request - switching identity requires these parameters: machine ('%s'), url ('%s'), to ('%s')", machine_guid?machine_guid:"UNSET", machine_url?machine_url:"UNSET", to_person_guid?to_person_guid:"UNSET"); + buffer_flush(w->response.data); + buffer_strcat(w->response.data, "Invalid registry Switch request."); + return HTTP_RESP_BAD_REQUEST; + } + + web_client_enable_tracking_required(w); + return registry_request_switch_json(host, w, person_guid, machine_guid, machine_url, to_person_guid, now_realtime_sec()); + + case 'H': + return registry_request_hello_json(host, w); + + default: + buffer_flush(w->response.data); + buffer_strcat(w->response.data, "Invalid registry request - you need to set an action: hello, access, delete, search"); + return HTTP_RESP_BAD_REQUEST; + } +} + +static inline void web_client_api_request_v1_info_summary_alarm_statuses(RRDHOST *host, BUFFER *wb) { + int alarm_normal = 0, alarm_warn = 0, alarm_crit = 0; + RRDCALC *rc; + rrdhost_rdlock(host); + for(rc = host->alarms; rc ; rc = rc->next) { + if(unlikely(!rc->rrdset || !rc->rrdset->last_collected_time.tv_sec)) + continue; + + switch(rc->status) { + case RRDCALC_STATUS_WARNING: + alarm_warn++; + break; + case RRDCALC_STATUS_CRITICAL: + alarm_crit++; + break; + default: + alarm_normal++; + } + } + rrdhost_unlock(host); + buffer_sprintf(wb, "\t\t\"normal\": %d,\n", alarm_normal); + buffer_sprintf(wb, "\t\t\"warning\": %d,\n", alarm_warn); + buffer_sprintf(wb, "\t\t\"critical\": %d\n", alarm_crit); +} + +static inline void web_client_api_request_v1_info_mirrored_hosts(BUFFER *wb) { + RRDHOST *host; + int count = 0; + + buffer_strcat(wb, "\t\"mirrored_hosts\": [\n"); + rrd_rdlock(); + rrdhost_foreach_read(host) { + if (rrdhost_flag_check(host, RRDHOST_FLAG_ARCHIVED)) + continue; + if (count > 0) + buffer_strcat(wb, ",\n"); + + buffer_sprintf(wb, "\t\t\"%s\"", host->hostname); + count++; + } + + buffer_strcat(wb, "\n\t],\n\t\"mirrored_hosts_status\": [\n"); + count = 0; + rrdhost_foreach_read(host) + { + if (rrdhost_flag_check(host, RRDHOST_FLAG_ARCHIVED)) + continue; + if (count > 0) + buffer_strcat(wb, ",\n"); + + netdata_mutex_lock(&host->receiver_lock); + buffer_sprintf( + wb, "\t\t{ \"guid\": \"%s\", \"reachable\": %s, \"claim_id\": ", host->machine_guid, + (host->receiver || host == localhost) ? "true" : "false"); + netdata_mutex_unlock(&host->receiver_lock); + + rrdhost_aclk_state_lock(host); + if (host->aclk_state.claimed_id) + buffer_sprintf(wb, "\"%s\" }", host->aclk_state.claimed_id); + else + buffer_strcat(wb, "null }"); + rrdhost_aclk_state_unlock(host); + + count++; + } + rrd_unlock(); + + buffer_strcat(wb, "\n\t],\n"); +} + +inline void host_labels2json(RRDHOST *host, BUFFER *wb, size_t indentation) { + char tabs[11]; + + if (indentation > 10) + indentation = 10; + + tabs[0] = '\0'; + while (indentation) { + strcat(tabs, "\t"); + indentation--; + } + + int count = 0; + rrdhost_rdlock(host); + netdata_rwlock_rdlock(&host->labels.labels_rwlock); + for (struct label *label = host->labels.head; label; label = label->next) { + if(count > 0) buffer_strcat(wb, ",\n"); + buffer_strcat(wb, tabs); + + char value[CONFIG_MAX_VALUE * 2 + 1]; + sanitize_json_string(value, label->value, CONFIG_MAX_VALUE * 2); + buffer_sprintf(wb, "\"%s\": \"%s\"", label->key, value); + + count++; + } + buffer_strcat(wb, "\n"); + netdata_rwlock_unlock(&host->labels.labels_rwlock); + rrdhost_unlock(host); +} + +extern int aclk_connected; +inline int web_client_api_request_v1_info_fill_buffer(RRDHOST *host, BUFFER *wb) +{ + buffer_strcat(wb, "{\n"); + buffer_sprintf(wb, "\t\"version\": \"%s\",\n", host->program_version); + buffer_sprintf(wb, "\t\"uid\": \"%s\",\n", host->machine_guid); + + web_client_api_request_v1_info_mirrored_hosts(wb); + + buffer_strcat(wb, "\t\"alarms\": {\n"); + web_client_api_request_v1_info_summary_alarm_statuses(host, wb); + buffer_strcat(wb, "\t},\n"); + + buffer_sprintf(wb, "\t\"os_name\": \"%s\",\n", (host->system_info->host_os_name) ? host->system_info->host_os_name : ""); + buffer_sprintf(wb, "\t\"os_id\": \"%s\",\n", (host->system_info->host_os_id) ? host->system_info->host_os_id : ""); + buffer_sprintf(wb, "\t\"os_id_like\": \"%s\",\n", (host->system_info->host_os_id_like) ? host->system_info->host_os_id_like : ""); + buffer_sprintf(wb, "\t\"os_version\": \"%s\",\n", (host->system_info->host_os_version) ? host->system_info->host_os_version : ""); + buffer_sprintf(wb, "\t\"os_version_id\": \"%s\",\n", (host->system_info->host_os_version_id) ? host->system_info->host_os_version_id : ""); + buffer_sprintf(wb, "\t\"os_detection\": \"%s\",\n", (host->system_info->host_os_detection) ? host->system_info->host_os_detection : ""); + buffer_sprintf(wb, "\t\"cores_total\": \"%s\",\n", (host->system_info->host_cores) ? host->system_info->host_cores : ""); + buffer_sprintf(wb, "\t\"total_disk_space\": \"%s\",\n", (host->system_info->host_disk_space) ? host->system_info->host_disk_space : ""); + buffer_sprintf(wb, "\t\"cpu_freq\": \"%s\",\n", (host->system_info->host_cpu_freq) ? host->system_info->host_cpu_freq : ""); + buffer_sprintf(wb, "\t\"ram_total\": \"%s\",\n", (host->system_info->host_ram_total) ? host->system_info->host_ram_total : ""); + + if (host->system_info->container_os_name) + buffer_sprintf(wb, "\t\"container_os_name\": \"%s\",\n", host->system_info->container_os_name); + if (host->system_info->container_os_id) + buffer_sprintf(wb, "\t\"container_os_id\": \"%s\",\n", host->system_info->container_os_id); + if (host->system_info->container_os_id_like) + buffer_sprintf(wb, "\t\"container_os_id_like\": \"%s\",\n", host->system_info->container_os_id_like); + if (host->system_info->container_os_version) + buffer_sprintf(wb, "\t\"container_os_version\": \"%s\",\n", host->system_info->container_os_version); + if (host->system_info->container_os_version_id) + buffer_sprintf(wb, "\t\"container_os_version_id\": \"%s\",\n", host->system_info->container_os_version_id); + if (host->system_info->container_os_detection) + buffer_sprintf(wb, "\t\"container_os_detection\": \"%s\",\n", host->system_info->container_os_detection); + if (host->system_info->is_k8s_node) + buffer_sprintf(wb, "\t\"is_k8s_node\": \"%s\",\n", host->system_info->is_k8s_node); + + buffer_sprintf(wb, "\t\"kernel_name\": \"%s\",\n", (host->system_info->kernel_name) ? host->system_info->kernel_name : ""); + buffer_sprintf(wb, "\t\"kernel_version\": \"%s\",\n", (host->system_info->kernel_version) ? host->system_info->kernel_version : ""); + buffer_sprintf(wb, "\t\"architecture\": \"%s\",\n", (host->system_info->architecture) ? host->system_info->architecture : ""); + buffer_sprintf(wb, "\t\"virtualization\": \"%s\",\n", (host->system_info->virtualization) ? host->system_info->virtualization : ""); + buffer_sprintf(wb, "\t\"virt_detection\": \"%s\",\n", (host->system_info->virt_detection) ? host->system_info->virt_detection : ""); + buffer_sprintf(wb, "\t\"container\": \"%s\",\n", (host->system_info->container) ? host->system_info->container : ""); + buffer_sprintf(wb, "\t\"container_detection\": \"%s\",\n", (host->system_info->container_detection) ? host->system_info->container_detection : ""); + + buffer_strcat(wb, "\t\"host_labels\": {\n"); + host_labels2json(host, wb, 2); + buffer_strcat(wb, "\t},\n"); + + buffer_strcat(wb, "\t\"collectors\": ["); + chartcollectors2json(host, wb); + buffer_strcat(wb, "\n\t],\n"); + +#ifdef DISABLE_CLOUD + buffer_strcat(wb, "\t\"cloud-enabled\": false,\n"); +#else + buffer_sprintf(wb, "\t\"cloud-enabled\": %s,\n", + appconfig_get_boolean(&cloud_config, CONFIG_SECTION_GLOBAL, "enabled", 1) ? "true" : "false"); +#endif + +#ifdef ENABLE_ACLK + buffer_strcat(wb, "\t\"cloud-available\": true,\n"); +#else + buffer_strcat(wb, "\t\"cloud-available\": false,\n"); +#endif + char *agent_id = is_agent_claimed(); + if (agent_id == NULL) + buffer_strcat(wb, "\t\"agent-claimed\": false,\n"); + else { + buffer_strcat(wb, "\t\"agent-claimed\": true,\n"); + freez(agent_id); + } +#ifdef ENABLE_ACLK + if (aclk_connected) + buffer_strcat(wb, "\t\"aclk-available\": true\n"); + else +#endif + buffer_strcat(wb, "\t\"aclk-available\": false\n"); // Intentionally valid with/without #ifdef above + + buffer_strcat(wb, "}"); + return 0; +} + +inline int web_client_api_request_v1_info(RRDHOST *host, struct web_client *w, char *url) { + (void)url; + if (!netdata_ready) return HTTP_RESP_BACKEND_FETCH_FAILED; + BUFFER *wb = w->response.data; + buffer_flush(wb); + wb->contenttype = CT_APPLICATION_JSON; + + web_client_api_request_v1_info_fill_buffer(host, wb); + + buffer_no_cacheable(wb); + return HTTP_RESP_OK; +} + +static struct api_command { + const char *command; + uint32_t hash; + WEB_CLIENT_ACL acl; + int (*callback)(RRDHOST *host, struct web_client *w, char *url); +} api_commands[] = { + { "info", 0, WEB_CLIENT_ACL_DASHBOARD, web_client_api_request_v1_info }, + { "data", 0, WEB_CLIENT_ACL_DASHBOARD, web_client_api_request_v1_data }, + { "chart", 0, WEB_CLIENT_ACL_DASHBOARD, web_client_api_request_v1_chart }, + { "charts", 0, WEB_CLIENT_ACL_DASHBOARD, web_client_api_request_v1_charts }, + { "archivedcharts", 0, WEB_CLIENT_ACL_DASHBOARD, web_client_api_request_v1_archivedcharts }, + + // registry checks the ACL by itself, so we allow everything + { "registry", 0, WEB_CLIENT_ACL_NOCHECK, web_client_api_request_v1_registry }, + + // badges can be fetched with both dashboard and badge permissions + { "badge.svg", 0, WEB_CLIENT_ACL_DASHBOARD|WEB_CLIENT_ACL_BADGE, web_client_api_request_v1_badge }, + + { "alarms", 0, WEB_CLIENT_ACL_DASHBOARD, web_client_api_request_v1_alarms }, + { "alarms_values", 0, WEB_CLIENT_ACL_DASHBOARD, web_client_api_request_v1_alarms_values }, + { "alarm_log", 0, WEB_CLIENT_ACL_DASHBOARD, web_client_api_request_v1_alarm_log }, + { "alarm_variables", 0, WEB_CLIENT_ACL_DASHBOARD, web_client_api_request_v1_alarm_variables }, + { "alarm_count", 0, WEB_CLIENT_ACL_DASHBOARD, web_client_api_request_v1_alarm_count }, + { "allmetrics", 0, WEB_CLIENT_ACL_DASHBOARD, web_client_api_request_v1_allmetrics }, + { "manage/health", 0, WEB_CLIENT_ACL_MGMT, web_client_api_request_v1_mgmt_health }, + // terminator + { NULL, 0, WEB_CLIENT_ACL_NONE, NULL }, +}; + +inline int web_client_api_request_v1(RRDHOST *host, struct web_client *w, char *url) { + static int initialized = 0; + int i; + + if(unlikely(initialized == 0)) { + initialized = 1; + + for(i = 0; api_commands[i].command ; i++) + api_commands[i].hash = simple_hash(api_commands[i].command); + } + + // get the command + if(url) { + debug(D_WEB_CLIENT, "%llu: Searching for API v1 command '%s'.", w->id, url); + uint32_t hash = simple_hash(url); + + for(i = 0; api_commands[i].command ;i++) { + if(unlikely(hash == api_commands[i].hash && !strcmp(url, api_commands[i].command))) { + if(unlikely(api_commands[i].acl != WEB_CLIENT_ACL_NOCHECK) && !(w->acl & api_commands[i].acl)) + return web_client_permission_denied(w); + + //return api_commands[i].callback(host, w, url); + return api_commands[i].callback(host, w, (w->decoded_query_string + 1)); + } + } + + buffer_flush(w->response.data); + buffer_strcat(w->response.data, "Unsupported v1 API command: "); + buffer_strcat_htmlescape(w->response.data, url); + return HTTP_RESP_NOT_FOUND; + } + else { + buffer_flush(w->response.data); + buffer_sprintf(w->response.data, "Which API v1 command?"); + return HTTP_RESP_BAD_REQUEST; + } +} diff --git a/web/api/web_api_v1.h b/web/api/web_api_v1.h new file mode 100644 index 0000000..445b0e4 --- /dev/null +++ b/web/api/web_api_v1.h @@ -0,0 +1,36 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#ifndef NETDATA_WEB_API_V1_H +#define NETDATA_WEB_API_V1_H 1 + +#include "daemon/common.h" +#include "web/api/badges/web_buffer_svg.h" +#include "web/api/formatters/rrd2json.h" +#include "web/api/health/health_cmdapi.h" + +extern uint32_t web_client_api_request_v1_data_options(char *o); +extern uint32_t web_client_api_request_v1_data_format(char *name); +extern uint32_t web_client_api_request_v1_data_google_format(char *name); + +extern int web_client_api_request_v1_alarms(RRDHOST *host, struct web_client *w, char *url); +extern int web_client_api_request_v1_alarms_values(RRDHOST *host, struct web_client *w, char *url); +extern int web_client_api_request_v1_alarm_log(RRDHOST *host, struct web_client *w, char *url); +extern int web_client_api_request_single_chart(RRDHOST *host, struct web_client *w, char *url, void callback(RRDSET *st, BUFFER *buf)); +extern int web_client_api_request_v1_alarm_variables(RRDHOST *host, struct web_client *w, char *url); +extern int web_client_api_request_v1_alarm_count(RRDHOST *host, struct web_client *w, char *url); +extern int web_client_api_request_v1_charts(RRDHOST *host, struct web_client *w, char *url); +extern int web_client_api_request_v1_archivedcharts(RRDHOST *host, struct web_client *w, char *url); +extern int web_client_api_request_v1_chart(RRDHOST *host, struct web_client *w, char *url); +extern int web_client_api_request_v1_data(RRDHOST *host, struct web_client *w, char *url); +extern int web_client_api_request_v1_registry(RRDHOST *host, struct web_client *w, char *url); +extern int web_client_api_request_v1_info(RRDHOST *host, struct web_client *w, char *url); +extern int web_client_api_request_v1(RRDHOST *host, struct web_client *w, char *url); +extern int web_client_api_request_v1_info_fill_buffer(RRDHOST *host, BUFFER *wb); +extern void host_labels2json(RRDHOST *host, BUFFER *wb, size_t indentation); + +extern void web_client_api_v1_init(void); +extern void web_client_api_v1_management_init(void); + +extern char *api_secret; + +#endif //NETDATA_WEB_API_V1_H |