diff options
| author | Daniel Baumann <daniel.baumann@progress-linux.org> | 2024-04-27 11:08:07 +0000 |
|---|---|---|
| committer | Daniel Baumann <daniel.baumann@progress-linux.org> | 2024-04-27 11:08:07 +0000 |
| commit | c69cb8cc094cc916adbc516b09e944cd3d137c01 (patch) | |
| tree | f2878ec41fb6d0e3613906c6722fc02b934eeb80 /web/api/queries | |
| parent | Initial commit. (diff) | |
| download | netdata-003423236c4cd249ed4246231d71a062f8f3d45a.tar.xz netdata-003423236c4cd249ed4246231d71a062f8f3d45a.zip | |
Adding upstream version 1.29.3.upstream/1.29.3upstream
Signed-off-by: Daniel Baumann <daniel.baumann@progress-linux.org>
Diffstat (limited to 'web/api/queries')
42 files changed, 3600 insertions, 0 deletions
diff --git a/web/api/queries/Makefile.am b/web/api/queries/Makefile.am new file mode 100644 index 0000000..34bfdb8 --- /dev/null +++ b/web/api/queries/Makefile.am @@ -0,0 +1,20 @@ +# SPDX-License-Identifier: GPL-3.0-or-later + +AUTOMAKE_OPTIONS = subdir-objects +MAINTAINERCLEANFILES = $(srcdir)/Makefile.in + +SUBDIRS = \ + average \ + des \ + incremental_sum \ + max \ + min \ + sum \ + median \ + ses \ + stddev \ + $(NULL) + +dist_noinst_DATA = \ + README.md \ + $(NULL) diff --git a/web/api/queries/README.md b/web/api/queries/README.md new file mode 100644 index 0000000..31ec496 --- /dev/null +++ b/web/api/queries/README.md @@ -0,0 +1,176 @@ +<!-- +title: "Database Queries" +custom_edit_url: https://github.com/netdata/netdata/edit/master/web/api/queries/README.md +--> + +# Database Queries + +Netdata database can be queried with `/api/v1/data` and `/api/v1/badge.svg` REST API methods. + +Every data query accepts the following parameters: + +|name|required|description| +|:--:|:------:|:----------| +|`chart`|yes|The chart to be queried.| +|`points`|no|The number of points to be returned. Netdata can reduce number of points by applying query grouping methods. If not given, the result will have the same granularity as the database (although this relates to `gtime`).| +|`before`|no|The absolute timestamp or the relative (to now) time the query should finish evaluating data. If not given, it defaults to the timestamp of the latest point in the database.| +|`after`|no|The absolute timestamp or the relative (to `before`) time the query should start evaluating data. if not given, it defaults to the timestamp of the oldest point in the database.| +|`group`|no|The grouping method to use when reducing the points the database has. If not given, it defaults to `average`.| +|`gtime`|no|A resampling period to change the units of the metrics (i.e. setting this to `60` will convert `per second` metrics to `per minute`. If not given it defaults to granularity of the database.| +|`options`|no|A bitmap of options that can affect the operation of the query. Only 2 options are used by the query engine: `unaligned` and `percentage`. All the other options are used by the output formatters. The default is to return aligned data.| +|`dimensions`|no|A simple pattern to filter the dimensions to be queried. The default is to return all the dimensions of the chart.| + +## Operation + +The query engine works as follows (in this order): + +#### Time-frame + +`after` and `before` define a time-frame, accepting: + +- **absolute timestamps** (unix timestamps, i.e. seconds since epoch). + +- **relative timestamps**: + + `before` is relative to now and `after` is relative to `before`. + + Example: `before=-60&after=-60` evaluates to the time-frame from -120 up to -60 seconds in + the past, relative to the latest entry of the database of the chart. + +The engine verifies that the time-frame requested is available at the database: + +- If the requested time-frame overlaps with the database, the excess requested + will be truncated. + +- If the requested time-frame does not overlap with the database, the engine will + return an empty data set. + +At the end of this operation, `after` and `before` are absolute timestamps. + +#### Data grouping + +Database points grouping is applied when the caller requests a time-frame to be +expressed with fewer points, compared to what is available at the database. + +There are 2 uses that enable this feature: + +- The caller requests a specific number of `points` to be returned. + + For example, for a time-frame of 10 minutes, the database has 600 points (1/sec), + while the caller requested these 10 minutes to be expressed in 200 points. + + This feature is used by Netdata dashboards when you zoom-out the charts. + The dashboard is requesting the number of points the user's screen has. + This saves bandwidth and speeds up the browser (fewer points to evaluate for drawing the charts). +- The caller requests a **re-sampling** of the database, by setting `gtime` to any value + above the granularity of the chart. + + For example, the chart's units is `requests/sec` and caller wants `requests/min`. + +Using `points` and `gtime` the query engine tries to find a best fit for **database-points** +vs **result-points** (we call this ratio `group points`). It always tries to keep `group points` +an integer. Keep in mind the query engine may shift `after` if required. See also the [example](#example). + +#### Time-frame Alignment + +Alignment is a very important aspect of Netdata queries. Without it, the animated +charts on the dashboards would constantly [change shape](#example) during incremental updates. + +To provide consistent grouping through time, the query engine (by default) aligns +`after` and `before` to be a multiple of `group points`. + +For example, if `group points` is 60 and alignment is enabled, the engine will return +each point with durations XX:XX:00 - XX:XX:59, matching whole minutes. + +To disable alignment, pass `&options=unaligned` to the query. + +#### Query Execution + +To execute the query, the engine evaluates all dimensions of the chart, one after another. + +The engine does not evaluate dimensions that do not match the [simple pattern](/libnetdata/simple_pattern/README.md) +given at the `dimensions` parameter, except when `options=percentage` is given (this option +requires all the dimensions to be evaluated to find the percentage of each dimension vs to chart +total). + +For each dimension, it starts evaluating values starting at `after` (not inclusive) towards +`before` (inclusive). + +For each value it calls the **grouping method** given with the `&group=` query parameter +(the default is `average`). + +## Grouping methods + +The following grouping methods are supported. These are given all the values in the time-frame +and they group the values every `group points`. + +-  finds the minimum value +-  finds the maximum value +-  finds the average value +-  adds all the values and returns the sum +-  sorts the values and returns the value in the middle of the list +-  finds the standard deviation of the values +-  finds the relative standard deviation (coefficient of variation) of the values +-  finds the exponential weighted moving average of the values +-  applies Holt-Winters double exponential smoothing +-  finds the difference of the last vs the first value + +The examples shown above, are live information from the `successful` web requests of the global Netdata registry. + +## Further processing + +The result of the query engine is always a structure that has dimensions and values +for each dimension. + +Formatting modules are then used to convert this result in many different formats and return it +to the caller. + +## Performance + +The query engine is highly optimized for speed. Most of its modules implement "online" +versions of the algorithms, requiring just one pass on the database values to produce +the result. + +## Example + +When Netdata is reducing metrics, it tries to return always the same boundaries. So, if we want 10s averages, it will always return points starting at a `unix timestamp % 10 = 0`. + +Let's see why this is needed, by looking at the error case. + +Assume we have 5 points: + +|time|value| +|:--:|:---:| +|00:01|1| +|00:02|2| +|00:03|3| +|00:04|4| +|00:05|5| + +At 00:04 you ask for 2 points for 4 seconds in the past. So `group = 2`. Netdata would return: + +|point|time|value| +|:---:|:--:|:---:| +|1|00:01 - 00:02|1.5| +|2|00:03 - 00:04|3.5| + +A second later the chart is to be refreshed, and makes again the same request at 00:05. These are the points that would have been returned: + +|point|time|value| +|:---:|:--:|:---:| +|1|00:02 - 00:03|2.5| +|2|00:04 - 00:05|4.5| + +**Wait a moment!** The chart was shifted just one point and it changed value! Point 2 was 3.5 and when shifted to point 1 is 2.5! If you see this in a chart, it's a mess. The charts change shape constantly. + +For this reason, Netdata always aligns the data it returns to the `group`. + +When you request `points=1`, Netdata understands that you need 1 point for the whole database, so `group = 3600`. Then it tries to find the starting point which would be `timestamp % 3600 = 0` Within a database of 3600 seconds, there is one such point for sure. Then it tries to find the average of 3600 points. But, most probably it will not find 3600 of them (for just 1 out of 3600 seconds this query will return something). + +So, the proper way to query the database is to also set at least `after`. The following call will returns 1 point for the last complete 10-second duration (it starts at `timestamp % 10 = 0`): + +<http://netdata.firehol.org/api/v1/data?chart=system.cpu&points=1&after=-10&options=seconds> + +When you keep calling this URL, you will see that it returns one new value every 10 seconds, and the timestamp always ends with zero. Similarly, if you say `points=1&after=-5` it will always return timestamps ending with 0 or 5. + +[](<>) diff --git a/web/api/queries/average/Makefile.am b/web/api/queries/average/Makefile.am new file mode 100644 index 0000000..161784b --- /dev/null +++ b/web/api/queries/average/Makefile.am @@ -0,0 +1,8 @@ +# SPDX-License-Identifier: GPL-3.0-or-later + +AUTOMAKE_OPTIONS = subdir-objects +MAINTAINERCLEANFILES = $(srcdir)/Makefile.in + +dist_noinst_DATA = \ + README.md \ + $(NULL) diff --git a/web/api/queries/average/README.md b/web/api/queries/average/README.md new file mode 100644 index 0000000..f32a675 --- /dev/null +++ b/web/api/queries/average/README.md @@ -0,0 +1,46 @@ +<!-- +title: "Average or Mean" +custom_edit_url: https://github.com/netdata/netdata/edit/master/web/api/queries/average/README.md +--> + +# Average or Mean + +> This query is available as `average` and `mean`. + +An average is a single number taken as representative of a list of numbers. + +It is calculated as: + +``` +average = sum(numbers) / count(numbers) +``` + +## how to use + +Use it in alarms like this: + +``` + alarm: my_alarm + on: my_chart +lookup: average -1m unaligned of my_dimension + warn: $this > 1000 +``` + +`average` does not change the units. For example, if the chart units is `requests/sec`, the result +will be again expressed in the same units. + +It can also be used in APIs and badges as `&group=average` in the URL. + +## Examples + +Examining last 1 minute `successful` web server responses: + +-  +-  +-  + +## References + +- <https://en.wikipedia.org/wiki/Average>. + +[](<>) diff --git a/web/api/queries/average/average.c b/web/api/queries/average/average.c new file mode 100644 index 0000000..2c64358 --- /dev/null +++ b/web/api/queries/average/average.c @@ -0,0 +1,62 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#include "average.h" + +// ---------------------------------------------------------------------------- +// average + +struct grouping_average { + calculated_number sum; + size_t count; +}; + +void *grouping_create_average(RRDR *r) { + (void)r; + return callocz(1, sizeof(struct grouping_average)); +} + +// resets when switches dimensions +// so, clear everything to restart +void grouping_reset_average(RRDR *r) { + struct grouping_average *g = (struct grouping_average *)r->internal.grouping_data; + g->sum = 0; + g->count = 0; +} + +void grouping_free_average(RRDR *r) { + freez(r->internal.grouping_data); + r->internal.grouping_data = NULL; +} + +void grouping_add_average(RRDR *r, calculated_number value) { + if(!isnan(value)) { + struct grouping_average *g = (struct grouping_average *)r->internal.grouping_data; + g->sum += value; + g->count++; + } +} + +calculated_number grouping_flush_average(RRDR *r, RRDR_VALUE_FLAGS *rrdr_value_options_ptr) { + struct grouping_average *g = (struct grouping_average *)r->internal.grouping_data; + + calculated_number value; + + if(unlikely(!g->count)) { + value = 0.0; + *rrdr_value_options_ptr |= RRDR_VALUE_EMPTY; + } + else { + if(unlikely(r->internal.resampling_group != 1)) { + if (unlikely(r->result_options & RRDR_RESULT_OPTION_VARIABLE_STEP)) + value = g->sum / g->count / r->internal.resampling_divisor; + else + value = g->sum / r->internal.resampling_divisor; + } else + value = g->sum / g->count; + } + + g->sum = 0.0; + g->count = 0; + + return value; +} diff --git a/web/api/queries/average/average.h b/web/api/queries/average/average.h new file mode 100644 index 0000000..9fb7de2 --- /dev/null +++ b/web/api/queries/average/average.h @@ -0,0 +1,15 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#ifndef NETDATA_API_QUERY_AVERAGE_H +#define NETDATA_API_QUERY_AVERAGE_H + +#include "../query.h" +#include "../rrdr.h" + +extern void *grouping_create_average(RRDR *r); +extern void grouping_reset_average(RRDR *r); +extern void grouping_free_average(RRDR *r); +extern void grouping_add_average(RRDR *r, calculated_number value); +extern calculated_number grouping_flush_average(RRDR *r, RRDR_VALUE_FLAGS *rrdr_value_options_ptr); + +#endif //NETDATA_API_QUERY_AVERAGE_H diff --git a/web/api/queries/des/Makefile.am b/web/api/queries/des/Makefile.am new file mode 100644 index 0000000..161784b --- /dev/null +++ b/web/api/queries/des/Makefile.am @@ -0,0 +1,8 @@ +# SPDX-License-Identifier: GPL-3.0-or-later + +AUTOMAKE_OPTIONS = subdir-objects +MAINTAINERCLEANFILES = $(srcdir)/Makefile.in + +dist_noinst_DATA = \ + README.md \ + $(NULL) diff --git a/web/api/queries/des/README.md b/web/api/queries/des/README.md new file mode 100644 index 0000000..5505de5 --- /dev/null +++ b/web/api/queries/des/README.md @@ -0,0 +1,73 @@ +<!-- +title: "double exponential smoothing" +custom_edit_url: https://github.com/netdata/netdata/edit/master/web/api/queries/des/README.md +--> + +# double exponential smoothing + +Exponential smoothing is one of many window functions commonly applied to smooth data in signal +processing, acting as low-pass filters to remove high frequency noise. + +Simple exponential smoothing does not do well when there is a trend in the data. +In such situations, several methods were devised under the name "double exponential smoothing" +or "second-order exponential smoothing.", which is the recursive application of an exponential +filter twice, thus being termed "double exponential smoothing". + +In simple terms, this is like an average value, but more recent values are given more weight +and the trend of the values influences significantly the result. + +> **IMPORTANT** +> +> It is common for `des` to provide "average" values that far beyond the minimum or the maximum +> values found in the time-series. +> `des` estimates these values because of it takes into account the trend. + +This module implements the "Holt-Winters double exponential smoothing". + +Netdata automatically adjusts the weight (`alpha`) and the trend (`beta`) based on the number +of values processed, using the formula: + +``` +window = max(number of values, 15) +alpha = 2 / (window + 1) +beta = 2 / (window + 1) +``` + +You can change the fixed value `15` by setting in `netdata.conf`: + +``` +[web] + des max window = 15 +``` + +## how to use + +Use it in alarms like this: + +``` + alarm: my_alarm + on: my_chart +lookup: des -1m unaligned of my_dimension + warn: $this > 1000 +``` + +`des` does not change the units. For example, if the chart units is `requests/sec`, the result +will be again expressed in the same units. + +It can also be used in APIs and badges as `&group=des` in the URL. + +## Examples + +Examining last 1 minute `successful` web server responses: + +-  +-  +-  +-  +-  + +## References + +- <https://en.wikipedia.org/wiki/Exponential_smoothing>. + +[](<>) diff --git a/web/api/queries/des/des.c b/web/api/queries/des/des.c new file mode 100644 index 0000000..c6236f3 --- /dev/null +++ b/web/api/queries/des/des.c @@ -0,0 +1,139 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#include <web/api/queries/rrdr.h> +#include "des.h" + + +// ---------------------------------------------------------------------------- +// single exponential smoothing + +struct grouping_des { + calculated_number alpha; + calculated_number alpha_other; + calculated_number beta; + calculated_number beta_other; + + calculated_number level; + calculated_number trend; + + size_t count; +}; + +static size_t max_window_size = 15; + +void grouping_init_des(void) { + long long ret = config_get_number(CONFIG_SECTION_WEB, "des max window", (long long)max_window_size); + if(ret <= 1) { + config_set_number(CONFIG_SECTION_WEB, "des max window", (long long)max_window_size); + } + else { + max_window_size = (size_t) ret; + } +} + +static inline calculated_number window(RRDR *r, struct grouping_des *g) { + (void)g; + + calculated_number points; + if(r->group == 1) { + // provide a running DES + points = r->internal.points_wanted; + } + else { + // provide a SES with flush points + points = r->group; + } + + // https://en.wikipedia.org/wiki/Moving_average#Exponential_moving_average + // A commonly used value for alpha is 2 / (N + 1) + return (points > max_window_size) ? max_window_size : points; +} + +static inline void set_alpha(RRDR *r, struct grouping_des *g) { + // https://en.wikipedia.org/wiki/Moving_average#Exponential_moving_average + // A commonly used value for alpha is 2 / (N + 1) + + g->alpha = 2.0 / (window(r, g) + 1.0); + g->alpha_other = 1.0 - g->alpha; + + //info("alpha for chart '%s' is " CALCULATED_NUMBER_FORMAT, r->st->name, g->alpha); +} + +static inline void set_beta(RRDR *r, struct grouping_des *g) { + // https://en.wikipedia.org/wiki/Moving_average#Exponential_moving_average + // A commonly used value for alpha is 2 / (N + 1) + + g->beta = 2.0 / (window(r, g) + 1.0); + g->beta_other = 1.0 - g->beta; + + //info("beta for chart '%s' is " CALCULATED_NUMBER_FORMAT, r->st->name, g->beta); +} + +void *grouping_create_des(RRDR *r) { + struct grouping_des *g = (struct grouping_des *)malloc(sizeof(struct grouping_des)); + set_alpha(r, g); + set_beta(r, g); + g->level = 0.0; + g->trend = 0.0; + g->count = 0; + return g; +} + +// resets when switches dimensions +// so, clear everything to restart +void grouping_reset_des(RRDR *r) { + struct grouping_des *g = (struct grouping_des *)r->internal.grouping_data; + g->level = 0.0; + g->trend = 0.0; + g->count = 0; + + // fprintf(stderr, "\nDES: "); + +} + +void grouping_free_des(RRDR *r) { + freez(r->internal.grouping_data); + r->internal.grouping_data = NULL; +} + +void grouping_add_des(RRDR *r, calculated_number value) { + struct grouping_des *g = (struct grouping_des *)r->internal.grouping_data; + + if(calculated_number_isnumber(value)) { + if(likely(g->count > 0)) { + // we have at least a number so far + + if(unlikely(g->count == 1)) { + // the second value we got + g->trend = value - g->trend; + g->level = value; + } + + // for the values, except the first + calculated_number last_level = g->level; + g->level = (g->alpha * value) + (g->alpha_other * (g->level + g->trend)); + g->trend = (g->beta * (g->level - last_level)) + (g->beta_other * g->trend); + } + else { + // the first value we got + g->level = g->trend = value; + } + + g->count++; + } + + //fprintf(stderr, "value: " CALCULATED_NUMBER_FORMAT ", level: " CALCULATED_NUMBER_FORMAT ", trend: " CALCULATED_NUMBER_FORMAT "\n", value, g->level, g->trend); +} + +calculated_number grouping_flush_des(RRDR *r, RRDR_VALUE_FLAGS *rrdr_value_options_ptr) { + struct grouping_des *g = (struct grouping_des *)r->internal.grouping_data; + + if(unlikely(!g->count || !calculated_number_isnumber(g->level))) { + *rrdr_value_options_ptr |= RRDR_VALUE_EMPTY; + return 0.0; + } + + //fprintf(stderr, " RESULT for %zu values = " CALCULATED_NUMBER_FORMAT " \n", g->count, g->level); + + return g->level; +} diff --git a/web/api/queries/des/des.h b/web/api/queries/des/des.h new file mode 100644 index 0000000..360513e --- /dev/null +++ b/web/api/queries/des/des.h @@ -0,0 +1,17 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#ifndef NETDATA_API_QUERIES_DES_H +#define NETDATA_API_QUERIES_DES_H + +#include "../query.h" +#include "../rrdr.h" + +extern void grouping_init_des(void); + +extern void *grouping_create_des(RRDR *r); +extern void grouping_reset_des(RRDR *r); +extern void grouping_free_des(RRDR *r); +extern void grouping_add_des(RRDR *r, calculated_number value); +extern calculated_number grouping_flush_des(RRDR *r, RRDR_VALUE_FLAGS *rrdr_value_options_ptr); + +#endif //NETDATA_API_QUERIES_DES_H diff --git a/web/api/queries/incremental_sum/Makefile.am b/web/api/queries/incremental_sum/Makefile.am new file mode 100644 index 0000000..161784b --- /dev/null +++ b/web/api/queries/incremental_sum/Makefile.am @@ -0,0 +1,8 @@ +# SPDX-License-Identifier: GPL-3.0-or-later + +AUTOMAKE_OPTIONS = subdir-objects +MAINTAINERCLEANFILES = $(srcdir)/Makefile.in + +dist_noinst_DATA = \ + README.md \ + $(NULL) diff --git a/web/api/queries/incremental_sum/README.md b/web/api/queries/incremental_sum/README.md new file mode 100644 index 0000000..e5f3dfc --- /dev/null +++ b/web/api/queries/incremental_sum/README.md @@ -0,0 +1,41 @@ +<!-- +title: "Incremental Sum (`incremental_sum`)" +custom_edit_url: https://github.com/netdata/netdata/edit/master/web/api/queries/incremental_sum/README.md +--> + +# Incremental Sum (`incremental_sum`) + +This modules finds the incremental sum of a period, which `last value - first value`. + +The result may be positive (rising) or negative (falling) depending on the first and last values. + +## how to use + +Use it in alarms like this: + +``` + alarm: my_alarm + on: my_chart +lookup: incremental_sum -1m unaligned of my_dimension + warn: $this > 1000 +``` + +`incremental_sum` does not change the units. For example, if the chart units is `requests/sec`, the result +will be again expressed in the same units. + +It can also be used in APIs and badges as `&group=incremental_sum` in the URL. + +## Examples + +Examining last 1 minute `successful` web server responses: + +-  +-  +-  +-  + +## References + +- none + +[](<>) diff --git a/web/api/queries/incremental_sum/incremental_sum.c b/web/api/queries/incremental_sum/incremental_sum.c new file mode 100644 index 0000000..131d85d --- /dev/null +++ b/web/api/queries/incremental_sum/incremental_sum.c @@ -0,0 +1,69 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#include "incremental_sum.h" + +// ---------------------------------------------------------------------------- +// incremental sum + +struct grouping_incremental_sum { + calculated_number first; + calculated_number last; + size_t count; +}; + +void *grouping_create_incremental_sum(RRDR *r) { + (void)r; + return callocz(1, sizeof(struct grouping_incremental_sum)); +} + +// resets when switches dimensions +// so, clear everything to restart +void grouping_reset_incremental_sum(RRDR *r) { + struct grouping_incremental_sum *g = (struct grouping_incremental_sum *)r->internal.grouping_data; + g->first = 0; + g->last = 0; + g->count = 0; +} + +void grouping_free_incremental_sum(RRDR *r) { + freez(r->internal.grouping_data); + r->internal.grouping_data = NULL; +} + +void grouping_add_incremental_sum(RRDR *r, calculated_number value) { + if(!isnan(value)) { + struct grouping_incremental_sum *g = (struct grouping_incremental_sum *)r->internal.grouping_data; + + if(unlikely(!g->count)) { + g->first = value; + g->count++; + } + else { + g->last = value; + g->count++; + } + } +} + +calculated_number grouping_flush_incremental_sum(RRDR *r, RRDR_VALUE_FLAGS *rrdr_value_options_ptr) { + struct grouping_incremental_sum *g = (struct grouping_incremental_sum *)r->internal.grouping_data; + + calculated_number value; + + if(unlikely(!g->count)) { + value = 0.0; + *rrdr_value_options_ptr |= RRDR_VALUE_EMPTY; + } + else if(unlikely(g->count == 1)) { + value = 0.0; + } + else { + value = g->last - g->first; + } + + g->first = 0.0; + g->last = 0.0; + g->count = 0; + + return value; +} diff --git a/web/api/queries/incremental_sum/incremental_sum.h b/web/api/queries/incremental_sum/incremental_sum.h new file mode 100644 index 0000000..990a2ac --- /dev/null +++ b/web/api/queries/incremental_sum/incremental_sum.h @@ -0,0 +1,15 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#ifndef NETDATA_API_QUERY_INCREMENTAL_SUM_H +#define NETDATA_API_QUERY_INCREMENTAL_SUM_H + +#include "../query.h" +#include "../rrdr.h" + +extern void *grouping_create_incremental_sum(RRDR *r); +extern void grouping_reset_incremental_sum(RRDR *r); +extern void grouping_free_incremental_sum(RRDR *r); +extern void grouping_add_incremental_sum(RRDR *r, calculated_number value); +extern calculated_number grouping_flush_incremental_sum(RRDR *r, RRDR_VALUE_FLAGS *rrdr_value_options_ptr); + +#endif //NETDATA_API_QUERY_INCREMENTAL_SUM_H diff --git a/web/api/queries/max/Makefile.am b/web/api/queries/max/Makefile.am new file mode 100644 index 0000000..161784b --- /dev/null +++ b/web/api/queries/max/Makefile.am @@ -0,0 +1,8 @@ +# SPDX-License-Identifier: GPL-3.0-or-later + +AUTOMAKE_OPTIONS = subdir-objects +MAINTAINERCLEANFILES = $(srcdir)/Makefile.in + +dist_noinst_DATA = \ + README.md \ + $(NULL) diff --git a/web/api/queries/max/README.md b/web/api/queries/max/README.md new file mode 100644 index 0000000..32b1d43 --- /dev/null +++ b/web/api/queries/max/README.md @@ -0,0 +1,38 @@ +<!-- +title: "Max" +custom_edit_url: https://github.com/netdata/netdata/edit/master/web/api/queries/max/README.md +--> + +# Max + +This module finds the max value in the time-frame given. + +## how to use + +Use it in alarms like this: + +``` + alarm: my_alarm + on: my_chart +lookup: max -1m unaligned of my_dimension + warn: $this > 1000 +``` + +`max` does not change the units. For example, if the chart units is `requests/sec`, the result +will be again expressed in the same units. + +It can also be used in APIs and badges as `&group=max` in the URL. + +## Examples + +Examining last 1 minute `successful` web server responses: + +-  +-  +-  + +## References + +- <https://en.wikipedia.org/wiki/Sample_maximum_and_minimum>. + +[](<>) diff --git a/web/api/queries/max/max.c b/web/api/queries/max/max.c new file mode 100644 index 0000000..a4be36a --- /dev/null +++ b/web/api/queries/max/max.c @@ -0,0 +1,60 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#include "max.h" + +// ---------------------------------------------------------------------------- +// max + +struct grouping_max { + calculated_number max; + size_t count; +}; + +void *grouping_create_max(RRDR *r) { + (void)r; + return callocz(1, sizeof(struct grouping_max)); +} + +// resets when switches dimensions +// so, clear everything to restart +void grouping_reset_max(RRDR *r) { + struct grouping_max *g = (struct grouping_max *)r->internal.grouping_data; + g->max = 0; + g->count = 0; +} + +void grouping_free_max(RRDR *r) { + freez(r->internal.grouping_data); + r->internal.grouping_data = NULL; +} + +void grouping_add_max(RRDR *r, calculated_number value) { + if(!isnan(value)) { + struct grouping_max *g = (struct grouping_max *)r->internal.grouping_data; + + if(!g->count || calculated_number_fabs(value) > calculated_number_fabs(g->max)) { + g->max = value; + g->count++; + } + } +} + +calculated_number grouping_flush_max(RRDR *r, RRDR_VALUE_FLAGS *rrdr_value_options_ptr) { + struct grouping_max *g = (struct grouping_max *)r->internal.grouping_data; + + calculated_number value; + + if(unlikely(!g->count)) { + value = 0.0; + *rrdr_value_options_ptr |= RRDR_VALUE_EMPTY; + } + else { + value = g->max; + } + + g->max = 0.0; + g->count = 0; + + return value; +} + diff --git a/web/api/queries/max/max.h b/web/api/queries/max/max.h new file mode 100644 index 0000000..d839fe3 --- /dev/null +++ b/web/api/queries/max/max.h @@ -0,0 +1,15 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#ifndef NETDATA_API_QUERY_MAX_H +#define NETDATA_API_QUERY_MAX_H + +#include "../query.h" +#include "../rrdr.h" + +extern void *grouping_create_max(RRDR *r); +extern void grouping_reset_max(RRDR *r); +extern void grouping_free_max(RRDR *r); +extern void grouping_add_max(RRDR *r, calculated_number value); +extern calculated_number grouping_flush_max(RRDR *r, RRDR_VALUE_FLAGS *rrdr_value_options_ptr); + +#endif //NETDATA_API_QUERY_MAX_H diff --git a/web/api/queries/median/Makefile.am b/web/api/queries/median/Makefile.am new file mode 100644 index 0000000..161784b --- /dev/null +++ b/web/api/queries/median/Makefile.am @@ -0,0 +1,8 @@ +# SPDX-License-Identifier: GPL-3.0-or-later + +AUTOMAKE_OPTIONS = subdir-objects +MAINTAINERCLEANFILES = $(srcdir)/Makefile.in + +dist_noinst_DATA = \ + README.md \ + $(NULL) diff --git a/web/api/queries/median/README.md b/web/api/queries/median/README.md new file mode 100644 index 0000000..25ce8b8 --- /dev/null +++ b/web/api/queries/median/README.md @@ -0,0 +1,45 @@ +<!-- +title: "Median" +description: "Use median in API queries and health entities to find the 'middle' value from a sample, eliminating any unwanted spikes in the returned metrics." +custom_edit_url: https://github.com/netdata/netdata/edit/master/web/api/queries/median/README.md +--> + +# Median + +The median is the value separating the higher half from the lower half of a data sample +(a population or a probability distribution). For a data set, it may be thought of as the +"middle" value. + +`median` is not an accurate average. However, it eliminates all spikes, by sorting +all the values in a period, and selecting the value in the middle of the sorted array. + +## how to use + +Use it in alarms like this: + +``` + alarm: my_alarm + on: my_chart +lookup: median -1m unaligned of my_dimension + warn: $this > 1000 +``` + +`median` does not change the units. For example, if the chart units is `requests/sec`, the result +will be again expressed in the same units. + +It can also be used in APIs and badges as `&group=median` in the URL. + +## Examples + +Examining last 1 minute `successful` web server responses: + +-  +-  +-  +-  + +## References + +- <https://en.wikipedia.org/wiki/Median>. + +[](<>) diff --git a/web/api/queries/median/median.c b/web/api/queries/median/median.c new file mode 100644 index 0000000..31916c5 --- /dev/null +++ b/web/api/queries/median/median.c @@ -0,0 +1,79 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#include "median.h" + + +// ---------------------------------------------------------------------------- +// median + +struct grouping_median { + size_t series_size; + size_t next_pos; + + LONG_DOUBLE series[]; +}; + +void *grouping_create_median(RRDR *r) { + long entries = r->group; + if(entries < 0) entries = 0; + + struct grouping_median *g = (struct grouping_median *)callocz(1, sizeof(struct grouping_median) + entries * sizeof(LONG_DOUBLE)); + g->series_size = (size_t)entries; + + return g; +} + +// resets when switches dimensions +// so, clear everything to restart +void grouping_reset_median(RRDR *r) { + struct grouping_median *g = (struct grouping_median *)r->internal.grouping_data; + g->next_pos = 0; +} + +void grouping_free_median(RRDR *r) { + freez(r->internal.grouping_data); + r->internal.grouping_data = NULL; +} + +void grouping_add_median(RRDR *r, calculated_number value) { + struct grouping_median *g = (struct grouping_median *)r->internal.grouping_data; + + if(unlikely(g->next_pos >= g->series_size)) { + error("INTERNAL ERROR: median buffer overflow on chart '%s' - next_pos = %zu, series_size = %zu, r->group = %ld.", r->st->name, g->next_pos, g->series_size, r->group); + } + else { + if(calculated_number_isnumber(value)) + g->series[g->next_pos++] = (LONG_DOUBLE)value; + } +} + +calculated_number grouping_flush_median(RRDR *r, RRDR_VALUE_FLAGS *rrdr_value_options_ptr) { + struct grouping_median *g = (struct grouping_median *)r->internal.grouping_data; + + calculated_number value; + + if(unlikely(!g->next_pos)) { + value = 0.0; + *rrdr_value_options_ptr |= RRDR_VALUE_EMPTY; + } + else { + if(g->next_pos > 1) { + sort_series(g->series, g->next_pos); + value = (calculated_number)median_on_sorted_series(g->series, g->next_pos); + } + else + value = (calculated_number)g->series[0]; + + if(!calculated_number_isnumber(value)) { + value = 0.0; + *rrdr_value_options_ptr |= RRDR_VALUE_EMPTY; + } + + //log_series_to_stderr(g->series, g->next_pos, value, "median"); + } + + g->next_pos = 0; + + return value; +} + diff --git a/web/api/queries/median/median.h b/web/api/queries/median/median.h new file mode 100644 index 0000000..dd2c1ff --- /dev/null +++ b/web/api/queries/median/median.h @@ -0,0 +1,15 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#ifndef NETDATA_API_QUERIES_MEDIAN_H +#define NETDATA_API_QUERIES_MEDIAN_H + +#include "../query.h" +#include "../rrdr.h" + +extern void *grouping_create_median(RRDR *r); +extern void grouping_reset_median(RRDR *r); +extern void grouping_free_median(RRDR *r); +extern void grouping_add_median(RRDR *r, calculated_number value); +extern calculated_number grouping_flush_median(RRDR *r, RRDR_VALUE_FLAGS *rrdr_value_options_ptr); + +#endif //NETDATA_API_QUERIES_MEDIAN_H diff --git a/web/api/queries/min/Makefile.am b/web/api/queries/min/Makefile.am new file mode 100644 index 0000000..161784b --- /dev/null +++ b/web/api/queries/min/Makefile.am @@ -0,0 +1,8 @@ +# SPDX-License-Identifier: GPL-3.0-or-later + +AUTOMAKE_OPTIONS = subdir-objects +MAINTAINERCLEANFILES = $(srcdir)/Makefile.in + +dist_noinst_DATA = \ + README.md \ + $(NULL) diff --git a/web/api/queries/min/README.md b/web/api/queries/min/README.md new file mode 100644 index 0000000..69ef4ea --- /dev/null +++ b/web/api/queries/min/README.md @@ -0,0 +1,38 @@ +<!-- +title: "Min" +custom_edit_url: https://github.com/netdata/netdata/edit/master/web/api/queries/min/README.md +--> + +# Min + +This module finds the min value in the time-frame given. + +## how to use + +Use it in alarms like this: + +``` + alarm: my_alarm + on: my_chart +lookup: min -1m unaligned of my_dimension + warn: $this > 1000 +``` + +`min` does not change the units. For example, if the chart units is `requests/sec`, the result +will be again expressed in the same units. + +It can also be used in APIs and badges as `&group=min` in the URL. + +## Examples + +Examining last 1 minute `successful` web server responses: + +-  +-  +-  + +## References + +- <https://en.wikipedia.org/wiki/Sample_maximum_and_minimum>. + +[](<>) diff --git a/web/api/queries/min/min.c b/web/api/queries/min/min.c new file mode 100644 index 0000000..9bd7460 --- /dev/null +++ b/web/api/queries/min/min.c @@ -0,0 +1,60 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#include "min.h" + +// ---------------------------------------------------------------------------- +// min + +struct grouping_min { + calculated_number min; + size_t count; +}; + +void *grouping_create_min(RRDR *r) { + (void)r; + return callocz(1, sizeof(struct grouping_min)); +} + +// resets when switches dimensions +// so, clear everything to restart +void grouping_reset_min(RRDR *r) { + struct grouping_min *g = (struct grouping_min *)r->internal.grouping_data; + g->min = 0; + g->count = 0; +} + +void grouping_free_min(RRDR *r) { + freez(r->internal.grouping_data); + r->internal.grouping_data = NULL; +} + +void grouping_add_min(RRDR *r, calculated_number value) { + if(!isnan(value)) { + struct grouping_min *g = (struct grouping_min *)r->internal.grouping_data; + + if(!g->count || calculated_number_fabs(value) < calculated_number_fabs(g->min)) { + g->min = value; + g->count++; + } + } +} + +calculated_number grouping_flush_min(RRDR *r, RRDR_VALUE_FLAGS *rrdr_value_options_ptr) { + struct grouping_min *g = (struct grouping_min *)r->internal.grouping_data; + + calculated_number value; + + if(unlikely(!g->count)) { + value = 0.0; + *rrdr_value_options_ptr |= RRDR_VALUE_EMPTY; + } + else { + value = g->min; + } + + g->min = 0.0; + g->count = 0; + + return value; +} + diff --git a/web/api/queries/min/min.h b/web/api/queries/min/min.h new file mode 100644 index 0000000..7470360 --- /dev/null +++ b/web/api/queries/min/min.h @@ -0,0 +1,15 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#ifndef NETDATA_API_QUERY_MIN_H +#define NETDATA_API_QUERY_MIN_H + +#include "../query.h" +#include "../rrdr.h" + +extern void *grouping_create_min(RRDR *r); +extern void grouping_reset_min(RRDR *r); +extern void grouping_free_min(RRDR *r); +extern void grouping_add_min(RRDR *r, calculated_number value); +extern calculated_number grouping_flush_min(RRDR *r, RRDR_VALUE_FLAGS *rrdr_value_options_ptr); + +#endif //NETDATA_API_QUERY_MIN_H diff --git a/web/api/queries/query.c b/web/api/queries/query.c new file mode 100644 index 0000000..663e4bd --- /dev/null +++ b/web/api/queries/query.c @@ -0,0 +1,1636 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#include "query.h" +#include "web/api/formatters/rrd2json.h" +#include "rrdr.h" + +#include "average/average.h" +#include "incremental_sum/incremental_sum.h" +#include "max/max.h" +#include "median/median.h" +#include "min/min.h" +#include "sum/sum.h" +#include "stddev/stddev.h" +#include "ses/ses.h" +#include "des/des.h" + +// ---------------------------------------------------------------------------- + +static struct { + const char *name; + uint32_t hash; + RRDR_GROUPING value; + + // One time initialization for the module. + // This is called once, when netdata starts. + void (*init)(void); + + // Allocate all required structures for a query. + // This is called once for each netdata query. + void *(*create)(struct rrdresult *r); + + // Cleanup collected values, but don't destroy the structures. + // This is called when the query engine switches dimensions, + // as part of the same query (so same chart, switching metric). + void (*reset)(struct rrdresult *r); + + // Free all resources allocated for the query. + void (*free)(struct rrdresult *r); + + // Add a single value into the calculation. + // The module may decide to cache it, or use it in the fly. + void (*add)(struct rrdresult *r, calculated_number value); + + // Generate a single result for the values added so far. + // More values and points may be requested later. + // It is up to the module to reset its internal structures + // when flushing it (so for a few modules it may be better to + // continue after a flush as if nothing changed, for others a + // cleanup of the internal structures may be required). + calculated_number (*flush)(struct rrdresult *r, RRDR_VALUE_FLAGS *rrdr_value_options_ptr); +} api_v1_data_groups[] = { + {.name = "average", + .hash = 0, + .value = RRDR_GROUPING_AVERAGE, + .init = NULL, + .create= grouping_create_average, + .reset = grouping_reset_average, + .free = grouping_free_average, + .add = grouping_add_average, + .flush = grouping_flush_average + }, + {.name = "mean", // alias on 'average' + .hash = 0, + .value = RRDR_GROUPING_AVERAGE, + .init = NULL, + .create= grouping_create_average, + .reset = grouping_reset_average, + .free = grouping_free_average, + .add = grouping_add_average, + .flush = grouping_flush_average + }, + {.name = "incremental_sum", + .hash = 0, + .value = RRDR_GROUPING_INCREMENTAL_SUM, + .init = NULL, + .create= grouping_create_incremental_sum, + .reset = grouping_reset_incremental_sum, + .free = grouping_free_incremental_sum, + .add = grouping_add_incremental_sum, + .flush = grouping_flush_incremental_sum + }, + {.name = "incremental-sum", + .hash = 0, + .value = RRDR_GROUPING_INCREMENTAL_SUM, + .init = NULL, + .create= grouping_create_incremental_sum, + .reset = grouping_reset_incremental_sum, + .free = grouping_free_incremental_sum, + .add = grouping_add_incremental_sum, + .flush = grouping_flush_incremental_sum + }, + {.name = "median", + .hash = 0, + .value = RRDR_GROUPING_MEDIAN, + .init = NULL, + .create= grouping_create_median, + .reset = grouping_reset_median, + .free = grouping_free_median, + .add = grouping_add_median, + .flush = grouping_flush_median + }, + {.name = "min", + .hash = 0, + .value = RRDR_GROUPING_MIN, + .init = NULL, + .create= grouping_create_min, + .reset = grouping_reset_min, + .free = grouping_free_min, + .add = grouping_add_min, + .flush = grouping_flush_min + }, + {.name = "max", + .hash = 0, + .value = RRDR_GROUPING_MAX, + .init = NULL, + .create= grouping_create_max, + .reset = grouping_reset_max, + .free = grouping_free_max, + .add = grouping_add_max, + .flush = grouping_flush_max + }, + {.name = "sum", + .hash = 0, + .value = RRDR_GROUPING_SUM, + .init = NULL, + .create= grouping_create_sum, + .reset = grouping_reset_sum, + .free = grouping_free_sum, + .add = grouping_add_sum, + .flush = grouping_flush_sum + }, + + // standard deviation + {.name = "stddev", + .hash = 0, + .value = RRDR_GROUPING_STDDEV, + .init = NULL, + .create= grouping_create_stddev, + .reset = grouping_reset_stddev, + .free = grouping_free_stddev, + .add = grouping_add_stddev, + .flush = grouping_flush_stddev + }, + {.name = "cv", // coefficient variation is calculated by stddev + .hash = 0, + .value = RRDR_GROUPING_CV, + .init = NULL, + .create= grouping_create_stddev, // not an error, stddev calculates this too + .reset = grouping_reset_stddev, // not an error, stddev calculates this too + .free = grouping_free_stddev, // not an error, stddev calculates this too + .add = grouping_add_stddev, // not an error, stddev calculates this too + .flush = grouping_flush_coefficient_of_variation + }, + {.name = "rsd", // alias of 'cv' + .hash = 0, + .value = RRDR_GROUPING_CV, + .init = NULL, + .create= grouping_create_stddev, // not an error, stddev calculates this too + .reset = grouping_reset_stddev, // not an error, stddev calculates this too + .free = grouping_free_stddev, // not an error, stddev calculates this too + .add = grouping_add_stddev, // not an error, stddev calculates this too + .flush = grouping_flush_coefficient_of_variation + }, + + /* + {.name = "mean", // same as average, no need to define it again + .hash = 0, + .value = RRDR_GROUPING_MEAN, + .setup = NULL, + .create= grouping_create_stddev, + .reset = grouping_reset_stddev, + .free = grouping_free_stddev, + .add = grouping_add_stddev, + .flush = grouping_flush_mean + }, + */ + + /* + {.name = "variance", // meaningless to offer + .hash = 0, + .value = RRDR_GROUPING_VARIANCE, + .setup = NULL, + .create= grouping_create_stddev, + .reset = grouping_reset_stddev, + .free = grouping_free_stddev, + .add = grouping_add_stddev, + .flush = grouping_flush_variance + }, + */ + + // single exponential smoothing + {.name = "ses", + .hash = 0, + .value = RRDR_GROUPING_SES, + .init = grouping_init_ses, + .create= grouping_create_ses, + .reset = grouping_reset_ses, + .free = grouping_free_ses, + .add = grouping_add_ses, + .flush = grouping_flush_ses + }, + {.name = "ema", // alias for 'ses' + .hash = 0, + .value = RRDR_GROUPING_SES, + .init = NULL, + .create= grouping_create_ses, + .reset = grouping_reset_ses, + .free = grouping_free_ses, + .add = grouping_add_ses, + .flush = grouping_flush_ses + }, + {.name = "ewma", // alias for ses + .hash = 0, + .value = RRDR_GROUPING_SES, + .init = NULL, + .create= grouping_create_ses, + .reset = grouping_reset_ses, + .free = grouping_free_ses, + .add = grouping_add_ses, + .flush = grouping_flush_ses + }, + + // double exponential smoothing + {.name = "des", + .hash = 0, + .value = RRDR_GROUPING_DES, + .init = grouping_init_des, + .create= grouping_create_des, + .reset = grouping_reset_des, + .free = grouping_free_des, + .add = grouping_add_des, + .flush = grouping_flush_des + }, + + // terminator + {.name = NULL, + .hash = 0, + .value = RRDR_GROUPING_UNDEFINED, + .init = NULL, + .create= grouping_create_average, + .reset = grouping_reset_average, + .free = grouping_free_average, + .add = grouping_add_average, + .flush = grouping_flush_average + } +}; + +void web_client_api_v1_init_grouping(void) { + int i; + + for(i = 0; api_v1_data_groups[i].name ; i++) { + api_v1_data_groups[i].hash = simple_hash(api_v1_data_groups[i].name); + + if(api_v1_data_groups[i].init) + api_v1_data_groups[i].init(); + } +} + +const char *group_method2string(RRDR_GROUPING group) { + int i; + + for(i = 0; api_v1_data_groups[i].name ; i++) { + if(api_v1_data_groups[i].value == group) { + return api_v1_data_groups[i].name; + } + } + + return "unknown-group-method"; +} + +RRDR_GROUPING web_client_api_request_v1_data_group(const char *name, RRDR_GROUPING def) { + int i; + + uint32_t hash = simple_hash(name); + for(i = 0; api_v1_data_groups[i].name ; i++) + if(unlikely(hash == api_v1_data_groups[i].hash && !strcmp(name, api_v1_data_groups[i].name))) + return api_v1_data_groups[i].value; + + return def; +} + +// ---------------------------------------------------------------------------- + +static void rrdr_disable_not_selected_dimensions(RRDR *r, RRDR_OPTIONS options, const char *dims, RRDDIM *temp_rd) { + rrdset_check_rdlock(r->st); + + if(unlikely(!dims || !*dims || (dims[0] == '*' && dims[1] == '\0'))) return; + + int match_ids = 0, match_names = 0; + + if(unlikely(options & RRDR_OPTION_MATCH_IDS)) + match_ids = 1; + if(unlikely(options & RRDR_OPTION_MATCH_NAMES)) + match_names = 1; + + if(likely(!match_ids && !match_names)) + match_ids = match_names = 1; + + SIMPLE_PATTERN *pattern = simple_pattern_create(dims, ",|\t\r\n\f\v", SIMPLE_PATTERN_EXACT); + + RRDDIM *d; + long c, dims_selected = 0, dims_not_hidden_not_zero = 0; + for(c = 0, d = temp_rd?temp_rd:r->st->dimensions; d ;c++, d = d->next) { + if( (match_ids && simple_pattern_matches(pattern, d->id)) + || (match_names && simple_pattern_matches(pattern, d->name)) + ) { + r->od[c] |= RRDR_DIMENSION_SELECTED; + if(unlikely(r->od[c] & RRDR_DIMENSION_HIDDEN)) r->od[c] &= ~RRDR_DIMENSION_HIDDEN; + dims_selected++; + + // since the user needs this dimension + // make it appear as NONZERO, to return it + // even if the dimension has only zeros + // unless option non_zero is set + if(unlikely(!(options & RRDR_OPTION_NONZERO))) + r->od[c] |= RRDR_DIMENSION_NONZERO; + + // count the visible dimensions + if(likely(r->od[c] & RRDR_DIMENSION_NONZERO)) + dims_not_hidden_not_zero++; + } + else { + r->od[c] |= RRDR_DIMENSION_HIDDEN; + if(unlikely(r->od[c] & RRDR_DIMENSION_SELECTED)) r->od[c] &= ~RRDR_DIMENSION_SELECTED; + } + } + simple_pattern_free(pattern); + + // check if all dimensions are hidden + if(unlikely(!dims_not_hidden_not_zero && dims_selected)) { + // there are a few selected dimensions + // but they are all zero + // enable the selected ones + // to avoid returning an empty chart + for(c = 0, d = temp_rd?temp_rd:r->st->dimensions; d ;c++, d = d->next) + if(unlikely(r->od[c] & RRDR_DIMENSION_SELECTED)) + r->od[c] |= RRDR_DIMENSION_NONZERO; + } +} + +// ---------------------------------------------------------------------------- +// helpers to find our way in RRDR + +static inline RRDR_VALUE_FLAGS *rrdr_line_options(RRDR *r, long rrdr_line) { + return &r->o[ rrdr_line * r->d ]; +} + +static inline calculated_number *rrdr_line_values(RRDR *r, long rrdr_line) { + return &r->v[ rrdr_line * r->d ]; +} + +static inline long rrdr_line_init(RRDR *r, time_t t, long rrdr_line) { + rrdr_line++; + + #ifdef NETDATA_INTERNAL_CHECKS + + if(unlikely(rrdr_line >= r->n)) + error("INTERNAL ERROR: requested to step above RRDR size for chart '%s'", r->st->name); + + if(unlikely(r->t[rrdr_line] != 0 && r->t[rrdr_line] != t)) + error("INTERNAL ERROR: overwriting the timestamp of RRDR line %zu from %zu to %zu, of chart '%s'", (size_t)rrdr_line, (size_t)r->t[rrdr_line], (size_t)t, r->st->name); + + #endif + + // save the time + r->t[rrdr_line] = t; + + return rrdr_line; +} + +static inline void rrdr_done(RRDR *r, long rrdr_line) { + r->rows = rrdr_line + 1; +} + + +// ---------------------------------------------------------------------------- +// fill RRDR for a single dimension + +static inline void do_dimension_variablestep( + RRDR *r + , long points_wanted + , RRDDIM *rd + , long dim_id_in_rrdr + , time_t after_wanted + , time_t before_wanted +){ +// RRDSET *st = r->st; + + time_t + now = after_wanted, + dt = r->update_every, + max_date = 0, + min_date = 0; + + long +// group_size = r->group, + points_added = 0, + values_in_group = 0, + values_in_group_non_zero = 0, + rrdr_line = -1; + + RRDR_VALUE_FLAGS + group_value_flags = RRDR_VALUE_NOTHING; + + struct rrddim_query_handle handle; + + calculated_number min = r->min, max = r->max; + size_t db_points_read = 0; + time_t db_now = now; + storage_number n_curr, n_prev = SN_EMPTY_SLOT; + calculated_number value; + + for(rd->state->query_ops.init(rd, &handle, now, before_wanted) ; points_added < points_wanted ; now += dt) { + // make sure we return data in the proper time range + if (unlikely(now > before_wanted)) { +#ifdef NETDATA_INTERNAL_CHECKS + r->internal.log = "stopped, because attempted to access the db after 'wanted before'"; +#endif + break; + } + if (unlikely(now < after_wanted)) { +#ifdef NETDATA_INTERNAL_CHECKS + r->internal.log = "skipped, because attempted to access the db before 'wanted after'"; +#endif + continue; + } + + while (now >= db_now && (!rd->state->query_ops.is_finished(&handle) || + does_storage_number_exist(n_prev))) { + value = NAN; + if (does_storage_number_exist(n_prev)) { + // use the previously read database value + n_curr = n_prev; + } else { + // read the value from the database + n_curr = rd->state->query_ops.next_metric(&handle, &db_now); + } + n_prev = SN_EMPTY_SLOT; + // db_now has a different value than above + if (likely(now >= db_now)) { + if (likely(does_storage_number_exist(n_curr))) { + value = unpack_storage_number(n_curr); + if (likely(value != 0.0)) + values_in_group_non_zero++; + + if (unlikely(did_storage_number_reset(n_curr))) + group_value_flags |= RRDR_VALUE_RESET; + } + } else { + // We must postpone processing the value and fill the result with gaps instead + if (likely(does_storage_number_exist(n_curr))) { + n_prev = n_curr; + } + } + // add this value to grouping + r->internal.grouping_add(r, value); + values_in_group++; + db_points_read++; + } + + if (0 == values_in_group) { + // add NAN to grouping + r->internal.grouping_add(r, NAN); + } + + rrdr_line = rrdr_line_init(r, now, rrdr_line); + + if(unlikely(!min_date)) min_date = now; + max_date = now; + + // find the place to store our values + RRDR_VALUE_FLAGS *rrdr_value_options_ptr = &r->o[rrdr_line * r->d + dim_id_in_rrdr]; + + // update the dimension options + if(likely(values_in_group_non_zero)) + r->od[dim_id_in_rrdr] |= RRDR_DIMENSION_NONZERO; + + // store the specific point options + *rrdr_value_options_ptr = group_value_flags; + + // store the value + value = r->internal.grouping_flush(r, rrdr_value_options_ptr); + r->v[rrdr_line * r->d + dim_id_in_rrdr] = value; + + if(likely(points_added || dim_id_in_rrdr)) { + // find the min/max across all dimensions + + if(unlikely(value < min)) min = value; + if(unlikely(value > max)) max = value; + + } + else { + // runs only when dim_id_in_rrdr == 0 && points_added == 0 + // so, on the first point added for the query. + min = max = value; + } + + points_added++; + values_in_group = 0; + group_value_flags = RRDR_VALUE_NOTHING; + values_in_group_non_zero = 0; + } + rd->state->query_ops.finalize(&handle); + + r->internal.db_points_read += db_points_read; + r->internal.result_points_generated += points_added; + + r->min = min; + r->max = max; + r->before = max_date; + r->after = min_date - (r->group - 1) * dt; + rrdr_done(r, rrdr_line); + + #ifdef NETDATA_INTERNAL_CHECKS + if(unlikely(r->rows != points_added)) + error("INTERNAL ERROR: %s.%s added %zu rows, but RRDR says I added %zu.", r->st->name, rd->name, (size_t)points_added, (size_t)r->rows); + #endif +} + +static inline void do_dimension_fixedstep( + RRDR *r + , long points_wanted + , RRDDIM *rd + , long dim_id_in_rrdr + , time_t after_wanted + , time_t before_wanted +){ + RRDSET *st = r->st; + + time_t + now = after_wanted, + dt = r->update_every / r->group, /* usually is st->update_every */ + max_date = 0, + min_date = 0; + + long + group_size = r->group, + points_added = 0, + values_in_group = 0, + values_in_group_non_zero = 0, + rrdr_line = -1; + + RRDR_VALUE_FLAGS + group_value_flags = RRDR_VALUE_NOTHING; + + struct rrddim_query_handle handle; + + calculated_number min = r->min, max = r->max; + size_t db_points_read = 0; + time_t db_now = now; + + for(rd->state->query_ops.init(rd, &handle, now, before_wanted) ; points_added < points_wanted ; now += dt) { + // make sure we return data in the proper time range + if(unlikely(now > before_wanted)) { +#ifdef NETDATA_INTERNAL_CHECKS + r->internal.log = "stopped, because attempted to access the db after 'wanted before'"; +#endif + break; + } + if(unlikely(now < after_wanted)) { +#ifdef NETDATA_INTERNAL_CHECKS + r->internal.log = "skipped, because attempted to access the db before 'wanted after'"; +#endif + continue; + } + // read the value from the database + //storage_number n = rd->values[slot]; +#ifdef NETDATA_INTERNAL_CHECKS + if ((rd->rrd_memory_mode != RRD_MEMORY_MODE_DBENGINE) && + (rrdset_time2slot(st, now) != (long unsigned)handle.slotted.slot)) { + error("INTERNAL CHECK: Unaligned query for %s, database slot: %lu, expected slot: %lu", rd->id, (long unsigned)handle.slotted.slot, rrdset_time2slot(st, now)); + } +#endif + db_now = now; // this is needed to set db_now in case the next_metric implementation does not set it + storage_number n = rd->state->query_ops.next_metric(&handle, &db_now); + if(unlikely(db_now > before_wanted)) { +#ifdef NETDATA_INTERNAL_CHECKS + r->internal.log = "stopped, because attempted to access the db after 'wanted before'"; +#endif + break; + } + for ( ; now <= db_now ; now += dt) { + calculated_number value = NAN; + if(likely(now >= db_now && does_storage_number_exist(n))) { +#if defined(NETDATA_INTERNAL_CHECKS) && defined(ENABLE_DBENGINE) + if ((rd->rrd_memory_mode == RRD_MEMORY_MODE_DBENGINE) && (now != handle.rrdeng.now)) { + error("INTERNAL CHECK: Unaligned query for %s, database time: %ld, expected time: %ld", rd->id, (long)handle.rrdeng.now, (long)now); + } +#endif + value = unpack_storage_number(n); + if(likely(value != 0.0)) + values_in_group_non_zero++; + + if(unlikely(did_storage_number_reset(n))) + group_value_flags |= RRDR_VALUE_RESET; + + } + + // add this value for grouping + r->internal.grouping_add(r, value); + values_in_group++; + db_points_read++; + + if(unlikely(values_in_group == group_size)) { + rrdr_line = rrdr_line_init(r, now, rrdr_line); + + if(unlikely(!min_date)) min_date = now; + max_date = now; + + // find the place to store our values + RRDR_VALUE_FLAGS *rrdr_value_options_ptr = &r->o[rrdr_line * r->d + dim_id_in_rrdr]; + + // update the dimension options + if(likely(values_in_group_non_zero)) + r->od[dim_id_in_rrdr] |= RRDR_DIMENSION_NONZERO; + + // store the specific point options + *rrdr_value_options_ptr = group_value_flags; + + // store the value + calculated_number value = r->internal.grouping_flush(r, rrdr_value_options_ptr); + r->v[rrdr_line * r->d + dim_id_in_rrdr] = value; + + if(likely(points_added || dim_id_in_rrdr)) { + // find the min/max across all dimensions + + if(unlikely(value < min)) min = value; + if(unlikely(value > max)) max = value; + + } + else { + // runs only when dim_id_in_rrdr == 0 && points_added == 0 + // so, on the first point added for the query. + min = max = value; + } + + points_added++; + values_in_group = 0; + group_value_flags = RRDR_VALUE_NOTHING; + values_in_group_non_zero = 0; + } + } + now = db_now; + } + rd->state->query_ops.finalize(&handle); + + r->internal.db_points_read += db_points_read; + r->internal.result_points_generated += points_added; + + r->min = min; + r->max = max; + r->before = max_date; + r->after = min_date - (r->group - 1) * dt; + rrdr_done(r, rrdr_line); + +#ifdef NETDATA_INTERNAL_CHECKS + if(unlikely(r->rows != points_added)) + error("INTERNAL ERROR: %s.%s added %zu rows, but RRDR says I added %zu.", r->st->name, rd->name, (size_t)points_added, (size_t)r->rows); +#endif +} + +// ---------------------------------------------------------------------------- +// fill RRDR for the whole chart + +#ifdef NETDATA_INTERNAL_CHECKS +static void rrd2rrdr_log_request_response_metdata(RRDR *r + , RRDR_GROUPING group_method + , int aligned + , long group + , long resampling_time + , long resampling_group + , time_t after_wanted + , time_t after_requested + , time_t before_wanted + , time_t before_requested + , long points_requested + , long points_wanted + //, size_t after_slot + //, size_t before_slot + , const char *msg + ) { + netdata_rwlock_rdlock(&r->st->rrdset_rwlock); + info("INTERNAL ERROR: rrd2rrdr() on %s update every %d with %s grouping %s (group: %ld, resampling_time: %ld, resampling_group: %ld), " + "after (got: %zu, want: %zu, req: %zu, db: %zu), " + "before (got: %zu, want: %zu, req: %zu, db: %zu), " + "duration (got: %zu, want: %zu, req: %zu, db: %zu), " + //"slot (after: %zu, before: %zu, delta: %zu), " + "points (got: %ld, want: %ld, req: %ld, db: %ld), " + "%s" + , r->st->name + , r->st->update_every + + // grouping + , (aligned) ? "aligned" : "unaligned" + , group_method2string(group_method) + , group + , resampling_time + , resampling_group + + // after + , (size_t)r->after + , (size_t)after_wanted + , (size_t)after_requested + , (size_t)rrdset_first_entry_t_nolock(r->st) + + // before + , (size_t)r->before + , (size_t)before_wanted + , (size_t)before_requested + , (size_t)rrdset_last_entry_t_nolock(r->st) + + // duration + , (size_t)(r->before - r->after + r->st->update_every) + , (size_t)(before_wanted - after_wanted + r->st->update_every) + , (size_t)(before_requested - after_requested) + , (size_t)((rrdset_last_entry_t_nolock(r->st) - rrdset_first_entry_t_nolock(r->st)) + r->st->update_every) + + // slot + /* + , after_slot + , before_slot + , (after_slot > before_slot) ? (r->st->entries - after_slot + before_slot) : (before_slot - after_slot) + */ + + // points + , r->rows + , points_wanted + , points_requested + , r->st->entries + + // message + , msg + ); + netdata_rwlock_unlock(&r->st->rrdset_rwlock); +} +#endif // NETDATA_INTERNAL_CHECKS + +// Returns 1 if an absolute period was requested or 0 if it was a relative period +static int rrdr_convert_before_after_to_absolute( + long long *after_requestedp + , long long *before_requestedp + , int update_every + , time_t first_entry_t + , time_t last_entry_t + , RRDR_OPTIONS options +) { + int absolute_period_requested = -1; + long long after_requested, before_requested; + + before_requested = *before_requestedp; + after_requested = *after_requestedp; + + if(before_requested == 0 && after_requested == 0) { + // dump the all the data + before_requested = last_entry_t; + after_requested = first_entry_t; + absolute_period_requested = 0; + } + + // allow relative for before (smaller than API_RELATIVE_TIME_MAX) + if(abs(before_requested) <= API_RELATIVE_TIME_MAX) { + if(abs(before_requested) % update_every) { + // make sure it is multiple of st->update_every + if(before_requested < 0) before_requested = before_requested - update_every - + before_requested % update_every; + else before_requested = before_requested + update_every - before_requested % update_every; + } + if(before_requested > 0) before_requested = first_entry_t + before_requested; + else before_requested = last_entry_t + before_requested; //last_entry_t is not really now_t + //TODO: fix before_requested to be relative to now_t + absolute_period_requested = 0; + } + + // allow relative for after (smaller than API_RELATIVE_TIME_MAX) + if(abs(after_requested) <= API_RELATIVE_TIME_MAX) { + if(after_requested == 0) after_requested = -update_every; + if(abs(after_requested) % update_every) { + // make sure it is multiple of st->update_every + if(after_requested < 0) after_requested = after_requested - update_every - after_requested % update_every; + else after_requested = after_requested + update_every - after_requested % update_every; + } + after_requested = before_requested + after_requested; + absolute_period_requested = 0; + } + + if(absolute_period_requested == -1) + absolute_period_requested = 1; + + // make sure they are within our timeframe + if(before_requested > last_entry_t) before_requested = last_entry_t; + if(before_requested < first_entry_t && !(options & RRDR_OPTION_ALLOW_PAST)) + before_requested = first_entry_t; + + if(after_requested > last_entry_t) after_requested = last_entry_t; + if(after_requested < first_entry_t && !(options & RRDR_OPTION_ALLOW_PAST)) + after_requested = first_entry_t; + + // check if they are reversed + if(after_requested > before_requested) { + time_t tmp = before_requested; + before_requested = after_requested; + after_requested = tmp; + } + + *before_requestedp = before_requested; + *after_requestedp = after_requested; + + return absolute_period_requested; +} + +static RRDR *rrd2rrdr_fixedstep( + RRDSET *st + , long points_requested + , long long after_requested + , long long before_requested + , RRDR_GROUPING group_method + , long resampling_time_requested + , RRDR_OPTIONS options + , const char *dimensions + , int update_every + , time_t first_entry_t + , time_t last_entry_t + , int absolute_period_requested + , struct context_param *context_param_list +) { + int aligned = !(options & RRDR_OPTION_NOT_ALIGNED); + + // the duration of the chart + time_t duration = before_requested - after_requested; + long available_points = duration / update_every; + + RRDDIM *temp_rd = context_param_list ? context_param_list->rd : NULL; + + if(duration <= 0 || available_points <= 0) + return rrdr_create(st, 1, context_param_list); + + // check the number of wanted points in the result + if(unlikely(points_requested < 0)) points_requested = -points_requested; + if(unlikely(points_requested > available_points)) points_requested = available_points; + if(unlikely(points_requested == 0)) points_requested = available_points; + + // calculate the desired grouping of source data points + long group = available_points / points_requested; + if(unlikely(group <= 0)) group = 1; + if(unlikely(available_points % points_requested > points_requested / 2)) group++; // rounding to the closest integer + + // resampling_time_requested enforces a certain grouping multiple + calculated_number resampling_divisor = 1.0; + long resampling_group = 1; + if(unlikely(resampling_time_requested > update_every)) { + if (unlikely(resampling_time_requested > duration)) { + // group_time is above the available duration + + #ifdef NETDATA_INTERNAL_CHECKS + info("INTERNAL CHECK: %s: requested gtime %ld secs, is greater than the desired duration %ld secs", st->id, resampling_time_requested, duration); + #endif + + after_requested = before_requested - resampling_time_requested; + duration = before_requested - after_requested; + available_points = duration / update_every; + group = available_points / points_requested; + } + + // if the duration is not aligned to resampling time + // extend the duration to the past, to avoid a gap at the chart + // only when the missing duration is above 1/10th of a point + if(duration % resampling_time_requested) { + time_t delta = duration % resampling_time_requested; + if(delta > resampling_time_requested / 10) { + after_requested -= resampling_time_requested - delta; + duration = before_requested - after_requested; + available_points = duration / update_every; + group = available_points / points_requested; + } + } + + // the points we should group to satisfy gtime + resampling_group = resampling_time_requested / update_every; + if(unlikely(resampling_time_requested % update_every)) { + #ifdef NETDATA_INTERNAL_CHECKS + info("INTERNAL CHECK: %s: requested gtime %ld secs, is not a multiple of the chart's data collection frequency %d secs", st->id, resampling_time_requested, update_every); + #endif + + resampling_group++; + } + + // adapt group according to resampling_group + if(unlikely(group < resampling_group)) group = resampling_group; // do not allow grouping below the desired one + if(unlikely(group % resampling_group)) group += resampling_group - (group % resampling_group); // make sure group is multiple of resampling_group + + //resampling_divisor = group / resampling_group; + resampling_divisor = (calculated_number)(group * update_every) / (calculated_number)resampling_time_requested; + } + + // now that we have group, + // align the requested timeframe to fit it. + + if(aligned) { + // alignement has been requested, so align the values + before_requested -= before_requested % (group * update_every); + after_requested -= after_requested % (group * update_every); + } + + // we align the request on requested_before + time_t before_wanted = before_requested; + if(likely(before_wanted > last_entry_t)) { + #ifdef NETDATA_INTERNAL_CHECKS + error("INTERNAL ERROR: rrd2rrdr() on %s, before_wanted is after db max", st->name); + #endif + + before_wanted = last_entry_t - (last_entry_t % ( ((aligned)?group:1) * update_every )); + } + //size_t before_slot = rrdset_time2slot(st, before_wanted); + + // we need to estimate the number of points, for having + // an integer number of values per point + long points_wanted = (before_wanted - after_requested) / (update_every * group); + + time_t after_wanted = before_wanted - (points_wanted * group * update_every) + update_every; + if(unlikely(after_wanted < first_entry_t)) { + // hm... we go to the past, calculate again points_wanted using all the db from before_wanted to the beginning + points_wanted = (before_wanted - first_entry_t) / group; + + // recalculate after wanted with the new number of points + after_wanted = before_wanted - (points_wanted * group * update_every) + update_every; + + if(unlikely(after_wanted < first_entry_t)) { + #ifdef NETDATA_INTERNAL_CHECKS + error("INTERNAL ERROR: rrd2rrdr() on %s, after_wanted is before db min", st->name); + #endif + + after_wanted = first_entry_t - (first_entry_t % ( ((aligned)?group:1) * update_every )) + ( ((aligned)?group:1) * update_every ); + } + } + //size_t after_slot = rrdset_time2slot(st, after_wanted); + + // check if they are reversed + if(unlikely(after_wanted > before_wanted)) { + #ifdef NETDATA_INTERNAL_CHECKS + error("INTERNAL ERROR: rrd2rrdr() on %s, reversed wanted after/before", st->name); + #endif + time_t tmp = before_wanted; + before_wanted = after_wanted; + after_wanted = tmp; + } + + // recalculate points_wanted using the final time-frame + points_wanted = (before_wanted - after_wanted) / update_every / group + 1; + if(unlikely(points_wanted < 0)) { + #ifdef NETDATA_INTERNAL_CHECKS + error("INTERNAL ERROR: rrd2rrdr() on %s, points_wanted is %ld", st->name, points_wanted); + #endif + points_wanted = 0; + } + +#ifdef NETDATA_INTERNAL_CHECKS + duration = before_wanted - after_wanted; + + if(after_wanted < first_entry_t) + error("INTERNAL CHECK: after_wanted %u is too small, minimum %u", (uint32_t)after_wanted, (uint32_t)first_entry_t); + + if(after_wanted > last_entry_t) + error("INTERNAL CHECK: after_wanted %u is too big, maximum %u", (uint32_t)after_wanted, (uint32_t)last_entry_t); + + if(before_wanted < first_entry_t) + error("INTERNAL CHECK: before_wanted %u is too small, minimum %u", (uint32_t)before_wanted, (uint32_t)first_entry_t); + + if(before_wanted > last_entry_t) + error("INTERNAL CHECK: before_wanted %u is too big, maximum %u", (uint32_t)before_wanted, (uint32_t)last_entry_t); + +/* + if(before_slot >= (size_t)st->entries) + error("INTERNAL CHECK: before_slot is invalid %zu, expected 0 to %ld", before_slot, st->entries - 1); + + if(after_slot >= (size_t)st->entries) + error("INTERNAL CHECK: after_slot is invalid %zu, expected 0 to %ld", after_slot, st->entries - 1); +*/ + + if(points_wanted > (before_wanted - after_wanted) / group / update_every + 1) + error("INTERNAL CHECK: points_wanted %ld is more than points %ld", points_wanted, (before_wanted - after_wanted) / group / update_every + 1); + + if(group < resampling_group) + error("INTERNAL CHECK: group %ld is less than the desired group points %ld", group, resampling_group); + + if(group > resampling_group && group % resampling_group) + error("INTERNAL CHECK: group %ld is not a multiple of the desired group points %ld", group, resampling_group); +#endif + + // ------------------------------------------------------------------------- + // initialize our result set + // this also locks the chart for us + + RRDR *r = rrdr_create(st, points_wanted, context_param_list); + if(unlikely(!r)) { + #ifdef NETDATA_INTERNAL_CHECKS + error("INTERNAL CHECK: Cannot create RRDR for %s, after=%u, before=%u, duration=%u, points=%ld", st->id, (uint32_t)after_wanted, (uint32_t)before_wanted, (uint32_t)duration, points_wanted); + #endif + return NULL; + } + + if(unlikely(!r->d || !points_wanted)) { + #ifdef NETDATA_INTERNAL_CHECKS + error("INTERNAL CHECK: Returning empty RRDR (no dimensions in RRDSET) for %s, after=%u, before=%u, duration=%zu, points=%ld", st->id, (uint32_t)after_wanted, (uint32_t)before_wanted, (size_t)duration, points_wanted); + #endif + return r; + } + + if(unlikely(absolute_period_requested == 1)) + r->result_options |= RRDR_RESULT_OPTION_ABSOLUTE; + else + r->result_options |= RRDR_RESULT_OPTION_RELATIVE; + + // find how many dimensions we have + long dimensions_count = r->d; + + // ------------------------------------------------------------------------- + // initialize RRDR + + r->group = group; + r->update_every = (int)group * update_every; + r->before = before_wanted; + r->after = after_wanted; + r->internal.points_wanted = points_wanted; + r->internal.resampling_group = resampling_group; + r->internal.resampling_divisor = resampling_divisor; + + + // ------------------------------------------------------------------------- + // assign the processor functions + + { + int i, found = 0; + for(i = 0; !found && api_v1_data_groups[i].name ;i++) { + if(api_v1_data_groups[i].value == group_method) { + r->internal.grouping_create= api_v1_data_groups[i].create; + r->internal.grouping_reset = api_v1_data_groups[i].reset; + r->internal.grouping_free = api_v1_data_groups[i].free; + r->internal.grouping_add = api_v1_data_groups[i].add; + r->internal.grouping_flush = api_v1_data_groups[i].flush; + found = 1; + } + } + if(!found) { + errno = 0; + #ifdef NETDATA_INTERNAL_CHECKS + error("INTERNAL ERROR: grouping method %u not found for chart '%s'. Using 'average'", (unsigned int)group_method, r->st->name); + #endif + r->internal.grouping_create= grouping_create_average; + r->internal.grouping_reset = grouping_reset_average; + r->internal.grouping_free = grouping_free_average; + r->internal.grouping_add = grouping_add_average; + r->internal.grouping_flush = grouping_flush_average; + } + } + + // allocate any memory required by the grouping method + r->internal.grouping_data = r->internal.grouping_create(r); + + + // ------------------------------------------------------------------------- + // disable the not-wanted dimensions + + rrdset_check_rdlock(st); + + if(dimensions) + rrdr_disable_not_selected_dimensions(r, options, dimensions, temp_rd); + + + // ------------------------------------------------------------------------- + // do the work for each dimension + + time_t max_after = 0, min_before = 0; + long max_rows = 0; + + RRDDIM *rd; + long c, dimensions_used = 0, dimensions_nonzero = 0; + for(rd = temp_rd?temp_rd:st->dimensions, c = 0 ; rd && c < dimensions_count ; rd = rd->next, c++) { + + // if we need a percentage, we need to calculate all dimensions + if(unlikely(!(options & RRDR_OPTION_PERCENTAGE) && (r->od[c] & RRDR_DIMENSION_HIDDEN))) { + if(unlikely(r->od[c] & RRDR_DIMENSION_SELECTED)) r->od[c] &= ~RRDR_DIMENSION_SELECTED; + continue; + } + r->od[c] |= RRDR_DIMENSION_SELECTED; + + // reset the grouping for the new dimension + r->internal.grouping_reset(r); + + do_dimension_fixedstep( + r + , points_wanted + , rd + , c + , after_wanted + , before_wanted + ); + + if(r->od[c] & RRDR_DIMENSION_NONZERO) + dimensions_nonzero++; + + // verify all dimensions are aligned + if(unlikely(!dimensions_used)) { + min_before = r->before; + max_after = r->after; + max_rows = r->rows; + } + else { + if(r->after != max_after) { + #ifdef NETDATA_INTERNAL_CHECKS + error("INTERNAL ERROR: 'after' mismatch between dimensions for chart '%s': max is %zu, dimension '%s' has %zu", + st->name, (size_t)max_after, rd->name, (size_t)r->after); + #endif + r->after = (r->after > max_after) ? r->after : max_after; + } + + if(r->before != min_before) { + #ifdef NETDATA_INTERNAL_CHECKS + error("INTERNAL ERROR: 'before' mismatch between dimensions for chart '%s': max is %zu, dimension '%s' has %zu", + st->name, (size_t)min_before, rd->name, (size_t)r->before); + #endif + r->before = (r->before < min_before) ? r->before : min_before; + } + + if(r->rows != max_rows) { + #ifdef NETDATA_INTERNAL_CHECKS + error("INTERNAL ERROR: 'rows' mismatch between dimensions for chart '%s': max is %zu, dimension '%s' has %zu", + st->name, (size_t)max_rows, rd->name, (size_t)r->rows); + #endif + r->rows = (r->rows > max_rows) ? r->rows : max_rows; + } + } + + dimensions_used++; + } + + #ifdef NETDATA_INTERNAL_CHECKS + if (dimensions_used) { + if(r->internal.log) + rrd2rrdr_log_request_response_metdata(r, group_method, aligned, group, resampling_time_requested, resampling_group, after_wanted, after_requested, before_wanted, before_requested, points_requested, points_wanted, /*after_slot, before_slot,*/ r->internal.log); + + if(r->rows != points_wanted) + rrd2rrdr_log_request_response_metdata(r, group_method, aligned, group, resampling_time_requested, resampling_group, after_wanted, after_requested, before_wanted, before_requested, points_requested, points_wanted, /*after_slot, before_slot,*/ "got 'points' is not wanted 'points'"); + + if(aligned && (r->before % group) != 0) + rrd2rrdr_log_request_response_metdata(r, group_method, aligned, group, resampling_time_requested, resampling_group, after_wanted, after_requested, before_wanted, before_requested, points_requested, points_wanted, /*after_slot, before_slot,*/ "'before' is not aligned but alignment is required"); + + // 'after' should not be aligned, since we start inside the first group + //if(aligned && (r->after % group) != 0) + // rrd2rrdr_log_request_response_metdata(r, group_method, aligned, group, resampling_time_requested, resampling_group, after_wanted, after_requested, before_wanted, before_requested, points_requested, points_wanted, after_slot, before_slot, "'after' is not aligned but alignment is required"); + + if(r->before != before_requested) + rrd2rrdr_log_request_response_metdata(r, group_method, aligned, group, resampling_time_requested, resampling_group, after_wanted, after_requested, before_wanted, before_requested, points_requested, points_wanted, /*after_slot, before_slot,*/ "chart is not aligned to requested 'before'"); + + if(r->before != before_wanted) + rrd2rrdr_log_request_response_metdata(r, group_method, aligned, group, resampling_time_requested, resampling_group, after_wanted, after_requested, before_wanted, before_requested, points_requested, points_wanted, /*after_slot, before_slot,*/ "got 'before' is not wanted 'before'"); + + // reported 'after' varies, depending on group + if(r->after != after_wanted) + rrd2rrdr_log_request_response_metdata(r, group_method, aligned, group, resampling_time_requested, resampling_group, after_wanted, after_requested, before_wanted, before_requested, points_requested, points_wanted, /*after_slot, before_slot,*/ "got 'after' is not wanted 'after'"); + } + #endif + + // free all resources used by the grouping method + r->internal.grouping_free(r); + + // when all the dimensions are zero, we should return all of them + if(unlikely(options & RRDR_OPTION_NONZERO && !dimensions_nonzero)) { + // all the dimensions are zero + // mark them as NONZERO to send them all + for(rd = temp_rd?temp_rd:st->dimensions, c = 0 ; rd && c < dimensions_count ; rd = rd->next, c++) { + if(unlikely(r->od[c] & RRDR_DIMENSION_HIDDEN)) continue; + r->od[c] |= RRDR_DIMENSION_NONZERO; + } + } + + rrdr_query_completed(r->internal.db_points_read, r->internal.result_points_generated); + return r; +} + +#ifdef ENABLE_DBENGINE +static RRDR *rrd2rrdr_variablestep( + RRDSET *st + , long points_requested + , long long after_requested + , long long before_requested + , RRDR_GROUPING group_method + , long resampling_time_requested + , RRDR_OPTIONS options + , const char *dimensions + , int update_every + , time_t first_entry_t + , time_t last_entry_t + , int absolute_period_requested + , struct rrdeng_region_info *region_info_array + , struct context_param *context_param_list +) { + int aligned = !(options & RRDR_OPTION_NOT_ALIGNED); + + // the duration of the chart + time_t duration = before_requested - after_requested; + long available_points = duration / update_every; + + RRDDIM *temp_rd = context_param_list ? context_param_list->rd : NULL; + + if(duration <= 0 || available_points <= 0) { + freez(region_info_array); + return rrdr_create(st, 1, context_param_list); + } + + // check the number of wanted points in the result + if(unlikely(points_requested < 0)) points_requested = -points_requested; + if(unlikely(points_requested > available_points)) points_requested = available_points; + if(unlikely(points_requested == 0)) points_requested = available_points; + + // calculate the desired grouping of source data points + long group = available_points / points_requested; + if(unlikely(group <= 0)) group = 1; + if(unlikely(available_points % points_requested > points_requested / 2)) group++; // rounding to the closest integer + + // resampling_time_requested enforces a certain grouping multiple + calculated_number resampling_divisor = 1.0; + long resampling_group = 1; + if(unlikely(resampling_time_requested > update_every)) { + if (unlikely(resampling_time_requested > duration)) { + // group_time is above the available duration + + #ifdef NETDATA_INTERNAL_CHECKS + info("INTERNAL CHECK: %s: requested gtime %ld secs, is greater than the desired duration %ld secs", st->id, resampling_time_requested, duration); + #endif + + after_requested = before_requested - resampling_time_requested; + duration = before_requested - after_requested; + available_points = duration / update_every; + group = available_points / points_requested; + } + + // if the duration is not aligned to resampling time + // extend the duration to the past, to avoid a gap at the chart + // only when the missing duration is above 1/10th of a point + if(duration % resampling_time_requested) { + time_t delta = duration % resampling_time_requested; + if(delta > resampling_time_requested / 10) { + after_requested -= resampling_time_requested - delta; + duration = before_requested - after_requested; + available_points = duration / update_every; + group = available_points / points_requested; + } + } + + // the points we should group to satisfy gtime + resampling_group = resampling_time_requested / update_every; + if(unlikely(resampling_time_requested % update_every)) { + #ifdef NETDATA_INTERNAL_CHECKS + info("INTERNAL CHECK: %s: requested gtime %ld secs, is not a multiple of the chart's data collection frequency %d secs", st->id, resampling_time_requested, update_every); + #endif + + resampling_group++; + } + + // adapt group according to resampling_group + if(unlikely(group < resampling_group)) group = resampling_group; // do not allow grouping below the desired one + if(unlikely(group % resampling_group)) group += resampling_group - (group % resampling_group); // make sure group is multiple of resampling_group + + //resampling_divisor = group / resampling_group; + resampling_divisor = (calculated_number)(group * update_every) / (calculated_number)resampling_time_requested; + } + + // now that we have group, + // align the requested timeframe to fit it. + + if(aligned) { + // alignement has been requested, so align the values + before_requested -= before_requested % (group * update_every); + after_requested -= after_requested % (group * update_every); + } + + // we align the request on requested_before + time_t before_wanted = before_requested; + if(likely(before_wanted > last_entry_t)) { + #ifdef NETDATA_INTERNAL_CHECKS + error("INTERNAL ERROR: rrd2rrdr() on %s, before_wanted is after db max", st->name); + #endif + + before_wanted = last_entry_t - (last_entry_t % ( ((aligned)?group:1) * update_every )); + } + //size_t before_slot = rrdset_time2slot(st, before_wanted); + + // we need to estimate the number of points, for having + // an integer number of values per point + long points_wanted = (before_wanted - after_requested) / (update_every * group); + + time_t after_wanted = before_wanted - (points_wanted * group * update_every) + update_every; + if(unlikely(after_wanted < first_entry_t)) { + // hm... we go to the past, calculate again points_wanted using all the db from before_wanted to the beginning + points_wanted = (before_wanted - first_entry_t) / group; + + // recalculate after wanted with the new number of points + after_wanted = before_wanted - (points_wanted * group * update_every) + update_every; + + if(unlikely(after_wanted < first_entry_t)) { + #ifdef NETDATA_INTERNAL_CHECKS + error("INTERNAL ERROR: rrd2rrdr() on %s, after_wanted is before db min", st->name); + #endif + + after_wanted = first_entry_t - (first_entry_t % ( ((aligned)?group:1) * update_every )) + ( ((aligned)?group:1) * update_every ); + } + } + //size_t after_slot = rrdset_time2slot(st, after_wanted); + + // check if they are reversed + if(unlikely(after_wanted > before_wanted)) { + #ifdef NETDATA_INTERNAL_CHECKS + error("INTERNAL ERROR: rrd2rrdr() on %s, reversed wanted after/before", st->name); + #endif + time_t tmp = before_wanted; + before_wanted = after_wanted; + after_wanted = tmp; + } + + // recalculate points_wanted using the final time-frame + points_wanted = (before_wanted - after_wanted) / update_every / group + 1; + if(unlikely(points_wanted < 0)) { + #ifdef NETDATA_INTERNAL_CHECKS + error("INTERNAL ERROR: rrd2rrdr() on %s, points_wanted is %ld", st->name, points_wanted); + #endif + points_wanted = 0; + } + +#ifdef NETDATA_INTERNAL_CHECKS + duration = before_wanted - after_wanted; + + if(after_wanted < first_entry_t) + error("INTERNAL CHECK: after_wanted %u is too small, minimum %u", (uint32_t)after_wanted, (uint32_t)first_entry_t); + + if(after_wanted > last_entry_t) + error("INTERNAL CHECK: after_wanted %u is too big, maximum %u", (uint32_t)after_wanted, (uint32_t)last_entry_t); + + if(before_wanted < first_entry_t) + error("INTERNAL CHECK: before_wanted %u is too small, minimum %u", (uint32_t)before_wanted, (uint32_t)first_entry_t); + + if(before_wanted > last_entry_t) + error("INTERNAL CHECK: before_wanted %u is too big, maximum %u", (uint32_t)before_wanted, (uint32_t)last_entry_t); + +/* + if(before_slot >= (size_t)st->entries) + error("INTERNAL CHECK: before_slot is invalid %zu, expected 0 to %ld", before_slot, st->entries - 1); + + if(after_slot >= (size_t)st->entries) + error("INTERNAL CHECK: after_slot is invalid %zu, expected 0 to %ld", after_slot, st->entries - 1); +*/ + + if(points_wanted > (before_wanted - after_wanted) / group / update_every + 1) + error("INTERNAL CHECK: points_wanted %ld is more than points %ld", points_wanted, (before_wanted - after_wanted) / group / update_every + 1); + + if(group < resampling_group) + error("INTERNAL CHECK: group %ld is less than the desired group points %ld", group, resampling_group); + + if(group > resampling_group && group % resampling_group) + error("INTERNAL CHECK: group %ld is not a multiple of the desired group points %ld", group, resampling_group); +#endif + + // ------------------------------------------------------------------------- + // initialize our result set + // this also locks the chart for us + + RRDR *r = rrdr_create(st, points_wanted, context_param_list); + if(unlikely(!r)) { + #ifdef NETDATA_INTERNAL_CHECKS + error("INTERNAL CHECK: Cannot create RRDR for %s, after=%u, before=%u, duration=%u, points=%ld", st->id, (uint32_t)after_wanted, (uint32_t)before_wanted, (uint32_t)duration, points_wanted); + #endif + freez(region_info_array); + return NULL; + } + + if(unlikely(!r->d || !points_wanted)) { + #ifdef NETDATA_INTERNAL_CHECKS + error("INTERNAL CHECK: Returning empty RRDR (no dimensions in RRDSET) for %s, after=%u, before=%u, duration=%zu, points=%ld", st->id, (uint32_t)after_wanted, (uint32_t)before_wanted, (size_t)duration, points_wanted); + #endif + freez(region_info_array); + return r; + } + + r->result_options |= RRDR_RESULT_OPTION_VARIABLE_STEP; + if(unlikely(absolute_period_requested == 1)) + r->result_options |= RRDR_RESULT_OPTION_ABSOLUTE; + else + r->result_options |= RRDR_RESULT_OPTION_RELATIVE; + + // find how many dimensions we have + long dimensions_count = r->d; + + // ------------------------------------------------------------------------- + // initialize RRDR + + r->group = group; + r->update_every = (int)group * update_every; + r->before = before_wanted; + r->after = after_wanted; + r->internal.points_wanted = points_wanted; + r->internal.resampling_group = resampling_group; + r->internal.resampling_divisor = resampling_divisor; + + + // ------------------------------------------------------------------------- + // assign the processor functions + + { + int i, found = 0; + for(i = 0; !found && api_v1_data_groups[i].name ;i++) { + if(api_v1_data_groups[i].value == group_method) { + r->internal.grouping_create= api_v1_data_groups[i].create; + r->internal.grouping_reset = api_v1_data_groups[i].reset; + r->internal.grouping_free = api_v1_data_groups[i].free; + r->internal.grouping_add = api_v1_data_groups[i].add; + r->internal.grouping_flush = api_v1_data_groups[i].flush; + found = 1; + } + } + if(!found) { + errno = 0; + #ifdef NETDATA_INTERNAL_CHECKS + error("INTERNAL ERROR: grouping method %u not found for chart '%s'. Using 'average'", (unsigned int)group_method, r->st->name); + #endif + r->internal.grouping_create= grouping_create_average; + r->internal.grouping_reset = grouping_reset_average; + r->internal.grouping_free = grouping_free_average; + r->internal.grouping_add = grouping_add_average; + r->internal.grouping_flush = grouping_flush_average; + } + } + + // allocate any memory required by the grouping method + r->internal.grouping_data = r->internal.grouping_create(r); + + + // ------------------------------------------------------------------------- + // disable the not-wanted dimensions + + rrdset_check_rdlock(st); + + if(dimensions) + rrdr_disable_not_selected_dimensions(r, options, dimensions, temp_rd); + + + // ------------------------------------------------------------------------- + // do the work for each dimension + + time_t max_after = 0, min_before = 0; + long max_rows = 0; + + RRDDIM *rd; + long c, dimensions_used = 0, dimensions_nonzero = 0; + for(rd = temp_rd?temp_rd:st->dimensions, c = 0 ; rd && c < dimensions_count ; rd = rd->next, c++) { + + // if we need a percentage, we need to calculate all dimensions + if(unlikely(!(options & RRDR_OPTION_PERCENTAGE) && (r->od[c] & RRDR_DIMENSION_HIDDEN))) { + if(unlikely(r->od[c] & RRDR_DIMENSION_SELECTED)) r->od[c] &= ~RRDR_DIMENSION_SELECTED; + continue; + } + r->od[c] |= RRDR_DIMENSION_SELECTED; + + // reset the grouping for the new dimension + r->internal.grouping_reset(r); + + do_dimension_variablestep( + r + , points_wanted + , rd + , c + , after_wanted + , before_wanted + ); + + if(r->od[c] & RRDR_DIMENSION_NONZERO) + dimensions_nonzero++; + + // verify all dimensions are aligned + if(unlikely(!dimensions_used)) { + min_before = r->before; + max_after = r->after; + max_rows = r->rows; + } + else { + if(r->after != max_after) { + #ifdef NETDATA_INTERNAL_CHECKS + error("INTERNAL ERROR: 'after' mismatch between dimensions for chart '%s': max is %zu, dimension '%s' has %zu", + st->name, (size_t)max_after, rd->name, (size_t)r->after); + #endif + r->after = (r->after > max_after) ? r->after : max_after; + } + + if(r->before != min_before) { + #ifdef NETDATA_INTERNAL_CHECKS + error("INTERNAL ERROR: 'before' mismatch between dimensions for chart '%s': max is %zu, dimension '%s' has %zu", + st->name, (size_t)min_before, rd->name, (size_t)r->before); + #endif + r->before = (r->before < min_before) ? r->before : min_before; + } + + if(r->rows != max_rows) { + #ifdef NETDATA_INTERNAL_CHECKS + error("INTERNAL ERROR: 'rows' mismatch between dimensions for chart '%s': max is %zu, dimension '%s' has %zu", + st->name, (size_t)max_rows, rd->name, (size_t)r->rows); + #endif + r->rows = (r->rows > max_rows) ? r->rows : max_rows; + } + } + + dimensions_used++; + } + + #ifdef NETDATA_INTERNAL_CHECKS + + if (dimensions_used) { + if(r->internal.log) + rrd2rrdr_log_request_response_metdata(r, group_method, aligned, group, resampling_time_requested, resampling_group, after_wanted, after_requested, before_wanted, before_requested, points_requested, points_wanted, /*after_slot, before_slot,*/ r->internal.log); + + if(r->rows != points_wanted) + rrd2rrdr_log_request_response_metdata(r, group_method, aligned, group, resampling_time_requested, resampling_group, after_wanted, after_requested, before_wanted, before_requested, points_requested, points_wanted, /*after_slot, before_slot,*/ "got 'points' is not wanted 'points'"); + + if(aligned && (r->before % group) != 0) + rrd2rrdr_log_request_response_metdata(r, group_method, aligned, group, resampling_time_requested, resampling_group, after_wanted, after_requested, before_wanted, before_requested, points_requested, points_wanted, /*after_slot, before_slot,*/ "'before' is not aligned but alignment is required"); + + // 'after' should not be aligned, since we start inside the first group + //if(aligned && (r->after % group) != 0) + // rrd2rrdr_log_request_response_metdata(r, group_method, aligned, group, resampling_time_requested, resampling_group, after_wanted, after_requested, before_wanted, before_requested, points_requested, points_wanted, after_slot, before_slot, "'after' is not aligned but alignment is required"); + + if(r->before != before_requested) + rrd2rrdr_log_request_response_metdata(r, group_method, aligned, group, resampling_time_requested, resampling_group, after_wanted, after_requested, before_wanted, before_requested, points_requested, points_wanted, /*after_slot, before_slot,*/ "chart is not aligned to requested 'before'"); + + if(r->before != before_wanted) + rrd2rrdr_log_request_response_metdata(r, group_method, aligned, group, resampling_time_requested, resampling_group, after_wanted, after_requested, before_wanted, before_requested, points_requested, points_wanted, /*after_slot, before_slot,*/ "got 'before' is not wanted 'before'"); + + // reported 'after' varies, depending on group + if(r->after != after_wanted) + rrd2rrdr_log_request_response_metdata(r, group_method, aligned, group, resampling_time_requested, resampling_group, after_wanted, after_requested, before_wanted, before_requested, points_requested, points_wanted, /*after_slot, before_slot,*/ "got 'after' is not wanted 'after'"); + } + #endif + + // free all resources used by the grouping method + r->internal.grouping_free(r); + + // when all the dimensions are zero, we should return all of them + if(unlikely(options & RRDR_OPTION_NONZERO && !dimensions_nonzero)) { + // all the dimensions are zero + // mark them as NONZERO to send them all + for(rd = temp_rd?temp_rd:st->dimensions, c = 0 ; rd && c < dimensions_count ; rd = rd->next, c++) { + if(unlikely(r->od[c] & RRDR_DIMENSION_HIDDEN)) continue; + r->od[c] |= RRDR_DIMENSION_NONZERO; + } + } + + rrdr_query_completed(r->internal.db_points_read, r->internal.result_points_generated); + freez(region_info_array); + return r; +} +#endif //#ifdef ENABLE_DBENGINE + +RRDR *rrd2rrdr( + RRDSET *st + , long points_requested + , long long after_requested + , long long before_requested + , RRDR_GROUPING group_method + , long resampling_time_requested + , RRDR_OPTIONS options + , const char *dimensions + , struct context_param *context_param_list +) +{ + int rrd_update_every; + int absolute_period_requested; + + time_t first_entry_t; + time_t last_entry_t; + if (context_param_list) { + first_entry_t = context_param_list->first_entry_t; + last_entry_t = context_param_list->last_entry_t; + } else { + rrdset_rdlock(st); + first_entry_t = rrdset_first_entry_t_nolock(st); + last_entry_t = rrdset_last_entry_t_nolock(st); + rrdset_unlock(st); + } + + rrd_update_every = st->update_every; + absolute_period_requested = rrdr_convert_before_after_to_absolute(&after_requested, &before_requested, + rrd_update_every, first_entry_t, + last_entry_t, options); + if (options & RRDR_OPTION_ALLOW_PAST) + if (first_entry_t > after_requested) + first_entry_t = after_requested; + + if (context_param_list) + rebuild_context_param_list(context_param_list, after_requested); + +#ifdef ENABLE_DBENGINE + if (st->rrd_memory_mode == RRD_MEMORY_MODE_DBENGINE) { + struct rrdeng_region_info *region_info_array; + unsigned regions, max_interval; + + /* This call takes the chart read-lock */ + regions = rrdeng_variable_step_boundaries(st, after_requested, before_requested, + ®ion_info_array, &max_interval, context_param_list); + if (1 == regions) { + if (region_info_array) { + if (rrd_update_every != region_info_array[0].update_every) { + rrd_update_every = region_info_array[0].update_every; + /* recalculate query alignment */ + absolute_period_requested = + rrdr_convert_before_after_to_absolute(&after_requested, &before_requested, rrd_update_every, + first_entry_t, last_entry_t, options); + } + freez(region_info_array); + } + return rrd2rrdr_fixedstep(st, points_requested, after_requested, before_requested, group_method, + resampling_time_requested, options, dimensions, rrd_update_every, + first_entry_t, last_entry_t, absolute_period_requested, context_param_list); + } else { + if (rrd_update_every != (uint16_t)max_interval) { + rrd_update_every = (uint16_t) max_interval; + /* recalculate query alignment */ + absolute_period_requested = rrdr_convert_before_after_to_absolute(&after_requested, &before_requested, + rrd_update_every, first_entry_t, + last_entry_t, options); + } + return rrd2rrdr_variablestep(st, points_requested, after_requested, before_requested, group_method, + resampling_time_requested, options, dimensions, rrd_update_every, + first_entry_t, last_entry_t, absolute_period_requested, region_info_array, context_param_list); + } + } +#endif + return rrd2rrdr_fixedstep(st, points_requested, after_requested, before_requested, group_method, + resampling_time_requested, options, dimensions, + rrd_update_every, first_entry_t, last_entry_t, absolute_period_requested, context_param_list); +}
\ No newline at end of file diff --git a/web/api/queries/query.h b/web/api/queries/query.h new file mode 100644 index 0000000..6b8a51c --- /dev/null +++ b/web/api/queries/query.h @@ -0,0 +1,24 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#ifndef NETDATA_API_DATA_QUERY_H +#define NETDATA_API_DATA_QUERY_H + +typedef enum rrdr_grouping { + RRDR_GROUPING_UNDEFINED = 0, + RRDR_GROUPING_AVERAGE, + RRDR_GROUPING_MIN, + RRDR_GROUPING_MAX, + RRDR_GROUPING_SUM, + RRDR_GROUPING_INCREMENTAL_SUM, + RRDR_GROUPING_MEDIAN, + RRDR_GROUPING_STDDEV, + RRDR_GROUPING_CV, + RRDR_GROUPING_SES, + RRDR_GROUPING_DES, +} RRDR_GROUPING; + +extern const char *group_method2string(RRDR_GROUPING group); +extern void web_client_api_v1_init_grouping(void); +extern RRDR_GROUPING web_client_api_request_v1_data_group(const char *name, RRDR_GROUPING def); + +#endif //NETDATA_API_DATA_QUERY_H diff --git a/web/api/queries/rrdr.c b/web/api/queries/rrdr.c new file mode 100644 index 0000000..ef237fa --- /dev/null +++ b/web/api/queries/rrdr.c @@ -0,0 +1,144 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#include "rrdr.h" + +/* +static void rrdr_dump(RRDR *r) +{ + long c, i; + RRDDIM *d; + + fprintf(stderr, "\nCHART %s (%s)\n", r->st->id, r->st->name); + + for(c = 0, d = r->st->dimensions; d ;c++, d = d->next) { + fprintf(stderr, "DIMENSION %s (%s), %s%s%s%s\n" + , d->id + , d->name + , (r->od[c] & RRDR_EMPTY)?"EMPTY ":"" + , (r->od[c] & RRDR_RESET)?"RESET ":"" + , (r->od[c] & RRDR_DIMENSION_HIDDEN)?"HIDDEN ":"" + , (r->od[c] & RRDR_DIMENSION_NONZERO)?"NONZERO ":"" + ); + } + + if(r->rows <= 0) { + fprintf(stderr, "RRDR does not have any values in it.\n"); + return; + } + + fprintf(stderr, "RRDR includes %d values in it:\n", r->rows); + + // for each line in the array + for(i = 0; i < r->rows ;i++) { + calculated_number *cn = &r->v[ i * r->d ]; + RRDR_DIMENSION_FLAGS *co = &r->o[ i * r->d ]; + + // print the id and the timestamp of the line + fprintf(stderr, "%ld %ld ", i + 1, r->t[i]); + + // for each dimension + for(c = 0, d = r->st->dimensions; d ;c++, d = d->next) { + if(unlikely(r->od[c] & RRDR_DIMENSION_HIDDEN)) continue; + if(unlikely(!(r->od[c] & RRDR_DIMENSION_NONZERO))) continue; + + if(co[c] & RRDR_EMPTY) + fprintf(stderr, "null "); + else + fprintf(stderr, CALCULATED_NUMBER_FORMAT " %s%s%s%s " + , cn[c] + , (co[c] & RRDR_EMPTY)?"E":" " + , (co[c] & RRDR_RESET)?"R":" " + , (co[c] & RRDR_DIMENSION_HIDDEN)?"H":" " + , (co[c] & RRDR_DIMENSION_NONZERO)?"N":" " + ); + } + + fprintf(stderr, "\n"); + } +} +*/ + + + + +inline static void rrdr_lock_rrdset(RRDR *r) { + if(unlikely(!r)) { + error("NULL value given!"); + return; + } + + rrdset_rdlock(r->st); + r->has_st_lock = 1; +} + +inline static void rrdr_unlock_rrdset(RRDR *r) { + if(unlikely(!r)) { + error("NULL value given!"); + return; + } + + if(likely(r->has_st_lock)) { + rrdset_unlock(r->st); + r->has_st_lock = 0; + } +} + +inline void rrdr_free(RRDR *r) +{ + if(unlikely(!r)) { + error("NULL value given!"); + return; + } + + rrdr_unlock_rrdset(r); + freez(r->t); + freez(r->v); + freez(r->o); + freez(r->od); + freez(r); +} + +RRDR *rrdr_create(struct rrdset *st, long n, struct context_param *context_param_list) +{ + if(unlikely(!st)) { + error("NULL value given!"); + return NULL; + } + + RRDR *r = callocz(1, sizeof(RRDR)); + r->st = st; + + rrdr_lock_rrdset(r); + + RRDDIM *temp_rd = context_param_list ? context_param_list->rd : NULL; + RRDDIM *rd; + if (temp_rd) { + RRDDIM *t = temp_rd; + while (t) { + r->d++; + t = t->next; + } + } else + rrddim_foreach_read(rd, st) r->d++; + + r->n = n; + + r->t = callocz((size_t)n, sizeof(time_t)); + r->v = mallocz(n * r->d * sizeof(calculated_number)); + r->o = mallocz(n * r->d * sizeof(RRDR_VALUE_FLAGS)); + r->od = mallocz(r->d * sizeof(RRDR_DIMENSION_FLAGS)); + + // set the hidden flag on hidden dimensions + int c; + for (c = 0, rd = temp_rd ? temp_rd : st->dimensions; rd; c++, rd = rd->next) { + if (unlikely(rrddim_flag_check(rd, RRDDIM_FLAG_HIDDEN))) + r->od[c] = RRDR_DIMENSION_HIDDEN; + else + r->od[c] = RRDR_DIMENSION_DEFAULT; + } + + r->group = 1; + r->update_every = 1; + + return r; +} diff --git a/web/api/queries/rrdr.h b/web/api/queries/rrdr.h new file mode 100644 index 0000000..4d349c3 --- /dev/null +++ b/web/api/queries/rrdr.h @@ -0,0 +1,115 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#ifndef NETDATA_QUERIES_RRDR_H +#define NETDATA_QUERIES_RRDR_H + +#include "libnetdata/libnetdata.h" + +typedef enum rrdr_options { + RRDR_OPTION_NONZERO = 0x00000001, // don't output dimensions with just zero values + RRDR_OPTION_REVERSED = 0x00000002, // output the rows in reverse order (oldest to newest) + RRDR_OPTION_ABSOLUTE = 0x00000004, // values positive, for DATASOURCE_SSV before summing + RRDR_OPTION_MIN2MAX = 0x00000008, // when adding dimensions, use max - min, instead of sum + RRDR_OPTION_SECONDS = 0x00000010, // output seconds, instead of dates + RRDR_OPTION_MILLISECONDS = 0x00000020, // output milliseconds, instead of dates + RRDR_OPTION_NULL2ZERO = 0x00000040, // do not show nulls, convert them to zeros + RRDR_OPTION_OBJECTSROWS = 0x00000080, // each row of values should be an object, not an array + RRDR_OPTION_GOOGLE_JSON = 0x00000100, // comply with google JSON/JSONP specs + RRDR_OPTION_JSON_WRAP = 0x00000200, // wrap the response in a JSON header with info about the result + RRDR_OPTION_LABEL_QUOTES = 0x00000400, // in CSV output, wrap header labels in double quotes + RRDR_OPTION_PERCENTAGE = 0x00000800, // give values as percentage of total + RRDR_OPTION_NOT_ALIGNED = 0x00001000, // do not align charts for persistent timeframes + RRDR_OPTION_DISPLAY_ABS = 0x00002000, // for badges, display the absolute value, but calculate colors with sign + RRDR_OPTION_MATCH_IDS = 0x00004000, // when filtering dimensions, match only IDs + RRDR_OPTION_MATCH_NAMES = 0x00008000, // when filtering dimensions, match only names + RRDR_OPTION_CUSTOM_VARS = 0x00010000, // when wraping response in a JSON, return custom variables in response + RRDR_OPTION_ALLOW_PAST = 0x00020000, // The after parameter can extend in the past before the first entry +} RRDR_OPTIONS; + +typedef enum rrdr_value_flag { + RRDR_VALUE_NOTHING = 0x00, // no flag set (a good default) + RRDR_VALUE_EMPTY = 0x01, // the database value is empty + RRDR_VALUE_RESET = 0x02, // the database value is marked as reset (overflown) +} RRDR_VALUE_FLAGS; + +typedef enum rrdr_dimension_flag { + RRDR_DIMENSION_DEFAULT = 0x00, + RRDR_DIMENSION_HIDDEN = 0x04, // the dimension is hidden (not to be presented to callers) + RRDR_DIMENSION_NONZERO = 0x08, // the dimension is non zero (contains non-zero values) + RRDR_DIMENSION_SELECTED = 0x10, // the dimension is selected for evaluation in this RRDR +} RRDR_DIMENSION_FLAGS; + +// RRDR result options +typedef enum rrdr_result_flags { + RRDR_RESULT_OPTION_ABSOLUTE = 0x00000001, // the query uses absolute time-frames + // (can be cached by browsers and proxies) + RRDR_RESULT_OPTION_RELATIVE = 0x00000002, // the query uses relative time-frames + // (should not to be cached by browsers and proxies) + RRDR_RESULT_OPTION_VARIABLE_STEP = 0x00000004, // the query uses variable-step time-frames +} RRDR_RESULT_FLAGS; + +typedef struct rrdresult { + struct rrdset *st; // the chart this result refers to + + RRDR_RESULT_FLAGS result_options; // RRDR_RESULT_OPTION_* + + int d; // the number of dimensions + long n; // the number of values in the arrays + long rows; // the number of rows used + + RRDR_DIMENSION_FLAGS *od; // the options for the dimensions + + time_t *t; // array of n timestamps + calculated_number *v; // array n x d values + RRDR_VALUE_FLAGS *o; // array n x d options for each value returned + + long group; // how many collected values were grouped for each row + int update_every; // what is the suggested update frequency in seconds + + calculated_number min; + calculated_number max; + + time_t before; + time_t after; + + int has_st_lock; // if st is read locked by us + + // internal rrd2rrdr() members below this point + struct { + long points_wanted; + long resampling_group; + calculated_number resampling_divisor; + + void *(*grouping_create)(struct rrdresult *r); + void (*grouping_reset)(struct rrdresult *r); + void (*grouping_free)(struct rrdresult *r); + void (*grouping_add)(struct rrdresult *r, calculated_number value); + calculated_number (*grouping_flush)(struct rrdresult *r, RRDR_VALUE_FLAGS *rrdr_value_options_ptr); + void *grouping_data; + + #ifdef NETDATA_INTERNAL_CHECKS + const char *log; + #endif + + size_t db_points_read; + size_t result_points_generated; + } internal; +} RRDR; + +#define rrdr_rows(r) ((r)->rows) + +#include "../../../database/rrd.h" +extern void rrdr_free(RRDR *r); +extern RRDR *rrdr_create(struct rrdset *st, long n, struct context_param *context_param_list); + +#include "../web_api_v1.h" +#include "web/api/queries/query.h" + +extern RRDR *rrd2rrdr( + RRDSET *st, long points_requested, long long after_requested, long long before_requested, + RRDR_GROUPING group_method, long resampling_time_requested, RRDR_OPTIONS options, const char *dimensions, + struct context_param *context_param_list); + +#include "query.h" + +#endif //NETDATA_QUERIES_RRDR_H diff --git a/web/api/queries/ses/Makefile.am b/web/api/queries/ses/Makefile.am new file mode 100644 index 0000000..161784b --- /dev/null +++ b/web/api/queries/ses/Makefile.am @@ -0,0 +1,8 @@ +# SPDX-License-Identifier: GPL-3.0-or-later + +AUTOMAKE_OPTIONS = subdir-objects +MAINTAINERCLEANFILES = $(srcdir)/Makefile.in + +dist_noinst_DATA = \ + README.md \ + $(NULL) diff --git a/web/api/queries/ses/README.md b/web/api/queries/ses/README.md new file mode 100644 index 0000000..c279701 --- /dev/null +++ b/web/api/queries/ses/README.md @@ -0,0 +1,61 @@ +<!-- +title: "Single (or Simple) Exponential Smoothing (`ses`)" +custom_edit_url: https://github.com/netdata/netdata/edit/master/web/api/queries/ses/README.md +--> + +# Single (or Simple) Exponential Smoothing (`ses`) + +> This query is also available as `ema` and `ewma`. + +An exponential moving average (`ema`), also known as an exponentially weighted moving average (`ewma`) +is a first-order infinite impulse response filter that applies weighting factors which decrease +exponentially. The weighting for each older datum decreases exponentially, never reaching zero. + +In simple terms, this is like an average value, but more recent values are given more weight. + +Netdata automatically adjusts the weight (`alpha`) based on the number of values processed, +using the formula: + +``` +window = max(number of values, 15) +alpha = 2 / (window + 1) +``` + +You can change the fixed value `15` by setting in `netdata.conf`: + +``` +[web] + ses max window = 15 +``` + +## how to use + +Use it in alarms like this: + +``` + alarm: my_alarm + on: my_chart +lookup: ses -1m unaligned of my_dimension + warn: $this > 1000 +``` + +`ses` does not change the units. For example, if the chart units is `requests/sec`, the exponential +moving average will be again expressed in the same units. + +It can also be used in APIs and badges as `&group=ses` in the URL. + +## Examples + +Examining last 1 minute `successful` web server responses: + +-  +-  +-  +-  + +## References + +- <https://en.wikipedia.org/wiki/Moving_average#exponential-moving-average> +- <https://en.wikipedia.org/wiki/Exponential_smoothing>. + +[](<>) diff --git a/web/api/queries/ses/ses.c b/web/api/queries/ses/ses.c new file mode 100644 index 0000000..772505f --- /dev/null +++ b/web/api/queries/ses/ses.c @@ -0,0 +1,92 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#include "ses.h" + + +// ---------------------------------------------------------------------------- +// single exponential smoothing + +struct grouping_ses { + calculated_number alpha; + calculated_number alpha_other; + calculated_number level; + size_t count; +}; + +static size_t max_window_size = 15; + +void grouping_init_ses(void) { + long long ret = config_get_number(CONFIG_SECTION_WEB, "ses max window", (long long)max_window_size); + if(ret <= 1) { + config_set_number(CONFIG_SECTION_WEB, "ses max window", (long long)max_window_size); + } + else { + max_window_size = (size_t) ret; + } +} + +static inline calculated_number window(RRDR *r, struct grouping_ses *g) { + (void)g; + + calculated_number points; + if(r->group == 1) { + // provide a running DES + points = r->internal.points_wanted; + } + else { + // provide a SES with flush points + points = r->group; + } + + return (points > max_window_size) ? max_window_size : points; +} + +static inline void set_alpha(RRDR *r, struct grouping_ses *g) { + // https://en.wikipedia.org/wiki/Moving_average#Exponential_moving_average + // A commonly used value for alpha is 2 / (N + 1) + g->alpha = 2.0 / (window(r, g) + 1.0); + g->alpha_other = 1.0 - g->alpha; +} + +void *grouping_create_ses(RRDR *r) { + struct grouping_ses *g = (struct grouping_ses *)callocz(1, sizeof(struct grouping_ses)); + set_alpha(r, g); + g->level = 0.0; + return g; +} + +// resets when switches dimensions +// so, clear everything to restart +void grouping_reset_ses(RRDR *r) { + struct grouping_ses *g = (struct grouping_ses *)r->internal.grouping_data; + g->level = 0.0; + g->count = 0; +} + +void grouping_free_ses(RRDR *r) { + freez(r->internal.grouping_data); + r->internal.grouping_data = NULL; +} + +void grouping_add_ses(RRDR *r, calculated_number value) { + struct grouping_ses *g = (struct grouping_ses *)r->internal.grouping_data; + + if(calculated_number_isnumber(value)) { + if(unlikely(!g->count)) + g->level = value; + + g->level = g->alpha * value + g->alpha_other * g->level; + g->count++; + } +} + +calculated_number grouping_flush_ses(RRDR *r, RRDR_VALUE_FLAGS *rrdr_value_options_ptr) { + struct grouping_ses *g = (struct grouping_ses *)r->internal.grouping_data; + + if(unlikely(!g->count || !calculated_number_isnumber(g->level))) { + *rrdr_value_options_ptr |= RRDR_VALUE_EMPTY; + return 0.0; + } + + return g->level; +} diff --git a/web/api/queries/ses/ses.h b/web/api/queries/ses/ses.h new file mode 100644 index 0000000..603fdb5 --- /dev/null +++ b/web/api/queries/ses/ses.h @@ -0,0 +1,17 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#ifndef NETDATA_API_QUERIES_SES_H +#define NETDATA_API_QUERIES_SES_H + +#include "../query.h" +#include "../rrdr.h" + +extern void grouping_init_ses(void); + +extern void *grouping_create_ses(RRDR *r); +extern void grouping_reset_ses(RRDR *r); +extern void grouping_free_ses(RRDR *r); +extern void grouping_add_ses(RRDR *r, calculated_number value); +extern calculated_number grouping_flush_ses(RRDR *r, RRDR_VALUE_FLAGS *rrdr_value_options_ptr); + +#endif //NETDATA_API_QUERIES_SES_H diff --git a/web/api/queries/stddev/Makefile.am b/web/api/queries/stddev/Makefile.am new file mode 100644 index 0000000..161784b --- /dev/null +++ b/web/api/queries/stddev/Makefile.am @@ -0,0 +1,8 @@ +# SPDX-License-Identifier: GPL-3.0-or-later + +AUTOMAKE_OPTIONS = subdir-objects +MAINTAINERCLEANFILES = $(srcdir)/Makefile.in + +dist_noinst_DATA = \ + README.md \ + $(NULL) diff --git a/web/api/queries/stddev/README.md b/web/api/queries/stddev/README.md new file mode 100644 index 0000000..7cd7d62 --- /dev/null +++ b/web/api/queries/stddev/README.md @@ -0,0 +1,93 @@ +<!-- +title: "standard deviation (`stddev`)" +custom_edit_url: https://github.com/netdata/netdata/edit/master/web/api/queries/stddev/README.md +--> + +# standard deviation (`stddev`) + +The standard deviation is a measure that is used to quantify the amount of variation or dispersion +of a set of data values. + +A low standard deviation indicates that the data points tend to be close to the mean (also called the +expected value) of the set, while a high standard deviation indicates that the data points are spread +out over a wider range of values. + +## how to use + +Use it in alarms like this: + +``` + alarm: my_alarm + on: my_chart +lookup: stddev -1m unaligned of my_dimension + warn: $this > 1000 +``` + +`stdev` does not change the units. For example, if the chart units is `requests/sec`, the standard +deviation will be again expressed in the same units. + +It can also be used in APIs and badges as `&group=stddev` in the URL. + +## Examples + +Examining last 1 minute `successful` web server responses: + +-  +-  +-  +-  + +## References + +Check <https://en.wikipedia.org/wiki/Standard_deviation>. + +--- + +# Coefficient of variation (`cv`) + +> This query is also available as `rsd`. + +The coefficient of variation (`cv`), also known as relative standard deviation (`rsd`), +is a standardized measure of dispersion of a probability distribution or frequency distribution. + +It is defined as the ratio of the **standard deviation** to the **mean**. + +In simple terms, it gives the percentage of change. So, if the average value of a metric is 1000 +and its standard deviation is 100 (meaning that it variates from 900 to 1100), then `cv` is 10%. + +This is an easy way to check the % variation, without using absolute values. + +For example, you may trigger an alarm if your web server requests/sec `cv` is above 20 (`%`) +over the last minute. So if your web server was serving 1000 reqs/sec over the last minute, +it will trigger the alarm if had spikes below 800/sec or above 1200/sec. + +## how to use + +Use it in alarms like this: + +``` + alarm: my_alarm + on: my_chart +lookup: cv -1m unaligned of my_dimension + units: % + warn: $this > 20 +``` + +The units reported by `cv` is always `%`. + +It can also be used in APIs and badges as `&group=cv` in the URL. + +## Examples + +Examining last 1 minute `successful` web server responses: + +-  +-  +-  +-  + +## References + +Check <https://en.wikipedia.org/wiki/Coefficient_of_variation>. + +[](<>) diff --git a/web/api/queries/stddev/stddev.c b/web/api/queries/stddev/stddev.c new file mode 100644 index 0000000..1625844 --- /dev/null +++ b/web/api/queries/stddev/stddev.c @@ -0,0 +1,176 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#include "stddev.h" + + +// ---------------------------------------------------------------------------- +// stddev + +// this implementation comes from: +// https://www.johndcook.com/blog/standard_deviation/ + +struct grouping_stddev { + long count; + calculated_number m_oldM, m_newM, m_oldS, m_newS; +}; + +void *grouping_create_stddev(RRDR *r) { + UNUSED (r); + return callocz(1, sizeof(struct grouping_stddev)); +} + +// resets when switches dimensions +// so, clear everything to restart +void grouping_reset_stddev(RRDR *r) { + struct grouping_stddev *g = (struct grouping_stddev *)r->internal.grouping_data; + g->count = 0; +} + +void grouping_free_stddev(RRDR *r) { + freez(r->internal.grouping_data); + r->internal.grouping_data = NULL; +} + +void grouping_add_stddev(RRDR *r, calculated_number value) { + struct grouping_stddev *g = (struct grouping_stddev *)r->internal.grouping_data; + + if(calculated_number_isnumber(value)) { + g->count++; + + // See Knuth TAOCP vol 2, 3rd edition, page 232 + if (g->count == 1) { + g->m_oldM = g->m_newM = value; + g->m_oldS = 0.0; + } + else { + g->m_newM = g->m_oldM + (value - g->m_oldM) / g->count; + g->m_newS = g->m_oldS + (value - g->m_oldM) * (value - g->m_newM); + + // set up for next iteration + g->m_oldM = g->m_newM; + g->m_oldS = g->m_newS; + } + } +} + +static inline calculated_number mean(struct grouping_stddev *g) { + return (g->count > 0) ? g->m_newM : 0.0; +} + +static inline calculated_number variance(struct grouping_stddev *g) { + return ( (g->count > 1) ? g->m_newS/(g->count - 1) : 0.0 ); +} +static inline calculated_number stddev(struct grouping_stddev *g) { + return sqrtl(variance(g)); +} + +calculated_number grouping_flush_stddev(RRDR *r, RRDR_VALUE_FLAGS *rrdr_value_options_ptr) { + struct grouping_stddev *g = (struct grouping_stddev *)r->internal.grouping_data; + + calculated_number value; + + if(likely(g->count > 1)) { + value = stddev(g); + + if(!calculated_number_isnumber(value)) { + value = 0.0; + *rrdr_value_options_ptr |= RRDR_VALUE_EMPTY; + } + } + else if(g->count == 1) { + value = 0.0; + } + else { + value = 0.0; + *rrdr_value_options_ptr |= RRDR_VALUE_EMPTY; + } + + grouping_reset_stddev(r); + + return value; +} + +// https://en.wikipedia.org/wiki/Coefficient_of_variation +calculated_number grouping_flush_coefficient_of_variation(RRDR *r, RRDR_VALUE_FLAGS *rrdr_value_options_ptr) { + struct grouping_stddev *g = (struct grouping_stddev *)r->internal.grouping_data; + + calculated_number value; + + if(likely(g->count > 1)) { + calculated_number m = mean(g); + value = 100.0 * stddev(g) / ((m < 0)? -m : m); + + if(unlikely(!calculated_number_isnumber(value))) { + value = 0.0; + *rrdr_value_options_ptr |= RRDR_VALUE_EMPTY; + } + } + else if(g->count == 1) { + // one value collected + value = 0.0; + } + else { + // no values collected + value = 0.0; + *rrdr_value_options_ptr |= RRDR_VALUE_EMPTY; + } + + grouping_reset_stddev(r); + + return value; +} + + +/* + * Mean = average + * +calculated_number grouping_flush_mean(RRDR *r, RRDR_VALUE_FLAGS *rrdr_value_options_ptr) { + struct grouping_stddev *g = (struct grouping_stddev *)r->internal.grouping_data; + + calculated_number value; + + if(unlikely(!g->count)) { + value = 0.0; + *rrdr_value_options_ptr |= RRDR_VALUE_EMPTY; + } + else { + value = mean(g); + + if(!isnormal(value)) { + value = 0.0; + *rrdr_value_options_ptr |= RRDR_VALUE_EMPTY; + } + } + + grouping_reset_stddev(r); + + return value; +} + */ + +/* + * It is not advised to use this version of variance directly + * +calculated_number grouping_flush_variance(RRDR *r, RRDR_VALUE_FLAGS *rrdr_value_options_ptr) { + struct grouping_stddev *g = (struct grouping_stddev *)r->internal.grouping_data; + + calculated_number value; + + if(unlikely(!g->count)) { + value = 0.0; + *rrdr_value_options_ptr |= RRDR_VALUE_EMPTY; + } + else { + value = variance(g); + + if(!isnormal(value)) { + value = 0.0; + *rrdr_value_options_ptr |= RRDR_VALUE_EMPTY; + } + } + + grouping_reset_stddev(r); + + return value; +} +*/
\ No newline at end of file diff --git a/web/api/queries/stddev/stddev.h b/web/api/queries/stddev/stddev.h new file mode 100644 index 0000000..7a46975 --- /dev/null +++ b/web/api/queries/stddev/stddev.h @@ -0,0 +1,18 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#ifndef NETDATA_API_QUERIES_STDDEV_H +#define NETDATA_API_QUERIES_STDDEV_H + +#include "../query.h" +#include "../rrdr.h" + +extern void *grouping_create_stddev(RRDR *r); +extern void grouping_reset_stddev(RRDR *r); +extern void grouping_free_stddev(RRDR *r); +extern void grouping_add_stddev(RRDR *r, calculated_number value); +extern calculated_number grouping_flush_stddev(RRDR *r, RRDR_VALUE_FLAGS *rrdr_value_options_ptr); +extern calculated_number grouping_flush_coefficient_of_variation(RRDR *r, RRDR_VALUE_FLAGS *rrdr_value_options_ptr); +// extern calculated_number grouping_flush_mean(RRDR *r, RRDR_VALUE_FLAGS *rrdr_value_options_ptr); +// extern calculated_number grouping_flush_variance(RRDR *r, RRDR_VALUE_FLAGS *rrdr_value_options_ptr); + +#endif //NETDATA_API_QUERIES_STDDEV_H diff --git a/web/api/queries/sum/Makefile.am b/web/api/queries/sum/Makefile.am new file mode 100644 index 0000000..161784b --- /dev/null +++ b/web/api/queries/sum/Makefile.am @@ -0,0 +1,8 @@ +# SPDX-License-Identifier: GPL-3.0-or-later + +AUTOMAKE_OPTIONS = subdir-objects +MAINTAINERCLEANFILES = $(srcdir)/Makefile.in + +dist_noinst_DATA = \ + README.md \ + $(NULL) diff --git a/web/api/queries/sum/README.md b/web/api/queries/sum/README.md new file mode 100644 index 0000000..aeace0a --- /dev/null +++ b/web/api/queries/sum/README.md @@ -0,0 +1,41 @@ +<!-- +title: "Sum" +custom_edit_url: https://github.com/netdata/netdata/edit/master/web/api/queries/sum/README.md +--> + +# Sum + +This module sums all the values in the time-frame requested. + +You can use `sum` to find the volume of something over a period. + +## how to use + +Use it in alarms like this: + +``` + alarm: my_alarm + on: my_chart +lookup: sum -1m unaligned of my_dimension + warn: $this > 1000 +``` + +`sum` does not change the units. For example, if the chart units is `requests/sec`, the result +will be again expressed in the same units. + +It can also be used in APIs and badges as `&group=sum` in the URL. + +## Examples + +Examining last 1 minute `successful` web server responses: + +-  +-  +-  +-  + +## References + +- <https://en.wikipedia.org/wiki/Summation>. + +[](<>) diff --git a/web/api/queries/sum/sum.c b/web/api/queries/sum/sum.c new file mode 100644 index 0000000..0da9937 --- /dev/null +++ b/web/api/queries/sum/sum.c @@ -0,0 +1,58 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#include "sum.h" + +// ---------------------------------------------------------------------------- +// sum + +struct grouping_sum { + calculated_number sum; + size_t count; +}; + +void *grouping_create_sum(RRDR *r) { + (void)r; + return callocz(1, sizeof(struct grouping_sum)); +} + +// resets when switches dimensions +// so, clear everything to restart +void grouping_reset_sum(RRDR *r) { + struct grouping_sum *g = (struct grouping_sum *)r->internal.grouping_data; + g->sum = 0; + g->count = 0; +} + +void grouping_free_sum(RRDR *r) { + freez(r->internal.grouping_data); + r->internal.grouping_data = NULL; +} + +void grouping_add_sum(RRDR *r, calculated_number value) { + if(!isnan(value)) { + struct grouping_sum *g = (struct grouping_sum *)r->internal.grouping_data; + g->sum += value; + g->count++; + } +} + +calculated_number grouping_flush_sum(RRDR *r, RRDR_VALUE_FLAGS *rrdr_value_options_ptr) { + struct grouping_sum *g = (struct grouping_sum *)r->internal.grouping_data; + + calculated_number value; + + if(unlikely(!g->count)) { + value = 0.0; + *rrdr_value_options_ptr |= RRDR_VALUE_EMPTY; + } + else { + value = g->sum; + } + + g->sum = 0.0; + g->count = 0; + + return value; +} + + diff --git a/web/api/queries/sum/sum.h b/web/api/queries/sum/sum.h new file mode 100644 index 0000000..9dc8d20 --- /dev/null +++ b/web/api/queries/sum/sum.h @@ -0,0 +1,15 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#ifndef NETDATA_API_QUERY_SUM_H +#define NETDATA_API_QUERY_SUM_H + +#include "../query.h" +#include "../rrdr.h" + +extern void *grouping_create_sum(RRDR *r); +extern void grouping_reset_sum(RRDR *r); +extern void grouping_free_sum(RRDR *r); +extern void grouping_add_sum(RRDR *r, calculated_number value); +extern calculated_number grouping_flush_sum(RRDR *r, RRDR_VALUE_FLAGS *rrdr_value_options_ptr); + +#endif //NETDATA_API_QUERY_SUM_H |