Merging upstream version 1.34.0.

Signed-off-by: Daniel Baumann <daniel.baumann@progress-linux.org>
author: Daniel Baumann <daniel.baumann@progress-linux.org> 2022-04-14 18:12:14 +0000
committer: Daniel Baumann <daniel.baumann@progress-linux.org> 2022-04-14 18:12:14 +0000
commit: bb50acdcb8073654ea667b8c0272e335bd43f844 (patch)
tree: 1e00c8a29871426f8182658928dcb62e42d57ce8 /web/api/queries
parent: Releasing debian version 1.33.1-1. (diff)
download: netdata-bb50acdcb8073654ea667b8c0272e335bd43f844.tar.xz
netdata-bb50acdcb8073654ea667b8c0272e335bd43f844.zip
12 files changed, 52 insertions, 20 deletions
diff --git a/web/api/queries/README.md b/web/api/queries/README.md
index 31ec49655..44cdd05b4 100644
--- a/web/api/queries/README.md
+++ b/web/api/queries/README.md
@@ -173,4 +173,4 @@ So, the proper way to query the database is to also set at least `after`. The fo
 
 When you keep calling this URL, you will see that it returns one new value every 10 seconds, and the timestamp always ends with zero. Similarly, if you say `points=1&after=-5` it will always return timestamps ending with 0 or 5.
 
-[![analytics](https://www.google-analytics.com/collect?v=1&aip=1&t=pageview&_s=1&ds=github&dr=https%3A%2F%2Fgithub.com%2Fnetdata%2Fnetdata&dl=https%3A%2F%2Fmy-netdata.io%2Fgithub%2Fweb%2Fapi%2Fqueries%2FREADME&_u=MAC~&cid=5792dfd7-8dc4-476b-af31-da2fdb9f93d2&tid=UA-64295674-3)](<>)
+
diff --git a/web/api/queries/average/README.md b/web/api/queries/average/README.md
index f32a67571..b8d4ba7e7 100644
--- a/web/api/queries/average/README.md
+++ b/web/api/queries/average/README.md
@@ -43,4 +43,4 @@ Examining last 1 minute `successful` web server responses:
 
 -   <https://en.wikipedia.org/wiki/Average>.
 
-[![analytics](https://www.google-analytics.com/collect?v=1&aip=1&t=pageview&_s=1&ds=github&dr=https%3A%2F%2Fgithub.com%2Fnetdata%2Fnetdata&dl=https%3A%2F%2Fmy-netdata.io%2Fgithub%2Fweb%2Fapi%2Fqueries%2Faverage%2FREADME&_u=MAC~&cid=5792dfd7-8dc4-476b-af31-da2fdb9f93d2&tid=UA-64295674-3)](<>)
+
diff --git a/web/api/queries/des/README.md b/web/api/queries/des/README.md
index 5505de5a6..33c5f1a0c 100644
--- a/web/api/queries/des/README.md
+++ b/web/api/queries/des/README.md
@@ -70,4 +70,4 @@ Examining last 1 minute `successful` web server responses:
 
 -   <https://en.wikipedia.org/wiki/Exponential_smoothing>.
 
-[![analytics](https://www.google-analytics.com/collect?v=1&aip=1&t=pageview&_s=1&ds=github&dr=https%3A%2F%2Fgithub.com%2Fnetdata%2Fnetdata&dl=https%3A%2F%2Fmy-netdata.io%2Fgithub%2Fweb%2Fapi%2Fqueries%2Fdes%2FREADME&_u=MAC~&cid=5792dfd7-8dc4-476b-af31-da2fdb9f93d2&tid=UA-64295674-3)](<>)
+
diff --git a/web/api/queries/incremental_sum/README.md b/web/api/queries/incremental_sum/README.md
index e5f3dfc0c..44301172e 100644
--- a/web/api/queries/incremental_sum/README.md
+++ b/web/api/queries/incremental_sum/README.md
@@ -38,4 +38,4 @@ Examining last 1 minute `successful` web server responses:
 
 -   none
 
-[![analytics](https://www.google-analytics.com/collect?v=1&aip=1&t=pageview&_s=1&ds=github&dr=https%3A%2F%2Fgithub.com%2Fnetdata%2Fnetdata&dl=https%3A%2F%2Fmy-netdata.io%2Fgithub%2Fweb%2Fapi%2Fqueries%2Fincremental_sum%2FREADME&_u=MAC~&cid=5792dfd7-8dc4-476b-af31-da2fdb9f93d2&tid=UA-64295674-3)](<>)
+
diff --git a/web/api/queries/max/README.md b/web/api/queries/max/README.md
index 32b1d434c..48da7cf08 100644
--- a/web/api/queries/max/README.md
+++ b/web/api/queries/max/README.md
@@ -35,4 +35,4 @@ Examining last 1 minute `successful` web server responses:
 
 -   <https://en.wikipedia.org/wiki/Sample_maximum_and_minimum>.
 
-[![analytics](https://www.google-analytics.com/collect?v=1&aip=1&t=pageview&_s=1&ds=github&dr=https%3A%2F%2Fgithub.com%2Fnetdata%2Fnetdata&dl=https%3A%2F%2Fmy-netdata.io%2Fgithub%2Fweb%2Fapi%2Fqueries%2Fmax%2FREADME&_u=MAC~&cid=5792dfd7-8dc4-476b-af31-da2fdb9f93d2&tid=UA-64295674-3)](<>)
+
diff --git a/web/api/queries/median/README.md b/web/api/queries/median/README.md
index 25ce8b8d6..bb7d4c66b 100644
--- a/web/api/queries/median/README.md
+++ b/web/api/queries/median/README.md
@@ -42,4 +42,4 @@ Examining last 1 minute `successful` web server responses:
 
 -   <https://en.wikipedia.org/wiki/Median>.
 
-[![analytics](https://www.google-analytics.com/collect?v=1&aip=1&t=pageview&_s=1&ds=github&dr=https%3A%2F%2Fgithub.com%2Fnetdata%2Fnetdata&dl=https%3A%2F%2Fmy-netdata.io%2Fgithub%2Fweb%2Fapi%2Fqueries%2Fmedian%2FREADME&_u=MAC~&cid=5792dfd7-8dc4-476b-af31-da2fdb9f93d2&tid=UA-64295674-3)](<>)
+
diff --git a/web/api/queries/min/README.md b/web/api/queries/min/README.md
index 69ef4ea12..495523c04 100644
--- a/web/api/queries/min/README.md
+++ b/web/api/queries/min/README.md
@@ -35,4 +35,4 @@ Examining last 1 minute `successful` web server responses:
 
 -   <https://en.wikipedia.org/wiki/Sample_maximum_and_minimum>.
 
-[![analytics](https://www.google-analytics.com/collect?v=1&aip=1&t=pageview&_s=1&ds=github&dr=https%3A%2F%2Fgithub.com%2Fnetdata%2Fnetdata&dl=https%3A%2F%2Fmy-netdata.io%2Fgithub%2Fweb%2Fapi%2Fqueries%2Fmin%2FREADME&_u=MAC~&cid=5792dfd7-8dc4-476b-af31-da2fdb9f93d2&tid=UA-64295674-3)](<>)
+
diff --git a/web/api/queries/query.c b/web/api/queries/query.c
index 216417ae8..c55a97060 100644
--- a/web/api/queries/query.c
+++ b/web/api/queries/query.c
@@ -347,11 +347,11 @@ static void rrdr_disable_not_selected_dimensions(RRDR *r, RRDR_OPTIONS options,
 // ----------------------------------------------------------------------------
 // helpers to find our way in RRDR
 
-static inline RRDR_VALUE_FLAGS *rrdr_line_options(RRDR *r, long rrdr_line) {
+static inline RRDR_VALUE_FLAGS *UNUSED_FUNCTION(rrdr_line_options)(RRDR *r, long rrdr_line) {
     return &r->o[ rrdr_line * r->d ];
 }
 
-static inline calculated_number *rrdr_line_values(RRDR *r, long rrdr_line) {
+static inline calculated_number *UNUSED_FUNCTION(rrdr_line_values)(RRDR *r, long rrdr_line) {
     return &r->v[ rrdr_line * r->d ];
 }
 
@@ -562,7 +562,7 @@ static inline void do_dimension_fixedstep(
     calculated_number min = r->min, max = r->max;
     size_t db_points_read = 0;
     time_t db_now = now;
-
+    time_t first_time_t = rrddim_first_entry_t(rd);
     for(rd->state->query_ops.init(rd, &handle, now, before_wanted) ; points_added < points_wanted ; now += dt) {
         // make sure we return data in the proper time range
         if(unlikely(now > before_wanted)) {
@@ -586,7 +586,11 @@ static inline void do_dimension_fixedstep(
         }
 #endif
         db_now = now; // this is needed to set db_now in case the next_metric implementation does not set it
-        storage_number n = rd->state->query_ops.next_metric(&handle, &db_now);
+        storage_number n;
+        if (rd->rrd_memory_mode != RRD_MEMORY_MODE_DBENGINE && now <= first_time_t)
+            n = SN_EMPTY_SLOT;
+        else
+            n = rd->state->query_ops.next_metric(&handle, &db_now);
         if(unlikely(db_now > before_wanted)) {
 #ifdef NETDATA_INTERNAL_CHECKS
             r->internal.log = "stopped, because attempted to access the db after 'wanted before'";
@@ -840,6 +844,7 @@ static RRDR *rrd2rrdr_fixedstep(
         , time_t last_entry_t
         , int absolute_period_requested
         , struct context_param *context_param_list
+        , int timeout
 ) {
     int aligned = !(options & RRDR_OPTION_NOT_ALIGNED);
 
@@ -1093,6 +1098,10 @@ static RRDR *rrd2rrdr_fixedstep(
 
     RRDDIM *rd;
     long c, dimensions_used = 0, dimensions_nonzero = 0;
+    struct timeval query_start_time;
+    struct timeval query_current_time;
+    if (timeout)
+        now_realtime_timeval(&query_start_time);
     for(rd = temp_rd?temp_rd:st->dimensions, c = 0 ; rd && c < dimensions_count ; rd = rd->next, c++) {
 
         // if we need a percentage, we need to calculate all dimensions
@@ -1114,6 +1123,8 @@ static RRDR *rrd2rrdr_fixedstep(
                 , before_wanted
                 , options
                 );
+        if (timeout)
+            now_realtime_timeval(&query_current_time);
 
         if(r->od[c] & RRDR_DIMENSION_NONZERO)
             dimensions_nonzero++;
@@ -1151,6 +1162,12 @@ static RRDR *rrd2rrdr_fixedstep(
         }
 
         dimensions_used++;
+        if (timeout && (dt_usec(&query_start_time, &query_current_time) / 1000.0) > timeout) {
+            log_access("QUERY CANCELED RUNTIME EXCEEDED %0.2f ms (LIMIT %d ms)",
+                       dt_usec(&query_start_time, &query_current_time) / 1000.0, timeout);
+            r->result_options |= RRDR_RESULT_OPTION_CANCEL;
+            break;
+        }
     }
 
     #ifdef NETDATA_INTERNAL_CHECKS
@@ -1184,7 +1201,7 @@ static RRDR *rrd2rrdr_fixedstep(
     r->internal.grouping_free(r);
 
     // when all the dimensions are zero, we should return all of them
-    if(unlikely(options & RRDR_OPTION_NONZERO && !dimensions_nonzero)) {
+    if(unlikely(options & RRDR_OPTION_NONZERO && !dimensions_nonzero && !(r->result_options & RRDR_RESULT_OPTION_CANCEL))) {
         // all the dimensions are zero
         // mark them as NONZERO to send them all
         for(rd = temp_rd?temp_rd:st->dimensions, c = 0 ; rd && c < dimensions_count ; rd = rd->next, c++) {
@@ -1213,6 +1230,7 @@ static RRDR *rrd2rrdr_variablestep(
         , int absolute_period_requested
         , struct rrdeng_region_info *region_info_array
         , struct context_param *context_param_list
+        , int timeout
 ) {
     int aligned = !(options & RRDR_OPTION_NOT_ALIGNED);
 
@@ -1470,6 +1488,10 @@ static RRDR *rrd2rrdr_variablestep(
 
     RRDDIM *rd;
     long c, dimensions_used = 0, dimensions_nonzero = 0;
+    struct timeval query_start_time;
+    struct timeval query_current_time;
+    if (timeout)
+        now_realtime_timeval(&query_start_time);
     for(rd = temp_rd?temp_rd:st->dimensions, c = 0 ; rd && c < dimensions_count ; rd = rd->next, c++) {
 
         // if we need a percentage, we need to calculate all dimensions
@@ -1491,6 +1513,8 @@ static RRDR *rrd2rrdr_variablestep(
                 , before_wanted
                 , options
         );
+        if (timeout)
+            now_realtime_timeval(&query_current_time);
 
         if(r->od[c] & RRDR_DIMENSION_NONZERO)
             dimensions_nonzero++;
@@ -1528,6 +1552,12 @@ static RRDR *rrd2rrdr_variablestep(
         }
 
         dimensions_used++;
+        if (timeout && (dt_usec(&query_start_time, &query_current_time) / 1000.0) > timeout) {
+            log_access("QUERY CANCELED RUNTIME EXCEEDED %0.2f ms (LIMIT %d ms)",
+                       dt_usec(&query_start_time, &query_current_time) / 1000.0, timeout);
+            r->result_options |= RRDR_RESULT_OPTION_CANCEL;
+            break;
+        }
     }
 
     #ifdef NETDATA_INTERNAL_CHECKS
@@ -1562,7 +1592,7 @@ static RRDR *rrd2rrdr_variablestep(
     r->internal.grouping_free(r);
 
     // when all the dimensions are zero, we should return all of them
-    if(unlikely(options & RRDR_OPTION_NONZERO && !dimensions_nonzero)) {
+    if(unlikely(options & RRDR_OPTION_NONZERO && !dimensions_nonzero && !(r->result_options & RRDR_RESULT_OPTION_CANCEL))) {
         // all the dimensions are zero
         // mark them as NONZERO to send them all
         for(rd = temp_rd?temp_rd:st->dimensions, c = 0 ; rd && c < dimensions_count ; rd = rd->next, c++) {
@@ -1587,6 +1617,7 @@ RRDR *rrd2rrdr(
         , RRDR_OPTIONS options
         , const char *dimensions
         , struct context_param *context_param_list
+        , int timeout
 )
 {
     int rrd_update_every;
@@ -1640,7 +1671,7 @@ RRDR *rrd2rrdr(
             }
             return rrd2rrdr_fixedstep(st, points_requested, after_requested, before_requested, group_method,
                                       resampling_time_requested, options, dimensions, rrd_update_every,
-                                      first_entry_t, last_entry_t, absolute_period_requested, context_param_list);
+                                      first_entry_t, last_entry_t, absolute_period_requested, context_param_list, timeout);
         } else {
             if (rrd_update_every != (uint16_t)max_interval) {
                 rrd_update_every = (uint16_t) max_interval;
@@ -1651,11 +1682,11 @@ RRDR *rrd2rrdr(
             }
             return rrd2rrdr_variablestep(st, points_requested, after_requested, before_requested, group_method,
                                          resampling_time_requested, options, dimensions, rrd_update_every,
-                                         first_entry_t, last_entry_t, absolute_period_requested, region_info_array, context_param_list);
+                                         first_entry_t, last_entry_t, absolute_period_requested, region_info_array, context_param_list, timeout);
         }
     }
 #endif
     return rrd2rrdr_fixedstep(st, points_requested, after_requested, before_requested, group_method,
                               resampling_time_requested, options, dimensions,
-                              rrd_update_every, first_entry_t, last_entry_t, absolute_period_requested, context_param_list);
+                              rrd_update_every, first_entry_t, last_entry_t, absolute_period_requested, context_param_list, timeout);
 }
diff --git a/web/api/queries/rrdr.h b/web/api/queries/rrdr.h
index 3637df687..bd94e56e2 100644
--- a/web/api/queries/rrdr.h
+++ b/web/api/queries/rrdr.h
@@ -47,6 +47,7 @@ typedef enum rrdr_result_flags {
     RRDR_RESULT_OPTION_RELATIVE      = 0x00000002, // the query uses relative time-frames
                                                    // (should not to be cached by browsers and proxies)
     RRDR_RESULT_OPTION_VARIABLE_STEP = 0x00000004, // the query uses variable-step time-frames
+    RRDR_RESULT_OPTION_CANCEL        = 0x00000008, // the query needs to be cancelled
 } RRDR_RESULT_FLAGS;
 
 typedef struct rrdresult {
@@ -110,7 +111,7 @@ extern RRDR *rrdr_create(struct rrdset *st, long n, struct context_param *contex
 extern RRDR *rrd2rrdr(
     RRDSET *st, long points_requested, long long after_requested, long long before_requested,
     RRDR_GROUPING group_method, long resampling_time_requested, RRDR_OPTIONS options, const char *dimensions,
-    struct context_param *context_param_list);
+    struct context_param *context_param_list, int timeout);
 
 #include "query.h"
 
diff --git a/web/api/queries/ses/README.md b/web/api/queries/ses/README.md
index c27970135..b835b8120 100644
--- a/web/api/queries/ses/README.md
+++ b/web/api/queries/ses/README.md
@@ -58,4 +58,4 @@ Examining last 1 minute `successful` web server responses:
 -   <https://en.wikipedia.org/wiki/Moving_average#exponential-moving-average>
 -   <https://en.wikipedia.org/wiki/Exponential_smoothing>.
 
-[![analytics](https://www.google-analytics.com/collect?v=1&aip=1&t=pageview&_s=1&ds=github&dr=https%3A%2F%2Fgithub.com%2Fnetdata%2Fnetdata&dl=https%3A%2F%2Fmy-netdata.io%2Fgithub%2Fweb%2Fapi%2Fqueries%2Fses%2FREADME&_u=MAC~&cid=5792dfd7-8dc4-476b-af31-da2fdb9f93d2&tid=UA-64295674-3)](<>)
+
diff --git a/web/api/queries/stddev/README.md b/web/api/queries/stddev/README.md
index 7cd7d62af..2fca47d5e 100644
--- a/web/api/queries/stddev/README.md
+++ b/web/api/queries/stddev/README.md
@@ -90,4 +90,4 @@ Examining last 1 minute `successful` web server responses:
 
 Check <https://en.wikipedia.org/wiki/Coefficient_of_variation>.
 
-[![analytics](https://www.google-analytics.com/collect?v=1&aip=1&t=pageview&_s=1&ds=github&dr=https%3A%2F%2Fgithub.com%2Fnetdata%2Fnetdata&dl=https%3A%2F%2Fmy-netdata.io%2Fgithub%2Fweb%2Fapi%2Fqueries%2Fstddev%2FREADME&_u=MAC~&cid=5792dfd7-8dc4-476b-af31-da2fdb9f93d2&tid=UA-64295674-3)](<>)
+
diff --git a/web/api/queries/sum/README.md b/web/api/queries/sum/README.md
index aeace0a16..d4465bd82 100644
--- a/web/api/queries/sum/README.md
+++ b/web/api/queries/sum/README.md
@@ -38,4 +38,4 @@ Examining last 1 minute `successful` web server responses:
 
 -   <https://en.wikipedia.org/wiki/Summation>.
 
-[![analytics](https://www.google-analytics.com/collect?v=1&aip=1&t=pageview&_s=1&ds=github&dr=https%3A%2F%2Fgithub.com%2Fnetdata%2Fnetdata&dl=https%3A%2F%2Fmy-netdata.io%2Fgithub%2Fweb%2Fapi%2Fqueries%2Fsum%2FREADME&_u=MAC~&cid=5792dfd7-8dc4-476b-af31-da2fdb9f93d2&tid=UA-64295674-3)](<>)
+
author	Daniel Baumann <daniel.baumann@progress-linux.org>	2022-04-14 18:12:14 +0000
committer	Daniel Baumann <daniel.baumann@progress-linux.org>	2022-04-14 18:12:14 +0000
commit	bb50acdcb8073654ea667b8c0272e335bd43f844 (patch)
tree	1e00c8a29871426f8182658928dcb62e42d57ce8 /web/api/queries
parent	Releasing debian version 1.33.1-1. (diff)
download	netdata-bb50acdcb8073654ea667b8c0272e335bd43f844.tar.xz netdata-bb50acdcb8073654ea667b8c0272e335bd43f844.zip