Merging upstream version 1.38.0.

Signed-off-by: Daniel Baumann <daniel.baumann@progress-linux.org>

6 files changed, 116 insertions, 69 deletions

diff --git a/web/server/README.md b/web/server/README.md
index 6485b84bc..407df6c03 100644
--- a/web/server/README.md
+++ b/web/server/README.md
@@ -1,7 +1,11 @@
 <!--
 title: "Web server"
 description: "The Netdata Agent's local static-threaded web server serves dashboards and real-time visualizations with security and DDoS protection."
-custom_edit_url: https://github.com/netdata/netdata/edit/master/web/server/README.md
+custom_edit_url: "https://github.com/netdata/netdata/edit/master/web/server/README.md"
+sidebar_label: "Web server"
+learn_status: "Published"
+learn_topic_type: "References"
+learn_rel_path: "References/Configuration"
 -->
 
 # Web server
@@ -47,7 +51,7 @@ Using the above, Netdata will bind to:
 -   IPv4 127.0.0.1 at port 19999 (port was used from `default port`). Only the UI (dashboard) and the read API will be accessible on this port. Both HTTP and HTTPS requests will be accepted.
 -   IPv4 10.1.1.1 at port 19998. The management API and `netdata.conf` will be accessible on this port.
 -   All the IPs `hostname` resolves to (both IPv4 and IPv6 depending on the resolved IPs) at port 19997. Only badges will be accessible on this port.
--   All IPv6 IPs at port 19996. Only metric streaming requests from other Netdata agents will be accepted on this port. Only encrypted streams will be allowed (i.e. child nodes also need to be [configured for TLS](/streaming/README.md).
+-   All IPv6 IPs at port 19996. Only metric streaming requests from other Netdata agents will be accepted on this port. Only encrypted streams will be allowed (i.e. child nodes also need to be [configured for TLS](https://github.com/netdata/netdata/blob/master/streaming/README.md).
 -   All the IPs `localhost` resolves to (both IPv4 and IPv6 depending the resolved IPs) at port 19996. This port will only accept registry API requests.
 -   All IPv4 and IPv6 IPs at port `http` as set in `/etc/services`. Only the UI (dashboard) and the read API will be accessible on this port. 
 -   Unix domain socket `/run/netdata/netdata.sock`. All requests are serviceable on this socket. Note that in some OSs like Fedora, every service sees a different `/tmp`, so don't create a Unix socket under `/tmp`. `/run` or `/var/run` is suggested.
@@ -136,7 +140,7 @@ Example:
     bind to = *=dashboard|registry|badges|management|streaming|netdata.conf^SSL=force
 ```
 
-For information how to configure the child to use TLS, check [securing the communication](/streaming/README.md#securing-streaming-communications) in the streaming documentation. There you will find additional details on the expected behavior for client and server nodes, when their respective TLS options are enabled.
+For information how to configure the child to use TLS, check [securing the communication](https://github.com/netdata/netdata/blob/master/streaming/README.md#securing-streaming-communications) in the streaming documentation. There you will find additional details on the expected behavior for client and server nodes, when their respective TLS options are enabled.
 
 When we define the use of SSL in a Netdata agent for different ports,  Netdata will apply the behavior specified on each port. For example, using the configuration line below:
 
@@ -192,7 +196,7 @@ Netdata supports access lists in `netdata.conf`:
 -   `allow netdata.conf from` checks the IP to allow `http://netdata.host:19999/netdata.conf`.
      The IPs listed are all the private IPv4 addresses, including link local IPv6 addresses. Keep in mind that connections to Netdata API ports are filtered by `allow connections from`. So, IPs allowed by `allow netdata.conf from` should also be allowed by `allow connections from`.
 
--   `allow management from` checks the IPs to allow API management calls. Management via the API is currently supported for [health](/web/api/health/README.md#health-management-api)
+-   `allow management from` checks the IPs to allow API management calls. Management via the API is currently supported for [health](https://github.com/netdata/netdata/blob/master/web/api/health/README.md#health-management-api)
 
 In order to check the FQDN of the connection without opening the Netdata agent to DNS-spoofing, a reverse-dns record
 must be setup for the connecting host. At connection time the reverse-dns of the peer IP address is resolved, and
@@ -218,13 +222,13 @@ present that may match DNS FQDNs.
 
 |setting|default|info|
 |:-----:|:-----:|:---|
-|ses max window|`15`|See [single exponential smoothing](/web/api/queries/des/README.md)|
-|des max window|`15`|See [double exponential smoothing](/web/api/queries/des/README.md)|
+|ses max window|`15`|See [single exponential smoothing](https://github.com/netdata/netdata/blob/master/web/api/queries/des/README.md)|
+|des max window|`15`|See [double exponential smoothing](https://github.com/netdata/netdata/blob/master/web/api/queries/des/README.md)|
 |listen backlog|`4096`|The port backlog. Check `man 2 listen`.|
 |disconnect idle clients after seconds|`60`|The time in seconds to disconnect web clients after being totally idle.|
 |timeout for first request|`60`|How long to wait for a client to send a request before closing the socket. Prevents slow request attacks.|
-|accept a streaming request every seconds|`0`|Can be used to set a limit on how often a parent node will accept streaming requests from child nodes in a [streaming and replication setup](/streaming/README.md)|
-|respect do not track policy|`no`|If set to `yes`, Netdata will respect the user's browser preferences for [Do Not Track](https://www.eff.org/issues/do-not-track) (DNT) and storing cookies. If DNT is _enabled_ in the browser, and this option is set to `yes`, users will not be able to sign in to Netdata Cloud via their local Agent dashboard, and their node will not connect to any [registry](/registry/README.md). For certain browsers, users must disable DNT and change this option to `yes` for full functionality.|
+|accept a streaming request every seconds|`0`|Can be used to set a limit on how often a parent node will accept streaming requests from child nodes in a [streaming and replication setup](https://github.com/netdata/netdata/blob/master/streaming/README.md)|
+|respect do not track policy|`no`|If set to `yes`, Netdata will respect the user's browser preferences for [Do Not Track](https://www.eff.org/issues/do-not-track) (DNT) and storing cookies. If DNT is _enabled_ in the browser, and this option is set to `yes`, users will not be able to sign in to Netdata Cloud via their local Agent dashboard, and their node will not connect to any [registry](https://github.com/netdata/netdata/blob/master/registry/README.md). For certain browsers, users must disable DNT and change this option to `yes` for full functionality.|
 |x-frame-options response header||[Avoid clickjacking attacks, by ensuring that the content is not embedded into other sites](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Frame-Options).|
 |enable gzip compression|`yes`|When set to `yes`, Netdata web responses will be GZIP compressed, if the web client accepts such responses.|
 |gzip compression strategy|`default`|Valid strategies are `default`, `filtered`, `huffman only`, `rle` and `fixed`|
diff --git a/web/server/static/static-threaded.c b/web/server/static/static-threaded.c
index 26e9a47bd..aca7d7ec0 100644
--- a/web/server/static/static-threaded.c
+++ b/web/server/static/static-threaded.c
@@ -307,7 +307,7 @@ static int web_server_rcv_callback(POLLINFO *pi, short int *events) {
         web_client_send(w);
     }
 
-    if(unlikely(w->mode == WEB_CLIENT_MODE_FILECOPY)) {
+    else if(unlikely(w->mode == WEB_CLIENT_MODE_FILECOPY)) {
         if(w->pollinfo_filecopy_slot == 0) {
             debug(D_WEB_CLIENT, "%llu: FILECOPY DETECTED ON FD %d", w->id, pi->fd);
 
@@ -408,6 +408,10 @@ static void socket_listen_main_static_threaded_worker_cleanup(void *ptr) {
     worker_unregister();
 }
 
+static bool web_server_should_stop(void) {
+    return !service_running(SERVICE_WEB_SERVER);
+}
+
 void *socket_listen_main_static_threaded_worker(void *ptr) {
     worker_private = (struct web_server_static_threaded_worker *)ptr;
     worker_private->running = 1;
@@ -430,6 +434,7 @@ void *socket_listen_main_static_threaded_worker(void *ptr) {
                         , web_server_rcv_callback
                         , web_server_snd_callback
                         , NULL
+                        , web_server_should_stop
                         , web_allow_connections_from
                         , web_allow_connections_dns
                         , NULL
@@ -452,35 +457,35 @@ static void socket_listen_main_static_threaded_cleanup(void *ptr) {
     struct netdata_static_thread *static_thread = (struct netdata_static_thread *)ptr;
     static_thread->enabled = NETDATA_MAIN_THREAD_EXITING;
 
-    int i, found = 0;
-    usec_t max = 2 * USEC_PER_SEC, step = 50000;
-
-    // we start from 1, - 0 is self
-    for(i = 1; i < static_threaded_workers_count; i++) {
-        if(static_workers_private_data[i].running) {
-            found++;
-            info("stopping worker %d", i + 1);
-            netdata_thread_cancel(static_workers_private_data[i].thread);
-        }
-        else
-            info("found stopped worker %d", i + 1);
-    }
-
-    while(found && max > 0) {
-        max -= step;
-        info("Waiting %d static web threads to finish...", found);
-        sleep_usec(step);
-        found = 0;
-
-        // we start from 1, - 0 is self
-        for(i = 1; i < static_threaded_workers_count; i++) {
-            if (static_workers_private_data[i].running)
-                found++;
-        }
-    }
-
-    if(found)
-        error("%d static web threads are taking too long to finish. Giving up.", found);
+//    int i, found = 0;
+//    usec_t max = 2 * USEC_PER_SEC, step = 50000;
+//
+//    // we start from 1, - 0 is self
+//    for(i = 1; i < static_threaded_workers_count; i++) {
+//        if(static_workers_private_data[i].running) {
+//            found++;
+//            info("stopping worker %d", i + 1);
+//            netdata_thread_cancel(static_workers_private_data[i].thread);
+//        }
+//        else
+//            info("found stopped worker %d", i + 1);
+//    }
+//
+//    while(found && max > 0) {
+//        max -= step;
+//        info("Waiting %d static web threads to finish...", found);
+//        sleep_usec(step);
+//        found = 0;
+//
+//        // we start from 1, - 0 is self
+//        for(i = 1; i < static_threaded_workers_count; i++) {
+//            if (static_workers_private_data[i].running)
+//                found++;
+//        }
+//    }
+//
+//    if(found)
+//        error("%d static web threads are taking too long to finish. Giving up.", found);
 
     info("closing all web server sockets...");
     listen_sockets_close(&api_sockets);
@@ -502,7 +507,7 @@ void *socket_listen_main_static_threaded(void *ptr) {
     // 6 threads is the optimal value
     // since 6 are the parallel connections browsers will do
     // so, if the machine has more CPUs, avoid using resources unnecessarily
-    int def_thread_count = (processors > 6) ? 6 : processors;
+    int def_thread_count = MIN(get_netdata_cpus(), 6);
 
     if (!strcmp(config_get(CONFIG_SECTION_WEB, "mode", ""),"single-threaded")) {
                 info("Running web server with one thread, because mode is single-threaded");
@@ -534,7 +539,7 @@ void *socket_listen_main_static_threaded(void *ptr) {
         static_workers_private_data[i].max_sockets = max_sockets / static_threaded_workers_count;
 
         char tag[50 + 1];
-        snprintfz(tag, 50, "WEB_SERVER[static%d]", i+1);
+        snprintfz(tag, 50, "WEB[%d]", i+1);
 
         info("starting worker %d", i+1);
         netdata_thread_create(&static_workers_private_data[i].thread, tag, NETDATA_THREAD_OPTION_DEFAULT,
diff --git a/web/server/web_client.c b/web/server/web_client.c
index b3c5ada7a..c14b86f3e 100644
--- a/web/server/web_client.c
+++ b/web/server/web_client.c
@@ -129,10 +129,10 @@ void web_client_request_done(struct web_client *w) {
                    , mode
                    , sent
                    , size
-                   , -((size > 0) ? ((size - sent) / (double) size * 100.0) : 0.0)
-                   , dt_usec(&w->tv_ready, &w->tv_in) / 1000.0
-                   , dt_usec(&tv, &w->tv_ready) / 1000.0
-                   , dt_usec(&tv, &w->tv_in) / 1000.0
+                   , -((size > 0) ? ((double)(size - sent) / (double) size * 100.0) : 0.0)
+                   , (double)dt_usec(&w->tv_ready, &w->tv_in) / 1000.0
+                   , (double)dt_usec(&tv, &w->tv_ready) / 1000.0
+                   , (double)dt_usec(&tv, &w->tv_in) / 1000.0
                    , w->response.code
                    , strip_control_characters(w->last_url)
         );
@@ -302,7 +302,7 @@ int mysendfile(struct web_client *w, char *filename) {
         }
     }
 
-    // if the filename contains a .. refuse to serve it
+    // if the filename contains a double dot refuse to serve it
     if(strstr(filename, "..") != 0) {
         debug(D_WEB_CLIENT_ACCESS, "%llu: File '%s' is not acceptable.", w->id, filename);
         w->response.data->contenttype = CT_TEXT_HTML;
@@ -831,9 +831,8 @@ static inline char *web_client_valid_method(struct web_client *w, char *s) {
  * @param s is the first address of the string.
  * @param ptr is the address of the separator.
  */
-static void web_client_set_path_query(struct web_client *w, char *s, char *ptr) {
+static void web_client_set_path_query(struct web_client *w, const char *s, char *ptr) {
     w->url_path_length = (size_t)(ptr -s);
-
     w->url_search_path = ptr;
 }
 
@@ -1250,12 +1249,15 @@ static inline void web_client_send_http_header(struct web_client *w) {
         if(bytes > 0)
             w->stats_sent_bytes += bytes;
 
-        error("HTTP headers failed to be sent (I sent %zu bytes but the system sent %zd bytes). Closing web client."
-              , buffer_strlen(w->response.header_output)
-              , bytes);
+        if (bytes < 0) {
 
-        WEB_CLIENT_IS_DEAD(w);
-        return;
+            error("HTTP headers failed to be sent (I sent %zu bytes but the system sent %zd bytes). Closing web client."
+                  , buffer_strlen(w->response.header_output)
+                  , bytes);
+
+            WEB_CLIENT_IS_DEAD(w);
+            return;
+        }
     }
     else
         w->stats_sent_bytes += bytes;
@@ -1314,6 +1316,9 @@ static inline int web_client_switch_host(RRDHOST *host, struct web_client *w, ch
 }
 
 static inline int web_client_process_url(RRDHOST *host, struct web_client *w, char *url) {
+    if(unlikely(!service_running(ABILITY_WEB_REQUESTS)))
+        return web_client_permission_denied(w);
+
     static uint32_t
             hash_api = 0,
             hash_netdata_conf = 0,
@@ -1423,7 +1428,7 @@ static inline int web_client_process_url(RRDHOST *host, struct web_client *w, ch
             // replace the zero bytes with spaces
             buffer_char_replace(w->response.data, '\0', ' ');
 
-            // just leave the buffer as is
+            // just leave the buffer as-is
             // it will be copied back to the client
 
             return HTTP_RESP_OK;
@@ -1540,7 +1545,7 @@ void web_client_process_request(struct web_client *w) {
             break;
     }
 
-    // keep track of the time we done processing
+    // keep track of the processing time
     now_realtime_timeval(&w->tv_ready);
 
     w->response.sent = 0;
@@ -1612,7 +1617,6 @@ ssize_t web_client_send_chunk_header(struct web_client *w, size_t len)
 
     else if(bytes == 0) {
         debug(D_WEB_CLIENT, "%llu: Did not send chunk header to the client.", w->id);
-        WEB_CLIENT_IS_DEAD(w);
     }
     else {
         debug(D_WEB_CLIENT, "%llu: Failed to send chunk header to client.", w->id);
@@ -1635,7 +1639,6 @@ ssize_t web_client_send_chunk_close(struct web_client *w)
 
     else if(bytes == 0) {
         debug(D_WEB_CLIENT, "%llu: Did not send chunk suffix to the client.", w->id);
-        WEB_CLIENT_IS_DEAD(w);
     }
     else {
         debug(D_WEB_CLIENT, "%llu: Failed to send chunk suffix to client.", w->id);
@@ -1658,7 +1661,6 @@ ssize_t web_client_send_chunk_finalize(struct web_client *w)
 
     else if(bytes == 0) {
         debug(D_WEB_CLIENT, "%llu: Did not send chunk finalize suffix to the client.", w->id);
-        WEB_CLIENT_IS_DEAD(w);
     }
     else {
         debug(D_WEB_CLIENT, "%llu: Failed to send chunk finalize suffix to client.", w->id);
@@ -1775,7 +1777,6 @@ ssize_t web_client_send_deflate(struct web_client *w)
         debug(D_WEB_CLIENT, "%llu: Did not send any bytes to the client (zhave = %zu, zsent = %zu, need to send = %zu).",
             w->id, w->response.zhave, w->response.zsent, w->response.zhave - w->response.zsent);
 
-        WEB_CLIENT_IS_DEAD(w);
     }
     else {
         debug(D_WEB_CLIENT, "%llu: Failed to send data to client.", w->id);
@@ -1828,7 +1829,6 @@ ssize_t web_client_send(struct web_client *w) {
     }
     else if(likely(bytes == 0)) {
         debug(D_WEB_CLIENT, "%llu: Did not send any bytes to the client.", w->id);
-        WEB_CLIENT_IS_DEAD(w);
     }
     else {
         debug(D_WEB_CLIENT, "%llu: Failed to send data to client.", w->id);
@@ -1846,7 +1846,7 @@ ssize_t web_client_read_file(struct web_client *w)
     if(unlikely(w->response.rlen <= w->response.data->len))
         return 0;
 
-    ssize_t left = w->response.rlen - w->response.data->len;
+    ssize_t left = (ssize_t)(w->response.rlen - w->response.data->len);
     ssize_t bytes = read(w->ifd, &w->response.data->buffer[w->response.data->len], (size_t)left);
     if(likely(bytes > 0)) {
         size_t old = w->response.data->len;
@@ -1896,7 +1896,7 @@ ssize_t web_client_receive(struct web_client *w)
         return web_client_read_file(w);
 
     ssize_t bytes;
-    ssize_t left = w->response.data->size - w->response.data->len;
+    ssize_t left = (ssize_t)(w->response.data->size - w->response.data->len);
 
     // do we have any space for more data?
     buffer_need_bytes(w->response.data, NETDATA_WEB_REQUEST_RECEIVE_SIZE);
@@ -1928,10 +1928,32 @@ ssize_t web_client_receive(struct web_client *w)
         debug(D_WEB_CLIENT, "%llu: Received %zd bytes.", w->id, bytes);
         debug(D_WEB_DATA, "%llu: Received data: '%s'.", w->id, &w->response.data->buffer[old]);
     }
-    else {
+    else if (bytes < 0) {
         debug(D_WEB_CLIENT, "%llu: receive data failed.", w->id);
         WEB_CLIENT_IS_DEAD(w);
-    }
+    } else
+        debug(D_WEB_CLIENT, "%llu: Received %zd bytes.", w->id, bytes);
 
     return(bytes);
 }
+
+
+int web_client_socket_is_now_used_for_streaming(struct web_client *w) {
+    // prevent the web_client from closing the streaming socket
+
+    WEB_CLIENT_IS_DEAD(w);
+
+    if(web_server_mode == WEB_SERVER_MODE_STATIC_THREADED) {
+        web_client_flag_set(w, WEB_CLIENT_FLAG_DONT_CLOSE_SOCKET);
+    }
+    else {
+        if(w->ifd == w->ofd)
+            w->ifd = w->ofd = -1;
+        else
+            w->ifd = -1;
+    }
+
+    buffer_flush(w->response.data);
+
+    return HTTP_RESP_OK;
+}
diff --git a/web/server/web_client.h b/web/server/web_client.h
index 630d71a8a..d0360f4f9 100644
--- a/web/server/web_client.h
+++ b/web/server/web_client.h
@@ -19,13 +19,16 @@ extern int web_enable_gzip, web_gzip_level, web_gzip_strategy;
 
 // HTTP_CODES 4XX Client Errors
 #define HTTP_RESP_BAD_REQUEST 400
+#define HTTP_RESP_UNAUTHORIZED 401
 #define HTTP_RESP_FORBIDDEN 403
 #define HTTP_RESP_NOT_FOUND 404
+#define HTTP_RESP_CONFLICT 409
 #define HTTP_RESP_PRECOND_FAIL 412
 
 // HTTP_CODES 5XX Server Errors
 #define HTTP_RESP_INTERNAL_SERVER_ERROR 500
-#define HTTP_RESP_BACKEND_FETCH_FAILED 503
+#define HTTP_RESP_BACKEND_FETCH_FAILED 503  // 503 is right
+#define HTTP_RESP_SERVICE_UNAVAILABLE 503   // 503 is right
 #define HTTP_RESP_GATEWAY_TIMEOUT 504
 #define HTTP_RESP_BACKEND_RESPONSE_INVALID 591
 
@@ -206,6 +209,8 @@ int mysendfile(struct web_client *w, char *filename);
 void web_client_build_http_header(struct web_client *w);
 char *strip_control_characters(char *url);
 
+int web_client_socket_is_now_used_for_streaming(struct web_client *w);
+
 #include "daemon/common.h"
 
 #endif
diff --git a/web/server/web_client_cache.c b/web/server/web_client_cache.c
index 1fa593580..4344209c8 100644
--- a/web/server/web_client_cache.c
+++ b/web/server/web_client_cache.c
@@ -11,7 +11,16 @@
 static void web_client_reuse_ssl(struct web_client *w) {
     if (netdata_ssl_srv_ctx) {
         if (w->ssl.conn) {
-            SSL_clear(w->ssl.conn);
+            SSL_SESSION *session = SSL_get_session(w->ssl.conn);
+            SSL *old = w->ssl.conn;
+            w->ssl.conn = SSL_new(netdata_ssl_srv_ctx);
+            if (session) {
+#if OPENSSL_VERSION_NUMBER >= OPENSSL_VERSION_111
+                if (SSL_SESSION_is_resumable(session))
+#endif
+                    SSL_set_session(w->ssl.conn, session);
+            }
+            SSL_free(old);
         }
     }
 }
@@ -56,13 +65,15 @@ static void web_client_free(struct web_client *w) {
     }
 #endif
     freez(w);
+    __atomic_sub_fetch(&netdata_buffers_statistics.buffers_web, sizeof(struct web_client), __ATOMIC_RELAXED);
 }
 
 static struct web_client *web_client_alloc(void) {
     struct web_client *w = callocz(1, sizeof(struct web_client));
-    w->response.data = buffer_create(NETDATA_WEB_RESPONSE_INITIAL_SIZE);
-    w->response.header = buffer_create(NETDATA_WEB_RESPONSE_HEADER_SIZE);
-    w->response.header_output = buffer_create(NETDATA_WEB_RESPONSE_HEADER_SIZE);
+    __atomic_add_fetch(&netdata_buffers_statistics.buffers_web, sizeof(struct web_client), __ATOMIC_RELAXED);
+    w->response.data = buffer_create(NETDATA_WEB_RESPONSE_INITIAL_SIZE, &netdata_buffers_statistics.buffers_web);
+    w->response.header = buffer_create(NETDATA_WEB_RESPONSE_HEADER_SIZE, &netdata_buffers_statistics.buffers_web);
+    w->response.header_output = buffer_create(NETDATA_WEB_RESPONSE_HEADER_SIZE, &netdata_buffers_statistics.buffers_web);
     return w;
 }
 
diff --git a/web/server/web_server.c b/web/server/web_server.c
index 4da08d431..d5645a947 100644
--- a/web/server/web_server.c
+++ b/web/server/web_server.c
@@ -37,7 +37,7 @@ LISTEN_SOCKETS api_sockets = {
 };
 
 void debug_sockets() {
-	BUFFER *wb = buffer_create(256 * sizeof(char));
+	BUFFER *wb = buffer_create(256 * sizeof(char), NULL);
 	int i;
 
 	for(i = 0 ; i < (int)api_sockets.opened ; i++) {