Merging upstream version 1.16.1.

Signed-off-by: Daniel Baumann <daniel.baumann@progress-linux.org>

3 files changed, 102 insertions, 13 deletions

diff --git a/streaming/README.md b/streaming/README.md
index 3e58f1f06..1bfbb2362 100644
--- a/streaming/README.md
+++ b/streaming/README.md
@@ -138,7 +138,7 @@ headless proxy|`none`|not `none`|`yes`|only for `data source = as collected`|not
 proxy with db|not `none`|not `none`|`yes`|possible|possible|yes
 central netdata|not `none`|not `none`|`no`|possible|possible|yes
 
-For the options to encrypt the data stream between the slave and the master, refer to [securing the communication](#securing-the-communication)
+For the options to encrypt the data stream between the slave and the master, refer to [securing the communication](#securing-streaming-communications)
 
 ##### options for the receiving node
 
@@ -213,40 +213,91 @@ The receiving end (`proxy` or `master`) logs entries like these:
 
 For netdata v1.9+, streaming can also be monitored via `access.log`.
 
-### Securing the communication
+### Securing streaming communications
 
-Netdata does not activate TLS encryption by default. To encrypt the connection, you first need to [enable TLS support](../web/server/#enabling-tls-support) on the master. With encryption enabled on the receiving side, we need to instruct the slave to use SSL as well. On the slave's `stream.conf`, configure the destination as follows:
+Netdata does not activate TLS encryption by default. To encrypt streaming connections, you first need to [enable TLS support](../web/server/#enabling-tls-support) on the master. With encryption enabled on the receiving side, you need to instruct the slave to use TLS/SSL as well. On the slave's `stream.conf`, configure the destination as follows:
 
 ```
 [stream]
     destination = host:port:SSL
 ```
 
-The word SSL appended to the end of the destination tells the slave that the connection must be encrypted.
+The word `SSL` appended to the end of the destination tells the slave that connections must be encrypted.
+
+??? info "Differences in TLS and SSL terminology"
+    While Netdata uses Transport Layer Security (TLS) 1.2 to encrypt communications rather than the obsolete SSL protocol, it's still common practice to refer to encrypted web connections as `SSL`. Many vendors, like Nginx and even Netdata itself, use `SSL` in configuration files, whereas documentation will always refer to encrypted communications as `TLS` or `TLS/SSL`.
 
 #### Certificate verification
 
-When SSL is enabled on the slave, the default behavior will be do not connect with the master unless the server's certificate can be verified via the default chain. In case you want to avoid this check, add to the slave's `stream.conf` the following:
+When TLS/SSL is enabled on the slave, the default behavior will be to not connect with the master unless the server's certificate can be verified via the default chain. In case you want to avoid this check, add the following to the slave's `stream.conf` file:
 
 ```
 [stream]
     ssl skip certificate verification = yes
 ```
 
+#### Trusted certificate
+
+If you've enabled [certificate verification](#certificate-verification), you might see errors from the OpenSSL library when there's a problem with checking the certificate chain (`X509_V_ERR_UNABLE_TO_GET_ISSUER_CERT_LOCALLY`). More importantly, OpenSSL will reject self-signed certificates.
+
+Given these known issues, you have two options. If you trust your certificate, you can set the options `CApath` and `CAfile` to inform Netdata where your certificates, and the certificate trusted file, are stored.
+
+For more details about these options, you can read about [verify locations](https://www.openssl.org/docs/man1.1.1/man3/SSL_CTX_load_verify_locations.html).
+
+Before you changed your streaming configuration, you need to copy your trusted certificate to your slave system and add the certificate to OpenSSL's list.
+
+On most Linux distributions, the `update-ca-certificates` command searches inside the `/usr/share/ca-certificates` directory for certificates. You should double-check by reading the `update-ca-certificate` manual (`man update-ca-certificate`), and then change the directory in the below commands if needed.
+
+If you have `sudo` configured on your slave system, you can use that to run the following commands. If not, you'll have to log in as `root` to complete them.
+
+```
+# mkdir /usr/share/ca-certificates/netdata
+# cp master_cert.pem /usr/share/ca-certificates/netdata/master_cert.crt
+# chown -R netdata.netdata /usr/share/ca-certificates/netdata/
+```
+
+First, you create a new directory to store your certificates for Netdata. Next, you need to change the extension on your certificate from `.pem` to `.crt` so it's compatible with `update-ca-certificate`. Finally, you need to change permissions so the user that runs Netdata can access the directory where you copied in your certificate.
+
+Next, edit the file `/etc/ca-certificates.conf` and add the following line:
+
+```
+netdata/master_cert.crt
+```
+
+Now you update the list of certificates running the following, again either as `sudo` or `root`:
+
+```
+# update-ca-certificates
+```
+
+!!! note
+    Some Linux distributions have different methods of updating the certificate list. For more details, please read this guide on [addding trusted root certificates](https://github.com/Busindre/How-to-Add-trusted-root-certificates).
+
+Once you update your certificate list, you can set the stream parameters for Netdata to trust the master certificate. Open `stream.conf` for editing and change the following lines:
+
+```
+[stream]
+    CApath = /etc/ssl/certs/
+    CAfile = /etc/ssl/certs/master_cert.pem
+```
+
+With this configuration, the `CApath` option tells Netdata to search for trusted certificates inside `/etc/ssl/certs`. The `CAfile` option specifies the Netdata master certificate is located at `/etc/ssl/certs/master_cert.pem`. With this configuration, you can skip using the system's entire list of certificates and use Netdata's master certificate instead.
+
 #### Expected behaviors
 
-With the introduction of SSL, the master-slave communication behaves as shown in the table below, depending on the following configurations:
-- Master TLS (Yes/No): Whether the `[web]` section in `netdata.conf` has `ssl key` and `ssl certificate`.
-- Master port SSL (-/force/optional): Depends on whether the `[web]` section `bind to` contains a `^SSL=force` or `^SSL=optional` directive on the port(s) used for streaming.
-- Slave TLS (Yes/No): Whether the destination in the slave's `stream.conf` has `:SSL` at the end.
-- Slave SSL Verification (yes/no): Value of the slave's `stream.conf` `ssl skip certificate verification` parameter (default is no).
+With the introduction of TLS/SSL, the master-slave communication behaves as shown in the table below, depending on the following configurations:
+
+- **Master TLS (Yes/No)**: Whether the `[web]` section in `netdata.conf` has `ssl key` and `ssl certificate`.
+- **Master port TLS (-/force/optional)**: Depends on whether the `[web]` section `bind to` contains a `^SSL=force` or `^SSL=optional` directive on the port(s) used for streaming.
+- **Slave TLS (Yes/No)**: Whether the destination in the slave's `stream.conf` has `:SSL` at the end.
+- **Slave TLS Verification (yes/no)**: Value of the slave's `stream.conf` `ssl skip certificate verification` parameter (default is no).
 
  Master TLS enabled | Master port SSL | Slave TLS | Slave SSL Ver. | Behavior
 :------:|:-----:|:-----:|:-----:|:--------
 No | - | No | no | Legacy behavior. The master-slave stream is unencrypted.
 Yes | force | No | no | The master rejects the slave connection.
 Yes | -/optional | No | no | The master-slave stream is unencrypted (expected situation for legacy slaves and newer masters)
-Yes | -/force/optional | Yes | no | The master-slave stream is encrypted, provided that the master has a valid SSL certificate. Otherwise, the slave refuses to connect.
+Yes | -/force/optional | Yes | no | The master-slave stream is encrypted, provided that the master has a valid TLS/SSL certificate. Otherwise, the slave refuses to connect.
 Yes | -/force/optional | Yes | yes | The master-slave stream is encrypted.
 
 ## Viewing remote host dashboards, using mirrored databases
diff --git a/streaming/rrdpush.c b/streaming/rrdpush.c
index 954b1d7d1..59913c24b 100644
--- a/streaming/rrdpush.c
+++ b/streaming/rrdpush.c
@@ -48,6 +48,11 @@ unsigned int default_rrdpush_enabled = 0;
 char *default_rrdpush_destination = NULL;
 char *default_rrdpush_api_key = NULL;
 char *default_rrdpush_send_charts_matching = NULL;
+#ifdef ENABLE_HTTPS
+int netdata_use_ssl_on_stream = NETDATA_SSL_OPTIONAL;
+char *netdata_ssl_ca_path = NULL;
+char *netdata_ssl_ca_file = NULL;
+#endif
 
 static void load_stream_conf() {
     errno = 0;
@@ -89,13 +94,17 @@ int rrdpush_init() {
             }
         }
     }
+
     char *invalid_certificate = appconfig_get(&stream_config, CONFIG_SECTION_STREAM, "ssl skip certificate verification", "no");
     if ( !strcmp(invalid_certificate,"yes")){
         if (netdata_validate_server == NETDATA_SSL_VALID_CERTIFICATE){
-            info("The Netdata is configured to accept invalid certificate.");
+            info("Netdata is configured to accept invalid SSL certificate.");
             netdata_validate_server = NETDATA_SSL_INVALID_CERTIFICATE;
         }
     }
+
+    netdata_ssl_ca_path = appconfig_get(&stream_config, CONFIG_SECTION_STREAM, "CApath", "/etc/ssl/certs/");
+    netdata_ssl_ca_file = appconfig_get(&stream_config, CONFIG_SECTION_STREAM, "CAfile", "/etc/ssl/certs/certs.pem");
 #endif
 
     return default_rrdpush_enabled;
@@ -652,6 +661,7 @@ void *rrdpush_sender_thread(void *ptr) {
 #ifdef ENABLE_HTTPS
     if (netdata_use_ssl_on_stream & NETDATA_SSL_FORCE ){
         security_start_ssl(NETDATA_SSL_CONTEXT_STREAMING);
+        security_location_for_context(netdata_client_ctx, netdata_ssl_ca_file, netdata_ssl_ca_path);
     }
 #endif
 
@@ -801,7 +811,17 @@ void *rrdpush_sender_thread(void *ptr) {
                         rrdpush_buffer_lock(host);
 
                         debug(D_STREAM, "STREAM: Sending data, starting from %zu, size %zu...", begin, buffer_strlen(host->rrdpush_sender_buffer));
-                        ssize_t ret = send(host->rrdpush_sender_socket, &host->rrdpush_sender_buffer->buffer[begin], buffer_strlen(host->rrdpush_sender_buffer) - begin, MSG_DONTWAIT);
+                        ssize_t ret;
+#ifdef ENABLE_HTTPS
+                        SSL *conn = host->ssl.conn ;
+                        if(conn && !host->ssl.flags) {
+                            ret = SSL_write(conn,&host->rrdpush_sender_buffer->buffer[begin], buffer_strlen(host->rrdpush_sender_buffer) - begin);
+                        } else {
+                            ret = send(host->rrdpush_sender_socket, &host->rrdpush_sender_buffer->buffer[begin], buffer_strlen(host->rrdpush_sender_buffer) - begin, MSG_DONTWAIT);
+                        }
+#else
+                        ret = send(host->rrdpush_sender_socket, &host->rrdpush_sender_buffer->buffer[begin], buffer_strlen(host->rrdpush_sender_buffer) - begin, MSG_DONTWAIT);
+#endif
                         if (unlikely(ret == -1)) {
                             if (errno != EAGAIN && errno != EINTR && errno != EWOULDBLOCK) {
                                 debug(D_STREAM, "STREAM: Send failed - closing socket...");
@@ -1059,6 +1079,8 @@ static int rrdpush_receive(int fd
 
     info("STREAM %s [receive from [%s]:%s]: initializing communication...", host->hostname, client_ip, client_port);
 #ifdef ENABLE_HTTPS
+    host->ssl.conn = ssl->conn;
+    host->ssl.flags = ssl->flags;
     if(send_timeout(ssl,fd, START_STREAMING_PROMPT, strlen(START_STREAMING_PROMPT), 0, 60) != strlen(START_STREAMING_PROMPT)) {
 #else
     if(send_timeout(fd, START_STREAMING_PROMPT, strlen(START_STREAMING_PROMPT), 0, 60) != strlen(START_STREAMING_PROMPT)) {
diff --git a/streaming/stream.conf b/streaming/stream.conf
index 0d360cc24..fdff1f25f 100644
--- a/streaming/stream.conf
+++ b/streaming/stream.conf
@@ -41,6 +41,22 @@
     #
     #ssl skip certificate verification = yes
 
+    # Certificate Authority Path
+    #
+    # OpenSSL has a default directory where the known certificates are stored,
+    # case it is necessary it is possible to change this rule using the variable
+    # "CApath"
+    #
+    #CApath = /etc/ssl/certs/
+
+    # Certificate Authority file
+    #
+    # When the Netdata master has certificate, that is not recognized as valid,
+    # we can add this certificate in the list of known certificates in CApath
+    # and give for Netdata as argument.
+    #
+    #CAfile = /etc/ssl/certs/cert.pem
+
     # The API_KEY to use (as the sender)
     api key =