Adding upstream version 1.17.0.upstream/1.17.0

Signed-off-by: Daniel Baumann <daniel.baumann@progress-linux.org>

3 files changed, 217 insertions, 128 deletions

diff --git a/collectors/python.d.plugin/mongodb/README.md b/collectors/python.d.plugin/mongodb/README.md
index 3ddf137d7..fd694c1e5 100644
--- a/collectors/python.d.plugin/mongodb/README.md
+++ b/collectors/python.d.plugin/mongodb/README.md
@@ -3,135 +3,161 @@
 Module monitor mongodb performance and health metrics
 
 **Requirements:**
- * `python-pymongo` package v2.4+.
 
-You need to install it manually.
+-   `python-pymongo` package v2.4+.
 
+You need to install it manually.
 
 Number of charts depends on mongodb version, storage engine and other features (replication):
 
-1. **Read requests**:
- * query
- * getmore (operation the cursor executes to get additional data from query)
-
-2. **Write requests**:
- * insert
- * delete
- * update
-
-3. **Active clients**:
- * readers (number of clients with read operations in progress or queued)
- * writers (number of clients with write operations in progress or queued)
-
-4. **Journal transactions**:
- * commits (count of transactions that have been written to the journal)
-
-5. **Data written to the journal**:
- * volume (volume of data)
-
-6. **Background flush** (MMAPv1):
- * average ms (average time taken by flushes to execute)
- * last ms (time taken by the last flush)
-
-8. **Read tickets** (WiredTiger):
- * in use (number of read tickets in use)
- * available (number of available read tickets remaining)
-
-9. **Write tickets** (WiredTiger):
- * in use (number of write tickets in use)
- * available (number of available write tickets remaining)
-
-10. **Cursors**:
- * opened (number of cursors currently opened by MongoDB for clients)
- * timedOut (number of cursors that have timed)
- * noTimeout (number of open cursors with timeout disabled)
-
-11. **Connections**:
- * connected (number of clients currently connected to the database server)
- * unused (number of unused connections available for new clients)
-
-12. **Memory usage metrics**:
- * virtual
- * resident (amount of memory used by the database process)
- * mapped
- * non mapped
-
-13. **Page faults**:
- * page faults (number of times MongoDB had to request from disk)
-
-14. **Cache metrics** (WiredTiger):
- * percentage of bytes currently in the cache (amount of space taken by cached data)
- * percantage of tracked dirty bytes in the cache (amount of space taken by dirty data)
-
-15. **Pages evicted from cache** (WiredTiger):
- * modified
- * unmodified
-
-16. **Queued requests**:
- * readers (number of read request currently queued)
- * writers (number of write request currently queued)
-
-17. **Errors**:
- * msg (number of message assertions raised)
- * warning (number of warning assertions raised)
- * regular (number of regular assertions raised)
- * user (number of assertions corresponding to errors generated by users)
-
-18. **Storage metrics** (one chart for every database)
- * dataSize (size of all documents + padding in the database)
- * indexSize (size of all indexes in the database)
- * storageSize (size of all extents in the database)
-
-19. **Documents in the database** (one chart for all databases)
- * documents (number of objects in the database among all the collections)
-
-20. **tcmalloc metrics**
- * central cache free
- * current total thread cache
- * pageheap free
- * pageheap unmapped
- * thread cache free
- * transfer cache free
- * heap size
-
-21. **Commands total/failed rate**
- * count
- * createIndex
- * delete
- * eval
- * findAndModify
- * insert
-
-22. **Locks metrics** (acquireCount metrics - number of times the lock was acquired in the specified mode)
- * Global lock
- * Database lock
- * Collection lock
- * Metadata lock
- * oplog lock
-
-23. **Replica set members state**
- * state
-
-24. **Oplog window**
-  * window (interval of time between the oldest and the latest entries in the oplog)
-
-25. **Replication lag**
-  * member (time when last entry from the oplog was applied for every member)
-
-26. **Replication set member heartbeat latency**
-  * member (time when last heartbeat was received from replica set member)
-
-### prerequisite
-Create a read-only user for the netdata in the admin database.
-
-1. Authenticate as the admin user.
+1.  **Read requests**:
+
+    -   query
+    -   getmore (operation the cursor executes to get additional data from query)
+
+2.  **Write requests**:
+
+    -   insert
+    -   delete
+    -   update
+
+3.  **Active clients**:
+
+    -   readers (number of clients with read operations in progress or queued)
+    -   writers (number of clients with write operations in progress or queued)
+
+4.  **Journal transactions**:
+
+    -   commits (count of transactions that have been written to the journal)
+
+5.  **Data written to the journal**:
+
+    -   volume (volume of data)
+
+6.  **Background flush** (MMAPv1):
+
+    -   average ms (average time taken by flushes to execute)
+    -   last ms (time taken by the last flush)
+
+7.  **Read tickets** (WiredTiger):
+
+    -   in use (number of read tickets in use)
+    -   available (number of available read tickets remaining)
+
+8.  **Write tickets** (WiredTiger):
+
+    -   in use (number of write tickets in use)
+    -   available (number of available write tickets remaining)
+
+9.  **Cursors**:
+
+-   opened (number of cursors currently opened by MongoDB for clients)
+-   timedOut (number of cursors that have timed)
+-   noTimeout (number of open cursors with timeout disabled)
+
+10. **Connections**:
+
+    -   connected (number of clients currently connected to the database server)
+    -   unused (number of unused connections available for new clients)
+
+11. **Memory usage metrics**:
+
+    -   virtual
+    -   resident (amount of memory used by the database process)
+    -   mapped
+    -   non mapped
+
+12. **Page faults**:
+
+    -   page faults (number of times MongoDB had to request from disk)
+
+13. **Cache metrics** (WiredTiger):
+
+    -   percentage of bytes currently in the cache (amount of space taken by cached data)
+    -   percantage of tracked dirty bytes in the cache (amount of space taken by dirty data)
+
+14. **Pages evicted from cache** (WiredTiger):
+
+    -   modified
+    -   unmodified
+
+15. **Queued requests**:
+
+    -   readers (number of read request currently queued)
+    -   writers (number of write request currently queued)
+
+16. **Errors**:
+
+    -   msg (number of message assertions raised)
+    -   warning (number of warning assertions raised)
+    -   regular (number of regular assertions raised)
+    -   user (number of assertions corresponding to errors generated by users)
+
+17. **Storage metrics** (one chart for every database)
+
+    -   dataSize (size of all documents + padding in the database)
+    -   indexSize (size of all indexes in the database)
+    -   storageSize (size of all extents in the database)
+
+18. **Documents in the database** (one chart for all databases)
+
+-   documents (number of objects in the database among all the collections)
+
+19. **tcmalloc metrics**
+
+    -   central cache free
+    -   current total thread cache
+    -   pageheap free
+    -   pageheap unmapped
+    -   thread cache free
+    -   transfer cache free
+    -   heap size
+
+20. **Commands total/failed rate**
+
+    -   count
+    -   createIndex
+    -   delete
+    -   eval
+    -   findAndModify
+    -   insert
+
+21. **Locks metrics** (acquireCount metrics - number of times the lock was acquired in the specified mode)
+
+    -   Global lock
+    -   Database lock
+    -   Collection lock
+    -   Metadata lock
+    -   oplog lock
+
+22. **Replica set members state**
+
+    -   state
+
+23. **Oplog window**
+
+    -   window (interval of time between the oldest and the latest entries in the oplog)
+
+24. **Replication lag**
+
+    -   member (time when last entry from the oplog was applied for every member)
+
+25. **Replication set member heartbeat latency**
+
+    -   member (time when last heartbeat was received from replica set member)
+
+## prerequisite
+
+Create a read-only user for Netdata in the admin database.
+
+1.  Authenticate as the admin user.
 
 ```
 use admin
 db.auth("admin", "<MONGODB_ADMIN_PASSWORD>")
 ```
 
-2. Create a user.
+2.  Create a user.
 
 ```
 # MongoDB 2.x.
@@ -161,11 +187,10 @@ local:
     port : 27017
     user : 'netdata'
     pass : 'netdata'
-
 ```
 
 If no configuration is given, module will attempt to connect to mongodb daemon on `127.0.0.1:27017` address
 
 ---
 
-[![analytics](https://www.google-analytics.com/collect?v=1&aip=1&t=pageview&_s=1&ds=github&dr=https%3A%2F%2Fgithub.com%2Fnetdata%2Fnetdata&dl=https%3A%2F%2Fmy-netdata.io%2Fgithub%2Fcollectors%2Fpython.d.plugin%2Fmongodb%2FREADME&_u=MAC~&cid=5792dfd7-8dc4-476b-af31-da2fdb9f93d2&tid=UA-64295674-3)]()
+[![analytics](https://www.google-analytics.com/collect?v=1&aip=1&t=pageview&_s=1&ds=github&dr=https%3A%2F%2Fgithub.com%2Fnetdata%2Fnetdata&dl=https%3A%2F%2Fmy-netdata.io%2Fgithub%2Fcollectors%2Fpython.d.plugin%2Fmongodb%2FREADME&_u=MAC~&cid=5792dfd7-8dc4-476b-af31-da2fdb9f93d2&tid=UA-64295674-3)](<>)
diff --git a/collectors/python.d.plugin/mongodb/mongodb.chart.py b/collectors/python.d.plugin/mongodb/mongodb.chart.py
index 5db48cb12..0dbe82ff9 100644
--- a/collectors/python.d.plugin/mongodb/mongodb.chart.py
+++ b/collectors/python.d.plugin/mongodb/mongodb.chart.py
@@ -3,6 +3,8 @@
 # Author: ilyam8
 # SPDX-License-Identifier: GPL-3.0-or-later
 
+import ssl
+
 from copy import deepcopy
 from datetime import datetime
 from sys import exc_info
@@ -418,18 +420,31 @@ CHARTS = {
     }
 }
 
+DEFAULT_HOST = '127.0.0.1'
+DEFAULT_PORT = 27017
+DEFAULT_TIMEOUT = 100
+DEFAULT_AUTHDB = 'admin'
+
+CONN_PARAM_HOST = 'host'
+CONN_PARAM_PORT = 'port'
+CONN_PARAM_SERVER_SELECTION_TIMEOUT_MS = 'serverselectiontimeoutms'
+CONN_PARAM_SSL_SSL = 'ssl'
+CONN_PARAM_SSL_CERT_REQS = 'ssl_cert_reqs'
+CONN_PARAM_SSL_CA_CERTS = 'ssl_ca_certs'
+CONN_PARAM_SSL_CRL_FILE = 'ssl_crlfile'
+CONN_PARAM_SSL_CERT_FILE = 'ssl_certfile'
+CONN_PARAM_SSL_KEY_FILE = 'ssl_keyfile'
+CONN_PARAM_SSL_PEM_PASSPHRASE = 'ssl_pem_passphrase'
+
 
 class Service(SimpleService):
     def __init__(self, configuration=None, name=None):
         SimpleService.__init__(self, configuration=configuration, name=name)
         self.order = ORDER[:]
         self.definitions = deepcopy(CHARTS)
-        self.authdb = self.configuration.get('authdb', 'admin')
+        self.authdb = self.configuration.get('authdb', DEFAULT_AUTHDB)
         self.user = self.configuration.get('user')
         self.password = self.configuration.get('pass')
-        self.host = self.configuration.get('host', '127.0.0.1')
-        self.port = self.configuration.get('port', 27017)
-        self.timeout = self.configuration.get('timeout', 100)
         self.metrics_to_collect = deepcopy(DEFAULT_METRICS)
         self.connection = None
         self.do_replica = None
@@ -705,14 +720,53 @@ class Service(SimpleService):
 
         return data
 
-    def _create_connection(self):
-        conn_vars = {'host': self.host, 'port': self.port}
+    def build_ssl_connection_params(self):
+        conf = self.configuration
+
+        def cert_req(v):
+            if v is None:
+                return None
+            if not v:
+                return ssl.CERT_NONE
+            return ssl.CERT_REQUIRED
+
+        ssl_params = {
+            CONN_PARAM_SSL_SSL: conf.get(CONN_PARAM_SSL_SSL),
+            CONN_PARAM_SSL_CERT_REQS: cert_req(conf.get(CONN_PARAM_SSL_CERT_REQS)),
+            CONN_PARAM_SSL_CA_CERTS: conf.get(CONN_PARAM_SSL_CA_CERTS),
+            CONN_PARAM_SSL_CRL_FILE: conf.get(CONN_PARAM_SSL_CRL_FILE),
+            CONN_PARAM_SSL_CERT_FILE: conf.get(CONN_PARAM_SSL_CERT_FILE),
+            CONN_PARAM_SSL_KEY_FILE: conf.get(CONN_PARAM_SSL_KEY_FILE),
+            CONN_PARAM_SSL_PEM_PASSPHRASE: conf.get(CONN_PARAM_SSL_PEM_PASSPHRASE),
+        }
+
+        ssl_params = dict((k, v) for k, v in ssl_params.items() if v is not None)
+
+        return ssl_params
+
+    def build_connection_params(self):
+        conf = self.configuration
+        params = {
+            CONN_PARAM_HOST: conf.get(CONN_PARAM_HOST, DEFAULT_HOST),
+            CONN_PARAM_PORT: conf.get(CONN_PARAM_PORT, DEFAULT_PORT),
+        }
         if hasattr(MongoClient, 'server_selection_timeout'):
-            conn_vars.update({'serverselectiontimeoutms': self.timeout})
+            params[CONN_PARAM_SERVER_SELECTION_TIMEOUT_MS] = conf.get('timeout', DEFAULT_TIMEOUT)
+
+        params.update(self.build_ssl_connection_params())
+        return params
+
+    def _create_connection(self):
+        params = self.build_connection_params()
+        self.debug('creating connection, connection params: {0}'.format(sorted(params)))
+
         try:
-            connection = MongoClient(**conn_vars)
+            connection = MongoClient(**params)
             if self.user and self.password:
+                self.debug('authenticating, user: {0}, password: {1}'.format(self.user, self.password))
                 getattr(connection, self.authdb).authenticate(name=self.user, password=self.password)
+            else:
+                self.debug('skip authenticating, user and password are not set')
             # elif self.user:
             #     connection.admin.authenticate(name=self.user, mechanism='MONGODB-X509')
             server_status = connection.admin.command('serverStatus')
diff --git a/collectors/python.d.plugin/mongodb/mongodb.conf b/collectors/python.d.plugin/mongodb/mongodb.conf
index 2dded40ae..9f660f594 100644
--- a/collectors/python.d.plugin/mongodb/mongodb.conf
+++ b/collectors/python.d.plugin/mongodb/mongodb.conf
@@ -71,6 +71,16 @@
 #     user: 'username'       # the mongodb username to use
 #     pass: 'password'       # the mongodb password to use
 #
+# SSL connection parameters (https://api.mongodb.com/python/current/examples/tls.html):
+#
+#     ssl: yes                             # connect to the server using TLS
+#     ssl_cert_reqs: yes                   # require a certificate from the server when TLS is enabled
+#     ssl_ca_certs: '/path/to/ca.pem'      # use a specific set of CA certificates
+#     ssl_crlfile: '/path/to/crl.pem'      # use a certificate revocation lists
+#     ssl_certfile: '/path/to/client.pem'  # use a client certificate
+#     ssl_keyfile: '/path/to/key.pem'      # use a specific client certificate key
+#     ssl_pem_passphrase: 'passphrase'     # use a passphrase to decrypt encrypted private keys
+#
 
 # ----------------------------------------------------------------------
 # to connect to the mongodb on localhost, without a password: