diff options
Diffstat (limited to 'collectors/python.d.plugin/mongodb')
| -rw-r--r-- | collectors/python.d.plugin/mongodb/README.md | 265 | ||||
| -rw-r--r-- | collectors/python.d.plugin/mongodb/mongodb.chart.py | 70 | ||||
| -rw-r--r-- | collectors/python.d.plugin/mongodb/mongodb.conf | 10 |
3 files changed, 217 insertions, 128 deletions
diff --git a/collectors/python.d.plugin/mongodb/README.md b/collectors/python.d.plugin/mongodb/README.md index 3ddf137d7..fd694c1e5 100644 --- a/collectors/python.d.plugin/mongodb/README.md +++ b/collectors/python.d.plugin/mongodb/README.md @@ -3,135 +3,161 @@ Module monitor mongodb performance and health metrics **Requirements:** - * `python-pymongo` package v2.4+. -You need to install it manually. +- `python-pymongo` package v2.4+. +You need to install it manually. Number of charts depends on mongodb version, storage engine and other features (replication): -1. **Read requests**: - * query - * getmore (operation the cursor executes to get additional data from query) - -2. **Write requests**: - * insert - * delete - * update - -3. **Active clients**: - * readers (number of clients with read operations in progress or queued) - * writers (number of clients with write operations in progress or queued) - -4. **Journal transactions**: - * commits (count of transactions that have been written to the journal) - -5. **Data written to the journal**: - * volume (volume of data) - -6. **Background flush** (MMAPv1): - * average ms (average time taken by flushes to execute) - * last ms (time taken by the last flush) - -8. **Read tickets** (WiredTiger): - * in use (number of read tickets in use) - * available (number of available read tickets remaining) - -9. **Write tickets** (WiredTiger): - * in use (number of write tickets in use) - * available (number of available write tickets remaining) - -10. **Cursors**: - * opened (number of cursors currently opened by MongoDB for clients) - * timedOut (number of cursors that have timed) - * noTimeout (number of open cursors with timeout disabled) - -11. **Connections**: - * connected (number of clients currently connected to the database server) - * unused (number of unused connections available for new clients) - -12. **Memory usage metrics**: - * virtual - * resident (amount of memory used by the database process) - * mapped - * non mapped - -13. **Page faults**: - * page faults (number of times MongoDB had to request from disk) - -14. **Cache metrics** (WiredTiger): - * percentage of bytes currently in the cache (amount of space taken by cached data) - * percantage of tracked dirty bytes in the cache (amount of space taken by dirty data) - -15. **Pages evicted from cache** (WiredTiger): - * modified - * unmodified - -16. **Queued requests**: - * readers (number of read request currently queued) - * writers (number of write request currently queued) - -17. **Errors**: - * msg (number of message assertions raised) - * warning (number of warning assertions raised) - * regular (number of regular assertions raised) - * user (number of assertions corresponding to errors generated by users) - -18. **Storage metrics** (one chart for every database) - * dataSize (size of all documents + padding in the database) - * indexSize (size of all indexes in the database) - * storageSize (size of all extents in the database) - -19. **Documents in the database** (one chart for all databases) - * documents (number of objects in the database among all the collections) - -20. **tcmalloc metrics** - * central cache free - * current total thread cache - * pageheap free - * pageheap unmapped - * thread cache free - * transfer cache free - * heap size - -21. **Commands total/failed rate** - * count - * createIndex - * delete - * eval - * findAndModify - * insert - -22. **Locks metrics** (acquireCount metrics - number of times the lock was acquired in the specified mode) - * Global lock - * Database lock - * Collection lock - * Metadata lock - * oplog lock - -23. **Replica set members state** - * state - -24. **Oplog window** - * window (interval of time between the oldest and the latest entries in the oplog) - -25. **Replication lag** - * member (time when last entry from the oplog was applied for every member) - -26. **Replication set member heartbeat latency** - * member (time when last heartbeat was received from replica set member) - -### prerequisite -Create a read-only user for the netdata in the admin database. - -1. Authenticate as the admin user. +1. **Read requests**: + + - query + - getmore (operation the cursor executes to get additional data from query) + +2. **Write requests**: + + - insert + - delete + - update + +3. **Active clients**: + + - readers (number of clients with read operations in progress or queued) + - writers (number of clients with write operations in progress or queued) + +4. **Journal transactions**: + + - commits (count of transactions that have been written to the journal) + +5. **Data written to the journal**: + + - volume (volume of data) + +6. **Background flush** (MMAPv1): + + - average ms (average time taken by flushes to execute) + - last ms (time taken by the last flush) + +7. **Read tickets** (WiredTiger): + + - in use (number of read tickets in use) + - available (number of available read tickets remaining) + +8. **Write tickets** (WiredTiger): + + - in use (number of write tickets in use) + - available (number of available write tickets remaining) + +9. **Cursors**: + +- opened (number of cursors currently opened by MongoDB for clients) +- timedOut (number of cursors that have timed) +- noTimeout (number of open cursors with timeout disabled) + +10. **Connections**: + + - connected (number of clients currently connected to the database server) + - unused (number of unused connections available for new clients) + +11. **Memory usage metrics**: + + - virtual + - resident (amount of memory used by the database process) + - mapped + - non mapped + +12. **Page faults**: + + - page faults (number of times MongoDB had to request from disk) + +13. **Cache metrics** (WiredTiger): + + - percentage of bytes currently in the cache (amount of space taken by cached data) + - percantage of tracked dirty bytes in the cache (amount of space taken by dirty data) + +14. **Pages evicted from cache** (WiredTiger): + + - modified + - unmodified + +15. **Queued requests**: + + - readers (number of read request currently queued) + - writers (number of write request currently queued) + +16. **Errors**: + + - msg (number of message assertions raised) + - warning (number of warning assertions raised) + - regular (number of regular assertions raised) + - user (number of assertions corresponding to errors generated by users) + +17. **Storage metrics** (one chart for every database) + + - dataSize (size of all documents + padding in the database) + - indexSize (size of all indexes in the database) + - storageSize (size of all extents in the database) + +18. **Documents in the database** (one chart for all databases) + +- documents (number of objects in the database among all the collections) + +19. **tcmalloc metrics** + + - central cache free + - current total thread cache + - pageheap free + - pageheap unmapped + - thread cache free + - transfer cache free + - heap size + +20. **Commands total/failed rate** + + - count + - createIndex + - delete + - eval + - findAndModify + - insert + +21. **Locks metrics** (acquireCount metrics - number of times the lock was acquired in the specified mode) + + - Global lock + - Database lock + - Collection lock + - Metadata lock + - oplog lock + +22. **Replica set members state** + + - state + +23. **Oplog window** + + - window (interval of time between the oldest and the latest entries in the oplog) + +24. **Replication lag** + + - member (time when last entry from the oplog was applied for every member) + +25. **Replication set member heartbeat latency** + + - member (time when last heartbeat was received from replica set member) + +## prerequisite + +Create a read-only user for Netdata in the admin database. + +1. Authenticate as the admin user. ``` use admin db.auth("admin", "<MONGODB_ADMIN_PASSWORD>") ``` -2. Create a user. +2. Create a user. ``` # MongoDB 2.x. @@ -161,11 +187,10 @@ local: port : 27017 user : 'netdata' pass : 'netdata' - ``` If no configuration is given, module will attempt to connect to mongodb daemon on `127.0.0.1:27017` address --- -[]() +[](<>) diff --git a/collectors/python.d.plugin/mongodb/mongodb.chart.py b/collectors/python.d.plugin/mongodb/mongodb.chart.py index 5db48cb12..0dbe82ff9 100644 --- a/collectors/python.d.plugin/mongodb/mongodb.chart.py +++ b/collectors/python.d.plugin/mongodb/mongodb.chart.py @@ -3,6 +3,8 @@ # Author: ilyam8 # SPDX-License-Identifier: GPL-3.0-or-later +import ssl + from copy import deepcopy from datetime import datetime from sys import exc_info @@ -418,18 +420,31 @@ CHARTS = { } } +DEFAULT_HOST = '127.0.0.1' +DEFAULT_PORT = 27017 +DEFAULT_TIMEOUT = 100 +DEFAULT_AUTHDB = 'admin' + +CONN_PARAM_HOST = 'host' +CONN_PARAM_PORT = 'port' +CONN_PARAM_SERVER_SELECTION_TIMEOUT_MS = 'serverselectiontimeoutms' +CONN_PARAM_SSL_SSL = 'ssl' +CONN_PARAM_SSL_CERT_REQS = 'ssl_cert_reqs' +CONN_PARAM_SSL_CA_CERTS = 'ssl_ca_certs' +CONN_PARAM_SSL_CRL_FILE = 'ssl_crlfile' +CONN_PARAM_SSL_CERT_FILE = 'ssl_certfile' +CONN_PARAM_SSL_KEY_FILE = 'ssl_keyfile' +CONN_PARAM_SSL_PEM_PASSPHRASE = 'ssl_pem_passphrase' + class Service(SimpleService): def __init__(self, configuration=None, name=None): SimpleService.__init__(self, configuration=configuration, name=name) self.order = ORDER[:] self.definitions = deepcopy(CHARTS) - self.authdb = self.configuration.get('authdb', 'admin') + self.authdb = self.configuration.get('authdb', DEFAULT_AUTHDB) self.user = self.configuration.get('user') self.password = self.configuration.get('pass') - self.host = self.configuration.get('host', '127.0.0.1') - self.port = self.configuration.get('port', 27017) - self.timeout = self.configuration.get('timeout', 100) self.metrics_to_collect = deepcopy(DEFAULT_METRICS) self.connection = None self.do_replica = None @@ -705,14 +720,53 @@ class Service(SimpleService): return data - def _create_connection(self): - conn_vars = {'host': self.host, 'port': self.port} + def build_ssl_connection_params(self): + conf = self.configuration + + def cert_req(v): + if v is None: + return None + if not v: + return ssl.CERT_NONE + return ssl.CERT_REQUIRED + + ssl_params = { + CONN_PARAM_SSL_SSL: conf.get(CONN_PARAM_SSL_SSL), + CONN_PARAM_SSL_CERT_REQS: cert_req(conf.get(CONN_PARAM_SSL_CERT_REQS)), + CONN_PARAM_SSL_CA_CERTS: conf.get(CONN_PARAM_SSL_CA_CERTS), + CONN_PARAM_SSL_CRL_FILE: conf.get(CONN_PARAM_SSL_CRL_FILE), + CONN_PARAM_SSL_CERT_FILE: conf.get(CONN_PARAM_SSL_CERT_FILE), + CONN_PARAM_SSL_KEY_FILE: conf.get(CONN_PARAM_SSL_KEY_FILE), + CONN_PARAM_SSL_PEM_PASSPHRASE: conf.get(CONN_PARAM_SSL_PEM_PASSPHRASE), + } + + ssl_params = dict((k, v) for k, v in ssl_params.items() if v is not None) + + return ssl_params + + def build_connection_params(self): + conf = self.configuration + params = { + CONN_PARAM_HOST: conf.get(CONN_PARAM_HOST, DEFAULT_HOST), + CONN_PARAM_PORT: conf.get(CONN_PARAM_PORT, DEFAULT_PORT), + } if hasattr(MongoClient, 'server_selection_timeout'): - conn_vars.update({'serverselectiontimeoutms': self.timeout}) + params[CONN_PARAM_SERVER_SELECTION_TIMEOUT_MS] = conf.get('timeout', DEFAULT_TIMEOUT) + + params.update(self.build_ssl_connection_params()) + return params + + def _create_connection(self): + params = self.build_connection_params() + self.debug('creating connection, connection params: {0}'.format(sorted(params))) + try: - connection = MongoClient(**conn_vars) + connection = MongoClient(**params) if self.user and self.password: + self.debug('authenticating, user: {0}, password: {1}'.format(self.user, self.password)) getattr(connection, self.authdb).authenticate(name=self.user, password=self.password) + else: + self.debug('skip authenticating, user and password are not set') # elif self.user: # connection.admin.authenticate(name=self.user, mechanism='MONGODB-X509') server_status = connection.admin.command('serverStatus') diff --git a/collectors/python.d.plugin/mongodb/mongodb.conf b/collectors/python.d.plugin/mongodb/mongodb.conf index 2dded40ae..9f660f594 100644 --- a/collectors/python.d.plugin/mongodb/mongodb.conf +++ b/collectors/python.d.plugin/mongodb/mongodb.conf @@ -71,6 +71,16 @@ # user: 'username' # the mongodb username to use # pass: 'password' # the mongodb password to use # +# SSL connection parameters (https://api.mongodb.com/python/current/examples/tls.html): +# +# ssl: yes # connect to the server using TLS +# ssl_cert_reqs: yes # require a certificate from the server when TLS is enabled +# ssl_ca_certs: '/path/to/ca.pem' # use a specific set of CA certificates +# ssl_crlfile: '/path/to/crl.pem' # use a certificate revocation lists +# ssl_certfile: '/path/to/client.pem' # use a client certificate +# ssl_keyfile: '/path/to/key.pem' # use a specific client certificate key +# ssl_pem_passphrase: 'passphrase' # use a passphrase to decrypt encrypted private keys +# # ---------------------------------------------------------------------- # to connect to the mongodb on localhost, without a password: |