diff options
Diffstat (limited to 'source/configuration/modules/impstats.rst')
| -rw-r--r-- | source/configuration/modules/impstats.rst | 405 |
1 files changed, 405 insertions, 0 deletions
diff --git a/source/configuration/modules/impstats.rst b/source/configuration/modules/impstats.rst new file mode 100644 index 0000000..ca95609 --- /dev/null +++ b/source/configuration/modules/impstats.rst @@ -0,0 +1,405 @@ +*********************************************************** +impstats: Generate Periodic Statistics of Internal Counters +*********************************************************** + +=========================== =========================================================================== +**Module Name:** **impstats** +**Author:** `Rainer Gerhards <https://rainer.gerhards.net/>`_ <rgerhards@adiscon.com> +=========================== =========================================================================== + + +Purpose +======= + +This module provides periodic output of rsyslog internal counters. + +The set of available counters will be output as a set of syslog +messages. This output is periodic, with the interval being configurable +(default is 5 minutes). Be sure that your configuration records the +counter messages (default is syslog.=info). Besides logging to the +regular syslog stream, the module can also be configured to write +statistics data into a (local) file. + +When logging to the regular syslog stream, impstats records are emitted +just like regular log messages. As such, +counters increase when processing these messages. This must be taken into +consideration when testing and troubleshooting. + +Note that loading this module has some impact on rsyslog performance. +Depending on settings, this impact may be noticeable for high-load +environments, but in general the overhead is pretty light. + +**Note that there is a** `rsyslog statistics online +analyzer <http://www.rsyslog.com/impstats-analyzer/>`_ **available.** It +can be given a impstats-generated file and will return problems it +detects. Note that the analyzer cannot replace a human in getting things +right, but it is expected to be a good aid in starting to understand and +gain information from the pstats logs. + +The rsyslog website has an overview of available `rsyslog +statistic counters <http://rsyslog.com/rsyslog-statistic-counter/>`_. +When browsing this page, please be sure to take note of which rsyslog +version is required to provide a specific counter. Counters are +continuously being added, and older versions do not support everything. + + +Notable Features +================ + +- :ref:`impstats-statistic-counter` + + + + +Configuration Parameters +======================== + +The configuration parameters for this module are designed for tailoring +the method and process for outputting the rsyslog statistics to file. + +.. note:: + + Parameter names are case-insensitive. + +.. note:: + + This module supports module parameters, only. + + +Module Parameters +----------------- + +Interval +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "300", "no", "none" + +Sets the interval, in **seconds** at which messages are generated. +Please note that the actual interval may be a bit longer. We do not +try to be precise and so the interval is actually a sleep period +which is entered after generating all messages. So the actual +interval is what is configured here plus the actual time required to +generate messages. In general, the difference should not really +matter. + + +Facility +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "5", "no", "none" + +The numerical syslog facility code to be used for generated +messages. Default is 5 (syslog). This is useful for filtering +messages. + + +Severity +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "6", "no", "none" + +The numerical syslog severity code to be used for generated +messages. Default is 6 (info).This is useful for filtering messages. + + +ResetCounters +^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +When set to "on", counters are automatically reset after they are +emitted. In that case, the contain only deltas to the last value +emitted. When set to "off", counters always accumulate their values. +Note that in auto-reset mode not all counters can be reset. Some +counters (like queue size) are directly obtained from internal object +and cannot be modified. Also, auto-resetting introduces some +additional slight inaccuracies due to the multi-threaded nature of +rsyslog and the fact that for performance reasons it cannot serialize +access to counter variables. As an alternative to auto-reset mode, +you can use rsyslog's statistics manipulation scripts to create delta +values from the regular statistic logs. This is the suggested method +if deltas are not necessarily needed in real-time. + + +Format +^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "legacy", "no", "none" + +.. versionadded:: 8.16.0 + +Specifies the format of emitted stats messages. The default of +"legacy" is compatible with pre v6-rsyslog. The other options provide +support for structured formats (note the "cee" is actually "project +lumberjack" logging). + +The json-elasticsearch format supports the broken ElasticSearch +JSON implementation. ES 2.0 no longer supports valid JSON and +disallows dots inside names. The "json-elasticsearch" format +option replaces those dots by the bang ("!") character. So +"discarded.full" becomes "discarded!full". +Options: json/json-elasticsearch/cee/legacy + + +log.syslog +^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "on", "no", "none" + +This is a boolean setting specifying if data should be sent to the +usual syslog stream. This is useful if custom formatting or more +elaborate processing is desired. However, output is placed under the +same restrictions as regular syslog data, especially in regard to the +queue position (stats data may sit for an extended period of time in +queues if they are full). If set "off", then you cannot bind the module to +ruleset. + + +log.file +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +If specified, statistics data is written to the specified file. For +robustness, this should be a local file. The file format cannot be +customized, it consists of a date header, followed by a colon, +followed by the actual statistics record, all on one line. Only very +limited error handling is done, so if things go wrong stats records +will probably be lost. Logging to file an be a useful alternative if +for some reasons (e.g. full queues) the regular syslog stream method +shall not be used solely. Note that turning on file logging does NOT +turn off syslog logging. If that is desired log.syslog="off" must be +explicitly set. + + +Ruleset +^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "none", "no", "none" + +Binds the listener to a specific :doc:`ruleset <../../concepts/multi_ruleset>`. + +**Note** that setting ``ruleset`` and ``log.syslog="off"`` are mutually +exclusive because syslog stream processing must be enabled to use a ruleset. + + +Bracketing +^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +.. versionadded:: 8.4.1 + +This is a utility setting for folks who post-process impstats logs +and would like to know the begin and end of a block of statistics. +When "bracketing" is set to "on", impstats issues a "BEGIN" message +before the first counter is issued, then all counter values +are issued, and then an "END" message follows. As such, if and only if messages +are kept in sequence, a block of stats counts can easily be identified +by those BEGIN and END messages. + +**Note well:** in general, sequence of syslog messages is **not** +strict and is not ordered in sequence of message generation. There +are various occasion that can cause message reordering, some +examples are: + +* using multiple threads +* using UDP forwarding +* using relay systems, especially with buffering enabled +* using disk-assisted queues + +This is not a problem with rsyslog, but rather the way a concurrent +world works. For strict order, a specific order predicate (e.g. a +sufficiently fine-grained timestamp) must be used. + +As such, BEGIN and END records may actually indicate the begin and +end of a block of statistics - or they may *not*. Any order is possible +in theory. So the bracketing option does not in all cases work as +expected. This is the reason why it is turned off by default. + +*However*, bracketing may still be useful for many use cases. First +and foremost, while there are many scenarios in which messages become +reordered, in practice it happens relatively seldom. So most of the +time the statistics records will come in as expected and actually +will be bracketed by the BEGIN and END messages. Consequently, if +an application can handle occasional out-of-order delivery (e.g. by +graceful degradation), bracketing may actually be a great solution. +It is, however, very important to know and +handle out of order delivery. For most real-world deployments, +a good way to handle it is to ignore unexpected +records and use the previous values for ones missing in the current +block. To guard against two or more blocks being mixed, it may also +be a good idea to never reset a value to a lower bound, except when +that lower bound is seen consistently (which happens due to a +restart). Note that such lower bound logic requires *resetCounters* +to be set to off. + + +.. _impstats-statistic-counter: + +Statistic Counter +================= + +The impstats plugin gathers some internal :doc:`statistics <../rsyslog_statistic_counter>`. +They have different names depending on the actual statistics. Obviously, they do not +relate to the plugin itself but rather to a broader object – most notably the +rsyslog process itself. The "resource-usage" counter maintains process +statistics. They base on the getrusage() system call. The counters are +named like getrusage returned data members. So for details, looking them +up in "man getrusage" is highly recommended, especially as value may be +different depending on the platform. A getrusage() call is done immediately +before the counter is emitted. The following individual counters are +maintained: + +- ``utime`` - this is the user time in microseconds (thus the timeval structure combined) +- ``stime`` - again, time given in microseconds +- ``maxrss`` +- ``minflt`` +- ``majflt`` +- ``inblock`` +- ``outblock`` +- ``nvcsw`` +- ``nivcsw`` +- ``openfiles`` - number of file handles used by rsyslog; includes actual files, sockets and others + + +Caveats/Known Bugs +================== + +- This module MUST be loaded right at the top of rsyslog.conf, + otherwise stats may not get turned on in all places. + + +Examples +======== + +Load module, send stats data to syslog stream +--------------------------------------------- + +This activates the module and records messages to /var/log/rsyslog-stats +in 10 minute intervals: + +.. code-block:: none + + module(load="impstats" + interval="600" + severity="7") + + # to actually gather the data: + syslog.=debug /var/log/rsyslog-stats + + +Load module, send stats data to local file +------------------------------------------ + +Here, the default interval of 5 minutes is used. However, this time, stats +data is NOT emitted to the syslog stream but to a local file instead. + +.. code-block:: none + + module(load="impstats" + interval="600" + severity="7" + log.syslog="off" + # need to turn log stream logging off! + log.file="/path/to/local/stats.log") + + +Load module, send stats data to local file and syslog stream +------------------------------------------------------------ + +Here we log to both the regular syslog log stream as well as a +file. Within the log stream, we forward the data records to another +server: + +.. code-block:: none + + module(load="impstats" + interval="600" + severity="7" + log.file="/path/to/local/stats.log") + + syslog.=debug @central.example.net + + +Explanation of output +===================== + +Example output for illustration:: + + Sep 17 11:43:49 localhost rsyslogd-pstats: imuxsock: submitted=16 + Sep 17 11:43:49 localhost rsyslogd-pstats: main Q: size=1 enqueued=2403 full=0 maxqsize=2 + +Explanation: + +All objects are shown in the results with a separate counter, one object per +line. + +Line 1: shows details for + +- ``imuxsock``, an object +- ``submitted=16``, a counter showing that 16 messages were received by the + imuxsock object. + +Line 2: shows details for the main queue: + +- ``main Q``, an object +- ``size``, messages in the queue +- ``enqueued``, all received messages thus far +- ``full``, how often was the queue was full +- ``maxqsize``, the maximum amount of messages that have passed through the + queue since rsyslog was started + +See Also +======== + +- `rsyslog statistics + counter <http://www.rsyslog.com/rsyslog-statistic-counter/>`_ +- `impstats delayed or + lost <http://www.rsyslog.com/impstats-delayed-or-lost/>`_ - cause and + cure |