diff options
author | Daniel Baumann <daniel.baumann@progress-linux.org> | 2024-04-15 16:27:18 +0000 |
---|---|---|
committer | Daniel Baumann <daniel.baumann@progress-linux.org> | 2024-04-15 16:27:18 +0000 |
commit | f7f20c3f5e0be02585741f5f54d198689ccd7866 (patch) | |
tree | 190d5e080f6cbcc40560b0ceaccfd883cb3faa01 /source/configuration/percentile_stats.rst | |
parent | Initial commit. (diff) | |
download | rsyslog-doc-f7f20c3f5e0be02585741f5f54d198689ccd7866.tar.xz rsyslog-doc-f7f20c3f5e0be02585741f5f54d198689ccd7866.zip |
Adding upstream version 8.2402.0+dfsg.upstream/8.2402.0+dfsg
Signed-off-by: Daniel Baumann <daniel.baumann@progress-linux.org>
Diffstat (limited to 'source/configuration/percentile_stats.rst')
-rw-r--r-- | source/configuration/percentile_stats.rst | 151 |
1 files changed, 151 insertions, 0 deletions
diff --git a/source/configuration/percentile_stats.rst b/source/configuration/percentile_stats.rst new file mode 100644 index 0000000..cd9781d --- /dev/null +++ b/source/configuration/percentile_stats.rst @@ -0,0 +1,151 @@ +Percentile Stats +================ + +The Percentile Stats component allows user to configure statistical namespaces +(called stats-buckets), which can then be used to record statistical values for +publishing periodic percentiles and summaries. + + +Percentile Stats Configuration +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Percentile-stats configuration involves a **two part setup**. First, define a ``percentile-stats`` bucket and +its attributes. Then, add statistics and observation values to the namespace, by making ``percentile_observation`` calls. + +percentile_stats(name="<bucket>", percentiles=[...], windowsize="<count>"...) (object) +----------------------------------------------------------------------------------------------------------- + +**Defines** the statistics bucket(identified by the bucket-name) and allows user to set some properties that control behavior of the bucket. + +.. code-block:: + + percentile_stats(name="bucket_name" percentiles=["95"] windowsize=1000) + +Parameters: + **name** <string literal, mandatory> : Bucket name of a set of statistics to sample. + + **percentiles** <array, mandatory> : A list of strings, with percentile statistic value between 1, 100 inclusive, user would like to publish to impstats. + This list of percentiles would apply for all statistics tracked under this bucket. + + **windowSize** <number, mandatory> : A max *sliding* window (FIFO) size - rounded *up* to the nearest power of 2, that is larger than the given window size. + Specifies the maximum number of observations stored over an impstats reporting interval. + This attribute would apply to all statistics tracked under this bucket. + + **delimiter** <string literal, default: "."> : A single character delimiter used in the published fully qualified statname. + This delimiter would apply to all statistics tracked under this bucket. + + +A definition setting all the parameters looks like: + +.. code-block:: + + percentile_stats(name="host_statistics" percentiles=["50", "95", "99"] windowsize="1000" delimiter="|") + + +percentile_observe("<bucket>", "<statname>", <value>) (function) +---------------------------------------------------------------- + +**Adds** a statistical sample to the statistic set identified by the <bucket>, and <statname>. +If the number of values exceed the defined **windowSize**, the earliest recorded value is dropped +and the new value is recorded. + +.. note:: + Sums and counts only include values found in the sliding window. + Min and Max summary values will include all values seen over an observation interval. + See :ref:`Reporting` below for more info on published summary statistics. + + + +Parameters: + **bucket name** <string literal, mandatory> : Name of the statistic bucket + + **stat name** <string literal, mandatory> : Name of the statistic to record (this name will be combined with a percentile and reported by impstats to identify the percentile) + + + **value** <expression resulting in an integer value, mandatory> : Value to record + +A ``percentile_observe`` call looks like: + +.. code-block:: + + set $.ret = percentile_observe("host_statistics", "msg_per_host", $.mycount); + + if ($.ret != 0) then { + .... + } + +``$.ret`` captures the error-code. It has value ``0`` when operation is successful and non-zero when it fails. It uses Rsyslog error-codes. + + +.. _Reporting: + +Reporting +^^^^^^^^^ + +The following example metrics would be reported by **impstats** + +Legacy format: + +.. code-block:: + + ... + global: origin=percentile host_statistics.new_metric_add=1 host_statistics.ops_overflow=0 + ... + host_statistics: origin=percentile.bucket msg_per_host|p95=1950 msg_per_host|p50=1500 msg_per_host|p99=1990 msg_per_host|window_min=1001 msg_per_host|window_max=2000 msg_per_host|window_sum=1500500 msg_per_host|window_count=1000 + ... + +Json(variants with the same structure are used in other Json based formats such as ``cee`` and ``json-elasticsearch``) format: + +.. code-block:: + + ... + { "name": "global", "origin": "percentile", "values": { "host_statistics.new_metric_add": 1, "host_statistics.ops_overflow": 0 } } + ... + { "name": "host_statistics", "origin": "percentile.bucket", "values": { "msg_per_host|p95": 1950, "msg_per_host|p50": 1500, "msg_per_host|p99": 1990, "msg_per_host|window_min": 1001, "msg_per_host|window_max": 2000, "msg_per_host|window_sum": 1500500, "msg_per_host|window_count": 1000 } } + ... + +In this case counters are encapsulated inside an object hanging off top-level-key ``values``. + +Fields +------ + +**global: origin=percentile.bucket**: + **new_metric_add**: Number of "new" metrics added (new counters created). + + **ops_overflow**: Number of operations ignored because number-of-counters-tracked has hit configured max-cardinality. + +**host_statistic: origin=percentile.bucket**: + **msg_per_host|<pXX>**: percentile value, identified by <stat name><delimiter><pXX>, where pXX is one of the requested ``percentiles`` requested. + + + **msg_per_host|window_min**: minimum recorded value in the statistical population window. Identified by ``<stat-name><delimiter>window_min`` + + + **msg_per_host|window_max**: maximum recorded value in the statistical population window. Identified by ``<stat-name><delimiter>window_max`` + + + **msg_per_host|window_sum**: sum of recorded values in the statistical population window. Identified by ``<stat-name>delimiter>window_sum`` + + + **msg_per_host|window_count**: count of recorded values in the statistical population window. Identified by ``<stat-name><delimiter>window_count`` + + +Implementation Details +---------------------- + +Percentile stats module uses a sliding window, sized according to the **windowSize** configuration parameter. +The percentile values are calculated, once every **impstats** interval. +Percentiles are calculated according to the standard “nearest-rank” method: + + + n = CEILING((P/100) x N) where: + +* n - index to percentile value +* P - percentile [1, 100] +* N - number of values recorded + + +.. note:: + In order to sort the values, a standard implementation of quicksort is used, which performs pretty well + on average. However quicksort quickly degrades when there are many repeated elements, thus it is best + to avoid repeated values if possible. |