Adding upstream version 8.2402.0+dfsg.upstream/8.2402.0+dfsg

Signed-off-by: Daniel Baumann <daniel.baumann@progress-linux.org>

1 files changed, 607 insertions, 0 deletions

diff --git a/source/rainerscript/queue_parameters.rst b/source/rainerscript/queue_parameters.rst
new file mode 100644
index 0000000..19e850c
--- /dev/null
+++ b/source/rainerscript/queue_parameters.rst
@@ -0,0 +1,607 @@
+************************
+General Queue Parameters
+************************
+
+===========================  ===========================================================================
+**Authors:**                  `Rainer Gerhards <https://rainer.gerhards.net/>`_ <rgerhards@adiscon.com>;
+===========================  ===========================================================================
+
+
+Usage
+=====
+
+Queue parameters can be used together with the following statements:
+
+- :doc:`action() <../configuration/actions>`
+- ruleset()
+- main\_queue()
+
+Queues need to be configured in the action or ruleset it should affect.
+If nothing is configured, default values will be used. Thus, the default
+ruleset has only the default main queue. Specific Action queues are not
+set up by default.
+
+To fully understand queue parameters and how they interact, be sure to
+read the :doc:`queues <../concepts/queues>` documentation.
+
+
+Configuration Parameters
+========================
+
+.. note::
+
+   As with other configuration objects, parameters for this
+   object are case-insensitive.
+
+
+queue.filename
+--------------
+
+.. csv-table::
+   :header: "type", "default", "mandatory", "|FmtObsoleteName| directive"
+   :widths: auto
+   :class: parameter-table
+
+   "word", "none", "no", "``$ActionQueueFileName``"
+
+File name to be used for the queue files. If specified, this parameter
+enables disk-assisted queue functionality. If *not* specified,
+the queue will operate without saving the queue to disk, either
+during its operation or when shut down. See the separate
+``queue.saveonshutdown`` parameter to configure that option.
+Please note that this is actually just the file name. A directory
+can NOT be specified in this parameter. If the files shall be
+created in a specific directory, specify ``queue.spoolDirectory``
+for this. The filename is used to build to complete path for queue
+files.
+
+
+queue.spoolDirectory
+--------------------
+
+.. csv-table::
+   :header: "type", "default", "mandatory", "|FmtObsoleteName| directive"
+   :widths: auto
+   :class: parameter-table
+
+   "word", "none", "no", "none"
+
+This is the directory into which queue files will be stored. Note
+that the directory must exist, it is NOT automatically created by
+rsyslog. If no spoolDirectory is specified, the work directory is
+used.
+
+
+queue.size
+----------
+
+.. csv-table::
+   :header: "type", "default", "mandatory", "|FmtObsoleteName| directive"
+   :widths: auto
+   :class: parameter-table
+
+   "integer", "1000/50000", "no", "``$ActionQueueSize``"
+
+This is the maximum size of the queue in number of messages. Note
+that setting the queue size to very small values (roughly below 100
+messages) is not supported and can lead to unpredictable results.
+For more information on the current status of this restriction see
+the `rsyslog FAQ: "lower bound for queue
+sizes" <http://www.rsyslog.com/lower-bound-for-queue-sizes/>`_.
+
+The default depends on queue type and rsyslog version, if you need
+a specific value, please specify it. Otherwise rsyslog selects what
+it considers appropriate for the version in question. In rsyslog
+rsyslog 8.30.0, for example, ruleset queues have a default size
+of 50000 and action queues which are configured to be non-direct
+have a size of 1000.
+
+
+queue.dequeueBatchSize
+----------------------
+
+.. csv-table::
+   :header: "type", "default", "mandatory", "|FmtObsoleteName| directive"
+   :widths: auto
+   :class: parameter-table
+
+   "integer", "128/1024", "no", "``$ActionQueueDequeueBatchSize``"
+
+Specifies the maximum batch size for dequeue operations. This setting affects performance.
+As a rule of thumb, larger batch sizes (up to a environment-induced upper limit)
+provide better performance. For the average system, there usually should be no need
+to adjust batch sizes as the defaults are sufficient. The default for ruleset queues
+is 1024 and for action queues 128.
+
+Note that this only specifies the **maximum** batch size. Batches will be slower if
+rsyslog does not have as many messages inside the queue at time of dequeuing it.
+If you want to set a minimum Batch size as well, you can use `queue.minDequeueBatchSize`.
+
+
+queue.minDequeueBatchSize
+-------------------------
+
+.. csv-table::
+   :header: "type", "default", "mandatory", "|FmtObsoleteName| directive"
+   :widths: auto
+   :class: parameter-table
+
+   "integer", "0", "no", "none"
+
+Specifies the **minimum** batch size for dequeue operations. This setting is especially
+useful with outputs like ElasticSearch or ClickHouse, where you want to limit the
+number of HTTP requests. With this setting, the queue engine waits up to
+`queue.minDequeueBatchSize.timeout` milliseconds when there are fewer messages
+currently queued. Note that the minimum batch size cannot be larger than the
+configured maximum batch size. If so, it is automatically adjusted to
+match the maximum. So if in doubt, you need to specify both parameters.
+
+
+queue.minDequeueBatchSize.timeout
+---------------------------------
+
+.. csv-table::
+   :header: "type", "default", "mandatory", "|FmtObsoleteName| directive"
+   :widths: auto
+   :class: parameter-table
+
+   "integer", "1000", "no", "none"
+
+This parameter is only meaningful if use together with `queue.minDequeueBatchSize`,
+otherwise it is ignored. It specifies the amount of time (in milliseconds) rsyslogs
+waits for new
+messages so that the minimum batch size can be reached. After this period, the
+batch is processed, *even if it is below minimum size*. This capability exists to
+prevent having messages stalled in an incomplete batch just because no new
+messages arrive. We would expect that it usually makes little sense to set it
+to higher than 60.000 (60 seconds), but this is permitted. Just be warned that
+this potentially delays log processing for that long.
+
+
+queue.maxDiskSpace
+------------------
+
+.. csv-table::
+   :header: "type", "default", "mandatory", "|FmtObsoleteName| directive"
+   :widths: auto
+   :class: parameter-table
+
+   "integer", "0", "no", "``$ActionQueueMaxDiskSpace``"
+
+The maximum size that all queue files together will use on disk. Note
+that the actual size may be slightly larger than the configured max,
+as rsyslog never writes partial queue records.
+
+
+queue.highWatermark
+-------------------
+
+.. csv-table::
+   :header: "type", "default", "mandatory", "|FmtObsoleteName| directive"
+   :widths: auto
+   :class: parameter-table
+
+   "integer", "90% of queue.size", "no", "``$ActionQueueHighWaterMark``"
+
+This applies to disk-assisted queues, only. When the queue fills up
+to this number of messages, the queue begins to spool messages to
+disk. Please note that this should not happen as part of usual
+processing, because disk queue mode is very considerably slower than
+in-memory queue mode. Going to disk should be reserved for cases
+where an output action destination is offline for some period.
+
+
+queue.lowWatermark
+------------------
+
+.. csv-table::
+   :header: "type", "default", "mandatory", "|FmtObsoleteName| directive"
+   :widths: auto
+   :class: parameter-table
+
+   "integer", "70% of queue.size", "no", "``$ActionQueueLowWaterMark``"
+
+This applies to disk-assisted queues, only. When the high watermark is
+reached, the queue will write data to disk. It does so until the low
+watermark is reached, then the queue reverts back to in-memory mode.
+
+
+queue.fullDelaymark
+-------------------
+
+.. csv-table::
+   :header: "type", "default", "mandatory", "|FmtObsoleteName| directive"
+   :widths: auto
+   :class: parameter-table
+
+   "integer", "97% of queue.size", "no", "none"
+
+Number of messages when the queue should block delayable messages.
+Messages are NO LONGER PROCESSED until the queue has sufficient space
+again. If a message is delayable depends on the input. For example,
+messages received via imtcp are delayable (because TCP can push back),
+but those received via imudp are not (as UDP does not permit a push back).
+The intent behind this setting is to leave some space in an almost-full
+queue for non-delayable messages, which would be lost if the queue runs
+out of space. Please note that if you use a DA queue, setting the
+fulldelaymark BELOW the highwatermark makes the queue never activate
+disk mode for delayable inputs. So this is probably not what you want.
+
+
+queue.lightDelayMark
+--------------------
+
+.. csv-table::
+   :header: "type", "default", "mandatory", "|FmtObsoleteName| directive"
+   :widths: auto
+   :class: parameter-table
+
+   "integer", "70% of queue.size", "no", "none"
+
+If this mark is reached the sender will be throttled if possible. The
+main idea to do this is leave some space inside the queue for inputs
+like UDP which cannot be throttled - and so any data arriving at
+"queue full" would be discarded.
+
+If the special value `0` is used, `queue.LightDelayMark` will be set
+to the value of `queue.size`. This effectively **disables** light delay
+functionality. This is useful if a queue is not used by non-delayable
+inputs like UDP. The special value was introduced in rsyslog 8.1904.0
+and is **not** available in earlier versions. There, you can achieve the
+same result by setting `queue.LightDelayMark` to a very large value.
+
+
+queue.discardMark
+-----------------
+
+.. csv-table::
+   :header: "type", "default", "mandatory", "|FmtObsoleteName| directive"
+   :widths: auto
+   :class: parameter-table
+
+   "integer", "80% of queue.size", "no", "``$ActionQueueDiscardMark``"
+
+Specifies the threshold at which rsyslog begins to discard less important
+messages. To define which messages should be discarded use the
+queue.discardseverity parameter.
+
+
+queue.discardSeverity
+---------------------
+
+.. csv-table::
+   :header: "type", "default", "mandatory", "|FmtObsoleteName| directive"
+   :widths: auto
+   :class: parameter-table
+
+   "integer", "8", "no", "``$ActionQueueDiscardSeverity``"
+
+As soon as the threshold of the parameter queue.discardMark is reached
+incoming as well as queued messages with a priority equal or lower than
+specified will be erased. With the default no messages will be erased.
+You have to specify a numeric severity value for this parameter.
+
+
+queue.checkpointInterval
+------------------------
+
+.. csv-table::
+   :header: "type", "default", "mandatory", "|FmtObsoleteName| directive"
+   :widths: auto
+   :class: parameter-table
+
+   "integer", "0", "no", "``$ActionQueueCheckpointInterval``"
+
+Disk queues by default do not update housekeeping structures every time
+the queue writes to disk. This is for performance reasons. In the event of failure,
+data will still be lost (except when data is mangled via the file structures).
+However, disk queues can be set to write bookkeeping information on checkpoints
+(every n records), so that this can be made ultra-reliable, too. If the
+checkpoint interval is set to one, no data can be lost, but the queue is
+exceptionally slow.
+
+
+queue.syncqueuefiles
+--------------------
+
+.. csv-table::
+   :header: "type", "default", "mandatory", "|FmtObsoleteName| directive"
+   :widths: auto
+   :class: parameter-table
+
+   "binary", "off", "no", "``$ActionQueueSyncQueueFiles``"
+
+Disk-based queues can be made very reliable by issuing a (f)sync after each
+write operation. This happens when you set the parameter to "on".
+Activating this option has a performance penalty, so it should not
+be turned on without a good reason. Note that the penalty also depends on
+*queue.checkpointInterval* frequency.
+
+
+queue.samplingInterval
+----------------------
+
+.. csv-table::
+   :header: "type", "default", "mandatory", "|FmtObsoleteName| directive"
+   :widths: auto
+   :class: parameter-table
+
+   "integer", "0", "no", "none"
+
+.. versionadded:: 8.23.0
+
+This option allows queues to be populated by events produced at a specific interval.
+It provides a way to sample data each N events, instead of processing all, in order
+to reduce resources usage (disk, bandwidth...)
+
+
+queue.type
+----------
+
+.. csv-table::
+   :header: "type", "default", "mandatory", "|FmtObsoleteName| directive"
+   :widths: auto
+   :class: parameter-table
+
+   "word", "Direct", "no", "``$ActionQueueType``"
+
+Specifies the type of queue that will be used. Possible options are "FixedArray",
+"LinkedList", "Direct" or "Disk". For more information read the documentation
+for :doc:`queues <../concepts/queues>`.
+
+
+queue.workerThreads
+-------------------
+
+.. csv-table::
+   :header: "type", "default", "mandatory", "|FmtObsoleteName| directive"
+   :widths: auto
+   :class: parameter-table
+
+   "integer", "1", "no", "``$ActionQueueWorkerThreads``"
+
+Specifies the maximum number of worker threads that can be run parallel.
+
+
+queue.workerThreadMinimumMessages
+---------------------------------
+
+.. csv-table::
+   :header: "type", "default", "mandatory", "|FmtObsoleteName| directive"
+   :widths: auto
+   :class: parameter-table
+
+   "integer", "queue.size/queue.workerthreads", "no", "``$ActionQueueWorkerThreadMinimumMessages``"
+
+Specify the number of messages a worker thread is processing before another
+worker thread is created. This number is limited by parameter queue.workerThreads.
+For example if this parameter is set to 200 and in the queue are 201 messages a
+second worker thread will be created.
+
+
+queue.timeoutWorkerthreadShutdown
+---------------------------------
+
+.. csv-table::
+   :header: "type", "default", "mandatory", "|FmtObsoleteName| directive"
+   :widths: auto
+   :class: parameter-table
+
+   "integer", "60000", "no", "``$ActionQueueTimeoutWorkerthreadShutdown``"
+
+After starting a worker thread, it will process messages until there are no
+messages for him to process. This parameter specifies the time the worker
+thread has to be inactive before it times out.
+The parameter must be specified in milliseconds. Which means the default of
+60000 is 1 minute.
+
+
+queue.timeoutshutdown
+---------------------
+
+.. csv-table::
+   :header: "type", "default", "mandatory", "|FmtObsoleteName| directive"
+   :widths: auto
+   :class: parameter-table
+
+   "integer", "10/1500", "no", "``$ActionQueueTimeoutShutdown``"
+
+If a queue that still contains messages is terminated it will wait the
+specified time interval for the worker thread to finish.
+The time is specified in milliseconds (1000ms is 1sec).
+Default for action queues is 10, for ruleset queues it is 1500.
+
+
+queue.timeoutActionCompletion
+-----------------------------
+
+.. csv-table::
+   :header: "type", "default", "mandatory", "|FmtObsoleteName| directive"
+   :widths: auto
+   :class: parameter-table
+
+   "integer", "1000", "no", "``$ActionQueueTimeoutActionCompletion``"
+
+When a queue is terminated, the timeout shutdown is over and there is
+still data in the queue, the queue will finish the current data element
+and then terminate. This parameter specifies the timeout for processing
+this last element.
+Parameter is specified in milliseconds (1000ms is 1sec).
+
+
+queue.timeoutEnqueue
+--------------------
+
+.. csv-table::
+   :header: "type", "default", "mandatory", "|FmtObsoleteName| directive"
+   :widths: auto
+   :class: parameter-table
+
+   "integer", "2000", "no", "``$ActionQueueTimeoutEnqueue``"
+
+This timeout value is used when the queue is full. If rsyslog cannot
+enqueue a message within the timeout period, the message is discarded.
+Note that this is setting of last resort (assuming defaults are used
+for the queue settings or proper parameters are set): all delayable
+inputs (like imtcp or imfile) have already been pushed back at this
+stage. Also, discarding of lower priority messages (if configured) has
+already happened. So we run into one of these situations if we do not
+timeout quickly enough:
+
+* if using imuxsock and no systemd journal is involved, the system
+  would become unresponsive and most probably a hard reset would be
+  required.
+* if using imuxsock with imjournal forwarding is active, messages are
+  lost because the journal discards them (more aggressive than rsyslog does)
+* if using imjournal, the journal will buffer messages. If journal
+  runs out of configured space, messages will be discarded. So in this
+  mode discarding is moved to a bit later place.
+* other non-delayable sources like imudp will also loose messages
+
+So this setting is provided in order to guard against problematic situations,
+which always will result either in message loss or system hang. For
+action queues, one may debate if it would be better to overflow rapidly
+to the main queue. If so desired, this is easy to accomplish by setting
+a very large timeout value. The same, of course, is true for the main
+queue, but you have been warned if you do so!
+
+In some other words, you can consider this scenario, using default values.
+With all progress blocked (unable to deliver a message):
+
+* all delayable inputs (tcp, relp, imfile, imjournal, etc) will block
+  indefinantly (assuming queue.lightdelaymark and queue.fulldelaymark
+  are set sensible, which they are by default).
+* imudp will be loosing messages because the OS will be dropping them
+* messages arriving via UDP or imuxsock that do make it to rsyslog,
+  and that are a severity high enough to not be filtered by
+  discardseverity, will block for 2 seconds trying to put the message in
+  the queue (in the hope that something happens to make space in the
+  queue) and then be dropped to avoid blocking the machine permanently.
+
+  Then the next message to be processed will also be tried for 2 seconds, etc.
+
+* If this is going into an action queue, the log message will remain
+  in the main queue during these 2 seconds, and additional logs that
+  arrive will accumulate behind this in the main queue.
+
+
+queue.maxFileSize
+-----------------
+
+.. csv-table::
+   :header: "type", "default", "mandatory", "|FmtObsoleteName| directive"
+   :widths: auto
+   :class: parameter-table
+
+   "integer", "1m/16m", "no", "``$ActionQueueMaxFileSize``"
+
+Specifies the maximum size for the disk-assisted queue file.
+Parameter can be specified in Mebibyte or Gibibyte, default for action
+queues is 1m and for ruleset queues 16m (1m = 1024*1024).
+
+
+queue.saveOnShutdown
+--------------------
+
+.. csv-table::
+   :header: "type", "default", "mandatory", "|FmtObsoleteName| directive"
+   :widths: auto
+   :class: parameter-table
+
+   "binary", "off", "no", "``$ActionQueueSaveOnShutdown``"
+
+This parameter specifies if data should be saved at shutdown.
+
+
+queue.dequeueSlowDown
+---------------------
+
+.. csv-table::
+   :header: "type", "default", "mandatory", "|FmtObsoleteName| directive"
+   :widths: auto
+   :class: parameter-table
+
+   "integer", "0", "no", "``$ActionQueueDequeueSlowDown``"
+
+Regulates how long dequeueing should be delayed. This value must be specified
+in microseconds (1000000us is 1sec). It can be used to slow down rsyslog so
+it won't send things to fast.
+For example if this parameter is set to 10000 on a UDP send action, the action
+won't be able to put out more than 100 messages per second.
+
+
+queue.dequeueTimeBegin
+----------------------
+
+.. csv-table::
+   :header: "type", "default", "mandatory", "|FmtObsoleteName| directive"
+   :widths: auto
+   :class: parameter-table
+
+   "integer", "0", "no", "``$ActionQueueDequeueTimeBegin``"
+
+With this parameter you can specify rsyslog to process queues during specific
+time periods. To define a time frame use the 24-hour format without minutes.
+This parameter specifies the begin and "queue.dequeuetimeend" the end of the
+time frame.
+
+
+queue.dequeueTimeEnd
+--------------------
+
+.. csv-table::
+   :header: "type", "default", "mandatory", "|FmtObsoleteName| directive"
+   :widths: auto
+   :class: parameter-table
+
+   "integer", "25", "no", "``$ActionQueueDequeueTimeEnd``"
+
+With this parameter you can specify rsyslog to process queues during specific
+time periods. To define a time frame use the 24-hour format without minutes.
+This parameter specifies the end and "queue.dequeuetimebegin" the begin of the
+time frame. The default 25 disables the time-window.
+
+
+queue.takeFlowCtlFromMsg
+------------------------
+
+.. csv-table::
+   :header: "type", "default", "mandatory", "|FmtObsoleteName| directive"
+   :widths: auto
+   :class: parameter-table
+
+   "boolean", "off", "no", "none"
+
+.. versionadded:: 8.1911.0
+
+This is a fine-tuning parameter which permits to control whether or not
+rsyslog shall always take the flow control setting from the message. If
+so, non-primary queues may also **block** when reaching high water mark.
+
+This permits to add some synchronous processing to rsyslog core engine.
+However, **this involves some risk**:  Improper use may make the core engine
+stall. As such, **enabling this parameter requires very careful planning
+of the rsyslog configuration and deep understanding of the consequences**.
+
+Note that the parameter is applied to individual queues, so a configuration
+with a large number of queues can (and must if used) be fine-tuned to
+the exact use case.
+
+**The rsyslog team strongly recommends to let this parameter turned off.**
+
+
+
+Examples
+========
+
+Example 1
+---------
+
+The following is a sample of a TCP forwarding action with its own queue.
+
+.. code-block:: none
+
+   action(type="omfwd" target="192.168.2.11" port="10514" protocol="tcp"
+          queue.filename="forwarding" queue.size="1000000" queue.type="LinkedList"
+         )
+