Adding upstream version 8.2402.0+dfsg.upstream/8.2402.0+dfsg

Signed-off-by: Daniel Baumann <daniel.baumann@progress-linux.org>

1 files changed, 320 insertions, 0 deletions

diff --git a/source/concepts/multi_ruleset.rst b/source/concepts/multi_ruleset.rst
new file mode 100644
index 0000000..512558e
--- /dev/null
+++ b/source/concepts/multi_ruleset.rst
@@ -0,0 +1,320 @@
+Multiple Rulesets in rsyslog
+============================
+
+Starting with version 4.5.0 and 5.1.1,
+`rsyslog <http://www.rsyslog.com>`_ supports multiple rulesets within a
+single configuration. This is especially useful for routing the
+reception of remote messages to a set of specific rules. Note that the
+input module must support binding to non-standard rulesets, so the
+functionality may not be available with all inputs.
+
+In this document, I am using :doc:`imtcp <../configuration/modules/imtcp>`, an input module that
+supports binding to non-standard rulesets since rsyslog started to
+support them.
+
+What is a Ruleset?
+------------------
+
+If you have worked with (r)syslog.conf, you know that it is made up of
+what I call rules (others tend to call them selectors, a sysklogd term).
+Each rule consist of a filter and one or more actions to be carried out
+when the filter evaluates to true. A filter may be as simple as a
+traditional syslog priority based filter (like "\*.\*" or "mail.info" or
+a as complex as a script-like expression. Details on that are covered in
+the config file documentation. After the filter come action specifiers,
+and an action is something that does something to a message, e.g. write
+it to a file or forward it to a remote logging server.
+
+A traditional configuration file is made up of one or more of these
+rules. When a new message arrives, its processing starts with the first
+rule (in order of appearance in rsyslog.conf) and continues for each
+rule until either all rules have been processed or a so-called "discard"
+action happens, in which case processing stops and the message is thrown
+away (what also happens after the last rule has been processed).
+
+The **multi-ruleset** support now permits to specify more than one such
+rule sequence. You can think of a traditional config file just as a
+single default rule set, which is automatically bound to each of the
+inputs. This is even what actually happens. When rsyslog.conf is
+processed, the config file parser looks for the directive
+
+::
+
+    ruleset(name="rulesetname")
+
+Where name is any name the user likes (but must not start with
+"RSYSLOG\_", which is the name space reserved for rsyslog use). If it
+finds this directive, it begins a new rule set (if the name was not yet
+know) or switches to an already-existing one (if the name was known).
+All rules defined between this $RuleSet directive and the next one are
+appended to the named ruleset. Note that the reserved name
+"RSYSLOG\_DefaultRuleset" is used to specify rsyslogd's default ruleset.
+You can use that name wherever you can use a ruleset name, including
+when binding an input to it.
+
+Inside a ruleset, messages are processed as described above: they start
+with the first rule and rules are processed in the order of appearance
+of the configuration file until either there are no more rules or the
+discard action is executed. Note that with multiple rulesets no longer
+**all** rsyslog.conf rules are executed but **only** those that are
+contained within the specific ruleset.
+
+Inputs must explicitly bind to rulesets. If they don't, the default
+ruleset is bound.
+
+This brings up the next question:
+
+What does "To bind to a Ruleset" mean?
+--------------------------------------
+
+This term is used in the same sense as "to bind an IP address to an
+interface": it means that a specific input, or part of an input (like a
+tcp listener) will use a specific ruleset to "pass its messages to". So
+when a new message arrives, it will be processed via the bound ruleset.
+Rules from all other rulesets are irrelevant and will never be processed.
+
+This makes multiple rulesets very handy to process local and remote
+message via separate means: bind the respective receivers to different
+rule sets, and you do not need to separate the messages by any other
+method.
+
+Binding to rulesets is input-specific. For imtcp, this is done via the
+
+::
+
+    input(type="imptcp" port="514" ruleset="rulesetname")
+
+directive. Note that "rulesetname" must be the name of a ruleset that is
+already defined at the time the bind directive is given. There are many
+ways to make sure this happens, but I personally think that it is best
+to define all rule sets at the top of rsyslog.conf and define the inputs
+at the bottom. This kind of reverses the traditional recommended
+ordering, but seems to be a really useful and straightforward way of
+doing things.
+
+Why are rulesets important for different parser configurations?
+---------------------------------------------------------------
+
+Custom message parsers, used to handle different (and potentially
+otherwise-invalid) message formats, can be bound to rulesets. So
+multiple rulesets can be a very useful way to handle devices sending
+messages in different malformed formats in a consistent way.
+Unfortunately, this is not uncommon in the syslog world. An in-depth
+explanation with configuration sample can be found at the
+:doc:`$RulesetParser <../configuration/ruleset/rsconf1_rulesetparser>` configuration directive.
+
+Can I use a different Ruleset as the default?
+---------------------------------------------
+
+This is possible by using the
+
+::
+
+    $DefaultRuleset <name>
+
+Directive. Please note, however, that this directive is actually global:
+that is, it does not modify the ruleset to which the next input is bound
+but rather provides a system-wide default rule set for those inputs that
+did not explicitly bind to one. As such, the directive can not be used
+as a work-around to bind inputs to non-default rulesets that do not
+support ruleset binding.
+
+Rulesets and Queues
+-------------------
+
+By default, rulesets do not have their own queue. It must be activated
+via the $RulesetCreateMainQueue directive, or if using rainerscript
+format, by specifying queue parameters on the ruleset directive, e.g.
+
+::
+
+   ruleset(name="whatever" queue.type="fixedArray" queue. ...)
+
+See :doc:`http://www.rsyslog.com/doc/master/rainerscript/queue\_parameters.html <../rainerscript/queue_parameters>`
+for more details.
+
+Please note that when a ruleset uses its own queue, processing of the ruleset
+happens **asynchronously** to the rest of processing. As such, any modifications
+made to the message object (e.g. message or local variables that are set) or
+discarding of the message object **have no effect outside that ruleset**. So
+if you want to modify the message object inside the ruleset, you **cannot**
+define a queue for it. Most importantly, you cannot call it and expect the
+modified properties to be present when the call returns. Even more so, the
+call will most probably return before the message is even begun to be processed
+by the ruleset in question.
+
+Note that in RainerScript format specifying any "queue.\*" can cause the
+creation of a dedicated queue and as such asynchronous processing. This is
+because queue parameters cannot be specified without a queue. Note, though,
+that the actual creation is **guaranteed** only if "queue.type" is specified
+as above. So if you intentionally want to assign a separate queue to the
+ruleset, do so as shown above.
+
+Examples
+--------
+
+Split local and remote logging
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Let's say you have a pretty standard system that logs its local messages
+to the usual bunch of files that are specified in the default
+rsyslog.conf. As an example, your rsyslog.conf might look like this:
+
+::
+
+    # ... module loading ...
+    # The authpriv file has restricted access.
+    authpriv.*  /var/log/secure
+    # Log all the mail messages in one place.
+    mail.*      /var/log/maillog
+    # Log cron stuff
+    cron.*      /var/log/cron
+    # Everybody gets emergency messages
+    *.emerg     *
+    ... more ...
+
+Now, you want to add receive messages from a remote system and log these
+to a special file, but you do not want to have these messages written to
+the files specified above. The traditional approach is to add a rule in
+front of all others that filters on the message, processes it and then
+discards it:
+
+::
+
+    # ... module loading ...
+    # process remote messages
+    if $fromhost-ip == '192.0.2.1' then {
+            action(type="omfile" file="/var/log/remotefile02")
+            stop
+        }
+
+
+    # only messages not from 192.0.2.1 make it past this point
+
+    # The authpriv file has restricted access.
+    authpriv.*                            /var/log/secure
+    # Log all the mail messages in one place.
+    mail.*                                /var/log/maillog
+    # Log cron stuff
+    cron.*                                /var/log/cron
+    # Everybody gets emergency messages
+    *.emerg                               *
+    ... more ...
+
+Note that "stop" is the discard action!. Also note that we assume that
+192.0.2.1 is the sole remote sender (to keep it simple).
+
+With multiple rulesets, we can simply define a dedicated ruleset for the
+remote reception case and bind it to the receiver. This may be written
+as follows:
+
+::
+
+    # ... module loading ...
+    # process remote messages
+    # define new ruleset and add rules to it:
+    ruleset(name="remote"){
+        action(type="omfile" file="/var/log/remotefile")
+    }
+    # only messages not from 192.0.2.1 make it past this point
+
+    # bind ruleset to tcp listener and activate it:
+    input(type="imptcp" port="10514" ruleset="remote")
+
+Split local and remote logging for three different ports
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This example is almost like the first one, but it extends it a little
+bit. While it is very similar, I hope it is different enough to provide
+a useful example why you may want to have more than two rulesets.
+
+Again, we would like to use the "regular" log files for local logging,
+only. But this time we set up three syslog/tcp listeners, each one
+listening to a different port (in this example 10514, 10515, and 10516).
+Logs received from these receivers shall go into different files. Also,
+logs received from 10516 (and only from that port!) with "mail.\*"
+priority, shall be written into a specific file and **not** be written to
+10516's general log file.
+
+This is the config:
+
+::
+
+    # ... module loading ...
+    # process remote messages
+
+    ruleset(name="remote10514"){
+        action(type="omfile" file="/var/log/remote10514")
+    }
+
+    ruleset(name="remote10515"){
+        action(type="omfile" file="/var/log/remote10515")
+    }
+
+    ruleset(name="remote10516"){
+        if prifilt("mail.*") then {
+            /var/log/mail10516
+            stop
+            # note that the stop-command will prevent this message from 
+            # being written to the remote10516 file - as usual...   
+        }
+        /var/log/remote10516
+    }
+
+
+    # and now define listeners bound to the relevant ruleset
+    input(type="imptcp" port="10514" ruleset="remote10514")
+    input(type="imptcp" port="10515" ruleset="remote10515")
+    input(type="imptcp" port="10516" ruleset="remote10516")
+
+Performance
+-----------
+
+Fewer Filters
+~~~~~~~~~~~~~
+
+No rule processing can be faster than not processing a rule at all. As
+such, it is useful for a high performance system to identify disjunct
+actions and try to split these off to different rule sets. In the
+example section, we had a case where three different tcp listeners need
+to write to three different files. This is a perfect example of where
+multiple rule sets are easier to use and offer more performance. The
+performance is better simply because there is no need to check the
+reception service - instead messages are automatically pushed to the
+right rule set and can be processed by very simple rules (maybe even
+with "\*.\*"-filters, the fastest ones available).
+
+Partitioning of Input Data
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Starting with rsyslog 5.3.4, rulesets permit higher concurrency. They
+offer the ability to run on their own "main" queue. What that means is
+that a own queue is associated with a specific rule set. That means that
+inputs bound to that ruleset do no longer need to compete with each
+other when they enqueue a data element into the queue. Instead, enqueue
+operations can be completed in parallel.
+
+An example: let us assume we have three TCP listeners. Without rulesets,
+each of them needs to insert messages into the main message queue. So if
+each of them wants to submit a newly arrived message into the queue at
+the same time, only one can do so while the others need to wait. With
+multiple rulesets, its own queue can be created for each ruleset. If now
+each listener is bound to its own ruleset, concurrent message submission
+is possible. On a machine with a sufficiently large number of cores,
+this can result in dramatic performance improvement.
+
+It is highly advised that high-performance systems define a dedicated
+ruleset, with a dedicated queue for each of the inputs.
+
+By default, rulesets do **not** have their own queue. It must be
+activated via the
+:doc:`$RulesetCreateMainQueue <../configuration/ruleset/rsconf1_rulesetcreatemainqueue>`
+directive.
+
+See Also
+--------
+.. toctree::
+   :maxdepth: 1
+   
+   ../historical/multi_ruleset_legacy_format_samples
+