diff options
Diffstat (limited to '')
| -rw-r--r-- | source/concepts/index.rst | 16 | ||||
| -rw-r--r-- | source/concepts/janitor.rst | 41 | ||||
| -rw-r--r-- | source/concepts/messageparser.rst | 304 | ||||
| -rw-r--r-- | source/concepts/multi_ruleset.rst | 320 | ||||
| -rw-r--r-- | source/concepts/netstrm_drvr.rst | 21 | ||||
| -rw-r--r-- | source/concepts/ns_gtls.rst | 91 | ||||
| -rw-r--r-- | source/concepts/ns_ossl.rst | 76 | ||||
| -rw-r--r-- | source/concepts/ns_ptcp.rst | 13 | ||||
| -rw-r--r-- | source/concepts/queues.rst | 524 | ||||
| -rw-r--r-- | source/concepts/rfc5424layers.png | bin | 0 -> 10605 bytes |
10 files changed, 1406 insertions, 0 deletions
diff --git a/source/concepts/index.rst b/source/concepts/index.rst new file mode 100644 index 0000000..2119e40 --- /dev/null +++ b/source/concepts/index.rst @@ -0,0 +1,16 @@ +Concepts +======== + +This chapter describes important rsyslog concepts and objects. Where +appropriate, it also refers to configurations settings to affect +the respective objects. + +.. toctree:: + :maxdepth: 2 + + queues + janitor + messageparser + multi_ruleset + netstrm_drvr + diff --git a/source/concepts/janitor.rst b/source/concepts/janitor.rst new file mode 100644 index 0000000..52765ed --- /dev/null +++ b/source/concepts/janitor.rst @@ -0,0 +1,41 @@ +The Janitor Process +=================== +The janitor process carries out periodic cleanup tasks. For example, +it is used by +:doc:`omfile <../configuration/modules/omfile>` +to close files after a timeout has expired. + +The janitor runs periodically. As such, all tasks carried out via the +janitor will be activated based on the interval at which it runs. This +means that all janitor-related times set are approximate and should be +considered as "no earlier than" (NET). If, for example, you set a timeout +to 5 minutes and the janitor is run in 10-minute intervals, the timeout +may actually happen after 5 minutes, but it may also take up to 20 +minutes for it to be detected. + +In general (see note about HUP below), janitor based activities scheduled +to occur after *n* minutes will occur after *n* and *(n + 2\*janitorInterval)* +minutes. + +To reduce the potential delay caused by janitor invocation, +:ref:`the interval at which the janitor runs can be be adjusted <global_janitorInterval>`\ . +If high precision is +required, it should be set to one minute. Janitor-based activities will +still be NET times, but the time frame will be much smaller. In the +example with the file timeout, it would be between 5 and 6 minutes if the +janitor is run at a one-minute interval. + +Note that the more frequent the janitor is run, the more frequent the +system needs to wakeup from potential low power state. This is no issue +for data center machines (which usually always run at full speed), but it +may be an issue for power-constrained environments like notebooks. For +such systems, a higher janitor interval may make sense. + +As a special case, sending a HUP signal to rsyslog also activate the +janitor process. This can lead to too-frequent wakeups of janitor-related +services. However, we don't expect this to cause any issues. If it does, +it could be solved by creating a separate thread for the janitor. But as +this takes up some system resources and is not not considered useful, we +have not implemented it that way. If the HUP/janitor interaction causes +problems, let the rsyslog team know and we can change the implementation. + diff --git a/source/concepts/messageparser.rst b/source/concepts/messageparser.rst new file mode 100644 index 0000000..f9886be --- /dev/null +++ b/source/concepts/messageparser.rst @@ -0,0 +1,304 @@ +Message parsers in rsyslog +========================== + +Written by `Rainer Gerhards <http://www.gerhards.net/rainer>`_ +(2009-11-06) + +Intro +----- + +Message parsers are a feature of rsyslog 5.3.4 and above. In this +article, I describe what message parsers are, what they can do and how +they relate to the relevant standards. I will also describe what you can +not do with time. Finally, I give some advice on implementing your own +custom parser. + +What are message parsers? +------------------------- + +Well, the quick answer is that message parsers are the component of +rsyslog that parses the syslog message after it is being received. Prior +to rsyslog 5.3.4, message parsers where built in into the rsyslog core +itself and could not be modified (other than by modifying the rsyslog +code). + +In 5.3.4, we changed that: message parsers are now loadable modules +(just like input and output modules). That means that new message +parsers can be added without modifying the rsyslog core, even without +contributing something back to the project. + +But that doesn't answer what a message parser really is. What does it +mean to "parse a message" and, maybe more importantly, what is a +message? To answer these questions correctly, we need to dig down into +the relevant standards. `RFC5424 <http://tools.ietf.org/html/rfc5424>`_ +specifies a layered architecture for the syslog protocol: + +.. figure:: rfc5424layers.png + :align: center + :alt: RFC5424 syslog protocol layers + + RFC5424 syslog protocol layers + +For us important is the distinction between the syslog transport and the +upper layers. The transport layer specifies how a stream of messages is +assembled at the sender side and how this stream of messages is +disassembled into the individual messages at the receiver side. In +networking terminology, this is called "framing". The core idea is that +each message is put into a so-called "frame", which then is transmitted +over the communications link. + +The framing used is depending on the protocol. For example, in UDP the +"frame"-equivalent is a packet that is being sent (this also means that +no two messages can travel within a single UDP packet). In "plain tcp +syslog", the industry standard, LF is used as a frame delimiter (which +also means that no multi-line message can properly be transmitted, a +"design" flaw in plain tcp syslog). In +`RFC5425 <http://tools.ietf.org/html/rfc5425>`_ there is a header in +front of each frame that contains the size of the message. With this +framing, any message content can properly be transferred. + +And now comes the important part: **message parsers do NOT operate at +the transport layer**, they operate, as their name implies, on messages. +So we can not use message parsers to change the underlying framing. For +example, if a sender splits (for whatever reason) a single message into +two and encapsulates these into two frames, there is no way a message +parser could undo that. + +A typical example may be a multi-line message: let's assume some +originator has generated a message for the format "A\\nB" (where \\n +means LF). If that message is being transmitted via plain tcp syslog, +the frame delimiter is LF. So the sender will delimit the frame with LF, +but otherwise send the message unmodified onto the wire (because that is +how things are -unfortunately- done in plain tcp syslog...). So wire +will see "A\\nB\\n". When this arrives at the receiver, the transport +layer will undo the framing. When it sees the LF after A, it thinks it +finds a valid frame delimiter (in fact, this is the correct view!). So +the receive will extract one complete message A and one complete message +B, not knowing that they once were both part of a large multi-line +message. These two messages are then passed to the upper layers, where +the message parsers receive them and extract information. However, the +message parsers never know (or even have a chance to see) that A and B +belonged together. Even further, in rsyslog there is no guarantee that A +will be parsed before B - concurrent operations may cause the reverse +order (and do so very validly). + +The important lesson is: **message parsers can not be used to fix a +broken framing**. You need a full protocol implementation to do that, +what is the domain of input and output modules. + +I have now told you what you can not do with message parsers. But what +they are good for? Thankfully, broken framing is not the primary problem +of the syslog world. A wealth of different formats is. Unfortunately, +many real-world implementations violate the relevant standards in one +way or another. That makes it often very hard to extract meaningful +information from a message or to process messages from different sources +by the same rules. In my article `syslog parsing in +rsyslog <syslog_parsing.html>`_ I have elaborated on all the real-world +evil that you can usually see. So I won't repeat that here. But in +short, the real problem is not the framing, but how to make malformed +messages well-looking. + +**This is what message parsers permit you to do: take a (well-known) +malformed message, parse it according to its semantics and generate +perfectly valid internal message representations from it.** So as long +as messages are consistently in the same wrong format (and they usually +are!), a message parser can look at that format, parse it, and make the +message processable just like it were well formed in the first place. +Plus, one can abuse the interface to do some other "interesting" tricks, +but that would take us to far. + +While this functionality may not sound exciting, it actually solves a +very big issue (that you only really understand if you have managed a +system with various different syslog sources). Note that we were often +able to process malformed messages in the past with the help of the +property replacer and regular expressions. While this is nice, it has a +performance hit. A message parser is a C code, compiled to native +language, and thus typically much faster than any regular expression +based method (depending, of course, on the quality of the +implementation...). + +How are message parsers used? +----------------------------- + +In a simplified view, rsyslog + +#. first receives messages (via the input module), +#. *then parses them (at the message level!)* and +#. then processes them (operating on the internal message + representation). + +Message parsers are utilized in the second step (written in italics). +Thus, they take the raw message (NOT frame!) received from the remote +system and create the internal structure out of it that the other parts +of rsyslog need in order to perform their processing. Parsing is vital, +because an unparsed message can not be processed in the third stage, the +actual application-level processing (like forwarding or writing to +files). + +Parser Chains and how they Operate +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Rsyslog chains parsers together to provide flexibility. A **parser +chain** contains all parsers that can potentially be used to parse a +message. It is assumed that there is some way a parser can detect if the +message it is being presented is supported by it. If so, the parser will +tell the rsyslog engine and parse the message. The rsyslog engine now +calls each parser inside the chain (in sequence!) until the first parser +is able to parse the message. After one parser has been found, the +message is considered parsed and no others parsers are called on that +message. + +Side-note: this method implies there are some "not-so-dirty" tricks +available to modify the message by a parser module that declares itself +as "unable to parse" but still does some message modification. This was +not a primary design goal, but may be utilized, and the interface +probably extended, to support generic filter modules. These would need +to go to the root of the parser chain. As mentioned, the current system +already supports this. + +The position inside the parser chain can be thought of as a priority: +parser sitting earlier in the chain take precedence over those sitting +later in it. So more specific parser should go earlier in the chain. A +good example of how this works is the default parser set provided by +rsyslog: rsyslog.rfc5424 and rsyslog.rfc3164, each one parses according +to the rfc that has named it. RFC5424 was designed to be distinguishable +from RFC3164 message by the sequence "1 " immediately after the +so-called PRI-part (don't worry about these words, it is sufficient if +you understand there is a well-defined sequence used to identify RFC5424 +messages). In contrary, RFC3164 actually permits everything as a valid +message. Thus the RFC3164 parser will always parse a message, sometimes +with quite unexpected outcome (there is a lot of guesswork involved in +that parser, which unfortunately is unavoidable due to existing +technology limits). So the default parser chain is to try the RFC5424 +parser first and after it the RFC3164 parser. If we have a +5424-formatted message, that parser will identify and parse it and the +rsyslog engine will stop processing. But if we receive a legacy syslog +message, the RFC5424 will detect that it can not parse it, return this +status to the engine which then calls the next parser inside the chain. +That usually happens to be the RFC3164 parser, which will always process +the message. But there could also be any other parser inside the chain, +and then each one would be called unless one that is able to parse can +be found. + +If we reversed the parser order, RFC5424 messages would incorrectly +parsed. Why? Because the RFC3164 parser will always parse every message, +so if it were asked first, it would parse (and misinterpret) the +5424-formatted message, return it did so and the rsyslog engine would +never call the 5424 parser. So order of sequence is very important. + +What happens if no parser in the chain could parse a message? Well, then +we could not obtain the in-memory representation that is needed to +further process the message. In that case, rsyslog has no other choice +than to discard the message. If it does so, it will emit a warning +message, but only in the first 1,000 incidents. This limit is a safety +measure against message-loops, which otherwise could quickly result from +a parser chain misconfiguration. **If you do not tolerate loss of +unparsable messages, you must ensure that each message can be parsed.** +You can easily achieve this by always using the "rsyslog-rfc3164" parser +as the *last* parser inside parser chains. That may result in invalid +parsing, but you will have a chance to see the invalid message (in debug +mode, a warning message will be written to the debug log each time a +message is dropped due to inability to parse it). + +Where are parser chains used? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +We now know what parser chains are and how they operate. The question is +now how many parser chains can be active and how it is decided which +parser chain is used on which message. This is controlled via +:doc:`rsyslog's rulesets <multi_ruleset>`. In short, multiple rulesets can be +defined and there always exist at least one ruleset. +A parser chain is bound to a +specific ruleset. This is done by virtue of defining parsers via the +:doc:`$RulesetParser <../configuration/ruleset/rsconf1_rulesetparser>` +configuration directive +(for specifics, see there). If no such directive is specified, the +default parser chain is used. As of this writing, the default parser +chain always consists of "rsyslog.rfc5424", "rsyslog.rfc3164", in that +order. As soon as a parser is configured, the default list is cleared +and the new parser is added to the end of the (initially empty) +ruleset's parser chain. + +The important point to know is that parser chains are defined on a +per-ruleset basis. + +Can I use different parser chains for different devices? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The correct answer is: generally yes, but it depends. First of all, +remember that input modules (and specific listeners) may be bound to +specific rulesets. As parser chains "reside" in rulesets, binding to a +ruleset also binds to the parser chain that is bound to that ruleset. As +a number one prerequisite, the input module must support binding to +different rulesets. Not all do, but their number is growing. For +example, the important `imudp <imudp.html>`_ and `imtcp <imtcp.html>`_ +input modules support that functionality. Those that do not (for example +`im3195 <im3195>`_) can only utilize the default ruleset and thus the +parser chain defined in that ruleset. + +If you do not know if the input module in question supports ruleset +binding, check its documentation page. Those that support it have the +required directives. + +Note that it is currently under evaluation if rsyslog will support +binding parser chains to specific inputs directly, without depending on +the ruleset. There are some concerns that this may not be necessary but +adds considerable complexity to the configuration. So this may or may +not be possible in the future. In any case, if we decide to add it, +input modules need to support it, so this functionality would require +some time to implement. + +The cookbook recipe for using different parsers for different devices +is given as an actual in-depth example in the +`$RulesetParser` configuration directive +doc page. In short, it is accomplished by defining specific rulesets for +the required parser chains, defining different listener ports for each +of the devices with different format and binding these listeners to the +correct ruleset (and thus parser chains). Using that approach, a variety +of different message formats can be supported via a single rsyslog +instance. + +Which message parsers are available +----------------------------------- + +As of this writing, there exist only two message parsers, one for +RFC5424 format and one for legacy syslog (loosely described in +`RFC3164 <http://tools.ietf.org/html/rfc3164>`_). These parsers are +built-in and must not be explicitly loaded. However, message parsers can +be added with relative ease by anyone knowing to code in C. Then, they +can be loaded via $ModLoad just like any other loadable module. It is +expected that the rsyslog project will be contributed additional message +parsers over time, so that at some point there hopefully is a rich +choice of them (I intend to add a browsable repository as soon as new +parsers pop up). + +How to write a message parser? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +As a prerequisite, you need to know the exact format that the device is +sending. Then, you need moderate C coding skills, and a little bit of +rsyslog internals. I guess the rsyslog specific part should not be that +hard, as almost all information can be gained from the existing parsers. +They are rather simple in structure and can be found under the "./tools" +directory. They are named pmrfc3164.c and pmrfc5424.c. You need to +follow the usual loadable module guidelines. It is my expectation that +writing a parser should typically not take longer than a single day, +with maybe a day more to get acquainted with rsyslog. Of course, I am +not sure if the number is actually right. + +If you can not program or have no time to do it, Adiscon can also write +a message parser for you as part of the `rsyslog professional services +offering <http://www.rsyslog.com/professional-services>`_. + +Conclusion +---------- + +Malformed syslog messages are a pain and unfortunately often seen in +practice. Message parsers provide a fast and efficient solution for this +problem. Different parsers can be defined for different devices, and +they all convert message information into rsyslog's well-defined +internal format. Message parsers were first introduced in rsyslog 5.3.4 +and also offer some interesting ideas that may be explored in the future +- up to full message normalization capabilities. It is strongly +recommended that anyone with a heterogeneous environment take a look at +message parser capabilities. diff --git a/source/concepts/multi_ruleset.rst b/source/concepts/multi_ruleset.rst new file mode 100644 index 0000000..512558e --- /dev/null +++ b/source/concepts/multi_ruleset.rst @@ -0,0 +1,320 @@ +Multiple Rulesets in rsyslog +============================ + +Starting with version 4.5.0 and 5.1.1, +`rsyslog <http://www.rsyslog.com>`_ supports multiple rulesets within a +single configuration. This is especially useful for routing the +reception of remote messages to a set of specific rules. Note that the +input module must support binding to non-standard rulesets, so the +functionality may not be available with all inputs. + +In this document, I am using :doc:`imtcp <../configuration/modules/imtcp>`, an input module that +supports binding to non-standard rulesets since rsyslog started to +support them. + +What is a Ruleset? +------------------ + +If you have worked with (r)syslog.conf, you know that it is made up of +what I call rules (others tend to call them selectors, a sysklogd term). +Each rule consist of a filter and one or more actions to be carried out +when the filter evaluates to true. A filter may be as simple as a +traditional syslog priority based filter (like "\*.\*" or "mail.info" or +a as complex as a script-like expression. Details on that are covered in +the config file documentation. After the filter come action specifiers, +and an action is something that does something to a message, e.g. write +it to a file or forward it to a remote logging server. + +A traditional configuration file is made up of one or more of these +rules. When a new message arrives, its processing starts with the first +rule (in order of appearance in rsyslog.conf) and continues for each +rule until either all rules have been processed or a so-called "discard" +action happens, in which case processing stops and the message is thrown +away (what also happens after the last rule has been processed). + +The **multi-ruleset** support now permits to specify more than one such +rule sequence. You can think of a traditional config file just as a +single default rule set, which is automatically bound to each of the +inputs. This is even what actually happens. When rsyslog.conf is +processed, the config file parser looks for the directive + +:: + + ruleset(name="rulesetname") + +Where name is any name the user likes (but must not start with +"RSYSLOG\_", which is the name space reserved for rsyslog use). If it +finds this directive, it begins a new rule set (if the name was not yet +know) or switches to an already-existing one (if the name was known). +All rules defined between this $RuleSet directive and the next one are +appended to the named ruleset. Note that the reserved name +"RSYSLOG\_DefaultRuleset" is used to specify rsyslogd's default ruleset. +You can use that name wherever you can use a ruleset name, including +when binding an input to it. + +Inside a ruleset, messages are processed as described above: they start +with the first rule and rules are processed in the order of appearance +of the configuration file until either there are no more rules or the +discard action is executed. Note that with multiple rulesets no longer +**all** rsyslog.conf rules are executed but **only** those that are +contained within the specific ruleset. + +Inputs must explicitly bind to rulesets. If they don't, the default +ruleset is bound. + +This brings up the next question: + +What does "To bind to a Ruleset" mean? +-------------------------------------- + +This term is used in the same sense as "to bind an IP address to an +interface": it means that a specific input, or part of an input (like a +tcp listener) will use a specific ruleset to "pass its messages to". So +when a new message arrives, it will be processed via the bound ruleset. +Rules from all other rulesets are irrelevant and will never be processed. + +This makes multiple rulesets very handy to process local and remote +message via separate means: bind the respective receivers to different +rule sets, and you do not need to separate the messages by any other +method. + +Binding to rulesets is input-specific. For imtcp, this is done via the + +:: + + input(type="imptcp" port="514" ruleset="rulesetname") + +directive. Note that "rulesetname" must be the name of a ruleset that is +already defined at the time the bind directive is given. There are many +ways to make sure this happens, but I personally think that it is best +to define all rule sets at the top of rsyslog.conf and define the inputs +at the bottom. This kind of reverses the traditional recommended +ordering, but seems to be a really useful and straightforward way of +doing things. + +Why are rulesets important for different parser configurations? +--------------------------------------------------------------- + +Custom message parsers, used to handle different (and potentially +otherwise-invalid) message formats, can be bound to rulesets. So +multiple rulesets can be a very useful way to handle devices sending +messages in different malformed formats in a consistent way. +Unfortunately, this is not uncommon in the syslog world. An in-depth +explanation with configuration sample can be found at the +:doc:`$RulesetParser <../configuration/ruleset/rsconf1_rulesetparser>` configuration directive. + +Can I use a different Ruleset as the default? +--------------------------------------------- + +This is possible by using the + +:: + + $DefaultRuleset <name> + +Directive. Please note, however, that this directive is actually global: +that is, it does not modify the ruleset to which the next input is bound +but rather provides a system-wide default rule set for those inputs that +did not explicitly bind to one. As such, the directive can not be used +as a work-around to bind inputs to non-default rulesets that do not +support ruleset binding. + +Rulesets and Queues +------------------- + +By default, rulesets do not have their own queue. It must be activated +via the $RulesetCreateMainQueue directive, or if using rainerscript +format, by specifying queue parameters on the ruleset directive, e.g. + +:: + + ruleset(name="whatever" queue.type="fixedArray" queue. ...) + +See :doc:`http://www.rsyslog.com/doc/master/rainerscript/queue\_parameters.html <../rainerscript/queue_parameters>` +for more details. + +Please note that when a ruleset uses its own queue, processing of the ruleset +happens **asynchronously** to the rest of processing. As such, any modifications +made to the message object (e.g. message or local variables that are set) or +discarding of the message object **have no effect outside that ruleset**. So +if you want to modify the message object inside the ruleset, you **cannot** +define a queue for it. Most importantly, you cannot call it and expect the +modified properties to be present when the call returns. Even more so, the +call will most probably return before the message is even begun to be processed +by the ruleset in question. + +Note that in RainerScript format specifying any "queue.\*" can cause the +creation of a dedicated queue and as such asynchronous processing. This is +because queue parameters cannot be specified without a queue. Note, though, +that the actual creation is **guaranteed** only if "queue.type" is specified +as above. So if you intentionally want to assign a separate queue to the +ruleset, do so as shown above. + +Examples +-------- + +Split local and remote logging +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Let's say you have a pretty standard system that logs its local messages +to the usual bunch of files that are specified in the default +rsyslog.conf. As an example, your rsyslog.conf might look like this: + +:: + + # ... module loading ... + # The authpriv file has restricted access. + authpriv.* /var/log/secure + # Log all the mail messages in one place. + mail.* /var/log/maillog + # Log cron stuff + cron.* /var/log/cron + # Everybody gets emergency messages + *.emerg * + ... more ... + +Now, you want to add receive messages from a remote system and log these +to a special file, but you do not want to have these messages written to +the files specified above. The traditional approach is to add a rule in +front of all others that filters on the message, processes it and then +discards it: + +:: + + # ... module loading ... + # process remote messages + if $fromhost-ip == '192.0.2.1' then { + action(type="omfile" file="/var/log/remotefile02") + stop + } + + + # only messages not from 192.0.2.1 make it past this point + + # The authpriv file has restricted access. + authpriv.* /var/log/secure + # Log all the mail messages in one place. + mail.* /var/log/maillog + # Log cron stuff + cron.* /var/log/cron + # Everybody gets emergency messages + *.emerg * + ... more ... + +Note that "stop" is the discard action!. Also note that we assume that +192.0.2.1 is the sole remote sender (to keep it simple). + +With multiple rulesets, we can simply define a dedicated ruleset for the +remote reception case and bind it to the receiver. This may be written +as follows: + +:: + + # ... module loading ... + # process remote messages + # define new ruleset and add rules to it: + ruleset(name="remote"){ + action(type="omfile" file="/var/log/remotefile") + } + # only messages not from 192.0.2.1 make it past this point + + # bind ruleset to tcp listener and activate it: + input(type="imptcp" port="10514" ruleset="remote") + +Split local and remote logging for three different ports +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This example is almost like the first one, but it extends it a little +bit. While it is very similar, I hope it is different enough to provide +a useful example why you may want to have more than two rulesets. + +Again, we would like to use the "regular" log files for local logging, +only. But this time we set up three syslog/tcp listeners, each one +listening to a different port (in this example 10514, 10515, and 10516). +Logs received from these receivers shall go into different files. Also, +logs received from 10516 (and only from that port!) with "mail.\*" +priority, shall be written into a specific file and **not** be written to +10516's general log file. + +This is the config: + +:: + + # ... module loading ... + # process remote messages + + ruleset(name="remote10514"){ + action(type="omfile" file="/var/log/remote10514") + } + + ruleset(name="remote10515"){ + action(type="omfile" file="/var/log/remote10515") + } + + ruleset(name="remote10516"){ + if prifilt("mail.*") then { + /var/log/mail10516 + stop + # note that the stop-command will prevent this message from + # being written to the remote10516 file - as usual... + } + /var/log/remote10516 + } + + + # and now define listeners bound to the relevant ruleset + input(type="imptcp" port="10514" ruleset="remote10514") + input(type="imptcp" port="10515" ruleset="remote10515") + input(type="imptcp" port="10516" ruleset="remote10516") + +Performance +----------- + +Fewer Filters +~~~~~~~~~~~~~ + +No rule processing can be faster than not processing a rule at all. As +such, it is useful for a high performance system to identify disjunct +actions and try to split these off to different rule sets. In the +example section, we had a case where three different tcp listeners need +to write to three different files. This is a perfect example of where +multiple rule sets are easier to use and offer more performance. The +performance is better simply because there is no need to check the +reception service - instead messages are automatically pushed to the +right rule set and can be processed by very simple rules (maybe even +with "\*.\*"-filters, the fastest ones available). + +Partitioning of Input Data +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Starting with rsyslog 5.3.4, rulesets permit higher concurrency. They +offer the ability to run on their own "main" queue. What that means is +that a own queue is associated with a specific rule set. That means that +inputs bound to that ruleset do no longer need to compete with each +other when they enqueue a data element into the queue. Instead, enqueue +operations can be completed in parallel. + +An example: let us assume we have three TCP listeners. Without rulesets, +each of them needs to insert messages into the main message queue. So if +each of them wants to submit a newly arrived message into the queue at +the same time, only one can do so while the others need to wait. With +multiple rulesets, its own queue can be created for each ruleset. If now +each listener is bound to its own ruleset, concurrent message submission +is possible. On a machine with a sufficiently large number of cores, +this can result in dramatic performance improvement. + +It is highly advised that high-performance systems define a dedicated +ruleset, with a dedicated queue for each of the inputs. + +By default, rulesets do **not** have their own queue. It must be +activated via the +:doc:`$RulesetCreateMainQueue <../configuration/ruleset/rsconf1_rulesetcreatemainqueue>` +directive. + +See Also +-------- +.. toctree:: + :maxdepth: 1 + + ../historical/multi_ruleset_legacy_format_samples + diff --git a/source/concepts/netstrm_drvr.rst b/source/concepts/netstrm_drvr.rst new file mode 100644 index 0000000..182fcdf --- /dev/null +++ b/source/concepts/netstrm_drvr.rst @@ -0,0 +1,21 @@ +NetStream Drivers +================= + +Network stream drivers are a layer between various parts of rsyslogd +(e.g. the imtcp module) and the transport layer. They provide sequenced +delivery, authentication and confidentiality to the upper layers. +Drivers implement different capabilities. + +Users need to know about netstream drivers because they need to +configure the proper driver, and proper driver properties, to achieve +desired results (e.g. a :doc:`../tutorials/tls`). + +Current Network Stream Drivers +------------------------------ + +.. toctree:: + :maxdepth: 2 + + ns_ptcp + ns_gtls + ns_ossl diff --git a/source/concepts/ns_gtls.rst b/source/concepts/ns_gtls.rst new file mode 100644 index 0000000..17fe29c --- /dev/null +++ b/source/concepts/ns_gtls.rst @@ -0,0 +1,91 @@ +gtls Network Stream Driver +========================== + +This network stream driver implements a TLS +protected transport via the `GnuTLS +library <http://www.gnu.org/software/gnutls/>`_. + +**Available since:** 3.19.0 (suggested minimum 3.19.8 and above) + +Supported Driver Modes +====================== + +- **0** - unencrypted transmission (just like `ptcp <ns_ptcp.html>`_ driver) +- **1** - TLS-protected operation + +.. note:: + + Mode 0 does not provide any benefit over the ptcp driver. This + mode exists for technical reasons, but should not be used. It may be + removed in the future. + +Supported Authentication Modes +============================== + +- **anon** - anonymous authentication as described in IETF's + draft-ietf-syslog-transport-tls-12 Internet draft + +- **x509/fingerprint** - certificate fingerprint authentication as + described in IETF's draft-ietf-syslog-transport-tls-12 Internet draft. + The fingerprint must be provided as the SHA1 or the SHA256 hex string of + the certificate. Multiple values must be separated by comma (,). + A valid configuration would be e.G. + :: + + StreamDriverPermittedPeers="SHA256:10:C4:26:1D:CB:3C:AB:12:DB:1A:F0:47:37:AE:6D:D2:DE:66:B5:71:B7:2E:5B:BB:AE:0C:7E:7F:5F:0D:E9:64,SHA1:DD:23:E3:E7:70:F5:B4:13:44:16:78:A5:5A:8C:39:48:53:A6:DD:25" + +- **x509/certvalid** - certificate validation only + +- **x509/name** - certificate validation and subject name authentication as + described in IETF's draft-ietf-syslog-transport-tls-12 Internet draft + +.. note:: + + "anon" does not permit to authenticate the remote peer. As such, + this mode is vulnerable to man in the middle attacks as well as + unauthorized access. It is recommended NOT to use this mode. + A certificate/key does not need to be configured in this authmode. + +.. note:: + + **Anon mode changes in:** v8.190 (or above) + + - Anonymous Ciphers (DH and ECDH) are available in ANON mode. + Note: ECDH is not available on GnuTLS Version below 3.x. + - Server does not require a certificate anymore in anon mode. + - If Server has a certificate and the Client does not, the highest possible + ciphers will be selected. + - If both Server and Client do not have a certificate, the highest available + anon cipher will be used. + +x509/certvalid is a nonstandard mode. It validates the remote peers +certificate, but does not check the subject name. This is weak +authentication that may be useful in scenarios where multiple devices +are deployed and it is sufficient proof of authenticity when their +certificates are signed by the CA the server trusts. This is better than +anon authentication, but still not recommended. **Known Problems** + + +CheckExtendedKeyPurpose +======================= + +- **off** - by default this binary argument is turned off, which means + that Extended Key Usage extension of GNUTls certificates is ignored + in cert validation. + +- **on** - if you turn this option on, it will check that peer's certificate + contains the value for GNUTLS_KP_TLS_WWW_SERVER or GNUTLS_KP_TLS_WWW_CLIENT + respectively, depending whether we are on sending or receiving end of a + connection. + +PrioritizeSAN +============= + +- **off** - by default this binary argument is turned off, which means + that validation of names in certificates goes per older RFC 5280 and either + Subject Alternative Name or Common Name match is good and connection is + allowed. + +- **on** - if you turn this option on, it will perform stricter name checking + as per newer RFC 6125, where, if any SAN is found, contents of CN are + completely ignored and name validity is decided based on SAN only. diff --git a/source/concepts/ns_ossl.rst b/source/concepts/ns_ossl.rst new file mode 100644 index 0000000..cbd002e --- /dev/null +++ b/source/concepts/ns_ossl.rst @@ -0,0 +1,76 @@ +***************************** +openssl Network Stream Driver +***************************** + +=========================== =========================================================================== +**Driver Name:** **ossl** +**Author:** Andre Lorbach <alorbach@adiscon.com> +**Available since:** 8.36.0 +=========================== =========================================================================== + + +Purpose +======= + +This network stream driver implements a TLS protected transport +via the `OpenSSL library <https://www.openssl.org/>`_. + + +Supported Driver Modes +====================== + +- **0** - unencrypted transmission (just like `ptcp <ns_ptcp.html>`_ driver) +- **1** - TLS-protected operation + +.. note:: + + Mode 0 does not provide any benefit over the ptcp driver. This + mode exists for technical reasons, but should not be used. It may be + removed in the future. + + +Supported Authentication Modes +============================== + +- **anon** - anonymous authentication as described in IETF's + draft-ietf-syslog-transport-tls-12 Internet draft + +- **x509/fingerprint** - certificate fingerprint authentication as + described in IETF's draft-ietf-syslog-transport-tls-12 Internet draft. + The fingerprint must be provided as the SHA1 or the SHA256 hex string of + the certificate. Multiple values must be separated by comma (,). + A valid configuration would be e.G. + :: + + StreamDriverPermittedPeers="SHA256:10:C4:26:1D:CB:3C:AB:12:DB:1A:F0:47:37:AE:6D:D2:DE:66:B5:71:B7:2E:5B:BB:AE:0C:7E:7F:5F:0D:E9:64,SHA1:DD:23:E3:E7:70:F5:B4:13:44:16:78:A5:5A:8C:39:48:53:A6:DD:25" + +- **x509/certvalid** - certificate validation only. x509/certvalid is + a nonstandard mode. It validates the remote peers certificate, but + does not check the subject name. This is weak authentication that may + be useful in scenarios where multiple devices are deployed and it is + sufficient proof of authenticity when their certificates are signed by + the CA the server trusts. This is better than anon authentication, but + still not recommended. **Known Problems** + +- **x509/name** - certificate validation and subject name authentication as + described in IETF's draft-ietf-syslog-transport-tls-12 Internet draft + +.. note:: + + "anon" does not permit to authenticate the remote peer. As such, + this mode is vulnerable to man in the middle attacks as well as + unauthorized access. It is recommended NOT to use this mode. + A certificate / key does not need to be configured in this authmode. + +.. note:: + + **Anon mode changes in:** v8.190 (or above) + + - Anonymous Ciphers (DH and ECDH) are available in ANON mode. + - Server does not require a certificate anymore in anon mode. + - If Server has a certificate and the Client does not, the highest possible + ciphers will be selected. + - If both Server and Client do not have a certificate, the highest available + anon cipher will be used. + + diff --git a/source/concepts/ns_ptcp.rst b/source/concepts/ns_ptcp.rst new file mode 100644 index 0000000..66d7062 --- /dev/null +++ b/source/concepts/ns_ptcp.rst @@ -0,0 +1,13 @@ +ptcp Network Stream Driver +========================== + +This network stream driver implement a plain tcp +transport without security properties. + +Supported Driver Modes + +- 0 - unencrypted transmission + +Supported Authentication Modes + +- "anon" - no authentication diff --git a/source/concepts/queues.rst b/source/concepts/queues.rst new file mode 100644 index 0000000..bb5647c --- /dev/null +++ b/source/concepts/queues.rst @@ -0,0 +1,524 @@ +Understanding rsyslog Queues +============================ + +Rsyslog uses queues whenever two activities need to be loosely coupled. +With a queue, one part of the system "produces" something while another +part "consumes" this something. The "something" is most often syslog +messages, but queues may also be used for other purposes. + +This document provides a good insight into technical details, operation +modes and implications. In addition to it, an :doc:`rsyslog queue concepts +overview <../whitepapers/queues_analogy>` document exists which tries to explain +queues with the help of some analogies. This may probably be a better +place to start reading about queues. I assume that once you have +understood that document, the material here will be much easier to grasp +and look much more natural. + +The most prominent example is the main message queue. Whenever rsyslog +receives a message (e.g. locally, via UDP, TCP or in whatever else way), +it places these messages into the main message queue. Later, it is +dequeued by the rule processor, which then evaluates which actions are +to be carried out. In front of each action, there is also a queue, which +potentially de-couples the filter processing from the actual action +(e.g. writing to file, database or forwarding to another host). + +Where are Queues Used? +---------------------- + +Currently, queues are used for the main message queue and for the +actions. + +There is a single main message queue inside rsyslog. Each input module +delivers messages to it. The main message queue worker filters messages +based on rules specified in rsyslog.conf and dispatches them to the +individual action queues. Once a message is in an action queue, it is +deleted from the main message queue. + +There are multiple action queues, one for each configured action. By +default, these queues operate in direct (non-queueing) mode. Action +queues are fully configurable and thus can be changed to whatever is +best for the given use case. + +Future versions of rsyslog will most probably utilize queues at other +places, too. + +Wherever "*<object>*\ " is used in the config file statements, +substitute "*<object>*\ " with either "MainMsg" or "Action". The former +will set main message queue parameters, the later parameters for the +next action that will be created. Action queue parameters can not be +modified once the action has been specified. For example, to tell the +main message queue to save its content on shutdown, use +*$MainMsgQueueSaveOnShutdown on*". + +If the same parameter is specified multiple times before a queue is +created, the last one specified takes precedence. The main message queue +is created after parsing the config file and all of its potential +includes. An action queue is created each time an action selector is +specified. Action queue parameters are reset to default after an action +queue has been created (to provide a clean environment for the next +action). + +Not all queues necessarily support the full set of queue configuration +parameters, because not all are applicable. For example, disk queues +always have exactly one worker thread. This cannot be overridden by +configuration parameters. Tries to do so are ignored. + +Queue Modes +----------- + +Rsyslog supports different queue modes, some with submodes. Each of them +has specific advantages and disadvantages. Selecting the right queue +mode is quite important when tuning rsyslogd. The queue mode (aka +"type") is set via the "*$<object>QueueType*\ " config directive. + +Direct Queues +~~~~~~~~~~~~~ + +Direct queues are **non**-queuing queues. A queue in direct mode does +neither queue nor buffer any of the queue elements but rather passes the +element directly (and immediately) from the producer to the consumer. +This sounds strange, but there is a good reason for this queue type. + +Direct mode queues allow to use queues generically, even in places where +queuing is not always desired. A good example is the queue in front of +output actions. While it makes perfect sense to buffer forwarding +actions or database writes, it makes only limited sense to build up a +queue in front of simple local file writes. Yet, rsyslog still has a +queue in front of every action. So for file writes, the queue mode can +simply be set to "direct", in which case no queuing happens. + +Please note that a direct queue also is the only queue type that passes +back the execution return code (success/failure) from the consumer to +the producer. This, for example, is needed for the backup action logic. +Consequently, backup actions require the to-be-checked action to use a +"direct" mode queue. + +To create a direct queue, use the "*$<object>QueueType Direct*\ " config +directive. + +Disk Queues +~~~~~~~~~~~ + +Disk queues use disk drives for buffering. The important fact is that +they always use the disk and do not buffer anything in memory. Thus, the +queue is ultra-reliable, but by far the slowest mode. For regular use +cases, this queue mode is not recommended. It is useful if log data is +so important that it must not be lost, even in extreme cases. + +When a disk queue is written, it is done in chunks. Each chunk receives +its individual file. Files are named with a prefix (set via the +"*$<object>QueueFilename*\ " config directive) and followed by a 7-digit +number (starting at one and incremented for each file). Chunks are 10mb +by default, a different size can be set via +the"*$<object>QueueMaxFileSize*\ " config directive. Note that the size +limit is not a sharp one: rsyslog always writes one complete queue +entry, even if it violates the size limit. So chunks are actually a +little bit (usually less than 1k) larger then the configured size. Each +chunk also has a different size for the same reason. If you observe +different chunk sizes, you can relax: this is not a problem. + +Writing in chunks is used so that processed data can quickly be deleted +and is free for other uses - while at the same time keeping no +artificial upper limit on disk space used. If a disk quota is set +(instructions further below), be sure that the quota/chunk size allows +at least two chunks to be written. Rsyslog currently does not check that +and will fail miserably if a single chunk is over the quota. + +Creating new chunks costs performance but provides quicker ability to +free disk space. The 10mb default is considered a good compromise +between these two. However, it may make sense to adapt these settings to +local policies. For example, if a disk queue is written on a dedicated +200gb disk, it may make sense to use a 2gb (or even larger) chunk size. + +Please note, however, that the disk queue by default does not update its +housekeeping structures every time it writes to disk. This is for +performance reasons. In the event of failure, data will still be lost +(except when manually is mangled with the file structures). However, +disk queues can be set to write bookkeeping information on checkpoints +(every n records), so that this can be made ultra-reliable, too. If the +checkpoint interval is set to one, no data can be lost, but the queue is +exceptionally slow. + +Each queue can be placed on a different disk for best performance and/or +isolation. This is currently selected by specifying different +*$WorkDirectory* config directives before the queue creation statement. + +To create a disk queue, use the "*$<object>QueueType Disk*\ " config +directive. Checkpoint intervals can be specified via +"*$<object>QueueCheckpointInterval*\ ", with 0 meaning no checkpoints. +Note that disk-based queues can be made very reliable by issuing a +(f)sync after each write operation. Starting with version 4.3.2, this +can be requested via "*<object>QueueSyncQueueFiles on/off* with the +default being off. Activating this option has a performance penalty, so +it should not be turned on without reason. + +If you happen to lose or otherwise need the housekeeping structures and +have all yours queue chunks you can use perl script included in rsyslog +package to generate it. +Usage: recover_qi.pl -w *$WorkDirectory* -f QueueFileName -d 8 > QueueFileName.qi + + +In-Memory Queues +~~~~~~~~~~~~~~~~ + +In-memory queue mode is what most people have on their mind when they +think about computing queues. Here, the enqueued data elements are held +in memory. Consequently, in-memory queues are very fast. But of course, +they do not survive any program or operating system abort (what usually +is tolerable and unlikely). Be sure to use an UPS if you use in-memory +mode and your log data is important to you. Note that even in-memory +queues may hold data for an infinite amount of time when e.g. an output +destination system is down and there is no reason to move the data out +of memory (lying around in memory for an extended period of time is NOT +a reason). Pure in-memory queues can't even store queue elements +anywhere else than in core memory. + +There exist two different in-memory queue modes: LinkedList and +FixedArray. Both are quite similar from the user's point of view, but +utilize different algorithms. + +A FixedArray queue uses a fixed, pre-allocated array that holds pointers +to queue elements. The majority of space is taken up by the actual user +data elements, to which the pointers in the array point. The pointer +array itself is comparatively small. However, it has a certain memory +footprint even if the queue is empty. As there is no need to dynamically +allocate any housekeeping structures, FixedArray offers the best run +time performance (uses the least CPU cycle). FixedArray is best if there +is a relatively low number of queue elements expected and performance is +desired. It is the default mode for the main message queue (with a limit +of 10,000 elements). + +A LinkedList queue is quite the opposite. All housekeeping structures +are dynamically allocated (in a linked list, as its name implies). This +requires somewhat more runtime processing overhead, but ensures that +memory is only allocated in cases where it is needed. LinkedList queues +are especially well-suited for queues where only occasionally a +than-high number of elements need to be queued. A use case may be +occasional message burst. Memory permitting, it could be limited to e.g. +200,000 elements which would take up only memory if in use. A FixedArray +queue may have a too large static memory footprint in such cases. + +**In general, it is advised to use LinkedList mode if in doubt**. The +processing overhead compared to FixedArray is low and may be outweighed by +the reduction in memory use. Paging in most-often-unused pointer array +pages can be much slower than dynamically allocating them. + +To create an in-memory queue, use the "*$<object>QueueType +LinkedList*\ " or "*$<object>QueueType FixedArray*\ " config directive. + +Disk-Assisted Memory Queues +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +If a disk queue name is defined for in-memory queues (via +*$<object>QueueFileName*), they automatically become "disk-assisted" +(DA). In that mode, data is written to disk (and read back) on an +as-needed basis. + +Actually, the regular memory queue (called the "primary queue") and a +disk queue (called the "DA queue") work in tandem in this mode. Most +importantly, the disk queue is activated if the primary queue is full or +needs to be persisted on shutdown. Disk-assisted queues combine the +advantages of pure memory queues with those of pure disk queues. Under +normal operations, they are very fast and messages will never touch the +disk. But if there is need to, an unlimited amount of messages can be +buffered (actually limited by free disk space only) and data can be +persisted between rsyslogd runs. + +With a DA-queue, both disk-specific and in-memory specific configuration +parameters can be set. From the user's point of view, think of a DA +queue like a "super-queue" which does all within a single queue [from +the code perspective, there is some specific handling for this case, so +it is actually much like a single object]. + +DA queues are typically used to de-couple potentially long-running and +unreliable actions (to make them reliable). For example, it is +recommended to use a disk-assisted linked list in-memory queue in front +of each database and "send via tcp" action. Doing so makes these actions +reliable and de-couples their potential low execution speed from the +rest of your rules (e.g. the local file writes). There is a howto on +`massive database inserts <rsyslog_high_database_rate.html>`_ which +nicely describes this use case. It may even be a good read if you do not +intend to use databases. + +With DA queues, we do not simply write out everything to disk and then +run as a disk queue once the in-memory queue is full. A much smarter +algorithm is used, which involves a "high watermark" and a "low +watermark". Both specify numbers of queued items. If the queue size +reaches high watermark elements, the queue begins to write data elements +to disk. It does so until it reaches the low water mark elements. At +this point, it stops writing until either high water mark is reached +again or the on-disk queue becomes empty, in which case the queue +reverts back to in-memory mode, only. While holding at the low +watermark, new elements are actually enqueued in memory. They are +eventually written to disk, but only if the high water mark is ever +reached again. If it isn't, these items never touch the disk. So even +when a queue runs disk-assisted, there is in-memory data present (this +is a big difference to pure disk queues!). + +This algorithm prevents unnecessary disk writes, but also leaves some +additional buffer space for message bursts. Remember that creating disk +files and writing to them is a lengthy operation. It is too lengthy to +e.g. block receiving UDP messages. Doing so would result in message +loss. Thus, the queue initiates DA mode, but still is able to receive +messages and enqueue them - as long as the maximum queue size is not +reached. The number of elements between the high water mark and the +maximum queue size serves as this "emergency buffer". Size it according +to your needs, if traffic is very bursty you will probably need a large +buffer here. Keep in mind, though, that under normal operations these +queue elements will probably never be used. Setting the high water mark +too low will cause disk-assistance to be turned on more often than +actually needed. + +The water marks can be set via the "*$<object>QueueHighWatermark*\ " +and "*$<object>QueueLowWatermark*\ " configuration file directives. +Note that these are actual numbers, not percentages. Be sure they make +sense (also in respect to "*$<object>QueueSize*\ "). Rsyslogd does +perform some checks on the numbers provided, and issues warning when +numbers are "suspicious". + +Limiting the Queue Size +----------------------- + +All queues, including disk queues, have a limit of the number of +elements they can enqueue. This is set via the "*$<object>QueueSize*\ " +config parameter. Note that the size is specified in number of enqueued +elements, not their actual memory size. Memory size limits can not be +set. A conservative assumption is that a single syslog messages takes up +512 bytes on average (in-memory, NOT on the wire, this \*is\* a +difference). + +Disk assisted queues are special in that they do **not** have any size +limit. The enqueue an unlimited amount of elements. To prevent running +out of space, disk and disk-assisted queues can be size-limited via the +"*$<object>QueueMaxDiskSpace*\ " configuration parameter. If it is not +set, the limit is only available free space (and reaching this limit is +currently not very gracefully handled, so avoid running into it!). If a +limit is set, the queue can not grow larger than it. Note, however, that +the limit is approximate. The engine always writes complete records. As +such, it is possible that slightly more than the set limit is used +(usually less than 1k, given the average message size). Keeping strictly +on the limit would be a performance hurt, and thus the design decision +was to favour performance. If you don't like that policy, simply specify +a slightly lower limit (e.g. 999,999K instead of 1G). + +In general, it is a good idea to limit the physical disk space even if +you dedicate a whole disk to rsyslog. That way, you prevent it from +running out of space (future version will have an auto-size-limit logic, +that then kicks in in such situations). + +Worker Thread Pools +------------------- + +Each queue (except in "direct" mode) has an associated pool of worker +threads. Worker threads carry out the action to be performed on the data +elements enqueued. As an actual sample, the main message queue's worker +task is to apply filter logic to each incoming message and enqueue them +to the relevant output queues (actions). + +Worker threads are started and stopped on an as-needed basis. On a +system without activity, there may be no worker at all running. One is +automatically started when a message comes in. Similarly, additional +workers are started if the queue grows above a specific size. The +"*$<object>QueueWorkerThreadMinimumMessages*\ " config parameter +controls worker startup. If it is set to the minimum number of elements +that must be enqueued in order to justify a new worker startup. For +example, let's assume it is set to 100. As long as no more than 100 +messages are in the queue, a single worker will be used. When more than +100 messages arrive, a new worker thread is automatically started. +Similarly, a third worker will be started when there are at least 300 +messages, a forth when reaching 400 and so on. + +It, however, does not make sense to have too many worker threads running +in parallel. Thus, the upper limit can be set via +"*$<object>QueueWorkerThreads*\ ". If it, for example, is set to four, +no more than four workers will ever be started, no matter how many +elements are enqueued. + +Worker threads that have been started are kept running until an +inactivity timeout happens. The timeout can be set via +"*$<object>QueueWorkerTimeoutThreadShutdown*\ " and is specified in +milliseconds. If you do not like to keep the workers running, simply set +it to 0, which means immediate timeout and thus immediate shutdown. But +consider that creating threads involves some overhead, and this is why +we keep them running. If you would like to never shutdown any worker +threads, specify -1 for this parameter. + +Discarding Messages +~~~~~~~~~~~~~~~~~~~ + +If the queue reaches the so called "discard watermark" (a number of +queued elements), less important messages can automatically be +discarded. This is in an effort to save queue space for more important +messages, which you even less like to lose. Please note that whenever +there are more than "discard watermark" messages, both newly incoming as +well as already enqueued low-priority messages are discarded. The +algorithm discards messages newly coming in and those at the front of +the queue. + +The discard watermark is a last resort setting. It should be set +sufficiently high, but low enough to allow for large message burst. +Please note that it take effect immediately and thus shows effect +promptly - but that doesn't help if the burst mainly consist of +high-priority messages... + +The discard watermark is set via the "*$<object>QueueDiscardMark*\ " +directive. The priority of messages to be discarded is set via +"*$<object>QueueDiscardSeverity*\ ". This directive accepts both the +usual textual severity as well as a numerical one. To understand it, you +must be aware of the numerical severity values. They are defined in RFC +3164: + + ==== ======== + Code Severity + ==== ======== + 0 Emergency: system is unusable + 1 Alert: action must be taken immediately + 2 Critical: critical conditions + 3 Error: error conditions + 4 Warning: warning conditions + 5 Notice: normal but significant condition + 6 Informational: informational messages + 7 Debug: debug-level messages + ==== ======== + +Anything of the specified severity and (numerically) above it is +discarded. To turn message discarding off, simply specify the discard +watermark to be higher than the queue size. An alternative is to specify +the numerical value 8 as DiscardSeverity. This is also the default +setting to prevent unintentional message loss. So if you would like to +use message discarding, you need to set" +*$<object>QueueDiscardSeverity*" to an actual value. + +An interesting application is with disk-assisted queues: if the discard +watermark is set lower than the high watermark, message discarding will +start before the queue becomes disk-assisted. This may be a good thing +if you would like to switch to disk-assisted mode only in cases where it +is absolutely unavoidable and you prefer to discard less important +messages first. + +Filled-Up Queues +---------------- + +If the queue has either reached its configured maximum number of entries +or disk space, it is finally full. If so, rsyslogd throttles the data +element submitter. If that, for example, is a reliable input (TCP, local +log socket), that will slow down the message originator which is a good +resolution for this scenario. + +During throttling, a disk-assisted queue continues to write to disk and +messages are also discarded based on severity as well as regular +dequeuing and processing continues. So chances are good the situation +will be resolved by simply throttling. Note, though, that throttling is +highly undesirable for unreliable sources, like UDP message reception. +So it is not a good thing to run into throttling mode at all. + +We can not hold processing infinitely, not even when throttling. For +example, throttling the local log socket too long would cause the +system at whole come to a standstill. To prevent this, rsyslogd times +out after a configured period ("*$<object>QueueTimeoutEnqueue*\ ", +specified in milliseconds) if no space becomes available. As a last +resort, it then discards the newly arrived message. + +If you do not like throttling, set the timeout to 0 - the message will +then immediately be discarded. If you use a high timeout, be sure you +know what you do. If a high main message queue enqueue timeout is set, +it can lead to something like a complete hang of the system. The same +problem does not apply to action queues. + +Rate Limiting +~~~~~~~~~~~~~ + +Rate limiting provides a way to prevent rsyslogd from processing things +too fast. It can, for example, prevent overrunning a receiver system. + +Currently, there are only limited rate-limiting features available. The +"*$<object>QueueDequeueSlowdown*\ " directive allows to specify how +long (in microseconds) dequeueing should be delayed. While simple, it +still is powerful. For example, using a DequeueSlowdown delay of 1,000 +microseconds on a UDP send action ensures that no more than 1,000 +messages can be sent within a second (actually less, as there is also +some time needed for the processing itself). + +Processing Timeframes +~~~~~~~~~~~~~~~~~~~~~ + +Queues can be set to dequeue (process) messages only during certain +timeframes. This is useful if you, for example, would like to transfer +the bulk of messages only during off-peak hours, e.g. when you have only +limited bandwidth on the network path to the central server. + +Currently, only a single timeframe is supported and, even worse, it can +only be specified by the hour. It is not hard to extend rsyslog's +capabilities in this regard - it was just not requested so far. So if +you need more fine-grained control, let us know and we'll probably +implement it. There are two configuration directives, both should be +used together or results are unpredictable:" +*$<object>QueueDequeueTimeBegin <hour>*" +and "*$<object>QueueDequeueTimeEnd <hour>*\ ". The hour parameter must +be specified in 24-hour format (so 10pm is 22). A use case for this +parameter can be found in the `rsyslog +wiki <http://wiki.rsyslog.com/index.php/OffPeakHours>`_. + +Performance +~~~~~~~~~~~ + +The locking involved with maintaining the queue has a potentially large +performance impact. How large this is, and if it exists at all, depends +much on the configuration and actual use case. However, the queue is +able to work on so-called "batches" when dequeueing data elements. With +batches, multiple data elements are dequeued at once (with a single +locking call). The queue dequeues all available elements up to a +configured upper limit (*<object>DequeueBatchSize <number>*). It is +important to note that the actual upper limit is dictated by +availability. The queue engine will never wait for a batch to fill. So +even if a high upper limit is configured, batches may consist of fewer +elements, even just one, if there are no more elements waiting in the +queue. + +Batching can improve performance considerably. Note, however, that it +affects the order in which messages are passed to the queue worker +threads, as each worker now receive as batch of messages. Also, the +larger the batch size and the higher the maximum number of permitted +worker threads, the more main memory is needed. For a busy server, large +batch sizes (around 1,000 or even more elements) may be useful. Please +note that with batching, the main memory must hold BatchSize \* +NumOfWorkers objects in memory (worst-case scenario), even if running in +disk-only mode. So if you use the default 5 workers at the main message +queue and set the batch size to 1,000, you need to be prepared that the +main message queue holds up to 5,000 messages in main memory **in +addition** to the configured queue size limits! + +The queue object's default maximum batch size is eight, but there exists +different defaults for the actual parts of rsyslog processing that +utilize queues. So you need to check these object's defaults. + +Terminating Queues +~~~~~~~~~~~~~~~~~~ + +Terminating a process sounds easy, but can be complex. Terminating a +running queue is in fact the most complex operation a queue object can +perform. You don't see that from a user's point of view, but its quite +hard work for the developer to do everything in the right order. + +The complexity arises when the queue has still data enqueued when it +finishes. Rsyslog tries to preserve as much of it as possible. As a +first measure, there is a regular queue time out +("*$<object>QueueTimeoutShutdown*\ ", specified in milliseconds): the +queue workers are given that time period to finish processing the queue. + +If after that period there is still data in the queue, workers are +instructed to finish the current data element and then terminate. This +essentially means any other data is lost. There is another timeout +("*$<object>QueueTimeoutActionCompletion*\ ", also specified in +milliseconds) that specifies how long the workers have to finish the +current element. If that timeout expires, any remaining workers are +cancelled and the queue is brought down. + +If you do not like to lose data on shutdown, the +"*$<object>QueueSaveOnShutdown*\ " parameter can be set to "on". This +requires either a disk or disk-assisted queue. If set, rsyslogd ensures +that any queue elements are saved to disk before it terminates. This +includes data elements there were begun being processed by workers that +needed to be cancelled due to too-long processing. For a large queue, +this operation may be lengthy. No timeout applies to a required shutdown +save. diff --git a/source/concepts/rfc5424layers.png b/source/concepts/rfc5424layers.png Binary files differnew file mode 100644 index 0000000..70192cc --- /dev/null +++ b/source/concepts/rfc5424layers.png |
