diff options
Diffstat (limited to 'source/compatibility')
| -rw-r--r-- | source/compatibility/index.rst | 12 | ||||
| -rw-r--r-- | source/compatibility/v3compatibility.rst | 260 | ||||
| -rw-r--r-- | source/compatibility/v4compatibility.rst | 132 | ||||
| -rw-r--r-- | source/compatibility/v5compatibility.rst | 49 | ||||
| -rw-r--r-- | source/compatibility/v6compatibility.rst | 230 | ||||
| -rw-r--r-- | source/compatibility/v7compatibility.rst | 173 | ||||
| -rw-r--r-- | source/compatibility/v8compatibility.rst | 200 |
7 files changed, 1056 insertions, 0 deletions
diff --git a/source/compatibility/index.rst b/source/compatibility/index.rst new file mode 100644 index 0000000..073c907 --- /dev/null +++ b/source/compatibility/index.rst @@ -0,0 +1,12 @@ +Compatibility +============= + +.. toctree:: + :glob: + + v8compatibility + v7compatibility + v6compatibility + v5compatibility + v4compatibility + v3compatibility diff --git a/source/compatibility/v3compatibility.rst b/source/compatibility/v3compatibility.rst new file mode 100644 index 0000000..502ca64 --- /dev/null +++ b/source/compatibility/v3compatibility.rst @@ -0,0 +1,260 @@ +Compatibility Notes for rsyslog v3 +================================== + +*Written by* `Rainer Gerhards <http://www.gerhards.net/rainer>`_ +*(2008-03-28)* + +Rsyslog aims to be a drop-in replacement for sysklogd. However, version +3 has some considerable enhancements, which lead to some backward +compatibility issues both in regard to sysklogd and rsyslog v1 and v2. +Most of these issues are avoided by default by not specifying the -c +option on the rsyslog command line. That will enable +backwards-compatibility mode. However, please note that things may be +suboptimal in backward compatibility mode, so the advise is to work +through this document, update your rsyslog.conf, remove the no longer +supported startup options and then add -c3 as the first option to the +rsyslog command line. That will enable native mode. + +Please note that rsyslogd helps you during that process by logging +appropriate messages about compatibility mode and +backwards-compatibility statements automatically generated. You may +want your syslogd log for those. They immediately follow rsyslogd's +startup message. + +Inputs +------ + +With v2 and below, inputs were automatically started together with +rsyslog. In v3, inputs are optional! They come in the form of plug-in +modules. **At least one input module must be loaded to make rsyslog do +any useful work.** The config file directives doc briefly lists which +config statements are available by which modules. + +It is suggested that input modules be loaded in the top part of the +config file. Here is an example, also highlighting the most important +modules: + +:: + + $ModLoad immark # provides --MARK-- message capability + $ModLoad imudp # provides UDP syslog reception + $ModLoad imtcp # provides TCP syslog reception + $ModLoad imgssapi # provides GSSAPI syslog reception + $ModLoad imuxsock # provides support for local system logging (e.g. via logger command) + $ModLoad imklog # provides kernel logging support (previously done by rklogd) + +Command Line Options +-------------------- + +A number of command line options have been removed. New config file +directives have been added for them. The -h and -e option have been +removed even in compatibility mode. They are ignored but an informative +message is logged. Please note that -h was never supported in v2, but +was silently ignored. It disappeared some time ago in the final v1 +builds. It can be replaced by applying proper filtering inside +syslog.conf. + +-c option / Compatibility Mode +------------------------------ + +The -c option is new and tells rsyslogd about the desired backward +compatibility mode. It must always be the first option on the command +line, as it influences processing of the other options. To use the +rsyslog v3 native interface, specify -c3. To use compatibility mode , +either do not use -c at all or use -c<vers> where vers is the rsyslog +version that it shall be compatible to. Use -c0 to be command-line +compatible to sysklogd. + +Please note that rsyslogd issues warning messages if the -c3 command +line option is not given. This is to alert you that your are running in +compatibility mode. Compatibility mode interferes with your rsyslog.conf +commands and may cause some undesired side-effects. It is meant to be +used with a plain old rsyslog.conf - if you use new features, things +become messy. So the best advise is to work through this document, +convert your options and config file and then use rsyslog in native +mode. In order to aid you in this process, rsyslog logs every +compatibility-mode config file directive it has generated. So you can +simply copy them from your logfile and paste them to the config. + +-e Option +--------- + +This option is no longer supported, as the "last message repeated n +times" feature is now turned off by default. We changed this default +because this feature is causing a lot of trouble and we need to make it +either go away or change the way it works. For more information, please +see our dedicated `forum thread on "last message repeated n +times" <http://www.rsyslog.com/PNphpBB2-viewtopic-p-1130.phtml>`_. This +thread also contains information on how to configure rsyslogd so that it +continues to support this feature (as long as it is not totally +removed). + +-m Option +--------- + +The -m command line option is emulated in compatibility mode. To replace +it, use the following config directives (compatibility mode +auto-generates them): + +:: + + $ModLoad immark + $MarkMessagePeriod 1800 # 30 minutes + +-r Option +--------- + +Is no longer available in native mode. However, it is understood in +compatibility mode (if no -c option is given). Use the ``$UDPSeverRun +<port>`` config file directives. You can now also set the local address +the server should listen to via ``$UDPServerAddress <ip>`` config +directive. + +The following example configures an UDP syslog server at the local +address 192.0.2.1 on port 514: + +:: + + $ModLoad imudp + $UDPServerAddress 192.0.2.1 # this MUST be before the $UDPServerRun directive! + $UDPServerRun 514 + +``$UDPServerAddress \*`` means listen on all local interfaces. This is the +default if no directive is specified. + +Please note that now multiple listeners are supported. For example, you +can do the following: + +:: + + $ModLoad imudp + $UDPServerAddress 192.0.2.1 # this MUST be before the $UDPServerRun directive! + $UDPServerRun 514 + $UDPServerAddress \* # all local interfaces + $UDPServerRun 1514 + +These config file settings run two listeners: one at 192.0.2.1:514 and +one on port 1514, which listens on all local interfaces. + +Default port for UDP (and TCP) Servers +-------------------------------------- + +Please note that with pre-v3 rsyslogd, a service database lookup was +made when a UDP server was started and no port was configured. Only if +that failed, the IANA default of 514 was used. For TCP servers, this +lookup was never done and 514 always used if no specific port was +configured. For consistency, both TCP and UDP now use port 514 as +default. If a lookup is desired, you need to specify it in the "Run" +directive, e.g. "*$UDPServerRun syslog*\ ". + +klogd +----- + +klogd has (finally) been replaced by a loadable input module. To enable +klogd functionality, do + +:: + + $ModLoad imklog + +Note that this can not be handled by the compatibility layer, as klogd +was a separate binary. A limited set of klogd command line settings is +now supported via rsyslog.conf. That set of configuration directives is +to be expanded. + +Output File Syncing +------------------- + +Rsyslogd tries to keep as compatible to stock syslogd as possible. As +such, it retained stock syslogd's default of syncing every file write if +not specified otherwise (by placing a dash in front of the output file +name). While this was a useful feature in past days where hardware was +much less reliable and UPS seldom, this no longer is useful in today's +world. Instead, the syncing is a high performance hit. With it, rsyslogd +writes files around 50 **times** slower than without it. It also affects +overall system performance due to the high IO activity. In rsyslog v3, +syncing has been turned off by default. This is done via a specific +configuration directive + +:: + $ActionFileEnableSync on/off + +which is off by +default. So even if rsyslogd finds sync selector lines, it ignores them +by default. In order to enable file syncing, the administrator must +specify ``$ActionFileEnableSync on`` at the top of rsyslog.conf. This +ensures that syncing only happens in some installations where the +administrator actually wanted that (performance-intense) feature. In the +fast majority of cases (if not all), this dramatically increases +rsyslogd performance without any negative effects. + +Output File Format +------------------ + +Rsyslog supports high precision RFC 3339 timestamps and puts these into +local log files by default. This is a departure from previous syslogd +behaviour. We decided to sacrifice some backward-compatibility in an +effort to provide a better logging solution. Rsyslog has been supporting +the high-precision timestamps for over three years as of this writing, +but nobody used them because they were not default (one may also assume +that most people didn't even know about them). Now, we are writing the +great high-precision time stamps, which greatly aid in getting the right +sequence of logging events. If you do not like that, you can easily turn +them off by placing + +:: + + $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat + +right at the start of your rsyslog.conf. This will use the previous +format. Please note that the name is case-sensitive and must be +specified exactly as shown above. Please also note that you can of +course use any other format of your liking. To do so, simply specify the +template to use or set a new default template via the +$ActionFileDefaultTemplate directive. Keep in mind, though, that +templates must be defined before they are used. + +Keep in mind that when receiving messages from remote hosts, the +timestamp is just as precise as the remote host provided it. In most +cases, this means you will only a receive a standard timestamp with +second precision. If rsyslog is running at the remote end, you can +configure it to provide high-precision timestamps (see below). + +Forwarding Format +----------------- + +When forwarding messages to remote syslog servers, rsyslogd by default +uses the plain old syslog format with second-level resolution inside the +timestamps. We could have made it emit high precision timestamps. +However, that would have broken almost all receivers, including earlier +versions of rsyslog. To avoid this hassle, high-precision timestamps +need to be explicitly enabled. To make this as painless as possible, +rsyslog comes with a canned template that contains everything necessary. + To enable high-precision timestamps, just use: + +:: + + $ActionForwardDefaultTemplate RSYSLOG_ForwardFormat # for plain TCP and UDP + $ActionGSSForwardDefaultTemplate RSYSLOG_ForwardFormat # for GSS-API + +And, of course, you can always set different forwarding formats by just +specifying the right template. + +If you are running in a system with only rsyslog 3.12.5 and above in the +receiver roles, it is suggested to add one (or both) of the above +statements to the top of your rsyslog.conf (but after the $ModLoad's!) - +that will enable you to use the best in timestamp support available. +Please note that when you use this format with other receivers, they +will probably become pretty confused and not detect the timestamp at +all. In earlier rsyslog versions, for example, that leads to duplication +of timestamp and hostname fields and disables the detection of the +original hostname in a relayed/NATed environment. So use the new format +with care. + +Queue Modes for the Main Message Queue +-------------------------------------- + +Either "FixedArray" or "LinkedList" is recommended. "Direct" is +available, but should not be used except for a very good reason +("Direct" disables queueing and will potentially lead to message loss on +the input side). diff --git a/source/compatibility/v4compatibility.rst b/source/compatibility/v4compatibility.rst new file mode 100644 index 0000000..ddbce3e --- /dev/null +++ b/source/compatibility/v4compatibility.rst @@ -0,0 +1,132 @@ +Compatibility Notes for rsyslog v4 +================================== + +*Written by* `Rainer Gerhards <http://www.gerhards.net/rainer>`_ +*(2009-07-15)* + +The changes introduced in rsyslog v4 are numerous, but not very +intrusive. This document describes things to keep in mind when moving +from v3 to v4. It does not list enhancements nor does it talk about +compatibility concerns introduced by v3 (for this, see the `rsyslog v3 +compatibility notes <v3compatibility.html>`_). + +HUP processing +-------------- + +With v3 and below, rsyslog used the traditional HUP behaviour. That +meant that all output files are closed and the configuration file is +re-read and the new configuration applied. + +With a program as simple and static as sysklogd, this was not much of an +issue. The most important config settings (like udp reception) of a +traditional syslogd can not be modified via the configuration file. So a +config file reload only meant setting up a new set of filters. It also +didn't account as problem that while doing so messages may be lost - +without any threading and queuing model, a traditional syslogd will +potentially always loose messages, so it is irrelevant if this happens, +too, during the short config re-read phase. + +In rsyslog, things are quite different: the program is more or less a +framework into which loadable modules are loaded as needed for a +particular configuration. The software that will actually be running is +tailored via the config file. Thus, a re-read of the config file +requires a full, very heavy restart, because the software actually +running with the new config can be totally different from what ran with +the old config. + +Consequently, the traditional HUP is a very heavy operation and may even +cause some data loss because queues must be shut down, listeners stopped +and so on. Some of these operations (depending on their configuration) +involve intentional message loss. The operation also takes up a lot of +system resources and needs quite some time (maybe seconds) to be +completed. During this restart period, the syslog subsystem is not fully +available. + +From the software developer's point of view, the full restart done by a +HUP is rather complex, especially if user-timeout limits set on action +completion are taken into consideration (for those in the know: at the +extreme ends this means we need to cancel threads as a last resort, but +than we need to make sure that such cancellation does not happen at +points where it would be fatal for a restart). A regular restart, where +the process is actually terminated, is much less complex, because the +operating system does a full cleanup after process termination, so +rsyslogd does not need to take care for exotic cleanup cases and leave +that to the OS. In the end result, restart-type HUPs clutter the code, +increase complexity (read: add bugs) and cost performance. + +On the contrary, a HUP is typically needed for log rotation, and the +real desire is to close files. This is a non-disruptive and very +lightweight operation. + +Many people have said that they are used to HUP the syslogd to apply +configuration changes. This is true, but it is questionable if that +really justifies all the cost that comes with it. After all, it is the +difference between typing + +:: + + $ kill -HUP `cat /var/run/rsyslogd.pid` + +versus + +:: + + $ /etc/init.d/rsyslog restart + +Semantically, both is mostly the same thing. The only difference is that +with the restart command rsyslogd can spit config error message to +stderr, so that the user is able to see any problems and fix them. With +a HUP, we do not have access to stderr and thus can log error messages +only to their configured destinations; experience tells that most users +will never find them there. What, by the way, is another strong argument +against restarting rsyslogd by HUPing it. + +So a restart via HUP is not strictly necessary and most other daemons +require that a restart command is typed in if a restart is required. + +Rsyslog will follow this paradigm in the next versions, resulting in +many benefits. In v4, we provide some support for the old-style +semantics. We introduced a setting $HUPisRestart which may be set to +"on" (traditional, heavy operation) or "off" (new, lightweight "file +close only" operation). The initial versions had the default set to +traditional behavior, but starting with 4.5.1 we are now using the new +behavior as the default. + +Most importantly, **this may break some scripts**, but my sincere belief +is that there are very few scripts that automatically **change** +rsyslog's config and then do a HUP to reload it. Anyhow, if you have +some of these, it may be a good idea to change them now instead of +turning restart-type HUPs on. Other than that, one mainly needs to +change the habit of how to restart rsyslog after a configuration change. + +**Please note that restart-type HUP is deprecated and will go away in +rsyslog v5.** So it is a good idea to become ready for the new version +now and also enjoy some of the benefits of the "real restart", like the +better error-reporting capability. + +Note that code complexity reduction (and thus performance improvement) +needs the restart-type HUP code to be removed, so these changes can (and +will) only happen in version 5. + +outchannels +----------- + +Note: as always documented, outchannels are an experimental feature that +may be removed and/or changed in the future. There is one concrete +change done starting with 4.6.7: let's assume an outchannel "mychannel" +was defined. Then, this channel could be used inside an + +\*.\* $mychannel + +This is still supported and will remain to be +supported in v4. However, there is a new variant which explicitly tells +this is to be handled by omfile. This new syntax is as follows: + +\*.\* :omfile:$mychannel + +Note that future versions, specifically +starting with v6, the older syntax is no longer supported. So users are +strongly advised to switch to the new syntax. As an aid to the +conversion process, rsyslog 4.7.4 and above issue a warning message if +the old-style directive is seen -- but still accept the old syntax +without any problems. diff --git a/source/compatibility/v5compatibility.rst b/source/compatibility/v5compatibility.rst new file mode 100644 index 0000000..c7e1865 --- /dev/null +++ b/source/compatibility/v5compatibility.rst @@ -0,0 +1,49 @@ +Compatibility Notes for rsyslog v5 +================================== + +*Written by* `Rainer Gerhards <http://www.gerhards.net/rainer>`_ +*(2009-07-15)* + +The changes introduced in rsyslog v5 are numerous, but not very +intrusive. This document describes things to keep in mind when moving +from v4 to v5. It does not list enhancements nor does it talk about +compatibility concerns introduced by earlier versions (for this, see +their respective compatibility documents). + +HUP processing +-------------- + +The $HUPisRestart directive is supported by some early v5 versions, but +has been removed in 5.1.3 and above. That means that restart-type HUP +processing is no longer available. This processing was redundant and had +a lot a drawbacks. For details, please see the `rsyslog v4 compatibility +notes <v4compatibility.html>`_ which elaborate on the reasons and the +(few) things you may need to change. + +Please note that starting with 5.8.11 HUP will also requery the local +hostname. + +Queue on-disk format +-------------------- + +The queue information file format has been changed. When upgrading from +v4 to v5, make sure that the queue is emptied and no on-disk structure +present. We did not go great length in understanding the old format, as +there was too little demand for that (and it being quite some effort if +done right). + +Queue Worker Thread Shutdown +---------------------------- + +Previous rsyslog versions had the capability to "run" on zero queue +worker if no work was required. This was done to save a very limited +number of resources. However, it came at the price of great complexity. +In v5, we have decided to let a minimum of one worker run all the time. +The additional resource consumption is probably not noticeable at all, +however, this enabled us to do some important code cleanups, resulting +in faster and more reliable code (complex code is hard to maintain and +error-prone). From the regular user's point of view, this change should +be barely noticeable. I am including the note for expert users, who will +notice it in rsyslog debug output and other analysis tools. So it is no +error if each queue in non-direct mode now always runs at least one +worker thread. diff --git a/source/compatibility/v6compatibility.rst b/source/compatibility/v6compatibility.rst new file mode 100644 index 0000000..1023acc --- /dev/null +++ b/source/compatibility/v6compatibility.rst @@ -0,0 +1,230 @@ +Compatibility Notes for rsyslog v6 +================================== + +This document describes things to keep in mind when moving from v5 to v6. It +does not list enhancements nor does it talk about compatibility concerns introduced +by earlier versions (for this, see their respective compatibility documents). Its focus +is primarily on what you need to know if you used a previous version and want to use the +current one without hassle. + +Version 6 offers a better config language and some other improvements. +As the config system has many ties into the rsyslog engine AND all plugins, +the changes are somewhat intrusive. Note, however, that core processing has +not been changed much in v6 and will not. So once the configuration is loaded, +the stability of v6 is quite comparable to v5. + +Property "pri-text" +------------------- +Traditionally, this property did not only return the textual form +of the pri ("local0.err"), but also appended the numerical value to it +("local0.err<133>"). This sounds odd and was left unnoticed for some years. +In October 2011, this odd behaviour was brought up on the rsyslog mailing list +by Gregory K. Ruiz-Ade. Code review showed that the behaviour was intentional, +but no trace of what the intention was when it was introduced could be found. +The documentation was also unclear, it said no numerical value was present, +but the samples had it. We agreed that the additional numerical value is +of disadvantage. We also guessed that this property is very rarely being used, +otherwise the problem should have been raised much earlier. However, we +didn't want to change behaviour in older builds. So v6 was set to clean up +the situation. In v6, text-pri will always return the textual part only +("local0.err") and the numerical value will not be contained any longer inside +the string. If you actually need that value, it can fairly easily be added +via the template system. +**If you have used this property previously and relied on the numerical +part, you need to update your rsyslog configuration files.** + +Plugin ABI +---------- +The plugin interface has considerably been changed to support the new +config language. All plugins need to be upgraded. This usually does not require +much coding. However, if the new config language shall be supported, more +changes must be made to plugin code. All project-supported plugins have been +upgraded, so this compatibility issue is only of interest for you if you have +custom plugins or use some user-contributed plugins from the rsyslog project +that are not maintained by the project itself (omoracle is an example). Please +expect some further plugin instability during the initial v6 releases. + +RainerScript based rsyslog.conf +------------------------------- +A better config format was the main release target for rsyslog v6. It comes in the +flavor of so-called RainerScript +`(why the name RainerScript?) +<https://rainer.gerhards.net/2008/02/introducing-rainerscript-and-some.html>`_ +RainerScript supports legacy syslog.conf format, much as you know it +from other syslogds (like sysklogd or the BSD syslogd) as well as previous versions +of rsyslog. Initial work on RainerScript began in v4, and the if-construct was already +supported in v4 and v5. Version 6 has now taken this further. After long discussions we +decided to use the legacy format as a basis, and lightly extend it by native RainerScript +constructs. The main goal was to make sure that previous knowledge and config systems +could still be used while offering a much more intuitive and powerful way of configuring +rsyslog. + +RainerScript has been implemented from scratch and with new tools (flex/bison, for those in the +know). Starting with 6.3.3, this new config file processor replaces the legacy one. Note that +the new processor handles all formats, extended RainerScript as well as legacy syslog.conf format. +There are some legacy construct that were especially hard to translate. You'll read about them in +other parts of this document (especially outchannels, which require a format change). + +In v6, all legacy formats are supported. In the long term, we may remove some of the ugly +rsyslog-specific constructs. Good candidates are all configuration commands starting with +a dollar sign, like "$ActionFileDefaultTemplate"). However, this will not be the case before +rsyslog v7 or (much more likely) v8/9. Right now, you also need to use these commands, because +not all have already been converted to the new RainerScript format. + +In 6.3.3, the new parser is used, but almost none of the extended RainerScript capabilities +are available. They will incrementally be introduced with the following releases. Note that for +some features (most importantly if-then-else nested blocks), the v6 core engine is not +capable enough. It is our aim to provide a much better config language to as many rsyslog +users as quickly as possible. As such, we refrain from doing big engine changes in v6. This +in turn means we cannot introduce some features into RainerScript that we really want to see. +These features will come up with rsyslog v7, which will have even better flow control +capabilities inside the core engine. Note that v7 will fully support v6 RainerScript. +Let us also say that the v6 version is not a low-end quick hack: it offers full-fledged +syslog message processing control, capable of doing the best you can find inside the +industry. We just say that v7 will come up with even more advanced capabilities. + +Please note that we tried hard to make the RainerScript parser compatible with +all legacy config files. However, we may have failed in one case or another. So if you +experience problems during config processing, chances are there may be a problem +on the rsyslog side. In that case, please let us know. + +Please see the +`blog post about rsyslog 6.3.3 config format +<https://rainer.gerhards.net/2011/07/rsyslog-633-config-format-improvements.html>`_ +for details of what is currently supported. + +compatibility mode +------------------ +Compatibility mode (specified via -c option) has been removed. This was a migration aid from +sysklogd and very early versions of rsyslog. As all major distros now have rsyslog as their +default, and thus ship rsyslog-compliant config files, there is no longer a need for +compatibility mode. Removing it provides easier to maintain code. Also, practice has shown +that many users were confused by compatibility mode (and even some package maintainers got +it wrong). So this not only cleans up the code but rather removes a frequent source of +error. + +It must be noted, though, that this means rsyslog is no longer a 100% drop-in replacement +for sysklogd. If you convert an extremely old system, you need to checks its config and +probably need to apply some very mild changes to the config file. + +abort on config errors +---------------------- +Previous versions accepted some malformedness inside the config file without aborting. This +could lead to some uncertainty about which configuration was actually running. In v6 there +are some situations where config file errors can not be ignored. In these cases rsyslog +emits error messages to stderr, and then exists with a non-zero exit code. It is important +to check for those cases as this means log data is potentially lost. +Please note that +the root problem is the same for earlier versions as well. With them, it was just harder +to spot why things went wrong (and if at all). + +Default Batch Sizes +------------------- +Due to their positive effect on performance and comparatively low overhead, +default batch sizes have been increased. Starting with 6.3.4, the action queues +have a default batch size of 128 messages. + +Default action queue enqueue timeout +------------------------------------ +This timeout previously was 2 seconds, and has been reduced to 50ms (starting with 6.5.0). This change +was made as a long timeout will caused delays in the associated main queue, something +that was quite unexpected to users. Now, this can still happen, but the effect is much +less harsh (but still considerable on a busy system). Also, 50ms should be fairly enough +for most output sources, except when they are really broken (like network disconnect). If +they are really broken, even a 2second timeout does not help, so we hopefully get the best +of both worlds with the new timeout. A specific timeout can of course still be configured, +it is just the timeout that changed. + +outchannels +----------- +Outchannels are a to-be-removed feature of rsyslog, at least as far as the config +syntax is concerned. Nevertheless, v6 still supports it, but a new syntax is required +for the action. Let's assume your outchannel is named "channel". The previous syntax was + +:: + + *.* $channel + +This was deprecated in v5 and no longer works in v6. Instead, you need to specify + +:: + + *.* :omfile:$channel + +Note that this syntax is available starting with rsyslog v4. It is important to keep on your +mind that future versions of rsyslog will require different syntax and/or drop outchannel support +completely. So if at all possible, avoid using this feature. If you must use it, be prepared for +future changes and watch announcements very carefully. + +ompipe default template +----------------------- +Starting with 6.5.0, ompipe does no longer use the omfile default template. +Instead, the default template must be set via the module load statement. +An example is + +:: + + module(load="builtin:ompipe" template="myDefaultTemplate") + +For obvious reasons, the default template must be defined somewhere in +the config file, otherwise errors will happen during the config load +phase. + +omusrmsg +-------- +The omusrmsg module is used to send messages to users. In legacy-legacy +config format (that is the very old sysklogd style), it was sufficient to use +just the user name to call this action, like in this example: + +:: + + *.* rgerhards + +This format is very ambiguous and causes headache (see +`blog post on omusrmsg <https://rainer.gerhards.net/2011/07/why-omusrmsg-is-evil-and-how-it-is.html>`_ +for details). Thus the format has been superseded by this syntax +(which is legacy format ;-)): + +:: + + *.* :omusrmsg:rgerhards + +That syntax is supported since later subversions of version 4. + +Rsyslog v6 still supports the legacy-legacy format, but in a very strict +sense. For example, if multiple users or templates are given, no spaces +must be included in the action line. For example, this works up to v5, but no +longer in v6: + +:: + + *.* rgerhards, bgerhards + +To fix it in a way that is compatible with pre-v4, use (note the removed space!): + +:: + + *.* rgerhards,bgerhards + +Of course, it probably is better to understand in native v6 format: + +:: + + *.* action(type="omusrmsg" users="rgerhards, bgerhards") + +As you see, here you may include spaces between user names. + +In the long term, legacy-legacy format will most probably totally disappear, +so it is a wise decision to change config files at least to the legacy +format (with ":omusrmsg:" in front of the name). + +Escape Sequences in Script-Based Filters +---------------------------------------- +In v5, escape sequences were very simplistic. Inside a string, "\x" meant +"x" with x being any character. This has been changed so that the usual set of +escapes is supported, must importantly "\n", "\t", "\xhh" (with hh being hex digits) +and "\ooo" with (o being octal digits). So if one of these sequences was used +previously, results are obviously different. However, that should not create any +real problems, because it is hard to envision why someone should have done that +(why write "\n" when you can also write "n"?). + diff --git a/source/compatibility/v7compatibility.rst b/source/compatibility/v7compatibility.rst new file mode 100644 index 0000000..ba6a1de --- /dev/null +++ b/source/compatibility/v7compatibility.rst @@ -0,0 +1,173 @@ +Compatibility Notes for rsyslog v7 +================================== + +This document describes things to keep in mind when moving from v6 to v7. It +does not list enhancements nor does it talk about compatibility concerns +introduced by earlier versions (for this, see their respective compatibility +documents). Its focus is primarily on what you need to know if you used v6 +and want to use v7 without hassle. + +Version 7 builds on the new config language introduced in v6 and extends it. +Other than v6, it not just only extends the config language, but provides +considerable changes to core elements as well. The result is much more power and +ease of use as well (this time that is not contradictionary). + +BSD-Style blocks +---------------- +BSD style blocks are no longer supported (for good reason). See the +`rsyslog BSD blocks info page <http://www.rsyslog.com/g/BSD>`_ +for more information and how to upgrade your config. + +CEE-Properties +-------------- + +In rsyslog v6, CEE properties could not be used across disk-based queues. If this was +done, their content was reset. This was a missing feature in v6. In v7, this feature +has been implemented. Consequently, situations where the previous behaviour were +desired need now to be solved differently. We do not think that this will cause any +problems to anyone, especially as in v6 this was announced as a missing feature. + +omusrmsg: using just a username or "*" is deprecated +---------------------------------------------------- +In legacy config format, the asterisk denotes writing the message to all users. +This is usually used for emergency messages and configured like this: + +:: + + *.emerg * + +Unfortunately, the use of this single character conflicts with other uses, for +example with the multiplication operator. While rsyslog up to versions v7.4 preserves the meaning of +asterisk as an action, it is deprecated and will probably be removed in future versions. +Consequently, a warning message is emitted. To make this warning go away, the action must +be explicitly given, as follows: + +:: + + *.emerg :omusrmsg:* + +The same holds true for user names. For example + +:: + + *.emerg john + +at a minimum should be rewritten as + +:: + + *.emerg :omusrmsg:john + +Of course, for even more clarity the new RainerScript style of action can +also be used: + +:: + + *.emerg action(type="omusrmsg" users="john") + +In Rainer's blog, there is more +`background information on why omusrmsg needed to be changed <https://rainer.gerhards.net/2011/07/why-omusrmsg-is-evil-and-how-it-is.html>`_ +available. + +omruleset and discard (~) action are deprecated +----------------------------------------------- +Both continue to work, but have been replaced by better alternatives. + +The discard action (tilde character) has been replaced by the "stop" +RainerScript directive. It is considered more intuitive and offers slightly +better performance. + +The omruleset module has been replaced by the "call" RainerScript directive. +Call permits to execute a ruleset like a subroutine, and does so with much +higher performance than omruleset did. Note that omruleset could be run off +an async queue. This was more a side than a desired effect and is not supported +by the call statement. If that effect was needed, it can simply be simulated by +running the called rulesets actions asynchronously (what in any case is the right +way to handle this). + +Note that the deprecated modules emit warning messages when being used. +They tell that the construct is deprecated and which statement is to be used +as replacement. This does **not** affect operations: both modules are still +fully operational and will not be removed in the v7 timeframe. + +Retries of output plugins that do not do proper replies +------------------------------------------------------- +Some output plugins may not be able to detect if their target is capable of +accepting data again after an error (technically, they always return OK when +TryResume is called). Previously, the rsyslog core engine suspended such an action +after 1000 successive failures. This lead to potentially a large amount of +errors and error messages. Starting with 7.2.1, this has been reduced to 10 +successive failures. This still gives the plugin a chance to recover. In extreme +cases, a plugin may now enter suspend mode where it previously did not do so. +In practice, we do NOT expect that. + +omfile: file creation time +-------------------------- +Originally, omfile created files immediately during startup, no matter if +they were written to or not. In v7, this has changed. Files are only created +when they are needed for the first time. + +Also, in pre-v7 files were created *before* privileges were dropped. This meant +that files could be written to locations where the actual desired rsyslog +user was *not* permitted to. In v7, this has been fixed. This is fix also +the prime reason that files are now created on demand (which is later in the +process and after the privilege drop). + +Notes for the 7.3/7.4 branch +---------------------------- + +"last message repeated n times" Processing +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +This processing has been optimized and moved to the input side. This results +in usually far better performance and also de-couples different sources +from the same +processing. It is now also integrated in to the more generic rate-limiting +processing. + +User-Noticable Changes +...................... +The code works almost as before, with two exceptions: + +* The suppression amount can be different, as the new algorithm + precisely check's a single source, and while that source is being + read. The previous algorithm worked on a set of mixed messages + from multiple sources. +* The previous algorithm wrote a "last message repeated n times" message + at least every 60 seconds. For performance reasons, we do no longer do + this but write this message only when a new message arrives or rsyslog + is shut down. + +Note that the new algorithms needs support from input modules. If old +modules which do not have the necessary support are used, duplicate +messages will most probably not be detected. Upgrading the module code is +simple, and all rsyslog-provided plugins support the new method, so this +should not be a real problem (crafting a solution would result in rather +complex code - for a case that most probably would never happen). + +Performance Implications +........................ +In general, the new method enables far faster output processing. However, it +needs to be noted that the "last message repeated n" processing needs parsed +messages in order to detect duplicated. Consequently, if it is enabled the +parser step cannot be deferred to the main queue processing thread and +thus must be done during input processing. The changes workload distribution +and may have (good or bad) effect on the overall performance. If you have +a very high performance installation, it is suggested to check the performance +profile before deploying the new version. + +**Note:** for high-performance +environments it is highly recommended NOT to use "last message repeated n times" +processing but rather the other (more efficient) rate-limiting methods. These +also do NOT require the parsing step to be done during input processing. + +Stricter string-template Processing +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Previously, no error message for invalid string template parameters +was generated. +Rather a malformed template was generated, and error information emitted +at runtime. However, this could be quite confusing. Note that the new code +changes user experience: formerly, rsyslog and the affected +actions properly started up, but the actions did not produce proper +data. Now, there are startup error messages and the actions are NOT +executed (due to missing template due to template error). diff --git a/source/compatibility/v8compatibility.rst b/source/compatibility/v8compatibility.rst new file mode 100644 index 0000000..e2abc31 --- /dev/null +++ b/source/compatibility/v8compatibility.rst @@ -0,0 +1,200 @@ +Compatibility Notes for rsyslog v8 +================================== + +This document describes things to keep in mind when moving from v7 to +v8. It does not list enhancements nor does it talk about compatibility +concerns introduced by earlier versions (for this, see their respective +compatibility documents). Its focus is primarily on what you need to +know if you used v7 and want to use v8 without hassle. + +Version 8 offers a completely rewritten core rsyslog engine. This +resulted in a number of changes that are visible to users and (plugin) +developers. Most importantly, pre-v8 plugins **do not longer work** and +need to be updated to support the new calling interfaces. If you +developed a plugin, be sure to review the developer section below. + +Mark Messages +------------- + +In previous versions, mark messages were by default only processed if an +action was not executed for some time. The default has now changed, and +mark messages are now always processed. Note that this enables faster +processing inside rsyslog. To change to previous behaviour, you need to +add action.writeAllMarkMessages="off" to the actions in question. + +Untested Modules +---------------- + +The following modules have been updated and successfully build, but no +"real" test were conducted. Users of these modules should use extra +care. + +- mmsequence +- plugins/omgssapi +- omsnmp +- mmfields +- mmpstrucdata +- plugins/mmaudit +- ommongodb - larger changes still outstanding +- ompgsql +- plugins/omrabbitmq - not project supported +- plugins/omzmq3 - not project supported +- plugins/omhdfs (transaction support should be improved, requires sponsor) +- omuxsock + +In addition to bug reports, success reports are also appreciated for +these modules (this may save us testing). + +What Developers need to Know +---------------------------- + +output plugin interface +~~~~~~~~~~~~~~~~~~~~~~~ + +To support the new core engine, the output interface has been +considerably changed. It is suggested to review some of the +project-provided plugins for full details. In this doc, we describe the +most important changes from a high level perspective. + +**Multi-thread awareness required** + +The new engine activates one **worker**\ instance of output actions on +each worker thread. This means an action has now three types of data: + +- global +- action-instance - previously known pData, one for each action inside + the config +- worker-action-instance - one for each worker thread (called + pWrkrData), note that this is specific to exactly one pData + +The plugin **must** now by multi-threading aware. It may be called by +multiple threads concurrently, but it is guaranteed that each call is +for a unique pWrkrData structure. This still permits to write plugins +easily, but enables the engine to work with much higher performance. +Note that plugin developers should assume it is the norm that multiple +concurrent worker action instances are active a the some time. + +**New required entry points** + +In order to support the new threading model, new entry points are +required. Most importantly, only the plugin knows which data must be +present in pData and pWrkrData, so it must created and destroy these +data structures on request of the engine. Note that pWrkrData may be +destroyed at any time and new ones re-created later. Depending on +workload structure and configuration, this can happen frequently. + +New entry points are: + +- createWrkrInstance +- freeWrkrInstance + +The calling interface for these entry points has changed. Basically, +they now receive a pWrkrData object instead pData. It is assumed that +createWrkrInstance populates pWrkrData->pData appropriately. + +- beginTransaction +- doAction +- endTransaction + +**Changed entry points** + +Some of the existing entry points have been changed. + +The **doAction** entry point formerly got a variable ``iMsgOpts`` which +is no longer provided. This variable was introduced in early days and +exposed some internal message state information to the output module. +Review of all known existing plugins showed that none except omfile ever +used that variable. And omfile only did so to do some no longer required +legacy handling. + +In essence, it is highly unlikely that you ever accessed this variable. +So we expect that nobody actually notices that the variable has gone +away. + +Removal of the variable provides a slight performance gain, as we do no +longer need to maintain it inside the output system (which leads to less +CPU instructions and better cache hits). + +**RS\_RET\_SUSPENDED is no longer supported when creating an action +instance** + +This means a plugin must not try to establish any connections or the +like before any of its processing entry points (like beginTransaction or +doAction) is called. This was generally also the case with v7, but was +not enforced in all cases. In v8, creating action fails if +anything but RS\_RET\_OK is returned. + +string generator interface +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Bottom line: string generators need to be changed or will abort. + +The BEGINstrgen() entry point has greatly changed. Instead of two +parameters for the output buffers, they now receive a single ``iparam`` +pointer, which contains all data items needed. Also, the message pointer +is now const to "prevent" (accidental) changes to the message via the +strgen interface. + +Note that strgen modules must now maintain the iparam->lenStr field, +which must hold the length of the generated string on exit. This is +necessary as we cache the string sizes in order to reduced strlen() +calls. Also, the numerical parameters are now unsigned and no longer +size\_t. This permits us to store them directly into optimized heap +structures. + +Specifics for Version 8.3 and 8.4 +--------------------------------- + +Unsupported Command Line Options Removed +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The command line options a, h, m, o, p, g, r, t and c were not +supported since many versions. However, they spit out an error +message that they were unsupported. This error message now no +longer appears, instead the regular usage() display happens. +This should not have any effect to users. + + +Specifics for Version 8.5 and 8.6 +--------------------------------- + +imfile changes +~~~~~~~~~~~~~~ + +Starting with 8.5.0, imfile supports wildcards in file names, but +does do so only in inotify mode. In order to support wildcards, the +handling of statefile needed to be changed. Most importantly, the +*statefile* input parameter has been deprecated. See +:doc:`imfile module documentation <../../configuration/modules/imfile>` +for more details. + +Command Line Options +~~~~~~~~~~~~~~~~~~~~ +There is a small set of configuration command line options available dating back +to the dark ages of syslog technology. Setting command-line options is +distro specific and a hassle for most users. As such, we are phasing out +these options, and will do so rather quickly. + +Some of them (most notably -l, -s) will completely be removed, as +feedback so far indicated they are no longer in use. Others will be +replaced by proper configuration objects. + +**Expect future rsyslog versions to no longer accept those configuration +command line options.** + +Please see this table to see what to use as a replacement for the +current options: + +========== =========================================================================== +**Option** **replacement** +-4 global(net.ipprotocol="ipv4-only") +-6 global(net.ipprotocol="ipv6-only") +-A omfwd input parameter "udp.sendToAll" +-l dropped, currently no replacement +-q global(net.aclAddHostnameOnFail="on") +-Q global(net.aclResolveHostname="off") +-s dropped, currently no replacement +-S omrelp action parameter "localclientip" +-w global(net.permitACLWarning="off") +-x global(net.enableDNS="off") +========== =========================================================================== + |