diff options
Diffstat (limited to 'source/configuration')
156 files changed, 29518 insertions, 0 deletions
diff --git a/source/configuration/.conf_formats.rst.swp b/source/configuration/.conf_formats.rst.swp Binary files differnew file mode 100644 index 0000000..3e9e0cf --- /dev/null +++ b/source/configuration/.conf_formats.rst.swp diff --git a/source/configuration/.templates.rst.swp b/source/configuration/.templates.rst.swp Binary files differnew file mode 100644 index 0000000..3b3805b --- /dev/null +++ b/source/configuration/.templates.rst.swp diff --git a/source/configuration/action/index.rst b/source/configuration/action/index.rst new file mode 100644 index 0000000..e71b7f6 --- /dev/null +++ b/source/configuration/action/index.rst @@ -0,0 +1,193 @@ +Legacy Action-Specific Configuration Statements +=============================================== + +Statements modify the next action(s) that is/are defined **via legacy syntax** +after the respective statement. +Actions defined via the action() object are **not** affected by the +legacy statements listed here. Use the action() object properties +instead. + +Generic action configuration Statements +--------------------------------------- +These statements can be used with all types of actions. + +.. toctree:: + :glob: + + *action* + *rsconf1_repeatedmsgreduction* + +- **$ActionName** <a\_single\_word> - used primarily for documentation, + e.g. when generating a configuration graph. Available since 4.3.1. +- **$ActionExecOnlyOnceEveryInterval** <seconds> - execute action only if + the last execute is at last <seconds> seconds in the past (more info + in `ommail <ommail.html>`_, but may be used with any action). To + disable this setting, use value 0. +- **$ActionExecOnlyEveryNthTime** <number> - If configured, the next + action will only be executed every n-th time. For example, if + configured to 3, the first two messages that go into the action will + be dropped, the 3rd will actually cause the action to execute, the + 4th and 5th will be dropped, the 6th executed under the action, ... + and so on. Note: this setting is automatically re-set when the actual + action is defined. +- **$ActionExecOnlyEveryNthTimeTimeout** <number-of-seconds> - has a + meaning only if $ActionExecOnlyEveryNthTime is also configured for + the same action. If so, the timeout setting specifies after which + period the counting of "previous actions" expires and a new action + count is begun. Specify 0 (the default) to disable timeouts. + *Why is this option needed?* Consider this case: a message comes in + at, eg., 10am. That's count 1. Then, nothing happens for the next 10 + hours. At 8pm, the next one occurs. That's count 2. Another 5 hours + later, the next message occurs, bringing the total count to 3. Thus, + this message now triggers the rule. + The question is if this is desired behavior? Or should the rule only + be triggered if the messages occur within an e.g. 20 minute window? + If the later is the case, you need a + $ActionExecOnlyEveryNthTimeTimeout 1200 + This directive will timeout previous messages seen if they are older + than 20 minutes. In the example above, the count would now be always + 1 and consequently no rule would ever be triggered. +- **$ActionResumeRetryCount** <number> [default 0, -1 means eternal] +- **$ActionWriteAllMarkMessages** [on/**off**]- [available since 5.1.5] + - normally, mark messages are written to actions only if the action + was not recently executed (by default, recently means within the past + 20 minutes). If this setting is switched to "on", mark messages are + always sent to actions, no matter how recently they have been + executed. In this mode, mark messages can be used as a kind of + heartbeat. Note that this option auto-resets to "off", so if you + intend to use it with multiple actions, it must be specified in front + off **all** selector lines that should provide this functionality. + +omfile-specific Configuration Statements +---------------------------------------- +These statements are specific to omfile-based actions. + +.. toctree:: + :glob: + + *omfile* + *dir* + *file* + +- **$CreateDirs** [**on**/off] - create directories on an as-needed + basis +- **$ActionFileDefaultTemplate** [templateName] - sets a new default + template for file actions +- **$ActionFileEnableSync [on/off]** - enables file syncing capability of + omfile +- **$OMFileAsyncWriting** [on/**off**], if turned on, the files will be + written in asynchronous mode via a separate thread. In that case, + double buffers will be used so that one buffer can be filled while + the other buffer is being written. Note that in order to enable + $OMFileFlushInterval, $OMFileAsyncWriting must be set to "on". + Otherwise, the flush interval will be ignored. Also note that when + $OMFileFlushOnTXEnd is "on" but $OMFileAsyncWriting is off, output + will only be written when the buffer is full. This may take several + hours, or even require a rsyslog shutdown. However, a buffer flush + can be forced in that case by sending rsyslogd a HUP signal. +- **$OMFileZipLevel** 0..9 [default 0] - if greater 0, turns on gzip + compression of the output file. The higher the number, the better the + compression, but also the more CPU is required for zipping. +- **$OMFileIOBufferSize** <size\_nbr>, default 4k, size of the buffer + used to writing output data. The larger the buffer, the potentially + better performance is. The default of 4k is quite conservative, it is + useful to go up to 64k, and 128K if you used gzip compression (then, + even higher sizes may make sense) +- **$OMFileFlushOnTXEnd** <[**on**/off]>, default on. Omfile has the + capability to write output using a buffered writer. Disk writes are + only done when the buffer is full. So if an error happens during that + write, data is potentially lost. In cases where this is unacceptable, + set $OMFileFlushOnTXEnd to on. Then, data is written at the end of + each transaction (for pre-v5 this means after **each** log message) + and the usual error recovery thus can handle write errors without + data loss. Note that this option severely reduces the effect of zip + compression and should be switched to off for that use case. Note + that the default -on- is primarily an aid to preserve the traditional + syslogd behaviour. + +omfwd-specific Configuration Statements +--------------------------------------- +These statements are specific to omfwd-based actions. + +- **$ActionForwardDefaultTemplate** [templateName] - sets a new default + template for UDP and plain TCP forwarding action +- **$ActionSendResendLastMsgOnReconnect** <[on/**off**]> specifies if the + last message is to be resend when a connection breaks and has been + reconnected. May increase reliability, but comes at the risk of + message duplication. +- **$ActionSendStreamDriver** <driver basename> just like + $DefaultNetstreamDriver, but for the specific action +- **$ActionSendStreamDriverMode** <mode>, default 0, mode to use with the + stream driver (driver-specific) +- **$ActionSendStreamDriverAuthMode** <mode>, authentication mode to use + with the stream driver. Note that this directive requires TLS + netstream drivers. For all others, it will be ignored. + (driver-specific) +- **$ActionSendStreamDriverPermittedPeer** <ID>, accepted fingerprint + (SHA1) or name of remote peer. Note that this directive requires TLS + netstream drivers. For all others, it will be ignored. + (driver-specific) - directive may go away! +- **$ActionSendTCPRebindInterval** nbr- [available since 4.5.1] - + instructs the TCP send action to close and re-open the connection to + the remote host every nbr of messages sent. Zero, the default, means + that no such processing is done. This directive is useful for use + with load-balancers. Note that there is some performance overhead + associated with it, so it is advisable to not too often "rebind" the + connection (what "too often" actually means depends on your + configuration, a rule of thumb is that it should be not be much more + often than once per second). +- **$ActionSendUDPRebindInterval** nbr- [available since 4.3.2] - + instructs the UDP send action to rebind the send socket every nbr of + messages sent. Zero, the default, means that no rebind is done. This + directive is useful for use with load-balancers. + +omgssapi-specific Configuration Statements +------------------------------------------ +These statements are specific to omgssapi actions. + +.. toctree:: + :glob: + + *gss* + +action-queue specific Configuration Statements +---------------------------------------------- +The following statements specify parameters for the action queue. +To understand queue parameters, read +:doc:`queues in rsyslog <../../concepts/queues>`. + +Action queue parameters usually affect the next action and auto-reset +to defaults thereafter. Most importantly, this means that when a +"real" (non-direct) queue type is defined, this affects the immediately +following action, only. The next and all other actions will be +in "direct" mode (no real queue) if not explicitly specified otherwise. + +- **$ActionQueueCheckpointInterval** <number> +- **$ActionQueueDequeueBatchSize** <number> [default 128] +- **$ActionQueueDequeueSlowdown** <number> [number is timeout in + *micro*\ seconds (1000000us is 1sec!), default 0 (no delay). Simple + rate-limiting!] +- **$ActionQueueDiscardMark** <number> [default 80% of queue size] +- **$ActionQueueDiscardSeverity** <number> [\*numerical\* severity! default + 8 (nothing discarded)] +- **$ActionQueueFileName** <name> +- **$ActionQueueHighWaterMark** <number> [default 90% of queue size] +- **$ActionQueueImmediateShutdown** [on/**off**] +- **$ActionQueueSize** <number> +- **$ActionQueueLowWaterMark** <number> [default 70% of queue size] +- **$ActionQueueMaxFileSize** <size\_nbr>, default 1m +- **$ActionQueueTimeoutActionCompletion** <number> [number is timeout in ms + (1000ms is 1sec!), default 1000, 0 means immediate!] +- **$ActionQueueTimeoutEnqueue** <number> [number is timeout in ms (1000ms + is 1sec!), default 2000, 0 means discard immediately] +- **$ActionQueueTimeoutShutdown** <number> [number is timeout in ms (1000ms + is 1sec!), default 0 (indefinite)] +- **$ActionQueueWorkerTimeoutThreadShutdown** <number> [number is timeout + in ms (1000ms is 1sec!), default 60000 (1 minute)] +- **$ActionQueueType** [FixedArray/LinkedList/**Direct**/Disk] +- **$ActionQueueSaveOnShutdown** [on/**off**] +- **$ActionQueueWorkerThreads** <number>, num worker threads, default 1, + recommended 1 +- $ActionQueueWorkerThreadMinumumMessages <number>, default 100 +- **$ActionGSSForwardDefaultTemplate** [templateName] - sets a new default + template for GSS-API forwarding action diff --git a/source/configuration/action/rsconf1_actionexeconlywhenpreviousissuspended.rst b/source/configuration/action/rsconf1_actionexeconlywhenpreviousissuspended.rst new file mode 100644 index 0000000..11251fe --- /dev/null +++ b/source/configuration/action/rsconf1_actionexeconlywhenpreviousissuspended.rst @@ -0,0 +1,41 @@ +$ActionExecOnlyWhenPreviousIsSuspended +-------------------------------------- + +**Type:** action configuration parameter + +**Default:** off + +**Description:** + +This parameter allows to specify if actions should always be executed +("off," the default) or only if the previous action is suspended ("on"). +This parameter works hand-in-hand with the multiple actions per selector +feature. It can be used, for example, to create rules that automatically +switch destination servers or databases to a (set of) backup(s), if the +primary server fails. Note that this feature depends on proper +implementation of the suspend feature in the output module. All built-in +output modules properly support it (most importantly the database write +and the syslog message forwarder). + +This selector processes all messages it receives (\*.\*). It tries to +forward every message to primary-syslog.example.com (via tcp). If it can +not reach that server, it tries secondary-1-syslog.example.com, if that +fails too, it tries secondary-2-syslog.example.com. If neither of these +servers can be connected, the data is stored in /var/log/localbuffer. +Please note that the secondaries and the local log buffer are only used +if the one before them does not work. So ideally, /var/log/localbuffer +will never receive a message. If one of the servers resumes operation, +it automatically takes over processing again. + +We strongly advise not to use repeated line reduction together with +ActionExecOnlyWhenPreviousIsSuspended. It may lead to "interesting" and +undesired results (but you can try it if you like). + +Example:: + + *.* @@primary-syslog.example.com + $ActionExecOnlyWhenPreviousIsSuspended on + & @@secondary-1-syslog.example.com # & is used to have more than one action for + & @@secondary-2-syslog.example.com # the same selector - the multi-action feature + & /var/log/localbuffer + $ActionExecOnlyWhenPreviousIsSuspended off # to re-set it for the next selector diff --git a/source/configuration/action/rsconf1_actionresumeinterval.rst b/source/configuration/action/rsconf1_actionresumeinterval.rst new file mode 100644 index 0000000..69d2114 --- /dev/null +++ b/source/configuration/action/rsconf1_actionresumeinterval.rst @@ -0,0 +1,26 @@ +$ActionResumeInterval +--------------------- + +**Type:** action configuration parameter + +**Default:** 30 + +**Description:** + +Sets the ActionResumeInterval for all following actions. The interval +provided is always in seconds. Thus, multiply by 60 if you need minutes +and 3,600 if you need hours (not recommended). + +When an action is suspended (e.g. destination can not be connected), the +action is resumed for the configured interval. Thereafter, it is +retried. If multiple retries fail, the interval is automatically +extended. This is to prevent excessive resource use for retries. After +each 10 retries, the interval is extended by itself. To be precise, the +actual interval is (numRetries / 10 + 1) \* $ActionResumeInterval. so +after the 10th try, it by default is 60 and after the 100th try it is +330. + +**Sample:** + +$ActionResumeInterval 30 + diff --git a/source/configuration/action/rsconf1_dirgroup.rst b/source/configuration/action/rsconf1_dirgroup.rst new file mode 100644 index 0000000..c965a43 --- /dev/null +++ b/source/configuration/action/rsconf1_dirgroup.rst @@ -0,0 +1,19 @@ +$DirGroup +--------- + +**Type:** global configuration parameter + +**Default:** + +**Description:** + +Set the group for directories newly created. Please note that this +setting does not affect the group of directories already existing. The +parameter is a group name, for which the groupid is obtained by rsyslogd +on during startup processing. Interim changes to the user mapping are +not detected. + +**Sample:** + +``$DirGroup loggroup`` + diff --git a/source/configuration/action/rsconf1_dirowner.rst b/source/configuration/action/rsconf1_dirowner.rst new file mode 100644 index 0000000..4bb2701 --- /dev/null +++ b/source/configuration/action/rsconf1_dirowner.rst @@ -0,0 +1,19 @@ +$DirOwner +--------- + +**Type:** global configuration parameter + +**Default:** + +**Description:** + +Set the file owner for directories newly created. Please note that this +setting does not affect the owner of directories already existing. The +parameter is a user name, for which the userid is obtained by rsyslogd +during startup processing. Interim changes to the user mapping are not +detected. + +**Sample:** + +``$DirOwner loguser`` + diff --git a/source/configuration/action/rsconf1_dynafilecachesize.rst b/source/configuration/action/rsconf1_dynafilecachesize.rst new file mode 100644 index 0000000..71ddc53 --- /dev/null +++ b/source/configuration/action/rsconf1_dynafilecachesize.rst @@ -0,0 +1,4 @@ +$DynaFileCacheSize +------------------ + +This is an :doc:`omfile <../modules/omfile>` parameter. See there for details. diff --git a/source/configuration/action/rsconf1_filecreatemode.rst b/source/configuration/action/rsconf1_filecreatemode.rst new file mode 100644 index 0000000..63f1ad7 --- /dev/null +++ b/source/configuration/action/rsconf1_filecreatemode.rst @@ -0,0 +1,43 @@ +$FileCreateMode +--------------- + +**Type:** global configuration parameter + +**Default:** 0644 + +**Description:** + +The $FileCreateMode parameter allows to specify the creation mode with +which rsyslogd creates new files. If not specified, the value 0644 is +used (which retains backward-compatibility with earlier releases). The +value given must always be a 4-digit octal number, with the initial +digit being zero. + +Please note that the actual permission depend on rsyslogd's process +umask. If in doubt, use "$umask 0000" right at the beginning of the +configuration file to remove any restrictions. + +$FileCreateMode may be specified multiple times. If so, it specifies the +creation mode for all selector lines that follow until the next +$FileCreateMode parameter. Order of lines is vitally important. + +**Sample:** + +``$FileCreateMode 0600`` + +This sample lets rsyslog create files with read and write access only +for the users it runs under. + +The following sample is deemed to be a complete rsyslog.conf:: + + $umask 0000 # make sure nothing interferes with the following definitions + *.* /var/log/file-with-0644-default + $FileCreateMode 0600 + *.* /var/log/file-with-0600 + $FileCreateMode 0644 + *.* /var/log/file-with-0644 + +As you can see, open modes depend on position in the config file. Note +the first line, which is created with the hardcoded default creation +mode. + diff --git a/source/configuration/action/rsconf1_filegroup.rst b/source/configuration/action/rsconf1_filegroup.rst new file mode 100644 index 0000000..7b33381 --- /dev/null +++ b/source/configuration/action/rsconf1_filegroup.rst @@ -0,0 +1,18 @@ +$FileGroup +---------- + +**Type:** global configuration parameter + +**Default:** + +**Description:** + +Set the group for dynaFiles newly created. Please note that this setting +does not affect the group of files already existing. The parameter is a +group name, for which the groupid is obtained by rsyslogd during startup +processing. Interim changes to the user mapping are not detected. + +**Sample:** + +``$FileGroup loggroup`` + diff --git a/source/configuration/action/rsconf1_fileowner.rst b/source/configuration/action/rsconf1_fileowner.rst new file mode 100644 index 0000000..1ad6a6e --- /dev/null +++ b/source/configuration/action/rsconf1_fileowner.rst @@ -0,0 +1,19 @@ +$FileOwner +---------- + +**Type:** global configuration parameter + +**Default:** + +**Description:** + +Set the file owner for dynaFiles newly created. Please note that this +setting does not affect the owner of files already existing. The +parameter is a user name, for which the userid is obtained by rsyslogd +during startup processing. Interim changes to the user mapping are not +detected. + +**Sample:** + +``$FileOwner loguser`` + diff --git a/source/configuration/action/rsconf1_gssforwardservicename.rst b/source/configuration/action/rsconf1_gssforwardservicename.rst new file mode 100644 index 0000000..c2e2fa1 --- /dev/null +++ b/source/configuration/action/rsconf1_gssforwardservicename.rst @@ -0,0 +1,21 @@ +$GssForwardServiceName +---------------------- + +**Type:** global configuration parameter + +**Default:** host + +**Provided by:** *omgssapi* + +**Description:** + +Specifies the service name used by the client when forwarding GSS-API +wrapped messages. + +The GSS-API service names are constructed by appending '@' and a +hostname following "@@" in each selector. + +**Sample:** + +``$GssForwardServiceName rsyslog`` + diff --git a/source/configuration/action/rsconf1_gssmode.rst b/source/configuration/action/rsconf1_gssmode.rst new file mode 100644 index 0000000..8e3b8b6 --- /dev/null +++ b/source/configuration/action/rsconf1_gssmode.rst @@ -0,0 +1,20 @@ +$GssMode +-------- + +**Type:** global configuration parameter + +**Default:** encryption + +**Provided by:** *omgssapi* + +**Description:** + +Specifies GSS-API mode to use, which can be "**integrity**\ " - clients +are authenticated and messages are checked for integrity, +"**encryption**\ " - same as "integrity", but messages are also +encrypted if both sides support it. + +**Sample:** + +``$GssMode Encryption`` + diff --git a/source/configuration/action/rsconf1_omfileforcechown.rst b/source/configuration/action/rsconf1_omfileforcechown.rst new file mode 100644 index 0000000..5eba119 --- /dev/null +++ b/source/configuration/action/rsconf1_omfileforcechown.rst @@ -0,0 +1,67 @@ +$omfileForceChown +----------------- + +**Type:** action configuration parameter + +**Parameter Values:** boolean (on/off, yes/no) + +**Available:** 4.7.0+, 5.3.0-5.8.x, **NOT** available in 5.9.x or higher + +**Note: this parameter has been removed and is no longer available. The +documentation is currently being retained for historical reaons.** +Expect it to go away at some later stage as well. + +**Default:** off + +**Description:** + +Forces rsyslogd to change the ownership for output files that already +exist. Please note that this tries to fix a potential problem that +exists outside the scope of rsyslog. Actually, it tries to fix invalid +ownership/permission settings set by the original file creator. + +Rsyslog changes the ownership during initial execution with root +privileges. When a privilege drop is configured, privileges are dropped +after the file owner ship is changed. Not that this currently is a +limitation in rsyslog's privilege drop code, which is on the TODO list +to be removed. See Caveats section below for the important implications. + +**Caveats:** + +This parameter tries to fix a problem that actually is outside the scope +of rsyslog. As such, there are a couple of restrictions and situations +in which it will not work. **Users are strongly encouraged to fix their +system instead of turning this parameter on** - it should only be used +as a last resort. + +At least in the following scenario, this parameter will fail expectedly: + +It does not address the situation that someone changes the ownership +\*after\* rsyslogd has started. Let's, for example, consider a log +rotation script. + +- rsyslog is started +- ownership is changed +- privileges dropped +- log rotation (lr) script starts +- lr removes files +- lr creates new files with root:adm (or whatever else) +- lr HUPs rsyslogd +- rsyslogd closes files +- rsyslogd tries to open files +- rsyslogd tries to change ownership --> fail as we are non-root now +- file open fails + +Please note that once the privilege drop code is refactored, this +parameter will no longer work, because then privileges will be dropped +before any action is performed, and thus we will no longer be able to +chown files that do not belong to the user rsyslogd is configured to run +under. + +So **expect the parameter to go away**. It will not be removed in +version 4, but may disappear at any time for any version greater than 4. + +**Sample:** + +``$FileOwner loguser $omfileForceChown on`` + diff --git a/source/configuration/action/rsconf1_repeatedmsgreduction.rst b/source/configuration/action/rsconf1_repeatedmsgreduction.rst new file mode 100644 index 0000000..33ddeb9 --- /dev/null +++ b/source/configuration/action/rsconf1_repeatedmsgreduction.rst @@ -0,0 +1,70 @@ +$RepeatedMsgReduction +--------------------- + +**Type:** global configuration parameter + +**Default:** off + +Description +^^^^^^^^^^^ + +This parameter models old sysklogd legacy. **Note that many people, +including the rsyslog authors, consider this to be a misfeature.** See +*Discussion* below to learn why. + +This parameter specifies whether or not repeated messages should be +reduced (this is the "Last line repeated n times" feature). If set to +*on*, repeated messages are reduced. If kept at *off*, every message is +logged. In very early versions of rsyslog, this was controlled by the +*-e* command line option. + +What is a repeated message +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +For a message to be classified as repeated, the following properties +must be **identical**: + +* msg +* hostname +* procid +* appname + +Note that rate-limiters are usually applied to specific input sources +or processes. So first and foremost the input source must be the same +to classify a messages as a duplicated. + +You may want to check out +`testing rsyslog ratelimiting <http://www.rsyslog.com/first-try-to-test-rate-limiting/>`_ +for some extra information on the per-process ratelimiting. + +Discussion +^^^^^^^^^^ + +* Very old versions of rsyslog did not have the ability to include the + repeated message itself within the repeat message. + +* Versions before 7.3.2 applied repeat message reduction to the output + side. This had some implications: + + - they did not account for the actual message origin, so two processes + emitting an equally-looking message triggered the repeated message + reduction code + + - repeat message processing could be set on a per-action basis, which + has switched to per-input basis for 7.3.2 and above + +* While turning this feature on can save some space in logs, most log analysis + tools need to see the repeated messages, they can't handle the + "last message repeated" format. +* This is a feature that worked decades ago when logs were small and reviewed + by a human, it fails badly on high volume logs processed by tools. + +Sample +^^^^^^ + +This turns on repeated message reduction (**not** recommended): + +:: + + $RepeatedMsgReduction on # do not log repeated messages + diff --git a/source/configuration/actions.rst b/source/configuration/actions.rst new file mode 100644 index 0000000..31dbce7 --- /dev/null +++ b/source/configuration/actions.rst @@ -0,0 +1,621 @@ +Actions +======= + +.. index:: ! action +.. _cfgobj_input: + +The Action object describe what is to be done with a message. They are +implemented via :doc:`output modules <modules/idx_output>`. + +The action object has different parameters: + +- those that apply to all actions and are action specific. These are + documented below. + +- parameters for the action queue. While they also apply to all + parameters, they are queue-specific, not action-specific (they are + the same that are used in rulesets, for example). The are documented + separately under :doc:`queue parameters <../rainerscript/queue_parameters>`. + +- action-specific parameters. These are specific to a certain type of + actions. They are documented by the :doc:`output modules<modules/idx_output>` + in question. + +General Action Parameters +------------------------- + +Note: parameter names are case-insensitive. + +- **name** word + + This names the action. The name is used for statistics gathering + and documentation. If no name is given, one is dynamically generated + based on the occurrence of this action inside the rsyslog configuration. + Actions are sequentially numbered from 1 to n. + +- **type** string + + Mandatory parameter for every action. The name of the module that + should be used. + +- **action.writeAllMarkMessages** *on*/off + + This setting tells if mark messages are always written ("on", the + default) or only if the action was not recently executed ("off"). By + default, recently means within the past 20 minutes. If this setting + is "on", mark messages are always sent to actions, no matter how + recently they have been executed. In this mode, mark messages can be + used as a kind of heartbeat. This mode also enables faster processing + inside the rule engine. So it should be set to "off" only when there + is a good reason to do so. + +- **action.execOnlyEveryNthTime** integer + + If configured, the next action will only be executed every n-th time. + For example, if configured to 3, the first two messages that go into + the action will be dropped, the 3rd will actually cause the action to + execute, the 4th and 5th will be dropped, the 6th executed under the + action, ... and so on. + +- **action.execOnlyEveryNthTimeout** integer + + Has a meaning only if Action.ExecOnlyEveryNthTime is also configured + for the same action. If so, the timeout setting specifies after which + period the counting of "previous actions" expires and a new action + count is begun. Specify 0 (the default) to disable timeouts. Why is + this option needed? Consider this case: a message comes in at, eg., + 10am. That's count 1. Then, nothing happens for the next 10 hours. At + 8pm, the next one occurs. That's count 2. Another 5 hours later, the + next message occurs, bringing the total count to 3. Thus, this + message now triggers the rule. The question is if this is desired + behavior? Or should the rule only be triggered if the messages occur + within an e.g. 20 minute window? If the later is the case, you need a + Action.ExecOnlyEveryNthTimeTimeout="1200" + This directive will timeout previous messages seen if they are older + than 20 minutes. In the example above, the count would now be always + 1 and consequently no rule would ever be triggered. + +- **action.errorfile** string + + .. versionadded:: 8.32.0 + + When an action is executed, some messages may permanently fail. + Depending on configuration, this could for example be caused by an + offline target or exceptionally non-numerical data inside a + numerical database field. If action.errorfile is specified, those + messages are written to the specified file. If it is not specified + (the default), messages are silently discarded. + + The error file format is JSON. It contains the failed messages as + provided to the action in question, the action name as well as + the rsyslog status code roughly explaining why it failed. + +- **action.errorfile.maxsize** integer + + In some cases, error file needs to be limited in size. + This option allows specifying a maximum size, in bytes, for the error file. + When error file reaches that size, no more errors are written to it. + +- **action.execOnlyOnceEveryInterval** integer + + Execute action only if the last execute is at last seconds in the + past (more info in ommail, but may be used with any action) + +- **action.execOnlyWhenPreviousIsSuspended** on/off + + This directive allows to specify if actions should always be executed + ("off," the default) or only if the previous action is suspended + ("on"). This directive works hand-in-hand with the multiple actions + per selector feature. It can be used, for example, to create rules + that automatically switch destination servers or databases to a (set + of) backup(s), if the primary server fails. Note that this feature + depends on proper implementation of the suspend feature in the output + module. All built-in output modules properly support it (most + importantly the database write and the syslog message forwarder). + Note, however, that a failed action may not immediately be detected. + For more information, see the `rsyslog + execOnlyWhenPreviousIsSuspended + preciseness <https://www.rsyslog.com/action-execonlywhenpreviousissuspended-preciseness/>`_ + FAQ article. + +- **action.repeatedmsgcontainsoriginalmsg** on/off + + "last message repeated n times" messages, if generated, have a + different format that contains the message that is being repeated. + Note that only the first "n" characters are included, with n to be at + least 80 characters, most probably more (this may change from version + to version, thus no specific limit is given). The bottom line is that + n is large enough to get a good idea which message was repeated but + it is not necessarily large enough for the whole message. (Introduced + with 4.1.5). + +- **action.resumeRetryCount** integer + + [default 0, -1 means eternal] + + Sets how often an action is retried before it is considered to have + failed. Failed actions discard messages. + +- **action.resumeInterval** integer + + Sets the action's resume interval. The interval provided + is always in seconds. Thus, multiply by 60 if you need minutes and + 3,600 if you need hours (not recommended). When an action is + suspended (e.g. destination can not be connected), the action is + resumed for the configured interval. Thereafter, it is retried. If + multiple retries fail, the interval is automatically extended. This + is to prevent excessive resource use for retries. After each 10 + retries, the interval is extended by itself. To be precise, the + actual interval is `(numRetries / 10 + 1) * action.resumeInterval`. + Using the default value of 30, this means that on the 10th try the + suspension interval will be 60 (seconds) and after the 100th try + it will be 330 (seconds). + +- **action.resumeIntervalMax** integer + + Default: 1800 (30 minutes) + + This sets an upper limit on the growth of action.resumeInterval. + No wait will be larger than the value configured here. Going higher + than the default is only recommended if you know that a system may + be offline for an extended period of time **and** if it is acceptable + that it may take quite long to detect it came online again. + +- **action.reportSuspension** on/off + + Configures rsyslog to report suspension and reactivation + of the action. This is useful to note which actions have + problems (e.g. connecting to a remote system) and when. + The default for this setting is the equally-named global + parameter. + +- **action.reportSuspensionContinuation** on/off + + Configures rsyslog to report continuation of action suspension. + This emits new messages whenever an action is to be retried, but + continues to fail. If set to "on", *action.reportSuspension* is + also automatically set to "on". + The default for this setting is the equally-named global + parameter. + +- **action.copyMsg** on/*off* + + Configures action to *copy* the message if *on*. Defaults to + *off* (which is how actions have worked traditionally), which + causes queue to refer to the original message object, with + reference-counting. (Introduced with 8.10.0). + +Useful Links +------------ + +- Rainer's blog posting on the performance of `main and action queue + worker + threads <https://rainer.gerhards.net/2013/06/rsyslog-performance-main-and-action.html>`_ + +Legacy Format +------------- + +.. _legacy-action-order: + +**Be warned that legacy action format is hard to get right. It is +recommended to use RainerScript-Style action format whenever possible!** +A key problem with legacy format is that a single action is defined via +multiple configurations lines, which may be spread all across +rsyslog.conf. Even the definition of multiple actions may be intermixed +(often not intentional!). If legacy actions format needs to be used +(e.g. some modules may not yet implement the RainerScript format), it is +strongly recommended to place all configuration statements pertaining to +a single action closely together. + +Please also note that legacy action parameters **do not** affect +RainerScript action objects. So if you define for example: + +:: + + $actionResumeRetryCount 10 + action(type="omfwd" target="server1.example.net") + @@server2.example.net + +server1's "action.resumeRetryCount" parameter is **not** set, instead +server2's is! + +A goal of the new RainerScript action format was to avoid confusion +which parameters are actually used. As such, it would be +counter-productive to honor legacy action parameters inside a +RainerScript definition. As result, both types of action definitions are +strictly (and nicely) separated from each other. The bottom line is that +if RainerScript actions are used, one does not need to care about which +legacy action parameters may (still...) be in effect. + +Note that not all modules necessarily support legacy action format. +Especially newer modules are recommended to NOT support it. + +Legacy Description +~~~~~~~~~~~~~~~~~~ + +Templates can be used with many actions. If used, the specified template +is used to generate the message content (instead of the default +template). To specify a template, write a semicolon after the action +value immediately followed by the template name. +Beware: templates MUST be defined BEFORE they are used. It is OK to +define some templates, then use them in selector lines, define more +templates and use use them in the following selector lines. But it is +NOT permitted to use a template in a selector line that is above its +definition. If you do this, the action will be ignored. + +**You can have multiple actions for a single selector** (or more +precisely a single filter of such a selector line). Each action must be +on its own line and the line must start with an ampersand ('&') +character and have no filters. An example would be + +:: + + *.=crit :omusrmsg:rger + & root + & /var/log/critmsgs + +These three lines send critical messages to the user rger and root and +also store them in /var/log/critmsgs. **Using multiple actions per +selector is** convenient and also **offers a performance benefit**. As +the filter needs to be evaluated only once, there is less computation +required to process the directive compared to the otherwise-equal config +directives below: + +:: + + *.=crit :omusrmsg:rger + *.=crit root + *.=crit /var/log/critmsgs + +Regular File +~~~~~~~~~~~~ + +Typically messages are logged to real files. The file usually is +specified by full pathname, beginning with a slash "/". Starting with +version 4.6.2 and 5.4.1 (previous v5 version do NOT support this) +relative file names can also be specified. To do so, these must begin +with a dot. For example, use "./file-in-current-dir.log" to specify a +file in the current directory. Please note that rsyslogd usually changes +its working directory to the root, so relative file names must be tested +with care (they were introduced primarily as a debugging vehicle, but +may have useful other applications as well). +You may prefix each entry with the minus "-'' sign to omit syncing the +file after every logging. Note that you might lose information if the +system crashes right behind a write attempt. Nevertheless this might +give you back some performance, especially if you run programs that use +logging in a very verbose manner. + +If your system is connected to a reliable UPS and you receive lots of +log data (e.g. firewall logs), it might be a very good idea to turn of +syncing by specifying the "-" in front of the file name. + +**The filename can be either static**\ (always the same) or **dynamic** +(different based on message received). The later is useful if you would +automatically split messages into different files based on some message +criteria. For example, dynamic file name selectors allow you to split +messages into different files based on the host that sent them. With +dynamic file names, everything is automatic and you do not need any +filters. + +It works via the template system. First, you define a template for the +file name. An example can be seen above in the description of template. +We will use the "DynFile" template defined there. Dynamic filenames are +indicated by specifying a questions mark "?" instead of a slash, +followed by the template name. Thus, the selector line for our dynamic +file name would look as follows: + + ``*.* ?DynFile`` + +That's all you need to do. Rsyslog will now automatically generate file +names for you and store the right messages into the right files. Please +note that the minus sign also works with dynamic file name selectors. +Thus, to avoid syncing, you may use + + ``*.* -?DynFile`` + +And of course you can use templates to specify the output format: + + ``*.* ?DynFile;MyTemplate`` + +**A word of caution:** rsyslog creates files as needed. So if a new host +is using your syslog server, rsyslog will automatically create a new +file for it. + +**Creating directories is also supported**. For example you can use the +hostname as directory and the program name as file name: + + ``$template DynFile,"/var/log/%HOSTNAME%/%programname%.log"`` + +Named Pipes +~~~~~~~~~~~ + +This version of rsyslogd(8) has support for logging output to named +pipes (fifos). A fifo or named pipe can be used as a destination for log +messages by prepending a pipe symbol ("\|'') to the name of the file. +This is handy for debugging. Note that the fifo must be created with the +mkfifo(1) command before rsyslogd(8) is started. + +Terminal and Console +~~~~~~~~~~~~~~~~~~~~ + +If the file you specified is a tty, special tty-handling is done, same +with /dev/console. + +Remote Machine +~~~~~~~~~~~~~~ + +Rsyslogd provides full remote logging, i.e. is able to send messages to +a remote host running rsyslogd(8) and to receive messages from remote +hosts. Using this feature you're able to control all syslog messages on +one host, if all other machines will log remotely to that. This tears +down administration needs. + +To forward messages to another host, prepend the hostname with the at +sign ("@"). A single at sign means that messages will be forwarded via +UDP protocol (the standard for syslog). If you prepend two at signs +("@@"), the messages will be transmitted via TCP. Please note that plain +TCP based syslog is not officially standardized, but most major syslogds +support it (e.g. syslog-ng or `WinSyslog <https://www.winsyslog.com/>`_). +The forwarding action indicator (at-sign) can be followed by one or more +options. If they are given, they must be immediately (without a space) +following the final at sign and be enclosed in parenthesis. The +individual options must be separated by commas. The following options +are right now defined: + +**z<number>** + +Enable zlib-compression for the message. The <number> is the compression +level. It can be 1 (lowest gain, lowest CPU overhead) to 9 (maximum +compression, highest CPU overhead). The level can also be 0, which means +"no compression". If given, the "z" option is ignored. So this does not +make an awful lot of sense. There is hardly a difference between level 1 +and 9 for typical syslog messages. You can expect a compression gain +between 0% and 30% for typical messages. Very chatty messages may +compress up to 50%, but this is seldom seen with typically traffic. +Please note that rsyslogd checks the compression gain. Messages with 60 +bytes or less will never be compressed. This is because compression gain +is pretty unlikely and we prefer to save CPU cycles. Messages over that +size are always compressed. However, it is checked if there is a gain in +compression and only if there is, the compressed message is transmitted. +Otherwise, the uncompressed messages is transmitted. This saves the +receiver CPU cycles for decompression. It also prevents small message to +actually become larger in compressed form. + +**Please note that when a TCP transport is used, compression will also +turn on syslog-transport-tls framing. See the "o" option for important +information on the implications.** + +Compressed messages are automatically detected and decompressed by the +receiver. There is nothing that needs to be configured on the receiver +side. + +**o** + +**This option is experimental. Use at your own risk and only if you know +why you need it! If in doubt, do NOT turn it on.** + +This option is only valid for plain TCP based transports. It selects a +different framing based on IETF internet draft syslog-transport-tls-06. +This framing offers some benefits over traditional LF-based framing. +However, the standardization effort is not yet complete. There may be +changes in upcoming versions of this standard. Rsyslog will be kept in +line with the standard. There is some chance that upcoming changes will +be incompatible to the current specification. In this case, all systems +using -transport-tls framing must be upgraded. There will be no effort +made to retain compatibility between different versions of rsyslog. The +primary reason for that is that it seems technically impossible to +provide compatibility between some of those changes. So you should take +this note very serious. It is not something we do not \*like\* to do +(and may change our mind if enough people beg...), it is something we +most probably \*can not\* do for technical reasons (aka: you can beg as +much as you like, it won't change anything...). + +The most important implication is that compressed syslog messages via +TCP must be considered with care. Unfortunately, it is technically +impossible to transfer compressed records over traditional syslog plain +tcp transports, so you are left with two evil choices... + + The hostname may be followed by a colon and the destination port. + +The following is an example selector line with forwarding: + +\*.\* @@(o,z9)192.168.0.1:1470 + +In this example, messages are forwarded via plain TCP with experimental +framing and maximum compression to the host 192.168.0.1 at port 1470. + +\*.\* @192.168.0.1 + +In the example above, messages are forwarded via UDP to the machine +192.168.0.1, the destination port defaults to 514. Messages will not be +compressed. + +Note that IPv6 addresses contain colons. So if an IPv6 address is +specified in the hostname part, rsyslogd could not detect where the IP +address ends and where the port starts. There is a syntax extension to +support this: put square brackets around the address (e.g. "[2001::1]"). +Square brackets also work with real host names and IPv4 addresses, too. + +A valid sample to send messages to the IPv6 host 2001::1 at port 515 is +as follows: + +\*.\* @[2001::1]:515 + +This works with TCP, too. + +**Note to sysklogd users:** sysklogd does **not** support RFC 3164 +format, which is the default forwarding template in rsyslog. As such, +you will experience duplicate hostnames if rsyslog is the sender and +sysklogd is the receiver. The fix is simple: you need to use a different +template. Use that one: + +$template sysklogd,"<%PRI%>%TIMESTAMP% %syslogtag%%msg%\\"" + \*.\* @192.168.0.1;sysklogd + +List of Users +~~~~~~~~~~~~~ + +Usually critical messages are also directed to "root'' on that machine. +You can specify a list of users that shall get the message by simply +writing ":omusrmsg: followed by the login name. For example, the send +messages to root, use ":omusrmsg:root". You may specify more than one +user by separating them with commas (",''). Do not repeat the +":omusrmsg:" prefix in this case. For example, to send data to users +root and rger, use ":omusrmsg:root,rger" (do not use +":omusrmsg:root,:omusrmsg:rger", this is invalid). If they're logged in +they get the message. + +Everyone logged on +~~~~~~~~~~~~~~~~~~ + +Emergency messages often go to all users currently online to notify them +that something strange is happening with the system. To specify this +wall(1)-feature use an asterisk as the user message +destination(":omusrmsg:\*''). + +Call Plugin +~~~~~~~~~~~ + +This is a generic way to call an output plugin. The plugin must support +this functionality. Actual parameters depend on the module, so see the +module's doc on what to supply. The general syntax is as follows: + +:modname:params;template + +Currently, the ommysql database output module supports this syntax (in +addition to the ">" syntax it traditionally supported). For ommysql, the +module name is "ommysql" and the params are the traditional ones. The +;template part is not module specific, it is generic rsyslog +functionality available to all modules. + +As an example, the ommysql module may be called as follows: + +:ommysql:dbhost,dbname,dbuser,dbpassword;dbtemplate + +For details, please see the "Database Table" section of this +documentation. + +Note: as of this writing, the ":modname:" part is hardcoded into the +module. So the name to use is not necessarily the name the module's +plugin file is called. + +Database Table +~~~~~~~~~~~~~~ + +This allows logging of the message to a database table. Currently, only +MySQL databases are supported. However, other database drivers will most +probably be developed as plugins. By default, a +`MonitorWare <https://www.mwagent.com/>`_-compatible schema is +required for this to work. You can create that schema with the +createDB.SQL file that came with the rsyslog package. You can also +use any other schema of your liking - you just need to define a proper +template and assign this template to the action. +The database writer is called by specifying a greater-then sign (">") +in front of the database connect information. Immediately after that +sign the database host name must be given, a comma, the database name, +another comma, the database user, a comma and then the user's password. +If a specific template is to be used, a semicolon followed by the +template name can follow the connect information. This is as follows: +>dbhost,dbname,dbuser,dbpassword;dbtemplate + +**Important: to use the database functionality, the MySQL output module +must be loaded in the config file** BEFORE the first database table +action is used. This is done by placing the + +:: + + $ModLoad ommysql + +directive some place above the first use of the database write (we +recommend doing at the beginning of the config file). + +Discard / Stop +~~~~~~~~~~~~~~ + +If the discard action is carried out, the received message is +immediately discarded. No further processing of it occurs. Discard has +primarily been added to filter out messages before carrying on any +further processing. For obvious reasons, the results of "discard" are +depending on where in the configuration file it is being used. Please +note that once a message has been discarded there is no way to retrieve +it in later configuration file lines. + +Discard can be highly effective if you want to filter out some annoying +messages that otherwise would fill your log files. To do that, place the +discard actions early in your log files. This often plays well with +property-based filters, giving you great freedom in specifying what you +do not want. + +Discard is just the word "stop" with no further parameters: + +stop + +For example, + +\*.\* stop + +discards everything (ok, you can achieve the same by not running rsyslogd +at all...). + +Note that in legacy configuration the tilde character "~" can also be +used instead of the word "stop". + +Output Channel +~~~~~~~~~~~~~~ + +Binds an output channel definition (see there for details) to this +action. Output channel actions must start with a $-sign, e.g. if you +would like to bind your output channel definition "mychannel" to the +action, use "$mychannel". Output channels support template definitions +like all all other actions. + +Shell Execute +~~~~~~~~~~~~~ + +**NOTE: This action is only supported for backwards compatibility. +For new configs, use** :doc:`omprog <modules/omprog>` **instead. +It provides a more solid +and secure solution with higher performance.** + +This executes a program in a subshell. The program is passed the +template-generated message as the only command line parameter. Rsyslog +waits until the program terminates and only then continues to run. + +^program-to-execute;template + +The program-to-execute can be any valid executable. It receives the +template string as a single parameter (argv[1]). + +**WARNING:** The Shell Execute action was added to serve an urgent need. +While it is considered reasonable save when used with some thinking, its +implications must be considered. The current implementation uses a +system() call to execute the command. This is not the best way to do it +(and will hopefully changed in further releases). Also, proper escaping +of special characters is done to prevent command injection. However, +attackers always find smart ways to circumvent escaping, so we can not +say if the escaping applied will really safe you from all hassles. +Lastly, rsyslog will wait until the shell command terminates. Thus, a +program error in it (e.g. an infinite loop) can actually disable +rsyslog. Even without that, during the programs run-time no messages are +processed by rsyslog. As the IP stacks buffers are quickly overflowed, +this bears an increased risk of message loss. You must be aware of these +implications. Even though they are severe, there are several cases where +the "shell execute" action is very useful. This is the reason why we +have included it in its current form. To mitigate its risks, always a) +test your program thoroughly, b) make sure its runtime is as short as +possible (if it requires a longer run-time, you might want to spawn your +own sub-shell asynchronously), c) apply proper firewalling so that only +known senders can send syslog messages to rsyslog. Point c) is +especially important: if rsyslog is accepting message from any hosts, +chances are much higher that an attacker might try to exploit the "shell +execute" action. + +Template Name +~~~~~~~~~~~~~ + +Every ACTION can be followed by a template name. If so, that template is +used for message formatting. If no name is given, a hard-coded default +template is used for the action. There can only be one template name for +each given action. The default template is specific to each action. For +a description of what a template is and what you can do with it, see the +:doc:`template<templates>` documentation. diff --git a/source/configuration/basic_structure.rst b/source/configuration/basic_structure.rst new file mode 100644 index 0000000..133da44 --- /dev/null +++ b/source/configuration/basic_structure.rst @@ -0,0 +1,233 @@ +Basic Structure +=============== + +This section describes how rsyslog configuration basically works. Think +of rsyslog as a big logging and event processing toolset. It can be considered +a framework with some basic processing that is fixed in the way data flows, +but is highly customizable in the details of this message flow. During +configuration, this customization is done by defining and customizing +the rsyslog objects. + +Quick overview of message flow and objects +------------------------------------------ +Messages enter rsyslog with the help of input modules. Then, they are +passed to a ruleset, where rules are conditionally applied. When a rule +matches, the message is transferred to an action, which then does +something to the message, e.g. writes it to a file, database or +forwards it to a remote host. + +Processing Principles +--------------------- + +- inputs submit received messages to rulesets + + * if the ruleset is not specifically bound, the default ruleset is used + +- by default, there is one ruleset (RSYSLOG_DefaultRuleset) + +- additional rulesets can be user-defined + +- each ruleset contains zero or more rules + + * while it is permitted to have zero rules inside a ruleset, + this obviously makes no sense + +- a rule consists of a filter and an action list + +- filters provide yes/no decisions and thus control-of-flow capability + +- if a filter "matches" (filter says "yes"), the corresponding + action list is executed. If it does not match, nothing special + happens + +- rules are evaluated in sequence from the first to the last rule + **inside** the given ruleset. No rules from unrelated rulesets are + ever executed. + +- all rules are **always** fully evaluated, no matter if a filter matches + or not (so we do **not** stop at the first match). If message processing + shall stop, the "discard" action (represented by the tilde character or the + stop command) must explicitly be executed. If discard is executed, + message processing immediately stops, without evaluating any further rules. + +- an action list contains one or many actions + +- inside an action list no further filters are possible + +- to have more than one action inside a list, the ampersand character + must be placed in the position of the filter, and this must immediately + follow the previous action + +- actions consist of the action call itself (e.g. ":omusrmsg:") as + well as all action-defining configuration statements ($Action... directives) + +- if legacy format is used (see below), $Action... directives **must** be + specified in front of the action they are intended to configure + +- some config directives automatically refer to their previous values + after being applied, others not. See the respective doc for details. Be + warned that this is currently not always properly documented. + +- in general, rsyslog v5 is heavily outdated and its native config language + is a pain. The rsyslog project strongly recommends using at least version 7, + where these problems are solved and configuration is much easier. + +- legacy configuration statements (those starting with $) do **not** affect + RainerScript objects (e.g. actions). + + +Configuration File +------------------ +Upon startup, rsyslog reads its configuration from the ``rsyslog.conf`` +file by default. This file may contain references to include other +config files. + +A different "root" configuration file can be specified via the ``-f <file>`` +rsyslogd command line option. This is usually done within some init +script or similar facility. + +Statement Types +--------------- +Rsyslog supports three different types of configuration statements +concurrently: + +- **sysklogd** - this is the plain old format, taught everywhere and + still pretty useful for simple use cases. Note that some + constructs are no longer supported because they are incompatible with + newer features. These are mentioned in the compatibility docs. +- **legacy rsyslog** - these are statements that begin with a dollar + sign. They set some (case-insensitive) configuration parameters and + modify e.g. the way actions operate. This is the only format supported + in pre-v6 versions of rsyslog. It is still fully supported in v6 and + above. Note that some plugins and features may still only be available + through legacy format (because plugins need to be explicitly upgraded + to use the new style format, and this hasn't happened to all plugins). +- **RainerScript** - the new style format. This is the best and most + precise format to be used for more complex cases. As with the legacy + format, RainerScript parameters are also case-insensitive. + The rest of this page assumes RainerScript based rsyslog.conf. + + +The rsyslog.conf files consists of statements. For old style (sysklogd & +legacy rsyslog), lines do matter. For new style (RainerScript) line +spacing is irrelevant. Most importantly, this means with new style +actions and all other objects can split across lines as users want to. + +Recommended use of Statement Types +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In general it is recommended to use RainerScript type statements, as +these provide clean and easy to read control-of-flow as well as +no doubt about which parameters are active. They also have no +side-effects with include files, which can be a major obstacle with +legacy rsyslog statements. + +For very simple things sysklogd statement types are still suggested, +especially if the full config consists of such simple things. The +classical sample is writing to files (or forwarding) via priority. +In sysklogd, this looks like: + +:: + + mail.info /var/log/mail.log + mail.err @server.example.net + +This is hard to beat in simplicity, still being taught in courses +and a lot of people know this syntax. It is perfectly fine to use +these constructs even in newly written config files. + +**As a rule of thumb, RainerScript config statements should be used +when** + +- configuration parameters are required (e.g. the ``Action...`` + type of legacy statements) +- more elaborate control-of-flow is required (e.g. when multiple + actions must be nested under the same condition) + +It is usually **not** recommended to use rsyslog legacy config format +(those directives starting with a dollar sign). However, a few +settings and modules have not yet been converted to RainerScript. In +those cases, the legacy syntax must be used. + +Comments +-------- + +There are two types of comments: + +- **#-Comments** - start with a hash sign (#) and run to the end of the + line +- **C-style Comments** - start with /\* and end with \*/, just like in + the C programming language. They can be used to comment out multiple + lines at once. Comment nesting is not supported, but #-Comments can be + contained inside a C-style comment. + +Processing Order +---------------- + +Directives are processed from the top of rsyslog.conf to the bottom. +Order matters. For example, if you stop processing of a message, +obviously all statements after the stop statement are never evaluated. + +Flow Control Statements +~~~~~~~~~~~~~~~~~~~~~~~ + +Flow control is provided by: + +- :doc:`Control Structures <../rainerscript/control_structures>` + +- :doc:`Filter Conditions <filters>` + + +Data Manipulation Statements +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Data manipulation is achieved by **set**, **unset** and **reset** statements +which are :doc:`documented here in detail <../rainerscript/variable_property_types>`. + +Inputs +------ + +Every input requires an input module to be loaded and a listener defined +for it. Full details can be found inside the :doc:`rsyslog +modules <modules/index>` documentation. Once loaded, inputs +are defined via the **input()** object. + +Outputs +------- + +Outputs are also called "actions". A small set of actions is pre-loaded +(like the output file writer, which is used in almost every +rsyslog.conf), others must be loaded just like inputs. + +An action is invoked via the **action(type="type" ...)** object. Type is +mandatory and must contain the name of the plugin to be called (e.g. +"omfile" or "ommongodb"). Other parameters may be present. Their type and +use depends on the output plugin in question. + +Rulesets and Rules +------------------ + +Rulesets and rules form the basis of rsyslog processing. In short, a +rule is a way how rsyslog shall process a specific message. Usually, +there is a type of filter (if-statement) in front of the rule. Complex +nesting of rules is possible, much like in a programming language. + +Rulesets are containers for rules. A single ruleset can contain many +rules. In the programming language analogy, one may think of a ruleset +like being a program. A ruleset can be "bound" (assigned) to a specific +input. In the analogy, this means that when a message comes in via that +input, the "program" (ruleset) bound to it will be executed (but not any +other!). + +There is detailed documentation available for +:doc:`rsyslog rulesets <../concepts/multi_ruleset>`. + +For quick reference, rulesets are defined as follows: + +:: + + ruleset(name="rulesetname") { + action(type="omfile" file="/path/to/file") + action(type="..." ...) + /* and so on... */ + } diff --git a/source/configuration/conf_formats.rst b/source/configuration/conf_formats.rst new file mode 100644 index 0000000..c85af8b --- /dev/null +++ b/source/configuration/conf_formats.rst @@ -0,0 +1,94 @@ +Configuration Formats +===================== + +Rsyslog has evolved over several decades. For this reason it supports three +different configuration formats ("languages"): + +- |FmtBasicName| - previously known as the :doc:`sysklogd <sysklogd_format>` + format, this is the format best used to express basic things, such as where + the statement fits on a single line. It stems back to the original + syslog.conf format, in use now for several decades. + + The most common use case is matching on facility/severity and writing + matching messages to a log file. + +- |FmtAdvancedName| - previously known as the ``RainerScript`` format, this + format was first available in rsyslog v6 and is the current, best and most + precise format for non-trivial use cases where more than one line is needed. + + Prior to v7, there was a performance impact when using this format that + encouraged use of the |FmtBasicName| format for best results. Current + versions of rsyslog do not suffer from this (historical) performance impact. + + This new style format is specifically targeted towards more advanced use + cases like forwarding to remote hosts that might be partially offline. + +- |FmtObsoleteName| - previously known simply as the ``legacy`` format, this + format is exactly what its name implies: it is obsolete and should not + be used when writing new configurations. It was created in the early + days (up to rsyslog version 5) where we expected that rsyslog would extend + sysklogd just mildly. Consequently, it was primarily aimed at small + additions to the original sysklogd format. Practice has shown that it + was notoriously hard to use for more advanced use cases, and thus we + replaced it with the |FmtAdvancedName| format. + + In essence, everything that needs to be written on a single line that + starts with a dollar sign is legacy format. Users of this format are + encouraged to migrate to the |FmtBasicName| or |FmtAdvancedName| formats. + +Which Format should I Use? +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +While rsyslog supports all three formats concurrently, you are **strongly** +encouraged to avoid using the |FmtObsoleteName| format. Instead, you should +use the |FmtBasicName| format for basic configurations and the |FmtAdvancedName| +format for anything else. + +While it is an older format, the |FmtBasicName| format is still suggested for +configurations that mostly consist of simple statements. The classic +example is writing to files (or forwarding) via priority. In |FmtBasicName| +format, this looks like: + +:: + + mail.info /var/log/mail.log + mail.err @@server.example.net + +This is hard to beat in simplicity, still being taught in courses +and a lot of people know this syntax. It is perfectly fine to use +these constructs even in newly written config files. Note that many +distributions use this format in their default rsyslog.conf, so you will +likely find it in existing configurations. + +**For anything more advanced, use** the |FmtAdvancedName| format. Advantages are: + +- fine control over rsyslog operations via advanced parameters +- easy to follow block structure +- easy to write +- safe for use with include files + +To continue with the above example, the |FmtAdvancedName| format is preferable +if you want to make sure that an offline remote destination will not slow down +local log file writing. In that case, forwarding is done via: + +:: + + mail.err action(type="omfwd" protocol="tcp" queue.type="linkedList") + +As can be seen by this example, the |FmtAdvancedName| format permits specifying +additional parameters to fine tune the behavior, whereas the |FmtBasicName| +format does not provide this level of control. + +**Do not use** |FmtObsoleteName| **format. It will make your life +miserable.** It is primarily supported in order to not break existing +configurations. + +Whatever you can do with the |FmtObsoleteName| format, you can also do +with the |FmtAdvancedName| format. The opposite is not true: Many newer features +cannot be turned on via the |FmtObsoleteName| format. The |FmtObsoleteName| +format is hard to understand and hard to get right. As you may inherit a rsyslog +configuration that makes use of it, this documentation gives you some clues +of what the obsolete statements do. For full details, obtain a +`v5 version of the rsyslog +documentation <http://www.rsyslog.com/doc/v5-stable/index.html>`_ (yes, this +format is dead since 2010!). diff --git a/source/configuration/config_param_types.rst b/source/configuration/config_param_types.rst new file mode 100644 index 0000000..36cc37c --- /dev/null +++ b/source/configuration/config_param_types.rst @@ -0,0 +1,48 @@ +Configuration Parameter Types +============================= +Configuration parameter values have different data types. +Unfortunately, the type currently must be guessed from the description +(consider contributing to the doc to help improve it). In general, the +following types are used: + +- **numbers** + + The traditional integer format. Numbers may include '.' and ',' + for readability. So you can for example specify either "1000" or + "1,000" with the same result. Please note that rsyslogd simply + *ignores* the punctuation. From it's point of view, "1,,0.0.,.,0" + also has the value 1000. + +- **sizes** + + Used for things like file size, main message queue sizes and the like. + These are integers, but support modifier after the number part. + For example, 1k means 1024. Supported are + k(ilo), m(ega), g(iga), t(era), p(eta) and e(xa). Lower case letters + refer to the traditional binary definition (e.g. 1m equals 1,048,576) + whereas upper case letters refer to their new 1000-based definition (e.g + 1M equals 1,000,000). + +- **complete line** + + A string consisting of multiple characters. This is relatively + seldom used and sometimes looks confusing (rsyslog v7+ has a much better + approach at these types of values). + +- **single word** + + This is used when only a single word can be provided. A "single + word" is a string without spaces in it. **No** quoting is necessary + nor permitted (the quotes would become part of the word). + +- **character** + + A single (printable) character. Must **not** be quoted. + +- **boolean** + + The traditional boolean type, specified as "on" (1) or "off" (0). + +Note that some other value types are occasionally used. However, the +majority of types is one of those listed above. The list is updated +as need arises and time permits. diff --git a/source/configuration/converting_to_new_format.rst b/source/configuration/converting_to_new_format.rst new file mode 100644 index 0000000..fcff061 --- /dev/null +++ b/source/configuration/converting_to_new_format.rst @@ -0,0 +1,196 @@ +Converting older formats to |FmtAdvancedName| +============================================= + +First of all, converting of older formats is not strictly necessary. All +formats can be mixed and matched and play well together. + +There are stil a number of reasons to convert older formats: + +* existing simple constructs need to be enhanced and become more complex +* aid future extensions +* ensure no side-effects accidentally occur +* unify rsyslog.conf language + +Do not overdo conversion +~~~~~~~~~~~~~~~~~~~~~~~~ + +Note: simple facility and severity based filters which trigger writing to +files can actually be very well expressd in |FmtBasicName|. So if you have +something like:: + + mail.info /var/log/maillog + +We suggest you leave it as-is without conversion. Equally, in our opinion it is +also fine to add new rules like the above. If you still want to convert, the +line may look as follows (completely in new format):: + + if prifilt("mail.info") then { + action(type="omfile" file="/var/log/maillog") + } + +More compact, this can also be written like:: + + if prifilt("mail.info") then action(type="omfile" file="/var/log/maillog") + +The older-selector-style filter is well-known, so this may also write it as:: + + mail.info action(type="omfile" file="/var/log/maillog") + +There are ample additional possibilities. We suggest to keep things consistent. + +Converting Module Load +~~~~~~~~~~~~~~~~~~~~~~ +This is very straight-forward. In |FmtObsoleteName| format we use:: + + $ModLoad module-name + +This is very simply converted to:: + + module(load="module-name") + +Sometimes modules provide global settings. In |FmtObsoleteName| format these are given in +individual lines **after** the \$ModLoad. In |FmtAdvancedName| format they are given inside +the module object. This makes it much clearer which module they belong to and +that they actually are global parameters (in contrast to per-action or per-listener +parameters). A typical example is `imtcp`:: + + $ModLoad imtcp + $InputTCPMaxSession 500 + +This is converted to:: + + module(load="imtcp" maxSessions="500") + +Note: in |FmtObsoleteName| format it is possible to provide global parameters more than once. +In this case it is unclear which one actually applies. For example:: + + $ModLoad imtcp + $InputTCPMaxSession 500 + ... + *.* /var/log/messages + ... + $InputTCPMaxSession 200 + +This is especially problematic if module-global parameters are used multiple times +in include files. + +In |FmtAdvancedName| format this is no longer possible. Module-global parameters can only +be applied once when the module is loaded. Any attempt to change them afterwards +results in an error message and will be ignored. The error messages will help you +find and fix multiple settings. Let us assume "200" is the setting actually intended +in above config snippet. So it would be converted to:: + + module(load="imtcp" maxSessions="200") + ... + *.* /var/log/messages + ... + + +Converting Actions +~~~~~~~~~~~~~~~~~~ +In general, you have lines like:: + + filter action + +where *filter* is any of the filters and *action* is ... the action to be +carried out. As could be seen above, the filter does **not** necessarily +need to be changed in order to convert the action. All filters also work +with all config formats. It often is best to keep existing filters, at +least while working on the conversion (do not change too many things at once). + +The following table lists traditional action syntax and how it can be +converted to new-style ``action()`` objects. The link will bring you to +detail documentation. In these detail documentations all parameters are given. +It is also specified which |FmtObsoleteName| directives map to |FmtAdvancedName| +properties. +This table is not conclusive but covers the most commonly used actions. + +.. csv-table:: + :header: "|FmtBasicName|", "|FmtAdvancedName|" + :widths: auto + :class: parameter-table + + "file path (`/var/log/...`)", "action(type="":doc:`omfile <modules/omfile>`"" file=""/var/log.../"" ...)" + "UDP forwarding (`@remote`)", "action(type="":doc:`omfwd <modules/omfwd>`"" target=""remote"" protocol=""udp"" ...)" + "TCP forwarding (`@@remote`)", "action(type="":doc:`omfwd <modules/omfwd>`"" target=""remote"" protocol=""tcp"" ...)" + "user notify (``:omusrmsg:user``)", "action(type="":doc:`omusrmsg <modules/omusrmsg>`"" users=""user"" ...)" + "module name (``:omxxx:..``)", "action(type="":doc:`omxxx <modules/idx_output>`"" ...)" + +Some concrete examples:: + + OLD: :hostname, contains, "remote-sender" @@central + NEW: :hostname, contains, "remote-sender" action(type="omfwd" target="central" protocol="tcp") + + OLD: if $msg contains "error" then @central + NEW: if $msg contains "error" then action(type="omfwd" target="central" protocol="udp") + + OLD: *.emerg :omusrmsg:* + NEW: *.emerg action(type="omusrmsg" users="*") + +**NOTE:** Some actions do not have a |FmtBasicName| format configuration line. They may +only be called via the ``action()`` syntax. Similarly, some very few actions, +mostly contributed, do not support ``action()`` syntax and thus can only be +configured via |FmtBasicName| and |FmtObsoleteName|. See module doc for details. + + +Action with Multiple Parameters +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +In many cases, actions have additional parameters, which also need to be converted. In +|FmtObsoleteName| format the action parameters are given before the actual action call. +To convert such constructs, you need to map all |FmtObsoleteName| parameters to |FmtAdvancedName| +ones. To look these up, you need to turn to three different documentation pages: + +* the :doc:`action object <actions>` itself +* the :doc:`output module <modules/idx_output>` that is called in the action (e.g. omfwd) +* the :doc:`queue documentation <../rainerscript/queue_parameters>` (if an action queue is used) + +To find the parameter in question, you can other do an on-page search via the browser on these +pages. Often it is very convenient to just use the +`rsyslog doc search engine <https://www.rsyslog.com/doc/master/search.html?q=>`_: +Type the |FmtObsoleteName| format statement into the search box. Most often, one of the first +search results is the matching object description doc page. + +Converting Action Chains +~~~~~~~~~~~~~~~~~~~~~~~~ + +Actions can be chained via the ampersand character ('``&``'). In |FmtAdvancedName| +format this has been replaced by blocks. For example:: + + *.error /var/log/errorlog + & @remote + +becomes:: + + *.error { + action(type="omfile" file="/var/log/errorlog") + action(type="omfwd" target="remote" protocol="udp") + } + +The latter is much easier to understand and less error-prone when extended. + +A common construct is to send messages to a remote host based on some message +content and then not further process it. This involves the ``stop`` statement +(or it's very old-time equivalent tilde ('``~``'). It may be specfied as such:: + + :msg, contains, "error" @remote + & ~ + +which is equavalent to:: + + :msg, contains, "error" @remote + & stop + +This format is often found in more modern distro's rsyslog.conf. It again is +fully equivalent to:: + + :msg, contains, "error" { + action(type="omfwd" target="remote" protocol="udp") + stop + } + +And, just to prove the point, this is also exactly the same like:: + + if $msg contains "error" then { + action(type="omfwd" target="remote" protocol="udp") + stop + } diff --git a/source/configuration/cryprov_gcry.rst b/source/configuration/cryprov_gcry.rst new file mode 100644 index 0000000..7dc32aa --- /dev/null +++ b/source/configuration/cryprov_gcry.rst @@ -0,0 +1,104 @@ +libgcrypt Log Crypto Provider (gcry) +==================================== + +**Crypto Provider Name:** gcry + +**Author:** Rainer Gerhards <rgerhards@adiscon.com> + +**Supported Since:** since 7.3.10 + +**Description**: + +Provides encryption support to rsyslog. + +**Configuration Parameters**: + +Crypto providers are loaded by omfile, when the provider is selected in +its "cry.providerName" parameter. Parameters for the provider are given +in the omfile action instance line. + +This provider creates an encryption information file with the same base +name but the extension ".encinfo" for each log file (both for fixed-name +files as well as dynafiles). Both files together form a set. So you need +to archive both in order to prove integrity. + +- **cry.algo** <Encryption Algorithm> + The algorithm (cipher) to be used for encryption. The default algorithm is "AES128". + Currently, the following Algorithms are supported: + + - 3DES + - CAST5 + - BLOWFISH + - AES128 + - AES192 + - AES256 + - TWOFISH + - TWOFISH128 + - ARCFOUR + - DES + - SERPENT128 + - SERPENT192 + - SERPENT256 + - RFC2268\_40 + - SEED + - CAMELLIA128 + - CAMELLIA192 + - CAMELLIA256 + + The actual availability of an algorithms depends on which ones are + compiled into libgcrypt. Note that some versions of libgcrypt simply + abort the process (rsyslogd in this case!) if a supported algorithm + is select but not available due to libgcrypt build settings. There is + nothing rsyslog can do against this. So in order to avoid production + downtime, always check carefully when you change the algorithm. + +- **cry.mode** <Algorithm Mode> + The encryption mode to be used. Default ist Cipher Block Chaining + (CBC). Note that not all encryption modes can be used together with + all algorithms. + Currently, the following modes are supported: + + - ECB + - CFB + - CBC + - STREAM + - OFB + - CTR + - AESWRAP + + +- **cry.key** <encryption key> + TESTING AID, NOT FOR PRODUCTION USE. This uses the KEY specified + inside rsyslog.conf. This is the actual key, and as such this mode is + highly insecure. However, it can be useful for initial testing steps. + This option may be removed in the future. + +- **cry.keyfile** <filename> + Reads the key from the specified file. The file must contain the + key, only, no headers or other meta information. Keyfiles can be + generated via the rscrytool utility. + +- **cry.keyprogram** <path to program> + If given, the key is provided by a so-called "key program". This + program is executed and must return the key (as well as some meta + information) via stdout. The core idea of key programs is that using + this interface the user can implement as complex (and secure) method + to obtain keys as desired, all without the need to make modifications + to rsyslog. + +**Caveats/Known Bugs:** + +- currently none known + +**Samples:** + +This encrypts a log file. Default parameters are used, they key is +provided via a keyfile. + +:: + + action(type="omfile" file="/var/log/somelog" cry.provider="gcry" + cry.keyfile="/secured/path/to/keyfile") + +Note that the keyfile can be generated via the rscrytool utility (see its +documentation for how to actually do that). diff --git a/source/configuration/droppriv.rst b/source/configuration/droppriv.rst new file mode 100644 index 0000000..8dc55a6 --- /dev/null +++ b/source/configuration/droppriv.rst @@ -0,0 +1,31 @@ +Dropping privileges in rsyslog +============================== + +**Available since**: 4.1.1 + +**Description**: + +Rsyslogd provides the ability to drop privileges by impersonating as +another user and/or group after startup. + +Please note that due to POSIX standards, rsyslogd always needs to start +up as root if there is a listener who must bind to a network port below +1024. For example, the UDP listener usually needs to listen to 514 and +therefore rsyslogd needs to start up as root. + +If you do not need this functionality, you can start rsyslog directly as +an ordinary user. That is probably the safest way of operations. +However, if a startup as root is required, you can use the +$PrivDropToGroup and $PrivDropToUser config directives to specify a +group and/or user that rsyslogd should drop to after initialization. +Once this happens, the daemon runs without high privileges (depending, +of course, on the permissions of the user account you specified). + +A special note for Docker and other container system users: user and +group names are usually not fully mirrored into containers. As such, +we strongly advise to use numerical IDs instead of user or group +names when configuring privilege drop. + +Privilege drop is configured via the +:doc:`global configuraton object<../rainerscript/global>` under the +"`privilege.`" set of parameters. diff --git a/source/configuration/dyn_stats.rst b/source/configuration/dyn_stats.rst new file mode 100644 index 0000000..b5d2ad9 --- /dev/null +++ b/source/configuration/dyn_stats.rst @@ -0,0 +1,106 @@ +Dynamic Stats +============= + +Rsyslog produces runtime-stats to allow user to study service health, performance, bottlenecks etc. Runtime-stats counters that Rsyslog components publish are statically defined. + +**Dynamic Stats** (called dyn-stats henceforth) component allows user to configure stats-namespaces (called stats-buckets) and increment counters within these buckets using Rainerscript function call. + +The metric-name in this case can be a message-property or a sub-string extracted from message etc. + +Dyn-stats configuration +^^^^^^^^^^^^^^^^^^^^^^^ + +Dyn-stats configuration involves a **two part setup**. + +dyn_stats(name="<bucket>"...) (object) +-------------------------------------- + +**Defines** the bucket(identified by the bucket-name) and allows user to set some properties that control behavior of the bucket. + +:: + + dyn_stats(name="msg_per_host") + +Parameters: + **name** <string literal, mandatory> : Name of the bucket. + + **resettable** <on|off, default: on> : Whether or not counters should be reset every time they are reported. This works independent of ``resetCounters`` config parameter in :doc:`modules/impstats`. + + **maxCardinality** <number, default: 2000> : Maximum number of unique counter-names to track. + + **unusedMetricLife** <number, default: 3600> : Interval between full purges (in seconds). This prevents unused counters from occupying resources forever. + + +A definition setting all the parameters looks like: + +:: + + dyn_stats(name="msg_per_host" resettable="on" maxCardinality="3000" unusedMetricLife="600") + + +dyn_inc("<bucket>", <expr>) (function) +-------------------------------------- + +**Increments** counter identified by value of variable in bucket identified by name. + +Parameters: + **name** <string literal, mandatory> : Name of the bucket + + **expr** <expression resulting in a string> : Name of counter (this name will be reported by impstats to identify the counter) + +A ``dyn_inc`` call looks like: + +:: + + set $.inc = dyn_inc("msg_per_host", $hostname); + + if ($.inc != 0) then { + .... + } + +``$.inc`` captures the error-code. It has value ``0`` when increment operation is successful and non-zero when it fails. It uses Rsyslog error-codes. + +Reporting +^^^^^^^^^ + +Legacy format: + +:: + + ... + global: origin=dynstats msg_per_host.ops_overflow=1 msg_per_host.new_metric_add=3 msg_per_host.no_metric=0 msg_per_host.metrics_purged=0 msg_per_host.ops_ignored=0 + ... + msg_per_host: origin=dynstats.bucket foo=2 bar=1 baz=1 + ... + +Json(variants with the same structure are used in other Json based formats such as ``cee`` and ``json-elasticsearch``) format: + +:: + + ... + { "name": "global", "origin": "dynstats", "values": { "msg_per_host.ops_overflow": 1, "msg_per_host.new_metric_add": 3, "msg_per_host.no_metric": 0, "msg_per_host.metrics_purged": 0, "msg_per_host.ops_ignored": 0 } } + ... + { "name": "msg_per_host", "origin": "dynstats.bucket", "values": { "foo": 2, "bar": 1, "baz": 1 } } + ... + +In this case counters are encapsulated inside an object hanging off top-level-key ``values``. + +Fields +------ + +**global: origin=dynstats**: + **ops_overflow**: Number of operations ignored because number-of-counters-tracked has hit configured max-cardinality. + + **new_metric_add**: Number of "new" metrics added (new counters created). + + **no_metric**: Counter-name given was invalid (length = 0). + + **metrics_purged**: Number of counters discarded at discard-cycle (controlled by **unusedMetricLife**). + + **ops_ignored**: Number of operations ignored due to potential performance overhead. Dyn-stats subsystem ignores operations to avoid performance-penalty if it can't get access to counter without delay(lock acquiring latency). + + **purge_triggered**: Indicates that a discard was performed (1 implies a discard-cycle run). + +**msg_per_host: origin=dynstats.bucket**: + **<metric_name>**: Value of counter identified by <metric-name>. + diff --git a/source/configuration/examples.rst b/source/configuration/examples.rst new file mode 100644 index 0000000..e035a0e --- /dev/null +++ b/source/configuration/examples.rst @@ -0,0 +1,214 @@ +Examples +-------- + +Below are example for templates and selector lines. I hope they are +self-explanatory. + +Templates +~~~~~~~~~ + +Please note that the samples are split across multiple lines. A template +MUST NOT actually be split across multiple lines. + +A template that resembles traditional syslogd file output: + $template TraditionalFormat,"%timegenerated% %HOSTNAME% + %syslogtag%%msg:::drop-last-lf%\\n" + +A template that tells you a little more about the message: + $template precise,"%syslogpriority%,%syslogfacility%,%timegenerated%,%HOSTNAME%, + %syslogtag%,%msg%\\n" + +A template for RFC 3164 format: + $template RFC3164fmt,"<%PRI%>%TIMESTAMP% %HOSTNAME% %syslogtag%%msg%" + +A template for the format traditionally used for user messages: + $template usermsg," XXXX%syslogtag%%msg%\\n\\r" + +And a template with the traditional wall-message format: + $template wallmsg,"\\r\\n\\7Message from syslogd@%HOSTNAME% at %timegenerated% + +A template that can be used for the database write (please note the SQL template option) + $template MySQLInsert,"insert iut, message, received at values + ('%iut%', '%msg:::UPPERCASE%', '%timegenerated:::date-mysql%') + into systemevents\\r\\n", SQL + +The following template emulates +`WinSyslog <http://www.winsyslog.com/en/>`_ format (it's an +`Adiscon <http://www.adiscon.com/>`_ format, you do not feel bad if +you don't know it ;)). It's interesting to see how it takes different +parts out of the date stamps. What happens is that the date stamp is +split into the actual date and time and the these two are combined with +just a comma in between them. + +:: + + $template WinSyslogFmt,"%HOSTNAME%,%timegenerated:1:10:date-rfc3339%, + %timegenerated:12:19:date-rfc3339%,%timegenerated:1:10:date-rfc3339%, + %timegenerated:12:19:date-rfc3339%,%syslogfacility%,%syslogpriority%, + %syslogtag%%msg%\\n" + +Selector lines +~~~~~~~~~~~~~~ + +:: + + # Store critical stuff in critical + # + *.=crit;kern.none /var/adm/critical + +This will store all messages with the priority crit in the file +/var/adm/critical, except for any kernel message. + +:: + + # Kernel messages are first, stored in the kernel + # file, critical messages and higher ones also go + # to another host and to the console. Messages to + # the host server.example.net are forwarded in RFC 3164 + # format (using the template defined above). + # + kern.* /var/adm/kernel + kern.crit @server.example.net;RFC3164fmt + kern.crit /dev/console + kern.info;kern.!err /var/adm/kernel-info + +The first rule direct any message that has the kernel facility to the +file /var/adm/kernel. + +The second statement directs all kernel messages of the priority crit +and higher to the remote host server.example.net. This is useful, because if the +host crashes and the disks get irreparable errors you might not be able +to read the stored messages. If they're on a remote host, too, you still +can try to find out the reason for the crash. + +The third rule directs these messages to the actual console, so the +person who works on the machine will get them, too. + +The fourth line tells rsyslogd to save all kernel messages that come +with priorities from info up to warning in the file /var/adm/kernel-info. +Everything from err and higher is excluded. + +:: + + # The tcp wrapper loggs with mail.info, we display + # all the connections on tty12 + # + mail.=info /dev/tty12 + +This directs all messages that uses mail.info (in source LOG\_MAIL \| +LOG\_INFO) to /dev/tty12, the 12th console. For example the tcpwrapper +tcpd(8) uses this as it's default. + +:: + + # Store all mail concerning stuff in a file + # + mail.\*;mail.!=info /var/adm/mail + +This pattern matches all messages that come with the mail facility, +except for the info priority. These will be stored in the file +/var/adm/mail. + +:: + + # Log all mail.info and news.info messages to info + # + mail,news.=info /var/adm/info + +This will extract all messages that come either with mail.info or with +news.info and store them in the file /var/adm/info. + +:: + + # Log info and notice messages to messages file + # + *.=info;*.=notice;\ + mail.none /var/log/messages + +This lets rsyslogd log all messages that come with either the info or +the notice facility into the file /var/log/messages, except for all +messages that use the mail facility. + +:: + + # Log info messages to messages file + # + *.=info;\ + mail,news.none /var/log/messages + +This statement causes rsyslogd to log all messages that come with the +info priority to the file /var/log/messages. But any message coming +either with the mail or the news facility will not be stored. + +:: + + # Emergency messages will be displayed to all users + # + *.=emerg :omusrmsg:* + +This rule tells rsyslogd to write all emergency messages to all +currently logged in users. + +:: + + # Messages of the priority alert will be directed + # to the operator + # + *.alert root,rgerhards + +This rule directs all messages with a priority of alert or higher to +the terminals of the operator, i.e. of the users "root'' and +"rgerhards'' if they're logged in. + +:: + + *.* @server.example.net + +This rule would redirect all messages to a remote host called +server.example.net. This is useful especially in a cluster of machines where all +syslog messages will be stored on only one machine. + +In the format shown above, UDP is used for transmitting the message. +The destination port is set to the default auf 514. Rsyslog is also +capable of using much more secure and reliable TCP sessions for message +forwarding. Also, the destination port can be specified. To select TCP, +simply add one additional @ in front of the host name (that is, @host is +UDP, @@host is TCP). For example: + +:: + + *.* @@server.example.net + +To specify the destination port on the remote machine, use a colon +followed by the port number after the machine name. The following +forwards to port 1514 on server.example.net: + +:: + + *.* @@server.example.net:1514 + +This syntax works both with TCP and UDP based syslog. However, you will +probably primarily need it for TCP, as there is no well-accepted port +for this transport (it is non-standard). For UDP, you can usually stick +with the default auf 514, but might want to modify it for security reasons. +If you would like to do that, it's quite easy: + +:: + + *.* @server.example.net:1514 + *.* >dbhost,dbname,dbuser,dbpassword;dbtemplate + +This rule writes all message to the database "dbname" hosted on +"dbhost". The login is done with user "dbuser" and password +"dbpassword". The actual table that is updated is specified within the +template (which contains the insert statement). The template is called +"dbtemplate" in this case. + +:: + + :msg,contains,"error" @server.example.net + +This rule forwards all messages that contain the word "error" in the msg +part to the server "errorServer". Forwarding is via UDP. Please note the +colon in fron + diff --git a/source/configuration/filters.rst b/source/configuration/filters.rst new file mode 100644 index 0000000..5357fad --- /dev/null +++ b/source/configuration/filters.rst @@ -0,0 +1,319 @@ +Filter Conditions +================= + +Rsyslog offers four different types "filter conditions": + +- "traditional" severity and facility based selectors +- property-based filters +- expression-based filters +- BSD-style blocks (not upward compatible) + +Selectors +--------- + +**Selectors are the traditional way of filtering syslog messages.** They +have been kept in rsyslog with their original syntax, because it is +well-known, highly effective and also needed for compatibility with +stock syslogd configuration files. If you just need to filter based on +priority and facility, you should do this with selector lines. They are +**not** second-class citizens in rsyslog and offer the simplest syntax +for this job. In versions of rsyslog prior to v7 there were significant +performance gains by using selector lines instead of the |FmtAdvancedName| +format. There is no longer any difference in performance between the two +formats. + +The selector field itself again consists of two parts, a facility and a +priority, separated by a period (".''). Both parts are case insensitive +and can also be specified as decimal numbers, but don't do that, you +have been warned. Both facilities and priorities are described in +syslog(3). The names mentioned below correspond to the similar +LOG\_-values in /usr/include/syslog.h. + +The facility is one of the following keywords: auth, authpriv, cron, +daemon, kern, lpr, mail, mark, news, security (same as auth), syslog, +user, uucp and local0 through local7. The keyword security should not be +used anymore and mark is only for internal use and therefore should not +be used in applications. Anyway, you may want to specify and redirect +these messages here. The facility specifies the subsystem that produced +the message, i.e. all mail programs log with the mail facility +(LOG\_MAIL) if they log using syslog. + +The priority is one of the following keywords, in ascending order: +debug, info, notice, warning, warn (same as warning), err, error (same +as err), crit, alert, emerg, panic (same as emerg). The keywords error, +warn and panic are deprecated and should not be used anymore. The +priority defines the severity of the message. + +The behavior of the original BSD syslogd is that all messages of the +specified priority and higher are logged according to the given action. +Rsyslogd behaves the same, but has some extensions. + +In addition to the above mentioned names the rsyslogd(8) understands +the following extensions: An asterisk ("\*'') stands for all facilities +or all priorities, depending on where it is used (before or after the +period). The keyword none stands for no priority of the given facility. +You can specify multiple facilities with the same priority pattern in +one statement using the comma (",'') operator. You may specify as much +facilities as you want. Remember that only the facility part from such a +statement is taken, a priority part would be skipped. + +Multiple selectors may be specified for a single action using the +semicolon (";'') separator. Remember that each selector in the selector +field is capable to overwrite the preceding ones. Using this behavior +you can exclude some priorities from the pattern. + +Rsyslogd has a syntax extension to the original BSD source, that makes +the use of selectors use more intuitively. You may precede every priority +with an equals +sign ("='') to specify only this single priority and not any of the +above. You may also (both is valid, too) precede the priority with an +exclamation mark ("!'') to ignore all that priorities, either exact this +one or this and any higher priority. If you use both extensions then the +exclamation mark must occur before the equals sign, just use it +intuitively. + +However, please note that there are some restrictions over the traditional +BSD syslog behaviour. These restrictions stem back to sysklogd, exist +probably since at least the 1990's and as such have always been in +rsyslog. + +Namely, in BSD syslogd you can craft a selector like this: + +\*.debug;local6.err + +The intent is to log all facilities at debug or higher, except for local6, +which should only log at err or higher. + +Unfortunately, local6.err will permit error severity and higher, but will +*not* exclude lower sevrity messages from facility local6. + +As an alternative, you can explicitely exclude all severities that you do +not want to match. For the above case, this selector is equivalent to the +BSD syslog selector: + +\*.debug;local6.!=info;local6.!=notice;local6.!=warn + +An easier approach is probably to do if ... then based matching in script. + +Property-Based Filters +---------------------- + +Property-based filters are unique to rsyslogd. They allow to filter on +any property, like HOSTNAME, syslogtag and msg. A list of all +currently-supported properties can be found in the :doc:`rsyslog properties +documentation <properties>`. With this filter, each property can be checked +against a specified value, using a specified compare operation. + +A property-based filter must start with a colon **in column 1**. This tells +rsyslogd that it is the new filter type. The colon must be followed by +the property name, a comma, the name of the compare operation to carry +out, another comma and then the value to compare against. This value +must be quoted. There can be spaces and tabs between the commas. +Property names and compare operations are case-sensitive, so "msg" +works, while "MSG" is an invalid property name. In brief, the syntax is +as follows: + +``:property, [!]compare-operation, "value"`` + +Compare-Operations +~~~~~~~~~~~~~~~~~~ + +The following **compare-operations** are currently supported: + +**contains** + Checks if the string provided in value is contained in the property. + There must be an exact match, wildcards are not supported. + +**isequal** + Compares the "value" string provided and the property contents. These + two values must be exactly equal to match. The difference to contains is + that contains searches for the value anywhere inside the property value, + whereas all characters must be identical for isequal. As such, isequal + is most useful for fields like syslogtag or FROMHOST, where you probably + know the exact contents. + +**startswith** + Checks if the value is found exactly at the beginning of the property + value. For example, if you search for "val" with + + ``:msg, startswith, "val"`` + + it will be a match if msg contains "values are in this message" but it + won't match if the msg contains "There are values in this message" (in + the later case, *"contains"* would match). Please note that "startswith" is + by far faster than regular expressions. So even once they are + implemented, it can make very much sense (performance-wise) to use + "startswith". + +**regex** + Compares the property against the provided POSIX BRE regular expression. + +**ereregex** + Compares the property against the provided POSIX ERE regular expression. + +You can use the bang-character (!) immediately in front of a +compare-operation, the outcome of this operation is negated. For +example, if msg contains "This is an informative message", the following +sample would not match: + +``:msg, contains, "error"`` + +but this one matches: + +``:msg, !contains, "error"`` + +Using negation can be useful if you would like to do some generic +processing but exclude some specific events. You can use the discard +action in conjunction with that. A sample would be: + +:: + + *.* /var/log/allmsgs-including-informational.log + :msg, contains, "informational" ~ + *.* /var/log/allmsgs-but-informational.log + +Do not overlook the tilde in line 2! In this sample, all messages +are written to the file allmsgs-including-informational.log. Then, all +messages containing the string "informational" are discarded. That means +the config file lines below the "discard line" (number 2 in our sample) +will not be applied to this message. Then, all remaining lines will also +be written to the file allmsgs-but-informational.log. + +Value Part +~~~~~~~~~~ + +**Value** is a quoted string. It supports some escape sequences: + +\\" - the quote character (e.g. "String with \\"Quotes\\"") + \\\\ - the backslash character (e.g. "C:\\\\tmp") + +Escape sequences always start with a backslash. Additional escape +sequences might be added in the future. Backslash characters **must** be +escaped. Any other sequence then those outlined above is invalid and may +lead to unpredictable results. + +Probably, "msg" is the most prominent use case of property based +filters. It is the actual message text. If you would like to filter +based on some message content (e.g. the presence of a specific code), +this can be done easily by: + +``:msg, contains, "ID-4711"`` + +This filter will match when the message contains the string "ID-4711". +Please note that the comparison is case-sensitive, so it would not match +if "id-4711" would be contained in the message. + +``:msg, regex, "fatal .* error"`` + +This filter uses a POSIX regular expression. It matches when the string +contains the words "fatal" and "error" with anything in between (e.g. +"fatal net error" and "fatal lib error" but not "fatal error" as two +spaces are required by the regular expression!). + +Getting property-based filters right can sometimes be challenging. In +order to help you do it with as minimal effort as possible, rsyslogd +spits out debug information for all property-based filters during their +evaluation. To enable this, run rsyslogd in foreground and specify the +"-d" option. + +Boolean operations inside property based filters (like 'message contains +"ID17" or message contains "ID18"') are currently not supported (except +for "not" as outlined above). Please note that while it is possible to +query facility and severity via property-based filters, it is far more +advisable to use classic selectors (see above) for those cases. + +Expression-Based Filters +------------------------ + +Expression based filters allow filtering on arbitrary complex +expressions, which can include boolean, arithmetic and string +operations. Expression filters will evolve into a full configuration +scripting language. Unfortunately, their syntax will slightly change +during that process. So if you use them now, you need to be prepared to +change your configuration files some time later. However, we try to +implement the scripting facility as soon as possible (also in respect to +stage work needed). So the window of exposure is probably not too long. + +Expression based filters are indicated by the keyword "if" in column 1 +of a new line. They have this format: + +:: + + if expr then action-part-of-selector-line + +"if" and "then" are fixed keywords that must be present. "expr" is a +(potentially quite complex) expression. So the :doc:`expression +documentation <../rainerscript/expressions>` for details. +"action-part-of-selector-line" is an action, just as you know it (e.g. +"/var/log/logfile" to write to that file). + +BSD-style Blocks +---------------- + +**Note:** rsyslog v7+ no longer supports BSD-style blocks +for technical reasons. So it is strongly recommended **not** to +use them. + +Rsyslogd supports BSD-style blocks inside rsyslog.conf. Each block of +lines is separated from the previous block by a program or hostname +specification. A block will only log messages corresponding to the most +recent program and hostname specifications given. Thus, a block which +selects ‘ppp’ as the program, directly followed by a block that selects +messages from the hostname ‘dialhost’, then the second block will only +log messages from the ppp program on dialhost. + +A program specification is a line beginning with ‘!prog’ and the +following blocks will be associated with calls to syslog from that +specific program. A program specification for ‘foo’ will also match any +message logged by the kernel with the prefix ‘foo: ’. Alternatively, a +program specification ‘-foo’ causes the following blocks to be applied +to messages from any program but the one specified. A hostname +specification of the form ‘+hostname’ and the following blocks will be +applied to messages received from the specified hostname. Alternatively, +a hostname specification ‘-hostname’ causes the following blocks to be +applied to messages from any host but the one specified. If the hostname +is given as ‘@’, the local hostname will be used. (NOT YET IMPLEMENTED) +A program or hostname specification may be reset by giving the program +or hostname as ‘\*’. + +Please note that the "#!prog", "#+hostname" and "#-hostname" syntax +available in BSD syslogd is not supported by rsyslogd. By default, no +hostname or program is set. + +Examples +-------- + +:: + + *.* /var/log/file1 # the traditional way + if $msg contains 'error' then /var/log/errlog # the expression-based way + +Right now, you need to specify numerical values if you would like to +check for facilities and severity. These can be found in :rfc:`5424`. +If you don't like that, +you can of course also use the textual property - just be sure to use +the right one. As expression support is enhanced, this will change. For +example, if you would like to filter on message that have facility +local0, start with "DEVNAME" and have either "error1" or "error0" in +their message content, you could use the following filter: + +:: + + if $syslogfacility-text == 'local0' and $msg startswith 'DEVNAME' and ($msg contains 'error1' or $msg contains 'error0') then /var/log/somelog + +Please note that the above must all be on one line! And if you would +like to store all messages except those that contain "error1" or +"error0", you just need to add a "not": + +:: + + if $syslogfacility-text == 'local0' and $msg startswith 'DEVNAME' and not ($msg contains 'error1' or $msg contains 'error0') then /var/log/somelog + +If you would like to do case-insensitive comparisons, use "contains\_i" +instead of "contains" and "startswith\_i" instead of "startswith". +Note that regular expressions are currently NOT supported in +expression-based filters. These will be added later when function +support is added to the expression engine (the reason is that regular +expressions will be a separate loadable module, which requires some more +prerequisites before it can be implemented). + diff --git a/source/configuration/global/index.rst b/source/configuration/global/index.rst new file mode 100644 index 0000000..2e6b840 --- /dev/null +++ b/source/configuration/global/index.rst @@ -0,0 +1,163 @@ +Legacy Global Configuration Statements +====================================== +Global configuration statements, as their name implies, usually affect +some global features. However, some also affect main queues, which +are "global" to a ruleset. + +True Global Directives +---------------------- + +.. toctree:: + :glob: + + options/rsconf1_abortonuncleanconfig + options/rsconf1_debugprintcfsyslinehandlerlist + options/rsconf1_debugprintmodulelist + options/rsconf1_debugprinttemplatelist + options/rsconf1_failonchownfailure + options/rsconf1_generateconfiggraph + options/rsconf1_includeconfig + options/rsconf1_mainmsgqueuesize + options/rsconf1_maxopenfiles + options/rsconf1_moddir + options/rsconf1_modload + options/rsconf1_umask + options/rsconf1_resetconfigvariables + +- **$MaxMessageSize** <size\_nbr>, default 8k - allows to specify + maximum supported message size (both for sending and receiving). The + default should be sufficient for almost all cases. Do not set this + below 1k, as it would cause interoperability problems with other + syslog implementations. + + **Important:** In order for this directive to work correctly, + it **must** be placed right at the top of ``rsyslog.conf`` + (before any input is defined). + + Change the setting to e.g. 32768 if you would like to support large + message sizes for IHE (32k is the current maximum needed for IHE). I + was initially tempted to set the default to 32k, but there is a some + memory footprint with the current implementation in rsyslog. + If you intend to receive Windows Event Log data (e.g. via + `EventReporter <http://www.eventreporter.com/>`_), you might want to + increase this number to an even higher value, as event log messages + can be very lengthy ("$MaxMessageSize 64k" is not a bad idea). Note: + testing showed that 4k seems to be the typical maximum for **UDP** + based syslog. This is an IP stack restriction. Not always ... but + very often. If you go beyond that value, be sure to test that + rsyslogd actually does what you think it should do ;) It is highly + suggested to use a TCP based transport instead of UDP (plain TCP + syslog, RELP). This resolves the UDP stack size restrictions. + Note that 2k, is the smallest size that must be + supported in order to be compliant to the upcoming new syslog RFC + series. +- **$LocalHostName** [name] - this directive permits to overwrite the + system hostname with the one specified in the directive. If the + directive is given multiple times, all but the last one will be + ignored. Please note that startup error messages may be issued with + the real hostname. This is by design and not a bug (but one may argue + if the design should be changed ;)). Available since 4.7.4+, 5.7.3+, + 6.1.3+. +- **$LogRSyslogStatusMessages** [**on**/off] - If set to on (the + default), rsyslog emits message on startup and shutdown as well as + when it is HUPed. This information might be needed by some log + analyzers. If set to off, no such status messages are logged, what + may be useful for other scenarios. [available since 4.7.0 and 5.3.0] +- **$DefaultRuleset** [name] - changes the default ruleset for unbound + inputs to the provided *name* (the default default ruleset is named + "RSYSLOG\_DefaultRuleset"). It is advised to also read our paper on + :doc:`using multiple rule sets in rsyslog <../../concepts/multi_ruleset>`. +- **$DefaultNetstreamDriver** <drivername>, the default + :doc:`network stream driver <../../concepts/netstrm_drvr>` to use. + Defaults to ptcp. +- **$DefaultNetstreamDriverCAFile** </path/to/cafile.pem> +- **$DefaultNetstreamDriverCertFile** </path/to/certfile.pem> +- **$DefaultNetstreamDriverKeyFile** </path/to/keyfile.pem> +- **$NetstreamDriverCaExtraFiles** </path/to/extracafile.pem> - This + directive allows to configure multiple additional extra CA files. + This is intended for SSL certificate chains to work appropriately, + as the different CA files in the chain need to be specified. + It must be remarked that this directive only works with the OpenSSL driver. +- **$RepeatedMsgContainsOriginalMsg** [on/**off**] - "last message + repeated n times" messages, if generated, have a different format + that contains the message that is being repeated. Note that only the + first "n" characters are included, with n to be at least 80 + characters, most probably more (this may change from version to + version, thus no specific limit is given). The bottom line is that n + is large enough to get a good idea which message was repeated but it + is not necessarily large enough for the whole message. (Introduced + with 4.1.5). Once set, it affects all following actions. +- **$OptimizeForUniprocessor** - This directive is no longer supported. + While present in versions prior to 8.32.0, the directive had no effect + for many years. Attempts to use the directive now results in a warning. +- **$PreserveFQDN** [on/**off**) - if set to off (legacy default to remain + compatible to sysklogd), the domain part from a name that is within + the same domain as the receiving system is stripped. If set to on, + full names are always used. +- **$WorkDirectory** <name> (directory for spool and other work files. Do + **not** use trailing slashes) +- `$PrivDropToGroup <droppriv.html>`_ +- `$PrivDropToGroupID <droppriv.html>`_ +- `$PrivDropToUser <droppriv.html>`_ +- `$PrivDropToUserID <droppriv.html>`_ +- **$Sleep** <seconds> - puts the rsyslog main thread to sleep for the + specified number of seconds immediately when the directive is + encountered. You should have a good reason for using this directive! +- **$LocalHostIPIF** <interface name> - (available since 5.9.6) - if + provided, the IP of the specified interface (e.g. "eth0") shall be + used as fromhost-ip for local-originating messages. If this + directive is not given OR the interface cannot be found (or has no IP + address), the default of "127.0.0.1" is used. Note that this + directive can be given only once. Trying to reset will result in an + error message and the new value will be ignored. Please note that + modules must have support for obtaining the local IP address set via + this directive. While this is the case for rsyslog-provided modules, + it may not always be the case for contributed plugins. + **Important:** This directive shall be placed **right at the top of + rsyslog.conf**. Otherwise, if error messages are triggered before + this directive is processed, rsyslog will fix the local host IP to + "127.0.0.1", what than can not be reset. +- **$ErrorMessagesToStderr** [**on**\ \|off] - direct rsyslogd error + message to stderr (in addition to other targets) +- **$SpaceLFOnReceive** [on/**off**] - instructs rsyslogd to replace LF + with spaces during message reception (sysklogd compatibility aid). + This is applied at the beginning of the parser stage and cannot + be overridden (neither at the input nor parser level). Consequently, + it affects all inputs and parsers. + +main queue specific Directives +------------------------------ +Note that these directives modify ruleset main message queues. +This includes the default ruleset's main message queue, the one +that is always present. To do this, specify directives outside of +a ruleset definition. + +To understand queue parameters, read +:doc:`queues in rsyslog <../../concepts/queues>`. + +- **$MainMsgQueueCheckpointInterval** <number> +- **$MainMsgQueueDequeueBatchSize** <number> [default 32] +- **$MainMsgQueueDequeueSlowdown** <number> [number is timeout in + *micro*\ seconds (1000000us is 1sec!), default 0 (no delay). Simple + rate-limiting!] +- **$MainMsgQueueDiscardMark** <number> [default 98% of queue size] +- **$MainMsgQueueDiscardSeverity** <severity> [either a textual or + numerical severity! default 4 (warning)] +- **$MainMsgQueueFileName** <name> +- **$MainMsgQueueHighWaterMark** <number> [default 90% of queue size] +- **$MainMsgQueueImmediateShutdown** [on/**off**] +- **$MainMsgQueueLowWaterMark** <number> [default 70% of queue size] +- **$MainMsgQueueMaxFileSize** <size\_nbr>, default 1m +- **$MainMsgQueueTimeoutActionCompletion** <number> [number is timeout in + ms (1000ms is 1sec!), default 1000, 0 means immediate!] +- **$MainMsgQueueTimeoutEnqueue** <number> [number is timeout in ms (1000ms + is 1sec!), default 2000, 0 means discard immediately] +- **$MainMsgQueueTimeoutShutdown** <number> [number is timeout in ms + (1000ms is 1sec!), default 0 (indefinite)] +- **$MainMsgQueueWorkerTimeoutThreadShutdown** <number> [number is timeout + in ms (1000ms is 1sec!), default 60000 (1 minute)] +- **$MainMsgQueueType** [**FixedArray**/LinkedList/Direct/Disk] +- **$MainMsgQueueSaveOnShutdown** [on/**off**] +- **$MainMsgQueueWorkerThreads** <number>, num worker threads, default 2, + recommended 1 +- **$MainMsgQueueWorkerThreadMinumumMessages** <number>, default queue size/number of workers diff --git a/source/configuration/global/options/rsconf1_abortonuncleanconfig.rst b/source/configuration/global/options/rsconf1_abortonuncleanconfig.rst new file mode 100644 index 0000000..f1a7fb3 --- /dev/null +++ b/source/configuration/global/options/rsconf1_abortonuncleanconfig.rst @@ -0,0 +1,40 @@ +`rsyslog.conf configuration parameter <rsyslog_conf_global.html>`_ + +$AbortOnUncleanConfig +---------------------- + +**Type:** global configuration parameter + +**Parameter Values:** boolean (on/off, yes/no) + +**Available since:** 5.3.1+ + +**Default:** off + +**Description:** + +This parameter permits to prevent rsyslog from running when the +configuration file is not clean. "Not Clean" means there are errors or +some other annoyances that rsyslogd reports on startup. This is a +user-requested feature to have a strict startup mode. Note that with the +current code base it is not always possible to differentiate between an +real error and a warning-like condition. As such, the startup will also +prevented if warnings are present. I consider this a good thing in being +"strict", but I admit there also currently is no other way of doing it. + +It is recommended to use the new config style. The equivalent of this +parameter in the new style is ``global(abortOnUncleanConfig="")``. + +**Caveats:** + +Note that the consequences of a failed rsyslogd startup can be much more +serious than a startup with only partial configuration. For example, log +data may be lost or systems that depend on the log server in question +will not be able to send logs, what in the ultimate result could result +in a system hang on those systems. Also, the local system may hang when +the local log socket has become full and is not read. There exist many +such scenarios. As such, it is strongly recommended not to turn on this +parameter. + +[`rsyslog site <http://www.rsyslog.com/>`_\ ] + diff --git a/source/configuration/global/options/rsconf1_debugprintcfsyslinehandlerlist.rst b/source/configuration/global/options/rsconf1_debugprintcfsyslinehandlerlist.rst new file mode 100644 index 0000000..70ceb4c --- /dev/null +++ b/source/configuration/global/options/rsconf1_debugprintcfsyslinehandlerlist.rst @@ -0,0 +1,13 @@ +$DebugPrintCFSyslineHandlerList +------------------------------- + +**Type:** global configuration parameter + +**Default:** on + +**Description:** + +Specifies whether or not the configuration file sysline handler list +should be written to the debug log. Possible values: on/off. Default is +on. Does not affect operation if debugging is disabled. + diff --git a/source/configuration/global/options/rsconf1_debugprintmodulelist.rst b/source/configuration/global/options/rsconf1_debugprintmodulelist.rst new file mode 100644 index 0000000..8ddfa79 --- /dev/null +++ b/source/configuration/global/options/rsconf1_debugprintmodulelist.rst @@ -0,0 +1,12 @@ +$DebugPrintModuleList +--------------------- + +**Type:** global configuration parameter + +**Default:** on + +**Description:** + +Specifies whether or not the module list should be written to the debug +log. Possible values: on/off. Default is on. Does not affect operation +if debugging is disabled. diff --git a/source/configuration/global/options/rsconf1_debugprinttemplatelist.rst b/source/configuration/global/options/rsconf1_debugprinttemplatelist.rst new file mode 100644 index 0000000..d67a05e --- /dev/null +++ b/source/configuration/global/options/rsconf1_debugprinttemplatelist.rst @@ -0,0 +1,12 @@ +$DebugPrintTemplateList +----------------------- + +**Type:** global configuration parameter + +**Default:** on + +**Description:** + +Specifies whether or not the template list should be written to the +debug log. Possible values: on/off. Default is on. Does not affect +operation if debugging is disabled.. diff --git a/source/configuration/global/options/rsconf1_failonchownfailure.rst b/source/configuration/global/options/rsconf1_failonchownfailure.rst new file mode 100644 index 0000000..01d6c5a --- /dev/null +++ b/source/configuration/global/options/rsconf1_failonchownfailure.rst @@ -0,0 +1,21 @@ +$FailOnChownFailure +------------------- + +**Type:** global configuration parameter + +**Default:** on + +**Description:** + +This option modifies behaviour of dynaFile creation. If different owners +or groups are specified for new files or directories and rsyslogd fails +to set these new owners or groups, it will log an error and NOT write to +the file in question if that option is set to "on". If it is set to +"off", the error will be ignored and processing continues. Keep in mind, +that the files in this case may be (in)accessible by people who should +not have permission. The default is "on". + +**Sample:** + +``$FailOnChownFailure off`` + diff --git a/source/configuration/global/options/rsconf1_generateconfiggraph.rst b/source/configuration/global/options/rsconf1_generateconfiggraph.rst new file mode 100644 index 0000000..4c70832 --- /dev/null +++ b/source/configuration/global/options/rsconf1_generateconfiggraph.rst @@ -0,0 +1,148 @@ +$GenerateConfigGraph +-------------------- + +**Type:** global configuration parameter + +**Default:** + +**Available Since:** 4.3.1 **CURRENTLY NOT AVAILABLE** + +**Description:** + +**This parameter is currently not supported. We had to disable it when +we improved the rule engine. It is considerable effort to re-enable it. +On the other hand, we are about to add a new config system, which will +make yet another config graph method necessary. As such we have decided +to currently disable this functionality and re-introduce it when the new +config system has been instantiated.** + +This parameter permits to create (hopefully) good-looking visualizations +of rsyslogd's configuration. It does not affect rsyslog operation. If +the parameter is specified multiple times, all but the last are ignored. +If it is specified, a graph is created. This happens both during a +regular startup as well a config check run. It is recommended to include +this parameter only for documentation purposes and remove it from a +production configuration. + +The graph is not drawn by rsyslog itself. Instead, it uses the great +open source tool `Graphviz <http://www.graphviz.org>`_ to do the actual +drawing. This has at least two advantages: + +- the graph drawing support code in rsyslog is extremely slim and + without overhead +- the user may change or further annotate the generated file, thus + potentially improving his documentation + +The drawback, of course, is that you need to run Graphviz once you have +generated the control file with rsyslog. Fortunately, the process to do +so is rather easy: + +#. add "$GenerateConfigGraph /path/to/file.dot" to rsyslog.conf (from + now on, I will call the file just file.dot). Optionally, add + "$ActionName" statement **in front of** those actions that you like + to use friendly names with. If you do this, keep the names short. +#. run rsyslog at least once (either in regular or configuration check + mode) +#. remember to remove the $GenerateConfigGraph parameter when you no + longer need it (or comment it out) +#. change your working directory to where you place the dot file +#. if you would like to edit the rsyslog-generated file, now is the time + to do so +#. do "dot -Tpng file.dot > file.png" +#. remember that you can use "convert -resize 50% file.png resized.png" + if dot's output is too large (likely) or too small. Resizing can be + especially useful if you intend to get a rough overview over your + configuration. + +After completing these steps, you should have a nice graph of your +configuration. Details are missing, but that is exactly the point. At +the start of the graph is always (at least in this version, could be +improved) a node called "inputs" in a triple hexagon shape. This +represents all inputs active in the system (assuming you have defined +some, what the current version does not check). Next comes the main +queue. It is given in a hexagon shape. That shape indicates that a queue +is present and used to de-couple the inbound from the outbound part of +the graph. In technical terms, here is a threading boundary. Action with +"real" queues (other than in direct mode) also utilize this shape. For +actions, notice that a "hexagon action" creates a deep copy of the +message. As such, a "discard hexagon action" actually does nothing, +because it duplicates the message and then discards **the duplicate**. +At the end of the diagram, you always see a "discard" action. This +indicates that rsyslog discards messages which have been run through all +available rules. + +Edges are labeled with information about when they are taken. For +filters, the type of filter, but not any specifics, are given. It is +also indicated if no filter is applied in the configuration file (by +using a "\*.\*" selector). Edges without labels are unconditionally +taken. The actions themselves are labeled with the name of the output +module that handles them. If provided, the name given via "ActionName" +is used instead. No further details are provided. + +If there is anything in red, this should draw your attention. In this +case, rsyslogd has detected something that does not look quite right. A +typical example is a discard action which is followed by some other +actions in an action unit. Even though something may be red, it can be +valid - rsyslogd's graph generator does not yet check each and every +specialty, so the configuration may just cover a very uncommon case. + +Now let's look at some examples. The graph below was generated on a +fairly standard Fedora rsyslog.conf file. It had only the usually +commented-out last forwarding action activated: + +.. figure:: rsyslog_confgraph_std.png + :align: center + :alt: rsyslog configuration graph for a default fedora rsyslog.conf + + rsyslog configuration graph for a default fedora rsyslog.conf + +This is the typical structure for a simple rsyslog configuration. There +are a couple of actions, each guarded by a filter. Messages run from top +to bottom and control branches whenever a filter evaluates to true. As +there is no discard action, all messages will run through all filters +and discarded in the system default discard action right after all +configured actions. + +A more complex example can be seen in the next graph. This is a +configuration I created for testing the graph-creation features, so it +contains a little bit of everything. However, real-world configurations +can look quite complex, too (and I wouldn't say this one is very +complex): + +.. figure:: rsyslog_confgraph_complex.png + :align: center + :alt: + +Here, we have a user-defined discard action. You can immediately see +this because processing branches after the first "builtin-file" action. +Those messages where the filter evaluates to true for will never run +through the left-hand action branch. However, there is also a +configuration error present: there are two more actions (now shown red) +after the discard action. As the message is discarded, these will never +be executed. Note that the discard branch contains no further filters. +This is because these actions are all part of the same action unit, +which is guarded only by an entry filter. The same is present a bit +further down at the node labeled "write\_system\_log\_2". This note has +one more special feature, that is label was set via "ActionName", thus +is does not have standard form (the same happened to the node named +"Forward" right at the top of the diagram. Inside this diagram, the +"Forward" node is executed asynchronously on its own queue. All others +are executed synchronously. + +Configuration graphs are useful for documenting a setup, but are also a +great `troubleshooting <troubleshoot.html>`_ resource. It is important +to remember that **these graphs are generated from rsyslogd's in-memory +action processing structures**. You can not get closer to understanding +on how rsyslog interpreted its configuration files. So if the graph does +not look what you intended to do, there is probably something wrong in +rsyslog.conf. + +If something is not working as expected, but you do not spot the error +immediately, I recommend to generate a graph and zoom it so that you see +all of it in one great picture. You may not be able to read anything, +but the structure should look good to you and so you can zoom into those +areas that draw your attention. + +**Sample:** + +``$DirOwner /path/to/graphfile-file.dot`` diff --git a/source/configuration/global/options/rsconf1_includeconfig.rst b/source/configuration/global/options/rsconf1_includeconfig.rst new file mode 100644 index 0000000..a177324 --- /dev/null +++ b/source/configuration/global/options/rsconf1_includeconfig.rst @@ -0,0 +1,74 @@ +$IncludeConfig +-------------- + +.. warning:: + + This legacy directive has been superseeded by the rsyslog + :doc:`include() <../../../rainerscript/include>` + configuration object. + While it is save to use the legacy statement, we highly + recommend to use it's modern counterpart. Among others, + the `include()` object provides enhanced functionality. + +**Type:** global configuration parameter + +**Default:** + +**Description:** + +This parameter allows to include other files into the main configuration +file. As soon as an IncludeConfig parameter is found, the contents of +the new file is processed. IncludeConfigs can be nested. Please note +that from a logical point of view the files are merged. Thus, if the +include modifies some parameters (e.g. $DynaFileCacheSize), these new +parameters are in place for the "calling" configuration file when the +include is completed. To avoid any side effects, do a +$ResetConfigVariables after the $IncludeConfig. It may also be a good +idea to do a $ResetConfigVariables right at the start of the include, so +that the module knows exactly what it does. Of course, one might +specifically NOT do this to inherit parameters from the main file. As +always, use it as it best fits... + +Note: if multiple files are included, they are processed in ascending +sort order of the file name. We use the "glob()" C library function +for obtaining the sorted list. On most platforms, especially Linux, +this means the sort order is the same as for bash. + +If all regular files in the /etc/rsyslog.d directory are included, then +files starting with "." are ignored - so you can use them to place +comments into the dir (e.g. "/etc/rsyslog.d/.mycomment" will be +ignored). `Michael Biebl had the idea to this +functionality <http://sourceforge.net/tracker/index.php?func=detail&aid=1764088&group_id=123448&atid=696555>`_. +Let me quote him: + + Say you can add an option + ``$IncludeConfig /etc/rsyslog.d/`` + (which probably would make a good default) + to ``/etc/rsyslog.conf``, which would then merge and include all + ``*.conf`` files + in ``/etc/rsyslog.d/``. + This way, a distribution can modify its packages easily to drop a + simple + config file into this directory upon installation. + As an example, the network-manager package could install a simple + config + file ``/etc/rsyslog.d/network-manager.conf`` which would contain. + ``:programname, contains, "NetworkManager" -/var/log/NetworkManager.log`` + Upon uninstallation, the file could be easily removed again. This + approach would be much cleaner and less error prone, than having to munge + around with the ``/etc/rsyslog.conf`` file directly. + +**Sample:** + +``$IncludeConfig /etc/some-included-file.conf`` + +Directories can also be included. To do so, the name must end on a +slash: + +``$IncludeConfig /etc/rsyslog.d/`` + +**And finally, only specific files matching a wildcard my be included +from a directory:** + +``$IncludeConfig /etc/rsyslog.d/*.conf`` + diff --git a/source/configuration/global/options/rsconf1_mainmsgqueuesize.rst b/source/configuration/global/options/rsconf1_mainmsgqueuesize.rst new file mode 100644 index 0000000..8b74b59 --- /dev/null +++ b/source/configuration/global/options/rsconf1_mainmsgqueuesize.rst @@ -0,0 +1,34 @@ +$MainMsgQueueSize +----------------- + +**Type:** global configuration parameter + +**Default:** 50000 (v8.30.0) - may change + +**Description:** + +This allows to specify the maximum size of the message queue. This +parameter is only available when rsyslogd has been compiled with +multithreading support. In this mode, receiver and output modules are +de-coupled via an in-memory queue. This queue buffers messages when the +output modules are not capable to process them as fast as they are +received. Once the queue size is exhausted, messages will be dropped. +The slower the output (e.g. MySQL), the larger the queue should be. +Buffer space for the actual queue entries is allocated on an as-needed +basis. Please keep in mind that a very large queue may exhaust available +system memory and swap space. Keep this in mind when configuring the max +size. The actual size of a message depends largely on its content and +the originator. As a rule of thumb, typically messages should not take +up more then roughly 1k (this is the memory structure, not what you see +in a network dump!). For typical linux messages, 512 bytes should be a +good bet. Please also note that there is a minimal amount of memory +taken for each queue entry, no matter if it is used or not. This is one +pointer value, so on 32bit systems, it should typically be 4 bytes and +on 64bit systems it should typically be 8 bytes. For example, the +default queue size of 10,000 entries needs roughly 40k fixed overhead on +a 32 bit system. + +**Sample:** + +``$MainMsgQueueSize 100000 # 100,000 may be a value to handle burst traffic`` + diff --git a/source/configuration/global/options/rsconf1_maxopenfiles.rst b/source/configuration/global/options/rsconf1_maxopenfiles.rst new file mode 100644 index 0000000..89e4021 --- /dev/null +++ b/source/configuration/global/options/rsconf1_maxopenfiles.rst @@ -0,0 +1,36 @@ +$MaxOpenFiles +------------- + +**Available Since:** 4.3.0 + +**Type:** global configuration parameter + +**Default:** *operating system default* + +**Description:** + +Set the maximum number of files that the rsyslog process can have open +at any given time. Note that this includes open tcp sockets, so this +setting is the upper limit for the number of open TCP connections as +well. If you expect a large number of concurrent connections, it is +suggested that the number is set to the max number connected plus 1000. +Please note that each dynafile also requires up to 100 open file +handles. + +The setting is similar to running "ulimit -n number-of-files". + +Please note that depending on permissions and operating system +configuration, the setrlimit() request issued by rsyslog may fail, in +which case the previous limit is kept in effect. Rsyslog will emit a +warning message in this case. + +**Sample:** + +``$MaxOpenFiles 2000`` + +**Bugs:** + +For some reason, this settings seems not to work on all platforms. If +you experience problems, please let us know so that we can (hopefully) +narrow down the issue. + diff --git a/source/configuration/global/options/rsconf1_moddir.rst b/source/configuration/global/options/rsconf1_moddir.rst new file mode 100644 index 0000000..c1fa503 --- /dev/null +++ b/source/configuration/global/options/rsconf1_moddir.rst @@ -0,0 +1,20 @@ +$ModDir +------- + +**Type:** global configuration parameter + +**Default:** system default for user libraries, e.g. +/usr/local/lib/rsyslog/ + +**Description:** + +Provides the default directory in which loadable modules reside. This +may be used to specify an alternate location that is not based on the +system default. If the system default is used, there is no need to +specify this parameter. Please note that it is vitally important to end +the path name with a slash, else module loads will fail. + +**Sample:** + +``$ModDir /usr/rsyslog/libs/ # note the trailing slash!`` + diff --git a/source/configuration/global/options/rsconf1_modload.rst b/source/configuration/global/options/rsconf1_modload.rst new file mode 100644 index 0000000..e600784 --- /dev/null +++ b/source/configuration/global/options/rsconf1_modload.rst @@ -0,0 +1,30 @@ +$ModLoad +-------- + +**Type:** global configuration parameter + +**Default:** + +**Description:** + +Dynamically loads a plug-in into rsyslog's address space and activates +it. The plug-in must obey the rsyslog module API. Currently, only MySQL +and Postgres output modules are available as a plugins, but users may +create their own. A plug-in must be loaded BEFORE any configuration file +lines that reference it. + +Modules must be present in the system default destination for rsyslog +modules. You can also set the directory via the +`$ModDir <rsconf1_moddir.html>`_ parameter. + +If a full path name is specified, the module is loaded from that path. +The default module directory is ignored in that case. + +**Sample:** + +.. code-block:: none + + $ModLoad ommysql # load MySQL functionality + $ModLoad /rsyslog/modules/ompgsql.so # load the postgres module via absolute path + + diff --git a/source/configuration/global/options/rsconf1_resetconfigvariables.rst b/source/configuration/global/options/rsconf1_resetconfigvariables.rst new file mode 100644 index 0000000..8d27c65 --- /dev/null +++ b/source/configuration/global/options/rsconf1_resetconfigvariables.rst @@ -0,0 +1,19 @@ +$ResetConfigVariables +--------------------- + +**Type:** global configuration parameter + +**Default:** + +**Description:** + +Resets all configuration variables to their default value. Any settings +made will not be applied to configuration lines following the +$ResetConfigVariables. This is a good method to make sure no +side-effects exists from previous parameters. This parameter has no +parameters. + +**Sample:** + +``$ResetConfigVariables`` + diff --git a/source/configuration/global/options/rsconf1_umask.rst b/source/configuration/global/options/rsconf1_umask.rst new file mode 100644 index 0000000..57ead52 --- /dev/null +++ b/source/configuration/global/options/rsconf1_umask.rst @@ -0,0 +1,25 @@ +$UMASK +------ + +**Type:** global configuration parameter + +**Default:** + +**Description:** + +The $umask parameter allows to specify the rsyslogd processes' umask. If +not specified, the system-provided default is used. The value given must +always be a 4-digit octal number, with the initial digit being zero. + +If $umask is specified multiple times in the configuration file, results +may be somewhat unpredictable. It is recommended to specify it only +once. + +**Sample:** + +``$umask 0000`` + +This sample removes all restrictions. + +[`rsyslog site <http://www.rsyslog.com/>`_\ ] + diff --git a/source/configuration/global/options/rsyslog_confgraph_complex.png b/source/configuration/global/options/rsyslog_confgraph_complex.png Binary files differnew file mode 100644 index 0000000..21c04c5 --- /dev/null +++ b/source/configuration/global/options/rsyslog_confgraph_complex.png diff --git a/source/configuration/global/options/rsyslog_confgraph_std.png b/source/configuration/global/options/rsyslog_confgraph_std.png Binary files differnew file mode 100644 index 0000000..655a7f8 --- /dev/null +++ b/source/configuration/global/options/rsyslog_confgraph_std.png diff --git a/source/configuration/index.rst b/source/configuration/index.rst new file mode 100644 index 0000000..d1055b4 --- /dev/null +++ b/source/configuration/index.rst @@ -0,0 +1,62 @@ +Configuration +============= + +**Rsyslog Configuration Reference Manual Introduction** + +This document serves as a detailed guide to rsyslog configuration, offering extensive information on the setup and management of system logging using +`rsyslog <https://www.rsyslog.com>`_ +It covers various aspects of rsyslog configuration, including constructs, statements, and key concepts, designed to assist users in customizing their logging infrastructure according to specific needs. + +The primary configuration file for rsyslog, located at `/etc/rsyslog.conf`, acts as the central point for establishing logging rules. This file is used to define input modules, filters, actions, and global directives, facilitating the processes of log collection, filtering, routing, and formatting. + +Please note that this documentation is currently in the process of being refined to improve its clarity, structure, and accessibility. We value your patience and understanding during this phase and are committed to delivering a comprehensive and easy-to-navigate guide to rsyslog. + +For further exploration of rsyslog's configuration intricacies, please refer to the links provided below. This manual is designed to be a valuable resource for both experienced system administrators and those new to the field, aiming to fully leverage the capabilities of rsyslog. + +Note that **configurations can be built interactively** via the online +`rsyslog configuration builder <http://www.rsyslog.com/rsyslog-configuration-builder/>`_ tool. + +.. toctree:: + :maxdepth: 2 + + conf_formats + converting_to_new_format + sysklogd_format + basic_structure + templates + properties + property_replacer + filters + ../rainerscript/index + actions + input + parser + timezone + examples + index_directives + rsyslog_statistic_counter + modules/index + output_channels + droppriv + ipv6 + cryprov_gcry + dyn_stats + lookup_tables + percentile_stats + +`Configuration file examples can be found in the rsyslog +wiki <http://wiki.rsyslog.com/index.php/Configuration_Samples>`_. Also +keep the `rsyslog config +snippets <http://www.rsyslog.com/config-snippets/>`_ on your mind. These +are ready-to-use real building blocks for rsyslog configuration. + +There is also one sample file provided together with the documentation +set. If you do not like to read, be sure to have at least a quick look +at :download:`rsyslog-example.conf <rsyslog-example.conf>`. + +While rsyslogd contains enhancements over standard syslogd, efforts have +been made to keep the configuration file as compatible as possible. +While, for obvious reasons, :doc:`enhanced features <../features>` require +a different config file syntax, rsyslogd should be able to work with a +standard syslog.conf file. This is especially useful while you are +migrating from syslogd to rsyslogd. diff --git a/source/configuration/index_directives.rst b/source/configuration/index_directives.rst new file mode 100644 index 0000000..fb93a19 --- /dev/null +++ b/source/configuration/index_directives.rst @@ -0,0 +1,21 @@ +Legacy Configuration Directives +=============================== +All legacy configuration directives need to be specified on a line by their own +and must start with a dollar-sign. + +Note that legacy configuration directives that set object options (e.g. for +inputs or actions) only affect those objects that are defined via legacy +constructs. Objects defined via new-style RainerScript objects (e.g. +action(), input()) are **not** affected by legacy directives. The reason +is that otherwise we would again have the ability to mess up a configuration +file with hard to understand constructs. This is avoided by not permitting +to mix and match the way object values are set. + +.. toctree:: + :maxdepth: 2 + + config_param_types + global/index + input_directives/index + action/index + ruleset/index diff --git a/source/configuration/input.rst b/source/configuration/input.rst new file mode 100644 index 0000000..3e214d1 --- /dev/null +++ b/source/configuration/input.rst @@ -0,0 +1,32 @@ +Input +===== + +.. index:: ! input +.. _cfgobj_input: + +The ``input`` object, as its name suggests, describes message input sources. +Without input, no processing happens at all, because no messages enter the +rsyslog system. +Inputs are implemented via :doc:`input modules <modules/idx_input>`. + +The input object has different parameters: + +- those that apply to all input and are generally available for + all inputs. These are documented below. +- input-specific parameters. These are specific to a certain type of + input. They are documented by the :doc:`input module <modules/idx_input>` + in question. + +General Input Parameters +------------------------ + +Note: parameter names are case-insensitive. + +.. function:: type <type-string> + + *Mandatory* + + The ``<type-string>`` is a string identifying the input module as given + it each module's documentation. For example, the + :doc:`UDP syslog input <modules/imudp>` is named "imudp". + diff --git a/source/configuration/input_directives/index.rst b/source/configuration/input_directives/index.rst new file mode 100644 index 0000000..796b73e --- /dev/null +++ b/source/configuration/input_directives/index.rst @@ -0,0 +1,28 @@ +Legacy Directives affecting Input Modules +========================================= + +Legacy Directives affecting multiple Input Modules +-------------------------------------------------- +While these directives only affect input modules, they are global in +the sense that they cannot be overwritten for specific input +instances. So they apply globally for all inputs that support these +directives. + +.. toctree:: + :glob: + + rsconf1_allowedsender + rsconf1_dropmsgswithmaliciousdnsptrrecords + rsconf1_controlcharacterescapeprefix + rsconf1_droptrailinglfonreception + rsconf1_escape8bitcharsonreceive + rsconf1_escapecontrolcharactersonreceive + +immark-specific Directives +-------------------------- + +.. toctree:: + :glob: + + rsconf1_markmessageperiod + diff --git a/source/configuration/input_directives/rsconf1_allowedsender.rst b/source/configuration/input_directives/rsconf1_allowedsender.rst new file mode 100644 index 0000000..eb30054 --- /dev/null +++ b/source/configuration/input_directives/rsconf1_allowedsender.rst @@ -0,0 +1,71 @@ +$AllowedSender +-------------- + +**Type:** input configuration parameter + +**Default:** all allowed + +**Description:** + +*Note:* this feature is supported for backward-compatibility, only. +The rsyslog team recommends to use proper firewalling instead of +this feature. + +Allowed sender lists can be used to specify which remote systems are +allowed to send syslog messages to rsyslogd. With them, further hurdles +can be placed between an attacker and rsyslogd. If a message from a +system not in the allowed sender list is received, that message is +discarded. A diagnostic message is logged, so that the fact is recorded +(this message can be turned off with the "-w" rsyslogd command line +option). + +Allowed sender lists can be defined for UDP and TCP senders separately. +There can be as many allowed senders as needed. The syntax to specify +them is: + +:: + + $AllowedSender <type>, ip[/bits], ip[/bits] + +"$AllowedSender" is the parameter - it must be written exactly as shown +and the $ must start at the first column of the line. "<type>" is either "UDP" +or "TCP" (or "GSS", if this is enabled during compilation). +It must immediately be followed by the comma, else you will +receive an error message. "ip[/bits]" is a machine or network ip address +as in "192.0.2.0/24" or "127.0.0.1". If the "/bits" part is omitted, a +single host is assumed (32 bits or mask 255.255.255.255). "/0" is not +allowed, because that would match any sending system. If you intend to +do that, just remove all $AllowedSender parameters. If more than 32 bits +are requested with IPv4, they are adjusted to 32. For IPv6, the limit is +128 for obvious reasons. Hostnames, with and without wildcards, may also +be provided. If so, the result of revers DNS resolution is used for +filtering. Multiple allowed senders can be specified in a +comma-delimited list. Also, multiple $AllowedSender lines can be given. +They are all combined into one UDP and one TCP list. Performance-wise, +it is good to specify those allowed senders with high traffic volume +before those with lower volume. As soon as a match is found, no further +evaluation is necessary and so you can save CPU cycles. + +Rsyslogd handles allowed sender detection very early in the code, nearly +as the first action after receiving a message. This keeps the access to +potential vulnerable code in rsyslog at a minimum. However, it is still +a good idea to impose allowed sender limitations via firewalling. + +**WARNING:** by UDP design, rsyslogd can not identify a spoofed sender +address in UDP syslog packets. As such, a malicious person could spoof +the address of an allowed sender, send such packets to rsyslogd and +rsyslogd would accept them as being from the faked sender. To prevent +this, use syslog via TCP exclusively. If you need to use UDP-based +syslog, make sure that you do proper egress and ingress filtering at the +firewall and router level. + +Rsyslog also detects some kind of malicious reverse DNS entries. In any +case, using DNS names adds an extra layer of vulnerability. We recommend +to stick with hard-coded IP addresses wherever possible. + +**Sample:** + +:: + + $AllowedSender UDP, 127.0.0.1, 192.0.2.0/24, [::1]/128, *.example.net, somehost.example.com + diff --git a/source/configuration/input_directives/rsconf1_controlcharacterescapeprefix.rst b/source/configuration/input_directives/rsconf1_controlcharacterescapeprefix.rst new file mode 100644 index 0000000..06ddb12 --- /dev/null +++ b/source/configuration/input_directives/rsconf1_controlcharacterescapeprefix.rst @@ -0,0 +1,24 @@ +$ControlCharacterEscapePrefix +----------------------------- + +**Type:** global configuration parameter + +**Default:** \\ + +**Description:** + +This option specifies the prefix character to be used for control +character escaping (see option $EscapeControlCharactersOnReceive). By +default, it is '\\', which is backwards-compatible with sysklogd. Change +it to '#' in order to be compliant to the value that is somewhat +suggested by Internet-Draft syslog-protocol. + +**IMPORTANT**: do not use the ' character. This is reserved and will +most probably be used in the future as a character delimiter. For the +same reason, the syntax of this parameter will probably change in future +releases. + +**Sample:** + +``$EscapeControlCharactersOnReceive # # as of syslog-protocol`` + diff --git a/source/configuration/input_directives/rsconf1_dropmsgswithmaliciousdnsptrrecords.rst b/source/configuration/input_directives/rsconf1_dropmsgswithmaliciousdnsptrrecords.rst new file mode 100644 index 0000000..81562ef --- /dev/null +++ b/source/configuration/input_directives/rsconf1_dropmsgswithmaliciousdnsptrrecords.rst @@ -0,0 +1,22 @@ +$DropMsgsWithMaliciousDnsPTRRecords +----------------------------------- + +**Type:** global configuration parameter + +**Default:** off + +**Description:** + +Rsyslog contains code to detect malicious DNS PTR records (reverse name +resolution). An attacker might use specially-crafted DNS entries to make +you think that a message might have originated on another IP address. +Rsyslog can detect those cases. It will log an error message in any +case. If this option here is set to "on", the malicious message will be +completely dropped from your logs. If the option is set to "off", the +message will be logged, but the original IP will be used instead of the +DNS name. + +**Sample:** + +``$DropMsgsWithMaliciousDnsPTRRecords on`` + diff --git a/source/configuration/input_directives/rsconf1_droptrailinglfonreception.rst b/source/configuration/input_directives/rsconf1_droptrailinglfonreception.rst new file mode 100644 index 0000000..1f69195 --- /dev/null +++ b/source/configuration/input_directives/rsconf1_droptrailinglfonreception.rst @@ -0,0 +1,21 @@ +$DropTrailingLFOnReception +-------------------------- + +**Type:** global configuration parameter + +**Default:** on + +**Description:** + +Syslog messages frequently have the line feed character (LF) as the last +character of the message. In almost all cases, this LF should not really +become part of the message. However, recent IETF syslog standardization +recommends against modifying syslog messages (e.g. to keep digital +signatures valid). This option allows to specify if trailing LFs should +be dropped or not. The default is to drop them, which is consistent with +what sysklogd does. + +**Sample:** + +``$DropTrailingLFOnReception on`` + diff --git a/source/configuration/input_directives/rsconf1_escape8bitcharsonreceive.rst b/source/configuration/input_directives/rsconf1_escape8bitcharsonreceive.rst new file mode 100644 index 0000000..7377f65 --- /dev/null +++ b/source/configuration/input_directives/rsconf1_escape8bitcharsonreceive.rst @@ -0,0 +1,39 @@ +$Escape8BitCharactersOnReceive +------------------------------ + +**Type:** global configuration parameter + +**Default:** off + +**Available Since:** 5.5.2 + +**Description:** + +This parameter instructs rsyslogd to replace non US-ASCII characters +(those that have the 8th bit set) during reception of the message. This +may be useful for some systems. Please note that this escaping breaks +Unicode and many other encodings. Most importantly, it can be assumed +that Asian and European characters will be rendered hardly readable by +this settings. However, it may still be useful when the logs themselves +are primarily in English and only occasionally contain local script. If +this option is turned on, all control-characters are converted to a +3-digit octal number and be prefixed with the +$ControlCharacterEscapePrefix character (being '#' by default). + +**Warning:** + +- turning on this option most probably destroys non-western character + sets (like Japanese, Chinese and Korean) as well as European + character sets. +- turning on this option destroys digital signatures if such exists + inside the message +- if turned on, the drop-cc, space-cc and escape-cc `property + replacer <property_replacer.html>`_ options do not work as expected + because control characters are already removed upon message + reception. If you intend to use these property replacer options, you + must turn off $Escape8BitCharactersOnReceive. + +**Sample:** + +``$Escape8BitCharactersOnReceive on`` + diff --git a/source/configuration/input_directives/rsconf1_escapecontrolcharactersonreceive.rst b/source/configuration/input_directives/rsconf1_escapecontrolcharactersonreceive.rst new file mode 100644 index 0000000..05cc623 --- /dev/null +++ b/source/configuration/input_directives/rsconf1_escapecontrolcharactersonreceive.rst @@ -0,0 +1,34 @@ +$EscapeControlCharactersOnReceive +--------------------------------- + +**Type:** global configuration parameter + +**Default:** on + +**Description:** + +This parameter instructs rsyslogd to replace control characters during +reception of the message. The intent is to provide a way to stop +non-printable messages from entering the syslog system as whole. If this +option is turned on, all control-characters are converted to a 3-digit +octal number and be prefixed with the $ControlCharacterEscapePrefix +character (being '\\' by default). For example, if the BEL character +(ctrl-g) is included in the message, it would be converted to "\\007". +To be compatible to sysklogd, this option must be turned on. + +**Warning:** + +- turning on this option most probably destroys non-western character + sets (like Japanese, Chinese and Korean) +- turning on this option destroys digital signatures if such exists + inside the message +- if turned on, the drop-cc, space-cc and escape-cc `property + replacer <property_replacer.html>`_ options do not work as expected + because control characters are already removed upon message + reception. If you intend to use these property replacer options, you + must turn off $EscapeControlCharactersOnReceive. + +**Sample:** + +``$EscapeControlCharactersOnReceive on`` + diff --git a/source/configuration/input_directives/rsconf1_markmessageperiod.rst b/source/configuration/input_directives/rsconf1_markmessageperiod.rst new file mode 100644 index 0000000..074cc95 --- /dev/null +++ b/source/configuration/input_directives/rsconf1_markmessageperiod.rst @@ -0,0 +1,26 @@ +$MarkMessagePeriod +------------------ + +**Type:** specific to immark input module + +**Default:** 1200 (20 minutes) + +**Description:** + +This specifies when mark messages are to be written to output modules. +The time specified is in seconds. Specifying 0 is possible and disables +mark messages. In that case, however, it is more efficient to NOT load +the immark input module. + +So far, there is only one mark message process and any subsequent +$MarkMessagePeriod overwrites the previous. + +**This parameter is only available after the immark input module has +been loaded.** + +**Sample:** + +``$MarkMessagePeriod 600 # mark messages appear every 10 Minutes`` + +**Available since:** rsyslog 3.0.0 + diff --git a/source/configuration/ipv6.rst b/source/configuration/ipv6.rst new file mode 100644 index 0000000..aeec21d --- /dev/null +++ b/source/configuration/ipv6.rst @@ -0,0 +1,47 @@ +Notes on IPv6 Handling in Rsyslog +================================= + +**Rsyslog fully\* supports sending and receiving syslog messages via +both IPv4 and IPv6.** IPv6 is natively supported for both UDP and TCP. +However, there are some options that control handling of IPv6 +operations. I thought it is is a good idea to elaborate a little about +them, so that you can probably find your way somewhat easier. + +First of all, you can restrict rsyslog to using IPv4 or IPv6 addresses +only by specifying the -4 or -6 command line option (now guess which one +does what...). If you do not provide any command line option, rsyslog +uses IPv4 and IPv6 addresses concurrently. In practice, that means the +listener binds to both addresses (provided they are configured). When +sending syslog messages, rsyslog uses IPv4 addresses when the receiver +can be reached via IPv4 and IPv6 addresses if it can be reached via +IPv6. If it can be reached on either IPv4 and v6, rsyslog leaves the +choice to the socket layer. The important point to know is that it uses +whatever connectivity is available to reach the destination. + +**There is one subtle difference between UDP and TCP.** With the new +IPv4/v6 ignorant code, rsyslog has potentially different ways to reach +destinations. The socket layer returns all of these paths in a sorted +array. For TCP, rsyslog loops through this array until a successful TCP +connect can be made. If that happens, the other addresses are ignored +and messages are sent via the successfully-connected socket. + +For UDP, there is no such definite success indicator. Sure, the socket +layer may detect some errors, but it may not notice other errors (due to +the unreliable nature of UDP). By default, the UDP sender also tries one +entry after the other in the sorted array of destination addresses. When +a send fails, the next address is tried. When the send function finally +succeeds, rsyslogd assumes the UDP packet has reached its final +destination. However, if rsyslogd is started with the "-A" (capital A!) +was given on the command line, rsyslogd will continue to send messages +until the end of the destination address array is reached. This may +result in duplicate messages, but it also provides some additional +reliability in case a message could not be received. You need to be sure +about the implications before applying this option. In general, it is +NOT recommended to use the -A option. + +**\***\ rsyslog does not support RFC 3195 over IPv6. The reason is that +the RFC 3195 library, `liblogging <http://www.liblogging.org/>`_, +supports IPv4, only. Currently, there are no plans to update either +rsyslog to another RFC 3195 stack or update liblogging. There is simply +no demand for 3195 solutions. + diff --git a/source/configuration/lookup_tables.rst b/source/configuration/lookup_tables.rst new file mode 100644 index 0000000..b673022 --- /dev/null +++ b/source/configuration/lookup_tables.rst @@ -0,0 +1,251 @@ +Lookup Tables +============= + +Lookup tables are a powerful construct to obtain *class* information based +on message content. It works on top of a data-file which maps key (to be looked +up) to value (the result of lookup). + +The idea is to use a message properties (or derivatives of it) as an index +into a table which then returns another value. For example, $fromhost-ip +could be used as an index, with the table value representing the type of +server or the department or remote office it is located in. + +This can be emulated using if and else-if stack, but implementing it as a +dedicated component allows ``lookup`` to be made fast. + +The lookup tables itself exists in a separate data file (one per +table). This file is loaded on Rsyslog startup and when a reload is requested. + +There are different types of lookup tables (identified by "type" field in json data-file). + +Types +^^^^^ + +string +------ + +The key to be looked up is an arbitrary string. + +**Match criterion**: The key must be exactly equal to index from one of the entries. + +array +----- + +The value to be looked up is an integer number from a consecutive set. +The set does not need to start at zero or one, but there must be no number missing. +So, for example ``5,6,7,8,9`` would be a valid set of index values, while ``1,2,4,5`` would +not be (due to missing ``3``). + +**Match criterion**: Looked-up number(key) must be exactly equal to index from one of the entries. + +sparseArray +----------- + +The value to be looked up is an integer value, but there may be gaps inside the +set of values (usually there are large gaps). A typical use case would be the +matching of IPv4 address information. + +**Match criterion**: A match happens on the first index that is less than or equal to the looked-up key. + +Note that index integer numbers are represented by unsigned 32 bits. + + +Lookup Table File Format +^^^^^^^^^^^^^^^^^^^^^^^^ + +Lookup table files contain a single JSON object. This object consists of a header and a table part. + +**Header** + +The header is the top-level json. +It has parameters "version", "nomatch", and "type". + +Parameters: + **version** <number, default: 1> : Version of table-definition format (so improvements in the future can be managed in a backward compatible way). + + **nomatch** <string literal, default: ""> : Value to be returned for a lookup when match fails. + + **type** <*string*, *array* or *sparseArray*, default: *string*> : Type of lookup-table (controls how matches are performed). + +**Table** + +This must be an array of elements, even if only a single value exists (for obvious reasons, +we do not expect this to occur often). Each array element must contain two fields "index" +and "value". + +This is a sample of how an ip-to-office mapping may look like: + +:: + + { "version" : 1, + "nomatch" : "unk", + "type" : "string", + "table" : [ + {"index" : "10.0.1.1", "value" : "A" }, + {"index" : "10.0.1.2", "value" : "A" }, + {"index" : "10.0.1.3", "value" : "A" }, + {"index" : "10.0.2.1", "value" : "B" }, + {"index" : "10.0.2.2", "value" : "B" }, + {"index" : "10.0.2.3", "value" : "B" }]} + + +Note: In the example above, if a different IP comes in, the value "unk" is returned thanks to the nomatch parameter in the first line. + +Lookup tables can be accessed via the ``lookup()`` built-in function. A common usage pattern is to set a local variable to the lookup result and later use that variable in templates. + + + +Lookup-table configuration +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Lookup-table configuration involves a **two part setup** (definition and usage(lookup)), with an optional third part, +which allows reloading table using internal trigger. + +lookup_table(name="<table>" file="</path/to/file>"...) (object) +--------------------------------------------------------------- + +**Defines** the table(identified by the table-name) and allows user to set some properties that control behavior of the table. + +:: + + lookup_table(name="msg_per_host") + +Parameters: + **name** <string literal, mandatory> : Name of the table. + + **file** <string literal, file path, mandatory> : Path to external json database file. + + **reloadOnHUP** <on|off, default: on> : Whether or not table should be reloaded when process receives HUP signal. + +A definition setting all the parameters looks like: + +:: + + lookup_table(name="host_bu" file="/var/lib/host_billing_unit_mapping.json" reloadOnHUP="on") + + +lookup("<table>", <expr>) (function) +------------------------------------ + +**Looks up** and returns the value that is associated with the given key (passed as <variable>) +in lookup table identified by table-name. If no match is found (according to table-type +matching-criteria specified above), the "nomatch" string is returned (or an empty string if it is not defined). + +Parameters: + **name** <string literal, mandatory> : Name of the table. + + **expr** <expression resulting in string or number according to lookup-table type, mandatory> : Key to be looked up. + +A ``lookup`` call looks like: + +:: + + set $.bu = lookup("host_bu", $hostname); + + if ($.bu == "unknown") then { + .... + } + +Some examples of different match/no-match scenarios: + +**string table**: + +:: + + { "nomatch" : "none", + "type" : "string", + "table":[ + {"index" : "foo", "value" : "bar" }, + {"index" : "baz", "value" : "quux" }]} + +Match/no-Match behaviour: + +====== ============== +key return +====== ============== +foo bar +baz quux +corge none +====== ============== + +**array table**: + +:: + + { "nomatch" : "nothing", + "type" : "array", + "table":[ + {"index" : 9, "value" : "foo" }, + {"index" : 10, "value" : "bar" }, + {"index" : 11, "value" : "baz" }]} + +Match/no-Match behaviour: + +====== ============== +key return +====== ============== +9 foo +11 baz +15 nothing +0 nothing +====== ============== + +**sparseArray table**: + +:: + + { "nomatch" : "no_num", + "type" : "sparseArray", + "table":[ + {"index" : "9", "value" : "foo" }, + {"index" : "11", "value" : "baz" }]} + +Match/no-Match behaviour: + +====== ============== +key return +====== ============== +8 no_num +9 foo +10 foo +11 baz +12 baz +100 baz +====== ============== + + +reload_lookup_table("<table>", "<stub value>") (statement) +---------------------------------------------------------- + +**Reloads** lookup table identified by given table name **asynchronously** (by internal trigger, as opposed to HUP). + +This statement isn't always useful. It needs to be used only when lookup-table-reload needs to be triggered in response to +a message. + +Messages will continue to be processed while table is asynchronously reloaded. + +Note: For performance reasons, message that triggers reload should be accepted only from a trusted source. + +Parameters: + **name** <string literal, mandatory> : Name of the table. + + **stub value** <string literal, optional> : Value to stub the table in-case reload-attempt fails. + +A ``reload_lookup_table`` invocation looks like: + +:: + + if ($.do_reload == "y") then { + reload_lookup_table("host_bu", "unknown") + } + + +Implementation Details +^^^^^^^^^^^^^^^^^^^^^^ + +The lookup table functionality is implemented via efficient algorithms. + +The string and sparseArray lookup have O(log(n)) time complexity, while array lookup is O(1). + +To preserve space and, more important, increase cache hit performance, equal data values are only stored once, +no matter how often a lookup index points to them. diff --git a/source/configuration/modules/gssapi.png b/source/configuration/modules/gssapi.png Binary files differnew file mode 100644 index 0000000..c82baa5 --- /dev/null +++ b/source/configuration/modules/gssapi.png diff --git a/source/configuration/modules/gssapi.rst b/source/configuration/modules/gssapi.rst new file mode 100644 index 0000000..157f5a3 --- /dev/null +++ b/source/configuration/modules/gssapi.rst @@ -0,0 +1,73 @@ +GSSAPI module support in rsyslog v3 +=================================== + +What is it good for. + +- client-serverauthentication +- Log messages encryption + +Requirements. + +- Kerberos infrastructure +- rsyslog, rsyslog-gssapi + +Configuration. + +Let's assume there are 3 machines in Kerberos Realm: + +- the first is running KDC (Kerberos Authentication Service and Key + Distribution Center), +- the second is a client sending its logs to the server, +- the third is receiver, gathering all logs. + +1. KDC: + +- Kerberos database must be properly set-up on KDC machine first. Use + kadmin/kadmin.local to do that. Two principals need to be add in our + case: + +#. sender@REALM.ORG + +- client must have ticket for principal sender +- REALM.ORG is kerberos Realm + +#. host/receiver.mydomain.com@REALM.ORG - service principal + +- Use ktadd to export service principal and transfer it to + /etc/krb5.keytab on receiver + +2. CLIENT: + +- set-up rsyslog, in /etc/rsyslog.conf +- $ModLoad omgssapi - load output gss module +- $GSSForwardServiceName otherThanHost - set the name of service + principal, "host" is the default one +- \*.\* :omgssapi:receiver.mydomain.com - action line, forward logs to + receiver +- kinit root - get the TGT ticket +- service rsyslog start + +3. SERVER: + +- set-up rsyslog, in /etc/rsyslog.conf + +- $ModLoad `imgssapi <imgssapi.html>`_ - load input gss module + +- $InputGSSServerServiceName otherThanHost - set the name of service + principal, "host" is the default one + +- $InputGSSServerPermitPlainTCP on - accept GSS and TCP connections + (not authenticated senders), off by default + +- $InputGSSServerRun 514 - run server on port + +- service rsyslog start + +The picture demonstrate how things work. + +.. figure:: gssapi.png + :align: center + :alt: rsyslog gssapi support + + rsyslog gssapi support + diff --git a/source/configuration/modules/idx_input.rst b/source/configuration/modules/idx_input.rst new file mode 100644 index 0000000..74f485c --- /dev/null +++ b/source/configuration/modules/idx_input.rst @@ -0,0 +1,13 @@ +Input Modules +------------- + +Input modules are used to gather messages from various sources. They +interface to message generators. They are generally defined via the +:doc:`input <../input>` configuration object. + +.. toctree:: + :glob: + :maxdepth: 1 + + im* + diff --git a/source/configuration/modules/idx_library.rst b/source/configuration/modules/idx_library.rst new file mode 100644 index 0000000..62452a1 --- /dev/null +++ b/source/configuration/modules/idx_library.rst @@ -0,0 +1,9 @@ +Library Modules +=============== + +Library modules provide dynamically loadable functionality for parts of +rsyslog, most often for other loadable modules. They can not be +user-configured and are loaded automatically by some components. They +are just mentioned so that error messages that point to library modules +can be understood. No module list is provided. + diff --git a/source/configuration/modules/idx_messagemod.rst b/source/configuration/modules/idx_messagemod.rst new file mode 100644 index 0000000..f29c6af --- /dev/null +++ b/source/configuration/modules/idx_messagemod.rst @@ -0,0 +1,15 @@ +Message Modification Modules +---------------------------- + +Message modification modules are used to change the content of messages +being processed. They can be implemented using either the output module +or the parser module interface. From the rsyslog core's point of view, +they actually are output or parser modules, it is their implementation +that makes them special. + +.. toctree:: + :maxdepth: 1 + :glob: + + mm* + diff --git a/source/configuration/modules/idx_output.rst b/source/configuration/modules/idx_output.rst new file mode 100644 index 0000000..0be3046 --- /dev/null +++ b/source/configuration/modules/idx_output.rst @@ -0,0 +1,16 @@ +Output Modules +-------------- + +Output modules process messages. With them, message formats can be +transformed and messages be transmitted to various different targets. +They are generally defined via :doc:`action <../actions>` configuration +objects. + +.. toctree:: + :glob: + :maxdepth: 1 + + om* + sigprov_gt + sigprov_ksi + sigprov_ksi12 diff --git a/source/configuration/modules/idx_parser.rst b/source/configuration/modules/idx_parser.rst new file mode 100644 index 0000000..220f2ca --- /dev/null +++ b/source/configuration/modules/idx_parser.rst @@ -0,0 +1,16 @@ +Parser Modules +-------------- + +Parser modules are used to parse message content, once the message has +been received. They can be used to process custom message formats or +invalidly formatted messages. For details, please see the :doc:`rsyslog +message parser documentation <../../concepts/messageparser>`. + +The current modules are currently provided as part of rsyslog: + +.. toctree:: + :glob: + :maxdepth: 1 + + pm* + diff --git a/source/configuration/modules/idx_stringgen.rst b/source/configuration/modules/idx_stringgen.rst new file mode 100644 index 0000000..221f0d4 --- /dev/null +++ b/source/configuration/modules/idx_stringgen.rst @@ -0,0 +1,43 @@ +String Generator Modules +======================== + +String generator modules are used, as the name implies, to generate +strings based on the message content. They are currently tightly coupled +with the template system. Their primary use is to speed up template +processing by providing a native C interface to template generation. +These modules exist since 5.5.6. To get an idea of the potential +speedup, the default file format, when generated by a string generator, +provides a roughly 5% speedup. For more complex strings, especially +those that include multiple regular expressions, the speedup may be +considerably higher. + +String generator modules are written to a quite simple interface. +However, a word of caution is due: they access the rsyslog message +object via a low-level interface. That interface is not guaranteed yet +to stay stable. So it may be necessary to modify string generator +modules if the interface changes. Obviously, we will not do that without +good reason, but it may happen. + +Rsyslog comes with a set of core, build-in string generators, which are +used to provide those default templates that we consider to be +time-critical: + +- smfile - the default rsyslog file format +- smfwd - the default rsyslog (network) forwarding format +- smtradfile - the traditional syslog file format +- smfwd - the traditional syslog (network) forwarding format + +Note that when you replace these defaults with some custom strings, you +will loose some performance (around 5%). For typical systems, this is +not really relevant. But for a high-performance systems, it may be very +relevant. To solve that issue, create a new string generator module for +your custom format, starting out from one of the default generators +provided. If you can not do this yourself, you may want to contact +`Adiscon <mailto:info%40adiscon.com>`_ as we offer custom development of +string generators at a very low price. + +Note that string generator modules can be dynamically loaded. However, +the default ones provided are so important that they are build right +into the executable. But this does not need to be done that way (and it +is straightforward to do it dynamic). + diff --git a/source/configuration/modules/im3195.rst b/source/configuration/modules/im3195.rst new file mode 100644 index 0000000..10a2d54 --- /dev/null +++ b/source/configuration/modules/im3195.rst @@ -0,0 +1,75 @@ +**************************** +im3195: RFC3195 Input Module +**************************** + +=========================== =========================================================================== +**Module Name:** **im3195** +**Author:** `Rainer Gerhards <https://rainer.gerhards.net/>`_ <rgerhards@adiscon.com> +=========================== =========================================================================== + + +Purpose +======= + +Receives syslog messages via RFC 3195. The RAW profile is fully +implemented and the COOKED profile is provided in an experimental state. +This module uses `liblogging <http://www.liblogging.org>`_ for the +actual protocol handling. + + +Configuration Parameters +======================== + +.. note:: + + Parameter names are case-insensitive. + + +Input Parameter +--------------- + +Input3195ListenPort +^^^^^^^^^^^^^^^^^^^ + +.. note:: + + Parameter is only available in Legacy Format. + + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "601", "no", "``$Input3195ListenPort``" + +The port on which imklog listens for RFC 3195 messages. The default +port is 601 (the IANA-assigned port) + + +Caveats/Known Bugs +================== + +Due to no demand at all for RFC3195, we have converted rfc3195d to this +input module, but we have NOT conducted any testing. Also, the module +does not yet properly handle the recovery case. If someone intends to +put this module into production, good testing should be conducted. It +also is a good idea to notify the rsyslog project that you intend to use +it in production. In this case, we'll probably give the module another +cleanup. We don't do this now because so far it looks just like a big +waste of time. + +Currently only a single listener can be defined. That one binds to all +interfaces. + +Example +======= + +The following sample accepts syslog messages via RFC 3195 on port 1601. + +.. code-block:: none + + $ModLoad im3195 + $Input3195ListenPort 1601 + + diff --git a/source/configuration/modules/imbatchreport.rst b/source/configuration/modules/imbatchreport.rst new file mode 100644 index 0000000..439eae4 --- /dev/null +++ b/source/configuration/modules/imbatchreport.rst @@ -0,0 +1,222 @@ +**************************************** +imbatchreport: Batch report input module +**************************************** + +================ ============================================================== +**Module Name:** **imbatchreport** +**Authors:** Jean-Philippe Hilaire <jean-philippe.hilaire@pmu.fr> & Philippe Duveau <philippe.duveau@free.fr> +================ ============================================================== + + +Purpose +======= + +This module allows rsyslog to manage batch reports. + +Batch are programs launched successively to process a large amount of +information. These programs are organized in stages with passing conditions. +The batch ends with a global execution summary. Each Batch produces a single +result file usually named with the name of the batch and its date of execution. + +Those files have sense only when they are complete in one log. When the file is +collected it becomes useless and, as a statefile, should be deleted or renamed. + +This module handle those characteristics : + +- reads the complete file, + +- extracts the structured data from the file (see managing structured data), + +- transmit the message to output module(s), + +- action is applied to the file to flag it as treated. Two different actions can be applied: delete or rename the file. + +If the file is too large to be handled in the message size defined by rsyslog, +the file is renamed as a "rejected file". See \$maxMessageSize + +**Managing structured data** + +As part of the batch summary, the structure data can be provided in the batch +report file as the last part of the file. + +The last non-space char has to be a closing brace ']' then all chars between +this char up to the closest opening brace '[' are computed as structured data. + +All the structured data has to be contained in the last 150 chars of the file. + +In general, structured data should contain the batch name (program) and the +start timestamp. Those two values can be extract to fill rsyslog message +attributes. + +Compile +======= + +To successfully compile imbatchreport module. + + ./configure --enable-imbatchreport ... + +Configuration Parameters +======================== + +Action Parameters +----------------- + +Reports +^^^^^^^ + +.. csv-table:: + :header: "type", "mandatory", "format", "default" + :widths: auto + :class: parameter-table + + "string", "yes", "glob definition", + +Glob definition used to identify reports to manage. + +Tag +^^^ + +.. csv-table:: + :header: "type", "mandatory", "format", "default" + :widths: auto + :class: parameter-table + + "string", "yes", ,"none" + +The tag to be assigned to messages read from this file. If you would like to +see the colon after the tag, you need to include it when you assign a tag +value, like so: ``tag="myTagValue:"``. + +Facility +^^^^^^^^ + +.. csv-table:: + :header: "type", "mandatory", "format", "default" + :widths: auto + :class: parameter-table + + "string", "no", "facility\|number", "local0" + +The syslog facility to be assigned to messages read from this file. Can be +specified in textual form (e.g. ``local0``, ``local1``, ...) or as numbers (e.g. +16 for ``local0``). Textual form is suggested. + +Severity +^^^^^^^^ + +.. csv-table:: + :header: "type", "mandatory", "format", "default" + :widths: auto + :class: parameter-table + + "string", "no", "severity\|number", "notice" + +The syslog severity to be assigned to lines read. Can be specified +in textual form (e.g. ``info``, ``warning``, ...) or as numbers (e.g. 6 +for ``info``). Textual form is suggested. + +DeduplicateSpaces +^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "mandatory", "format", "default" + :widths: auto + :class: parameter-table + + "binary", "no", "", "on" + +The parameter modify the way consecutive spaces like chars are managed. +When it is setted to "on", consecutive spaces like chars are reduced to a single one +and trailing space like chars are suppressed. + +Delete +^^^^^^ + +.. csv-table:: + :header: "type", "mandatory", "format", "default" + :widths: auto + :class: parameter-table + + "string", "no", "<regex> <reject>", + +This parameter informs the module to delete the report to flag it as treated. +If the file is too large (or failed to be removed) it is renamed using the +<regex> to identify part of the file name that has to be replaced it by +<reject>. See Examples + +Rename +^^^^^^ + +.. csv-table:: + :header: "type", "mandatory", "format", "default" + :widths: auto + :class: parameter-table + + "string", "no", "<regex> <sent> <reject>", + +This parameter informs the module to rename the report to flag it as treated. +The file is renamed using the <regex> to identify part of the file name that +has to be replaced it: + +- by <rename> if the file was successfully treated, + +- by <reject> if the file is too large to be sent. + +See #Examples + +Programkey +^^^^^^^^^^ + +.. csv-table:: + :header: "type", "mandatory", "format", "default" + :widths: auto + :class: parameter-table + + "string", "no", , + +The attribute in structured data which contains the rsyslog APPNAME. +This attribute has to be a String between double quotes ("). + +Timestampkey +^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "mandatory", "format", "default" + :widths: auto + :class: parameter-table + + "string", "no", , + +The attribute in structured data which contains the rsyslog TIMESTAMP. +This attribute has to be a Number (Unix TimeStamp). + +Examples +======== + +The example show the delete action. All files corresponding to +"/test/\*.ok" will be treated as batch reports and will be deleted +on success or renamed from <file>.ok to <file>.rejected in other +cases. + +.. code-block:: none + + module(load="imbatchreport") + input(type="imbatchreport" reports="/test/\*.ok" + ruleset="myruleset" tag="batch" + delete=".ok$ .rejected" + programkey="SHELL" timestampkey="START" + ) + +The example show the rename action. All files corresponding to +"/test/\*.ok" will be treated as batch reports and will be renamed +from <file>.ok to <file>.sent on success or +renamed from <file>.ok to <file>.rejected in other cases. + +.. code-block:: none + + module(load="imbatchreport") + input(type="imbatchreport" reports="/test/\*.ok" + ruleset="myruleset" tag="batch" + rename=".ok$ .sent .rejected" + programkey="SHELL" timestampkey="START" + ) diff --git a/source/configuration/modules/imdocker.rst b/source/configuration/modules/imdocker.rst new file mode 100644 index 0000000..8c47b8f --- /dev/null +++ b/source/configuration/modules/imdocker.rst @@ -0,0 +1,268 @@ +*************************************** +imdocker: Docker Input Module +*************************************** + +=========================== =========================================================================== +**Module Name:** **imdocker** +**Author:** Nelson Yen +**Available since:** 8.41.0 +=========================== =========================================================================== + + +Purpose +======= + +The imdocker input plug-in provides the ability to receive container logs from Docker (engine) +via the Docker Rest API. + +Other features include: + +- filter containers through the plugin options +- handle long log lines (greater than 16kb) and obtain +- container metadata, such as container id, name, image id, labels, etc. + +**Note**: Multiple docker instances are not supported at the time of this writing. + + +Configuration Parameters +======================== + +The configuration parameters for this module are designed for tailoring +the behavior of imdocker. + +.. note:: + + Parameter names are case-insensitive. + +.. note:: + + This module supports module parameters, only. + + + +Module Parameters +----------------- + + +DockerApiUnixSockAddr +^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "/var/run/docker.sock", "no", "none" + +Specifies the Docker unix socket address to use. + +ApiVersionStr +^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "v1.27", "no", "none" + +Specifies the version of Docker API to use. Must be in the format specified by the +Docker api, e.g. similar to the default above (v1.27, v1.28, etc). + + +PollingInterval +^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "60", "no", "none" + +Specifies the polling interval in seconds, imdocker will poll for new containers by +calling the 'List containers' API from the Docker engine. + + +ListContainersOptions +^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "", "no", "none" + +Specifies the http query component of the a 'List Containers' HTTP API request. +See Docker API for more information about available options. +**Note**: It is not necessary to prepend the string with '?'. + + +GetContainerLogOptions +^^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "timestamp=0&follow=1&stdout=1&stderr=1&tail=1", "no", "none" + +Specifies the http query component of the a 'Get container logs' HTTP API request. +See Docker API for more information about available options. +**Note**: It is not necessary to prepend the string with '?'. + + +RetrieveNewLogsFromStart +^^^^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "1", "no", "none" + +This option specifies the whether imdocker will process newly found container logs from the beginning. +The exception is for containers found on start-up. The container logs for containers +that were active at imdocker start-up are controlled via 'GetContainerLogOptions', the +'tail' in particular. + + +DefaultFacility +^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer or string (preferred)", "user", "no", "``$InputFileFacility``" + +The syslog facility to be assigned to log messages received. Specified as numbers. + +.. seealso:: + + https://en.wikipedia.org/wiki/Syslog + + +DefaultSeverity +^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer or string (preferred)", "notice", "no", "``$InputFileSeverity``" + +The syslog severity to be assigned to log messages received. Specified as numbers (e.g. 6 +for ``info``). Textual form is suggested. Default is ``notice``. + +.. seealso:: + + https://en.wikipedia.org/wiki/Syslog + + +escapeLF +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "on", "no", "none" + +This is only meaningful if multi-line messages are to be processed. +LF characters embedded into syslog messages cause a lot of trouble, +as most tools and even the legacy syslog TCP protocol do not expect +these. If set to "on", this option avoid this trouble by properly +escaping LF characters to the 4-byte sequence "#012". This is +consistent with other rsyslog control character escaping. By default, +escaping is turned on. If you turn it off, make sure you test very +carefully with all associated tools. Please note that if you intend +to use plain TCP syslog with embedded LF characters, you need to +enable octet-counted framing. +For more details, see Rainer's blog posting on imfile LF escaping. + + +Metadata +======== +The imdocker module supports message metadata. It supports the following +data items: + +- **Id** - the container id associated with the message. + +- **Names** - the first container associated with the message. + +- **ImageID** - the image id of the container associated with the message. + +- **Labels** - all the labels of the container associated with the message in json format. + +**Note**: At the time of this writing, metadata is always enabled. + + +Statistic Counter +================= + +This plugin maintains `statistics <http://www.rsyslog.com/rsyslog-statistic-counter/>`. The statistic is named "imdocker". + +The following properties are maintained for each listener: + +- **submitted** - total number of messages submitted to main queue after reading from journal for processing + since startup. All records may not be submitted due to rate-limiting. + +- **ratelimit.discarded** - number of messages discarded due to rate-limiting within configured + rate-limiting interval. + +- **curl.errors** - total number of curl errors. + + +Caveats/Known Bugs +================== + +- At the moment, this plugin only supports a single instance of docker on a host. + + +Configuration Examples +====================== + +Load module, with only defaults +-------------------------------- + +This activates the module with all the default options: + +.. code-block:: none + + module(load="imdocker") + + +Load module, with container filtering +------------------------------------- + +This activates the module with container filtering on a label: + +.. code-block:: none + + module(load="imdocker" + DockerApiUnixSockAddr="/var/run/docker.sock" + ApiVersionStr="v1.27" + PollingInterval="60" + ListContainersOptions="filters={\"label\":[\"log_opt_enabled\"]}" + GetContainerLogOptions="timestamps=0&follow=1&stdout=1&stderr=0&tail=1" + ) + + +Example template to get container metadata +------------------------------------------ + +An example of how to create a template with container metadata + +.. code-block:: none + + template (name="ImdockerFormat" type="string" + string="program:%programname% tag:%syslogtag% id:%$!metadata!Id% name:%$!metadata!Names% imageid:%$!metadata!ImageID% labels:%$!metadata!Labels% msg: %msg%\n" + ) + diff --git a/source/configuration/modules/imfile.rst b/source/configuration/modules/imfile.rst new file mode 100644 index 0000000..cc11742 --- /dev/null +++ b/source/configuration/modules/imfile.rst @@ -0,0 +1,948 @@ +****************************** +imfile: Text File Input Module +****************************** + +.. index:: ! imfile + +=========================== =========================================================================== +**Module Name:** **imfile** +**Author:** `Rainer Gerhards <https://rainer.gerhards.net/>`_ <rgerhards@adiscon.com> +=========================== =========================================================================== + +Purpose +======= + +This module provides the ability to convert any standard text file +into a syslog +message. A standard text file is a file consisting of printable +characters with lines being delimited by LF. + +The file is read line-by-line and any line read is passed to rsyslog's +rule engine. The rule engine applies filter conditions and selects which +actions needs to be carried out. Empty lines are **not** processed, as +they would result in empty syslog records. They are simply ignored. + +As new lines are written they are taken from the file and processed. +Depending on the selected mode, this happens via inotify or based on +a polling interval. Especially in polling mode, file reading doesn't +happen immediately. But there are also slight delays (due to process +scheduling and internal processing) in inotify mode. + +The file monitor supports file rotation. To fully work, +rsyslogd must run while the file is rotated. Then, any remaining lines +from the old file are read and processed and when done with that, the +new file is being processed from the beginning. If rsyslogd is stopped +during rotation, the new file is read, but any not-yet-reported lines +from the previous file can no longer be obtained. + +When rsyslogd is stopped while monitoring a text file, it records the +last processed location and continues to work from there upon restart. +So no data is lost during a restart (except, as noted above, if the file +is rotated just in this very moment). + +Notable Features +================ + +- :ref:`Metadata` +- :ref:`State-Files` +- :ref:`WildCards` +- presentation on `using wildcards with imfile <http://www.slideshare.net/rainergerhards1/using-wildcards-with-rsyslogs-file-monitor-imfile>`_ + + +Configuration +============= + +.. note:: + + Parameter names are case-insensitive. + + +Module Parameters +----------------- + +Mode +^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "inotify", "no", "none" + +.. versionadded:: 8.1.5 + +This specifies if imfile is shall run in inotify ("inotify") or polling +("polling") mode. Traditionally, imfile used polling mode, which is +much more resource-intense (and slower) than inotify mode. It is +suggested that users turn on "polling" mode only if they experience +strange problems in inotify mode. In theory, there should never be a +reason to enable "polling" mode and later versions will most probably +remove it. + +Note: if a legacy "$ModLoad" statement is used, the default is *polling*. +This default was kept to prevent problems with old configurations. It +might change in the future. + +.. versionadded:: 8.32.0 + +On Solaris, the FEN API is used instead of INOTIFY. You can set the mode +to fen or inotify (which is automatically mapped to fen on Solaris OS). +Please note that the FEN is limited compared to INOTIFY. Deep wildcard +matches may not work because of the API limits for now. + + +readTimeout +^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "0", "no", "none" + +*Default: 0 (no timeout)* + +.. versionadded:: 8.23.0 + +This sets the default value for input *timeout* parameters. See there +for exact meaning. Parameter value is the number of seconds. + + +timeoutGranularity +^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "1", "no", "none" + +.. versionadded:: 8.23.0 + +This sets the interval in which multi-line-read timeouts are checked. +The interval is specified in seconds. Note that +this establishes a lower limit on the length of the timeout. For example, if +a timeoutGranularity of 60 seconds is selected and a readTimeout value of 10 seconds +is used, the timeout is nevertheless only checked every 60 seconds (if there is +no other activity in imfile). This means that the readTimeout is also only +checked every 60 seconds, which in turn means a timeout can occur only after 60 +seconds. + +Note that timeGranularity has some performance implication. The more frequently +timeout processing is triggered, the more processing time is needed. This +effect should be negligible, except if a very large number of files is being +monitored. + + +sortFiles +^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +.. versionadded:: 8.32.0 + +If this parameter is set to on, the files will be processed in sorted order, else +not. However, due to the inherent asynchronicity of the whole operations involved +in tracking files, it is not possible to guarantee this sorted order, as it also +depends on operation mode and OS timing. + + +PollingInterval +^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "10", "no", "none" + +This setting specifies how often files are to be +polled for new data. For obvious reasons, it has effect only if +imfile is running in polling mode. +The time specified is in seconds. During each +polling interval, all files are processed in a round-robin fashion. + +A short poll interval provides more rapid message forwarding, but +requires more system resources. While it is possible, we strongly +recommend not to set the polling interval to 0 seconds. That will +make rsyslogd become a CPU hog, taking up considerable resources. It +is supported, however, for the few very unusual situations where this +level may be needed. Even if you need quick response, 1 seconds +should be well enough. Please note that imfile keeps reading files as +long as there is any data in them. So a "polling sleep" will only +happen when nothing is left to be processed. + +**We recommend to use inotify mode.** + + +statefile.directory +^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "global(WorkDirectory) value", "no", "none" + +.. versionadded:: 8.1905.0 + +This parameter permits to specify a dedicated directory for the storage of +imfile state files. An absolute path name should be specified (e.g. +`/var/rsyslog/imfilestate`). This permits to keep imfile state files separate +from other rsyslog work items. + +If not specified the global `workDirectory` setting is used. + +**Important: The directory must exist before rsyslog is started.** Also, +rsyslog needs write permissions to work correctly. Keep in mind that this +also might require SELinux definitions (or similar for other enhanced security +systems). + + +Input Parameters +---------------- + +File +^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "none", "yes", "``$InputFileName``" + +The file being monitored. So far, this must be an absolute name (no +macros or templates). Note that wildcards are supported at the file +name level (see **WildCards** below for more details). + + +Tag +^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "none", "yes", "``$InputFileTag``" + +The tag to be assigned to messages read from this file. If you would like to +see the colon after the tag, you need to include it when you assign a tag +value, like so: ``tag="myTagValue:"``. + + +Facility +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer or string (preferred)", "local0", "no", "``$InputFileFacility``" + +The syslog facility to be assigned to messages read from this file. Can be +specified in textual form (e.g. ``local0``, ``local1``, ...) or as numbers (e.g. +16 for ``local0``). Textual form is suggested. Default is ``local0``. + +.. seealso:: + + https://en.wikipedia.org/wiki/Syslog + + +Severity +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer or string (preferred)", "notice", "no", "``$InputFileSeverity``" + +The syslog severity to be assigned to lines read. Can be specified +in textual form (e.g. ``info``, ``warning``, ...) or as numbers (e.g. 6 +for ``info``). Textual form is suggested. Default is ``notice``. + +.. seealso:: + + https://en.wikipedia.org/wiki/Syslog + + +PersistStateInterval +^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "0", "no", "``$InputFilePersistStateInterval``" + +Specifies how often the state file shall be written when processing +the input file. The **default** value is 0, which means a new state +file is at least being written when the monitored files is being closed (end of +rsyslogd execution). Any other value n means that the state file is +written at least every time n file lines have been processed. This setting can +be used to guard against message duplication due to fatal errors +(like power fail). Note that this setting affects imfile performance, +especially when set to a low value. Frequently writing the state file +is very time consuming. + +Note further that rsyslog may write state files +more frequently. This happens if rsyslog has some reason to do so. +There is intentionally no more precise description of when state files +are being written, as this is an implementation detail and may change +as needed. + +**Note: If this parameter is not set, state files are not created.** + + +startmsg.regex +^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "none", "no", "none" + +.. versionadded:: 8.10.0 + +This permits the processing of multi-line messages. When set, a +messages is terminated when the next one begins, and +``startmsg.regex`` contains the regex that identifies the start +of a message. As this parameter is using regular expressions, it +is more flexible than ``readMode`` but at the cost of lower +performance. +Note that ``readMode`` and ``startmsg.regex`` and ``endmsg.regex`` cannot all be +defined for the same input. + + +endmsg.regex +^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "none", "no", "none" + +.. versionadded:: 8.38.0 + +This permits the processing of multi-line messages. When set, a message is +terminated when ``endmsg.regex`` matches the line that +identifies the end of a message. As this parameter is using regular +expressions, it is more flexible than ``readMode`` but at the cost of lower +performance. +Note that ``readMode`` and ``startmsg.regex`` and ``endmsg.regex`` cannot all be +defined for the same input. +The primary use case for this is multiline container log files which look like +this: + +.. code-block:: none + + date stdout P start of message + date stdout P middle of message + date stdout F end of message + +The `F` means this is the line which contains the final part of the message. +The fully assembled message should be `start of message middle of message end of +message`. `endmsg.regex="^[^ ]+ stdout F "` will match. + +readTimeout +^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "0", "no", "none" + +.. versionadded:: 8.23.0 + +This can be used with *startmsg.regex* (but not *readMode*). If specified, +partial multi-line reads are timed out after the specified timeout interval. +That means the current message fragment is being processed and the next +message fragment arriving is treated as a completely new message. The +typical use case for this parameter is a file that is infrequently being +written. In such cases, the next message arrives relatively late, maybe hours +later. Specifying a readTimeout will ensure that those "last messages" are +emitted in a timely manner. In this use case, the "partial" messages being +processed are actually full messages, so everything is fully correct. + +To guard against accidental too-early emission of a (partial) message, the +timeout should be sufficiently large (5 to 10 seconds or more recommended). +Specifying a value of zero turns off timeout processing. Also note the +relationship to the *timeoutGranularity* global parameter, which sets the +lower bound of *readTimeout*. + +Setting timeout values slightly increases processing time requirements; the +effect should only be visible of a very large number of files is being +monitored. + + +readMode +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "0", "no", "``$InputFileReadMode``" + +This provides support for processing some standard types of multiline +messages. It is less flexible than ``startmsg.regex`` or ``endmsg.regex`` but +offers higher performance than regex processing. Note that ``readMode`` and +``startmsg.regex`` and ``endmsg.regex`` cannot all be defined for the same +input. + +The value can range from 0-2 and determines the multiline +detection method. + +0 - (**default**) line based (each line is a new message) + +1 - paragraph (There is a blank line between log messages) + +2 - indented (new log messages start at the beginning of a line. If a +line starts with a space or tab "\t" it is part of the log message before it) + + +escapeLF +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "1", "no", "none" + +This is only meaningful if multi-line messages are to be processed. +LF characters embedded into syslog messages cause a lot of trouble, +as most tools and even the legacy syslog TCP protocol do not expect +these. If set to "on", this option avoid this trouble by properly +escaping LF characters to the 4-byte sequence "#012". This is +consistent with other rsyslog control character escaping. By default, +escaping is turned on. If you turn it off, make sure you test very +carefully with all associated tools. Please note that if you intend +to use plain TCP syslog with embedded LF characters, you need to +enable octet-counted framing. For more details, see +`Rainer Gerhards' blog posting on imfile LF escaping <https://rainer.gerhards.net/2013/09/imfile-multi-line-messages.html>`_. + + +escapeLF.replacement +^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "depending on use", "no", "none" + +.. versionadded:: 8.2001.0 + +This parameter works in conjunction with `escapeLF`. It is only +honored if `escapeLF="on"`. + +It permits to replace the default escape sequence by a different character +sequence. The default historically is inconsistent and denpends on which +functionality is used to read the file. It can be either "#012" or "\\n". If +you want to retain that default, do not configure this parameter. + +If it is configured, any sequence may be used. For example, to replace a LF +with a simple space, use:: + + escapeLF.replacement=" " + +It is also possible to configure longer replacements. An example for this is:: + + escapeLF.replacement="[LF]" + +Finally, it is possible to completely remove the LF. This is done by specifying +an empty replacement sequence:: + + escapeLF.replacement="" + + +MaxLinesAtOnce +^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "0", "no", "``$InputFileMaxLinesAtOnce``" + +This is a legacy setting that only is supported in *polling* mode. +In *inotify* mode, it is fixed at 0 and all attempts to configure +a different value will be ignored, but will generate an error +message. + +Please note that future versions of imfile may not support this +parameter at all. So it is suggested to not use it. + +In *polling* mode, if set to 0, each file will be fully processed and +then processing switches to the next file. If it is set to any other +value, a maximum of [number] lines is processed in sequence for each file, +and then the file is switched. This provides a kind of multiplexing +the load of multiple files and probably leads to a more natural +distribution of events when multiple busy files are monitored. For +*polling* mode, the **default** is 10240. + + +MaxSubmitAtOnce +^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "1024", "no", "none" + +This is an expert option. It can be used to set the maximum input +batch size that imfile can generate. The **default** is 1024, which +is suitable for a wide range of applications. Be sure to understand +rsyslog message batch processing before you modify this option. If +you do not know what this doc here talks about, this is a good +indication that you should NOT modify the default. + + +deleteStateOnFileDelete +^^^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "on", "no", "none" + +This parameter controls if state files are deleted if their associated +main file is deleted. Usually, this is a good idea, because otherwise +problems would occur if a new file with the same name is created. In +that case, imfile would pick up reading from the last position in +the **deleted** file, which usually is not what you want. + +However, there is one situation where not deleting associated state +file makes sense: this is the case if a monitored file is modified +with an editor (like vi or gedit). Most editors write out modifications +by deleting the old file and creating a new now. If the state file +would be deleted in that case, all of the file would be reprocessed, +something that's probably not intended in most case. As a side-note, +it is strongly suggested *not* to modify monitored files with +editors. In any case, in such a situation, it makes sense to +disable state file deletion. That also applies to similar use +cases. + +In general, this parameter should only by set if the users +knows exactly why this is required. + + +Ruleset +^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "none", "no", "``$InputFileBindRuleset``" + +Binds the listener to a specific :doc:`ruleset <../../concepts/multi_ruleset>`. + + +addMetadata +^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "-1", "no", "none" + +**Default: see intro section on Metadata** + +This is used to turn on or off the addition of metadata to the +message object. + + +addCeeTag +^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +This is used to turn on or off the addition of the "@cee:" cookie to the +message object. + + +reopenOnTruncate +^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +This is an **experimental** feature that tells rsyslog to reopen input file +when it was truncated (inode unchanged but file size on disk is less than +current offset in memory). + + +MaxLinesPerMinute +^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "0", "no", "none" + +Instructs rsyslog to enqueue up to the specified maximum number of lines +as messages per minute. Lines above this value are discarded. + +The **default** value is 0, which means that no lines are discarded. + + +MaxBytesPerMinute +^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "0", "no", "none" + +Instructs rsyslog to enqueue a maximum number of bytes as messages per +minute. Once MaxBytesPerMinute is reached, subsequent messages are +discarded. + +Note that messages are not truncated as a result of MaxBytesPerMinute, +rather the entire message is discarded if part of it would be above the +specified maximum bytes per minute. + +The **default** value is 0, which means that no messages are discarded. + + +trimLineOverBytes +^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "0", "no", "none" + +This is used to tell rsyslog to truncate the line which length is greater +than specified bytes. If it is positive number, rsyslog truncate the line +at specified bytes. Default value of 'trimLineOverBytes' is 0, means never +truncate line. + +This option can be used when ``readMode`` is 0 or 2. + + +freshStartTail +^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +This is used to tell rsyslog to seek to the end/tail of input files +(discard old logs) **at its first start(freshStart)** and process only new +log messages. + +When deploy rsyslog to a large number of servers, we may only care about +new log messages generated after the deployment. set **freshstartTail** +to **on** will discard old logs. Otherwise, there may be vast useless +message burst on the remote central log receiver + +This parameter only applies to files that are already existing during +rsyslog's initial processing of the file monitors. + +.. warning:: + + Depending on the number and location of existing files, this initial + startup processing may take some time as well. If another process + creates a new file at exactly the time of startup processing and writes + data to it, rsyslog might detect this file and it's data as prexisting + and may skip it. This race is inevitable. So when freshStartTail is used, + some risk of data loss exists. The same holds true if between the last + shutdown of rsyslog and its restart log file content has been added. + As such, the rsyslog team advises against activating the freshStartTail + option. + + +discardTruncatedMsg +^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +When messages are too long they are truncated and the following part is +processed as a new message. When this parameter is turned on the +truncated part is not processed but discarded. + + +msgDiscardingError +^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "on", "no", "none" + +Upon truncation an error is given. When this parameter is turned off, no +error will be shown upon truncation. + + +needParse +^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +.. versionadded:: 8.1903.0 + +By default, read message are sent to output modules without passing through +parsers. This parameter informs rsyslog to use also defined parser module(s). + + + +persistStateAfterSubmission +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +.. versionadded:: 8.2006.0 + +This setting makes imfile persist state file information after a batch of +messages has been submitted. It can be activated (switched to "on") in order +to provide enhanced robustness against unclean shutdowns. Depending on the +configuration of the rest of rsyslog (most importantly queues), persisting +the state file after each message submission prevents message loss +when reading files and the system is shutdown in an unclean way (e.g. +loss of power). + +Please note that this setting may cause frequent state file writes and +as such may cause some performance degradation. + + +ignoreOlderThan +^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "0", "no", "none" + +.. versionadded:: 8.2108.0 + +Instructs imfile to ignore a discovered file that has not been modified in the +specified number of seconds. Once a file is discovered, the file is no longer +ignored and new data will be read. This option is disabled (set to 0) by default. + + + +.. _Metadata: + +Metadata +======== +The imfile module supports message metadata. It supports the following +data items: + +- filename + + Name of the file where the message originated from. This is most + useful when using wildcards inside file monitors, because it then + is the only way to know which file the message originated from. + The value can be accessed using the %$!metadata!filename% property. + **Note**: For symlink-ed files this does **not** contain name of the + actual file (source of the data) but name of the symlink (file which + matched configured input). + +- fileoffset + + Offset of the file in bytes at the time the message was read. The + offset reported is from the **start** of the line. + This information can be useful when recreating multi-line files + that may have been accessed or transmitted non-sequentially. + The value can be accessed using the %$!metadata!fileoffset% property. + +Metadata is only present if enabled. By default it is enabled for +input() statements that contain wildcards. For all others, it is +disabled by default. It can explicitly be turned on or off via the +*addMetadata* input() parameter, which always overrides the default. + + +.. _State-Files: + +State Files +=========== +Rsyslog must keep track of which parts of the monitored file +are already processed. This is done in so-called "state files" that +are created in the rsyslog working directory and are read on startup to +resume monitoring after a shutdown. The location of the rsyslog +working directory is configurable via the ``global(workDirectory)`` +|FmtAdvancedName| format parameter. + +**Note**: The ``PersistStateInterval`` parameter must be set, otherwise state +files will NOT be created. + +Rsyslog automatically generates state file names. These state file +names will begin with the string ``imfile-state:`` and be followed +by some suffix rsyslog generates. + +There is intentionally no more precise description of when state file +naming, as this is an implementation detail and may change as needed. + +Note that it is possible to set a fixed state file name via the +deprecated ``stateFile`` parameter. It is suggested to avoid this, as +the user must take care of name clashes. Most importantly, if +"stateFile" is set for file monitors with wildcards, the **same** +state file is used for all occurrences of these files. In short, +this will usually not work and cause confusion. Upon startup, +rsyslog tries to detect these cases and emit warning messages. +However, the detection simply checks for the presence of "*" +and as such it will not cover more complex cases. + +Note that when the ``global(workDirectory)`` |FmtAdvancedName| format +parameter is points to a non-writable location, the state file +**will not be generated**. In those cases, the file content will always +be completely re-sent by imfile, because the module does not know that it +already processed parts of that file. if The parameter is not set to all, it +defaults to the file system root, which may or may not be writable by +the rsyslog process. + + +.. _WildCards: + +WildCards +========= + +**Before Version: 8.25.0** + Wildcards are only supported in the filename part, not in directory names. + +* /var/log/\*.log **works**. * +* /var/log/\*/syslog.log does **not work**. * + + +**Since Version: 8.25.0** + Wildcards are supported in filename and paths which means these samples will work: + +* /var/log/\*.log **works**. * +* /var/log/\*/syslog.log **works**. * +* /var/log/\*/\*.log **works**. * + + + All matching files in all matching subfolders will work. + Note that this may decrease performance in imfile depending on how + many directories and files are being watched dynamically. + + + + +Caveats/Known Bugs +================== + +* symlink may not always be properly processed + +Configuration Examples +====================== + +The following sample monitors two files. If you need just one, remove +the second one. If you need more, add them according to the sample ;). +This code must be placed in /etc/rsyslog.conf (or wherever your distro +puts rsyslog's config files). Note that only commands actually needed +need to be specified. The second file uses less commands and uses +defaults instead. + +.. code-block:: none + + module(load="imfile" PollingInterval="10") #needs to be done just once + + # File 1 + input(type="imfile" + File="/path/to/file1" + Tag="tag1" + Severity="error" + Facility="local7") + + # File 2 + input(type="imfile" + File="/path/to/file2" + Tag="tag2") + + # ... and so on ... # + + +Deprecated parameters +===================== + +**Note:** While these parameters are still accepted, they should no longer be +used for newly created configurations. + +stateFile +--------- + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "none", "no", "``$InputFileStateFile``" + +This is the name of this file's state file. This parameter should +usually **not** be used. Check the section on "State Files" above +for more details. + + diff --git a/source/configuration/modules/imgssapi.rst b/source/configuration/modules/imgssapi.rst new file mode 100644 index 0000000..de51b36 --- /dev/null +++ b/source/configuration/modules/imgssapi.rst @@ -0,0 +1,154 @@ +************************************ +imgssapi: GSSAPI Syslog Input Module +************************************ + +=========================== =========================================================================== +**Module Name:** **imgssapi** +**Author:** varmojfekoj +=========================== =========================================================================== + + +Purpose +======= + +Provides the ability to receive syslog messages from the network +protected via Kerberos 5 encryption and authentication. This module also +accept plain tcp syslog messages on the same port if configured to do +so. If you need just plain tcp, use :doc:`imtcp <imtcp>` instead. + +Note: This is a contributed module, which is not supported by the +rsyslog team. We recommend to use RFC5425 TLS-protected syslog +instead. + +.. toctree:: + :maxdepth: 1 + + gssapi + + +Configuration Parameters +======================== + +.. note:: + + Parameter names are case-insensitive. + + +Input Parameter +--------------- + +.. note:: + + Parameter are only available in Legacy Format. + + +InputGSSServerRun +^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "``$InputGSSServerRun``" + +Starts a GSSAPI server on selected port - note that this runs +independently from the TCP server. + + +InputGSSServerServiceName +^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "``$InputGSSServerServiceName``" + +The service name to use for the GSS server. + + +InputGSSServerPermitPlainTCP +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "0", "no", "``$InputGSSServerPermitPlainTCP``" + +Permits the server to receive plain tcp syslog (without GSS) on the +same port. + + +InputGSSServerMaxSessions +^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "200", "no", "``$InputGSSServerMaxSessions``" + +Sets the maximum number of sessions supported. + + +InputGSSServerKeepAlive +^^^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "0", "no", "``$InputGSSServerKeepAlive``" + +.. versionadded:: 8.5.0 + +Enables or disable keep-alive handling. + + +InputGSSListenPortFileName +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "``$InputGSSListenPortFileName``" + +.. versionadded:: 8.38.0 + +With this parameter you can specify the name for a file. In this file the +port, imtcp is connected to, will be written. +This parameter was introduced because the testbench works with dynamic ports. + +.. note:: + + If this parameter is set, 0 will be accepted as the port. Otherwise it + is automatically changed to port 514 + + +Caveats/Known Bugs +================== + +- module always binds to all interfaces +- only a single listener can be bound + +Example +======= + +This sets up a GSS server on port 1514 that also permits to receive +plain tcp syslog messages (on the same port): + +.. code-block:: none + + $ModLoad imgssapi # needs to be done just once + $InputGSSServerRun 1514 + $InputGSSServerPermitPlainTCP on + + diff --git a/source/configuration/modules/imhiredis.rst b/source/configuration/modules/imhiredis.rst new file mode 100644 index 0000000..3574332 --- /dev/null +++ b/source/configuration/modules/imhiredis.rst @@ -0,0 +1,356 @@ + +.. include:: <isonum.txt> + +***************************** +Imhiredis: Redis input plugin +***************************** + +==================== ===================================== +**Module Name:** **imhiredis** +**Author:** Jeremie Jourdin <jeremie.jourdin@advens.fr> +**Contributors:** Theo Bertin <theo.bertin@advens.fr> +==================== ===================================== + +Purpose +======= + +Imhiredis is an input module reading arbitrary entries from Redis. +It uses the `hiredis library <https://github.com/redis/hiredis.git>`_ to query Redis instances using 3 modes: + +- **queues**, using `LIST <https://redis.io/commands#list>`_ commands +- **channels**, using `SUBSCRIBE <https://redis.io/commands#pubsub>`_ commands +- **streams**, using `XREAD/XREADGROUP <https://redis.io/commands/?group=stream>`_ commands + + +.. _imhiredis_queue_mode: + +Queue mode +---------- + +The **queue mode** uses Redis LISTs to push/pop messages to/from lists. It allows simple and efficient uses of Redis as a queueing system, providing both LIFO and FIFO methods. + +This mode should be preferred if the user wants to use Redis as a caching system, with one (or many) Rsyslog instances POP'ing out entries. + +.. Warning:: + This mode was configured to provide optimal performances while not straining Redis, but as imhiredis has to poll the instance some trade-offs had to be made: + + - imhiredis POPs entries by batches of 10 to improve performances (size of batch is configurable via the batchsize parameter) + - when no entries are left in the list, the module sleeps for 1 second before checking the list again. This means messages might be delayed by as much as 1 second between a push to the list and a pop by imhiredis (entries will still be POP'ed out as fast as possible while the list is not empty) + + +.. _imhiredis_channel_mode: + +Channel mode +------------ + +The **subscribe** mode uses Redis PUB/SUB system to listen to messages published to Redis' channels. It allows performant use of Redis as a message broker. + +This mode should be preferred to use Redis as a message broker, with zero, one or many subscribers listening to new messages. + +.. Warning:: + This mode shouldn't be used if messages are to be reliably processed, as messages published when no Imhiredis is listening will result in the loss of the message. + + +.. _imhiredis_stream_mode: + +Stream mode +------------ + +The **stream** mode uses `Redis Streams system <https://redis.io/docs/data-types/streams/>`_ to read entries published to Redis' streams. It is a good alternative when: + - sharing work is desired + - not losing any log (even in the case of a crash) is mandatory + +This mode is especially useful to define pools of workers that do various processing along the way, while ensuring not a single log is lost during processing by a worker. + +.. note:: + As Redis streams do not insert simple values in keys, but rather fleid/value pairs, this mode can also be useful when handling structured data. This is better shown with the examples for the parameter :ref:`imhiredis_fields`. + + This mode also adds additional internal metadata to the message, it won't be included in json data or regular fields, but + + - **$.redis!stream** will be added to the message, with the value of the source stream + - **$.redis!index** will be added to the message, with the exact ID of the entry + - **$.redis!group** will be added in the message (if :ref:`imhiredis_stream_consumergroup` is set), with the value of the group used to read the entry + - **$.redis!consumer** will be added in the message (if :ref:`imhiredis_stream_consumername` is set), with the value of the consumer name used to read the entry + + This is especially useful when used with the omhiredis module, to allow it to get the required information semi-automatically (custom templates will still be required in the user configuration) + +.. Warning:: + This mode is the most reliable to handle entries stored in Redis, but it might also be the one with the most overhead. Although still minimal, make sure to test the different options and determine if this mode is right for you! + + +Master/Replica +-------------- + +This module is able to automatically connect to the master instance of a master/replica(s) cluster. Simply providing a valid connection entry point (being the current master or a valid replica), Imhiredis is able to redirect to the master node on startup and when states change between nodes. + + +Configuration Parameters +======================== + +.. note:: + Parameter names are case-insensitive + + +Input Parameters +---------------- + +.. _imhiredis_mode: + +mode +^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "subscribe", "yes", "none" + +| Defines the mode to use for the module. +| Should be either "**subscribe**" (:ref:`imhiredis_channel_mode`), "**queue**" (:ref:`imhiredis_queue_mode`) or "**stream**" (:ref:`imhiredis_stream_mode`) (case-sensitive). + + +ruleset +^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +Assign messages from this input to a specific Rsyslog ruleset. + + +batchsize +^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "number", "10", "yes", "none" + +Defines the dequeue batch size for redis pipelining. +imhiredis will read "**batchsize**" elements from redis at a time. + +When using the :ref:`imhiredis_queue_mode`, defines the size of the batch to use with LPOP / RPOP. + + +.. _imhiredis_key: + +key +^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "yes", "none" + +Defines either the name of the list to use (for :ref:`imhiredis_queue_mode`) or the channel to listen to (for :ref:`imhiredis_channel_mode`). + + +.. _imhiredis_socketPath: + +socketPath +^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "no", "if no :ref:`imhiredis_server` provided", "none" + +Defines the socket to use when trying to connect to Redis. Will be ignored if both :ref:`imhiredis_server` and :ref:`imhiredis_socketPath` are given. + + +.. _imhiredis_server: + +server +^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "ip", "127.0.0.1", "if no :ref:`imhiredis_socketPath` provided", "none" + +The Redis server's IP to connect to. + + +.. _imhiredis_port: + +port +^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "number", "6379", "no", "none" + +The Redis server's port to use when connecting via IP. + + +.. _imhiredis_password: + +password +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +The password to use when connecting to a Redis node, if necessary. + + +.. _imhiredis_uselpop: + +uselpop +^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "boolean", "no", "no", "none" + +| When using the :ref:`imhiredis_queue_mode`, defines if imhiredis should use a LPOP instruction instead of a RPOP (the default). +| Has no influence on the :ref:`imhiredis_channel_mode` and will be ignored if set with this mode. + + +.. _imhiredis_stream_consumergroup: + +stream.consumerGroup +^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "", "no", "none" + +| When using the :ref:`imhiredis_stream_mode`, defines a consumer group name to use (see `the XREADGROUP documentation <https://redis.io/commands/xreadgroup/>`_ for details). This parameter activates the use of **XREADGROUP** commands, in replacement to simple XREADs. +| Has no influence in the other modes (queue or channel) and will be ignored. + +.. note:: + If this parameter is set, :ref:`imhiredis_stream_consumername` should also be set + + +.. _imhiredis_stream_consumername: + +stream.consumerName +^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "", "no", "none" + +| When using the :ref:`imhiredis_stream_mode`, defines a consumer name to use (see `the XREADGROUP documentation <https://redis.io/commands/xreadgroup/>`_ for details). This parameter activates the use of **XREADGROUP** commands, in replacement to simple XREADs. +| Has no influence in the other modes (queue or channel) and will be ignored. + +.. note:: + If this parameter is set, :ref:`imhiredis_stream_consumergroup` should also be set + + +.. _imhiredis_stream_readfrom: + +stream.readFrom +^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "$", "no", "none" + +| When using the :ref:`imhiredis_stream_mode`, defines the `starting ID <https://redis.io/docs/data-types/streams-tutorial/#entry-ids>`_ for XREAD/XREADGROUP commands (can also use special IDs, see `documentation <https://redis.io/docs/data-types/streams-tutorial/#special-ids-in-the-streams-api>`_). +| Has no influence in the other modes (queue or channel) and will be ignored. + + +.. _imhiredis_stream_consumerack: + +stream.consumerACK +^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "boolean", "on", "no", "none" + +| When using :ref:`imhiredis_stream_mode` with :ref:`imhiredis_stream_consumergroup` and :ref:`imhiredis_stream_consumername`, determines if the module should directly acknowledge the ID once read from the Consumer Group. +| Has no influence in the other modes (queue or channel) and will be ignored. + +.. note:: + When using Consumer Groups and imhiredis, omhiredis can also integrate with this workflow to acknowledge a processed message once put back in another stream (or somewhere else). This parameter is then useful set to **off** to let the omhiredis module acknowledge the input ID once the message is correctly sent. + + +.. _imhiredis_stream_autoclaimidletime: + +stream.autoclaimIdleTime +^^^^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "positive number", "0", "no", "none" + +| When using :ref:`imhiredis_stream_mode` with :ref:`imhiredis_stream_consumergroup` and :ref:`imhiredis_stream_consumername`, determines if the module should check for pending IDs that exceed this time (**in milliseconds**) to assume the original consumer failed to acknowledge the log and claim them for their own (see `the redis ducumentation <https://redis.io/docs/data-types/streams-tutorial/#automatic-claiming>`_ on this subject for more details on how that works). +| Has no influence in the other modes (queue or channel) and will be ignored. + +.. note:: + If this parameter is set, the AUTOCLAIM operation will also take into account the specified :ref:`imhiredis_stream_readfrom` parameter. **If its value is '$' (default), the AUTOCLAIM commands will use '0-0' as the starting ID**. + + + +.. _imhiredis_fields: + +fields +^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "array", "[]", "no", "none" + +| When using :ref:`imhiredis_stream_mode`, the module won't get a simple entry but will instead get hashes, with field/value pairs. +| By default, the module will insert every value into their respective field in the **$!** object, but this parameter can change this behaviour, for each entry the value will be a string where: + + - if the entry begins with a **!** or a **.**, it will be taken as a key to take into the original entry + - if the entry doesn't begin with a **!** or a **.**, the value will be taken verbatim + - in addition, if the value is prefixed with a **:<key>:** pattern, the value (verbatim or taken from the entry) will be inserted in this specific key (or subkey) + +*Examples*: + +.. csv-table:: + :header: "configuration", "result" + :widths: auto + :class: parameter-table + + ``["static_value"]``, the value "static_value" will be inserted in $!static_value + ``[":key:static_value"]``, the value "static_value" will be inserted in $!key + ``["!field"]``, the value of the field "field" will be inserted in $!field + ``[":key!subkey:!field"]``, the value of the field "field" will be inserted in $!key!subkey + diff --git a/source/configuration/modules/imhttp.rst b/source/configuration/modules/imhttp.rst new file mode 100644 index 0000000..1200c39 --- /dev/null +++ b/source/configuration/modules/imhttp.rst @@ -0,0 +1,369 @@ +************************* +imhttp: http input module +************************* + +=========================== =========== +**Module Name:** **imhttp** +**Author:** Nelson Yen +=========================== =========== + + +Purpose +======= + +Provides the ability to receive adhoc and plaintext syslog messages via http. The format of messages accepted, +depends on configuration. imhttp exposes the capabilities and the underlying options of the http library +used, which currently is civetweb. + +Civetweb documentation: + +- `Civetweb User Manual <https://github.com/civetweb/civetweb/blob/master/docs/UserManual.md>`_ +- `Civetweb Configuration Options <https://github.com/civetweb/civetweb/blob/master/docs/UserManual.md#configuration-options>`_ + +Notable Features +================ + +- :ref:`imhttp-statistic-counter` +- :ref:`imhttp-error-messages` + + +Configuration Parameters +======================== + +.. note:: + + Parameter names are case-insensitive. + + +Module Parameters +----------------- + +Ports +^^^^^^^ + +.. csv-table:: + :header: "type", "mandatory", "format", "default" + :widths: auto + :class: parameter-table + + "string", "no", "ports", "8080" + +Configures "listening_ports" in the civetweb library. This option may also be configured using the +liboptions_ (below) however, this option will take precendence. + +- `Civetweb listening_ports <https://github.com/civetweb/civetweb/blob/master/docs/UserManual.md#listening_ports-8080>`_ + + +documentroot +^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "mandatory", "format", "default" + :widths: auto + :class: parameter-table + + "string", "no", "none", "." + +Configures "document_root" in the civetweb library. This option may also be configured using liboptions_, however +this option will take precedence. + +- `Civetweb document_root <https://github.com/civetweb/civetweb/blob/master/docs/UserManual.md#document_root->`_ + + +.. _liboptions: + +liboptions +^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "mandatory", "format", "default" + :widths: auto + :class: parameter-table + + "string", "no", "none", "none" + +Configures civetweb library "Options". + +- `Civetweb Options <https://github.com/civetweb/civetweb/blob/master/docs/UserManual.md#options-from-civetwebc>`_ + + +Input Parameters +---------------- + +These parameters can be used with the "input()" statement. They apply to +the input they are specified with. + + +Endpoint +^^^^^^^^ + +.. csv-table:: + :header: "type", "mandatory", "format", "default" + :widths: auto + :class: parameter-table + + "string", "yes", "path that begins with '/' ", "none" + +Sets a request path for an http input. Path should always start with a '/'. + + +DisableLFDelimiter +^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "mandatory", "format", "default" + :widths: auto + :class: parameter-table + + "binary", "no", "", "off" + +By default LF is used to delimit msg frames, for data is sent in batches. +Set this to ‘on’ if this behavior is not needed. + + +Name +^^^^ + +.. csv-table:: + :header: "type", "mandatory", "format", "default" + :widths: auto + :class: parameter-table + + "string", "no", "", "imhttp" + +Sets a name for the inputname property. If no name is set "imhttp" +is used by default. Setting a name is not strictly necessary, but can +be useful to apply filtering based on which input the message was +received from. + + +Ruleset +^^^^^^^ + +.. csv-table:: + :header: "type", "mandatory", "format", "default" + :widths: auto + :class: parameter-table + + "string", "no", "", "default ruleset" + +Binds specified ruleset to this input. If not set, the default +ruleset is bound. + + +SupportOctetCountedFraming +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "mandatory", "format", "default" + :widths: auto + :class: parameter-table + + "binary", "no", "", "off" + +Useful to send data using syslog style message framing, disabled by default. Message framing is described by `RFC 6587 <https://tools.ietf.org/html/rfc6587#section-3.4.1>`_ . + + +RateLimit.Interval +^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "mandatory", "format", "default" + :widths: auto + :class: parameter-table + + "integer", "no", "none", "0" + +Specifies the rate-limiting interval in seconds. Set it to a number +of seconds to activate rate-limiting. + + +RateLimit.Burst +^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "mandatory", "format", "default" + :widths: auto + :class: parameter-table + + "integer", "no", "none", "10000" + +Specifies the rate-limiting burst in number of messages. + + + +flowControl +^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "mandatory", "format", "default" + :widths: auto + :class: parameter-table + + "binary", "no", "none", "on" + +Flow control is used to throttle the sender if the receiver queue is +near-full preserving some space for input that can not be throttled. + + + +addmetadata +^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "mandatory", "format", "default" + :widths: auto + :class: parameter-table + + "binary", "no", "none", "off" + +Enables metadata injection into `$!metadata` property. Currently, only header data is supported. +The following metadata will be injected into the following properties: + +- `$!metadata!httpheaders`: http header data will be injected here as key-value pairs. All header names will automatically be lowercased + for case-insensitive access. + +- `$!metadata!queryparams`: query parameters from the http request will be injected here as key-value pairs. All header names will automatically be lowercased + for case-insensitive access. + + +basicAuthFile +^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "mandatory", "format", "default" + :widths: auto + :class: parameter-table + + "string", "no", "none", "" + +Enables access control to this endpoint using http basic authentication. Option is disabled by default. +To enable it, set this option to a `htpasswd file`, which can be generated using a standard `htpasswd` tool. + +See also: + +- `HTTP Authorization <https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Authorization>`_ +- `HTTP Basic Authentication <https://developer.mozilla.org/en-US/docs/Web/HTTP/Authentication#basic_authentication_scheme>`_ +- `htpasswd utility <https://httpd.apache.org/docs/2.4/programs/htpasswd.html>`_ + + +.. _imhttp-statistic-counter: + + +basicAuthFile +^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "mandatory", "format", "default" + :widths: auto + :class: parameter-table + + "string", "no", "none", "none" + +Configures a `htpasswd <https://httpd.apache.org/docs/2.4/programs/htpasswd.html>`_ file and enables `basic authentication <https://en.wikipedia.org/wiki/Basic_access_authentication>`_ on http request received on this input. +If this option is not set, basic authentation will not be enabled. + + +Statistic Counter +================= + +This plugin maintains global imhttp :doc:`statistics <../rsyslog_statistic_counter>`. The statistic's origin and name is "imhttp" and is +accumulated for all inputs. The statistic has the following counters: + + +- **submitted** - Total number of messages successfully submitted for processing since startup. +- **failed** - Total number of messages failed since startup, due to processing a request. +- **discarded** - Total number of messages discarded since startup, due to rate limiting or similar. + + +.. _imhttp-error-messages: + +Error Messages +============== + +When a message is to long it will be truncated and an error will show the remaining length of the message and the beginning of it. It will be easier to comprehend the truncation. + + +Caveats/Known Bugs +================== + +- module currently only a single http instance, however multiple ports may be bound. + + +Examples +======== + +Example 1 +--------- + +This sets up a http server instance on port 8080 with two inputs. +One input path at '/postrequest', and another at '/postrequest2': + +.. code-block:: none + + # ports=8080 + # document root='.' + module(load="imhttp") # needs to be done just once + + # Input using default LF delimited framing + # For example, the following http request, with data body "Msg0001\nMsg0002\nMsg0003" + ## + # - curl -si http://localhost:$IMHTTP_PORT/postrequest -d $'Msg0001\nMsg0002\nMsg0003' + ## + # Results in the 3 message objects being submitted into rsyslog queues. + # - Message object with `msg` property set to `Msg0001` + # - Message object with `msg` property set to `Msg0002` + # - Message object with `msg` property set to `Msg0003` + + input(type="imhttp" + name="myinput1" + endpoint="/postrequest" + ruleset="postrequest_rs") + + # define 2nd input path, using octet-counted framing, + # and routing to different ruleset + input(type="imhttp" + name="myinput2" + endpoint="/postrequest2" + SupportOctetCountedFraming="on" + ruleset="postrequest_rs") + + # handle the messages in ruleset + ruleset(name="postrequest_rs") { + action(type="omfile" file="/var/log/http_messages" template="myformat") + } + + +Example 2 +--------- + +This sets up a http server instance on ports 80 and 443s (use 's' to indicate ssl) with an input path at '/postrequest': + +.. code-block:: none + + # ports=8080, 443 (ssl) + # document root='.' + module(load="imhttp" ports=8080,443s) + input(type="imhttp" + endpoint="/postrequest" + ruleset="postrequest_rs") + + + +Example 3 +--------- + +imhttp can also support the underlying options of `Civetweb <https://github.com/civetweb/civetweb/blob/master/docs/UserManual.md>`_ using the liboptions_ option. + +.. code-block:: none + + module(load="imhttp" + liboptions=[ + "error_log_file=my_log_file_path", + "access_log_file=my_http_access_log_path", + ]) + + input(type="imhttp" + endpoint="/postrequest" + ruleset="postrequest_rs" + ) diff --git a/source/configuration/modules/imjournal.rst b/source/configuration/modules/imjournal.rst new file mode 100644 index 0000000..5c0404c --- /dev/null +++ b/source/configuration/modules/imjournal.rst @@ -0,0 +1,474 @@ +*************************************** +imjournal: Systemd Journal Input Module +*************************************** + +=========================== =========================================================================== +**Module Name:** **imjournal** +**Author:** Jiri Vymazal <jvymazal@redhat.com> (This module is **not** project-supported) +**Available since:** 7.3.11 +=========================== =========================================================================== + + +Purpose +======= + +Provides the ability to import structured log messages from systemd +journal to syslog. + +Note that this module reads the journal database, what is considered a +relatively performance-intense operation. As such, the performance of a +configuration utilizing this module may be notably slower than when +using `imuxsock <imuxsock.html>`_. The journal provides imuxsock with a +copy of all "classical" syslog messages, however, it does not provide +structured data. Only if that structured data is needed, imjournal must be used. +Otherwise, imjournal may simply be replaced by imuxsock, and we highly +suggest doing so. + +We suggest to check out our short presentation on `rsyslog journal +integration <http://youtu.be/GTS7EuSdFKE>`_ to learn more details of +anticipated use cases. + +**Warning:** Some versions of systemd journal have problems with +database corruption, which leads to the journal to return the same data +endlessly in a tight loop. This results in massive message duplication +inside rsyslog probably resulting in a denial-of-service when the system +resources get exhausted. This can be somewhat mitigated by using proper +rate-limiters, but even then there are spikes of old data which are +endlessly repeated. By default, ratelimiting is activated and permits to +process 20,000 messages within 10 minutes, what should be well enough +for most use cases. If insufficient, use the parameters described below +to adjust the permitted volume. **It is strongly recommended to use this +plugin only if there is hard need to do so.** + + +Notable Features +================ + +- statistics counters + + +Configuration Parameters +======================== + +.. note:: + + Parameter names are case-insensitive. + + +Module Parameters +================= + + +PersistStateInterval +^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "10", "no", "``$imjournalPersistStateInterval``" + +This is a global setting. It specifies how often should the journal +state be persisted. The persists happens after each *number-of-messages*. +This option is useful for rsyslog to start reading from the last journal +message it read. + +FileCreateMode +^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "octalNumber", "0644", "no", "none" + +Set the access permissions for the state file. The value given must +always be a 4-digit octal number, with the initial digit being zero. +Please note that the actual permission depend on rsyslogd's process +umask. If in doubt, use "$umask 0000" right at the beginning of the +configuration file to remove any restrictions. The state file's only +consumer is rsyslog, so it's recommended to adjust the value according +to that. + + +StateFile +^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "``$imjournalStateFile``" + +This is a global setting. It specifies where the state file for +persisting journal state is located. If a full path name is given +(starting with "/"), that path is used. Otherwise the given name +is created inside the working directory. + + +Ratelimit.Interval +^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "600", "no", "``$imjournalRatelimitInterval``" + +Specifies the interval in seconds onto which rate-limiting is to be +applied. If more than ratelimit.burst messages are read during that +interval, further messages up to the end of the interval are +discarded. The number of messages discarded is emitted at the end of +the interval (if there were any discards). + +**Setting this value to 0 turns off ratelimiting.** + +Note that it is *not recommended to turn off ratelimiting*, +except that you know for +sure journal database entries will never be corrupted. Without +ratelimiting, a corrupted systemd journal database may cause a kind +of denial of service We are stressing this point as multiple users +have reported us such problems with the journal database - in June +of 2013 and occasionally also after this time (up until the time of +this writing in January 2019). + + +Ratelimit.Burst +^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "20000", "no", "``$imjournalRatelimitBurst``" + +Specifies the maximum number of messages that can be emitted within +the ratelimit.interval interval. For further information, see +description there. + + +IgnorePreviousMessages +^^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "``$ImjournalIgnorePreviousMessages``" + +This option specifies whether imjournal should ignore messages +currently in journal and read only new messages. This option is only +used when there is no StateFile to avoid message loss. + + +DefaultSeverity +^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "severity", "5", "no", "``$ImjournalDefaultSeverity``" + +Some messages coming from journald don't have the SYSLOG_PRIORITY +field. These are typically the messages logged through journald's +native API. This option specifies the default severity for these +messages. Can be given either as a name or a number. Defaults to 'notice'. + + +DefaultFacility +^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "facility", "LOG_USER", "no", "``$ImjournalDefaultFacility``" + +Some messages coming from journald don't have the SYSLOG_FACILITY +field. These are typically the messages logged through journald's +native API. This option specifies the default facility for these +messages. Can be given either as a name or a number. Defaults to 'user'. + + +UsePidFromSystem +^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "0", "no", "none" + +Retrieves the trusted systemd parameter, _PID, instead of the user +systemd parameter, SYSLOG_PID, which is the default. +This option override the "usepid" option. +This is now deprecated. It is better to use usepid="syslog" instead. + + +UsePid +^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "both", "no", "none" + +Sets the PID source from journal. + +*syslog* + *imjournal* retrieves SYSLOG_PID from journal as PID number. + +*system* + *imjournal* retrieves _PID from journal as PID number. + +*both* + *imjournal* trying to retrieve SYSLOG_PID first. When it is not + available, it is also trying to retrieve _PID. When none of them is available, + message is parsed without PID number. + + +IgnoreNonValidStatefile +^^^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "on", "no", "none" + +When a corrupted statefile is read imjournal ignores the statefile and continues +with logging from the beginning of the journal (from its end if IgnorePreviousMessages +is on). After PersistStateInterval or when rsyslog is stopped invalid statefile +is overwritten with a new valid cursor. + + +WorkAroundJournalBug +^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "on", "no", "none" + +.. versionadded:: 8.37.0 + +**Deprecated.** This option was intended as temporary and has no effect now +(since 8.1910.0). Left for backwards compatibility only. + + +FSync +^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +.. versionadded:: 8.1908.0 + +When there is a hard crash, power loss or similar abrupt end of rsyslog process, +there is a risk of state file not being written to persistent storage or possibly +being corrupted. This then results in imjournal starting reading elsewhere then +desired and most probably message duplication. To mitigate this problem you can +turn this option on which will force state file writes to persistent physical +storage. Please note that fsync calls are costly, so especially with lower +PersistStateInterval value, this may present considerable performance hit. + + +Remote +^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +.. versionadded:: 8.1910.0 + +When this option is turned on, imjournal will pull not only all local journal +files (default behavior), but also any journal files on machine originating from +remote sources. + +defaultTag +^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +.. versionadded:: 8.2312.0 + +The DefaultTag option specifies the default value for the tag field. +In imjournal, this can happen when one of the following is missing: + +* identifier string provided by the application (SYSLOG_IDENTIFIER) or +* name of the process the journal entry originates from (_COMM) + +Under normal circumstances, at least one of the previously mentioned fields +is always part of the journal message. But there are some corner cases +where this is not the case. This parameter provides the ability to alter +the content of the tag field. + + +Input Module Parameters +======================= + +Parameters specific to the input module. + +Main +^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "journal", "no", "none" + +.. versionadded:: 8.2312.0 + +When this option is turned on within the input module, imjournal will run the +target ruleset in the main thread and will be stop taking input if the output +module is not accepting data. If multiple input moduels set `main` to true, only +the first one will be affected. The non `main` rulesets will run in the +background thread and not affected by the output state. + + + +Statistic Counter +================= + +.. _imjournal-statistic-counter: + +This plugin maintains :doc:`statistics <../rsyslog_statistic_counter>` for each listener and for each worker thread. The listener statistic is named "imjournal". + +The following properties are maintained for each listener: + +- **read** - total number of message read from journal since startup. + +- **submitted** - total number of messages submitted to main queue after reading from journal for processing + since startup. All records may not be submitted due to rate-limiting. + +- **discarded** - total number of messages that were read but not submitted to main queue due to rate-limiting. + +- **failed** - total number of failures to read messages from journal. + +- **poll_failed** - total number of journal poll failures. + +- **rotations** - total number of journal file rotations. + +- **recovery_attempts** - total number of recovery attempts by imjournal after unknown errors by closing and + re-opening journal. + +- **ratelimit_discarded_in_interval** - number of messages discarded due to rate-limiting within configured + rate-limiting interval. + +- **disk_usage_bytes** - total size of journal obtained from sd_journal_get_usage(). + +Here is an example output of corresponding imjournal impstat message, which is produced by loading imjournal +with default rate-limit interval and burst and running a docker container with log-driver as journald that +spews lots of logs to stdout: + +.. code-block:: none + + Jun 13 15:02:48 app1-1.example.com rsyslogd-pstats: imjournal: origin=imjournal submitted=20000 read=216557 + discarded=196557 failed=0 poll_failed=0 rotations=6 recovery_attempts=0 ratelimit_discarded_in_interval=196557 + disk_usage_bytes=106610688 + +Although these counters provide insight into imjournal end message submissions to main queue as well as losses due to +rate-limiting or other problems to extract messages from journal, they don't offer full visibility into journal end +issues. While these counters measure journal rotations and disk usage, they do not offer visibility into message +loss due to journal rate-limiting. sd_journal_* API does not provide any visibility into messages that are +discarded by the journal due to rate-limiting. Journald does emit a syslog message when log messages cannot make +it into the journal due to rate-limiting: + +.. code-block:: none + + Jun 13 15:50:32 app1-1.example.com systemd-journal[333]: Suppressed 102 messages from /system.slice/docker.service + +Such messages can be processed after they are read through imjournal to get a signal for message loss due to journal +end rate-limiting using a dynamic statistics counter for such log lines with a rule like this: + +.. code-block:: none + + dyn_stats(name="journal" resettable="off") + if $programname == 'journal' and $msg contains 'Suppressed' and $msg contains 'messages from' then { + set $.inc = dyn_inc("journal", "suppressed_count"); + } + +Caveats/Known Bugs: +=================== + +- As stated above, a corrupted systemd journal database can cause major + problems, depending on what the corruption results in. This is beyond + the control of the rsyslog team. + +- imjournal does not check if messages received actually originated + from rsyslog itself (via omjournal or other means). Depending on + configuration, this can also lead to a loop. With imuxsock, this + problem does not exist. + + +Build Requirements: +=================== + +Development headers for systemd, version >= 197. + + +Example 1 +========= + +The following example shows pulling structured imjournal messages and +saving them into /var/log/ceelog. + +.. code-block:: none + + module(load="imjournal" PersistStateInterval="100" + StateFile="/path/to/file") #load imjournal module + module(load="mmjsonparse") #load mmjsonparse module for structured logs + + template(name="CEETemplate" type="string" string="%TIMESTAMP% %HOSTNAME% %syslogtag% @cee: %$!all-json%\n" ) #template for messages + + action(type="mmjsonparse") + action(type="omfile" file="/var/log/ceelog" template="CEETemplate") + + +Example 2 +========= + +The following example is the same as `Example 1`, but with the input module. + +.. code-block:: none + + ruleset(name="imjournam-example" queue.type="direct"){ + action(type="mmjsonparse") + action(type="omfile" file="/var/log/ceelog" template="CEETemplate") + } + + input( + type="imjournal" + ruleset="imjournam-example" + main="on" + ) diff --git a/source/configuration/modules/imkafka.rst b/source/configuration/modules/imkafka.rst new file mode 100644 index 0000000..e589b49 --- /dev/null +++ b/source/configuration/modules/imkafka.rst @@ -0,0 +1,177 @@ +******************************* +imkafka: read from Apache Kafka +******************************* + +=========================== =========================================================================== +**Module Name:** **imkafka** +**Author:** Andre Lorbach <alorbach@adiscon.com> +**Available since:** 8.27.0 +=========================== =========================================================================== + + +Purpose +======= + +The imkafka plug-in implements an Apache Kafka consumer, permitting +rsyslog to receive data from Kafka. + + +Configuration Parameters +======================== + +Note that imkafka supports some *Array*-type parameters. While the parameter +name can only be set once, it is possible to set multiple values with that +single parameter. + +For example, to select a broker, you can use + +.. code-block:: none + + input(type="imkafka" topic="mytopic" broker="localhost:9092" consumergroup="default") + +which is equivalent to + +.. code-block:: none + + input(type="imkafka" topic="mytopic" broker=["localhost:9092"] consumergroup="default") + +To specify multiple values, just use the bracket notation and create a +comma-delimited list of values as shown here: + +.. code-block:: none + + input(type="imkafka" topic="mytopic" + broker=["localhost:9092", + "localhost:9093", + "localhost:9094"] + ) + + +.. note:: + + Parameter names are case-insensitive. + + +Module Parameters +----------------- + +Currently none. + + +Action Parameters +----------------- + +Broker +^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "array", "localhost:9092", "no", "none" + +Specifies the broker(s) to use. + + +Topic +^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "none", "yes", "none" + +Specifies the topic to produce to. + + +ConfParam +^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "array", "none", "no", "none" + +Permits to specify Kafka options. Rather than offering a myriad of +config settings to match the Kafka parameters, we provide this setting +here as a vehicle to set any Kafka parameter. This has the big advantage +that Kafka parameters that come up in new releases can immediately be used. + +Note that we use librdkafka for the Kafka connection, so the parameters +are actually those that librdkafka supports. As of our understanding, this +is a superset of the native Kafka parameters. + + +ConsumerGroup +^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "none", "no", "none" + +With this parameter the group.id for the consumer is set. All consumers +sharing the same group.id belong to the same group. + + +Ruleset +^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "none", "no", "none" + +Specifies the ruleset to be used. + + +ParseHostname +^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +.. versionadded:: 8.38.0 + +If this parameter is set to on, imkafka will parse the hostname in log +if it exists. The result can be retrieved from $hostname. If it's off, +for compatibility reasons, the local hostname is used, same as the previous +version. + + +Caveats/Known Bugs +================== + +- currently none + + +Examples +======== + +Example 1 +--------- + +In this sample a consumer for the topic static is created and will forward the messages to the omfile action. + +.. code-block:: none + + module(load="imkafka") + input(type="imkafka" topic="static" broker="localhost:9092" + consumergroup="default" ruleset="pRuleset") + + ruleset(name="pRuleset") { + action(type="omfile" file="path/to/file") + } diff --git a/source/configuration/modules/imklog.rst b/source/configuration/modules/imklog.rst new file mode 100644 index 0000000..2de34ed --- /dev/null +++ b/source/configuration/modules/imklog.rst @@ -0,0 +1,230 @@ +******************************* +imklog: Kernel Log Input Module +******************************* + +=========================== =========================================================================== +**Module Name:** **imklog** +**Author:** `Rainer Gerhards <https://rainer.gerhards.net/>`_ <rgerhards@adiscon.com> +=========================== =========================================================================== + + +Purpose +======= + +Reads messages from the kernel log and submits them to the syslog +engine. + + +Configuration Parameters +======================== + +.. note:: + + Parameter names are case-insensitive. + + +Module Parameters +----------------- + +InternalMsgFacility +^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "facility", "(see description)", "no", "``$KLogInternalMsgFacility``" + +The facility which messages internally generated by imklog will +have. imklog generates some messages of itself (e.g. on problems, +startup and shutdown) and these do not stem from the kernel. +Historically, under Linux, these too have "kern" facility. Thus, on +Linux platforms the default is "kern" while on others it is +"syslogd". You usually do not need to specify this configuration +directive - it is included primarily for few limited cases where it +is needed for good reason. Bottom line: if you don't have a good idea +why you should use this setting, do not touch it. + + +PermitNonKernelFacility +^^^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "``$KLogPermitNonKernelFacility``" + +At least under BSD the kernel log may contain entries with +non-kernel facilities. This setting controls how those are handled. +The default is "off", in which case these messages are ignored. +Switch it to on to submit non-kernel messages to rsyslog processing. + + +ConsoleLogLevel +^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "-1", "no", "``$klogConsoleLogLevel``" + +Sets the console log level. If specified, only messages with up to +the specified level are printed to the console. The default is -1, +which means that the current settings are not modified. To get this +behavior, do not specify $klogConsoleLogLevel in the configuration +file. Note that this is a global parameter. Each time it is changed, +the previous definition is re-set. The one activate will be that one +that is active when imklog actually starts processing. In short +words: do not specify this directive more than once! + +**Linux only**, ignored on other platforms (but may be specified) + + +ParseKernelTimestamp +^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "``$klogParseKernelTimestamp``" + +If enabled and the kernel creates a timestamp for its log messages, +this timestamp will be parsed and converted into regular message time +instead to use the receive time of the kernel message (as in 5.8.x +and before). Default is 'off' to prevent parsing the kernel timestamp, +because the clock used by the kernel to create the timestamps is not +supposed to be as accurate as the monotonic clock required to convert +it. Depending on the hardware and kernel, it can result in message +time differences between kernel and system messages which occurred at +same time. + + +KeepKernelTimestamp +^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "``$klogKeepKernelTimestamp``" + +If enabled, this option causes to keep the [timestamp] provided by +the kernel at the begin of in each message rather than to remove it, +when it could be parsed and converted into local time for use as +regular message time. Only used, when $klogParseKernelTimestamp is +on. + + +LogPath +^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "(see description)", "no", "``$klogpath``" + +Defines the path to the log file that is used. +If this parameter is not set a default will be used. +On Linux "/proc/kmsg" and else "/dev/klog". + + +RatelimitInterval +^^^^^^^^^^^^^^^^^ +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "0", "no", "none" + +.. versionadded:: 8.35.0 + +The rate-limiting interval in seconds. Value 0 turns off rate limiting. +Set it to a number of seconds (5 recommended) to activate rate-limiting. + + +RatelimitBurst +^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "10000", "no", "none" + +.. versionadded:: 8.35.0 + +Specifies the rate-limiting burst in number of messages. Set it high to +preserve all bootup messages. + + +Caveats/Known Bugs +================== + +This is obviously platform specific and requires platform drivers. +Currently, imklog functionality is available on Linux and BSD. + +This module is **not supported on Solaris** and not needed there. For +Solaris kernel input, use :doc:`imsolaris <imsolaris>`. + + +Example 1 +========= + +The following sample pulls messages from the kernel log. All parameters +are left by default, which is usually a good idea. Please note that +loading the plugin is sufficient to activate it. No directive is needed +to start pulling kernel messages. + +.. code-block:: none + + module(load="imklog") + + +Example 2 +========= + +The following sample adds a ratelimiter. The burst and interval are +set high to allow for a large volume of messages on boot. + +.. code-block:: none + + module(load="imklog" RatelimitBurst="5000" RatelimitInterval="5") + + +Unsupported |FmtObsoleteName| directives +======================================== + +.. function:: $DebugPrintKernelSymbols on/off + + Linux only, ignored on other platforms (but may be specified). + Defaults to off. + +.. function:: $klogLocalIPIF + + This directive is no longer supported. Instead, use the global + $localHostIPIF directive instead. + + +.. function:: $klogUseSyscallInterface on/off + + Linux only, ignored on other platforms (but may be specified). + Defaults to off. + +.. function:: $klogSymbolsTwice on/off + + Linux only, ignored on other platforms (but may be specified). + Defaults to off. + + diff --git a/source/configuration/modules/imkmsg.rst b/source/configuration/modules/imkmsg.rst new file mode 100644 index 0000000..318c9f5 --- /dev/null +++ b/source/configuration/modules/imkmsg.rst @@ -0,0 +1,188 @@ +********************************** +imkmsg: /dev/kmsg Log Input Module +********************************** + +=========================== =========================================================================== +**Module Name:** **imkmsg** +**Author:** `Rainer Gerhards <https://rainer.gerhards.net/>`_ <rgerhards@adiscon.com> + Milan Bartos <mbartos@redhat.com> +=========================== =========================================================================== + + +Purpose +======= + +Reads messages from the /dev/kmsg structured kernel log and submits them +to the syslog engine. + +The printk log buffer contains log records. These records are exported +by /dev/kmsg device as structured data in the following format: +"level,sequnum,timestamp;<message text>\\n" +There could be continuation lines starting with space that contains +key/value pairs. +Log messages are parsed as necessary into rsyslog msg\_t structure. +Continuation lines are parsed as json key/value pairs and added into +rsyslog's message json representation. + + +Configuration Parameters +======================== + +.. note:: + + Parameter names are case-insensitive. + +Module Parameters +----------------- + +Mode +^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "parseKernelTimestamp", "no", "none" + +.. versionadded:: 8.2312.0 + +This parameter configures which timestamps will be used. It is an advanced +setting and most users should probably keep the default mode ("startup"). + +The linux kernel message buffer contains a timestamp, which reflects the time +the message was created. However, this is not the "normal" time one expects, but +in a so-called monotonic time in seconds since kernel start. For a datacenter +system which runs 24 hours by 7 days a week, kernel time and actual +wall clock time is mostly the same. Problems may occur during daylight +savings time switches. + +For desktops and laptops this is not necessarily the case. The reason is, as +it looks, that during low power states (energy save mode, hibernation), kernel +monotonic time **does not advance**. This is also **not** corrected when the +system comes back to normal operations. As such, on systems using low power +states from time to time, kernel time and wallclock time drift apart. We have +been told cases where this is in the magnitude of days. Just think about +desktops which are in hibernate during the night, missing several hours +each day. So this is a real-world problem. + +To work around this, we usually do **not** use the kernel timstamp when +we calculate the message time. Instead, we use wallclock time (obtained +from the respective linux timer) of the instant when imkmsg reads the +message from the kernel log. As message creation and imkmsg reading it +is usually in very close time proximity, this approach works very well. + +**However**, this is not helpful for e.g. early boot messages. These +were potentially generated some seconds to a minute or two before rsyslog +startup. To provide a proper meaning of time for these events, we use +the kernel timstamp instead of wallclock time during rsyslog startup. +This is most probably correct, because it is extremely unlikely (close +to impossible) that the system entered a low-power state before rsyslog +startup. + +**Note well:** When rsyslog is restarted during normal system operations, +existing imkmsg messages are re-read and this is done with the kernel +timestamp. This causes message duplication, but is what imkmsg always +did. It is planned to provide ehance the module to improve this +behaviour. This documentation page here will be updated when changes are +made. + +The *parseKernelTimestamp* parameter provides fine-grain control over +the processing of kernel vs. wallclock time. Adjustments should only +be needed rarely and if there is a dedicated use case for it. So use +this parameter only if you have a good reason to do so. + +Supported modes are: + +* **startup** - This is the **DEFAULT setting**. + + Uses the kernel time stamp during the initial read + loop of /dev/kmsg, but uses system wallclock time once the initial + read is completed. This behavior is described in the text above in + detail. + +* **on** - kernel timestamps are always used and wallclock time never + +* **off** - kernel timestamps are never used, system wallclock time is + always used + + +readMode +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "full-boot", "no", "none" + +.. versionadded:: 8.2312.0 + +This parameter permits to control when imkmsg reads the full kernel. + +It provides the following options: + +* **full-boot** - (default) read full klog, but only "immediately" after + boot. "Immediately" is hereby meant in seconds of system uptime + given in "expectedBootCompleteSeconds" + +* **full-always** - read full klog on every rsyslog startup. Most probably + causes message duplication + +* **new-only** - never emit existing kernel log message, read only new ones. + +Note that some message loss can happen if rsyslog is stopped in "full-boot" and +"new-only" read mode. The longer rsyslog is inactive, the higher the message +loss probability and potential number of messages lost. For typical restart +scenarios, this should be minimal. On HUP, no message loss occurs as rsyslog +is not actually stopped. + + +expectedBootCompleteSeconds +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "positive integer", "90", "no", "none" + +.. versionadded:: 8.2312.0 + +This parameter works in conjunction with **readMode** and specifies how +many seconds after startup the system should be considered to be +"just booted", which means in **readMode** "full-boot" imkmsg reads and +forwards to rsyslog processing all existing messages. + +In any other **readMode** the **expectedBootCompleteSettings** is +ignored. + +Caveats/Known Bugs: +=================== + +This module cannot be used together with imklog module. When using one of +them, make sure the other one is not enabled. + +This is Linux specific module and requires /dev/kmsg device with +structured kernel logs. + +This module does not support rulesets. All messages are delivered to the +default rulseset. + + + +Examples +======== + +The following sample pulls messages from the /dev/kmsg log device. All +parameters are left by default, which is usually a good idea. Please +note that loading the plugin is sufficient to activate it. No directive +is needed to start pulling messages. + +.. code-block:: none + + module(load="imkmsg") + + diff --git a/source/configuration/modules/immark.rst b/source/configuration/modules/immark.rst new file mode 100644 index 0000000..cbae437 --- /dev/null +++ b/source/configuration/modules/immark.rst @@ -0,0 +1,41 @@ +********************************** +immark: Mark Message Input Module +********************************** + +=========================== =========================================================================== +**Module Name:** **immark** +**Author:** `Rainer Gerhards <http://www.gerhards.net/rainer>`_ <rgerhards@adiscon.com> +=========================== =========================================================================== + +Purpose +======= + +This module provides the ability to inject periodic "mark" messages to +the input of rsyslog. This is useful to allow for verification that +the logging system is functioning. + +Configuration Parameters +======================== + +.. note:: + + Parameter names are case-insensitive. + +Module Parameters +----------------- + +interval +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "max", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "1200", "", "no", "``$MarkMessagePeriod``" + +Specifies the mark message injection interval in seconds. + +.. seealso:: + + The Action Parameter ``action.writeAllMarkMessages`` in :doc:`../actions`. diff --git a/source/configuration/modules/impcap.rst b/source/configuration/modules/impcap.rst new file mode 100644 index 0000000..99e0f49 --- /dev/null +++ b/source/configuration/modules/impcap.rst @@ -0,0 +1,255 @@ + +******************************* +Impcap: network traffic capture +******************************* + +==================== ===================================== +**Module Name:** **impcap** +**Author:** Theo Bertin <theo.bertin@advens.fr> +==================== ===================================== + +Purpose +======= + +Impcap is an input module based upon `tcpdump's libpcap <https://www.tcpdump.org/>`_ library for network traffic capture. + +Its goal is to capture network traffic with efficiency, parse network packets metadata AND data, and allow users/modules +to make full use of it. + + + +Configuration Parameters +======================== + +.. note:: + Parameter names are case-insensitive + +Module Parameter +---------------- + +metadata_container +^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "!impcap", "no", "none" + +Defines the container to place all the parsed metadata of the network packet. + +.. Warning:: + if overwritten, this parameter should always begin with '!' to define the JSON object accompanying messages. No checks are done to ensure that + and not complying with this rule will prevent impcap/rsyslog from running, or will result in unexpected behaviours. + + +data_container +^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "!data", "no", "none" + +Defines the container to place all the data of the network packet. 'data' here defines everything above transport layer +in the OSI model, and is a string representation of the hexadecimal values of the stream. + +.. Warning:: + if overwritten, this parameter should always begin with '!' to define the JSON object accompanying messages. No checks are done to ensure that + and not complying with this rule will prevent impcap/rsyslog from running, or will result in unexpected behaviours. + + + +snap_length +^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "number", "65535", "no", "none" + +Defines the maximum size of captured packets. +If captured packets are longer than the defined value, they will be capped. +Default value allows any type of packet to be captured entirely but can be much shorter if only metadata capture is +desired (500 to 2000 should still be safe, depending on network protocols). +Be wary though, as impcap won't be able to parse metadata correctly if the value is not high enough. + + +Input Parameters +---------------- + +interface +^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +This parameter specifies the network interface to listen to. **If 'interface' is not specified, 'file' must be in order +for the module to run.** + +.. note:: + The name must be a valid network interface on the system (such as 'lo'). + see :ref:`Supported interface types` for an exhaustive list of all supported interface link-layer types. + + +file +^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +This parameter specifies a pcap file to read. +The file must respect the `pcap file format specification <https://www.tcpdump.org/pcap/pcap.html>`_. **If 'file' is not specified, 'interface' must be in order +for the module to run.** + +.. Warning:: + This functionality is not intended for production environnments, + it is designed for development/tests. + + +promiscuous +^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "boolean", "off", "no", "none" + +When a valid interface is provided, sets the capture to promiscuous for this interface. + +.. warning:: + Setting your network interface to promiscuous can come against your local laws and + regulations, maintainers cannot be held responsible for improper use of the module. + + +filter +^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "none", "no", "none" + +Set a filter for the capture. +Filter semantics are defined `on pcap manpages <https://www.tcpdump.org/manpages/pcap-filter.7.html>`_. + + +tag +^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +Set a tag to messages coming from this input. + + +ruleset +^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +Assign messages from thi simput to a specific Rsyslog ruleset. + + +.. _no_buffer: + +no_buffer +^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "boolean", "off", "no", "none" + +Disable buffering during capture. +By default, impcap asks the system to bufferize packets (see parameters :ref:`buffer_size`, :ref:`buffer_timeout` and +:ref:`packet_count`), this parameter disables buffering completely. This means packets will be handled as soon as they +arrive, but impcap will make more system calls to get them and might miss some depending on the incoming rate and system +performances. + + +.. _buffer_size: + +buffer_size +^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "number (octets)", "15740640", "no", "none" + +Set a buffer size in bytes to the capture handle. +This parameter is only relevant when :ref:`no_buffer` is not active, and should be set depending on input packet rates, +:ref:`buffer_timeout` and :ref:`packet_count` values. + + +.. _buffer_timeout: + +buffer_timeout +^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "number (ms)", "10", "no", "none" + +Set a timeout in milliseconds between two system calls to get bufferized packets. This parameter prevents low input rate +interfaces to keep packets in buffers for too long, but does not guarantee fetch every X seconds (see `pcap manpage <https://www.tcpdump.org/manpages/pcap.3pcap.html>`_ for more details). + + + +.. _packet_count: + +packet_count +^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "number", "5", "no", "none" + +Set a maximum number of packets to process at a time. This parameter allows to limit batch calls to a maximum of X +packets at a time. + + +.. _Supported interface types: + +Supported interface types +========================= + +Impcap currently supports IEEE 802.3 Ethernet link-layer type interfaces. +Please contact the maintainer if you need a different interface type ! diff --git a/source/configuration/modules/improg.rst b/source/configuration/modules/improg.rst new file mode 100644 index 0000000..a0fd746 --- /dev/null +++ b/source/configuration/modules/improg.rst @@ -0,0 +1,170 @@ +**************************************** +improg: Program integration input module +**************************************** + +================ ============================================================== +**Module Name:** **improg** +**Authors:** Jean-Philippe Hilaire <jean-philippe.hilaire@pmu.fr> & Philippe Duveau <philippe.duveau@free.fr> +================ ============================================================== + + +Purpose +======= + +This module allows rsyslog to spawn external command(s) and consume message +from pipe(s) (stdout of the external process). + +**Limitation:** `select()` seems not to support usage of `printf(...)` or +`fprintf(stdout,...)`. Only `write(STDOUT_FILENO,...)` seems to be efficient. + +The imput module consume pipes form all external programs in a mono-threaded +`runInput` method. This means that data treatments will be serialized. + +Optionally, the module manage the external program through keyword sent to +it using a second pipe to stdin of the external process. + +An operational sample in C can be found @ "github.com/pduveau/jsonperfmon" + +Also a bash's script is provided as tests/improg-simul.sh. The `echo` and `read` (built-in) can be used to communicate with the module. +External commands can not be used to communicate. `printf` is unable to send data directly to the module but can used through a variable and `echo`. + + +Compile +======= + +To successfully compile improg module. + + ./configure --enable-improg ... + +Configuration Parameters +======================== + +Action Parameters +----------------- + +Binary +^^^^^^ + +.. csv-table:: + :header: "type", "mandatory", "format", "default" + :widths: auto + :class: parameter-table + + "string", "yes", "command arguments...", + +Command line : external program and arguments + +Tag +^^^ + +.. csv-table:: + :header: "type", "mandatory", "format", "default" + :widths: auto + :class: parameter-table + + "string", "yes", ,"none" + +The tag to be assigned to messages read from this file. If you would like to +see the colon after the tag, you need to include it when you assign a tag +value, like so: ``tag="myTagValue:"``. + +Facility +^^^^^^^^ + +.. csv-table:: + :header: "type", "mandatory", "format", "default" + :widths: auto + :class: parameter-table + + "string", "no", "facility\|number", "local0" + +The syslog facility to be assigned to messages read from this file. Can be +specified in textual form (e.g. ``local0``, ``local1``, ...) or as numbers (e.g. +16 for ``local0``). Textual form is suggested. + +Severity +^^^^^^^^ + +.. csv-table:: + :header: "type", "mandatory", "format", "default" + :widths: auto + :class: parameter-table + + "string", "no", "severity\|number", "notice" + +The syslog severity to be assigned to lines read. Can be specified +in textual form (e.g. ``info``, ``warning``, ...) or as numbers (e.g. 6 +for ``info``). Textual form is suggested. + +confirmMessages +^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "mandatory", "format", "default" + :widths: auto + :class: parameter-table + + "binary", "no", "on\|off", "on" + +Specifies whether the external program needs feedback from rsyslog via stdin. +When this switch is set to "on", rsyslog confirms each received message. +This feature facilitates error handling: instead of having to implement a retry +logic, the external program can rely on the rsyslog queueing capabilities. +The program receives a line with the word ``ACK`` from its standard input. + +Also, the program receives a ``STOP`` when rsyslog ask the module to stop. + +signalOnClose +^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "mandatory", "format", "default" + :widths: auto + :class: parameter-table + + "binary", "no", "on\|off", "off" + +Specifies whether a TERM signal must be sent to the external program before +closing it (when either the worker thread has been unscheduled, a restart +of the program is being forced, or rsyslog is about to shutdown). + +closeTimeout +^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "mandatory", "format", "default" + :widths: auto + :class: parameter-table + + "number", "no", ,"200" + +Specifies whether a KILL signal must be sent to the external program in case +it does not terminate within the timeout indicated by closeTimeout_ +(when either the worker thread has been unscheduled, a restart of the program +is being forced, or rsyslog is about to shutdown). + +killUnresponsive +^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "mandatory", "format", "default" + :widths: auto + :class: parameter-table + + "binary", "no", "on\|off", "on" + +Specifies whether a KILL signal must be sent to the external program in case +it does not terminate within the timeout indicated by closeTimeout +(when either the worker thread has been unscheduled, a restart of the program +is being forced, or rsyslog is about to shutdown). + +Stop sequence +============= + +1. If `confirmMessages` is set to on, a `STOP` is written in stdin of the child. +2. If `signalOnClose` is set to "on", a TERM signal is sent to the child. +3. The pipes with the child process are closed (the child will receive EOF on stdin), +4. Then, rsyslog waits for the child process to terminate during closeTimeout, +5. If the child has not terminated within the timeout, a KILL signal is sent to it. + + diff --git a/source/configuration/modules/impstats.rst b/source/configuration/modules/impstats.rst new file mode 100644 index 0000000..ca95609 --- /dev/null +++ b/source/configuration/modules/impstats.rst @@ -0,0 +1,405 @@ +*********************************************************** +impstats: Generate Periodic Statistics of Internal Counters +*********************************************************** + +=========================== =========================================================================== +**Module Name:** **impstats** +**Author:** `Rainer Gerhards <https://rainer.gerhards.net/>`_ <rgerhards@adiscon.com> +=========================== =========================================================================== + + +Purpose +======= + +This module provides periodic output of rsyslog internal counters. + +The set of available counters will be output as a set of syslog +messages. This output is periodic, with the interval being configurable +(default is 5 minutes). Be sure that your configuration records the +counter messages (default is syslog.=info). Besides logging to the +regular syslog stream, the module can also be configured to write +statistics data into a (local) file. + +When logging to the regular syslog stream, impstats records are emitted +just like regular log messages. As such, +counters increase when processing these messages. This must be taken into +consideration when testing and troubleshooting. + +Note that loading this module has some impact on rsyslog performance. +Depending on settings, this impact may be noticeable for high-load +environments, but in general the overhead is pretty light. + +**Note that there is a** `rsyslog statistics online +analyzer <http://www.rsyslog.com/impstats-analyzer/>`_ **available.** It +can be given a impstats-generated file and will return problems it +detects. Note that the analyzer cannot replace a human in getting things +right, but it is expected to be a good aid in starting to understand and +gain information from the pstats logs. + +The rsyslog website has an overview of available `rsyslog +statistic counters <http://rsyslog.com/rsyslog-statistic-counter/>`_. +When browsing this page, please be sure to take note of which rsyslog +version is required to provide a specific counter. Counters are +continuously being added, and older versions do not support everything. + + +Notable Features +================ + +- :ref:`impstats-statistic-counter` + + + + +Configuration Parameters +======================== + +The configuration parameters for this module are designed for tailoring +the method and process for outputting the rsyslog statistics to file. + +.. note:: + + Parameter names are case-insensitive. + +.. note:: + + This module supports module parameters, only. + + +Module Parameters +----------------- + +Interval +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "300", "no", "none" + +Sets the interval, in **seconds** at which messages are generated. +Please note that the actual interval may be a bit longer. We do not +try to be precise and so the interval is actually a sleep period +which is entered after generating all messages. So the actual +interval is what is configured here plus the actual time required to +generate messages. In general, the difference should not really +matter. + + +Facility +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "5", "no", "none" + +The numerical syslog facility code to be used for generated +messages. Default is 5 (syslog). This is useful for filtering +messages. + + +Severity +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "6", "no", "none" + +The numerical syslog severity code to be used for generated +messages. Default is 6 (info).This is useful for filtering messages. + + +ResetCounters +^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +When set to "on", counters are automatically reset after they are +emitted. In that case, the contain only deltas to the last value +emitted. When set to "off", counters always accumulate their values. +Note that in auto-reset mode not all counters can be reset. Some +counters (like queue size) are directly obtained from internal object +and cannot be modified. Also, auto-resetting introduces some +additional slight inaccuracies due to the multi-threaded nature of +rsyslog and the fact that for performance reasons it cannot serialize +access to counter variables. As an alternative to auto-reset mode, +you can use rsyslog's statistics manipulation scripts to create delta +values from the regular statistic logs. This is the suggested method +if deltas are not necessarily needed in real-time. + + +Format +^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "legacy", "no", "none" + +.. versionadded:: 8.16.0 + +Specifies the format of emitted stats messages. The default of +"legacy" is compatible with pre v6-rsyslog. The other options provide +support for structured formats (note the "cee" is actually "project +lumberjack" logging). + +The json-elasticsearch format supports the broken ElasticSearch +JSON implementation. ES 2.0 no longer supports valid JSON and +disallows dots inside names. The "json-elasticsearch" format +option replaces those dots by the bang ("!") character. So +"discarded.full" becomes "discarded!full". +Options: json/json-elasticsearch/cee/legacy + + +log.syslog +^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "on", "no", "none" + +This is a boolean setting specifying if data should be sent to the +usual syslog stream. This is useful if custom formatting or more +elaborate processing is desired. However, output is placed under the +same restrictions as regular syslog data, especially in regard to the +queue position (stats data may sit for an extended period of time in +queues if they are full). If set "off", then you cannot bind the module to +ruleset. + + +log.file +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +If specified, statistics data is written to the specified file. For +robustness, this should be a local file. The file format cannot be +customized, it consists of a date header, followed by a colon, +followed by the actual statistics record, all on one line. Only very +limited error handling is done, so if things go wrong stats records +will probably be lost. Logging to file an be a useful alternative if +for some reasons (e.g. full queues) the regular syslog stream method +shall not be used solely. Note that turning on file logging does NOT +turn off syslog logging. If that is desired log.syslog="off" must be +explicitly set. + + +Ruleset +^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "none", "no", "none" + +Binds the listener to a specific :doc:`ruleset <../../concepts/multi_ruleset>`. + +**Note** that setting ``ruleset`` and ``log.syslog="off"`` are mutually +exclusive because syslog stream processing must be enabled to use a ruleset. + + +Bracketing +^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +.. versionadded:: 8.4.1 + +This is a utility setting for folks who post-process impstats logs +and would like to know the begin and end of a block of statistics. +When "bracketing" is set to "on", impstats issues a "BEGIN" message +before the first counter is issued, then all counter values +are issued, and then an "END" message follows. As such, if and only if messages +are kept in sequence, a block of stats counts can easily be identified +by those BEGIN and END messages. + +**Note well:** in general, sequence of syslog messages is **not** +strict and is not ordered in sequence of message generation. There +are various occasion that can cause message reordering, some +examples are: + +* using multiple threads +* using UDP forwarding +* using relay systems, especially with buffering enabled +* using disk-assisted queues + +This is not a problem with rsyslog, but rather the way a concurrent +world works. For strict order, a specific order predicate (e.g. a +sufficiently fine-grained timestamp) must be used. + +As such, BEGIN and END records may actually indicate the begin and +end of a block of statistics - or they may *not*. Any order is possible +in theory. So the bracketing option does not in all cases work as +expected. This is the reason why it is turned off by default. + +*However*, bracketing may still be useful for many use cases. First +and foremost, while there are many scenarios in which messages become +reordered, in practice it happens relatively seldom. So most of the +time the statistics records will come in as expected and actually +will be bracketed by the BEGIN and END messages. Consequently, if +an application can handle occasional out-of-order delivery (e.g. by +graceful degradation), bracketing may actually be a great solution. +It is, however, very important to know and +handle out of order delivery. For most real-world deployments, +a good way to handle it is to ignore unexpected +records and use the previous values for ones missing in the current +block. To guard against two or more blocks being mixed, it may also +be a good idea to never reset a value to a lower bound, except when +that lower bound is seen consistently (which happens due to a +restart). Note that such lower bound logic requires *resetCounters* +to be set to off. + + +.. _impstats-statistic-counter: + +Statistic Counter +================= + +The impstats plugin gathers some internal :doc:`statistics <../rsyslog_statistic_counter>`. +They have different names depending on the actual statistics. Obviously, they do not +relate to the plugin itself but rather to a broader object – most notably the +rsyslog process itself. The "resource-usage" counter maintains process +statistics. They base on the getrusage() system call. The counters are +named like getrusage returned data members. So for details, looking them +up in "man getrusage" is highly recommended, especially as value may be +different depending on the platform. A getrusage() call is done immediately +before the counter is emitted. The following individual counters are +maintained: + +- ``utime`` - this is the user time in microseconds (thus the timeval structure combined) +- ``stime`` - again, time given in microseconds +- ``maxrss`` +- ``minflt`` +- ``majflt`` +- ``inblock`` +- ``outblock`` +- ``nvcsw`` +- ``nivcsw`` +- ``openfiles`` - number of file handles used by rsyslog; includes actual files, sockets and others + + +Caveats/Known Bugs +================== + +- This module MUST be loaded right at the top of rsyslog.conf, + otherwise stats may not get turned on in all places. + + +Examples +======== + +Load module, send stats data to syslog stream +--------------------------------------------- + +This activates the module and records messages to /var/log/rsyslog-stats +in 10 minute intervals: + +.. code-block:: none + + module(load="impstats" + interval="600" + severity="7") + + # to actually gather the data: + syslog.=debug /var/log/rsyslog-stats + + +Load module, send stats data to local file +------------------------------------------ + +Here, the default interval of 5 minutes is used. However, this time, stats +data is NOT emitted to the syslog stream but to a local file instead. + +.. code-block:: none + + module(load="impstats" + interval="600" + severity="7" + log.syslog="off" + # need to turn log stream logging off! + log.file="/path/to/local/stats.log") + + +Load module, send stats data to local file and syslog stream +------------------------------------------------------------ + +Here we log to both the regular syslog log stream as well as a +file. Within the log stream, we forward the data records to another +server: + +.. code-block:: none + + module(load="impstats" + interval="600" + severity="7" + log.file="/path/to/local/stats.log") + + syslog.=debug @central.example.net + + +Explanation of output +===================== + +Example output for illustration:: + + Sep 17 11:43:49 localhost rsyslogd-pstats: imuxsock: submitted=16 + Sep 17 11:43:49 localhost rsyslogd-pstats: main Q: size=1 enqueued=2403 full=0 maxqsize=2 + +Explanation: + +All objects are shown in the results with a separate counter, one object per +line. + +Line 1: shows details for + +- ``imuxsock``, an object +- ``submitted=16``, a counter showing that 16 messages were received by the + imuxsock object. + +Line 2: shows details for the main queue: + +- ``main Q``, an object +- ``size``, messages in the queue +- ``enqueued``, all received messages thus far +- ``full``, how often was the queue was full +- ``maxqsize``, the maximum amount of messages that have passed through the + queue since rsyslog was started + +See Also +======== + +- `rsyslog statistics + counter <http://www.rsyslog.com/rsyslog-statistic-counter/>`_ +- `impstats delayed or + lost <http://www.rsyslog.com/impstats-delayed-or-lost/>`_ - cause and + cure diff --git a/source/configuration/modules/imptcp.rst b/source/configuration/modules/imptcp.rst new file mode 100644 index 0000000..e495155 --- /dev/null +++ b/source/configuration/modules/imptcp.rst @@ -0,0 +1,711 @@ +************************ +imptcp: Plain TCP Syslog +************************ + +=========================== =========================================================================== +**Module Name:** **imptcp** +**Author:** `Rainer Gerhards <https://rainer.gerhards.net/>`_ <rgerhards@adiscon.com> +=========================== =========================================================================== + + +Purpose +======= + +Provides the ability to receive syslog messages via plain TCP syslog. +This is a specialised input plugin tailored for high performance on +Linux. It will probably not run on any other platform. Also, it does not +provide TLS services. Encryption can be provided by using +`stunnel <rsyslog_stunnel.html>`_. + +This module has no limit on the number of listeners and sessions that +can be used. + + +Notable Features +================ + +- :ref:`imptcp-statistic-counter` +- :ref:`error-messages` + + +Configuration Parameters +======================== + +.. note:: + + Parameter names are case-insensitive. + + +Module Parameters +----------------- + +Threads +^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "2", "no", "``$InputPTCPServerHelperThreads``" + +Number of helper worker threads to process incoming messages. These +threads are utilized to pull data off the network. On a busy system, +additional helper threads (but not more than there are CPUs/Cores) +can help improving performance. The default value is two, which means +there is a default thread count of three (the main input thread plus +two helpers). No more than 16 threads can be set (if tried to, +rsyslog always resorts to 16). + + +MaxSessions +^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "0", "no", "none" + +Maximum number of open sessions allowed. This is inherited to each +"input()" config, however it is not a global maximum, rather just +setting the default per input. + +A setting of zero or less than zero means no limit. + +ProcessOnPoller +^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "on", "no", "none" + +Instructs imptcp to process messages on poller thread opportunistically. +This leads to lower resource footprint(as poller thread doubles up as +message-processing thread too). "On" works best when imptcp is handling +low ingestion rates. + +At high throughput though, it causes polling delay(as poller spends time +processing messages, which keeps connections in read-ready state longer +than they need to be, filling socket-buffer, hence eventually applying +backpressure). + +It defaults to allowing messages to be processed on poller (for backward +compatibility). + + +Input Parameters +---------------- + +These parameters can be used with the "input()" statement. They apply to +the input they are specified with. + + +Port +^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "none", "no", "``$InputPTCPServerRun``" + +Select a port to listen on. It is an error to specify +both `path` and `port`. + + +Path +^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "none", "no", "none" + +A path on the filesystem for a unix domain socket. It is an error to specify +both `path` and `port`. + + +DiscardTruncatedMsg +^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +When a message is split because it is to long the second part is normally +processed as the next message. This can cause Problems. When this parameter +is turned on the part of the message after the truncation will be discarded. + +FileOwner +^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "UID", "system default", "no", "none" + +Set the file owner for the domain socket. The +parameter is a user name, for which the userid is obtained by +rsyslogd during startup processing. Interim changes to the user +mapping are *not* detected. + + +FileOwnerNum +^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "system default", "no", "none" + +Set the file owner for the domain socket. The +parameter is a numerical ID, which which is used regardless of +whether the user actually exists. This can be useful if the user +mapping is not available to rsyslog during startup. + + +FileGroup +^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "GID", "system default", "no", "none" + +Set the group for the domain socket. The parameter is +a group name, for which the groupid is obtained by rsyslogd during +startup processing. Interim changes to the user mapping are not +detected. + + +FileGroupNum +^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "system default", "no", "none" + +Set the group for the domain socket. The parameter is +a numerical ID, which is used regardless of whether the group +actually exists. This can be useful if the group mapping is not +available to rsyslog during startup. + + +FileCreateMode +^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "octalNumber", "0644", "no", "none" + +Set the access permissions for the domain socket. The value given must +always be a 4-digit octal number, with the initial digit being zero. +Please note that the actual permission depend on rsyslogd's process +umask. If in doubt, use "$umask 0000" right at the beginning of the +configuration file to remove any restrictions. + + +FailOnChOwnFailure +^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "on", "no", "none" + +Rsyslog will not start if this is on and changing the file owner, group, +or access permissions fails. Disable this to ignore these errors. + + +Unlink +^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +If a unix domain socket is being used this controls whether or not the socket +is unlinked before listening and after closing. + + +Name +^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "imptcp", "no", "``$InputPTCPServerInputName``" + +Sets a name for the inputname property. If no name is set "imptcp" +is used by default. Setting a name is not strictly necessary, but can +be useful to apply filtering based on which input the message was +received from. Note that the name also shows up in +:doc:`impstats <impstats>` logs. + + +Ruleset +^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "none", "no", "``$InputPTCPServerBindRuleset``" + +Binds specified ruleset to this input. If not set, the default +ruleset is bound. + + +MaxFrameSize +^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "200000", "no", "none" + +When in octet counted mode, the frame size is given at the beginning +of the message. With this parameter the max size this frame can have +is specified and when the frame gets to large the mode is switched to +octet stuffing. +The max value this parameter can have was specified because otherwise +the integer could become negative and this would result in a +Segmentation Fault. (Max Value: 200000000) + + +MaxSessions +^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "0", "no", "none" + +Maximum number of open sessions allowed. If more tcp connections +are created then rsyslog will drop those connections. Warning, +this defaults to 0 which means unlimited, so take care to set this +if you have limited memory and/or processing power. + +A setting of zero or negative integer means no limit. + +Address +^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "none", "no", "``$InputPTCPServerListenIP``" + +On multi-homed machines, specifies to which local address the +listener should be bound. + + +AddtlFrameDelimiter +^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "-1", "no", "``$InputPTCPServerAddtlFrameDelimiter``" + +This directive permits to specify an additional frame delimiter for +plain tcp syslog. The industry-standard specifies using the LF +character as frame delimiter. Some vendors, notable Juniper in their +NetScreen products, use an invalid frame delimiter, in Juniper's case +the NUL character. This directive permits to specify the ASCII value +of the delimiter in question. Please note that this does not +guarantee that all wrong implementations can be cured with this +directive. It is not even a sure fix with all versions of NetScreen, +as I suggest the NUL character is the effect of a (common) coding +error and thus will probably go away at some time in the future. But +for the time being, the value 0 can probably be used to make rsyslog +handle NetScreen's invalid syslog/tcp framing. For additional +information, see this `forum +thread <http://kb.monitorware.com/problem-with-netscreen-log-t1652.html>`_. +**If this doesn't work for you, please do not blame the rsyslog team. +Instead file a bug report with Juniper!** + +Note that a similar, but worse, issue exists with Cisco's IOS +implementation. They do not use any framing at all. This is confirmed +from Cisco's side, but there seems to be very limited interest in +fixing this issue. This directive **can not** fix the Cisco bug. That +would require much more code changes, which I was unable to do so +far. Full details can be found at the `Cisco tcp syslog +anomaly <http://www.rsyslog.com/Article321.phtml>`_ page. + + +SupportOctetCountedFraming +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "on", "no", "``$InputPTCPServerSupportOctetCountedFraming``" + +The legacy octed-counted framing (similar to RFC5425 +framing) is activated. This is the default and should be left +unchanged until you know very well what you do. It may be useful to +turn it off, if you know this framing is not used and some senders +emit multi-line messages into the message stream. + + +NotifyOnConnectionClose +^^^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "``$InputPTCPServerNotifyOnConnectionClose``" + +Instructs imptcp to emit a message if a remote peer closes the +connection. + + +NotifyOnConnectionOpen +^^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +Instructs imptcp to emit a message if a remote peer opens a +connection. Hostname of the remote peer is given in the message. + + +KeepAlive +^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "``$InputPTCPServerKeepAlive``" + +Enable of disable keep-alive packets at the tcp socket layer. The +default is to disable them. + + +KeepAlive.Probes +^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "0", "no", "``$InputPTCPServerKeepAlive_probes``" + +The number of unacknowledged probes to send before considering the +connection dead and notifying the application layer. The default, 0, +means that the operating system defaults are used. This has only +effect if keep-alive is enabled. The functionality may not be +available on all platforms. + + +KeepAlive.Interval +^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "0", "no", "``$InputPTCPServerKeepAlive_intvl``" + +The interval between subsequential keepalive probes, regardless of +what the connection has exchanged in the meantime. The default, 0, +means that the operating system defaults are used. This has only +effect if keep-alive is enabled. The functionality may not be +available on all platforms. + + +KeepAlive.Time +^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "0", "no", "``$InputPTCPServerKeepAlive_time``" + +The interval between the last data packet sent (simple ACKs are not +considered data) and the first keepalive probe; after the connection +is marked to need keepalive, this counter is not used any further. +The default, 0, means that the operating system defaults are used. +This has only effect if keep-alive is enabled. The functionality may +not be available on all platforms. + + +RateLimit.Interval +^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "0", "no", "none" + +Specifies the rate-limiting interval in seconds. Set it to a number +of seconds (5 recommended) to activate rate-limiting. + + +RateLimit.Burst +^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "10000", "no", "none" + +Specifies the rate-limiting burst in number of messages. + + +Compression.mode +^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +This is the counterpart to the compression modes set in +:doc:`omfwd <omfwd>`. +Please see it's documentation for details. + + +flowControl +^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "on", "no", "none" + +Flow control is used to throttle the sender if the receiver queue is +near-full preserving some space for input that can not be throttled. + + +MultiLine +^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +Experimental parameter which causes rsyslog to recognise a new message +only if the line feed is followed by a '<' or if there are no more characters. + + +framing.delimiter.regex +^^^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "off", "no", "none" + +Experimental parameter. It is similar to "MultiLine", but provides greater +control of when a log message ends. You can specify a regular expression that +characterizes the header to expect at the start of the next message. As such, +it indicates the end of the current message. For example, one can use this +setting to use a RFC3164 header as frame delimiter:: + + framing.delimiter.regex="^<[0-9]{1,3}>(Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)" + +Note that when oversize messages arrive this mode may have problems finding +the proper frame terminator. There are some provisions inside imptcp to make +these kinds of problems unlikely, but if the messages are very much over the +configured MaxMessageSize, imptcp emits an error messages. Chances are great +it will properly recover from such a situation. + + +SocketBacklog +^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "5", "no", "none" + +Specifies the backlog parameter sent to the listen() function. +It defines the maximum length to which the queue of pending connections may grow. +See man page of listen(2) for more information. +The parameter controls both TCP and UNIX sockets backlog parameter. +Default value is arbitrary set to 5. + + +Defaulttz +^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "none", "no", "none" + +Set default time zone. At most seven chars are set, as we would otherwise +overrun our buffer. + + +Framingfix.cisco.asa +^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +Cisco very occasionally sends a space after a line feed, which thrashes framing +if not taken special care of. When this parameter is set to "on", we permit +space *in front of the next frame* and ignore it. + + +ListenPortFileName +^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "none", "no", "none" + +.. versionadded:: 8.38.0 + +With this parameter you can specify the name for a file. In this file the +port, imptcp is connected to, will be written. +This parameter was introduced because the testbench works with dynamic ports. + + +.. _imptcp-statistic-counter: + +Statistic Counter +================= + +This plugin maintains :doc:`statistics <../rsyslog_statistic_counter>` for each listener. The statistic is +named "imtcp" , followed by the bound address, listener port and IP +version in parenthesis. For example, the counter for a listener on port +514, bound to all interfaces and listening on IPv6 is called +"imptcp(\*/514/IPv6)". + +The following properties are maintained for each listener: + +- **submitted** - total number of messages submitted for processing since startup + + +.. _error-messages: + +Error Messages +============== + +When a message is to long it will be truncated and an error will show the remaining length of the message and the beginning of it. It will be easier to comprehend the truncation. + + +Caveats/Known Bugs +================== + +- module always binds to all interfaces + + +Examples +======== + +Example 1 +--------- + +This sets up a TCP server on port 514: + +.. code-block:: none + + module(load="imptcp") # needs to be done just once + input(type="imptcp" port="514") + + +Example 2 +--------- + +This creates a listener that listens on the local loopback +interface, only. + +.. code-block:: none + + module(load="imptcp") # needs to be done just once + input(type="imptcp" port="514" address="127.0.0.1") + + +Example 3 +--------- + +Create a unix domain socket: + +.. code-block:: none + + module(load="imptcp") # needs to be done just once + input(type="imptcp" path="/tmp/unix.sock" unlink="on") + + diff --git a/source/configuration/modules/imrelp.rst b/source/configuration/modules/imrelp.rst new file mode 100644 index 0000000..eb0d9c5 --- /dev/null +++ b/source/configuration/modules/imrelp.rst @@ -0,0 +1,595 @@ +************************* +imrelp: RELP Input Module +************************* + +=========================== =========================================================================== +**Module Name:** **imrelp** +**Author:** `Rainer Gerhards <https://rainer.gerhards.net/>`_ <rgerhards@adiscon.com> +=========================== =========================================================================== + + +Purpose +======= + +Provides the ability to receive syslog messages via the reliable RELP +protocol. This module requires `librelp <http://www.librelp.com>`__ to +be present on the system. From the user's point of view, imrelp works +much like imtcp or imgssapi, except that no message loss can occur. +Please note that with the currently supported RELP protocol version, a +minor message duplication may occur if a network connection between the +relp client and relp server breaks after the client could successfully +send some messages but the server could not acknowledge them. The window +of opportunity is very slim, but in theory this is possible. Future +versions of RELP will prevent this. Please also note that rsyslogd may +lose a few messages if rsyslog is shutdown while a network connection to +the server is broken and could not yet be recovered. Future versions of +RELP support in rsyslog will prevent that issue. Please note that both +scenarios also exist with plain TCP syslog. RELP, even with the small +nits outlined above, is a much more reliable solution than plain TCP +syslog and so it is highly suggested to use RELP instead of plain TCP. +Clients send messages to the RELP server via omrelp. + + +Notable Features +================ + +- :ref:`imrelp-statistic-counter` + + +Configuration Parameters +======================== + +.. note:: + + Parameter names are case-insensitive. + + +Module Parameters +----------------- + +Ruleset +^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "``$InputRELPServerBindRuleset``" + +.. versionadded:: 7.5.0 + +Binds the specified ruleset to **all** RELP listeners. This can be +overridden at the instance level. + +tls.tlslib +^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +.. versionadded:: 8.1903.0 + +Permits to specify the TLS library used by librelp. +All RELP protocol operations are actually performed by librelp and +not rsyslog itself. The value specified is directly passed down to +librelp. Depending on librelp version and build parameters, supported +TLS libraries differ (or TLS may not be supported at all). In this case +rsyslog emits an error message. + +Usually, the following options should be available: "openssl", "gnutls". + +Note that "gnutls" is the current default for historic reasons. We actually +recommend to use "openssl". It provides better error messages and accepts +a wider range of certificate types. + +If you have problems with the default setting, we recommend to switch to +"openssl". + + +Input Parameters +---------------- + +Port +^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "none", "yes", "``$InputRELPServerRun``" + +Starts a RELP server on selected port + +Address +^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "none", "no", "none" + +.. versionadded:: 8.37.0 + +Bind the RELP server to that address. If not specified, the server will be +bound to the wildcard address. + +Name +^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "imrelp", "no", "none" + + +Ruleset +^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "none", "no", "none" + +Binds specified ruleset to this listener. This overrides the +module-level Ruleset parameter. + + +MaxDataSize +^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "size_nbr", ":doc:`global(maxMessageSize) <../../rainerscript/global>`", "no", "none" + +Sets the max message size (in bytes) that can be received. Messages that +are too long are handled as specified in parameter oversizeMode. Note that +maxDataSize cannot be smaller than the global parameter maxMessageSize. + + +TLS +^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +If set to "on", the RELP connection will be encrypted by TLS, so +that the data is protected against observers. Please note that both +the client and the server must have set TLS to either "on" or "off". +Other combinations lead to unpredictable results. + +*Attention when using GnuTLS 2.10.x or older* + +Versions older than GnuTLS 2.10.x may cause a crash (Segfault) under +certain circumstances. Most likely when an imrelp inputs and an +omrelp output is configured. The crash may happen when you are +receiving/sending messages at the same time. Upgrade to a newer +version like GnuTLS 2.12.21 to solve the problem. + + +TLS.Compression +^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +The controls if the TLS stream should be compressed (zipped). While +this increases CPU use, the network bandwidth should be reduced. Note +that typical text-based log records usually compress rather well. + + +TLS.dhbits +^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "0", "no", "none" + +This setting controls how many bits are used for Diffie-Hellman key +generation. If not set, the librelp default is used. For security +reasons, at least 1024 bits should be used. Please note that the +number of bits must be supported by GnuTLS. If an invalid number is +given, rsyslog will report an error when the listener is started. We +do this to be transparent to changes/upgrades in GnuTLS (to check at +config processing time, we would need to hardcode the supported bits +and keep them in sync with GnuTLS - this is even impossible when +custom GnuTLS changes are made...). + + +TLS.PermittedPeer +^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "array", "none", "no", "none" + +PermittedPeer places access restrictions on this listener. Only peers which +have been listed in this parameter may connect. The certificate presented +by the remote peer is used for it's validation. + +The *peer* parameter lists permitted certificate fingerprints. Note +that it is an array parameter, so either a single or multiple +fingerprints can be listed. When a non-permitted peer connects, the +refusal is logged together with it's fingerprint. So if the +administrator knows this was a valid request, he can simply add the +fingerprint by copy and paste from the logfile to rsyslog.conf. + +To specify multiple fingerprints, just enclose them in braces like +this: + +.. code-block:: none + + tls.permittedPeer=["SHA1:...1", "SHA1:....2"] + +To specify just a single peer, you can either specify the string +directly or enclose it in braces. You may also use wildcards to match +a larger number of permitted peers, e.g. ``*.example.com``. + +When using wildcards to match larger number of permitted peers, please +know that the implementation is similar to Syslog RFC5425 which means: +This wildcard matches any left-most DNS label in the server name. +That is, the subject ``*.example.com`` matches the server names ``a.example.com`` +and ``b.example.com``, but does not match ``example.com`` or ``a.b.example.com``. + + +TLS.AuthMode +^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "none", "no", "none" + +Sets the mode used for mutual authentication. + +Supported values are either "*fingerprint*\ " or "*name"*. + +Fingerprint mode basically is what SSH does. It does not require a +full PKI to be present, instead self-signed certs can be used on all +peers. Even if a CA certificate is given, the validity of the peer +cert is NOT verified against it. Only the certificate fingerprint +counts. + +In "name" mode, certificate validation happens. Here, the matching is +done against the certificate's subjectAltName and, as a fallback, the +subject common name. If the certificate contains multiple names, a +match on any one of these names is considered good and permits the +peer to talk to rsyslog. + + +About Chained Certificates +-------------------------- + +.. versionadded:: 8.2008.0 + +With librelp 1.7.0, you can use chained certificates. +If using "openssl" as tls.tlslib, we recommend at least OpenSSL Version 1.1 +or higher. Chained certificates will also work with OpenSSL Version 1.0.2, but +they will be loaded into the main OpenSSL context object making them available +to all librelp instances (omrelp/imrelp) within the same process. + +If this is not desired, you will require to run rsyslog in multiple instances +with different omrelp configurations and certificates. + + +TLS.CaCert +^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "none", "no", "none" + +The CA certificate that is being used to verify the client certificates. +Has to be configured if TLS.AuthMode is set to "*fingerprint*\ " or "*name"*. + + +TLS.MyCert +^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "none", "no", "none" + +The machine certificate that is being used for TLS communication. + + +TLS.MyPrivKey +^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "none", "no", "none" + +The machine private key for the configured TLS.MyCert. + + +TLS.PriorityString +^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "none", "no", "none" + +This parameter allows passing the so-called "priority string" to +GnuTLS. This string gives complete control over all crypto +parameters, including compression settings. For this reason, when the +prioritystring is specified, the "tls.compression" parameter has no +effect and is ignored. + +Full information about how to construct a priority string can be +found in the GnuTLS manual. At the time of writing, this +information was contained in `section 6.10 of the GnuTLS +manual <http://gnutls.org/manual/html_node/Priority-Strings.html>`_. + +**Note: this is an expert parameter.** Do not use if you do not +exactly know what you are doing. + +tls.tlscfgcmd +^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "none", "no", "none" + +.. versionadded:: 8.2001.0 + +The setting can be used if tls.tlslib is set to "openssl" to pass configuration commands to +the openssl libray. +OpenSSL Version 1.0.2 or higher is required for this feature. +A list of possible commands and their valid values can be found in the documentation: +https://www.openssl.org/docs/man1.0.2/man3/SSL_CONF_cmd.html + +The setting can be single or multiline, each configuration command is separated by linefeed (\n). +Command and value are separated by equal sign (=). Here are a few samples: + +Example 1 +--------- + +This will allow all protocols except for SSLv2 and SSLv3: + +.. code-block:: none + + tls.tlscfgcmd="Protocol=ALL,-SSLv2,-SSLv3" + + +Example 2 +--------- + +This will allow all protocols except for SSLv2, SSLv3 and TLSv1. +It will also set the minimum protocol to TLSv1.2 + +.. code-block:: none + + tls.tlscfgcmd="Protocol=ALL,-SSLv2,-SSLv3,-TLSv1 + MinProtocol=TLSv1.2" + + +KeepAlive +^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +Enable or disable keep-alive packets at the TCP socket layer. By +defauly keep-alives are disabled. + + +KeepAlive.Probes +^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "0", "no", "none" + +The number of keep-alive probes to send before considering the +connection dead and notifying the application layer. The default, 0, +means that the operating system defaults are used. This only has an +effect if keep-alives are enabled. The functionality may not be +available on all platforms. + + +KeepAlive.Interval +^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "0", "no", "none" + +The interval between subsequent keep-alive probes, regardless of what +the connection has been exchanged in the meantime. The default, 0, +means that the operating system defaults are used. This only has an effect +if keep-alive is enabled. The functionality may not be available on all +platforms. + + +KeepAlive.Time +^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "0", "no", "none" + +The interval between the last data packet sent (simple ACKs are not +considered data) and the first keepalive probe; after the connection +is marked with keep-alive, this counter is not used any further. +The default, 0, means that the operating system defaults are used. +This only has an effect if keep-alive is enabled. The functionality may +not be available on all platforms. + + +oversizeMode +^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "truncate", "no", "none" + +.. versionadded:: 8.35.0 + +This parameter specifies how messages that are too long will be handled. +For this parameter the length of the parameter maxDataSize is used. + +- truncate: Messages will be truncated to the maximum message size. +- abort: This is the behaviour until version 8.35.0. Upon receiving a + message that is too long imrelp will abort. +- accept: Messages will be accepted even if they are too long and an error + message will be output. Using this option does have associated risks. + + +flowControl +^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "light", "no", "none" + +.. versionadded:: 8.1911.0 + + +This parameter permits the fine-tuning of the flowControl parameter. +Possible values are "no", "light", and "full". With "light" being the default +and previously only value. + +Changing the flow control setting may be useful for some rare applications, +this is an advanced setting and should only be changed if you know what you +are doing. Most importantly, **rsyslog block incoming data and become +unresponsive if you change flowcontrol to "full"**. While this may be a +desired effect when intentionally trying to make it most unlikely that +rsyslog needs to lose/discard messages, usually this is not what you want. + +General rule of thumb: **if you do not fully understand what this decription +here talks about, leave the parameter at default value**. + +This part of the +documentation is intentionally brief, as one needs to have deep understanding +of rsyslog to evaluate usage of this parameter. If someone has the insight, +the meaning of this parameter is crystal-clear. If not, that someone will +most likely make the wrong decision when changing this parameter away +from the default value. + + +.. _imrelp-statistic-counter: + +Statistic Counter +================= + +This plugin maintains :doc:`statistics <../rsyslog_statistic_counter>` for each listener. +The statistic by default is named "imrelp" , followed by the listener port in +parenthesis. For example, the counter for a listener on port 514 is called "imprelp(514)". +If the input is given a name, that input name is used instead of "imrelp". This counter is +available starting rsyslog 7.5.1 + +The following properties are maintained for each listener: + +- **submitted** - total number of messages submitted for processing since startup + + +Caveats/Known Bugs +================== + +- see description +- To obtain the remote system's IP address, you need to have at least + librelp 1.0.0 installed. Versions below it return the hostname + instead of the IP address. + + +Examples +======== + +Example 1 +--------- + +This sets up a RELP server on port 2514 with a max message size of 10,000 bytes. + +.. code-block:: none + + module(load="imrelp") # needs to be done just once + input(type="imrelp" port="2514" maxDataSize="10k") + + + +Receive RELP traffic via TLS +---------------------------- + +This receives RELP traffic via TLS using the recommended "openssl" library. +Except for encryption support the scenario is the same as in Example 1. + +Certificate files must exist at configured locations. Note that authmode +"certvalid" is not very strong - you may want to use a different one for +actual deployments. For details, see parameter descriptions. + +.. code-block:: none + + module(load="imrelp" tls.tlslib="openssl") + input(type="imrelp" port="2514" maxDataSize="10k" + tls="on" + tls.cacert="/tls-certs/ca.pem" + tls.mycert="/tls-certs/cert.pem" + tls.myprivkey="/tls-certs/key.pem" + tls.authmode="certvalid" + tls.permittedpeer="rsyslog") + diff --git a/source/configuration/modules/imsolaris.rst b/source/configuration/modules/imsolaris.rst new file mode 100644 index 0000000..f215714 --- /dev/null +++ b/source/configuration/modules/imsolaris.rst @@ -0,0 +1,59 @@ +******************************* +imsolaris: Solaris Input Module +******************************* + +=========================== =========================================================================== +**Module Name:** **imsolaris** +**Author:** `Rainer Gerhards <https://rainer.gerhards.net/>`_ <rgerhards@adiscon.com> +=========================== =========================================================================== + + +Purpose +======= + +Reads local Solaris log messages including the kernel log. + +This module is specifically tailored for Solaris. Under Solaris, there +is no special kernel input device. Instead, both kernel messages as well +as messages emitted via syslog() are received from a single source. + +This module obeys the Solaris door() mechanism to detect a running +syslogd instance. As such, only one can be active at one time. If it +detects another active instance at startup, the module disables itself, +but rsyslog will continue to run. + + +Configuration Parameters +======================== + +.. note:: + + Parameter names are case-insensitive. + + +|FmtObsoleteName| Directives +---------------------------- + +| functions:: $IMSolarisLogSocketName <name> + + This is the name of the log socket (stream) to read. If not given, + /dev/log is read. + + +Caveats/Known Bugs +================== + +None currently known. For obvious reasons, works on Solaris, only (and +compilation will most probably fail on any other platform). + + +Examples +======== + +The following sample pulls messages from the default log source + +.. code-block:: none + + $ModLoad imsolaris + + diff --git a/source/configuration/modules/imtcp.rst b/source/configuration/modules/imtcp.rst new file mode 100644 index 0000000..052fa6b --- /dev/null +++ b/source/configuration/modules/imtcp.rst @@ -0,0 +1,1086 @@ +****************************** +imtcp: TCP Syslog Input Module +****************************** + +=========================== =========================================================================== +**Module Name:** **imtcp** +**Author:** `Rainer Gerhards <https://rainer.gerhards.net/>`_ <rgerhards@adiscon.com> +=========================== =========================================================================== + + +Purpose +======= + +Provides the ability to receive syslog messages via TCP. Encryption is +natively provided by selecting the appropriate network stream driver +and can also be provided by using `stunnel <rsyslog_stunnel.html>`_ (an +alternative is the use the `imgssapi <imgssapi.html>`_ module). + + +Notable Features +================ + +- :ref:`imtcp-statistic-counter` + + +Configuration Parameters +======================== + +.. note:: + + Parameter names are case-insensitive. + + +Module Parameters +----------------- + +AddtlFrameDelimiter +^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "-1", "no", "``$InputTCPServerAddtlFrameDelimiter``" + +.. versionadded:: 4.3.1 + +This directive permits to specify an additional frame delimiter for +Multiple receivers may be configured by specifying $InputTCPServerRun +multiple times. + + +DisableLFDelimiter +^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "``$InputTCPServerDisableLFDelimiter``" + +Industry-standard plain text tcp syslog uses the LF to delimit +syslog frames. However, some users brought up the case that it may be +useful to define a different delimiter and totally disable LF as a +delimiter (the use case named were multi-line messages). This mode is +non-standard and will probably come with a lot of problems. However, +as there is need for it and it is relatively easy to support, we do +so. Be sure to turn this setting to "on" only if you exactly know +what you are doing. You may run into all sorts of troubles, so be +prepared to wrangle with that! + + +MaxFrameSize +^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "200000", "no", "none" + +When in octet counted mode, the frame size is given at the beginning +of the message. With this parameter the max size this frame can have +is specified and when the frame gets to large the mode is switched to +octet stuffing. +The max value this parameter can have was specified because otherwise +the integer could become negative and this would result in a +Segmentation Fault. (Max Value = 200000000) + + +NotifyOnConnectionOpen +^^^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "" + +Instructs imtcp to emit a message if the remote peer closes a +connection. + + +NotifyOnConnectionOpen +^^^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +Instructs imtcp to emit a message if the remote peer opens a +connection. + + +NotifyOnConnectionClose +^^^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "``$InputTCPServerNotifyOnConnectionClose``" + +Instructs imtcp to emit a message if the remote peer closes a +connection. + + + +KeepAlive +^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "``$InputTCPServerKeepAlive``" + +Enable or disable keep-alive packets at the tcp socket layer. The +default is to disable them. + + +KeepAlive.Probes +^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "0", "no", "``$InputTCPServerKeepAlive_probes``" + +The number of unacknowledged probes to send before considering the +connection dead and notifying the application layer. The default, 0, +means that the operating system defaults are used. This has only +effect if keep-alive is enabled. The functionality may not be +available on all platforms. + + +KeepAlive.Time +^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "0", "no", "``$InputTCPServerKeepAlive_time``" + +The interval between the last data packet sent (simple ACKs are not +considered data) and the first keepalive probe; after the connection +is marked to need keepalive, this counter is not used any further. +The default, 0, means that the operating system defaults are used. +This has only effect if keep-alive is enabled. The functionality may +not be available on all platforms. + + +KeepAlive.Interval +^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "0", "no", "" + +.. versionadded:: 8.2106.0 + +The interval for keep alive packets. + + + + +FlowControl +^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "on", "no", "``$InputTCPFlowControl``" + +This setting specifies whether some message flow control shall be +exercised on the related TCP input. If set to on, messages are +handled as "light delayable", which means the sender is throttled a +bit when the queue becomes near-full. This is done in order to +preserve some queue space for inputs that can not throttle (like +UDP), but it may have some undesired effect in some configurations. +Still, we consider this as a useful setting and thus it is the +default. To turn the handling off, simply configure that explicitly. + + +MaxListeners +^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "20", "no", "``$InputTCPMaxListeners``" + +Sets the maximum number of listeners (server ports) supported. +This must be set before the first $InputTCPServerRun directive. + + +MaxSessions +^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "200", "no", "``$InputTCPMaxSessions``" + +Sets the maximum number of sessions supported. This must be set +before the first $InputTCPServerRun directive. + + +StreamDriver.Name +^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "none", "no", "none" + +Selects :doc:`network stream driver <../../concepts/netstrm_drvr>` +for all inputs using this module. + + +StreamDriver.Mode +^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "0", "no", "``$InputTCPServerStreamDriverMode``" + +Sets the driver mode for the currently selected +:doc:`network stream driver <../../concepts/netstrm_drvr>`. +<number> is driver specific. + + +StreamDriver.AuthMode +^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "none", "no", "``$InputTCPServerStreamDriverAuthMode``" + +Sets stream driver authentication mode. Possible values and meaning +depend on the +:doc:`network stream driver <../../concepts/netstrm_drvr>`. +used. + + +StreamDriver.PermitExpiredCerts +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "warn", "no", "none" + +Controls how expired certificates will be handled when stream driver is in TLS mode. +It can have one of the following values: + +- on = Expired certificates are allowed + +- off = Expired certificates are not allowed (Default, changed from warn to off since Version 8.2012.0) + +- warn = Expired certificates are allowed but warning will be logged + + +StreamDriver.CheckExtendedKeyPurpose +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +Whether to check also purpose value in extended fields part of certificate +for compatibility with rsyslog operation. (driver-specific) + + +StreamDriver.PrioritizeSAN +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +Whether to use stricter SAN/CN matching. (driver-specific) + + +StreamDriver.TlsVerifyDepth +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "TLS library default", "no", "none" + + +Specifies the allowed maximum depth for the certificate chain verification. +Support added in v8.2001.0, supported by GTLS and OpenSSL driver. +If not set, the API default will be used. +For OpenSSL, the default is 100 - see the doc for more: +https://www.openssl.org/docs/man1.1.1/man3/SSL_set_verify_depth.html +For GnuTLS, the default is 5 - see the doc for more: +https://www.gnutls.org/manual/gnutls.html + + +PermittedPeer +^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "array", "none", "no", "``$InputTCPServerStreamDriverPermittedPeer``" + +Sets permitted peer IDs. Only these peers are able to connect to +the listener. <id-string> semantics depend on the currently +selected AuthMode and +:doc:`network stream driver <../../concepts/netstrm_drvr>`. +PermittedPeer may not be set in anonymous modes. PermittedPeer may +be set either to a single peer or an array of peers either of type +IP or name, depending on the tls certificate. + +Single peer: +PermittedPeer="127.0.0.1" + +Array of peers: +PermittedPeer=["test1.example.net","10.1.2.3","test2.example.net","..."] + + +DiscardTruncatedMsg +^^^^^^^^^^^^^^^^^^^ + +Normally when a message is truncated in octet stuffing mode the part that +is cut off is processed as the next message. When this parameter is activated, +the part that is cut off after a truncation is discarded and not processed. + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + + +gnutlsPriorityString +^^^^^^^^^^^^^^^^^^^^ + +The "gnutls priority string" parameter in rsyslog offers enhanced +customization for secure communications, allowing detailed configuration +of TLS driver properties. This includes specifying handshake algorithms +and other settings for GnuTLS, as well as implementing OpenSSL +configuration commands. Initially developed for GnuTLS, the "gnutls +priority string" has evolved since version v8.1905.0 to also support +OpenSSL, broadening its application and utility in network security +configurations. This update signifies a key advancement in rsyslog's +capabilities, making the "gnutls priority string" an essential +feature for advanced TLS configuration. + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "none", "no", "none" + +.. versionadded:: 8.29.0 + + +**Configuring Driver-Specific Properties** + +This configuration string is used to set properties specific to different drivers. Originally designed for the GnuTLS driver, it has been extended to support OpenSSL configuration commands from version v8.1905.0 onwards. + +**GNUTLS Configuration** + +In GNUTLS, this setting determines the handshake algorithms and options for the TLS session. It's designed to allow user overrides of the library's default settings. If you leave this parameter unset (NULL), the system will revert to the default settings. For more detailed information on priority strings in GNUTLS, you can refer to the GnuTLS Priority Strings Documentation available at [GnuTLS Website](https://gnutls.org/manual/html_node/Priority-Strings.html). + +**OpenSSL Configuration** + +This feature is compatible with OpenSSL Version 1.0.2 and above. It enables the passing of configuration commands to the OpenSSL library. You can find a comprehensive list of commands and their acceptable values in the OpenSSL Documentation, accessible at [OpenSSL Documentation](https://www.openssl.org/docs/man1.0.2/man3/SSL_CONF_cmd.html). + +**General Configuration Guidelines** + +The configuration can be formatted as a single line or across multiple lines. Each command within the configuration is separated by a linefeed (`\n`). To differentiate between a command and its corresponding value, use an equal sign (`=`). Below are some examples to guide you in formatting these commands. + + +Example 1 +--------- + +This will allow all protocols except for SSLv2 and SSLv3: + +.. code-block:: none + + gnutlsPriorityString="Protocol=ALL,-SSLv2,-SSLv3" + + +Example 2 +--------- + +This will allow all protocols except for SSLv2, SSLv3 and TLSv1. +It will also set the minimum protocol to TLSv1.2 + +.. code-block:: none + + gnutlsPriorityString="Protocol=ALL,-SSLv2,-SSLv3,-TLSv1 + MinProtocol=TLSv1.2" + + +PreserveCase +^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "boolean", "on", "no", "none" + +.. versionadded:: 8.37.0 + +This parameter is for controlling the case in fromhost. If preservecase is set to "off", the case in fromhost is not preserved. E.g., 'host1.example.org' the message was received from 'Host1.Example.Org'. Default to "on" for the backword compatibility. + + +Input Parameters +---------------- + +Port +^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "none", "yes", "``$InputTCPServerRun``" + +Starts a TCP server on selected port. If port zero is selected, the OS automatically +assigens a free port. Use `listenPortFileName` in this case to obtain the information +of which port was assigned. + + +ListenPortFileName +^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "none", "no", "none" + +This parameter specifies a file name into which the port number this input listens +on is written. It is primarily intended for cases when `port` is set to 0 to let +the OS automatically assign a free port number. + + +Address +^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "none", "no", "none" + +On multi-homed machines, specifies to which local address the +listener should be bound. + + +Name +^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "imtcp", "no", "``$InputTCPServerInputName``" + +Sets a name for the inputname property. If no name is set "imtcp" is +used by default. Setting a name is not strictly necessary, but can be +useful to apply filtering based on which input the message was +received from. + + +Ruleset +^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "none", "no", "``$InputTCPServerBindRuleset``" + +Binds the listener to a specific :doc:`ruleset <../../concepts/multi_ruleset>`. + + +SupportOctetCountedFraming +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "on", "no", "``$InputTCPServerSupportOctetCountedFraming``" + +If set to "on", the legacy octed-counted framing (similar to RFC5425 +framing) is activated. This should be left unchanged until you know +very well what you do. It may be useful to turn it off, if you know +this framing is not used and some senders emit multi-line messages +into the message stream. + + +RateLimit.Interval +^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "0", "no", "none" + +Specifies the rate-limiting interval in seconds. Default value is 0, +which turns off rate limiting. Set it to a number of seconds (5 +recommended) to activate rate-limiting. + + +RateLimit.Burst +^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "10000", "no", "none" + +Specifies the rate-limiting burst in number of messages. Default is +10,000. + + +listenPortFileName +^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "none", "no", "none" + +.. versionadded:: 8.38.0 + +With this parameter you can specify the name for a file. In this file the +port, imtcp is connected to, will be written. +This parameter was introduced because the testbench works with dynamic ports. + +.. note:: + + If this parameter is set, 0 will be accepted as the port. Otherwise it + is automatically changed to port 514 + + +StreamDriver.Name +^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "module parameter", "no", "none" + +.. versionadded:: 8.2106.0 + +This permits to override the equally-named module parameter on the input() +level. For further details, see the module parameter. + + +StreamDriver.Mode +^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "module parameter", "no", "``$InputTCPServerStreamDriverMode``" + +.. versionadded:: 8.2106.0 + +This permits to override the equally-named module parameter on the input() +level. For further details, see the module parameter. + + +StreamDriver.AuthMode +^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "module parameter", "no", "``$InputTCPServerStreamDriverAuthMode``" + +.. versionadded:: 8.2106.0 + +This permits to override the equally-named module parameter on the input() +level. For further details, see the module parameter. + + +StreamDriver.PermitExpiredCerts +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "module parameter", "no", "none" + +.. versionadded:: 8.2106.0 + +This permits to override the equally-named module parameter on the input() +level. For further details, see the module parameter. + + +StreamDriver.CheckExtendedKeyPurpose +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "module parameter", "no", "none" + +.. versionadded:: 8.2106.0 + +This permits to override the equally-named module parameter on the input() +level. For further details, see the module parameter. + + +StreamDriver.PrioritizeSAN +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "module parameter", "no", "none" + +.. versionadded:: 8.2106.0 + +This permits to override the equally-named module parameter on the input() +level. For further details, see the module parameter. + + +StreamDriver.TlsVerifyDepth +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "module parameter", "no", "none" + +.. versionadded:: 8.2106.0 + +This permits to override the equally-named module parameter on the input() +level. For further details, see the module parameter. + + +streamDriver.CAFile +^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "global parameter", "no", "none" + +.. versionadded:: 8.2108.0 + +This permits to override the DefaultNetstreamDriverCAFile global parameter on the input() +level. For further details, see the global parameter. + +streamDriver.CRLFile +^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "optional", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "global parameter", "no", "none" + +.. versionadded:: 8.2308.0 + +This permits to override the CRL (Certificate revocation list) file set via `global()` config +object at the per-action basis. This parameter is ignored if the netstream driver and/or its +mode does not need or support certificates. + +streamDriver.KeyFile +^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "global parameter", "no", "none" + +.. versionadded:: 8.2108.0 + +This permits to override the DefaultNetstreamDriverKeyFile global parameter on the input() +level. For further details, see the global parameter. + + +streamDriver.CertFile +^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "global parameter", "no", "none" + +.. versionadded:: 8.2108.0 + +This permits to override the DefaultNetstreamDriverCertFile global parameter on the input() +level. For further details, see the global parameter. + + +PermittedPeer +^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "array", "none", "no", "equally-named module parameter" +.. versionadded:: 8.2112.0 + +This permits to override the equally-named module parameter on the input() +level. For further details, see the module parameter. + + +gnutlsPriorityString +^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "module parameter", "no", "none" +.. versionadded:: 8.2106.0 + +This permits to override the equally-named module parameter on the input() +level. For further details, see the module parameter. + + +MaxSessions +^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "module parameter", "no", "" + +.. versionadded:: 8.2106.0 + +This permits to override the equally-named module parameter on the input() +level. For further details, see the module parameter. + + +MaxListeners +^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "module parameter", "no", "" + +.. versionadded:: 8.2106.0 + +This permits to override the equally-named module parameter on the input() +level. For further details, see the module parameter. + + +FlowControl +^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "module parameter", "no", "" + +.. versionadded:: 8.2106.0 + +This permits to override the equally-named module parameter on the input() +level. For further details, see the module parameter. + + +DisableLFDelimiter +^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "module parameter", "no", "" + + +.. versionadded:: 8.2106.0 + +This permits to override the equally-named module parameter on the input() +level. For further details, see the module parameter. + + +DiscardTruncatedMsg +^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "module parameter", "no", "none" + +.. versionadded:: 8.2106.0 + +This permits to override the equally-named module parameter on the input() +level. For further details, see the module parameter. + + +NotifyOnConnectionClose +^^^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "module parameter", "no", "none" + +.. versionadded:: 8.2106.0 + +This permits to override the equally-named module parameter on the input() +level. For further details, see the module parameter. + + +AddtlFrameDelimiter +^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "module parameter", "no", "" + +.. versionadded:: 8.2106.0 + +This permits to override the equally-named module parameter on the input() +level. For further details, see the module parameter. + + +MaxFrameSize +^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "module parameter", "no", "none" + +.. versionadded:: 8.2106.0 + +This permits to override the equally-named module parameter on the input() +level. For further details, see the module parameter. + + +PreserveCase +^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "boolean", "module parameter", "no", "none" + +.. versionadded:: 8.2106.0 + +This permits to override the equally-named module parameter on the input() +level. For further details, see the module parameter. + + +KeepAlive +^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "module parameter", "no", "" + +.. versionadded:: 8.2106.0 + +This permits to override the equally-named module parameter on the input() +level. For further details, see the module parameter. + + +KeepAlive.Probes +^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "module parameter", "no", "" + +.. versionadded:: 8.2106.0 + +This permits to override the equally-named module parameter on the input() +level. For further details, see the module parameter. + + +KeepAlive.Time +^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "module parameter", "no", "" + +.. versionadded:: 8.2106.0 + +This permits to override the equally-named module parameter on the input() +level. For further details, see the module parameter. + + +KeepAlive.Interval +^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "module parameter", "no", "" + +.. versionadded:: 8.2106.0 + +This permits to override the equally-named module parameter on the input() +level. For further details, see the module parameter. + + + +.. _imtcp-statistic-counter: + +Statistic Counter +================= + +This plugin maintains :doc:`statistics <../rsyslog_statistic_counter>` for each listener. The statistic is named +after the given input name (or "imtcp" if none is configured), followed by +the listener port in parenthesis. For example, the counter for a listener +on port 514 with no set name is called "imtcp(514)". + +The following properties are maintained for each listener: + +- **submitted** - total number of messages submitted for processing since startup + + +Caveats/Known Bugs +================== + +- module always binds to all interfaces +- can not be loaded together with `imgssapi <imgssapi.html>`_ (which + includes the functionality of imtcp) + + +Examples +======== + +Example 1 +--------- + +This sets up a TCP server on port 514 and permits it to accept up to 500 +connections: + +.. code-block:: none + + module(load="imtcp" MaxSessions="500") + input(type="imtcp" port="514") + + +Note that the global parameters (here: max sessions) need to be set when +the module is loaded. Otherwise, the parameters will not apply. + + +Additional Resources +==================== + +- `rsyslog video tutorial on how to store remote messages in a separate file <http://www.rsyslog.com/howto-store-remote-messages-in-a-separate-file/>`_ (for legacy syntax, but you get the idea). + diff --git a/source/configuration/modules/imtuxedoulog.rst b/source/configuration/modules/imtuxedoulog.rst new file mode 100644 index 0000000..56f72a9 --- /dev/null +++ b/source/configuration/modules/imtuxedoulog.rst @@ -0,0 +1,146 @@ +************************************** +imtuxedoulog: Tuxedo ULOG input module +************************************** + +================ ============================================================== +**Module Name:** **imtuxedoulog** +**Authors:** Jean-Philippe Hilaire <jean-philippe.hilaire@pmu.fr> & Philippe Duveau <philippe.duveau@free.fr> +================ ============================================================== + + +Purpose +======= + +This module allows rsyslog to process Tuxedo ULOG files. +Tuxedo create an ULOG file each new log of the day this file is defined + +- a prefix configured in the tuxedo configuration + +- a suffix based on the date ".MMDDYY" + +This module is a copy of the polling mode of imfile but the file name is +computed each polling. The previous one is closed to limit the number of +opened file descriptor simultaneously. + +Another particularity of ULOG is that the lines contains only the time in +day. So the module use the date in filename and time in log to fill log +timestamp. + +Compile +======= + +To successfully compile improg module. + + ./configure --enable-imtuxedoulog ... + +Configuration Parameters +======================== + +Action Parameters +----------------- + +ulogbase +^^^^^^^^ + +.. csv-table:: + :header: "type", "mandatory", "format", "default" + :widths: auto + :class: parameter-table + + "string", "yes", "path of ULOG file", + +Path of ULOG file as it is defined in Tuxedo Configuration ULOGPFX. +Dot and date is added a end to build full file path + +Tag +^^^ + +.. csv-table:: + :header: "type", "mandatory", "format", "default" + :widths: auto + :class: parameter-table + + "string", "yes", ,"none" + +The tag to be assigned to messages read from this file. If you would like to +see the colon after the tag, you need to include it when you assign a tag +value, like so: ``tag="myTagValue:"``. + +Facility +^^^^^^^^ + +.. csv-table:: + :header: "type", "mandatory", "format", "default" + :widths: auto + :class: parameter-table + + "string", "no", "facility\|number", "local0" + +The syslog facility to be assigned to messages read from this file. Can be +specified in textual form (e.g. ``local0``, ``local1``, ...) or as numbers (e.g. +16 for ``local0``). Textual form is suggested. + +Severity +^^^^^^^^ + +.. csv-table:: + :header: "type", "mandatory", "format", "default" + :widths: auto + :class: parameter-table + + "string", "no", "severity\|number", "notice" + +The syslog severity to be assigned to lines read. Can be specified +in textual form (e.g. ``info``, ``warning``, ...) or as numbers (e.g. 6 +for ``info``). Textual form is suggested. + +PersistStateInterval +^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "0", "no", + +Specifies how often the state file shall be written when processing +the input file. The **default** value is 0, which means a new state +file is only written when the monitored files is being closed (end of +rsyslogd execution). Any other value n means that the state file is +written every time n file lines have been processed. This setting can +be used to guard against message duplication due to fatal errors +(like power fail). Note that this setting affects imfile performance, +especially when set to a low value. Frequently writing the state file +is very time consuming. + +MaxLinesAtOnce +^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "0", "no", + +If set to 0, the file will be fully processed. If it is set to any other +value, a maximum of [number] lines is processed in sequence. The **default** +is 10240. + +MaxSubmitAtOnce +^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "1024", "no", "none" + +This is an expert option. It can be used to set the maximum input +batch size that the module can generate. The **default** is 1024, which +is suitable for a wide range of applications. Be sure to understand +rsyslog message batch processing before you modify this option. If +you do not know what this doc here talks about, this is a good +indication that you should NOT modify the default. diff --git a/source/configuration/modules/imudp.rst b/source/configuration/modules/imudp.rst new file mode 100644 index 0000000..cfaf9e6 --- /dev/null +++ b/source/configuration/modules/imudp.rst @@ -0,0 +1,600 @@ +****************************** +imudp: UDP Syslog Input Module +****************************** + +.. index:: ! imudp + +=========================== =========================================================================== +**Module Name:** **imudp** +**Author:** `Rainer Gerhards <https://rainer.gerhards.net/>`_ <rgerhards@adiscon.com> +=========================== =========================================================================== + + +Purpose +======= + +Provides the ability to receive syslog messages via UDP. + +Multiple receivers may be configured by specifying multiple input +statements. + +Note that in order to enable UDP reception, Firewall rules probably +need to be modified as well. Also, SELinux may need additional rules. + + +Notable Features +================ + +- :ref:`imudp-statistic-counter` + + +Configuration Parameters +======================== + +.. note:: + + Parameter names are case-insensitive. + +.. index:: imudp; module parameters + + +Module Parameters +----------------- + +TimeRequery +^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "2", "no", "``$UDPServerTimeRequery``" + +This is a performance optimization. Getting the system time is very +costly. With this setting, imudp can be instructed to obtain the +precise time only once every n-times. This logic is only activated if +messages come in at a very fast rate, so doing less frequent time +calls should usually be acceptable. The default value is two, because +we have seen that even without optimization the kernel often returns +twice the identical time. You can set this value as high as you like, +but do so at your own risk. The higher the value, the less precise +the timestamp. + +**Note:** the timeRequery is done based on executed system calls +(**not** messages received). So when batch sizes are used, multiple +messages are received with one system call. All of these messages +always receive the same timestamp, as they are effectively received +at the same time. When there is very high traffic and successive +system calls immediately return the next batch of messages, the time +requery logic kicks in, which means that by default time is only +queried for every second batch. Again, this should not cause a +too-much deviation as it requires messages to come in very rapidly. +However, we advise not to set the "timeRequery" parameter to a large +value (larger than 10) if input batches are used. + + +SchedulingPolicy +^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "``$IMUDPSchedulingPolicy``" + +Can be used the set the scheduler priority, if the necessary +functionality is provided by the platform. Most useful to select +"fifo" for real-time processing under Linux (and thus reduce chance +of packet loss). Other options are "rr" and "other". + + +SchedulingPriority +^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "none", "no", "``$IMUDPSchedulingPriority``" + +Scheduling priority to use. + + +BatchSize +^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "32", "no", "none" + +This parameter is only meaningful if the system support recvmmsg() +(newer Linux OSs do this). The parameter is silently ignored if the +system does not support it. If supported, it sets the maximum number +of UDP messages that can be obtained with a single OS call. For +systems with high UDP traffic, a relatively high batch size can +reduce system overhead and improve performance. However, this +parameter should not be overdone. For each buffer, max message size +bytes are statically required. Also, a too-high number leads to +reduced efficiency, as some structures need to be completely +initialized before the OS call is done. We would suggest to not set +it above a value of 128, except if experimental results show that +this is useful. + + +Threads +^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "1", "no", "none" + +.. versionadded:: 7.5.5 + +Number of worker threads to process incoming messages. These threads +are utilized to pull data off the network. On a busy system, +additional threads (but not more than there are CPUs/Cores) can help +improving performance and avoiding message loss. Note that with too +many threads, performance can suffer. There is a hard upper limit on +the number of threads that can be defined. Currently, this limit is +set to 32. It may increase in the future when massive multicore +processors become available. + + +PreserveCase +^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "boolean", "off", "no", "none" + +.. versionadded:: 8.37.0 + +This parameter is for controlling the case in fromhost. If preservecase is set to "on", the case in fromhost is preserved. E.g., 'Host1.Example.Org' when the message was received from 'Host1.Example.Org'. Default to "off" for the backword compatibility. + + +.. index:: imudp; input parameters + + +Input Parameters +---------------- + +.. index:: imudp; address (input parameter) + +Address +^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "none", "no", "``$UDPServerAddress``" + +Local IP address (or name) the UDP server should bind to. Use "*" +to bind to all of the machine's addresses. + + +Port +^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "array", "514", "yes", "``$UDPServerRun``" + +Specifies the port the server shall listen to.. Either a single port can +be specified or an array of ports. If multiple ports are specified, a +listener will be automatically started for each port. Thus, no +additional inputs need to be configured. + +Single port: Port="514" + +Array of ports: Port=["514","515","10514","..."] + + +IpFreeBind +^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "2", "no", "none" + +.. versionadded:: 8.18.0 + +Manages the IP_FREEBIND option on the UDP socket, which allows binding it to +an IP address that is nonlocal or not (yet) associated to any network interface. + +The parameter accepts the following values: + +- 0 - does not enable the IP_FREEBIND option on the + UDP socket. If the *bind()* call fails because of *EADDRNOTAVAIL* error, + socket initialization fails. + +- 1 - silently enables the IP_FREEBIND socket + option if it is required to successfully bind the socket to a nonlocal address. + +- 2 - enables the IP_FREEBIND socket option and + warns when it is used to successfully bind the socket to a nonlocal address. + + +Device +^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "none", "no", "none" + +Bind socket to given device (e.g., eth0) + +For Linux with VRF support, the Device option can be used to specify the +VRF for the Address. + + +Ruleset +^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "RSYSLOG_DefaultRuleset", "no", "``$InputUDPServerBindRuleset``" + +Binds the listener to a specific :doc:`ruleset <../../concepts/multi_ruleset>`. + + +RateLimit.Interval +^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "0", "no", "none" + +.. versionadded:: 7.3.1 + +The rate-limiting interval in seconds. Value 0 turns off rate limiting. +Set it to a number of seconds (5 recommended) to activate rate-limiting. + + +RateLimit.Burst +^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "10000", "no", "none" + +.. versionadded:: 7.3.1 + +Specifies the rate-limiting burst in number of messages. + + +Name +^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "imudp", "no", "none" + +.. versionadded:: 8.3.3 + +Specifies the value of the inputname property. In older versions, +this was always "imudp" for all +listeners, which still is the default. Starting with 7.3.9 it can be +set to different values for each listener. Note that when a single +input statement defines multiple listener ports, the inputname will be +the same for all of them. If you want to differentiate in that case, +use "name.appendPort" to make them unique. Note that the +"name" parameter can be an empty string. In that case, the +corresponding inputname property will obviously also be the empty +string. This is primarily meant to be used together with +"name.appendPort" to set the inputname equal to the port. + + +Name.appendPort +^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +.. versionadded:: 7.3.9 + +Appends the port the inputname property. Note that when no "name" is +specified, the default of "imudp" is used and the port is appended to +that default. So, for example, a listener port of 514 in that case +will lead to an inputname of "imudp514". The ability to append a port +is most useful when multiple ports are defined for a single input and +each of the inputnames shall be unique. Note that there currently is +no differentiation between IPv4/v6 listeners on the same port. + + +DefaultTZ +^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "none", "no", "none" + +This is an **experimental** parameter; details may change at any +time and it may also be discontinued without any early warning. +Permits to set a default timezone for this listener. This is useful +when working with legacy syslog (RFC3164 et al) residing in different +timezones. If set it will be used as timezone for all messages **that +do not contain timezone info**. Currently, the format **must** be +"+/-hh:mm", e.g. "-05:00", "+01:30". Other formats, including TZ +names (like EST) are NOT yet supported. Note that consequently no +daylight saving settings are evaluated when working with timezones. +If an invalid format is used, "interesting" things can happen, among +them malformed timestamps and rsyslogd segfaults. This will obviously +be changed at the time this feature becomes non-experimental. + + +RcvBufSize +^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "size", "none", "no", "none" + +.. versionadded:: 7.3.9 + +This request a socket receive buffer of specific size from the operating system. It +is an expert parameter, which should only be changed for a good reason. +Note that setting this parameter disables Linux auto-tuning, which +usually works pretty well. The default value is 0, which means "keep +the OS buffer size unchanged". This is a size value. So in addition +to pure integer values, sizes like "256k", "1m" and the like can be +specified. Note that setting very large sizes may require root or +other special privileges. Also note that the OS may slightly adjust +the value or shrink it to a system-set max value if the user is not +sufficiently privileged. Technically, this parameter will result in a +setsockopt() call with SO\_RCVBUF (and SO\_RCVBUFFORCE if it is +available). (Maximum Value: 1G) + + +.. _imudp-statistic-counter: + +Statistic Counter +================= + +This plugin maintains :doc:`statistics <../rsyslog_statistic_counter>` for each listener and for each worker thread. + +The listener statistic is named starting with "imudp", followed followed by the +listener IP, a colon and port in parenthesis. For example, the counter for a +listener on port 514 (on all IPs) with no set name is called "imudp(\*:514)". + +If an "inputname" is defined for a listener, that inputname is used instead of +"imudp" as statistic name. For example, if the inputname is set to "myudpinut", +that corresponding statistic name in above case would be "myudpinput(\*:514)". +This has been introduced in 7.5.3. + +The following properties are maintained for each listener: + +- **submitted** - total number of messages submitted for processing since startup + +The worker thread (in short: worker) statistic is named "imudp(wX)" where "X" is +the worker thread ID, which is an monotonically increasing integer starting at 0. +This means the first worker will have the name "imudp(w0)", the second "imudp(w1)" +and so on. Note that workers are all equal. It doesn’t really matter which worker +processes which messages, so the actual worker ID is not of much concern. More +interesting is to check how the load is spread between the worker. Also note that +there is no fixed worker-to-listener relationship: all workers process messages +from all listeners. + +Note: worker thread statistics are available starting with rsyslog 7.5.5. + +- **disallowed** - total number of messages discarded due to disallowed sender + +This counts the number of messages that have been discarded because they have +been received by an disallowed sender. Note that if no allowed senders are +configured (the default), this counter will always be zero. + +This counter was introduced by rsyslog 8.35.0. + + +The following properties are maintained for each worker thread: + +- **called.recvmmsg** - number of recvmmsg() OS calls done + +- **called.recvmsg** - number of recvmsg() OS calls done + +- **msgs.received** - number of actual messages received + + +Caveats/Known Bugs +================== + +- Scheduling parameters are set **after** privileges have been dropped. + In most cases, this means that setting them will not be possible + after privilege drop. This may be worked around by using a + sufficiently-privileged user account. + +Examples +======== + +Example 1 +--------- + +This sets up an UDP server on port 514: + +.. code-block:: none + + module(load="imudp") # needs to be done just once + input(type="imudp" port="514") + + +Example 2 +--------- + +This sets up a UDP server on port 514 bound to device eth0: + +.. code-block:: none + + module(load="imudp") # needs to be done just once + input(type="imudp" port="514" device="eth0") + + +Example 3 +--------- + +The following sample is mostly equivalent to the first one, but request +a larger rcvuf size. Note that 1m most probably will not be honored by +the OS until the user is sufficiently privileged. + +.. code-block:: none + + module(load="imudp") # needs to be done just once + input(type="imudp" port="514" rcvbufSize="1m") + + +Example 4 +--------- + +In the next example, we set up three listeners at ports 10514, 10515 and +10516 and assign a listener name of "udp" to it, followed by the port +number: + +.. code-block:: none + + module(load="imudp") + input(type="imudp" port=["10514","10515","10516"] + inputname="udp" inputname.appendPort="on") + + +Example 5 +--------- + +The next example is almost equal to the previous one, but now the +inputname property will just be set to the port number. So if a message +was received on port 10515, the input name will be "10515" in this +example whereas it was "udp10515" in the previous one. Note that to do +that we set the inputname to the empty string. + +.. code-block:: none + + module(load="imudp") + input(type="imudp" port=["10514","10515","10516"] + inputname="" inputname.appendPort="on") + + +Additional Information on Performance Tuning +============================================ + +Threads and Ports +----------------- + +The maximum number of threads is a module parameter. Thus there is no direct +relation to the number of ports. + +Every worker thread processes all inbound ports in parallel. To do so, it +adds all listen ports to an `epoll()` set and waits for packets to arrive. If +the system supports the `recvmmsg()` call, it tries to receive up to `batchSize` +messages at once. This reduces the number of transitions between user and +kernel space and as such overhead. + +After the packages have been received, imudp processes each message and creates +input batches which are then submitted according to the config file's queue +definition. After that the a new cycle beings and imudp return to wait for +new packets to arrive. + +When multiple threads are defined, each thread performs the processing +described above. All worker threads are created when imudp is started. +Each of them will individually awoken from epoll as data +is present. Each one reads as much available data as possible. With a low +incoming volume this can be inefficient in that the threads compete against +inbound data. At sufficiently high volumes this is not a problem because +multiple workers permit to read data from the operating system buffers +while other workers process the data they have read. It must be noted +that "sufficiently high volume" is not a precise concept. A single thread +can be very efficient. As such it is recommended to run impstats inside a +performance testing lab to find out a good number of worker threads. If +in doubt, start with a low number and increase only if performance +actually increases by adding threads. + +A word of caution: just looking at thread CPU use is **not** a proper +way to monitor imudp processing capabilities. With too many threads +the overhead can increase, even strongly. This can result in a much higher +CPU utilization but still overall less processing capability. + +Please also keep in your mind that additional input worker threads may +cause more mutex contention when adding data to processing queues. + +Too many threads may also reduce the number of messages received via +a single recvmmsg() call, which in turn increases kernel/user space +switching and thus system overhead. + +If **real time** priority is used it must be ensured that not all +operating system cores are used by imudp threads. The reason is that +otherwise for heavy workloads there is no ability to actually process +messages. While this may be desirable in some cases where queue settings +permit for large bursts, it in general can lead to pushback from the +queues. + +For lower volumes, real time priority can increase the operating system +overhead by awaking imudp more often than strictly necessary and thus +reducing the effectiveness of `recvmmsg()`. + +imudp threads and queue worker threads +-------------------------------------- +There is no direct relationship between these two entities. Imudp submits +messages to the configured rulesets and places them into the respective +queues. It is then up the the queue config, and outside of the scope +or knowledge of imudp, how many queue worker threads will be spawned by +the queue in question. + +Note, however, that queue worker threads and imudp input worker threads +compete for system resources. As such the combined overall value should +not overload the system. There is no strict rule to follow when sizing +overall worker numbers: for queue workers it strongly depends on how +compute-intense the workload is. For example, omfile actions need +few worker threads as they are fast. On the contrary, omelasticsearch often +waits for server replies and as such more worker threads can be beneficial. +The queue subsystem auto-tuning of worker threads should handle the +different needs in a useful way. + +Additional Resources +==================== + +- `rsyslog video tutorial on how to store remote messages in a separate file <http://www.rsyslog.com/howto-store-remote-messages-in-a-separate-file/>`_. +- Description of `rsyslog statistic + counters <http://www.rsyslog.com/rsyslog-statistic-counter/>`_. + This also describes all imudp counters. + diff --git a/source/configuration/modules/imuxsock.rst b/source/configuration/modules/imuxsock.rst new file mode 100644 index 0000000..8c05108 --- /dev/null +++ b/source/configuration/modules/imuxsock.rst @@ -0,0 +1,966 @@ +********************************** +imuxsock: Unix Socket Input Module +********************************** + +=========================== =========================================================================== +**Module Name:** **imuxsock** +**Author:** `Rainer Gerhards <http://www.gerhards.net/rainer>`_ <rgerhards@adiscon.com> +=========================== =========================================================================== + + +Purpose +======= + +This module provides the ability to accept syslog messages from applications +running on the local system via Unix sockets. Most importantly, this is the +mechanism by which the :manpage:`syslog(3)` call delivers syslog messages +to rsyslogd. + +.. seealso:: + + :doc:`omuxsock` + + +Notable Features +================ + +- :ref:`imuxsock-rate-limiting-label` +- :ref:`imuxsock-trusted-properties-label` +- :ref:`imuxsock-flow-control-label` +- :ref:`imuxsock-application-timestamps-label` +- :ref:`imuxsock-systemd-details-label` + + +Configuration Parameters +======================== + +.. note:: + + Parameter names are case-insensitive. + +Module Parameters +----------------- + +.. warning:: + + When running under systemd, **many "sysSock." parameters are ignored**. + See parameter descriptions and the :ref:`imuxsock-systemd-details-label` section for + details. + + +SysSock.IgnoreTimestamp +^^^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "on", "no", "``$SystemLogSocketIgnoreMsgTimestamp``" + +Ignore timestamps included in the messages, applies to messages received via +the system log socket. + + +SysSock.IgnoreOwnMessages +^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "on", "no", "none" + +Ignores messages that originated from the same instance of rsyslogd. +There usually is no reason to receive messages from ourselves. This +setting is vital when writing messages to the systemd journal. + +.. versionadded:: 7.3.7 + +.. seealso:: + + See :doc:`omjournal <omjournal>` module documentation for a more + in-depth description. + + +SysSock.Use +^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "on", "no", "``$OmitLocalLogging``" + +Listen on the default local log socket (``/dev/log``) or, if provided, use +the log socket value assigned to the ``SysSock.Name`` parameter instead +of the default. This is most useful if you run multiple instances of +rsyslogd where only one shall handle the system log socket. Unless +disabled by the ``SysSock.Unlink`` setting, this socket is created +upon rsyslog startup and deleted upon shutdown, according to +traditional syslogd behavior. + +The behavior of this parameter is different for systemd systems. For those +systems, ``SysSock.Use`` still needs to be enabled, but the value of +``SysSock.Name`` is ignored and the socket provided by systemd is used +instead. If this parameter is *not* enabled, then imuxsock will only be +of use if a custom input is configured. + +See the :ref:`imuxsock-systemd-details-label` section for details. + + +SysSock.Name +^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "/dev/log", "no", "``$SystemLogSocketName``" + +Specifies an alternate log socket to be used instead of the default system +log socket, traditionally ``/dev/log``. Unless disabled by the +``SysSock.Unlink`` setting, this socket is created upon rsyslog startup +and deleted upon shutdown, according to traditional syslogd behavior. + +The behavior of this parameter is different for systemd systems. See the + :ref:`imuxsock-systemd-details-label` section for details. + + +SysSock.FlowControl +^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "``$SystemLogFlowControl``" + +Specifies if flow control should be applied to the system log socket. + + +SysSock.UsePIDFromSystem +^^^^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "``$SystemLogUsePIDFromSystem``" + +Specifies if the pid being logged shall be obtained from the log socket +itself. If so, the TAG part of the message is rewritten. It is recommended +to turn this option on, but the default is "off" to keep compatible +with earlier versions of rsyslog. + +.. versionadded:: 5.7.0 + + +SysSock.RateLimit.Interval +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "max", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "0", "", "no", "``$SystemLogRateLimitInterval``" + +Specifies the rate-limiting interval in seconds. Default value is 0, +which turns off rate limiting. Set it to a number of seconds (5 +recommended) to activate rate-limiting. The default of 0 has been +chosen as people experienced problems with this feature activated +by default. Now it needs an explicit opt-in by setting this parameter. + + +SysSock.RateLimit.Burst +^^^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "max", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "200", "(2^31)-1", "no", "``$SystemLogRateLimitBurst``" + +Specifies the rate-limiting burst in number of messages. + +.. versionadded:: 5.7.1 + + +SysSock.RateLimit.Severity +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "max", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "1", "", "no", "``$SystemLogRateLimitSeverity``" + +Specifies the severity of messages that shall be rate-limited. + +.. seealso:: + + https://en.wikipedia.org/wiki/Syslog#Severity_level + + +SysSock.UseSysTimeStamp +^^^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "on", "no", "``$SystemLogUseSysTimeStamp``" + +The same as the input parameter ``UseSysTimeStamp``, but for the system log +socket. This parameter instructs ``imuxsock`` to obtain message time from +the system (via control messages) instead of using time recorded inside +the message. This may be most useful in combination with systemd. Due to +the usefulness of this functionality, we decided to enable it by default. +As such, the behavior is slightly different than previous versions. +However, we do not see how this could negatively affect existing environments. + +.. versionadded:: 5.9.1 + + +SysSock.Annotate +^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "``$SystemLogSocketAnnotate``" + +Turn on annotation/trusted properties for the system log socket. See +the :ref:`imuxsock-trusted-properties-label` section for more info. + + +SysSock.ParseTrusted +^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "``$SystemLogParseTrusted``" + +If ``SysSock.Annotation`` is turned on, create JSON/lumberjack properties +out of the trusted properties (which can be accessed via |FmtAdvancedName| +JSON Variables, e.g. ``$!pid``) instead of adding them to the message. + +.. versionadded:: 7.2.7 + |FmtAdvancedName| directive introduced + +.. versionadded:: 7.3.8 + |FmtAdvancedName| directive introduced + +.. versionadded:: 6.5.0 + |FmtObsoleteName| directive introduced + + +SysSock.Unlink +^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "on", "no", "none" + +If turned on (default), the system socket is unlinked and re-created +when opened and also unlinked when finally closed. Note that this +setting has no effect when running under systemd control (because +systemd handles the socket. See the :ref:`imuxsock-systemd-details-label` +section for details. + +.. versionadded:: 7.3.9 + + +SysSock.UseSpecialParser +^^^^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "on", "no", "none" + +The equivalent of the ``UseSpecialParser`` input parameter, but +for the system socket. If turned on (the default) a special parser is +used that parses the format that is usually used +on the system log socket (the one :manpage:`syslog(3)` creates). If set to +"off", the regular parser chain is used, in which case the format on the +log socket can be arbitrary. + +.. note:: + + When the special parser is used, rsyslog is able to inject a more precise + timestamp into the message (it is obtained from the log socket). If the + regular parser chain is used, this is not possible. + +.. versionadded:: 8.9.0 + The setting was previously hard-coded "on" + + +SysSock.ParseHostname +^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +.. note:: + + This option only has an effect if ``SysSock.UseSpecialParser`` is + set to "off". + +Normally, the local log sockets do *not* contain hostnames. If set +to on, parsers will expect hostnames just like in regular formats. If +set to off (the default), the parser chain is instructed to not expect +them. + +.. versionadded:: 8.9.0 + + +Input Parameters +---------------- + +Ruleset +^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "default ruleset", "no", "none" + +Binds specified ruleset to this input. If not set, the default +ruleset is bound. + +.. versionadded:: 8.17.0 + + +IgnoreTimestamp +^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "on", "no", "``$InputUnixListenSocketIgnoreMsgTimestamp``" + +Ignore timestamps included in messages received from the input being +defined. + + +IgnoreOwnMessages +^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "on", "no", "none" + +Ignore messages that originated from the same instance of rsyslogd. +There usually is no reason to receive messages from ourselves. This +setting is vital when writing messages to the systemd journal. + +.. versionadded:: 7.3.7 + +.. seealso:: + + See :doc:`omjournal <omjournal>` module documentation for a more + in-depth description. + + + + +FlowControl +^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "``$InputUnixListenSocketFlowControl``" + +Specifies if flow control should be applied to the input being defined. + + +RateLimit.Interval +^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "max", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "0", "", "no", "``$IMUXSockRateLimitInterval``" + +Specifies the rate-limiting interval in seconds. Default value is 0, which +turns off rate limiting. Set it to a number of seconds (5 recommended) +to activate rate-limiting. The default of 0 has been chosen as people +experienced problems with this feature activated by default. Now it +needs an explicit opt-in by setting this parameter. + + +RateLimit.Burst +^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "max", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "200", "(2^31)-1", "no", "``$IMUXSockRateLimitBurst``" + +Specifies the rate-limiting burst in number of messages. + +.. versionadded:: 5.7.1 + + +RateLimit.Severity +^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "max", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "1", "", "no", "``$IMUXSockRateLimitSeverity``" + +Specifies the severity of messages that shall be rate-limited. + +.. seealso:: + + https://en.wikipedia.org/wiki/Syslog#Severity_level + + +UsePIDFromSystem +^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "``$InputUnixListenSocketUsePIDFromSystem``" + +Specifies if the pid being logged shall be obtained from the log socket +itself. If so, the TAG part of the message is rewritten. It is +recommended to turn this option on, but the default is "off" to keep +compatible with earlier versions of rsyslog. + + +UseSysTimeStamp +^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "on", "no", "``$InputUnixListenSocketUseSysTimeStamp``" + +This parameter instructs ``imuxsock`` to obtain message time from +the system (via control messages) instead of using time recorded inside +the message. This may be most useful in combination with systemd. Due to +the usefulness of this functionality, we decided to enable it by default. +As such, the behavior is slightly different than previous versions. +However, we do not see how this could negatively affect existing environments. + +.. versionadded:: 5.9.1 + + +CreatePath +^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "``$InputUnixListenSocketCreatePath``" + +Create directories in the socket path if they do not already exist. +They are created with 0755 permissions with the owner being the +process under which rsyslogd runs. The default is not to create +directories. Keep in mind, though, that rsyslogd always creates +the socket itself if it does not exist (just not the directories +by default). + +This option is primarily considered useful for defining additional +sockets that reside on non-permanent file systems. As rsyslogd probably +starts up before the daemons that create these sockets, it is a vehicle +to enable rsyslogd to listen to those sockets even though their directories +do not yet exist. + +.. versionadded:: 4.7.0 +.. versionadded:: 5.3.0 + + +Socket +^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "none", "no", "``$AddUnixListenSocket``" + +Adds additional unix socket. Formerly specified with the ``-a`` option. + + +HostName +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "NULL", "no", "``$InputUnixListenSocketHostName``" + +Allows overriding the hostname that shall be used inside messages +taken from the input that is being defined. + + +Annotate +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "``$InputUnixListenSocketAnnotate``" + +Turn on annotation/trusted properties for the input that is being defined. +See the :ref:`imuxsock-trusted-properties-label` section for more info. + + +ParseTrusted +^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "``$ParseTrusted``" + +Equivalent to the ``SysSock.ParseTrusted`` module parameter, but applies +to the input that is being defined. + + +Unlink +^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "on", "no", "``none``" + +If turned on (default), the socket is unlinked and re-created when opened +and also unlinked when finally closed. Set it to off if you handle socket +creation yourself. + +.. note:: + + Note that handling socket creation oneself has the + advantage that a limited amount of messages may be queued by the OS + if rsyslog is not running. + +.. versionadded:: 7.3.9 + + +UseSpecialParser +^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "on", "no", "none" + +Equivalent to the ``SysSock.UseSpecialParser`` module parameter, but applies +to the input that is being defined. + +.. versionadded:: 8.9.0 + The setting was previously hard-coded "on" + + +ParseHostname +^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +Equivalent to the ``SysSock.ParseHostname`` module parameter, but applies +to the input that is being defined. + +.. versionadded:: 8.9.0 + + +.. _imuxsock-rate-limiting-label: + +Input rate limiting +=================== + +rsyslog supports (optional) input rate limiting to guard against the problems +of a wild running logging process. If more than +``SysSock.RateLimit.Interval`` \* ``SysSock.RateLimit.Burst`` log messages +are emitted from the same process, those messages with +``SysSock.RateLimit.Severity`` or lower will be dropped. It is not possible +to recover anything about these messages, but imuxsock will tell you how +many it has dropped once the interval has expired AND the next message is +logged. Rate-limiting depends on ``SCM\_CREDENTIALS``. If the platform does +not support this socket option, rate limiting is turned off. If multiple +sockets are configured, rate limiting works independently on each of +them (that should be what you usually expect). + +The same functionality is available for additional log sockets, in which +case the config statements just use the prefix RateLimit... but otherwise +works exactly the same. When working with severities, please keep in mind +that higher severity numbers mean lower severity and configure things +accordingly. To turn off rate limiting, set the interval to zero. + +.. versionadded:: 5.7.1 + + +.. _imuxsock-trusted-properties-label: + +Trusted (syslog) properties +=========================== + +rsyslog can annotate messages from system log sockets (via imuxsock) with +so-called `Trusted syslog +properties <http://www.rsyslog.com/what-are-trusted-properties/>`_, (or just +"Trusted Properties" for short). These are message properties not provided by +the logging client application itself, but rather obtained from the system. +As such, they can not be faked by the user application and are trusted in +this sense. This feature is based on a similar idea introduced in systemd. + +This feature requires a recent enough Linux Kernel and access to +the ``/proc`` file system. In other words, this may not work on all +platforms and may not work fully when privileges are dropped (depending +on how they are dropped). Note that trusted properties can be very +useful, but also typically cause the message to grow rather large. Also, +the format of log messages is changed by adding the trusted properties at +the end. For these reasons, the feature is **not enabled by default**. +If you want to use it, you must turn it on (via +``SysSock.Annotate`` and ``Annotate``). + +.. versionadded:: 5.9.4 + +.. seealso:: + + `What are "trusted properties"? + <http://www.rsyslog.com/what-are-trusted-properties/>`_ + + +.. _imuxsock-flow-control-label: + +Flow-control of Unix log sockets +================================ + +If processing queues fill up, the unix socket reader is blocked for a +short while to help prevent overrunning the queues. If the queues are +overrun, this may cause excessive disk-io and impact performance. + +While turning on flow control for many systems does not hurt, it `can` lead +to a very unresponsive system and as such is disabled by default. + +This means that log records are placed as quickly as possible into the +processing queues. If you would like to have flow control, you +need to enable it via the ``SysSock.FlowControl`` and ``FlowControl`` config +directives. Just make sure you have thought about the implications and have +tested the change on a non-production system first. + + +.. _imuxsock-application-timestamps-label: + +Control over application timestamps +=================================== + +Application timestamps are ignored by default. This is needed, as some +programs (e.g. sshd) log with inconsistent timezone information, what +messes up the local logs (which by default don't even contain time zone +information). This seems to be consistent with what sysklogd has done for +many years. Alternate behaviour may be desirable if gateway-like processes +send messages via the local log slot. In that case, it can be enabled via +the ``SysSock.IgnoreTimestamp`` and ``IgnoreTimestamp`` config directives. + + +.. _imuxsock-systemd-details-label: + +Coexistence with systemd +======================== + +Rsyslog should by default be configured for systemd support on all platforms +that usually run systemd (which means most Linux distributions, but not, for +example, Solaris). + +Rsyslog is able to coexist with systemd with minimal changes on the part of the +local system administrator. While the ``systemd journal`` now assumes full +control of the local ``/dev/log`` system log socket, systemd provides +access to logging data via the ``/run/systemd/journal/syslog`` log socket. +This log socket is provided by the ``syslog.socket`` file that is shipped +with systemd. + +The imuxsock module can still be used in this setup and provides superior +performance over :doc:`imjournal <imjournal>`, the alternative journal input +module. + +.. note:: + + It must be noted, however, that the journal tends to drop messages + when it becomes busy instead of forwarding them to the system log socket. + This is because the journal uses an async log socket interface for forwarding + instead of the traditional synchronous one. + +.. versionadded:: 8.32.0 + rsyslog emits an informational message noting the system log socket provided + by systemd. + +.. seealso:: + + :doc:`imjournal` + + +Handling of sockets +------------------- + +What follows is a brief description of the process rsyslog takes to determine +what system socket to use, which sockets rsyslog should listen on, whether +the sockets should be created and how rsyslog should handle the sockets when +shutting down. + +.. seealso:: + + `Writing syslog Daemons Which Cooperate Nicely With systemd + <https://www.freedesktop.org/wiki/Software/systemd/syslog/>`_ + + +Step 1: Select name of system socket +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +#. If the user has not explicitly chosen to set ``SysSock.Use="off"`` then + the default listener socket (aka, "system log socket" or simply "system + socket") name is set to ``/dev/log``. Otherwise, if the user `has` + explicitly set ``SysSock.Use="off"``, then rsyslog will not listen on + ``/dev/log`` OR any socket defined by the ``SysSock.Name`` parameter and + the rest of this section does not apply. + +#. If the user has specified ``sysSock.Name="/path/to/custom/socket"`` (and not + explicitly set ``SysSock.Use="off"``), then the default listener socket name + is overwritten with ``/path/to/custom/socket``. + +#. Otherwise, if rsyslog is running under systemd AND + ``/run/systemd/journal/syslog`` exists, (AND the user has not + explicitly set ``SysSock.Use="off"``) then the default listener socket name + is overwritten with ``/run/systemd/journal/syslog``. + + +Step 2: Listen on specified sockets +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. note:: + + This is true for all sockets, be it system socket or not. But if + ``SysSock.Use="off"``, the system socket will not be listened on. + +rsyslog evaluates the list of sockets it has been asked to activate: + +- the system log socket (if still enabled after completion of the last section) +- any custom inputs defined by the user + +and then checks to see if it has been passed in via systemd (name is checked). +If it was passed in via systemd, the socket is used as-is (e.g., not recreated +upon rsyslog startup), otherwise if not passed in via systemd the log socket +is unlinked, created and opened. + + +Step 3: Shutdown log sockets +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. note:: + + This is true for all sockets, be it system socket or not. + +Upon shutdown, rsyslog processes each socket it is listening on and evaluates +it. If the socket was originally passed in via systemd (name is checked), then +rsyslog does nothing with the socket (systemd maintains the socket). + +If the socket was `not` passed in via systemd AND the configuration permits +rsyslog to do so (the default setting), rsyslog will unlink/remove the log +socket. If not permitted to do so (the user specified otherwise), then rsyslog +will not unlink the log socket and will leave that cleanup step to the +user or application that created the socket to remove it. + + +Statistic Counter +================= + +This plugin maintains a global :doc:`statistics <../rsyslog_statistic_counter>` with the following properties: + +- ``submitted`` - total number of messages submitted for processing since startup + +- ``ratelimit.discarded`` - number of messages discarded due to rate limiting + +- ``ratelimit.numratelimiters`` - number of currently active rate limiters + (small data structures used for the rate limiting logic) + + +Caveats/Known Bugs +================== + +- When running under systemd, **many "sysSock." parameters are ignored**. + See parameter descriptions and the :ref:`imuxsock-systemd-details-label` section for + details. + +- On systems where systemd is used this module is often not loaded by default. + See the :ref:`imuxsock-systemd-details-label` section for details. + +- Application timestamps are ignored by default. See the + :ref:`imuxsock-application-timestamps-label` section for details. + +- `imuxsock does not work on Solaris + <http://www.rsyslog.com/why-does-imuxsock-not-work-on-solaris/>`_ + +.. todolist:: + + +Examples +======== + +Minimum setup +------------- + +The following sample is the minimum setup required to accept syslog +messages from applications running on the local system. + +.. code-block:: none + + module(load="imuxsock") + +This only needs to be done once. + + +Enable flow control +------------------- + +.. code-block:: none + :emphasize-lines: 2 + + module(load="imuxsock" # needs to be done just once + SysSock.FlowControl="on") # enable flow control (use if needed) + +Enable trusted properties +------------------------- + +As noted in the :ref:`imuxsock-trusted-properties-label` section, trusted properties +are disabled by default. If you want to use them, you must turn the feature +on via ``SysSock.Annotate`` for the system log socket and ``Annotate`` for +inputs. + +Append to end of message +^^^^^^^^^^^^^^^^^^^^^^^^ + +The following sample is used to activate message annotation and thus +trusted properties on the system log socket. These trusted properties +are appended to the end of each message. + +.. code-block:: none + :emphasize-lines: 2 + + module(load="imuxsock" # needs to be done just once + SysSock.Annotate="on") + + +Store in JSON message properties +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The following sample is similar to the first one, but enables parsing of +trusted properties, which places the results into JSON/lumberjack variables. + +.. code-block:: none + :emphasize-lines: 2 + + module(load="imuxsock" + SysSock.Annotate="on" SysSock.ParseTrusted="on") + +Read log data from jails +------------------------ + +The following sample is a configuration where rsyslogd pulls logs from +two jails, and assigns different hostnames to each of the jails: + +.. code-block:: none + :emphasize-lines: 3,4,5 + + module(load="imuxsock") # needs to be done just once + input(type="imuxsock" + HostName="jail1.example.net" + Socket="/jail/1/dev/log") input(type="imuxsock" + HostName="jail2.example.net" Socket="/jail/2/dev/log") + +Read from socket on temporary file system +----------------------------------------- + +The following sample is a configuration where rsyslogd reads the openssh +log messages via a separate socket, but this socket is created on a +temporary file system. As rsyslogd starts up before the sshd daemon, it needs +to create the socket directories, because it otherwise can not open the +socket and thus not listen to openssh messages. + +.. code-block:: none + :emphasize-lines: 3,4 + + module(load="imuxsock") # needs to be done just once + input(type="imuxsock" + Socket="/var/run/sshd/dev/log" + CreatePath="on") + + +Disable rate limiting +--------------------- + +The following sample is used to turn off input rate limiting on the +system log socket. + +.. code-block:: none + :emphasize-lines: 2 + + module(load="imuxsock" # needs to be done just once + SysSock.RateLimit.Interval="0") # turn off rate limiting + diff --git a/source/configuration/modules/index.rst b/source/configuration/modules/index.rst new file mode 100644 index 0000000..be423cd --- /dev/null +++ b/source/configuration/modules/index.rst @@ -0,0 +1,33 @@ +Modules +======= + +Rsyslog has a modular design. This enables functionality to be +dynamically loaded from modules, which may also be written by any third +party. Rsyslog itself offers all non-core functionality as modules. +Consequently, there is a growing number of modules. Here is the entry +point to their documentation and what they do (list is currently not +complete) + +Please note that each module provides (case-insensitive) configuration +parameters, which are NOT necessarily being listed below. Also remember, +that a modules configuration parameter (and functionality) is only +available if it has been loaded. + +It is relatively easy to write a rsyslog module. If none of the +provided modules solve your need, you may consider writing one or have +one written for you by `Adiscon's professional services for +rsyslog <http://www.rsyslog.com/professional-services>`_ (this often +is a very cost-effective and efficient way of getting what you need). + +There exist different classes of loadable modules: + +.. toctree:: + :maxdepth: 1 + + idx_output + idx_input + idx_parser + idx_messagemod + idx_stringgen + idx_library + workflow diff --git a/source/configuration/modules/mmanon.rst b/source/configuration/modules/mmanon.rst new file mode 100644 index 0000000..b1d1a4b --- /dev/null +++ b/source/configuration/modules/mmanon.rst @@ -0,0 +1,370 @@ +**************************************** +IP Address Anonymization Module (mmanon) +**************************************** + +=========================== =========================================================================== +**Module Name:** **mmanon** +**Author:** `Rainer Gerhards <https://rainer.gerhards.net/>`_ <rgerhards@adiscon.com> +**Available since:** 7.3.7 +=========================== =========================================================================== + + +Purpose +======= + +The mmanon module permits to anonymize IP addresses. It is a message +modification module that actually changes the IP address inside the +message, so after calling mmanon, the original message can no longer be +obtained. Note that anonymization will break digital signatures on the +message, if they exist. + +Please note that log files can also be anonymized via +`SLFA <http://jan.gerhards.net/p/slfa.html>`_ after they +have been created. + +*How are IP-Addresses defined?* + +We assume that an IPv4 address consists of four octets in dotted notation, +where each of the octets has a value between 0 and 255, inclusively. + +An IPv6 is defined by being bewtween zero and eight hex values between 0 +and ffff. These are separated by ':'. Leading zeros in blocks can be omitted +and blocks full of zeros can be abbreviated by using '::'. However, this +can ony happen once in an IP address. + +An IPv6 address with embedded IPv4 is an IPv6 address where the last two blocks +have been replaced by an IPv4 address. (see also: RFC4291, 2.2.3) + + +Configuration Parameters +======================== + +.. note:: + + Parameter names are case-insensitive. + + +Action Parameters +----------------- + +Parameters starting with 'IPv4.' will configure IPv4 anonymization, +while 'IPv6.' parameters do the same for IPv6 anonymization. + + +ipv4.enable +^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "on", "no", "none" + +Allows to enable or disable the anonymization of IPv4 addresses. + + +ipv4.mode +^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "zero", "no", "none" + +There exist the "simple", "random", "random-consitent", and "zero" +modes. In simple mode, only octets as whole can be anonymized +and the length of the message is never changed. This means +that when the last three octets of the address 10.1.12.123 are +anonymized, the result will be 10.0.00.000. This means that +the length of the original octets is still visible and may be used +to draw some privacy-evasive conclusions. This mode is slightly +faster than the other modes, and this may matter in high +throughput environments. + +The modes "random" and "random-consistent" are very similar, in +that they both anonymize ip-addresses by randomizing the last bits (any +number) of a given address. However, while "random" mode assigns a new +random ip-address for every address in a message, "random-consitent" will +assign the same randomized address to every instance of the same original address. + +The default "zero" mode will do full anonymization of any number +of bits and it will also normalize the address, so that no information +about the original IP address is available. So in the above example, +10.1.12.123 would be anonymized to 10.0.0.0. + + +ipv4.bits +^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "positive integer", "16", "no", "none" + +This sets the number of bits that should be anonymized (bits are from +the right, so lower bits are anonymized first). This setting permits +to save network information while still anonymizing user-specific +data. The more bits you discard, the better the anonymization +obviously is. The default of 16 bits reflects what German data +privacy rules consider as being sufficinetly anonymized. We assume, +this can also be used as a rough but conservative guideline for other +countries. +Note: when in simple mode, only bits on a byte boundary can be +specified. As such, any value other than 8, 16, 24 or 32 is invalid. +If an invalid value is given, it is rounded to the next byte boundary +(so we favor stronger anonymization in that case). For example, a bit +value of 12 will become 16 in simple mode (an error message is also +emitted). + + +ipv4.replaceChar +^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "char", "x", "no", "none" + +In simple mode, this sets the character that the to-be-anonymized +part of the IP address is to be overwritten with. In any other +mode the parameter is ignored if set. + + +ipv6.enable +^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "on", "no", "none" + +Allows to enable or disable the anonymization of IPv6 addresses. + + +ipv6.anonmode +^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "zero", "no", "none" + +This defines the mode, in which IPv6 addresses will be anonymized. +There exist the "random", "random-consistent", and "zero" modes. + +The modes "random" and "random-consistent" are very similar, in +that they both anonymize ip-addresses by randomizing the last bits (any +number) of a given address. However, while "random" mode assigns a new +random ip-address for every address in a message, "random-consistent" will +assign the same randomized address to every instance of the same original address. + +The default "zero" mode will do full anonymization of any number +of bits and it will also normalize the address, so that no information +about the original IP address is available. + +Also note that an anonymmized IPv6 address will be normalized, meaning +there will be no abbreviations, leading zeros will **not** be displayed, +and capital letters in the hex numerals will be lowercase. + + +ipv6.bits +^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "positive integer", "96", "no", "none" + +This sets the number of bits that should be anonymized (bits are from +the right, so lower bits are anonymized first). This setting permits +to save network information while still anonymizing user-specific +data. The more bits you discard, the better the anonymization +obviously is. The default of 96 bits reflects what German data +privacy rules consider as being sufficinetly anonymized. We assume, +this can also be used as a rough but conservative guideline for other +countries. + + +embeddedipv4.enable +^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "on", "no", "none" + +Allows to enable or disable the anonymization of IPv6 addresses with embedded IPv4. + + +embeddedipv4.anonmode +^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "zero", "no", "none" + +This defines the mode, in which IPv6 addresses will be anonymized. +There exist the "random", "random-consistent", and "zero" modes. + +The modes "random" and "random-consistent" are very similar, in +that they both anonymize ip-addresses by randomizing the last bits (any +number) of a given address. However, while "random" mode assigns a new +random ip-address for every address in a message, "random-consistent" will +assign the same randomized address to every instance of the same original address. + +The default "zero" mode will do full anonymization of any number +of bits and it will also normalize the address, so that no information +about the original IP address is available. + +Also note that an anonymmized IPv6 address will be normalized, meaning +there will be no abbreviations, leading zeros will **not** be displayed, +and capital letters in the hex numerals will be lowercase. + + +embeddedipv4.bits +^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "positive integer", "96", "no", "none" + +This sets the number of bits that should be anonymized (bits are from +the right, so lower bits are anonymized first). This setting permits +to save network information while still anonymizing user-specific +data. The more bits you discard, the better the anonymization +obviously is. The default of 96 bits reflects what German data +privacy rules consider as being sufficinetly anonymized. We assume, +this can also be used as a rough but conservative guideline for other +countries. + + +See Also +======== + +- `Howto anonymize messages that go to specific + files <http://www.rsyslog.com/howto-anonymize-messages-that-go-to-specific-files/>`_ + + +Caveats/Known Bugs +================== + +- will **not** anonymize addresses in the header + + +Examples +======== + +Anonymizing messages +-------------------- + +In this snippet, we write one file without anonymization and another one +with the message anonymized. Note that once mmanon has run, access to +the original message is no longer possible (execept if stored in user +variables before anonymization). + +.. code-block:: none + + module(load="mmanon") + action(type="omfile" file="/path/to/non-anon.log") + action(type="mmanon" ipv6.enable="off") + action(type="omfile" file="/path/to/anon.log") + + +Anonymizing a specific part of the ip address +--------------------------------------------- + +This next snippet is almost identical to the first one, but here we +anonymize the full IPv4 address. Note that by modifying the number of +bits, you can anonymize different parts of the address. Keep in mind +that in simple mode (used here), the bit values must match IP address +bytes, so for IPv4 only the values 8, 16, 24 and 32 are valid. Also, in +this example the replacement is done via asterisks instead of lower-case +"x"-letters. Also keep in mind that "replacementChar" can only be set in +simple mode. + +.. code-block:: none + + module(load="mmanon") action(type="omfile" file="/path/to/non-anon.log") + action(type="mmanon" ipv4.bits="32" ipv4.mode="simple" replacementChar="\*" ipv6.enable="off") + action(type="omfile" file="/path/to/anon.log") + + +Anonymizing an odd number of bits +--------------------------------- + +The next snippet is also based on the first one, but anonymizes an "odd" +number of bits, 12. The value of 12 is used by some folks as a +compromise between keeping privacy and still permitting to gain some more +in-depth insight from log files. Note that anonymizing 12 bits may be +insufficient to fulfill legal requirements (if such exist). + +.. code-block:: none + + module(load="mmanon") action(type="omfile" file="/path/to/non-anon.log") + action(type="mmanon" ipv4.bits="12" ipv6.enable="off") action(type="omfile" + file="/path/to/anon.log") + + +Anonymizing ipv4 and ipv6 addresses +----------------------------------- + +You can also anonymize IPv4 and IPv6 in one go using a configuration like this. + +.. code-block:: none + + module(load="mmanon") action(type="omfile" file="/path/to/non-anon.log") + action(type="mmanon" ipv4.bits="12" ipv6.bits="128" ipv6.anonmode="random") action(type="omfile" + file="/path/to/anon.log") + + +Anonymizing with default values +------------------------------- + +It is also possible to use the default configuration for both types of +anonymization. This will result in IPv4 addresses being anonymized in zero +mode anonymizing 16 bits. IPv6 addresses will also be anonymized in zero +mode anonymizing 96 bits. + +.. code-block:: none + + module(load="mmanon") + action(type="omfile" file="/path/to/non-anon.log") + action(type="mmanon") + action(type="omfile" file="/path/to/anon.log") + + +Anonymizing only ipv6 addresses +------------------------------- + +Another option is to only anonymize IPv6 addresses. When doing this you have to +disable IPv4 aonymization. This example will lead to only IPv6 addresses anonymized +(using the random-consistent mode). + +.. code-block:: none + + module(load="mmanon") + action(type="omfile" file="/path/to/non-anon.log") + action(type="mmanon" ipv4.enable="off" ipv6.anonmode="random-consistent") + action(type="omfile" file="/path/to/anon.log") + diff --git a/source/configuration/modules/mmcount.rst b/source/configuration/modules/mmcount.rst new file mode 100644 index 0000000..de3247d --- /dev/null +++ b/source/configuration/modules/mmcount.rst @@ -0,0 +1,56 @@ +******* +mmcount +******* + +=========================== =========================================================================== +**Module Name:** **mmcount** +**Author:** Bala.FA <barumuga@redhat.com> +**Available since:** 7.5.0 +=========================== =========================================================================== + + +**Status:**\ Non project-supported module - contact author or rsyslog +mailing list for questions + + +Purpose +======= + +Message modification plugin which counts messages. + +This module provides the capability to count log messages by severity +or json property of given app-name. The count value is added into the +log message as json property named 'mmcount'. + + +Examples +======== + +Example usage of the module in the configuration file. + +.. code-block:: none + + module(load="mmcount") + + # count each severity of appname gluster + action(type="mmcount" appname="gluster") + + # count each value of gf_code of appname gluster + action(type="mmcount" appname="glusterd" key="!gf_code") + + # count value 9999 of gf_code of appname gluster + action(type="mmcount" appname="glusterfsd" key="!gf_code" value="9999") + + # send email for every 50th mmcount + if $app-name == 'glusterfsd' and $!mmcount <> 0 and $!mmcount % 50 == 0 then { + $ActionMailSMTPServer smtp.example.com + $ActionMailFrom rsyslog@example.com + $ActionMailTo glusteradmin@example.com + $template mailSubject,"50th message of gf_code=9999 on %hostname%" + $template mailBody,"RSYSLOG Alert\r\nmsg='%msg%'" + $ActionMailSubject mailSubject + $ActionExecOnlyOnceEveryInterval 30 + :ommail:;RSYSLOG_SyslogProtocol23Format + } + + diff --git a/source/configuration/modules/mmdarwin.rst b/source/configuration/modules/mmdarwin.rst new file mode 100644 index 0000000..17d8c0d --- /dev/null +++ b/source/configuration/modules/mmdarwin.rst @@ -0,0 +1,229 @@ +.. index:: ! mmdarwin + +.. role:: json(code) + :language: json + +*************************** +Darwin connector (mmdarwin) +*************************** + +================ =========================================== +**Module Name:** **mmdarwin** +**Author:** Guillaume Catto <guillaume.catto@advens.fr>, + Theo Bertin <theo.bertin@advens.fr> +================ =========================================== + +Purpose +======= + +Darwin is an open source Artificial Intelligence Framework for CyberSecurity. The mmdarwin module allows us to call Darwin in order to enrich our JSON-parsed logs with a score, and/or to allow Darwin to generate alerts. + +How to build the module +======================= + +To compile Rsyslog with mmdarwin you'll need to: + +* set *--enable-mmdarwin* on configure + +Configuration Parameter +======================= + +Input Parameters +---------------- + +key +^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "yes", "none" + +The key name to use to store the returned data. + +For example, given the following log line: + +.. code-block:: json + + { + "from": "192.168.1.42", + "date": "2012-12-21 00:00:00", + "status": "200", + "data": { + "status": true, + "message": "Request processed correctly" + } + } + +and the :json:`"certitude"` key, the enriched log line would be: + +.. code-block:: json + :emphasize-lines: 9 + + { + "from": "192.168.1.42", + "date": "2012-12-21 00:00:00", + "status": "200", + "data": { + "status": true, + "message": "Request processed correctly" + }, + "certitude": 0 + } + +where :json:`"certitude"` represents the score returned by Darwin. + + +socketpath +^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "yes", "none" + +The Darwin filter socket path to use. + + +response +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", \"no", "no", "none" + +Tells the Darwin filter what to do next: + +* :json:`"no"`: no response will be sent, nothing will be sent to next filter. +* :json:`"back"`: a score for the input will be returned by the filter, nothing will be forwarded to the next filter. +* :json:`"darwin"`: the data provided will be forwarded to the next filter (in the format specified in the filter's configuration), no response will be given to mmdarwin. +* :json:`"both"`: the filter will respond to mmdarwin with the input's score AND forward the data (in the format specified in the filter's configuration) to the next filter. + +.. note:: + + Please be mindful when setting this parameter, as the called filter will only forward data to the next configured filter if you ask the filter to do so with :json:`"darwin"` or :json:`"both"`, if a next filter if configured but you ask for a :json:`"back"` response, the next filter **WILL NOT** receive anything! + +filtercode +^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "0x00000000", "no", "none" + +Each Darwin module has a unique filter code. For example, the code of the hostlookup filter is :json:`"0x66726570"`. +This code was mandatory but has now become obsolete. you can leave it as it is. + +fields +^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "array", "none", "yes", "none" + +Array containing values to be sent to Darwin as parameters. + +Two types of values can be set: + +* if it starts with a bang (:json:`"!"`), mmdarwin will search in the JSON-parsed log line the associated value. You can search in subkeys as well: just add a bang to go to a deeper level. +* otherwise, the value is considered static, and will be forwarded directly to Darwin. + +For example, given the following log line: + +.. code-block:: json + + { + "from": "192.168.1.42", + "date": "2012-12-21 00:00:00", + "status": "200", + "data": { + "status": true, + "message": "Request processed correctly" + } + } + +and the :json:`"fields"` array: + +.. code-block:: none + + ["!from", "!data!status", "rsyslog"] + +The parameters sent to Darwin would be :json:`"192.168.1.42"`, :json:`true` and :json:`"rsyslog"`. + +.. note:: + The order of the parameters is important. Thus, you have to be careful when providing the fields in the array. + Refer to `Darwin documentation`_ to see what each filter requires as parameters. + +.. _`Darwin documentation`: https://github.com/VultureProject/darwin/wiki + +send_partial +^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "boolean", "off", "no", "none" + +Whether to send to Darwin if not all :json:`"fields"` could be found in the message, or not. +All current Darwin filters required a strict number (and format) of parameters as input, so they will most likely not process the data if some fields are missing. This should be kept to "off", unless you know what you're doing. + +For example, for the following log line: + +.. code-block:: json + + { + "from": "192.168.1.42", + "date": "2012-12-21 00:00:00", + "status": "200", + "data": { + "status": true, + "message": "Request processed correctly" + } + } + +and the :json:`"fields"` array: + +.. code-block:: none + + ["!from", "!data!status", "!this!field!is!not!in!message"] + +the third field won't be found, so the call to Darwin will be dropped. + + +Configuration example +===================== + +This example shows a possible configuration of mmdarwin. + +.. code-block:: none + + module(load="imtcp") + module(load="mmjsonparse") + module(load="mmdarwin") + + input(type="imtcp" port="8042" Ruleset="darwin_ruleset") + + ruleset(name="darwin_ruleset") { + action(type="mmjsonparse" cookie="") + action(type="mmdarwin" socketpath="/path/to/reputation_1.sock" fields=["!srcip", "ATTACK;TOR"] key="reputation" response="back" filtercode="0x72657075") + + call darwin_output + } + + ruleset(name="darwin_output") { + action(type="omfile" file="/path/to/darwin_output.log") + } diff --git a/source/configuration/modules/mmdblookup.rst b/source/configuration/modules/mmdblookup.rst new file mode 100644 index 0000000..d92f849 --- /dev/null +++ b/source/configuration/modules/mmdblookup.rst @@ -0,0 +1,141 @@ +.. index:: ! mmdblookup + +************************************ +MaxMind/GeoIP DB lookup (mmdblookup) +************************************ + +================ ================================== +**Module Name:** mmdblookup +**Author:** `chenryn <rao.chenlin@gmail.com>`_ +**Available:** 8.24+ +================ ================================== + + +Purpose +======= + +MaxMindDB is the new file format for storing information about IP addresses +in a highly optimized, flexible database format. GeoIP2 Databases are +available in the MaxMind DB format. + +Plugin author claimed a MaxMindDB vs GeoIP speed around 4 to 6 times. + + +How to build the module +======================= + +To compile Rsyslog with mmdblookup you'll need to: + +* install *libmaxminddb-devel* package +* set *--enable-mmdblookup* on configure + + +Configuration Parameter +======================= + +.. note:: + + Parameter names are case-insensitive. + + +Module Parameters +----------------- + +container +^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "!iplocation", "no", "none" + +.. versionadded:: 8.28.0 + +Specifies the container to be used to store the fields amended by +mmdblookup. + + +Input Parameters +---------------- + +key +^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "yes", "none" + +Name of field containing IP address. + + +mmdbfile +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "yes", "none" + +Location of Maxmind DB file. + + +fields +^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "array", "none", "yes", "none" + +Fields that will be appended to processed message. The fields will +always be appended in the container used by mmdblookup (which may be +overridden by the "container" parameter on module load). + +By default, the maxmindb field name is used for variables. This can +be overridden by specifying a custom name between colons at the +beginning of the field name. As usual, bang signs denote path levels. +So for example, if you want to extract "!city!names!en" but rename it +to "cityname", you can use ":cityname:!city!names!en" as field name. + + +Examples +======== + +Minimum configuration +--------------------- + +This example shows the minimum configuration. + +.. code-block:: none + + # load module + module( load="mmdblookup" ) + + action( type="mmdblookup" mmdbfile="/etc/rsyslog.d/GeoLite2-City.mmdb" + fields=["!continent!code","!location"] key="!clientip" ) + + +Custom container and field name +------------------------------- + +The following example uses a custom container and custom field name + +.. code-block:: none + + # load module + module( load="mmdblookup" container="!geo_ip") + + action( type="mmdblookup" mmdbfile="/etc/rsyslog.d/GeoLite2-City.mmdb" + fields=[":continent:!continent!code", ":loc:!location"] + key="!clientip") + + diff --git a/source/configuration/modules/mmexternal.rst b/source/configuration/modules/mmexternal.rst new file mode 100644 index 0000000..afc85cc --- /dev/null +++ b/source/configuration/modules/mmexternal.rst @@ -0,0 +1,110 @@ +******************************************************** +Support module for external message modification modules +******************************************************** + +=========================== =========================================================================== +**Module Name:** **mmexternal** +**Author:** `Rainer Gerhards <https://rainer.gerhards.net/>`_ <rgerhards@adiscon.com> +**Available since:** 8.3.0 +=========================== =========================================================================== + + +Purpose +======= + +This module permits to integrate external message modification plugins +into rsyslog. + +For details on the interface specification, see rsyslog's source in the +./plugins/external/INTERFACE.md. + + +Configuration Parameters +======================== + +.. note:: + + Parameter names are case-insensitive. + + +Action Parameters +----------------- + +binary +^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "none", "yes", "none" + +The name of the external message modification plugin to be called. This +can be a full path name. + + +interface.input +^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "msg", "no", "none" + +This can either be "msg", "rawmsg" or "fulljson". In case of "fulljson", the +message object is provided as a json object. Otherwise, the respective +property is provided. This setting **must** match the external plugin's +expectations. Check the external plugin documentation for what needs to be used. + + +output +^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "none", "no", "none" + +This is a debug aid. If set, this is a filename where the plugins output +is logged. Note that the output is also being processed as usual by rsyslog. +Setting this parameter thus gives insight into the internal processing +that happens between plugin and rsyslog core. + + +forceSingleInstance +^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +This is an expert parameter, just like the equivalent *omprog* parameter. +See the message modification plugin's documentation if it is needed. + + +Examples +======== + +Execute external module +----------------------- + +The following config file snippet is used to write execute an external +message modification module "mmexternal.py". Note that the path to the +module is specified here. This is necessary if the module is not in the +default search path. + +.. code-block:: none + + module (load="mmexternal") # needs to be done only once inside the config + + action(type="mmexternal" binary="/path/to/mmexternal.py") + + diff --git a/source/configuration/modules/mmfields.rst b/source/configuration/modules/mmfields.rst new file mode 100644 index 0000000..876a7ca --- /dev/null +++ b/source/configuration/modules/mmfields.rst @@ -0,0 +1,132 @@ +*********************************** +Fields Extraction Module (mmfields) +*********************************** + +=========================== =========================================================================== +**Module Name:** **mmfields** +**Author:** `Rainer Gerhards <https://rainer.gerhards.net/>`_ <rgerhards@adiscon.com> +**Available since:** 7.5.1 +=========================== =========================================================================== + + +Purpose +======= + +The mmfield module permits to extract fields. It is an alternate to +using the property replacer field extraction capabilities. In contrast +to the property replacer, all fields are extracted as once and stored +inside the structured data part (more precisely: they become Lumberjack +[JSON] properties). + +Using this module is of special advantage if a field-based log format is +to be processed, like for example CEF **and** either a large number +of fields is needed or a specific field is used multiple times inside +filters. In these scenarios, mmfields potentially offers better +performance than the property replacer of the RainerScript field +extraction method. The reason is that mmfields extracts all fields as +one big sweep, whereas the other methods extract fields individually, +which requires multiple passes through the same data. On the other hand, +adding field content to the rsyslog property dictionary also has some +overhead, so for high-performance use cases it is suggested to do some +performance testing before finally deciding which method to use. This is +most important if only a smaller subset of the fields is actually +needed. + +In any case, mmfields provides a very handy and easy to use way to parse +structured data into a it's individual data items. Again, a primary use +case was support for CEF (Common Event Format), which is made extremely +easy to do with this module. + +This module is implemented via the action interface. Thus it can be +conditionally used depending on some prerequisites. + + +Configuration Parameters +======================== + +.. note:: + + Parameter names are case-insensitive. + + +Action Parameters +----------------- + +separator +^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "char", ",", "no", "none" + +This is the character used to separate fields. Currently, only a +single character is permitted, while the RainerScript method permits +to specify multi-character separator strings. For CEF, this is not +required. If there is actual need to support multi-character +separator strings, support can relatively easy be added. It is +suggested to request it on the rsyslog mailing list, together with +the use case - we intend to add functionality only if there is a real +use case behind the request (in the past we too-often implemented +things that actually never got used). +The fields are named f\ *nbr*, where *nbr* is the field number +starting with one and being incremented for each field. + + +jsonRoot +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "!", "no", "none" + +This parameters specifies into which json path the extracted fields +shall be written. The default is to use the json root object itself. + + +Examples +======== + +Parsing messages and writing them to file +----------------------------------------- + +This is a very simple use case where each message is parsed. The default +separator character of comma is being used. + +.. code-block:: none + + module(load="mmfields") + template(name="ftpl" + type="string" + string="%$!%\\n") + action(type="mmfields") + action(type="omfile" + file="/path/to/logfile" + template="ftpl") + + +Writing into a specific json path +--------------------------------- + +The following sample is similar to the previous one, but this time the +colon is used as separator and data is written into the "$!mmfields" +json path. + +.. code-block:: none + + module(load="mmfields") + template(name="ftpl" + type="string" + string="%$!%\\n") + action(type="mmfields" + separator=":" + jsonRoot="!mmfields") + action(type="omfile" + file="/path/to/logfile" + template="ftpl") + diff --git a/source/configuration/modules/mmjsonparse.rst b/source/configuration/modules/mmjsonparse.rst new file mode 100644 index 0000000..3d47c4f --- /dev/null +++ b/source/configuration/modules/mmjsonparse.rst @@ -0,0 +1,147 @@ +*********************************************************** +JSON/CEE Structured Content Extraction Module (mmjsonparse) +*********************************************************** + +=========================== =========================================================================== +**Module Name:** **mmjsonparse** +**Author:** `Rainer Gerhards <https://rainer.gerhards.net/>`_ <rgerhards@adiscon.com> +**Available since:** 6.6.0 +=========================== =========================================================================== + + +Purpose +======= + +This module provides support for parsing structured log messages that +follow the CEE/lumberjack spec. The so-called "CEE cookie" is checked +and, if present, the JSON-encoded structured message content is parsed. +The properties are then available as original message properties. + +As a convenience, mmjsonparse will produce a valid CEE/lumberjack log +message if passed a message without the CEE cookie. A JSON structure +will be created and the "msg" field will be the only field and it will +contain the message. Note that in this case, mmjsonparse will +nonetheless return that the JSON parsing has failed. + +The "CEE cookie" is the character squence "@cee:" which must prepend the +actual JSON. Note that the JSON must be valid and MUST NOT be followed +by any non-JSON message. If either of these conditions is not true, +mmjsonparse will **not** parse the associated JSON. This is based on the +cookie definition used in CEE/project lumberjack and is meant to aid +against an erroneous detection of a message as being CEE where it is +not. + +This also means that mmjsonparse currently is NOT a generic JSON parser +that picks up JSON from wherever it may occur in the message. This is +intentional, but future versions may support config parameters to relax +the format requirements. + + +Notable Features +================ + +- :ref:`mmjsonparse-parsing-result` + + +Configuration Parameters +======================== + +.. note:: + + Parameter names are case-insensitive. + + +Action Parameters +----------------- + +cookie +^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "@cee:", "no", "none" + +Permits to set the cookie that must be present in front of the +JSON part of the message. + +Most importantly, this can be set to the empty string ("") in order +to not require any cookie. In this case, leading spaces are permitted +in front of the JSON. No non-whitespace characters are permitted +after the JSON. If such is required, mmnormalize must be used. + + +useRawMsg +^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +Specifies if the raw message should be used for normalization (on) +or just the MSG part of the message (off). + + +container +^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "$!", "no", "none" + +Specifies the JSON container (path) under which parsed elements should be +placed. By default, all parsed properties are merged into root of +message properties. You can place them under a subtree, instead. You +can place them in local variables, also, by setting path="$.". + + +.. _mmjsonparse-parsing-result: + +Check parsing result +==================== + +You can check whether rsyslogd was able to successfully parse the +message by reading the $parsesuccess variable : + +.. code-block:: none + + action(type="mmjsonparse") + if $parsesuccess == "OK" then { + action(type="omfile" File="/tmp/output") + } + else if $parsesuccess == "FAIL" then { + action(type="omfile" File="/tmp/parsing_failure") + } + + +Examples +======== + +Apply default normalization +--------------------------- + +This activates the module and applies normalization to all messages. + +.. code-block:: none + + module(load="mmjsonparse") + action(type="mmjsonparse") + + +Permit parsing messages without cookie +-------------------------------------- + +To permit parsing messages without cookie, use this action statement + +.. code-block:: none + + action(type="mmjsonparse" cookie="") + diff --git a/source/configuration/modules/mmkubernetes.rst b/source/configuration/modules/mmkubernetes.rst new file mode 100644 index 0000000..83efc18 --- /dev/null +++ b/source/configuration/modules/mmkubernetes.rst @@ -0,0 +1,630 @@ +***************************************** +Kubernetes Metadata Module (mmkubernetes) +***************************************** + +=========================== =========================================================================== +**Module Name:** **mmkubernetes** +**Author:** `Tomáš Heinrich` + `Rich Megginson` <rmeggins@redhat.com> +=========================== =========================================================================== + +Purpose +======= + +This module is used to add `Kubernetes <https://kubernetes.io/>` +metadata to log messages logged by containers running in Kubernetes. +It will add the namespace uuid, pod uuid, pod and namespace labels and +annotations, and other metadata associated with the pod and +namespace. + +.. note:: + + This **only** works with log files in `/var/log/containers/*.log` (docker + `--log-driver=json-file`, or CRI-O log files), or with journald entries with + message properties `CONTAINER_NAME` and `CONTAINER_ID_FULL` (docker + `--log-driver=journald`), and when the application running inside the + container writes logs to `stdout`/`stderr`. This **does not** currently + work with other log drivers. + +For json-file and CRI-O logs, you must use the `imfile` module with the +`addmetadata="on"` parameter, and the filename must match the +liblognorm rules specified by the `filenamerules` +(:ref:`filenamerules`) or `filenamerulebase` (:ref:`filenamerulebase`) +parameter values. + +For journald logs, there must be a message property `CONTAINER_NAME` +which matches the liblognorm rules specified by the `containerrules` +(:ref:`containerrules`) or `containerrulebase` +(:ref:`containerrulebase`) parameter values. The record must also have +the message property `CONTAINER_ID_FULL`. + +This module is implemented via the output module interface. This means +that mmkubernetes should be called just like an action. After it has +been called, there will be two new message properties: `kubernetes` +and `docker`. There will be subfields of each one for the various +metadata items: `$!kubernetes!namespace_name` +`$!kubernetes!labels!this-is-my-label`, etc. There is currently only +1 docker subfield: `$!docker!container_id`. See +https://github.com/ViaQ/elasticsearch-templates/blob/master/namespaces/kubernetes.yml +and +https://github.com/ViaQ/elasticsearch-templates/blob/master/namespaces/docker.yml +for more details. + +Configuration Parameters +======================== + +.. note:: + + Parameter names are case-insensitive. + +Module Parameters and Action Parameters +--------------------------------------- + +.. _kubernetesurl: + +KubernetesURL +^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "https://kubernetes.default.svc.cluster.local:443", "no", "none" + +The URL of the Kubernetes API server. Example: `https://localhost:8443`. + +.. _mmkubernetes-tls.cacert: + +tls.cacert +^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +Full path and file name of file containing the CA cert of the +Kubernetes API server cert issuer. Example: `/etc/rsyslog.d/mmk8s-ca.crt`. +This parameter is not mandatory if using an `http` scheme instead of `https` in +`kubernetesurl`, or if using `allowunsignedcerts="yes"`. + +.. _mmkubernetes-tls.mycert: + +tls.mycert +^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +This is the full path and file name of the file containing the client cert for +doing client cert auth against Kubernetes. This file is in PEM format. For +example: `/etc/rsyslog.d/k8s-client-cert.pem` + +.. _mmkubernetes-tls.myprivkey: + +tls.myprivkey +^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +This is the full path and file name of the file containing the private key +corresponding to the cert `tls.mycert` used for doing client cert auth against +Kubernetes. This file is in PEM format, and must be unencrypted, so take +care to secure it properly. For example: `/etc/rsyslog.d/k8s-client-key.pem` + +.. _tokenfile: + +tokenfile +^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +The file containing the token to use to authenticate to the Kubernetes API +server. One of `tokenfile` or `token` is required if Kubernetes is configured +with access control. Example: `/etc/rsyslog.d/mmk8s.token` + +.. _token: + +token +^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +The token to use to authenticate to the Kubernetes API server. One of `token` +or `tokenfile` is required if Kubernetes is configured with access control. +Example: `UxMU46ptoEWOSqLNa1bFmH` + +.. _annotation_match: + +annotation_match +^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "array", "none", "no", "none" + +By default no pod or namespace annotations will be added to the +messages. This parameter is an array of patterns to match the keys of +the `annotations` field in the pod and namespace metadata to include +in the `$!kubernetes!annotations` (for pod annotations) or the +`$!kubernetes!namespace_annotations` (for namespace annotations) +message properties. Example: `["k8s.*master","k8s.*node"]` + +.. _srcmetadatapath: + +srcmetadatapath +^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "$!metadata!filename", "no", "none" + +When reading json-file logs, with `imfile` and `addmetadata="on"`, +this is the property where the filename is stored. + +.. _dstmetadatapath: + +dstmetadatapath +^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "$!", "no", "none" + +This is the where the `kubernetes` and `docker` properties will be +written. By default, the module will add `$!kubernetes` and +`$!docker`. + +.. _allowunsignedcerts: + +allowunsignedcerts +^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "boolean", "off", "no", "none" + +If `"on"`, this will set the curl `CURLOPT_SSL_VERIFYPEER` option to +`0`. You are strongly discouraged to set this to `"on"`. It is +primarily useful only for debugging or testing. + +.. _skipverifyhost: + +skipverifyhost +^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "boolean", "off", "no", "none" + +If `"on"`, this will set the curl `CURLOPT_SSL_VERIFYHOST` option to +`0`. You are strongly discouraged to set this to `"on"`. It is +primarily useful only for debugging or testing. + +.. _de_dot: + +de_dot +^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "boolean", "on", "no", "none" + +When processing labels and annotations, if this parameter is set to +`"on"`, the key strings will have their `.` characters replaced with +the string specified by the `de_dot_separator` parameter. + +.. _de_dot_separator: + +de_dot_separator +^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "_", "no", "none" + +When processing labels and annotations, if the `de_dot` parameter is +set to `"on"`, the key strings will have their `.` characters replaced +with the string specified by the string value of this parameter. + +.. _filenamerules: + +filenamerules +^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "SEE BELOW", "no", "none" + +.. note:: + + This directive is not supported with liblognorm 2.0.2 and earlier. + +When processing json-file logs, these are the lognorm rules to use to +match the filename and extract metadata. The default value is:: + + rule=:/var/log/containers/%pod_name:char-to:_%_%namespace_name:char-to:_%_%contai\ + ner_name_and_id:char-to:.%.log + +.. note:: + + In the above rules, the slashes ``\`` ending each line indicate + line wrapping - they are not part of the rule. + +.. _filenamerulebase: + +filenamerulebase +^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "/etc/rsyslog.d/k8s_filename.rulebase", "no", "none" + +When processing json-file logs, this is the rulebase used to match the filename +and extract metadata. For the actual rules, see :ref:`filenamerules`. + +.. _containerrules: + +containerrules +^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "SEE BELOW", "no", "none" + +.. note:: + + This directive is not supported with liblognorm 2.0.2 and earlier. + +For journald logs, there must be a message property `CONTAINER_NAME` +which has a value matching these rules specified by this parameter. +The default value is:: + + rule=:%k8s_prefix:char-to:_%_%container_name:char-to:.%.%container_hash:char-to:\ + _%_%pod_name:char-to:_%_%namespace_name:char-to:_%_%not_used_1:char-to:_%_%not_u\ + sed_2:rest% + rule=:%k8s_prefix:char-to:_%_%container_name:char-to:_%_%pod_name:char-to:_%_%na\ + mespace_name:char-to:_%_%not_used_1:char-to:_%_%not_used_2:rest% + +.. note:: + + In the above rules, the slashes ``\`` ending each line indicate + line wrapping - they are not part of the rule. + +There are two rules because the `container_hash` is optional. + +.. _containerrulebase: + +containerrulebase +^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "/etc/rsyslog.d/k8s_container_name.rulebase", "no", "none" + +When processing json-file logs, this is the rulebase used to match the +CONTAINER_NAME property value and extract metadata. For the actual rules, see +:ref:`containerrules`. + +.. _mmkubernetes-busyretryinterval: + +busyretryinterval +^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "5", "no", "none" + +The number of seconds to wait before retrying operations to the Kubernetes API +server after receiving a `429 Busy` response. The default `"5"` means that the +module will retry the connection every `5` seconds. Records processed during +this time will _not_ have any additional metadata associated with them, so you +will need to handle cases where some of your records have all of the metadata +and some do not. + +If you want to have rsyslog suspend the plugin until the Kubernetes API server +is available, set `busyretryinterval` to `"0"`. This will cause the plugin to +return an error to rsyslog. + +.. _mmkubernetes-sslpartialchain: + +sslpartialchain +^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "boolean", "off", "no", "none" + +This option is only available if rsyslog was built with support for OpenSSL and +only if the `X509_V_FLAG_PARTIAL_CHAIN` flag is available. If you attempt to +set this parameter on other platforms, you will get an `INFO` level log +message. This was done so that you could use the same configuration on +different platforms. +If `"on"`, this will set the OpenSSL certificate store flag +`X509_V_FLAG_PARTIAL_CHAIN`. This will allow you to verify the Kubernetes API +server cert with only an intermediate CA cert in your local trust store, rather +than having to have the entire intermediate CA + root CA chain in your local +trust store. See also `man s_client` - the `-partial_chain` flag. +If you get errors like this, you probably need to set `sslpartialchain="on"`: + +.. code-block:: none + + rsyslogd: mmkubernetes: failed to connect to [https://...url...] - + 60:Peer certificate cannot be authenticated with given CA certificates + +.. _mmkubernetes-cacheexpireinterval: + +cacheexpireinterval +^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "-1", "no", "none" + +This parameter allows you to expire entries from the metadata cache. The +values are: + +- -1 (default) - disables metadata cache expiration +- 0 - check cache for expired entries before every cache lookup +- 1 or higher - the number is a number of seconds - check the cache + for expired entries every this many seconds, when processing an + entry + +The cache is only checked if processing a record from Kubernetes. There +isn't some sort of housekeeping thread that continually runs cleaning up +the cache. When an record from Kubernetes is processed: + +If `cacheexpireinterval` is -1, then do not check for cache expiration. +If `cacheexpireinterval` is 0, then check for cache expiration. +If `cacheexpireinterval` is greater than 0, check for cache expiration +if the last time we checked was more than this many seconds ago. + +When cache expiration is checked, it will delete all cache entries which +have a ttl less than or equal to the current time. The cache entry ttl +is set using the `cacheentryttl`. + +.. _mmkubernetes-cacheentryttl: + +cacheentryttl +^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "3600", "no", "none" + +This parameter allows you to set the maximum age (time-to-live, or ttl) of +an entry in the metadata cache. The value is in seconds. The default value +is `3600` (one hour). When cache expiration is checked, if a cache entry +has a ttl less than or equal to the current time, it will be removed from +the cache. + +This option is only used if `cacheexpireinterval` is 0 or greater. + +This value must be 0 or greater, otherwise, if `cacheexpireinterval` is 0 +or greater, you will get an error. + +.. _mmkubernetes-statistic-counter: + +Statistic Counter +================= + +This plugin maintains per-action :doc:`statistics +<../rsyslog_statistic_counter>`. The statistic is named +"mmkubernetes($kubernetesurl)", where `$kubernetesurl` is the +:ref:`kubernetesurl` setting for the action. + +Parameters are: + +- **recordseen** - number of messages seen by the action which the action has + determined have Kubernetes metadata associated with them + +- **namespacemetadatasuccess** - the number of times a successful request was + made to the Kubernetes API server for namespace metadata. + +- **namespacemetadatanotfound** - the number of times a request to the + Kubernetes API server for namespace metadata was returned with a `404 Not + Found` error code - the namespace did not exist at that time. + +- **namespacemetadatabusy** - the number of times a request to the Kubernetes + API server for namespace metadata was returned with a `429 Busy` error + code - the server was too busy to send a proper response. + +- **namespacemetadataerror** - the number of times a request to the Kubernetes + API server for namespace metadata was returned with some other error code + not handled above. These are typically "hard" errors which require some + sort of intervention to fix e.g. Kubernetes server down, credentials incorrect. + +- **podmetadatasuccess** - the number of times a successful request was made + to the Kubernetes API server for pod metadata. + +- **podmetadatanotfound** - the number of times a request to the Kubernetes + API server for pod metadata was returned with a `404 Not Found` error code - + the pod did not exist at that time. + +- **podmetadatabusy** - the number of times a request to the Kubernetes API + server for pod metadata was returned with a `429 Busy` error code - the + server was too busy to send a proper response. + +- **podmetadataerror** - the number of times a request to the Kubernetes API + server for pod metadata was returned with some other error code not handled + above. These are typically "hard" errors which require some sort of + intervention to fix e.g. Kubernetes server down, credentials incorrect. + +- **podcachenumentries** - the number of entries in the pod metadata cache. + +- **namespacecachenumentries** - the number of entries in the namespace metadata + cache. + +- **podcachehits** - the number of times a requested entry was found in the + pod metadata cache. + +- **namespacecachehits** - the number of times a requested entry was found in the + namespace metadata cache. + +- **podcachemisses** - the number of times a requested entry was not found in the + pod metadata cache, and had to be requested from Kubernetes. + +- **namespacecachemisses** - the number of times a requested entry was not found + in the namespace metadata cache, and had to be requested from Kubernetes. + +Fields +------ + +These are the fields added from the metadata in the json-file filename, or from +the `CONTAINER_NAME` and `CONTAINER_ID_FULL` fields from the `imjournal` input: + +`$!kubernetes!namespace_name`, `$!kubernetes!pod_name`, +`$!kubernetes!container_name`, `$!docker!id`, `$!kubernetes!master_url`. + +If mmkubernetes can extract the above fields from the input, the following +fields will always be present. If they are not present, mmkubernetes +failed to look up the namespace or pod in Kubernetes: + +`$!kubernetes!namespace_id`, `$!kubernetes!pod_id`, +`$!kubernetes!creation_timestamp`, `$!kubernetes!host` + +The following fields may be present, depending on how the namespace and pod are +defined in Kubernetes, and depending on the value of the directive +`annotation_match`: + +`$!kubernetes!labels`, `$!kubernetes!annotations`, `$!kubernetes!namespace_labels`, +`$!kubernetes!namespace_annotations` + +More fields may be added in the future. + +Error Handling +-------------- +If the plugin encounters a `404 Not Found` in response to a request for +namespace or pod metadata, that is, the pod or namespace is missing, the plugin +will cache that result, and no metadata will be available for that pod or +namespace forever. If the pod or namespace is recreated, you will need to +restart rsyslog in order to clear the cache and allow it to find that metadata. + +If the plugin gets a `429 Busy` response, the plugin will _not_ cache that +result, and will _not_ add the metadata to the record. This can happen in very +large Kubernetes clusters when you run into the upper limit on the number of +concurrent Kubernetes API service connections. You may have to increase that +limit. In the meantime, you can control what the plugin does with those +records using the :ref:`mmkubernetes-busyretryinterval` setting. If you want +to continue to process the records, but with incomplete metadata, set +`busyretryinterval` to a non-zero value, which will be the number of seconds +after which mmkubernetes will retry the connection. The default value is `5`, +so by default, the plugin will retry the connection every `5` seconds. If the +`429` error condition in the Kubernetes API server is brief and transient, this +means you will have some (hopefully small) number of records without the +metadata such as the uuids, labels, and annotations, but your pipeline will not +stop. If the `429` error condition in the Kubernetes API server is persistent, +it may require Kubernetes API server administrator intervention to address, and +you may want to use the `busyretryinterval` value of `"0"`. This will cause +the module to return a "hard" error (see below). + +For other errors, the plugin will assume they are "hard" errors requiring admin +intervention, return an error code, and rsyslog will suspend the plugin. Use +the :ref:`mmkubernetes-statistic-counter` to monitor for problems getting data +from the Kubernetes API service. + +Example +------- + +Assuming you have an `imfile` input reading from docker json-file container +logs managed by Kubernetes, with `addmetadata="on"` so that mmkubernetes can +get the basic necessary Kubernetes metadata from the filename: + +.. code-block:: none + + input(type="imfile" file="/var/log/containers/*.log" + tag="kubernetes" addmetadata="on") + +(Add `reopenOnTruncate="on"` if using Docker, not required by CRI-O). + +and/or an `imjournal` input for docker journald container logs annotated by +Kubernetes: + +.. code-block:: none + + input(type="imjournal") + +Then mmkubernetes can be used to annotate log records like this: + +.. code-block:: none + + module(load="mmkubernetes") + + action(type="mmkubernetes") + +After this, you should have log records with fields described in the `Fields` +section above. + +Credits +------- + +This work is based on +https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter +and has many of the same features. diff --git a/source/configuration/modules/mmnormalize.rst b/source/configuration/modules/mmnormalize.rst new file mode 100644 index 0000000..7d526a5 --- /dev/null +++ b/source/configuration/modules/mmnormalize.rst @@ -0,0 +1,178 @@ +Log Message Normalization Module (mmnormalize) +============================================== + +**Module Name: mmnormalize** + +**Available since:** 6.1.2+ + +**Author:** Rainer Gerhards <rgerhards@adiscon.com> + +**Description**: + +This module provides the capability to normalize log messages via +`liblognorm <http://www.liblognorm.com>`_. Thanks to liblognorm, +unstructured text, like usually found in log messages, can very quickly +be parsed and put into a normal form. This is done so quickly, that it +should be possible to normalize events in realtime. + +This module is implemented via the output module interface. This means +that mmnormalize should be called just like an action. After it has been +called, the normalized message properties are available and can be +accessed. These properties are called the "CEE/lumberjack" properties, +because liblognorm creates a format that is inspired by the +CEE/lumberjack approach. + +**Please note:** CEE/lumberjack properties are different from regular +properties. They have always "$!" prepended to the property name given +in the rulebase. Such a property needs to be called with +**%$!propertyname%**. + +Note that from a performance point of view mmnormalize should only be called +once on each message, if possible. To do so, place all rules into a single +rule base. If that is not possible, you can safely call mmnormalize multiple +times. This incurs a small performance drawback. + +Module Parameters +~~~~~~~~~~~~~~~~~ + +Note: parameter names are case-insensitive. + +.. function:: allow_regex <boolean> + + **Default**: off + + Specifies if regex field-type should be allowed. Regex field-type has + significantly higher computational overhead compared to other fields, + so it should be avoided when another field-type can achieve the desired + effect. Needs to be "on" for regex field-type to work. + +Action Parameters +~~~~~~~~~~~~~~~~~ + +Note: parameter names are case-insensitive. + +.. function:: ruleBase <word> + + Specifies which rulebase file is to use. If there are multiple + mmnormalize instances, each one can use a different file. However, a + single instance can use only a single file. This parameter or **rule** MUST be + given, because normalization can only happen based on a rulebase. It + is recommended that an absolute path name is given. Information on + how to create the rulebase can be found in the `liblognorm + manual <http://www.liblognorm.com/files/manual/index.html>`_. + +.. function:: rule <array> + + *(Available since: 8.26.0)* + + Contains an array of strings which will be put together as the rulebase. This parameter + or **rulebase** MUST be given, because normalization can only happen based on a rulebase. + +.. function:: useRawMsg <boolean> + + **Default**: off + + Specifies if the raw message should be used for normalization (on) + or just the MSG part of the message (off). + +.. function:: path <word> + + **Default**: $! + + Specifies the JSON path under which parsed elements should be + placed. By default, all parsed properties are merged into root of + message properties. You can place them under a subtree, instead. You + can place them in local variables, also, by setting path="$.". + +.. function:: variable <word> + + *(Available since: 8.5.1)* + + Specifies if a variable insteed of property 'msg' should be used for + normalization. A variable can be property, local variable, json-path etc. + Please note that **useRawMsg** overrides this parameter, so if **useRawMsg** + is set, **variable** will be ignored and raw message will be used. + + + + +Legacy Configuration Parameters +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Note: parameter names are case-insensitive. + +- $mmnormalizeRuleBase <rulebase-file> - equivalent to the "ruleBase" + parameter. +- $mmnormalizeUseRawMsg <on/off> - equivalent to the "useRawMsg" + parameter. + +See Also +~~~~~~~~ + +- `First steps for + mmnormalize <http://www.rsyslog.com/normalizer-first-steps-for-mmnormalize/>`_ +- `Log normalization and special + characters <http://www.rsyslog.com/log-normalization-and-special-characters/>`_ +- `Log normalization and the leading + space <http://www.rsyslog.com/log-normalization-and-the-leading-space/>`_ +- `Using mmnormalize effectively with Adiscon + LogAnalyzer <http://www.rsyslog.com/using-rsyslog-mmnormalize-module-effectively-with-adiscon-loganalyzer/>`_ + +Caveats/Known Bugs +~~~~~~~~~~~~~~~~~~ + +None known at this time. + +Example +~~~~~~~ + +**Sample 1:** + +In this sample messages are received via imtcp. Then they are normalized with the given rulebase. +After that they are written in a file. + +:: + + module(load="mmnormalize") + module(load="imtcp") + + input(type="imtcp" port="10514" ruleset="outp") + + ruleset(name="outp") { + action(type="mmnormalize" rulebase="/tmp/rules.rulebase") + action(type="omfile" File="/tmp/output") + } + +**Sample 2:** + +In this sample messages are received via imtcp. Then they are normalized based on the given rules. +The strings from **rule** are put together and are equal to a rulebase with the same content. + +:: + + module(load="mmnormalize") + module(load="imtcp") + + input(type="imtcp" port="10514" ruleset="outp") + + ruleset(name="outp") { + action(type="mmnormalize" rule=["rule=:%host:word% %tag:char-to:\\x3a%: no longer listening on %ip:ipv4%#%port:number%", "rule=:%host:word% %ip:ipv4% user was logged out"]) + action(type="omfile" File="/tmp/output") + } + +**Sample 3:** + +This activates the module and applies normalization to all messages: + +:: + + module(load="mmnormalize") + action(type="mmnormalize" ruleBase="/path/to/rulebase.rb") + +The same in legacy format: + +:: + + $ModLoad mmnormalize + $mmnormalizeRuleBase /path/to/rulebase.rb + *.* :mmnormalize: diff --git a/source/configuration/modules/mmpstrucdata.rst b/source/configuration/modules/mmpstrucdata.rst new file mode 100644 index 0000000..481f2e0 --- /dev/null +++ b/source/configuration/modules/mmpstrucdata.rst @@ -0,0 +1,101 @@ +RFC5424 structured data parsing module (mmpstrucdata) +===================================================== + +**Module Name:** mmpstrucdata + +**Author:** Rainer Gerhards <rgerhards@adiscon.com> + +**Available since**: 7.5.4 + +**Description**: + +The mmpstrucdata parses the structured data of `RFC5424 <https://tools.ietf.org/html/rfc5424>`_ into the message json variable tree. The data parsed, if available, is stored under "jsonRoot!rfc5424-sd!...". Please note that only RFC5424 messages will be processed. + +The difference of RFC5424 is in the message layout: the SYSLOG-MSG part only contains the structured-data part instead of the normal message part. Further down you can find a example of a structured-data part. + +**Module Configuration Parameters**: + +Note: parameter names are case-insensitive. + +Currently none. + + + +**Action Confguration Parameters**: + +Note: parameter names are case-insensitive. + +- **jsonRoot** - default "!" + Specifies into which json container the data shall be parsed to. + +- **sd_name.lowercase** - default "on" + + Available: rsyslog 8.32.0 and above + + Specifies if sd names (SDID) shall be lowercased. If set to "on", this + is the case, if "off" than not. The default of "on" is used because that + was the traditional mode of operations. It it generally advised to + change the parameter to "off" if not otherwise required. + +**See Also** + +- `Howto anonymize messages that go to specific + files <http://www.rsyslog.com/howto-anonymize-messages-that-go-to-specific-files/>`_ + +**Caveats/Known Bugs:** + +- this module is currently experimental; feedback is appreciated +- property names are treated case-insensitive in rsyslog. As such, + RFC5424 names are treated case-insensitive as well. If such names + only differ in case (what is not recommended anyways), problems will + occur. +- structured data with duplicate SD-IDs and SD-PARAMS is not properly + processed + +**Samples:** + +Below you can find a structured data part of a random message which has three parameters. + +:: + + [exampleSDID@32473 iut="3" eventSource="Application"eventID="1011"] + + +In this snippet, we parse the message and emit all json variable to a +file with the message anonymized. Note that once mmpstrucdata has run, +access to the original message is no longer possible (execept if stored +in user variables before anonymization). + +:: + + module(load="mmpstrucdata") action(type="mmpstrucdata") + template(name="jsondump" type="string" string="%msg%: %$!%\\n") + action(type="omfile" file="/path/to/log" template="jsondump") + + +**A more practical one:** + +Take this example message (inspired by RFC5424 sample;)): + +``<34>1 2003-10-11T22:14:15.003Z mymachine.example.com su - ID47 [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011"][id@2 test="tast"] BOM'su root' failed for lonvick on /dev/pts/8`` + +We apply this configuration: + +:: + + module(load="mmpstrucdata") action(type="mmpstrucdata") + template(name="sample2" type="string" string="ALL: %$!%\\nSD: + %$!RFC5424-SD%\\nIUT:%$!rfc5424-sd!exampleSDID@32473!iut%\\nRAWMSG: + %rawmsg%\\n\\n") action(type="omfile" file="/path/to/log" + template="sample2") + + + +This will output: + +``ALL: { "rfc5424-sd": { "examplesdid@32473": { "iut": "3", "eventsource": "Application", "eventid": "1011" }, "id@2": { "test": "tast" } } } SD: { "examplesdid@32473": { "iut": "3", "eventsource": "Application", "eventid": "1011" }, "id@2": { "test": "tast" } } IUT:3 RAWMSG: <34>1 2003-10-11T22:14:15.003Z mymachine.example.com su - ID47 [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011"][id@2 test="tast"] BOM'su root' failed for lonvick on /dev/pts/8`` + +As you can seem, you can address each of the individual items. Note that +the case of the RFC5424 parameter names has been converted to lower +case. + diff --git a/source/configuration/modules/mmrfc5424addhmac.rst b/source/configuration/modules/mmrfc5424addhmac.rst new file mode 100644 index 0000000..d3e5333 --- /dev/null +++ b/source/configuration/modules/mmrfc5424addhmac.rst @@ -0,0 +1,93 @@ +mmrfc5424addhmac +================ + +**Module Name: mmrfc5424addhmac** + +**Author:**\ Rainer Gerhards <rgerhards@adiscon.com> + +**Available since**: 7.5.6 + +**Description**: + +This module adds a hmac to RFC5424 structured data if not already +present. This is a custom module and uses openssl as requested by the +sponsor. This works exclusively for RFC5424 formatted messages; all +others are ignored. + +If both `mmpstrucdata <mmpstrucdata.html>`_ and mmrfc5424addhmac are to +be used, the recommended calling sequence is + +#. mmrfc5424addhmac +#. mmpstrucdata + +with that sequence, the generated hash will become available for +mmpstrucdata. + + + +**Module Configuration Parameters**: + +Note: parameter names are case-insensitive. + +Currently none. + + + +**Action Confguration Parameters**: + +Note: parameter names are case-insensitive. + +- **key** + The "key" (string) to be used to generate the hmac. +- **hashfunction** + An openssl hash function name for the function to be used. This is + passed on to openssl, so see the openssl list of supported function + names. +- **sd\_id** + The RFC5424 structured data ID to be used by this module. This is + the SD-ID that will be added. Note that nothing is added if this + SD-ID is already present. + +**Verification method** + +rsyslog does not contain any tools to verify a log file (this was not +part of the custom project). So you need to write your own verifier. + +When writing the verifier, keep in mind that the log file contains +messages with the hash SD-ID included. For obvious reasons, this SD-ID +was not present when the hash was created. So before the actual +verification is done, this SD-ID must be removed, and the remaining +(original) message be verified. Also, it is important to note that the +output template must write the exact same message format that was +received. Otherwise, a verification failure will obviously occur - and +must so, because the message content actually was altered. + +So in a more formal description, verification of a message m can be done +as follows: + +#. let m' be m with the configured SD-ID removed (everything between + []). Otherwise, m' must be an exact duplicate of m. +#. call openssl's HMAC function as follows: + ``HMAC(hashfunction, key, len(key), m', len(m'), hash, &hashlen);`` + Where hashfunction and key are the configured values and hash is an + output buffer for the hash. +#. let h be the extracted hash value obtained from m within the relevant + SD-ID. Be sure to convert the hex string back to the actual byte + values. +#. now compare hash and h under consideration of the sizes. If these + values match the verification succeeds, otherwise the message was + modified. + +If you need help implementing a verifier function or want to sponsor +development of a verification tool, please simply email +`sales@adiscon.com <sales@adiscon.com>`_ for a quote. + +**See Also** + +- `How to add a HMAC to RFC5424 + messages <http://www.rsyslog.com/how-to-add-a-hmac-to-rfc5424-structured-data-messages/>`_ + +**Caveats/Known Bugs:** + +- none + diff --git a/source/configuration/modules/mmrm1stspace.rst b/source/configuration/modules/mmrm1stspace.rst new file mode 100644 index 0000000..915f026 --- /dev/null +++ b/source/configuration/modules/mmrm1stspace.rst @@ -0,0 +1,30 @@ +mmrm1stspace: First Space Modification Module +============================================= + +**Author:** Pascal Withopf <pascalwithopf1@gmail.com> + +In rfc3164 the msg begins at the first letter after the tag. It is often the +case that this is a unnecessary space. This module removes this first character +if it is a space. + +Configuration Parameters +------------------------ + +Note: parameter names are case-insensitive. + +Currently none. + +Examples +-------- + +This example receives messages over imtcp and modifies them, before sending +them to a file. + +:: + + module(load="imtcp") + module(load="mmrm1stspace") + input(type="imtcp" port="13514") + action(type="mmrm1stspace") + action(type="omfile" file="output.log") + diff --git a/source/configuration/modules/mmsequence.rst b/source/configuration/modules/mmsequence.rst new file mode 100644 index 0000000..a35599f --- /dev/null +++ b/source/configuration/modules/mmsequence.rst @@ -0,0 +1,156 @@ +Number generator and counter module (mmsequence) +================================================ + +**Module Name: mmsequence** + +**Author:**\ Pavel Levshin <pavel@levshin.spb.ru> + +**Status:**\ Non project-supported module - contact author or rsyslog +mailing list for questions + +**This module is deprecated** in v8 and solely provided for backward +compatibility reasons. It was written as a work-around for missing +global variable support in v7. Global variables are available in v8, +and at some point in time this module will entirely be removed. + +**Do not use this module for newly crafted config files.** +Use global variables instead. + + +**Available since**: 7.5.6 + +**Description**: + +This module generates numeric sequences of different kinds. It can be +used to count messages up to a limit and to number them. It can generate +random numbers in a given range. + +This module is implemented via the output module interface, so it is +called just as an action. The number generated is stored in a variable. + + + +**Action Parameters**: + +Note: parameter names are case-insensitive. + +- **mode** "random" or "instance" or "key" + + Specifies mode of the action. In "random" mode, the module generates + uniformly distributed integer numbers in a range defined by "from" + and "to". + + In "instance" mode, which is default, the action produces a counter + in range [from, to). This counter is specific to this action + instance. + + In "key" mode, the counter can be shared between multiple instances. + This counter is identified by a name, which is defined with "key" + parameter. + +- **from** [non-negative integer], default "0" + + Starting value for counters and lower margin for random generator. + +- **to** [positive integer], default "INT\_MAX" + + Upper margin for all sequences. Note that this margin is not + inclusive. When next value for a counter is equal or greater than + this parameter, the counter resets to the starting value. + +- **step** [non-negative integer], default "1" + + Increment for counters. If step is "0", it can be used to fetch + current value without modification. The latter not applies to + "random" mode. This is useful in "key" mode or to get constant values + in "instance" mode. + +- **key** [word], default "" + + Name of the global counter which is used in this action. + +- **var** [word], default "$!mmsequence" + + Name of the variable where the number will be stored. Should start + with "$". + +**Sample**: + +:: + + # load balance + Ruleset( + name="logd" + queue.workerthreads="5" + ){ + + Action( + type="mmsequence" + mode="instance" + from="0" + to="2" + var="$.seq" + ) + + if $.seq == "0" then { + Action( + type="mmnormalize" + userawmsg="on" + rulebase="/etc/rsyslog.d/rules.rb" + ) + } else { + Action( + type="mmnormalize" + userawmsg="on" + rulebase="/etc/rsyslog.d/rules.rb" + ) + } + + # output logic here + } + # generate random numbers + action( + type="mmsequence" + mode="random" + to="100" + var="$!rndz" + ) + # count from 0 to 99 + action( + type="mmsequence" + mode="instance" + to="100" + var="$!cnt1" + ) + # the same as before but the counter is global + action( + type="mmsequence" + mode="key" + key="key1" + to="100" + var="$!cnt2" + ) + # count specific messages but place the counter in every message + if $msg contains "txt" then + action( + type="mmsequence" + mode="key" + to="100" + var="$!cnt3" + ) + else + action( + type="mmsequence" + mode="key" + to="100" + step="0" + var="$!cnt3" + key="" + ) + +**Legacy Configuration Parameters**: + +Note: parameter names are case-insensitive. + +Not supported. + diff --git a/source/configuration/modules/mmsnmptrapd.rst b/source/configuration/modules/mmsnmptrapd.rst new file mode 100644 index 0000000..0f75d8c --- /dev/null +++ b/source/configuration/modules/mmsnmptrapd.rst @@ -0,0 +1,103 @@ +mmsnmptrapd message modification module +======================================= + +**Module Name:** mmsnmptrapd + +**Author:** Rainer Gerhards <rgerhards@adiscon.com> (custom-created) + +**Multi-Ruleset Support:** since 5.8.1 + +**Description**: + +This module uses a specific configuration of snmptrapd's tag values to +obtain information of the original source system and the severity +present inside the original SNMP trap. It then replaces these fields +inside the syslog message. + +Let's look at an example. Essentially, SNMPTT will invoke something like +this: + +:: + + logger -t snmptrapd/warning/realhost Host 003c.abcd.ffff in vlan 17 is flapping between port Gi4/1 and port Gi3/2 + +This message modification module will change the tag (removing the +additional information), hostname and severity (not shown in example), +so the log entry will look as follows: + +:: + + 2011-04-21T16:43:09.101633+02:00 realhost snmptrapd: Host 003c.abcd.ffff in vlan 122 is flapping between port Gi4/1 and port Gi3/2 + +The following logic is applied to all message being processed: + +#. The module checks incoming syslog entries. If their TAG field starts + with "snmptrapd/" (configurable), they are modified, otherwise not. + If the are modified, this happens as follows: +#. It will derive the hostname from the tag field which has format + snmptrapd/severity/hostname +#. It should derive the severity from the tag field which has format + snmptrapd/severity/hostname. A configurable mapping table will be + used to drive a new severity value from that severity string. If no + mapping has been defined, the original severity is not changed. +#. It replaces the "FromHost" value with the derived value from step 2 +#. It replaces the "Severity" value with the derived value from step 3 + +Note that the placement of this module inside the configuration is +important. All actions before this modules is called will work on the +unmodified message. All messages after it's call will work on the +modified message. Please also note that there is some extra power in +case it is required: as this module is implemented via the output module +interface, a filter can be used (actually must be used) in order to tell +when it is called. Usually, the catch-all filter (\*.\*) is used, but +more specific filters are fully supported. So it is possible to define +different parameters for this module depending on different filters. It +is also possible to just run messages from one remote system through +this module, with the help of filters or multiple rulesets and ruleset +bindings. In short words, all capabilities rsyslog offers to control +output modules are also available to mmsnmptrapd. + +**Configuration Parameters**: + +Note: parameter names are case-insensitive. + +- **$mmsnmptrapdTag** [tagname] + + Tells the module which start string inside the tag to look for. The + default is "snmptrapd". Note that a slash is automatically added to + this tag when it comes to matching incoming messages. It MUST not be + given, except if two slashes are required for whatever reasons (so + "tag/" results in a check for "tag//" at the start of the tag field). + +- **$mmsnmptrapdSeverityMapping** [severitymap] + This specifies the severity mapping table. It needs to be specified + as a list. Note that due to the current config system **no + whitespace** is supported inside the list, so be sure not to use any + whitespace inside it. + The list is constructed of Severity-Name/Severity-Value pairs, + delimited by comma. Severity-Name is a case-sensitive string, e.g. + "warning" and an associated numerical value (e.g. 4). Possible values + are in the rage 0..7 and are defined in RFC5424, table 2. The given + sample would be specified as "warning/4". + If multiple instances of mmsnmptrapd are used, each instance uses + the most recently defined $mmsnmptrapdSeverityMapping before itself. + +**Caveats/Known Bugs:** + +- currently none known + +**Example:** + +This enables to rewrite messages from snmptrapd and configures error and +warning severities. The default tag is used. + +:: + + $ModLoad mmsnmptrapd # needs to be done just once + # ... other module loads and listener setup ... + *.* /path/to/file/with/originalMessage # this file receives unmodified messages + $mmsnmptrapdSeverityMapping warning/4,error/3 + *.* :mmsnmptrapd: # now message is modified + *.* /path/to/file/with/modifiedMessage # this file receives modified messages + # ... rest of config ... + diff --git a/source/configuration/modules/mmtaghostname.rst b/source/configuration/modules/mmtaghostname.rst new file mode 100644 index 0000000..114f723 --- /dev/null +++ b/source/configuration/modules/mmtaghostname.rst @@ -0,0 +1,89 @@ +****************************************** +mmtaghostname: message modification module +****************************************** + +================ ============================================================== +**Module Name:** **mmtaghostname** +**Authors:** Jean-Philippe Hilaire <jean-philippe.hilaire@pmu.fr> & Philippe Duveau <philippe.duveau@free.fr> +================ ============================================================== + + +Purpose +======= + +As a message modification, it can be used in a different step of the +message processing without interfering in the parsers' chain process +and can be applied before or after parsing process using rulesets. + +The purposes are : + +- to add a tag on message produce by input module which does not provide + a tag like imudp or imtcp. Useful when the tag is used for routing the + message. + +- to force message hostname to the rsyslog valeur. + AWS Use case : applications in auto-scaling systems provides logs to rsyslog + through udp/tcp. As a result of auto-scaling, the name of the host is based + on an ephemeral IPs (short term meaning). In this situation rsyslog local + hostname is generally closed to business rule. So replacing hostanme received + by the rsyslog local Hostname provide values to the logs collected. + +Compile +======= + +To successfully compile mmtaghostname module. + +.. code-block:: none + + ./configure --enable-mmtaghostname ... + +Configuration Parameters +======================== + +Tag +^^^ + +.. csv-table:: + :header: "type", "mandatory", "format", "default" + :widths: auto + :class: parameter-table + + "string", "no", ,"none" + +The tag to be assigned to messages modified. If you would like to see the +colon after the tag, you need to include it when you assign a tag value, +like so: ``tag="myTagValue:"``. + +If this attribute is no provided, messages tags are not modified. + +ForceLocalHostname +^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "mandatory", "format", "default" + :widths: auto + :class: parameter-table + + "Binary", "no", ,"off" + +This attribute force to set the HOSTNAME of the message to the rsyslog +value "localHostName". This allow to set a valid value to message received +received from local application through imudp or imtcp. + +Sample +====== + +In this sample, the message received is parsed by RFC5424 parser and then +the HOSTNAME is overwritten and a tag is setted. + +.. code-block:: none + + module(load='mmtaghostname') + module(load='imudp') + global(localhostname="sales-front") + + ruleset(name="TagUDP" parser=[ "rsyslog.rfc5424" ]) { + action(type="mmtaghostname" tag="front" forcelocalhostname="on") + call ... + } + input(type="imudp" port="514" ruleset="TagUDP") diff --git a/source/configuration/modules/mmutf8fix.rst b/source/configuration/modules/mmutf8fix.rst new file mode 100644 index 0000000..594bb90 --- /dev/null +++ b/source/configuration/modules/mmutf8fix.rst @@ -0,0 +1,112 @@ +Fix invalid UTF-8 Sequences (mmutf8fix) +======================================= + +**Module Name:** mmutf8fix + +**Author:** Rainer Gerhards <rgerhards@adiscon.com> + +**Available since**: 7.5.4 + +**Description**: + +The mmutf8fix module permits to fix invalid UTF-8 sequences. Most often, +such invalid sequences result from syslog sources sending in non-UTF +character sets, e.g. ISO 8859. As syslog does not have a way to convey +the character set information, these sequences are not properly handled. +While they are typically uncritical with plain text files, they can +cause big headache with database sources as well as systems like +ElasticSearch. + +The module supports different "fixing" modes and fixes. The current +implementation will always replace invalid bytes with a single US ASCII +character. Additional replacement modes will probably be added in the +future, depending on user demand. In the longer term it could also be +evolved into an any-charset-to-UTF8 converter. But first let's see if it +really gets into widespread enough use. + +**Proper Usage**: + +Some notes are due for proper use of this module. This is a message +modification module utilizing the action interface, which means you call +it like an action. This gives great flexibility on the question on when +and how to call this module. Note that once it has been called, it +actually modifies the message. The original message is then no longer +available. However, this does **not** change any properties set, used or +extracted before the modification is done. + +One potential use case is to normalize all messages. This is done by +simply calling mmutf8fix right in front of all other actions. + +If only a specific source (or set of sources) is known to cause +problems, mmutf8fix can be conditionally called only on messages from +them. This also offers performance benefits. If such multiple sources +exists, it probably is a good idea to define different listeners for +their incoming traffic, bind them to specific +`ruleset <multi_ruleset.html>`_ and call mmutf8fix as first action in +this ruleset. + +**Module Configuration Parameters**: + +Note: parameter names are case-insensitive. + +Currently none. + + + +**Action Confguration Parameters**: + +Note: parameter names are case-insensitive. + +- **mode** - **utf-8**/controlcharacters + + This sets the basic detection mode. + In **utf-8** mode (the default), proper UTF-8 encoding is checked and + bytes which are not proper UTF-8 sequences are acted on. If a proper + multi-byte start sequence byte is detected but any of the following + bytes is invalid, the whole sequence is replaced by the replacement + method. This mode is most useful with non-US-ASCII character sets, + which validly includes multibyte sequences. Note that in this mode + control characters are NOT being replaced, because they are valid + UTF-8. + In **controlcharacters** mode, all bytes which do not represent a + printable US-ASCII character (codes 32 to 126) are replaced. Note + that this also mangles valid UTF-8 multi-byte sequences, as these are + (deliberately) outside of that character range. This mode is most + useful if it is known that no characters outside of the US-ASCII + alphabet need to be processed. +- **replacementChar** - default " " (space), a single character + + This is the character that invalid sequences are replaced by. + Currently, it MUST be a **printable** US-ASCII character. + +**Caveats/Known Bugs:** + +- overlong UTF-8 encodings are currently not detected in utf-8 mode. + +**Samples:** + +In this snippet, we write one file without fixing UTF-8 and another one +with the message fixed. Note that once mmutf8fix has run, access to the +original message is no longer possible. + +:: + + module(load="mmutf8fix") action(type="omfile" + file="/path/to/non-fixed.log") action(type="mmutf8fix") + action(type="omfile" file="/path/to/fixed.log") + +In this sample, we fix only message originating from host 10.0.0.1. + +:: + + module(load="mmutf8fix") if $fromhost-ip == "10.0.0.1" then + action(type="mmutf8fix") # all other actions here... + +This is mostly the same as the previous sample, but uses +"controlcharacters" processing mode. + +:: + + module(load="mmutf8fix") if $fromhost-ip == "10.0.0.1" then + action(type="mmutf8fix" mode="controlcharacters") # all other actions here... + diff --git a/source/configuration/modules/module_workflow.png b/source/configuration/modules/module_workflow.png Binary files differnew file mode 100644 index 0000000..e1a72e9 --- /dev/null +++ b/source/configuration/modules/module_workflow.png diff --git a/source/configuration/modules/omamqp1.rst b/source/configuration/modules/omamqp1.rst new file mode 100644 index 0000000..c5e2e07 --- /dev/null +++ b/source/configuration/modules/omamqp1.rst @@ -0,0 +1,476 @@ +***************************************** +omamqp1: AMQP 1.0 Messaging Output Module +***************************************** + +=========================== =========================================================================== +**Module Name:** **omamqp1** +**Available Since:** **8.17.0** +**Original Author:** Ken Giusti <kgiusti@gmail.com> +=========================== =========================================================================== + + +Purpose +======= + +This module provides the ability to send logging via an AMQP 1.0 +compliant message bus. It puts the log messages into an AMQP +message and sends the message to a destination on the bus. + + +Notable Features +================ + +- :ref:`omamqp1-message-format` +- :ref:`omamqp1-interoperability` + + +Configuration Parameters +======================== + +.. note:: + + Parameter names are case-insensitive. + + +Action Parameters +----------------- + +Host +^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "5672", "yes", "none" + +The address of the message bus in *host[:port]* format. +The port defaults to 5672 if absent. Examples: *"localhost"*, +*"127.0.0.1:9999"*, *"bus.someplace.org"* + + +Target +^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "yes", "none" + +The destination for the generated messages. This can be +the name of a queue or topic. On some messages buses it may be +necessary to create this target manually. Example: *"amq.topic"* + + +Username +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +Used by SASL to authenticate with the message bus. + + +Password +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +Used by SASL to authenticate with the message bus. + + +Template +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "RSYSLOG_FileFormat", "no", "none" + +Format for the log messages. + + +idleTimeout +^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "0", "no", "none" + +The idle timeout in seconds. This enables connection +heartbeats and is used to detect a failed connection to the message +bus. Set to zero to disable. + + +reconnectDelay +^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "5", "no", "none" + +The time in seconds this module will delay before +attempting to re-established a failed connection to the message bus. + + +MaxRetries +^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "10", "no", "none" + +The number of times an undeliverable message is +re-sent to the message bus before it is dropped. This is unrelated +to rsyslog's action.resumeRetryCount. Once the connection to the +message bus is active this module is ready to receive log messages +from rsyslog (i.e. the module has 'resumed'). Even though the +connection is active, any particular message may be rejected by the +message bus (e.g. 'unrouteable'). The module will retry +(e.g. 'suspend') for up to *maxRetries* attempts before discarding +the message as undeliverable. Setting this to zero disables the +limit and unrouteable messages will be retried as long as the +connection stays up. You probably do not want that to +happen. + + +DisableSASL +^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "0", "no", "none" + +Setting this to a non-zero value will disable SASL +negotiation. Only necessary if the message bus does not offer SASL +support. + + +Dependencies +============ + +* `libqpid-proton <http://qpid.apache.org/proton>`_ + +Configure +========= + +.. code-block:: none + + ./configure --enable-omamqp1 + + +.. _omamqp1-message-format: + +Message Format +============== + +Messages sent from this module to the message bus contain an AMQP List +in the message body. This list contains one or more log messages as +AMQP String types. Each string entry is a single log message. The +list is ordered such that the oldest log appears at the front of the +list (e.g. list index 0), whilst the most recent log is at the end of +the list. + + +.. _omamqp1-interoperability: + +Interoperability +================ + +The output plugin has been tested against the following messaging systems: + +* `QPID C++ Message Broker <http://qpid.apache.org/components/cpp-broker>`_ +* `QPID Dispatch Message Router <http://qpid.apache.org/components/dispatch-router>`_ + + +TODO +==== + +- Add support for SSL connections. + + +Examples +======== + +Example 1 +--------- + +This example shows a minimal configuration. The module will attempt +to connect to a QPID broker at *broker.amqp.org*. Messages are +sent to the *amq.topic* topic, which exists on the broker by default: + +.. code-block:: none + + module(load="omamqp1") + action(type="omamqp1" + host="broker.amqp.org" + target="amq.topic") + + +Example 2 +--------- + +This example forces rsyslogd to authenticate with the message bus. +The message bus must be provisioned such that user *joe* is allowed to +send to the message bus. All messages are sent to *log-queue*. It is +assumed that *log-queue* has already been provisioned: + +.. code-block:: none + + module(load="omamqp1") + + action(type="omamqp1" + host="bus.amqp.org" + target="log-queue" + username="joe" + password="trustno1") + + +Notes on use with the QPID C++ broker (qpidd) +============================================= + +*Note well*: These notes assume use of version 0.34 of the QPID C++ +broker. Previous versions may not be fully compatible. + +To use the Apache QPID C++ broker **qpidd** as the message bus, a +version of qpidd that supports the AMQP 1.0 protocol must be used. + +Since qpidd can be packaged without AMQP 1.0 support you should verify +AMQP 1.0 has been enabled by checking for AMQP 1.0 related options in +the qpidd help text. For example: + +.. code-block:: none + + qpidd --help + + ... + + AMQP 1.0 Options: + --domain DOMAIN Domain of this broker + --queue-patterns PATTERN Pattern for on-demand queues + --topic-patterns PATTERN Pattern for on-demand topics + + +If no AMQP 1.0 related options appear in the help output then your +instance of qpidd does not support AMQP 1.0 and cannot be used with +this output module. + +The destination for message (target) *must* be created before log +messages arrive. This can be done using the qpid-config tool. + +Example: + +.. code-block:: none + + qpid-config add queue rsyslogd + + +Alternatively the target can be created on demand by configuring a +queue-pattern (or topic-pattern) that matches the target. To do this, +add a *queue-patterns* or *topic_patterns* configuration directive to +the qpidd configuration file /etc/qpid/qpidd.conf. + +For example to have qpidd automatically create a queue named +*rsyslogd* add the following to the qpidd configuration file: + +.. code-block:: none + + queue-patterns=rsyslogd + + +or, if a topic behavior is desired instead of a queue: + +.. code-block:: none + + topic-patterns=rsyslogd + + +These dynamic targets are auto-delete and will be destroyed once there +are no longer any subscribers or queue-bound messages. + +Versions of qpidd <= 0.34 also need to have the SASL service name set +to *"amqp"* if SASL authentication is used. Add this to the qpidd.conf +file: + +.. code-block:: none + + sasl-service-name=amqp + + +Notes on use with the QPID Dispatch Router (qdrouterd) +====================================================== + +*Note well*: These notes assume use of version 0.5 of the QPID Dispatch +Router **qdrouterd**. Previous versions may not be fully compatible. + +The default qdrouterd configuration does not have SASL authentication +turned on. If SASL authentication is required you must configure SASL +in the qdrouter configuration file /etc/qpid-dispatch/qdrouterd.conf + +First create a SASL configuration file for qdrouterd. This +configuration file is usually /etc/sasl2/qdrouterd.conf, but its +default location may vary depending on your platform's configuration. + +This document assumes you understand how to properly configure Cyrus +SASL. + +Here is an example qdrouterd SASL configuration file that allows the +client to use either the **DIGEST-MD5** or **PLAIN** authentication +mechanisms and specifies the path to the SASL user credentials +database: + +.. code-block:: none + + pwcheck_method: auxprop + auxprop_plugin: sasldb + sasldb_path: /var/lib/qdrouterd/qdrouterd.sasldb + mech_list: DIGEST-MD5 PLAIN + + +Once a SASL configuration file has been set up for qdrouterd the path +to the directory holding the configuration file and the name of the +configuration file itself **without the '.conf' suffix** must be added +to the /etc/qpid-dispatch/qdrouterd.conf configuration file. This is +done by adding *saslConfigPath* and *saslConfigName* to the +*container* section of the configuration file. For example, assuming +the file /etc/sasl2/qdrouterd.conf holds the qdrouterd SASL +configuration: + +.. code-block:: none + + container { + workerThreads: 4 + containerName: Qpid.Dispatch.Router.A + saslConfigPath: /etc/sasl2 + saslConfigName: qdrouterd + } + + +In addition the address used by the omamqp1 module to connect to +qdrouterd must have SASL authentication turned on. This is done by +adding the *authenticatePeer* attribute set to 'yes' to the +corresponding *listener* entry: + +.. code-block:: none + + listener { + addr: 0.0.0.0 + port: amqp + authenticatePeer: yes + } + + +This should complete the SASL setup needed by qdrouterd. + +The target address used as the destination for the log messages must +be picked with care. qdrouterd uses the prefix of the target address +to determine the forwarding pattern used for messages sent to that +target address. Addresses starting with the prefix *queue* are +distributed to only one message receiver. If there are multiple +message consumers listening to that target address only one listener +will receive the message - mimicking the behavior of a queue with +competing subscribers. For example: *queue/rsyslogd* + +If a multicast pattern is desired - where all active listeners receive +their own copy of the message - the target address prefix *multicast* +may be used. For example: *multicast/rsyslogd* + +Note well: if there are no active receivers for the log messages the +messages will be rejected by qdrouterd since the messages are +undeliverable. In this case the omamqp1 module will return a +**SUSPENDED** status to the rsyslogd main task. rsyslogd may then +re-submit the rejected log messages to the module which will attempt +to send them again. This retry option is configured via rsyslogd - it +is not part of this module. Refer to the rsyslogd actions +documentation. + + +Using qdrouterd in combination with qpidd +========================================= + +A qdrouterd-based message bus can use a broker as a message storage +mechanism for those that require broker-based message services (such +as a message store). This section explains how to configure qdrouterd +and qpidd for this type of deployment. Please read the above notes +for deploying qpidd and qdrouterd first. + +Each qdrouterd instance that is to connect the broker to the message +bus must define a *connector* section in the qdrouterd.conf file. +This connector contains the addressing information necessary to have +the message bus set up a connection to the broker. For example, if a +broker is available on host broker.host.com at port 5672: + +.. code-block:: none + + connector { + name: mybroker + role: on-demand + addr: broker.host.com + port: 5672 + } + + +In order to route messages to and from the broker, a static *link +route* must be configured on qdrouterd. This link route contains a +target address prefix and the name of the connector to use for +forwarding matching messages. + +For example, to have qdrouterd forward messages that have a target +address prefixed by "Broker" to the connector defined above, the +following link pattern must be added to the qdrouterd.conf +configuration: + +.. code-block:: none + + linkRoutePattern { + prefix: /Broker/ + connector: mybroker + } + + +A queue must then be created on the broker. The name of the queue +must be prefixed by the same prefix specified in the linkRoutePattern +entry. For example: + +.. code-block:: none + + $ qpid-config add queue Broker/rsyslogd + + +Lastly use the name of the queue for the target address for the omamqp +module action. For example, assuming qdrouterd is listening on local +port 5672: + +.. code-block:: none + + action(type="omamqp1" + host="localhost:5672" + target="Broker/rsyslogd") + + diff --git a/source/configuration/modules/omazureeventhubs.rst b/source/configuration/modules/omazureeventhubs.rst new file mode 100644 index 0000000..2c27660 --- /dev/null +++ b/source/configuration/modules/omazureeventhubs.rst @@ -0,0 +1,412 @@ +********************************************************** +omazureeventhubs: Microsoft Azure Event Hubs Output Module +********************************************************** + +=========================== =========================================================================== +**Module Name:** **omazureeventhubs** +**Author:** Andre Lorbach <alorbach@adiscon.com> +**Available since:** v8.2304 +=========================== =========================================================================== + + +Purpose +======= + +The purpose of the rsyslog output plugin omazureeventhubs is to provide a +fast and reliable way to send log data from rsyslog to Microsoft Azure Event Hubs. +This plugin uses the Advanced Message Queuing Protocol (AMQP) to securely transmit +log data from rsyslog to Microsoft Azure, where it can be centralized, analyzed, and stored. +The plugin uses the "Qpid Proton C API" library to implement the AMQP protocol, +providing a flexible and efficient solution for sending log data to Microsoft Azure Event Hubs. + +AMQP is a reliable and secure binary protocol for exchanging messages between applications, +and it is widely used in the cloud and enterprise messaging systems. The use of AMQP in the +omazureeventhubs plugin, in combination with the Qpid Proton C API library, ensures that +log data is transmitted in a robust and reliable manner, even in the presence of network +outages or other disruptions. + +The omazureeventhubs plugin supports various configuration options, allowing organizations to +customize their log data pipeline to meet their specific requirements. +This includes options for specifying the Event Hubs endpoint, port, and authentication credentials. +With this plugin, organizations can easily integrate their rsyslog infrastructure with +Microsoft Azure Event Hubs, providing a scalable and secure solution for log management. +The plugin is designed to work with the latest versions of rsyslog and Microsoft Azure, +ensuring compatibility and reliability. + + +Requirements +============ + +To output logs from rsyslog to Microsoft Azure Event Hubs, you will need to fulfill the +following requirements: + +- Qpid Proton C Library Version 0.13 or higher including Qpid Proton ProActor +- The AMQP Protocol needs to have firewall Ports 5671 and 443 TCP to be open for outgoing connections. + + +Configuration Parameters +======================== + +.. note:: + + Parameter names are case-insensitive. + + +Action Parameters +----------------- + +azurehost +^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "yes", "none" + +Specifies the fully qualified domain name (FQDN) of the Event Hubs instance that +the rsyslog output plugin should connect to. The format of the hostname should +be **<namespace>.servicebus.windows.net**, where **<namespace>** is the name +of the Event Hubs namespace that was created in Microsoft Azure. + + +azureport +^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "5671", "no", "none" + +Specifies the TCP port number used by the Event Hubs instance for incoming connections. +The default port number for Event Hubs is 5671 for connections over the +AMQP Secure Sockets Layer (SSL) protocol. This property is usually optional in the configuration +file of the rsyslog output plugin, as the default value of 5671 is typically used. + + +azure_key_name +^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive", "Available since" + :widths: auto + :class: parameter-table + + "word", "none", "yes", "none" + +The configuration property for the Azure key name used to connect to Microsoft Azure Event Hubs is +typically referred to as the "Event Hubs shared access key name". It specifies the name of +the shared access key that is used to authenticate and authorize connections to the Event Hubs instance. +The shared access key is a secret string that is used to securely sign and validate requests +to the Event Hubs instance. + + +azure_key +^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "yes", "none" + +The configuration property for the Azure key used to connect to Microsoft Azure Event Hubs is +typically referred to as the "Event Hubs shared access key". It specifies the value of the +shared access key that is used to authenticate and authorize connections to the Event Hubs instance. +The shared access key is a secret string that is used to securely sign and validate requests +to the Event Hubs instance. + + +container +^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "yes", "none" + +The configuration property for the Azure container used to connect to Microsoft Azure Event Hubs is +typically referred to as the "Event Hubs Instance". It specifies the name of the Event Hubs Instance, +to which log data should be sent. + + +template +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "RSYSLOG_FileFormat", "no", "none" + +Specifies the template used to format and structure the log messages that will be sent from rsyslog to +Microsoft Azure Event Hubs. + +The message template can include rsyslog variables, such as the timestamp, hostname, or process name, +and it can use rsyslog macros, such as $rawmsg or $json, to control the formatting of log data. + +For a message template sample with valid JSON output see the sample below: + +.. code-block:: none + + template(name="generic" type="list" option.jsonf="on") { + property(outname="timestamp" name="timereported" dateFormat="rfc3339" format="jsonf") + constant(value="\"source\": \"EventHubMessage\", ") + property(outname="host" name="hostname" format="jsonf") + property(outname="severity" name="syslogseverity" caseConversion="upper" format="jsonf" datatype="number") + property(outname="facility" name="syslogfacility" format="jsonf" datatype="number") + property(outname="appname" name="syslogtag" format="jsonf") + property(outname="message" name="msg" format="jsonf" ) + property(outname="etlsource" name="$myhostname" format="jsonf") + } + + +amqp_address +^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +The configuration property for the AMQP address used to connect to Microsoft Azure Event Hubs is +typically referred to as the "Event Hubs connection string". It specifies the URL that is used to connect +to the target Event Hubs instance in Microsoft Azure. If the amqp_address is configured, the configuration +parameters for **azurehost**, **azureport**, **azure_key_name** and **azure_key** will be ignored. + +A sample Event Hubs connection string URL is: + +.. code-block:: none + + amqps://[Shared access key name]:[Shared access key]@[Event Hubs namespace].servicebus.windows.net/[Event Hubs Instance] + + +eventproperties +^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "array", "none", "no", "none" + +The **eventproperties** configuration property is an array property used to add key-value pairs as additional properties to the +encoded AMQP message object, providing additional information about the log event. +These properties can be used for filtering, routing, and grouping log events in Azure Event Hubs. + +The event properties property is specified as a list of key-value pairs separated by comma, +with the key and value separated by an equal sign. + +For example, the following configuration setting adds two event properties: + +.. code-block:: none + + eventproperties=[ "Table=TestTable", + "Format=JSON"] + +In this example, the Table and Format keys are added to the message object as event properties, +with the corresponding values of TestTable and JSON, respectively. + + +closeTimeout +^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "2000", "no", "none" + +The close timeout configuration property is used in the rsyslog output module +to specify the amount of time the output module should wait for a response +from Microsoft Azure Event Hubs before timing out and closing the connection. + +This property is used to control the amount of time the output module will wait +for a response from the target Event Hubs instance before giving up and +assuming that the connection has failed. The close timeout property is specified in milliseconds. + + +statsname +^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "omazureeventhubs", "no", "none" + +The name assigned to statistics specific to this action instance. The supported set of +statistics tracked for this action instance are **submitted**, **accepted**, **failures** and **failures_other**. +See the :ref:`statistics-counter_omazureeventhubs_label` section for more details. + + +.. _statistics-counter_omazureeventhubs_label: + +Statistic Counter +================= + +This plugin maintains global :doc:`statistics <../rsyslog_statistic_counter>` for omazureeventhubs that +accumulate all action instances. The statistic origin is named "omazureeventhubs" with following counters: + + +- **submitted** - This counter tracks the number of log messages that have been submitted by the rsyslog process + to the output module for delivery to Microsoft Azure Event Hubs. + +- **accepted** - This counter tracks the number of log messages that have been successfully delivered to + Microsoft Azure Event Hubs by the output module. + +- **failures** - This counter tracks the number of log messages that have failed to be delivered to + Microsoft Azure Event Hubs due to various error conditions, such as network connectivity issues, + incorrect configuration settings, or other technical problems. This counter provides important information about + any issues that may be affecting the delivery of log data to Microsoft Azure Event Hubs. + +- **failures_other** - This counter tracks the number of log messages that have failed to be delivered due to + other error conditions, such as incorrect payload format or unexpected data. + +These statistics counters are updated in real-time by the rsyslog output module as log data is processed, +and they provide valuable information about the performance and operation of the output module. + +For multiple actions using statistics callback, there will be one record for each action. + +.. _omazureeventhubs-examples-label: + +Examples +======== + +Example 1: Use AMQP URL +----------------------- + +The following sample does the following: + +- loads the omazureeventhubs module +- outputs all logs to Microsoft Azure Event Hubs with standard template +- Uses amqp_address parameter + +.. code-block:: none + + module(load="omazureeventhubs") + action(type="omazureeventhubs" amqp_address="amqps://<AccessKeyName>:<AccessKey>@<EventHubsNamespace>.servicebus.windows.net/<EventHubsInstance>") + + +Example 2: RAW Format +--------------------- + +The following sample does the following: + +- loads the omazureeventhubs module +- outputs all logs to Microsoft Azure Event Hubs with simple custom template +- Uses **azurehost**, **azureport**, **azure_key_name** and **azure_key** + parameters instead of **amqp_address** parameter + +.. code-block:: none + + module(load="omazureeventhubs") + template(name="outfmt" type="string" string="%msg%\n") + + action(type="omazureeventhubs" + azurehost="<EventHubsNamespace>.servicebus.windows.net" + azureport="5671" + azure_key_name="<AccessKeyName>" + azure_key="<AccessKey>" + container="<EventHubsInstance>" + template="outfmt" + ) + + +Example 3: JSON Format +---------------------- + +The following sample does the following: + +- loads the omazureeventhubs module +- outputs all logs to Microsoft Azure Event Hubs with JSON custom template +- Uses **azurehost**, **azureport**, **azure_key_name** and **azure_key** + parameters instead of **amqp_address** parameter +- Uses **eventproperties** array parameter to set additional message properties + +.. code-block:: none + + module(load="omazureeventhubs") + template(name="outfmtjson" type="list" option.jsonf="on") { + property(outname="timestamp" name="timereported" dateFormat="rfc3339" format="jsonf") + constant(value="\"source\": \"EventHubMessage\", ") + property(outname="host" name="hostname" format="jsonf") + property(outname="severity" name="syslogseverity" caseConversion="upper" format="jsonf" datatype="number") + property(outname="facility" name="syslogfacility" format="jsonf" datatype="number") + property(outname="appname" name="syslogtag" format="jsonf") + property(outname="message" name="msg" format="jsonf" ) + property(outname="etlsource" name="$myhostname" format="jsonf") + } + + action(type="omazureeventhubs" + azurehost="<EventHubsNamespace>.servicebus.windows.net" + azureport="5671" + azure_key_name="<AccessKeyName>" + azure_key="<AccessKey>" + container="<EventHubsInstance>" + template="outfmtjson" + eventproperties=[ "Table=CustomTable", + "Format=JSON"] + ) + +Example 4: High Performance +--------------------------- + +To achieve high performance when sending syslog data to Azure Event Hubs, you should consider configuring your output module to use multiple worker instances. This can be done by setting the "workerthreads" parameter in the configuration file. + +The following example is for high performance (Azure Premium Tier) and does the following: + +- loads the omazureeventhubs module +- outputs all logs to Microsoft Azure Event Hubs with JSON custom template +- Uses **azurehost**, **azureport**, **azure_key_name** and **azure_key** + parameters instead of **amqp_address** parameter +- Uses **eventproperties** array parameter to set additional message properties +- Uses **Linkedlist** In-Memory Queue which enables multiple omazureeventhubs workers running at the same time. Using a dequeue size of 2000 and a dequeue timeout of 1000 has shown very good results in performance tests. +- Uses 8 workerthreads in this example, which will be spawn automatically if more than 2000 messages are waiting in the Queue. To achieve more performance, the number can be incremented. + +.. code-block:: none + + module(load="omazureeventhubs") + template(name="outfmtjson" type="list" option.jsonf="on") { + property(outname="timestamp" name="timereported" dateFormat="rfc3339" format="jsonf") + constant(value="\"source\": \"EventHubMessage\", ") + property(outname="host" name="hostname" format="jsonf") + property(outname="severity" name="syslogseverity" caseConversion="upper" format="jsonf" datatype="number") + property(outname="facility" name="syslogfacility" format="jsonf" datatype="number") + property(outname="appname" name="syslogtag" format="jsonf") + property(outname="message" name="msg" format="jsonf" ) + property(outname="etlsource" name="$myhostname" format="jsonf") + } + + action(type="omazureeventhubs" + azurehost="<EventHubsNamespace>.servicebus.windows.net" + azureport="5671" + azure_key_name="<AccessKeyName>" + azure_key="<AccessKey>" + container="<EventHubsInstance>" + template="outfmtjson" + eventproperties=[ "Table=CustomTable", + "Format=JSON"] + queue.type="linkedList" + queue.size="200000" + queue.saveonshutdown="on" + queue.dequeueBatchSize="2000" + queue.minDequeueBatchSize.timeout="1000" + queue.workerThreads="8" + queue.workerThreadMinimumMessages="2000" + queue.timeoutWorkerthreadShutdown="10000" + queue.timeoutshutdown="1000" + ) + diff --git a/source/configuration/modules/omclickhouse.rst b/source/configuration/modules/omclickhouse.rst new file mode 100644 index 0000000..499d8bd --- /dev/null +++ b/source/configuration/modules/omclickhouse.rst @@ -0,0 +1,324 @@ +************************************** +omclickhouse: ClickHouse Output Module +************************************** + +=========================== =========================================================================== +**Module Name:** **omclickhouse** +**Author:** Pascal Withopf <pwithopf@adiscon.com> +=========================== =========================================================================== + + +Purpose +======= + +This module provides native support for logging to +`ClickHouse <https://clickhouse.yandex/>`_. +To enable the module use "--enable-clickhouse" while configuring rsyslog. +Tests for the testbench can be enabled with "--enable-clickhouse-tests". + + +Notable Features +================ + +- :ref:`omclickhouse-statistic-counter` + + +Configuration Parameters +======================== + +.. note:: + + Parameter names are case-insensitive. + + +Action Parameters +----------------- + +Server +^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "localhost", "no", "none" + +The address of a ClickHouse server. + +.. _port: + +Port +^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "8123", "no", "none" + +HTTP port to use to connect to ClickHouse. + + +usehttps +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "on", "no", "none" + +Default scheme to use when sending events to ClickHouse if none is +specified on a server. + + +template +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "StdClickHouseFmt", "no", "none" + +This is the message format that will be sent to ClickHouse. The +resulting string needs to be a valid INSERT Query, otherwise ClickHouse +will return an error. Defaults to: + +.. code-block:: none + + "\"INSERT INTO rsyslog.SystemEvents (severity, facility, " + "timestamp, hostname, tag, message) VALUES (%syslogseverity%, %syslogfacility%, " + "'%timereported:::date-unixtimestamp%', '%hostname%', '%syslogtag%', '%msg%')\"" + + +bulkmode +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "on", "no", "none" + +The "off" setting means logs are shipped one by one. Each in +its own HTTP request. +The default "on" will send multiple logs in the same request. This is +recommended, because it is many times faster than when bulkmode is turned off. +The maximum number of logs sent in a single bulk request depends on your +maxbytes and queue settings - usually limited by the `dequeue batch +size <http://www.rsyslog.com/doc/node35.html>`_. More information +about queues can be found +`here <http://www.rsyslog.com/doc/node32.html>`_. + + +maxbytes +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "Size", "104857600/100mb", "no", "none" + +When shipping logs with bulkmode **on**, maxbytes specifies the maximum +size of the request body sent to ClickHouse. Logs are batched until +either the buffer reaches maxbytes or the `dequeue batch +size <http://www.rsyslog.com/doc/node35.html>`_ is reached. + + +user +^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "default", "no", "none" + +If you have basic HTTP authentication deployed you can specify your user-name here. + + +pwd +^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "", "no", "none" + +Password for basic authentication. + + +errorFile +^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +If specified, records failed in bulk mode are written to this file, including +their error cause. Rsyslog itself does not process the file any more, but the +idea behind that mechanism is that the user can create a script to periodically +inspect the error file and react appropriately. As the complete request is +included, it is possible to simply resubmit messages from that script. + +*Please note:* when rsyslog has problems connecting to clickhouse, a general +error is assumed. However, if we receive negative responses during batch +processing, we assume an error in the data itself (like a mandatory field is +not filled in, a format error or something along those lines). Such errors +cannot be solved by simply resubmitting the record. As such, they are written +to the error file so that the user (script) can examine them and act appropriately. +Note that e.g. after search index reconfiguration (e.g. dropping the mandatory +attribute) a resubmit may be successful. + + +allowUnsignedCerts +^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "on", "no", "none" + +The module accepts connections to servers, which have unsigned certificates. +If this parameter is disabled, the module will verify whether the certificates +are authentic. + + +skipverifyhost +^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "boolean", "off", "no", "none" + +If `"on"`, this will set the curl `CURLOPT_SSL_VERIFYHOST` option to +`0`. You are strongly discouraged to set this to `"on"`. It is +primarily useful only for debugging or testing. + + +healthCheckTimeout +^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "int", "3500", "no", "none" + +This parameter sets the timeout for checking the availability +of ClickHouse. Value is given in milliseconds. + + +timeout +^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "int", "0", "no", "none" + +This parameter sets the timeout for sending data to ClickHouse. +Value is given in milliseconds. + + +.. _omclickhouse-statistic-counter: + +Statistic Counter +================= + +This plugin maintains global :doc:`statistics <../rsyslog_statistic_counter>`, +which accumulate all action instances. The statistic is named "omclickhouse". +Parameters are: + +- **submitted** - number of messages submitted for processing (with both + success and error result) + +- **fail.httprequests** - the number of times a http request failed. Note + that a single http request may be used to submit multiple messages, so this + number may be (much) lower than failed.http. + +- **failed.http** - number of message failures due to connection like-problems + (things like remote server down, broken link etc) + +- **fail.clickhouse** - number of failures due to clickhouse error reply; Note that + this counter does NOT count the number of failed messages but the number of + times a failure occurred (a potentially much smaller number). Counting messages + would be quite performance-intense and is thus not done. + +- **response.success** - number of records successfully sent in bulk index + requests - counts the number of successful responses + + +**The fail.httprequests and failed.http counters reflect only failures that +omclickhouse detected.** Once it detects problems, it (usually, depends on +circumstances) tell the rsyslog core that it wants to be suspended until the +situation clears (this is a requirement for rsyslog output modules). Once it is +suspended, it does NOT receive any further messages. Depending on the user +configuration, messages will be lost during this period. Those lost messages will +NOT be counted by impstats (as it does not see them). + + +Examples +======== + +Example 1 +--------- + +The following sample does the following: + +- loads the omclickhouse module +- outputs all logs to ClickHouse using the default settings + +.. code-block:: none + + module(load="omclickhouse") + action(type="omclickhouse") + + +Example 2 +--------- + +In this example the URL will use http and the specified parameters to create +the REST URL. + +.. code-block:: none + + module(load="omclickhouse") + action(type="omclickhouse" server="127.0.0.1" port="8124" user="user1" pwd="pwd1" + usehttps="off") + + +Example 3 +--------- + +This example will send messages in batches up to 10MB. +If an error occurs it will be written in the error file. + +.. code-block:: none + + module(load="omclickhouse") + action(type="omclickhouse" maxbytes="10mb" errorfile="clickhouse-error.log") + + diff --git a/source/configuration/modules/omelasticsearch.rst b/source/configuration/modules/omelasticsearch.rst new file mode 100644 index 0000000..ee561fc --- /dev/null +++ b/source/configuration/modules/omelasticsearch.rst @@ -0,0 +1,1102 @@ +******************************************** +omelasticsearch: Elasticsearch Output Module +******************************************** + +=========================== =========================================================================== +**Module Name:** **omelasticsearch** +**Author:** `Rainer Gerhards <https://rainer.gerhards.net/>`_ <rgerhards@adiscon.com> +=========================== =========================================================================== + + +Purpose +======= + +This module provides native support for logging to +`Elasticsearch <http://www.elasticsearch.org/>`_. + + +Notable Features +================ + +- :ref:`omelasticsearch-statistic-counter` + + +Configuration Parameters +======================== + +.. note:: + + Parameter names are case-insensitive. + + +Action Parameters +----------------- + +Server +^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "array", "none", "no", "none" + +An array of Elasticsearch servers in the specified format. If no scheme +is specified, it will be chosen according to usehttps_. If no port is +specified, serverport_ will be used. Defaults to "localhost". + +Requests to Elasticsearch will be load-balanced between all servers in +round-robin fashion. + +.. code-block:: none + + Examples: + server="localhost:9200" + server=["elasticsearch1", "elasticsearch2"] + + +.. _serverport: + +Serverport +^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "9200", "no", "none" + +Default HTTP port to use to connect to Elasticsearch if none is specified +on a server_. Defaults to 9200 + + +.. _healthchecktimeout: + +HealthCheckTimeout +^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "3500", "no", "none" + +Specifies the number of milliseconds to wait for a successful health check +on a server_. Before trying to submit events to Elasticsearch, rsyslog will +execute an *HTTP HEAD* to ``/_cat/health`` and expect an *HTTP OK* within +this timeframe. Defaults to 3500. + +*Note, the health check is verifying connectivity only, not the state of +the Elasticsearch cluster.* + + +.. _esVersion_major: + +esVersion.major +^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "0", "no", "none" + +ElasticSearch is notoriously bad at maintaining backwards compatibility. For this +reason, the setting can be used to configure the server's major version number (e.g. 7, 8, ...). +As far as we know breaking changes only happen with major version changes. +As of now, only value 8 triggers API changes. All other values select +pre-version-8 API usage. + +.. _searchIndex: + +searchIndex +^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +`Elasticsearch +index <http://www.elasticsearch.org/guide/appendix/glossary.html#index>`_ +to send your logs to. Defaults to "system" + + +.. _dynSearchIndex: + +dynSearchIndex +^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +Whether the string provided for searchIndex_ should be taken as a +`rsyslog template <http://www.rsyslog.com/doc/rsyslog_conf_templates.html>`_. +Defaults to "off", which means the index name will be taken +literally. Otherwise, it will look for a template with that name, and +the resulting string will be the index name. For example, let's +assume you define a template named "date-days" containing +"%timereported:1:10:date-rfc3339%". Then, with dynSearchIndex="on", +if you say searchIndex="date-days", each log will be sent to and +index named after the first 10 characters of the timestamp, like +"2013-03-22". + + +.. _searchType: + +searchType +^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +`Elasticsearch +type <http://www.elasticsearch.org/guide/appendix/glossary.html#type>`_ +to send your index to. Defaults to "events". +Setting this parameter to an empty string will cause the type to be omitted, +which is required since Elasticsearch 7.0. See +`Elasticsearch documentation <https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html>`_ +for more information. + + +.. _dynSearchType: + +dynSearchType +^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +Like dynSearchIndex_, it allows you to specify a +`rsyslog template <http://www.rsyslog.com/doc/rsyslog_conf_templates.html>`_ +for searchType_, instead of a static string. + + +.. _pipelineName: + +pipelineName +^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +The `ingest node <https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest.html>`_ +pipeline name to be included in the request. This allows pre processing +of events before indexing them. By default, events are not send to a pipeline. + + +.. _dynPipelineName: + +dynPipelineName +^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +Like dynSearchIndex_, it allows you to specify a +`rsyslog template <http://www.rsyslog.com/doc/rsyslog_conf_templates.html>`_ +for pipelineName_, instead of a static string. + + +.. _skipPipelineIfEmpty: + +skipPipelineIfEmpty +^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +When POST'ing a document, Elasticsearch does not allow an empty pipeline +parameter value. If boolean option skipPipelineIfEmpty is set to `"on"`, the +pipeline parameter won't be posted. Default is `"off"`. + + +.. _asyncrepl: + +asyncrepl +^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +No longer supported as ElasticSearch no longer supports it. + + +.. _usehttps: + +usehttps +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +Default scheme to use when sending events to Elasticsearch if none is +specified on a server_. Good for when you have +Elasticsearch behind Apache or something else that can add HTTPS. +Note that if you have a self-signed certificate, you'd need to install +it first. This is done by copying the certificate to a trusted path +and then running *update-ca-certificates*. That trusted path is +typically */usr/local/share/ca-certificates* but check the man page of +*update-ca-certificates* for the default path of your distro + + +.. _timeout: + +timeout +^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "1m", "no", "none" + +How long Elasticsearch will wait for a primary shard to be available +for indexing your log before sending back an error. Defaults to "1m". + + +.. _indextimeout: + +indexTimeout +^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "0", "no", "none" + +.. versionadded:: 8.2204.0 + +Specifies the number of milliseconds to wait for a successful log indexing +request on a server_. By default there is no timeout. + + +.. _template: + +template +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "see below", "no", "none" + +This is the JSON document that will be indexed in Elasticsearch. The +resulting string needs to be a valid JSON, otherwise Elasticsearch +will return an error. Defaults to: + +.. code-block:: none + + $template StdJSONFmt, "{\"message\":\"%msg:::json%\",\"fromhost\":\"%HOSTNAME:::json%\",\"facility\":\"%syslogfacility-text%\",\"priority\":\"%syslogpriority-text%\",\"timereported\":\"%timereported:::date-rfc3339%\",\"timegenerated\":\"%timegenerated:::date-rfc3339%\"}" + +Which will produce this sort of documents (pretty-printed here for +readability): + +.. code-block:: none + + { + "message": " this is a test message", + "fromhost": "test-host", + "facility": "user", + "priority": "info", + "timereported": "2013-03-12T18:05:01.344864+02:00", + "timegenerated": "2013-03-12T18:05:01.344864+02:00" + } + +Another template, FullJSONFmt, is available that includes more fields including programname, PROCID (usually the process ID), and MSGID. + +.. _bulkmode: + +bulkmode +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +The default "off" setting means logs are shipped one by one. Each in +its own HTTP request, using the `Index +API <http://www.elasticsearch.org/guide/reference/api/index_.html>`_. +Set it to "on" and it will use Elasticsearch's `Bulk +API <http://www.elasticsearch.org/guide/reference/api/bulk.html>`_ to +send multiple logs in the same request. The maximum number of logs +sent in a single bulk request depends on your maxbytes_ +and queue settings - +usually limited by the `dequeue batch +size <http://www.rsyslog.com/doc/node35.html>`_. More information +about queues can be found +`here <http://www.rsyslog.com/doc/node32.html>`_. + + +.. _maxbytes: + +maxbytes +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "100m", "no", "none" + +.. versionadded:: 8.23.0 + +When shipping logs with bulkmode_ **on**, maxbytes specifies the maximum +size of the request body sent to Elasticsearch. Logs are batched until +either the buffer reaches maxbytes or the `dequeue batch +size <http://www.rsyslog.com/doc/node35.html>`_ is reached. In order to +ensure Elasticsearch does not reject requests due to content length, verify +this value is set according to the `http.max_content_length +<https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-http.html>`_ +setting in Elasticsearch. Defaults to 100m. + + +.. _parent: + +parent +^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +Specifying a string here will index your logs with that string the +parent ID of those logs. Please note that you need to define the +`parent +field <http://www.elasticsearch.org/guide/reference/mapping/parent-field.html>`_ +in your +`mapping <http://www.elasticsearch.org/guide/reference/mapping/>`_ +for that to work. By default, logs are indexed without a parent. + + +.. _dynParent: + +dynParent +^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +Using the same parent for all the logs sent in the same action is +quite unlikely. So you'd probably want to turn this "on" and specify +a +`rsyslog template <http://www.rsyslog.com/doc/rsyslog_conf_templates.html>`_ +that will provide meaningful parent IDs for your logs. + + +.. _uid: + +uid +^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +If you have basic HTTP authentication deployed (eg through the +`elasticsearch-basic +plugin <https://github.com/Asquera/elasticsearch-http-basic>`_), you +can specify your user-name here. + + +.. _pwd: + +pwd +^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +Password for basic authentication. + + +.. _errorfile: + +errorFile +^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +If specified, records failed in bulk mode are written to this file, including +their error cause. Rsyslog itself does not process the file any more, but the +idea behind that mechanism is that the user can create a script to periodically +inspect the error file and react appropriately. As the complete request is +included, it is possible to simply resubmit messages from that script. + +*Please note:* when rsyslog has problems connecting to elasticsearch, a general +error is assumed and the submit is retried. However, if we receive negative +responses during batch processing, we assume an error in the data itself +(like a mandatory field is not filled in, a format error or something along +those lines). Such errors cannot be solved by simply resubmitting the record. +As such, they are written to the error file so that the user (script) can +examine them and act appropriately. Note that e.g. after search index +reconfiguration (e.g. dropping the mandatory attribute) a resubmit may +be successful. + +.. _omelasticsearch-tls.cacert: + +tls.cacert +^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +This is the full path and file name of the file containing the CA cert for the +CA that issued the Elasticsearch server cert. This file is in PEM format. For +example: `/etc/rsyslog.d/es-ca.crt` + +.. _tls.mycert: + +tls.mycert +^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +This is the full path and file name of the file containing the client cert for +doing client cert auth against Elasticsearch. This file is in PEM format. For +example: `/etc/rsyslog.d/es-client-cert.pem` + +.. _tls.myprivkey: + +tls.myprivkey +^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +This is the full path and file name of the file containing the private key +corresponding to the cert `tls.mycert` used for doing client cert auth against +Elasticsearch. This file is in PEM format, and must be unencrypted, so take +care to secure it properly. For example: `/etc/rsyslog.d/es-client-key.pem` + +.. _omelasticsearch-allowunsignedcerts: + +allowunsignedcerts +^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "boolean", "off", "no", "none" + +If `"on"`, this will set the curl `CURLOPT_SSL_VERIFYPEER` option to +`0`. You are strongly discouraged to set this to `"on"`. It is +primarily useful only for debugging or testing. + +.. _omelasticsearch-skipverifyhost: + +skipverifyhost +^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "boolean", "off", "no", "none" + +If `"on"`, this will set the curl `CURLOPT_SSL_VERIFYHOST` option to +`0`. You are strongly discouraged to set this to `"on"`. It is +primarily useful only for debugging or testing. + +.. _omelasticsearch-bulkid: + +bulkid +^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +This is the unique id to assign to the record. The `bulk` part is misleading - this +can be used in both bulk mode :ref:`bulkmode` or in index +(record at a time) mode. Although you can specify a static value for this +parameter, you will almost always want to specify a *template* for the value of +this parameter, and set `dynbulkid="on"` :ref:`omelasticsearch-dynbulkid`. NOTE: +you must use `bulkid` and `dynbulkid` in order to use `writeoperation="create"` +:ref:`omelasticsearch-writeoperation`. + +.. _omelasticsearch-dynbulkid: + +dynbulkid +^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +If this parameter is set to `"on"`, then the `bulkid` parameter :ref:`omelasticsearch-bulkid` +specifies a *template* to use to generate the unique id value to assign to the record. If +using `bulkid` you will almost always want to set this parameter to `"on"` to assign +a different unique id value to each record. NOTE: +you must use `bulkid` and `dynbulkid` in order to use `writeoperation="create"` +:ref:`omelasticsearch-writeoperation`. + +.. _omelasticsearch-writeoperation: + +writeoperation +^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "index", "no", "none" + +The value of this parameter is either `"index"` (the default) or `"create"`. If `"create"` is +used, this means the bulk action/operation will be `create` - create a document only if the +document does not already exist. The record must have a unique id in order to use `create`. +See :ref:`omelasticsearch-bulkid` and :ref:`omelasticsearch-dynbulkid`. See +:ref:`omelasticsearch-writeoperation-example` for an example. + +.. _omelasticsearch-retryfailures: + +retryfailures +^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +If this parameter is set to `"on"`, then the module will look for an +`"errors":true` in the bulk index response. If found, each element in the +response will be parsed to look for errors, since a bulk request may have some +records which are successful and some which are failures. Failed requests will +be converted back into records and resubmitted back to rsyslog for +reprocessing. Each failed request will be resubmitted with a local variable +called `$.omes`. This is a hash consisting of the fields from the metadata +header in the original request, and the fields from the response. If the same +field is in the request and response, the value from the field in the *request* +will be used, to facilitate retries that want to send the exact same request, +and want to know exactly what was sent. +See below :ref:`omelasticsearch-retry-example` for an example of how retry +processing works. +*NOTE* The retried record will be resubmitted at the "top" of your processing +pipeline. If your processing pipeline is not idempotent (that is, your +processing pipeline expects "raw" records), then you can specify a ruleset to +redirect retries to. See :ref:`omelasticsearch-retryruleset` below. + +`$.omes` fields: + +* writeoperation - the operation used to submit the request - for rsyslog + omelasticsearch this currently means either `"index"` or `"create"` +* status - the HTTP status code - typically an error will have a `4xx` or `5xx` + code - of particular note is `429` - this means Elasticsearch was unable to + process this bulk record request due to a temporary condition e.g. the bulk + index thread pool queue is full, and rsyslog should retry the operation. +* _index, _type, _id, pipeline, _parent - the metadata associated with the + request - not all of these fields will be present with every request - for + example, if you do not use `"pipelinename"` or `"dynpipelinename"`, there + will be no `$.omes!pipeline` field. +* error - a hash containing one or more, possibly nested, fields containing + more detailed information about a failure. Typically there will be fields + `$.omes!error!type` (a keyword) and `$.omes!error!reason` (a longer string) + with more detailed information about the rejection. NOTE: The format is + apparently not described in great detail, so code must not make any + assumption about the availability of `error` or any specific sub-field. + +There may be other fields too - the code just copies everything in the +response. Here is an example of a detailed error response, in JSON format, from +Elasticsearch 5.6.9: + +.. code-block:: json + + {"omes": + {"writeoperation": "create", + "_index": "rsyslog_testbench", + "_type": "test-type", + "_id": "92BE7AF79CD44305914C7658AF846A08", + "status": 400, + "error": + {"type": "mapper_parsing_exception", + "reason": "failed to parse [msgnum]", + "caused_by": + {"type": "number_format_exception", + "reason": "For input string: \"x00000025\""}}}} + +Reference: https://www.elastic.co/guide/en/elasticsearch/guide/current/bulk.html#bulk + +.. _omelasticsearch-retryruleset: + +retryruleset +^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "", "no", "none" + +If `retryfailures` is not `"on"` (:ref:`omelasticsearch-retryfailures`) then +this parameter has no effect. This parameter specifies the name of a ruleset +to use to route retries. This is useful if you do not want retried messages to +be processed starting from the top of your processing pipeline, or if you have +multiple outputs but do not want to send retried Elasticsearch failures to all +of your outputs, and you do not want to clutter your processing pipeline with a +lot of conditionals. See below :ref:`omelasticsearch-retry-example` for an +example of how retry processing works. + +.. _omelasticsearch-ratelimit.interval: + +ratelimit.interval +^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "600", "no", "none" + +If `retryfailures` is not `"on"` (:ref:`omelasticsearch-retryfailures`) then +this parameter has no effect. Specifies the interval in seconds onto which +rate-limiting is to be applied. If more than ratelimit.burst messages are read +during that interval, further messages up to the end of the interval are +discarded. The number of messages discarded is emitted at the end of the +interval (if there were any discards). +Setting this to value zero turns off ratelimiting. + +.. _omelasticsearch-ratelimit.burst: + +ratelimit.burst +^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "20000", "no", "none" + +If `retryfailures` is not `"on"` (:ref:`omelasticsearch-retryfailures`) then +this parameter has no effect. Specifies the maximum number of messages that +can be emitted within the ratelimit.interval interval. For further information, +see description there. + +.. _omelasticsearch-rebindinterval: + +rebindinterval +^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "-1", "no", "none" + +This parameter tells omelasticsearch to close the connection and reconnect +to Elasticsearch after this many operations have been submitted. The default +value `-1` means that omelasticsearch will not reconnect. A value greater +than `-1` tells omelasticsearch, after this many operations have been +submitted to Elasticsearch, to drop the connection and establish a new +connection. This is useful when rsyslog connects to multiple Elasticsearch +nodes through a router or load balancer, and you need to periodically drop +and reestablish connections to help the router balance the connections. Use +the counter `rebinds` to monitor the number of times this has happened. + +.. _omelasticsearch-statistic-counter: + +Statistic Counter +================= + +This plugin maintains global :doc:`statistics <../rsyslog_statistic_counter>`, +which accumulate all action instances. The statistic is named "omelasticsearch". +Parameters are: + +- **submitted** - number of messages submitted for processing (with both + success and error result) + +- **fail.httprequests** - the number of times a http request failed. Note + that a single http request may be used to submit multiple messages, so this + number may be (much) lower than fail.http. + +- **fail.http** - number of message failures due to connection like-problems + (things like remote server down, broken link etc) + +- **fail.es** - number of failures due to elasticsearch error reply; Note that + this counter does NOT count the number of failed messages but the number of + times a failure occurred (a potentially much smaller number). Counting messages + would be quite performance-intense and is thus not done. + +The following counters are available when `retryfailures="on"` is used: + +- **response.success** - number of records successfully sent in bulk index + requests - counts the number of successful responses + +- **response.bad** - number of times omelasticsearch received a response in a + bulk index response that was unrecognized or unable to be parsed. This may + indicate that omelasticsearch is attempting to communicate with a version of + Elasticsearch that is incompatible, or is otherwise sending back data in the + response that cannot be handled + +- **response.duplicate** - number of records in the bulk index request that + were duplicates of already existing records - this will only be reported if + using `writeoperation="create"` and `bulkid` to assign each record a unique + ID + +- **response.badargument** - number of times omelasticsearch received a + response that had a status indicating omelasticsearch sent bad data to + Elasticsearch. For example, status `400` and an error message indicating + omelasticsearch attempted to store a non-numeric string value in a numeric + field. + +- **response.bulkrejection** - number of times omelasticsearch received a + response that had a status indicating Elasticsearch was unable to process + the record at this time - status `429`. The record can be retried. + +- **response.other** - number of times omelasticsearch received a + response not recognized as one of the above responses, typically some other + `4xx` or `5xx` http status. + +- **rebinds** - if using `rebindinterval` this will be the number of + times omelasticsearch has reconnected to Elasticsearch + +**The fail.httprequests and fail.http counters reflect only failures that +omelasticsearch detected.** Once it detects problems, it (usually, depends on +circumstances) tell the rsyslog core that it wants to be suspended until the +situation clears (this is a requirement for rsyslog output modules). Once it is +suspended, it does NOT receive any further messages. Depending on the user +configuration, messages will be lost during this period. Those lost messages will +NOT be counted by impstats (as it does not see them). + +Note that some previous (pre 7.4.5) versions of this plugin had different counters. +These were experimental and confusing. The only ones really used were "submits", +which were the number of successfully processed messages and "connfail" which were +equivalent to "failed.http". + +How Retries Are Handled +======================= + +When using `retryfailures="on"` (:ref:`omelasticsearch-retryfailures`), the +original `Message` object (that is, the original `smsg_t *msg` object) **is not +available**. This means none of the metadata associated with that object, such +as various timestamps, hosts/ip addresses, etc. are not available for the retry +operation. The only thing available are the metadata header (_index, _type, +_id, pipeline, _parent) and original JSON string sent in the original request, +and whatever data is returned in the error response. All of these are made +available in the `$.omes` fields. If the same field name exists in the request +metadata and the response, the field from the request will be used, in order to +facilitate retrying the exact same request. For the message to retry, the code +will take the original JSON string and parse it back into an internal `Message` +object. This means you **may need to use a different template** to output +messages for your retry ruleset. For example, if you used the following +template to format the Elasticsearch message for the initial submission: + +.. code-block:: none + + template(name="es_output_template" + type="list" + option.json="on") { + constant(value="{") + constant(value="\"timestamp\":\"") property(name="timereported" dateFormat="rfc3339") + constant(value="\",\"message\":\"") property(name="msg") + constant(value="\",\"host\":\"") property(name="hostname") + constant(value="\",\"severity\":\"") property(name="syslogseverity-text") + constant(value="\",\"facility\":\"") property(name="syslogfacility-text") + constant(value="\",\"syslogtag\":\"") property(name="syslogtag") + constant(value="\"}") + } + +You would have to use a different template for the retry, since none of the +`timereported`, `msg`, etc. fields will have the same values for the retry as +for the initial try. + +Same with the other omelasticsearch parameters which can be constructed with +templates, such as `"dynpipelinename"`, `"dynsearchindex"`, `"dynsearchtype"`, +`"dynparent"`, and `"dynbulkid"`. For example, if you generate the `_id` to +use in the request, you will want to reuse the same `_id` for each subsequent +retry: + +.. code-block:: none + + template(name="id-template" type="string" string="%$.es_msg_id%") + if strlen($.omes!_id) > 0 then { + set $.es_msg_id = $.omes!_id; + } else { + # NOTE: depends on rsyslog being compiled with --enable-uuid + set $.es_msg_id = $uuid; + } + action(type="omelasticsearch" bulkid="id-template" ...) + +That is, if this is a retry, `$.omes!_id` will be set, so use that value for +the bulk id for this record, otherwise, generate a new one with `$uuid`. Note +that the template uses the temporary variable `$.es_msg_id` which must be set +each time, to either `$.omes!_id` or `$uuid`. + +The `rawmsg` field is a special case. If the original request had a field +called `message`, then when constructing the new message from the original to +retry, the `rawmsg` message property will be set to the value of the `message` +field. Otherwise, the `rawmsg` property value will be set to the entire +original request - the data part, not the metadata. In previous versions, +without the `message` field, the `rawmsg` property was set to the value of the +data plus the Elasticsearch metadata, which caused problems with retries. See +`rsyslog issue 3573 <https://github.com/rsyslog/rsyslog/issues/3573>`_ + +Examples +======== + +Example 1 +--------- + +The following sample does the following: + +- loads the omelasticsearch module +- outputs all logs to Elasticsearch using the default settings + +.. code-block:: none + + module(load="omelasticsearch") + *.* action(type="omelasticsearch") + + +Example 2 +--------- + +The following sample does the following: + +- loads the omelasticsearch module +- outputs all logs to Elasticsearch using the full JSON logging template including program name + +.. code-block:: none + + module(load="omelasticsearch") + *.* action(type="omelasticsearch" template="FullJSONFmt") + + +Example 3 +--------- + +The following sample does the following: + +- loads the omelasticsearch module +- defines a template that will make the JSON contain the following + properties + + - RFC-3339 timestamp when the event was generated + - the message part of the event + - hostname of the system that generated the message + - severity of the event, as a string + - facility, as a string + - the tag of the event + +- outputs to Elasticsearch with the following settings + + - host name of the server is myserver.local + - port is 9200 + - JSON docs will look as defined in the template above + - index will be "test-index" + - type will be "test-type" + - activate bulk mode. For that to work effectively, we use an + in-memory queue that can hold up to 5000 events. The maximum bulk + size will be 300 + - retry indefinitely if the HTTP request failed (eg: if the target + server is down) + +.. code-block:: none + + module(load="omelasticsearch") + template(name="testTemplate" + type="list" + option.json="on") { + constant(value="{") + constant(value="\"timestamp\":\"") property(name="timereported" dateFormat="rfc3339") + constant(value="\",\"message\":\"") property(name="msg") + constant(value="\",\"host\":\"") property(name="hostname") + constant(value="\",\"severity\":\"") property(name="syslogseverity-text") + constant(value="\",\"facility\":\"") property(name="syslogfacility-text") + constant(value="\",\"syslogtag\":\"") property(name="syslogtag") + constant(value="\"}") + } + action(type="omelasticsearch" + server="myserver.local" + serverport="9200" + template="testTemplate" + searchIndex="test-index" + searchType="test-type" + bulkmode="on" + maxbytes="100m" + queue.type="linkedlist" + queue.size="5000" + queue.dequeuebatchsize="300" + action.resumeretrycount="-1") + + +.. _omelasticsearch-writeoperation-example: + +Example 4 +--------- + +The following sample shows how to use :ref:`omelasticsearch-writeoperation` +with :ref:`omelasticsearch-dynbulkid` and :ref:`omelasticsearch-bulkid`. For +simplicity, it assumes rsyslog has been built with `--enable-libuuid` which +provides the `uuid` property for each record: + +.. code-block:: none + + module(load="omelasticsearch") + set $!es_record_id = $uuid; + template(name="bulkid-template" type="list") { property(name="$!es_record_id") } + action(type="omelasticsearch" + ... + bulkmode="on" + bulkid="bulkid-template" + dynbulkid="on" + writeoperation="create") + + +.. _omelasticsearch-retry-example: + +Example 5 +--------- + +The following sample shows how to use :ref:`omelasticsearch-retryfailures` to +process, discard, or retry failed operations. This uses +`writeoperation="create"` with a unique `bulkid` so that we can check for and +discard duplicate messages as successful. The `try_es` ruleset is used both +for the initial attempt and any subsequent retries. The code in the ruleset +assumes that if `$.omes!status` is set and is non-zero, this is a retry for a +previously failed operation. If the status was successful, or Elasticsearch +said this was a duplicate, the record is already in Elasticsearch, so we can +drop the record. If there was some error processing the response +e.g. Elasticsearch sent a response formatted in some way that we did not know +how to process, then submit the record to the `error_es` ruleset. If the +response was a "hard" error like `400`, then submit the record to the +`error_es` ruleset. In any other case, such as a status `429` or `5xx`, the +record will be resubmitted to Elasticsearch. In the example, the `error_es` +ruleset just dumps the records to a file. + +.. code-block:: none + + module(load="omelasticsearch") + module(load="omfile") + template(name="bulkid-template" type="list") { property(name="$.es_record_id") } + + ruleset(name="error_es") { + action(type="omfile" template="RSYSLOG_DebugFormat" file="es-bulk-errors.log") + } + + ruleset(name="try_es") { + if strlen($.omes!status) > 0 then { + # retry case + if ($.omes!status == 200) or ($.omes!status == 201) or (($.omes!status == 409) and ($.omes!writeoperation == "create")) then { + stop # successful + } + if ($.omes!writeoperation == "unknown") or (strlen($.omes!error!type) == 0) or (strlen($.omes!error!reason) == 0) then { + call error_es + stop + } + if ($.omes!status == 400) or ($.omes!status < 200) then { + call error_es + stop + } + # else fall through to retry operation + } + if strlen($.omes!_id) > 0 then { + set $.es_record_id = $.omes!_id; + } else { + # NOTE: depends on rsyslog being compiled with --enable-uuid + set $.es_record_id = $uuid; + } + action(type="omelasticsearch" + ... + bulkmode="on" + bulkid="bulkid-template" + dynbulkid="on" + writeoperation="create" + retryfailures="on" + retryruleset="try_es") + } + call try_es diff --git a/source/configuration/modules/omfile.rst b/source/configuration/modules/omfile.rst new file mode 100644 index 0000000..b5d1b22 --- /dev/null +++ b/source/configuration/modules/omfile.rst @@ -0,0 +1,930 @@ +************************** +omfile: File Output Module +************************** + +=========================== =========================================================================== +**Module Name:** **omfile** +**Author:** `Rainer Gerhards <https://rainer.gerhards.net/>`_ <rgerhards@adiscon.com> +=========================== =========================================================================== + + +Purpose +======= + +The omfile plug-in provides the core functionality of writing messages +to files residing inside the local file system (which may actually be +remote if methods like NFS are used). Both files named with static names +as well files with names based on message content are supported by this +module. + + +Notable Features +================ + +- :ref:`omfile-statistic-counter` + + +Configuration Parameters +======================== + +Omfile is a built-in module that does not need to be loaded. In order to +specify module parameters, use + +.. code-block:: none + + module(load="builtin:omfile" ...parameters...) + + +Note that legacy parameters **do not** affect new-style RainerScript configuration +objects. See :doc:`basic configuration structure doc <../basic_structure>` to +learn about different configuration languages in use by rsyslog. + +.. note:: + + Parameter names are case-insensitive. + + +General Notes +------------- + +As can be seen in the parameters below, owner and groups can be set either by +name or by direct id (uid, gid). While using a name is more convenient, using +the id is more robust. There may be some situations where the OS is not able +to do the name-to-id resolution, and these cases the owner information will be +set to the process default. This seems to be uncommon and depends on the +authentication provider and service start order. In general, using names +is fine. + + +Module Parameters +----------------- + +Template +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "RSYSLOG_FileFormat", "no", "``$ActionFileDefaultTemplate``" + +Set the default template to be used if an action is not configured +to use a specific template. + + +DirCreateMode +^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "FileCreateMode", "0700", "no", "``$DirCreateMode``" + +Sets the default dirCreateMode to be used for an action if no +explicit one is specified. + + +FileCreateMode +^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "FileCreateMode", "0644", "no", "``$FileCreateMode``" + +Sets the default fileCreateMode to be used for an action if no +explicit one is specified. + + +fileOwner +^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "UID", "process user", "no", "``$FileOwner``" + +Sets the default fileOwner to be used for an action if no +explicit one is specified. + + +fileOwnerNum +^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "process user", "no", "none" + +Sets the default fileOwnerNum to be used for an action if no +explicit one is specified. + + +fileGroup +^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "GID", "process user's primary group", "no", "``$FileGroup``" + +Sets the default fileGroup to be used for an action if no +explicit one is specified. + + +fileGroupNum +^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "process user's primary group", "no", "none" + +Sets the default fileGroupNum to be used for an action if no +explicit one is specified. + + +dirOwner +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "UID", "process user", "no", "``$DirOwner``" + +Sets the default dirOwner to be used for an action if no +explicit one is specified. + + +dirOwnerNum +^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "process user", "no", "none" + +Sets the default dirOwnerNum to be used for an action if no +explicit one is specified. + + +dirGroup +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "GID", "process user's primary group", "no", "``$DirGroup``" + +Sets the default dirGroup to be used for an action if no +explicit one is specified. + + +dirGroupNum +^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "process user's primary group", "no", "none" + +Sets the default dirGroupNum to be used for an action if no +explicit one is specified. + + +dynafile.donotsuspend +^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "on", "no", "none" + +This permits SUSPENDing dynafile actions. Traditionally, SUSPEND mode was +never entered for dynafiles as it would have blocked overall processing +flow. Default is not to suspend (and thus block). + + +compression.driver +^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "zlib", "no", "none" + +.. versionadded:: 8.2208.0 + +For compressed operation ("zlib mode"), this permits to set the compression +driver to be used. Originally, only zlib was supported and still is the +default. Since 8.2208.0 zstd is also supported. It provides much better +compression ratios and performance, especially with multiple zstd worker +threads enabled. + +Possible values are: +- zlib +- zstd + + +compression.zstd.workers +^^^^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "positive integer", "zlib library default", "no", "none" + +.. versionadded:: 8.2208.0 + +In zstd mode, this enables to configure zstd-internal compression worker threads. +This setting has nothing to do with rsyslog workers. The zstd library provides +an enhanced worker thread pool which permits multithreaed compression of serial +data streams. Rsyslog fully supports this mode for optimal performance. + +Please note that for this parameter to have an effect, the zstd library must +be compiled with multithreading support. As of this writing (2022), this is +**not** the case for many frequently used distros and distro versions. In this +case, you may want to custom install the zstd library with threading enabled. Note +that this does not require a rsyslog rebuild. + + +Action Parameters +----------------- + +Note that **one** of the parameters *file* or *dynaFile* must be specified. This +selects whether a static or dynamic file (name) shall be written to. + + +File +^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "none", "no", "none" + +This creates a static file output, always writing into the same file. +If the file already exists, new data is appended to it. Existing +data is not truncated. If the file does not already exist, it is +created. Files are kept open as long as rsyslogd is active. This +conflicts with external log file rotation. In order to close a file +after rotation, send rsyslogd a HUP signal after the file has been +rotated away. Either file or dynaFile can be used, but not both. If both +are given, dynaFile will be used. + + +dynaFile +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "none", "no", "none" + +For each message, the file name is generated based on the given +template. Then, this file is opened. As with the *file* property, +data is appended if the file already exists. If the file does not +exist, a new file is created. The template given in "templateName" +is just a regular :doc:`rsyslog template <../templates>`, so all +you have full control over how to format the file name. Either file +or dynaFile can be used, but not both. If both are given, dynaFile +will be used. + +A cache of recent files is kept. Note +that this cache can consume quite some memory (especially if large +buffer sizes are used). Files are kept open as long as they stay +inside the cache. +Files are removed from the cache when a HUP signal is sent, the +*closeTimeout* occurs, or the cache runs out of space, in which case +the least recently used entry is evicted. + + +Template +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "template set via module parameter", "no", "``$ActionFileDefaultTemplate``" + +Sets the template to be used for this action. + + +closeTimeout +^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "File: 0 DynaFile: 10", "no", "none" + +.. versionadded:: 8.3.3 + +Specifies after how many minutes of inactivity a file is +automatically closed. Note that this functionality is implemented +based on the +:doc:`janitor process <../../concepts/janitor>`. +See its doc to understand why and how janitor-based times are +approximate. + + +dynaFileCacheSize +^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "10", "no", "``$DynaFileCacheSize``" + +This parameter specifies the maximum size of the cache for +dynamically-generated file names (dynafile= parmeter). +This setting specifies how many open file handles should +be cached. If, for example, the file name is generated with the hostname +in it and you have 100 different hosts, a cache size of 100 would ensure +that files are opened once and then stay open. This can be a great way +to increase performance. If the cache size is lower than the number of +different files, the least recently used one is discarded (and the file +closed). + +Note that this is a per-action value, so if you have +multiple dynafile actions, each of them have their individual caches +(which means the numbers sum up). Ideally, the cache size exactly +matches the need. You can use :doc:`impstats <impstats>` to tune +this value. Note that a too-low cache size can be a very considerable +performance bottleneck. + + +zipLevel +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "0", "no", "``$OMFileZipLevel``" + +If greater than 0, turns on gzip compression of the output file. The +higher the number, the better the compression, but also the more CPU +is required for zipping. + + +veryRobustZip +^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +.. versionadded:: 7.3.0 + +If *zipLevel* is greater than 0, +then this setting controls if extra headers are written to make the +resulting file extra hardened against malfunction. If set to off, +data appended to previously unclean closed files may not be +accessible without extra tools (like `gztool <https://github.com/circulosmeos/gztool>`_ with: ``gztool -p``). +Note that this risk is usually +expected to be bearable, and thus "off" is the default mode. The +extra headers considerably degrade compression, files with this +option set to "on" may be four to five times as large as files +processed in "off" mode. + +**In order to avoid this degradation in compression** both +*flushOnTXEnd* and *asyncWriting* parameters must be set to "off" +and also *ioBufferSize* must be raised from default "4k" value to +at least "32k". This way a reasonable compression factor is +maintained, similar to a non-blocked gzip file: + +.. code-block:: none + + veryRobustZip="on" ioBufferSize="64k" flushOnTXEnd="off" asyncWriting="off" + + +Do not forget to add your desired *zipLevel* to this configuration line. + + +flushInterval +^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "1", "no", "``$OMFileFlushInterval``" + +Defines, in seconds, the interval after which unwritten data is +flushed. + + +asyncWriting +^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "``$OMFileASyncWriting``" + +If turned on, the files will be written in asynchronous mode via a +separate thread. In that case, double buffers will be used so that +one buffer can be filled while the other buffer is being written. +Note that in order to enable FlushInterval, AsyncWriting must be set +to "on". Otherwise, the flush interval will be ignored. + + +flushOnTXEnd +^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "on", "no", "``$OMFileFlushOnTXEnd``" + +Omfile has the capability to write output using a buffered writer. +Disk writes are only done when the buffer is full. So if an error +happens during that write, data is potentially lost. Bear in mind that +the buffer may become full only after several hours or a rsyslog +shutdown (however a buffer flush can still be forced by sending rsyslogd +a HUP signal). In cases where this is unacceptable, set FlushOnTXEnd +to "on". Then, data is written at the end of each transaction +(for pre-v5 this means after each log message) and the usual error +recovery thus can handle write errors without data loss. +Note that this option severely reduces the effect of zip compression +and should be switched to "off" for that use case. +Also note that the default -on- is primarily an aid to preserve the +traditional syslogd behaviour. + +If you are using dynamic file names (dynafiles), flushes can actually +happen more frequently. In this case, a flush can also happen when +the file name changes within a transaction. + + +ioBufferSize +^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "size", "4 KiB", "no", "``$OMFileIOBufferSize``" + +Size of the buffer used to writing output data. The larger the +buffer, the potentially better performance is. The default of 4k is +quite conservative, it is useful to go up to 64k, and 128k if you +used gzip compression (then, even higher sizes may make sense) + + +dirOwner +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "UID", "system default", "no", "``$DirOwner``" + +Set the file owner for directories newly created. Please note that +this setting does not affect the owner of directories already +existing. The parameter is a user name, for which the userid is +obtained by rsyslogd during startup processing. Interim changes to +the user mapping are not detected. + + +dirOwnerNum +^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "system default", "no", "``$DirOwnerNum``" + +.. versionadded:: 7.5.8 + +Set the file owner for directories newly created. Please note that +this setting does not affect the owner of directories already +existing. The parameter is a numerical ID, which is used regardless +of whether the user actually exists. This can be useful if the user +mapping is not available to rsyslog during startup. + + +dirGroup +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "GID", "system default", "no", "``$DirGroup``" + +Set the group for directories newly created. Please note that this +setting does not affect the group of directories already existing. +The parameter is a group name, for which the groupid is obtained by +rsyslogd on during startup processing. Interim changes to the user +mapping are not detected. + + +dirGroupNum +^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "system default", "no", "``$DirGroupNum``" + +Set the group for directories newly created. Please note that this +setting does not affect the group of directories already existing. +The parameter is a numerical ID, which is used regardless of whether +the group actually exists. This can be useful if the group mapping is +not available to rsyslog during startup. + + +fileOwner +^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "UID", "system default", "no", "``$FileOwner``" + +Set the file owner for files newly created. Please note that this +setting does not affect the owner of files already existing. The +parameter is a user name, for which the userid is obtained by +rsyslogd during startup processing. Interim changes to the user +mapping are *not* detected. + + +fileOwnerNum +^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "system default", "no", "``$FileOwnerNum``" + +.. versionadded:: 7.5.8 + +Set the file owner for files newly created. Please note that this +setting does not affect the owner of files already existing. The +parameter is a numerical ID, which which is used regardless of +whether the user actually exists. This can be useful if the user +mapping is not available to rsyslog during startup. + + +fileGroup +^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "GID", "system default", "no", "``$FileGroup``" + +Set the group for files newly created. Please note that this setting +does not affect the group of files already existing. The parameter is +a group name, for which the groupid is obtained by rsyslogd during +startup processing. Interim changes to the user mapping are not +detected. + + +fileGroupNum +^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "system default", "no", "``$FileGroupNum``" + +.. versionadded:: 7.5.8 + +Set the group for files newly created. Please note that this setting +does not affect the group of files already existing. The parameter is +a numerical ID, which is used regardless of whether the group +actually exists. This can be useful if the group mapping is not +available to rsyslog during startup. + + +fileCreateMode +^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "equally-named module parameter", "no", "``$FileCreateMode``" + +The FileCreateMode directive allows to specify the creation mode +with which rsyslogd creates new files. If not specified, the value +0644 is used (which retains backward-compatibility with earlier +releases). The value given must always be a 4-digit octal number, +with the initial digit being zero. +Please note that the actual permission depend on rsyslogd's process +umask. If in doubt, use "$umask 0000" right at the beginning of the +configuration file to remove any restrictions. + + +dirCreateMode +^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "equally-named module parameter", "no", "``$DirCreateMode``" + +This is the same as FileCreateMode, but for directories +automatically generated. + + +failOnChOwnFailure +^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "on", "no", "``$FailOnCHOwnFailure``" + +This option modifies behaviour of file creation. If different owners +or groups are specified for new files or directories and rsyslogd +fails to set these new owners or groups, it will log an error and NOT +write to the file in question if that option is set to "on". If it is +set to "off", the error will be ignored and processing continues. +Keep in mind, that the files in this case may be (in)accessible by +people who should not have permission. The default is "on". + + +createDirs +^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "on", "no", "``$CreateDirs``" + +Create directories on an as-needed basis + + +sync +^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "``$ActionFileEnableSync``" + +Enables file syncing capability of omfile. + +When enabled, rsyslog does a sync to the data file as well as the +directory it resides after processing each batch. There currently +is no way to sync only after each n-th batch. + +Enabling sync causes a severe performance hit. Actually, +it slows omfile so much down, that the probability of loosing messages +**increases**. In short, +you should enable syncing only if you know exactly what you do, and +fully understand how the rest of the engine works, and have tuned +the rest of the engine to lossless operations. + + +sig.Provider +^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "no signature provider", "no", "none" + +Selects a signature provider for log signing. By selecting a provider, +the signature feature is turned on. + +Currently there is one signature provider available: ":doc:`ksi_ls12 <sigprov_ksi12>`". + +Previous signature providers ":doc:`gt <sigprov_gt>`" and ":doc:`ksi <sigprov_ksi>`" are deprecated. + + +cry.Provider +^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "no crypto provider", "no", "none" + +Selects a crypto provider for log encryption. By selecting a provider, +the encryption feature is turned on. + +Currently, there only is one provider called ":doc:`gcry <../cryprov_gcry>`". + + +rotation.sizeLimit +^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "size", "0 (disabled)", "no", "`$outchannel` (partly)" + +This permits to set a size limit on the output file. When the limit is reached, +rotation of the file is tried. The rotation script needs to be configured via +`rotation.sizeLimitCommand`. + +Please note that the size limit is not exact. Some excess bytes are permitted +to prevent messages from being split across two files. Also, a full batch of +messages is not terminated in between. As such, in practice, the size of the +output file can grow some KiB larger than configured. + +Also avoid to configer a too-low limit, especially for busy files. Calling the +rotation script is relatively performance intense. As such, it could negatively +affect overall rsyslog performance. + + +rotation.sizeLimitCommand +^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "(empty)", "no", "`$outchannel` (partly)" + +This permits to configure the script to be called whe a size limit on the output +file is reached. The actual size limit needs to be configured via +`rotation.sizeLimit`. + + +.. _omfile-statistic-counter: + +Statistic Counter +================= + +This plugin maintains :doc:`statistics <../rsyslog_statistic_counter>` for each +dynafile cache. Dynafile cache performance is critical for overall system performance, +so reviewing these counters on a busy system (especially one experiencing performance +problems) is advisable. The statistic is named "dynafile cache", followed by the +template name used for this dynafile action. + +The following properties are maintained for each dynafile: + +- **request** - total number of requests made to obtain a dynafile + +- **level0** - requests for the current active file, so no real cache + lookup needed to be done. These are extremely good. + +- **missed** - cache misses, where the required file did not reside in + cache. Even with a perfect cache, there will be at least one miss per + file. That happens when the file is being accessed for the first time + and brought into cache. So "missed" will always be at least as large + as the number of different files processed. + +- **evicted** - the number of times a file needed to be evicted from + the cache as it ran out of space. These can simply happen when + date-based files are used, and the previous date files are being + removed from the cache as time progresses. It is better, though, to + set an appropriate "closeTimeout" (counter described below), so that + files are removed from the cache after they become no longer accessed. + It is bad if active files need to be evicted from the cache. This is a + very costly operation as an evict requires to close the file (thus a + full flush, no matter of its buffer state) and a later access requires + a re-open – and the eviction of another file, as the cache obviously has + run out of free entries. If this happens frequently, it can severely + affect performance. So a high eviction rate is a sign that the dynafile + cache size should be increased. If it is already very high, it is + recommended to re-think about the design of the file store, at least if + the eviction process causes real performance problems. + +- **maxused** - the maximum number of cache entries ever used. This can + be used to trim the cache down to a value that’s actually useful but + does not waste resources. Note that when date-based files are used and + rsyslog is run for an extended period of time, the cache gradually fills + up to the max configured value as older files are migrated out of it. + This will make "maxused" questionable after some time. Frequently enough + purging the cache can prevent this (usually, once a day is sufficient). + +- **closetimeouts** - available since 8.3.3 – tells how often a file was + closed due to timeout settings ("closeTimeout" action parameter). These + are cases where dynafiles or static files have been closed by rsyslog due + to inactivity. Note that if no "closeTimeout" is specified for the action, + this counter always is zero. A high or low number in itself doesn’t mean + anything good or bad. It totally depends on the use case, so no general + advise can be given. + + +Caveats/Known Bugs +================== + +- people often report problems that dynafiles are not properly created. + The common cause for this problem is SELinux rules, which do not permit + the create of those files (check generated file names and paths!). The + same happens for generic permission issues (this is often a problem + under Ubuntu where permissions are dropped by default) + +- One needs to be careful with log rotation if signatures and/or + encryption are being used. These create side-files, which form a set + and must be kept together. + For signatures, the ".sigstate" file must NOT be rotated away if + signature chains are to be build across multiple files. This is + because .sigstate contains just global information for the whole file + set. However, all other files need to be rotated together. The proper + sequence is to + + #. move all files inside the file set + #. only AFTER this is completely done, HUP rsyslog + + This sequence will ensure that all files inside the set are + atomically closed and in sync. HUPing only after a subset of files + have been moved results in inconsistencies and will most probably + render the file set unusable. + +- If ``zipLevel`` is greater than 0 and ``veryRobustZip`` is set to off, + data appended to previously unclean closed files will not be + accessible with ``gunzip`` if rsyslog writes again in the same + file. Nonetheless, data is still there and can be correctly accessed + with other tools like `gztool <https://github.com/circulosmeos/gztool>`_ (v>=1.1) with: ``gztool -p``. + + +Examples +======== + +Example 1 +--------- + +The following command writes all syslog messages into a file. + +.. code-block:: none + + action(type="omfile" dirCreateMode="0700" FileCreateMode="0644" + File="/var/log/messages") + + diff --git a/source/configuration/modules/omfwd.rst b/source/configuration/modules/omfwd.rst new file mode 100644 index 0000000..06adcd0 --- /dev/null +++ b/source/configuration/modules/omfwd.rst @@ -0,0 +1,795 @@ +************************************** +omfwd: syslog Forwarding Output Module +************************************** + +=========================== =========================================================================== +**Module Name:** **omfwd** +**Author:** `Rainer Gerhards <https://rainer.gerhards.net/>`_ <rgerhards@adiscon.com> +=========================== =========================================================================== + + +Purpose +======= + +The omfwd plug-in provides the core functionality of traditional message +forwarding via UDP and plain TCP. It is a built-in module that does not +need to be loaded. + + +Notable Features +================ + + +Configuration Parameters +======================== + +.. note:: + + Parameter names are case-insensitive. + +Module Parameters +----------------- + +Template +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "RSYSLOG_TraditionalForwardFormat", "no", "``$ActionForwardDefaultTemplateName``" + +Sets a non-standard default template for this module. + + +Action Parameters +----------------- + +Target +^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +Name or IP-Address of the system that shall receive messages. Any +resolvable name is fine. + + +Port +^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "514", "no", "none" + +Name or numerical value of port to use when connecting to target. + + +Protocol +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "udp", "no", "none" + +Type of protocol to use for forwarding. Note that \`\`tcp'' means +both legacy plain tcp syslog as well as RFC5425-based TLS-encrypted +syslog. Which one is selected depends on the StreamDriver parameter. +If StreamDriver is set to "ossl" or "gtls" it will use TLS-encrypted syslog. + + +NetworkNamespace +^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +Name of a network namespace as in /var/run/netns/ to use for forwarding. + +If the setns() system call is not available on the system (e.g. BSD +kernel, linux kernel before v2.6.24) the given namespace will be +ignored. + +Address +^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +.. versionadded:: 8.35.0 + +Bind socket to a given local IP address. This option is only supported +for UDP, not TCP. + +IpFreeBind +^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "2", "no", "none" + +.. versionadded:: 8.35.0 + +Manages the IP_FREEBIND option on the UDP socket, which allows binding it to +an IP address that is not yet associated to any network interface. This option +is only relevant if the address option is set. + +The parameter accepts the following values: + +- 0 - does not enable the IP_FREEBIND option on the + UDP socket. If the *bind()* call fails because of *EADDRNOTAVAIL* error, + socket initialization fails. + +- 1 - silently enables the IP_FREEBIND socket + option if it is required to successfully bind the socket to a nonlocal address. + +- 2 - enables the IP_FREEBIND socket option and + warns when it is used to successfully bind the socket to a nonlocal address. + +Device +^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +Bind socket to given device (e.g., eth0) + +For Linux with VRF support, the Device option can be used to specify the +VRF for the Target address. + + +TCP_Framing +^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "traditional", "no", "none" + +Framing-Mode to be used for forwarding, either "traditional" or +"octet-counted". This affects only TCP-based protocols, it is ignored for UDP. +In protocol engineering, "framing" means how multiple messages over the same +connection are separated. Usually, this is transparent to users. Unfortunately, +the early syslog protocol evolved and so there are cases where users need to +specify the framing. The "traditional" framing is nontransparent. With it, +messages end when an LF (aka "line break", "return") is encountered, and the +next message starts immediately after the LF. If multi-line messages are +received, these are essentially broken up into multiple message, usually with +all but the first message segment being incorrectly formatted. The +"octet-counted" framing solves this issue. With it, each message is prefixed +with the actual message length, so that a receiver knows exactly where the +message ends. Multi-line messages cause no problem here. This mode is very +close to the method described in RFC5425 for TLS-enabled syslog. Unfortunately, +only few syslogd implementations support "octet-counted" framing. As such, the +"traditional" framing is set as default, even though it has defects. If it is +known that the receiver supports "octet-counted" framing, it is suggested to +use that framing mode. + + +TCP_FrameDelimiter +^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "10", "no", "none" + +Sets a custom frame delimiter for TCP transmission when running TCP\_Framing +in "traditional" mode. The delimiter has to be a number between 0 and 255 +(representing the ASCII-code of said character). The default value for this +parameter is 10, representing a '\\n'. When using Graylog, the parameter +must be set to 0. + + +ZipLevel +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "0", "no", "none" + +Compression level for messages. + +Up until rsyslog 7.5.1, this was the only compression setting that +rsyslog understood. Starting with 7.5.1, we have different +compression modes. All of them are affected by the ziplevel. If, +however, no mode is explicitly set, setting ziplevel also turns on +"single" compression mode, so pre 7.5.1 configuration will continue +to work as expected. + +The compression level is specified via the usual factor of 0 to 9, +with 9 being the strongest compression (taking up most processing +time) and 0 being no compression at all (taking up no extra +processing time). + + +compression.Mode +^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +*mode* is one of "none", "single", or "stream:always". The default +is "none", in which no compression happens at all. +In "single" compression mode, Rsyslog implements a proprietary +capability to zip transmitted messages. That compression happens on a +message-per-message basis. As such, there is a performance gain only +for larger messages. Before compressing a message, rsyslog checks if +there is some gain by compression. If so, the message is sent +compressed. If not, it is sent uncompressed. As such, it is totally +valid that compressed and uncompressed messages are intermixed within +a conversation. + +In "stream:always" compression mode the full stream is being +compressed. This also uses non-standard protocol and is compatible +only with receives that have the same abilities. This mode offers +potentially very high compression ratios. With typical syslog +messages, it can be as high as 95+% compression (so only one +twentieth of data is actually transmitted!). Note that this mode +introduces extra latency, as data is only sent when the compressor +emits new compressed data. For typical syslog messages, this can mean +that some hundred messages may be held in local buffers before they +are actually sent. This mode has been introduced in 7.5.1. + +**Note: currently only imptcp supports receiving stream-compressed +data.** + + +compression.stream.flushOnTXEnd +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "on", "no", "none" + +.. versionadded:: 7.5.3 + +This setting affects stream compression mode, only. If enabled (the +default), the compression buffer will by emptied at the end of a +rsyslog batch. If set to "off", end of batch will not affect +compression at all. + +While setting it to "off" can potentially greatly improve +compression ratio, it will also introduce severe delay between when a +message is being processed by rsyslog and actually sent out to the +network. We have seen cases where for several thousand message not a +single byte was sent. This is good in the sense that it can happen +only if we have a great compression ratio. This is most probably a +very good mode for busy machines which will process several thousand +messages per second and the resulting short delay will not pose any +problems. However, the default is more conservative, while it works +more "naturally" with even low message traffic. Even in flush mode, +notable compression should be achievable (but we do not yet have +practice reports on actual compression ratios). + + +RebindInterval +^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "0", "no", "``$ActionSendTCPRebindInterval`` or ``$ActionSendUDPRebindInterval``" + +Permits to specify an interval at which the current connection is +broken and re-established. This setting is primarily an aid to load +balancers. After the configured number of batches (equals roughly to +messages for UDP traffic, dependent on batch size for TCP) has been +transmitted, the current connection is terminated and a new one +started. Note that this setting applies to both TCP and UDP traffic. +For UDP, the new \`\`connection'' uses a different source port (ports +are cycled and not reused too frequently). This usually is perceived +as a \`\`new connection'' by load balancers, which in turn forward +messages to another physical target system. + + +KeepAlive +^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +Enable or disable keep-alive packets at the tcp socket layer. The +default is to disable them. + + +KeepAlive.Probes +^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "0", "no", "none" + +The number of unacknowledged probes to send before considering the +connection dead and notifying the application layer. The default, 0, +means that the operating system defaults are used. This has only +effect if keep-alive is enabled. The functionality may not be +available on all platforms. + + +KeepAlive.Interval +^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "0", "no", "none" + +The interval between subsequential keepalive probes, regardless of +what the connection has exchanged in the meantime. The default, 0, +means that the operating system defaults are used. This has only +effect if keep-alive is enabled. The functionality may not be +available on all platforms. + + +KeepAlive.Time +^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "0", "no", "none" + +The interval between the last data packet sent (simple ACKs are not +considered data) and the first keepalive probe; after the connection +is marked to need keepalive, this counter is not used any further. +The default, 0, means that the operating system defaults are used. +This has only effect if keep-alive is enabled. The functionality may +not be available on all platforms. + +ConErrSkip +^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "0", "no", "none" + +The ConErrSkip can be used to limit the number of network errors +recorded in logs. For example, value 10 means that each 10th error +message is logged. Note that this options should be used as the last +resort since the necessity of its use indicates network issues. +The default behavior is that all network errors are logged. + +RateLimit.Interval +^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "max", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "0", "", "no", "none" + +Specifies the rate-limiting interval in seconds. Default value is 0, +which turns off rate limiting. + +RateLimit.Burst +^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "max", "mandatory", "none" + :widths: auto + :class: parameter-table + + "integer", "200", "(2^32)-1", "no", "none" + +Specifies the rate-limiting burst in number of messages. + + +StreamDriver +^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "``$ActionSendStreamDriver``" + +Choose the stream driver to be used. Default is plain tcp, but +you can also choose "ossl" or "gtls" for TLS encryption. + + +StreamDriverMode +^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "0", "no", "``$ActionSendStreamDriverMode``" + +Mode to use with the stream driver (driver-specific) + + +StreamDriverAuthMode +^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "none", "no", "``$ActionSendStreamDriverAuthMode``" + +Authentication mode to use with the stream driver. Note that this +parameter requires TLS netstream drivers. For all others, it will be +ignored. (driver-specific). + + +StreamDriver.PermitExpiredCerts +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "warn", "no", "none" + +Controls how expired certificates will be handled when stream driver is in TLS mode. +It can have one of the following values: + +- on = Expired certificates are allowed + +- off = Expired certificates are not allowed (Default, changed from warn to off since Version 8.2012.0) + +- warn = Expired certificates are allowed but warning will be logged + + +StreamDriverPermittedPeers +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "``$ActionSendStreamDriverPermittedPeers``" + +Accepted fingerprint (SHA1) or name of remote peer. Note that this +parameter requires TLS netstream drivers. For all others, it will be +ignored. (driver-specific) + + +StreamDriver.CheckExtendedKeyPurpose +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +Whether to check also purpose value in extended fields part of certificate +for compatibility with rsyslog operation. (driver-specific) + + +StreamDriver.PrioritizeSAN +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +Whether to use stricter SAN/CN matching. (driver-specific) + + +StreamDriver.TlsVerifyDepth +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "TLS library default", "no", "none" + + +Specifies the allowed maximum depth for the certificate chain verification. +Support added in v8.2001.0, supported by GTLS and OpenSSL driver. +If not set, the API default will be used. +For OpenSSL, the default is 100 - see the doc for more: +https://www.openssl.org/docs/man1.1.1/man3/SSL_set_verify_depth.html +For GnuTLS, the default is 5 - see the doc for more: +https://www.gnutls.org/manual/gnutls.html + +StreamDriver.CAFile +^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "global() default", "no", "none" + +.. versionadded:: 8.2108.0 + +This permits to override the CA file set via `global()` config object at the +per-action basis. This parameter is ignored if the netstream driver and/or its +mode does not need or support certificates. + +StreamDriver.CRLFile +^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "optional", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "global() default", "no", "none" + +.. versionadded:: 8.2308.0 + +This permits to override the CRL (Certificate revocation list) file set via `global()` config +object at the per-action basis. This parameter is ignored if the netstream driver and/or its +mode does not need or support certificates. + +StreamDriver.KeyFile +^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "global() default", "no", "none" + +.. versionadded:: 8.2108.0 + +This permits to override the CA file set via `global()` config object at the +per-action basis. This parameter is ignored if the netstream driver and/or its +mode does not need or support certificates. + +StreamDriver.CertFile +^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "global() default", "no", "none" + +.. versionadded:: 8.2108.0 + +This permits to override the CA file set via `global()` config object at the +per-action basis. This parameter is ignored if the netstream driver and/or its +mode does not need or support certificates. + + +ResendLastMSGOnReconnect +^^^^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "``$ActionSendResendLastMsgOnReconnect``" + +Permits to resend the last message when a connection is reconnected. +This setting affects TCP-based syslog, only. It is most useful for +traditional, plain TCP syslog. Using this protocol, it is not always +possible to know which messages were successfully transmitted to the +receiver when a connection breaks. In many cases, the last message +sent is lost. By switching this setting to "yes", rsyslog will always +retransmit the last message when a connection is reestablished. This +reduces potential message loss, but comes at the price that some +messages may be duplicated (what usually is more acceptable). + +Please note that busy systems probably loose more than a +single message in such cases. This is caused by an +`inherant unreliability in plain tcp syslog +<https://rainer.gerhards.net/2008/04/on-unreliability-of-plain-tcp-syslog.html>`_ +and there is no way rsyslog could prevent this from happening +(if you read the detail description, be sure to follow the link +to the follow-up posting). In order to prevent these problems, +we recommend the use of :doc:`omrelp <omrelp>`. + + +udp.SendToAll +^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +When sending UDP messages, there are potentially multiple paths to +the target destination. By default, rsyslogd +only sends to the first target it can successfully send to. If this +option is set to "on", messages are sent to all targets. This may improve +reliability, but may also cause message duplication. This option +should be enabled only if it is fully understood. + +Note: this option replaces the former -A command line option. In +contrast to the -A option, this option must be set once per +input() definition. + + +udp.SendDelay +^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "0", "no", "none" + +.. versionadded:: 8.7.0 + +This is an **expert option**, do only use it if you know very well +why you are using it! + +This options permits to introduce a small delay after *each* send +operation. The integer specifies the delay in microseconds. This +option can be used in cases where too-quick sending of UDP messages +causes message loss (UDP is permitted to drop packets if e.g. a device +runs out of buffers). Usually, you do not want this delay. The parameter +was introduced in order to support some testbench tests. Be sure +to think twice before you use it in production. + + +gnutlsPriorityString +^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "none", "no", "none" + +.. versionadded:: 8.29.0 + +This strings setting is used to configure driver specific properties. +Historically, the setting was only meant for gnutls driver. However +with version v8.1905.0 and higher, the setting can also be used to set openssl configuration commands. + +For GNUTls, the setting specifies the TLS session's handshake algorithms and +options. These strings are intended as a user-specified override of the library +defaults. If this parameter is NULL, the default settings are used. More +information about priority Strings +`here <https://gnutls.org/manual/html_node/Priority-Strings.html>`_. + +For OpenSSL, the setting can be used to pass configuration commands to openssl library. +OpenSSL Version 1.0.2 or higher is required for this feature. +A list of possible commands and their valid values can be found in the documentation: +https://www.openssl.org/docs/man1.0.2/man3/SSL_CONF_cmd.html + +The setting can be single or multiline, each configuration command is separated by linefeed (\n). +Command and value are separated by equal sign (=). Here are a few samples: + +Example 1 +--------- + +This will allow all protocols except for SSLv2 and SSLv3: + +.. code-block:: none + + gnutlsPriorityString="Protocol=ALL,-SSLv2,-SSLv3" + + +Example 2 +--------- + +This will allow all protocols except for SSLv2, SSLv3 and TLSv1. +It will also set the minimum protocol to TLSv1.2 + +.. code-block:: none + + gnutlsPriorityString="Protocol=ALL,-SSLv2,-SSLv3,-TLSv1 + MinProtocol=TLSv1.2" + + +Statistic Counter +================= + +This plugin maintains :doc:`statistics <../rsyslog_statistic_counter>` for each forwarding action. +The statistic is named "target-port-protocol" where "target", "port", and +"protocol" are the respective configuration parameters. So an actual name might be +"192.0.2.1-514-TCP" or "example.net-10514-UDP". + +The following properties are maintained for each action: + +- **bytes.sent** - total number of bytes sent to the network + +See Also +======== + +- `Encrypted Disk + Queues <http://www.rsyslog.com/encrypted-disk-queues/>`_ + + +Examples +======== + +Example 1 +--------- + +The following command sends all syslog messages to a remote server via +TCP port 10514. + +.. code-block:: none + + action(type="omfwd" Target="192.168.2.11" Port="10514" Protocol="tcp" Device="eth0") + + +Example 2 +--------- + +In case the system in use has multiple (maybe virtual) network interfaces network +namespaces come in handy, each with its own routing table. To be able to distribute +syslogs to remote servers in different namespaces specify them as separate actions. + +.. code-block:: none + + action(type="omfwd" Target="192.168.1.13" Port="10514" Protocol="tcp" NetworkNamespace="ns_eth0.0") + action(type="omfwd" Target="192.168.2.24" Port="10514" Protocol="tcp" NetworkNamespace="ns_eth0.1") + action(type="omfwd" Target="192.168.3.38" Port="10514" Protocol="tcp" NetworkNamespace="ns_eth0.2") diff --git a/source/configuration/modules/omhdfs.rst b/source/configuration/modules/omhdfs.rst new file mode 100644 index 0000000..ccdf114 --- /dev/null +++ b/source/configuration/modules/omhdfs.rst @@ -0,0 +1,114 @@ +*************************************** +omhdfs: Hadoop Filesystem Output Module +*************************************** + +=========================== =========================================================================== +**Module Name:** **omhdfs** +**Author:** `Rainer Gerhards <https://rainer.gerhards.net/>`_ <rgerhards@adiscon.com> +=========================== =========================================================================== + + +Purpose +======= + +This module supports writing message into files on Hadoop's HDFS file +system. + + +Configuration Parameters +======================== + +.. note:: + + Parameter names are case-insensitive. + + +|FmtObsoleteName| Directives +---------------------------- + +.. csv-table:: + :header: "|FmtObsoleteName| directive", "type", "default", "mandatory" + :widths: auto + :class: parameter-table + + "``$OMHDFSFileName``", "word", "none", "no" + +The name of the file to which the output data shall be written. + +.. csv-table:: + :header: "|FmtObsoleteName| directive", "type", "default", "mandatory" + :widths: auto + :class: parameter-table + + "``$OMHDFSHost``", "word", "default", "no" + +Name or IP address of the HDFS host to connect to. + +.. csv-table:: + :header: "|FmtObsoleteName| directive", "type", "default", "mandatory" + :widths: auto + :class: parameter-table + + "``$OMHDFSPort``", "integer", "0", "no" + +Port on which to connect to the HDFS host. + +.. csv-table:: + :header: "|FmtObsoleteName| directive", "type", "default", "mandatory" + :widths: auto + :class: parameter-table + + "``$OMHDFSDefaultTemplate``", "word", "RSYSLOG_FileFormat", "no" + +Default template to be used when none is specified. This saves the work of +specifying the same template ever and ever again. Of course, the default +template can be overwritten via the usual method. + + +Caveats/Known Bugs +================== + +Building omhdfs is a challenge because we could not yet find out how to +integrate Java properly into the autotools build process. The issue is +that HDFS is written in Java and libhdfs uses JNI to talk to it. That +requires that various system-specific environment options and paths be +set correctly. At this point, we leave this to the user. If someone knows +how to do it better, please drop us a line! + +- In order to build, you need to set these environment variables BEFORE + running ./configure: + + - JAVA\_INCLUDES - must have all include paths that are needed to + build JNI C programs, including the -I options necessary for gcc. + An example is + # export + JAVA\_INCLUDES="-I/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86\_64/include + -I/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86\_64/include/linux" + - JAVA\_LIBS - must have all library paths that are needed to build + JNI C programs, including the -l/-L options necessary for gcc. An + example is + # export export + JAVA\_LIBS="-L/usr/java/jdk1.6.0\_21/jre/lib/amd64 + -L/usr/java/jdk1.6.0\_21/jre/lib/amd64/server -ljava -ljvm + -lverify" + +- As of HDFS architecture, you must make sure that all relevant + environment variables (the usual Java stuff and HADOOP's home + directory) are properly set. +- As it looks, libhdfs makes Java throw exceptions to stdout. There is + no known work-around for this (and it usually should not case any + troubles. + + +Examples +======== + +Example 1 +--------- + +.. code-block:: none + + $ModLoad omhdfs + $OMHDFSFileName /var/log/logfile \*.\* :omhdfs: + + diff --git a/source/configuration/modules/omhiredis.rst b/source/configuration/modules/omhiredis.rst new file mode 100644 index 0000000..fb1a98c --- /dev/null +++ b/source/configuration/modules/omhiredis.rst @@ -0,0 +1,779 @@ +****************************** +omhiredis: Redis Output Module +****************************** + +=========================== =========================================================================== +**Module Name:** **omhiredis** +**Author:** Brian Knox <bknox@digitalocean.com> +**Contributors:** Theo Bertin <theo.bertin@advens.fr> +=========================== =========================================================================== + + +Purpose +======= + +This module provides native support for writing to Redis, +using the hiredis client library. + + +Configuration Parameters +======================== + +.. note:: + + Parameter names are case-insensitive. + + +Action Parameters +----------------- + +Server +^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +Name or address of the Redis server + + +ServerPort +^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "6379", "no", "none" + +Port of the Redis server if the server is not listening on the default port. + + +ServerPassword +^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +Password to support authenticated redis database server to push messages +across networks and datacenters. Parameter is optional if not provided +AUTH command wont be sent to the server. + + +Mode +^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "template", "no", "none" + +Mode to run the output action in: "queue", "publish", "set" or "stream" If not supplied, the +original "template" mode is used. + +.. note:: + + Due to a config parsing bug in 8.13, explicitly setting this to "template" mode will result in a config parsing + error. + + If mode is "set", omhiredis will send SET commands. If "expiration" parameter is provided (see parameter below), + omhiredis will send SETEX commands. + + If mode is "stream", logs will be sent using XADD. In that case, the template-formatted message will be inserted in + the **msg** field of the stream ID (this behaviour can be controlled through the :ref:`omhiredis_streamoutfield` parameter) + +.. _omhiredis_template: + +Template +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "RSYSLOG_ForwardFormat", "no", "none" + +.. Warning:: + Template is required if using "template" mode. + + +Key +^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +Key is required if using "publish", "queue" or "set" modes. + + +Dynakey +^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +If set to "on", the key value will be considered a template by Rsyslog. +Useful when dynamic key generation is desired. + +Userpush +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +If set to on, use RPUSH instead of LPUSH, if not set or off, use LPUSH. + + +Expiration +^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "number", "0", "no", "none" + +Only applicable with mode "set". Specifies an expiration for the keys set by omhiredis. +If this parameter is not specified, the value will be 0 so keys will last forever, otherwise they will last for X +seconds. + +.. _omhiredis_streamoutfield: + +stream.outField +^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "msg", "no", "none" + +| Only applicable with mode "stream". +| The stream ID's field to use to insert the generated log. + +.. note:: + Currently, the module cannot use the full message object, so it can only insert templated messages to a single stream entry's specific field + + +.. _omhiredis_streamcapacitylimit: + +stream.capacityLimit +^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "positive integer", "0", "no", "none" + +| Only applicable with mode "stream". +| If set to a value greater than 0 (zero), the XADD will add a `MAXLEN <https://redis.io/docs/data-types/streams-tutorial/#capped-streams>`_ option with **approximate** trimming, limiting the amount of stored entries in the stream at each insertion. + +.. Warning:: + This parameter has no way to check if the deleted entries have been ACK'ed once or even used, this should be set if you're sure the insertion rate in lower that the dequeuing to avoid losing entries! + + +.. _omhiredis_streamack: + +stream.ack +^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "boolean", "off", "no", "none" + +| Only applicable with mode "stream". +| If set, the module will send an acknowledgement to Redis, for the stream defined by :ref:`omhiredis_streamkeyack`, with the group defined by :ref:`omhiredis_streamgroupack` and the ID defined by :ref:`omhiredis_streamindexack`. +| This is especially useful when used with the :ref:`imhiredis_stream_consumerack` deactivated, as it allows omhiredis to acknowledge the correct processing of the log once the job is effectively done. + + +.. _omhiredis_streamdel: + +stream.del +^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "boolean", "off", "no", "none" + +| Only applicable with mode "stream". +| If set, the module will send a XDEL command to remove an entry, for the stream defined by :ref:`omhiredis_streamkeyack`, and the ID defined by :ref:`omhiredis_streamindexack`. +| This can be useful to automatically remove processed entries extracted on a previous stream by imhiredis. + + +.. _omhiredis_streamkeyack: + +stream.keyAck +^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "", "no", "none" + +| Only applicable with mode "stream". +| Is **required**, if one of :ref:`omhiredis_streamack` or :ref:`omhiredis_streamdel` are **on**. +| Defines the value to use for acknowledging/deleting while inserting a new entry, can be either a constant value or a template name if :ref:`omhiredis_streamdynakeyack` is set. +| This can be useful to automatically acknowledge/remove processed entries extracted on a previous stream by imhiredis. + + +.. _omhiredis_streamdynakeyack: + +stream.dynaKeyAck +^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "boolean", "off", "no", "none" + +| Only applicable with mode "stream". +| If set to **on**, the value of :ref:`omhiredis_streamkeyack` is taken as an existing Rsyslog template. + + + +.. _omhiredis_streamgroupack: + +stream.groupAck +^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "", "no", "none" + +| Only applicable with mode "stream". +| Is **required**, if :ref:`omhiredis_streamack` is **on**. +| Defines the value to use for acknowledging while inserting a new entry, can be either a constant value or a template name if :ref:`omhiredis_streamdynagroupack` is set. +| This can be useful to automatically acknowledge processed entries extracted on a previous stream by imhiredis. + + +.. _omhiredis_streamdynagroupack: + +stream.dynaGroupAck +^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "boolean", "off", "no", "none" + +| Only applicable with mode "stream". +| If set to **on**, the value of :ref:`omhiredis_streamgroupack` is taken as an existing Rsyslog template. + + +.. _omhiredis_streamindexack: + +stream.indexAck +^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "", "no", "none" + +| Only applicable with mode "stream". +| Is **required**, if one of :ref:`omhiredis_streamack` or :ref:`omhiredis_streamdel` are **on**. +| Defines the index value to use for acknowledging/deleting while inserting a new entry, can be either a constant value or a template name if :ref:`omhiredis_streamdynaindexack` is set. +| This can be useful to automatically acknowledge/remove processed entries extracted on a previous stream by imhiredis. + + +.. _omhiredis_streamdynaindexack: + +stream.dynaIndexAck +^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "boolean", "off", "no", "none" + +| Only applicable with mode "stream". +| If set to **on**, the value of :ref:`omhiredis_streamindexack` is taken as an existing Rsyslog template. + +Examples +======== + +Example 1: Template mode +------------------------ + +In "template" mode, the string constructed by the template is sent +to Redis as a command. + +.. note:: + + This mode has problems with strings with spaces in them - full message won't work correctly. In this mode, the template argument is required, and the key argument is meaningless. + +.. code-block:: none + + module(load="omhiredis") + + template( + name="program_count_tmpl" + type="string" + string="HINCRBY progcount %programname% 1") + + action( + name="count_programs" + server="my-redis-server.example.com" + serverport="6379" + type="omhiredis" + mode="template" + template="program_count_tmpl") + + +Results +^^^^^^^ + +Here's an example redis-cli session where we HGETALL the counts: + +.. code-block:: none + + > redis-cli + 127.0.0.1:6379> HGETALL progcount + 1) "rsyslogd" + 2) "35" + 3) "rsyslogd-pstats" + 4) "4302" + + +Example 2: Queue mode +--------------------- + +In "queue" mode, the syslog message is pushed into a Redis list +at "key", using the LPUSH command. If a template is not supplied, +the plugin will default to the RSYSLOG_ForwardFormat template. + +.. code-block:: none + + module(load="omhiredis") + + action( + name="push_redis" + server="my-redis-server.example.com" + serverport="6379" + type="omhiredis" + mode="queue" + key="my_queue") + + +Results +^^^^^^^ + +Here's an example redis-cli session where we RPOP from the queue: + +.. code-block:: none + + > redis-cli + 127.0.0.1:6379> RPOP my_queue + + "<46>2015-09-17T10:54:50.080252-04:00 myhost rsyslogd: [origin software=\"rsyslogd\" swVersion=\"8.13.0.master\" x-pid=\"6452\" x-info=\"http://www.rsyslog.com\"] start" + + 127.0.0.1:6379> + + +Example 3: Publish mode +----------------------- + +In "publish" mode, the syslog message is published to a Redis +topic set by "key". If a template is not supplied, the plugin +will default to the RSYSLOG_ForwardFormat template. + +.. code-block:: none + + module(load="omhiredis") + + action( + name="publish_redis" + server="my-redis-server.example.com" + serverport="6379" + type="omhiredis" + mode="publish" + key="my_channel") + + +Results +^^^^^^^ + +Here's an example redis-cli session where we SUBSCRIBE to the topic: + +.. code-block:: none + + > redis-cli + + 127.0.0.1:6379> subscribe my_channel + + Reading messages... (press Ctrl-C to quit) + + 1) "subscribe" + + 2) "my_channel" + + 3) (integer) 1 + + 1) "message" + + 2) "my_channel" + + 3) "<46>2015-09-17T10:55:44.486416-04:00 myhost rsyslogd-pstats: {\"name\":\"imuxsock\",\"origin\":\"imuxsock\",\"submitted\":0,\"ratelimit.discarded\":0,\"ratelimit.numratelimiters\":0}" + + +Example 4: Set mode +------------------- + +In "set" mode, the syslog message is set as a Redis key at "key". If a template is not supplied, +the plugin will default to the RSYSLOG_ForwardFormat template. + +.. code-block:: none + + module(load="omhiredis") + + action( + name="set_redis" + server="my-redis-server.example.com" + serverport="6379" + type="omhiredis" + mode="set" + key="my_key") + + +Results +^^^^^^^ +Here's an example redis-cli session where we get the key: + +.. code-block:: none + + > redis-cli + + 127.0.0.1:6379> get my_key + + "<46>2019-12-17T20:16:54.781239+00:00 localhost rsyslogd-pstats: { \"name\": \"main Q\", \"origin\": \"core.queue\", + \"size\": 3, \"enqueued\": 7, \"full\": 0, \"discarded.full\": 0, \"discarded.nf\": 0, \"maxqsize\": 3 }" + + 127.0.0.1:6379> ttl my_key + + (integer) -1 + + +Example 5: Set mode with expiration +----------------------------------- + +In "set" mode when "expiration" is set to a positive integer, the syslog message is set as a Redis key at "key", +with expiration "expiration". +If a template is not supplied, the plugin will default to the RSYSLOG_ForwardFormat template. + +.. code-block:: none + + module(load="omhiredis") + + action( + name="set_redis" + server="my-redis-server.example.com" + serverport="6379" + type="omhiredis" + mode="set" + key="my_key" + expiration="10") + + +Results +^^^^^^^ + +Here's an example redis-cli session where we get the key and test the expiration: + +.. code-block:: none + + > redis-cli + + 127.0.0.1:6379> get my_key + + "<46>2019-12-17T20:16:54.781239+00:00 localhost rsyslogd-pstats: { \"name\": \"main Q\", \"origin\": \"core.queue\", + \"size\": 3, \"enqueued\": 7, \"full\": 0, \"discarded.full\": 0, \"discarded.nf\": 0, \"maxqsize\": 3 }" + + 127.0.0.1:6379> ttl my_key + + (integer) 10 + + 127.0.0.1:6379> ttl my_key + + (integer) 3 + + 127.0.0.1:6379> ttl my_key + + (integer) -2 + + 127.0.0.1:6379> get my_key + + (nil) + + +Example 6: Set mode with dynamic key +------------------------------------ + +In any mode with "key" defined and "dynakey" as "on", the key used during operation will be dynamically generated +by Rsyslog using templating. + +.. code-block:: none + + module(load="omhiredis") + + template(name="example-template" type="string" string="%hostname%") + + action( + name="set_redis" + server="my-redis-server.example.com" + serverport="6379" + type="omhiredis" + mode="set" + key="example-template" + dynakey="on") + + +Results +^^^^^^^ +Here's an example redis-cli session where we get the dynamic key: + +.. code-block:: none + + > redis-cli + + 127.0.0.1:6379> keys * + + (empty list or set) + + 127.0.0.1:6379> keys * + + 1) "localhost" + + +Example 7: "Simple" stream mode +------------------------------- + +| By using the **stream mode**, the template-formatted log is inserted in a stream using the :ref:`omhiredis_streamoutfield` parameter as key (or *msg* as default). +| The output template can be explicitely set with the :ref:`omhiredis_template` option (or the default *RSYSLOG_ForwardFormat* template will be used). + +.. code-block:: none + + module(load="omhiredis") + + template(name="example-template" type="string" string="%hostname%") + + action( + type="omhiredis" + server="my-redis-server.example.com" + serverport="6379" + template="example-template" + mode="stream" + key="stream_output" + stream.outField="data") + + +Results +^^^^^^^ +Here's an example redis-cli session where we get the newly inserted stream index: + +.. code-block:: none + + > redis-cli + + 127.0.0.1:6379> XLEN stream_output + 1 + + 127.0.0.1:6379> xread STREAMS stream_output 0 + 1) 1) "stream_output" + 2) 1) 1) "1684507855284-0" + 2) 1) "data" + 2) "localhost" + + +Example 8: Get from a stream with imhiredis, then insert in another one with omhiredis +-------------------------------------------------------------------------------------- + +| When you use the omhiredis in stream mode with the imhiredis in stream mode as input, you might want to acknowledge + entries taken with imhiredis once you insert them back somewhere else with omhiredis. +| The module allows to acknowledge input entries using information either provided by the user through configuration + or through information accessible in the log entry. +| Under the hood, imhiredis adds metadata to the generated logs read from redis streams, this data includes + the stream name, ID of the index, group name and consumer name (when read from a consumer group). +| This information is added in the **$.redis** object and can be retrieved with the help of specific templates. + +.. code-block:: none + + module(load="imhiredis") + module(load="omhiredis") + + template(name="redisJsonMessage" type="list") { + property(name="$!output") + } + + template(name="indexTemplate" type="list") { + property(name="$.redis!index") + } + # Not used in this example, but can be used to replace the static declarations in omhiredis' configuration below + template(name="groupTemplate" type="list") { + property(name="$.redis!group") + } + template(name="keyTemplate" type="list") { + property(name="$.redis!stream") + } + + input(type="imhiredis" + server="127.0.0.1" + port="6379" + mode="stream" + key="stream_input" + stream.consumerGroup="group1" + stream.consumerName="consumer1" + stream.consumerACK="off" + ruleset="receive_redis") + + ruleset(name="receive_redis") { + + action(type="omhiredis" + server="127.0.0.1" + serverport="6379" + mode="stream" + key="stream_output" + stream.ack="on" + # The key and group values are set statically, but the index value is taken from imhiredis metadata + stream.dynaKeyAck="off" + stream.keyAck="stream_input" + stream.dynaGroupAck="off" + stream.groupAck="group1" + stream.indexAck="indexTemplate" + stream.dynaIndexAck="on" + template="redisJsonMessage" + ) + } + + +Results +^^^^^^^ +Here's an example redis-cli session where we get the pending entries at the end of the log re-insertion: + +.. code-block:: none + + > redis-cli + + 127.0.0.1:6379> XINFO GROUPS stream_input + 1) 1) "name" + 1) "group1" + 2) "consumers" + 3) (integer) 1 + 4) "pending" + 5) (integer) 0 + 6) "last-delivered-id" + 7) "1684509391900-0" + 8) "entries-read" + 9) (integer) 1 + 10) "lag" + 11) (integer) 0 + + + +Example 9: Ensuring streams don't grow indefinitely +--------------------------------------------------- + +| While using Redis streams, index entries are not automatically evicted, even if you acknowledge entries. +| You have several options to ensure your streams stays under reasonable memoyr usage, while making sure your data is + not evicted before behing processed. +| To do that, you have 2 available options, that can be used independently from each other + (as they don't apply to the same source): + + - **stream.del** to delete processed entries (can also be used as a complement of ACK'ing) + - **stream.capacityLimit** that allows to ensure a hard-limit of logs can be inserted before dropping entries + +.. code-block:: none + + module(load="imhiredis") + module(load="omhiredis") + + template(name="redisJsonMessage" type="list") { + property(name="$!output") + } + + template(name="indexTemplate" type="list") { + property(name="$.redis!index") + } + template(name="keyTemplate" type="list") { + property(name="$.redis!stream") + } + + input(type="imhiredis" + server="127.0.0.1" + port="6379" + mode="stream" + key="stream_input" + ruleset="receive_redis") + + ruleset(name="receive_redis") { + + action(type="omhiredis" + server="127.0.0.1" + serverport="6379" + mode="stream" + key="stream_output" + stream.capacityLimit="1000000" + stream.del="on" + stream.dynaKeyAck="on" + stream.keyAck="keyTemplate" + stream.dynaIndexAck="on" + stream.indexAck="indexTemplate" + template="redisJsonMessage" + ) + } + + +Results +^^^^^^^ +Here, the result of this configuration is: + + - entries are deleted from the source stream *stream_input* after being inserted by omhiredis to *stream_output* + - *stream_output* won't hold more than (approximately) a million entries at a time + +.. Warning:: + The **stream.capacityLimit** is an approximate maximum! see `redis documentation on MAXLEN and the '~' option <https://redis.io/commands/xadd>`_ to understand how it works! diff --git a/source/configuration/modules/omhttp.rst b/source/configuration/modules/omhttp.rst new file mode 100644 index 0000000..3eee0aa --- /dev/null +++ b/source/configuration/modules/omhttp.rst @@ -0,0 +1,869 @@ +******************************************** +omhttp: HTTP Output Module +******************************************** + +=========================== =========================================================================== +**Module Name:** **omhttp** +**Module Type:** **contributed** - not maintained by rsyslog core team +**Current Maintainer:** `Nelson Yen <https://github.com/n2yen/>`_ +Original Author: `Christian Tramnitz <https://github.com/ctramnitz/>`_ +=========================== =========================================================================== + +.. warning:: + + This page is incomplete, if you want to contribute you can do this on + github in the `rsyslog-doc repository <https://github.com/rsyslog/rsyslog-doc>`_. + + + +Purpose +======= + +This module provides the ability to send messages over an HTTP REST interface. + +This module supports sending messages in individual requests (the default), and batching multiple messages into a single request. Support for retrying failed requests is available in both modes. GZIP compression is configurable with the compress_ parameter. TLS encryption is configurable with the useHttps_ parameter and associated tls parameters. + +In the default mode, every message is sent in its own HTTP request and it is a drop-in replacement for any other output module. In batch_ mode, the module implements several batch formatting options that are configurable via the batch.format_ parameter. Some additional attention to message formatting and retry_ strategy is required in this mode. + +See the `Examples`_ section for some configuration examples. + + +Notable Features +================ + +- `Statistic Counter`_ +- `Message Batching`_, supports several formats. + - Newline concatenation, like the Elasticsearch bulk format. + - JSON Array as a generic batching strategy. + - Kafka REST Proxy format, to support sending data through the `Confluent Kafka REST API <https://docs.confluent.io/current/kafka-rest/docs/index.html>`_ to a Kafka cluster. + +Configuration Parameters +======================== + +.. note:: + + Parameter names are case-insensitive. + + +Action Parameters +----------------- + +Server +^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "array", "localhost", "no", "none" + +The server address you want to connect to. + + +Serverport +^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "443", "no", "none" + +The port you want to connect to. + + +healthchecktimeout +^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "3500", "no", "none" + +The time after which the health check will time out in milliseconds. + +httpcontenttype +^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "application/json; charset=utf-8", "no", "none" + +The HTTP "Content-Type" header sent with each request. This parameter will override other defaults. If a batching mode is specified, the correct content type is automatically configured. The "Content-Type" header can also be configured using the httpheaders_ parameter, it should be configured in only one of the parameters. + + +httpheaderkey +^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +The header key. Currently only a single additional header/key pair is configurable, to specify multiple headers see the httpheaders_ parameter. This parameter along with httpheadervalue_ may be deprecated in the future. + + +httpheadervalue +^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +The header value for httpheaderkey_. + +httpheaders +^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "array", "none", "no", "none" + +An array of strings that defines a list of one or more HTTP headers to send with each message. Keep in mind that some HTTP headers are added using other parameters, "Content-Type" can be configured using httpcontenttype_ and "Content-Encoding: gzip" is added when using the compress_ parameter. + +.. code-block:: text + + action( + type="omhttp" + ... + httpheaders=[ + "X-Insert-Key: key", + "X-Event-Source: logs" + ] + ... + ) + + +uid +^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +The username for basic auth. + + +pwd +^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +The password for the user for basic auth. + + +restpath +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +The rest path you want to use. Do not include the leading slash character. If the full path looks like "localhost:5000/my/path", restpath should be "my/path". + + +dynrestpath +^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +When this parameter is set to "on" you can specify a template name in the parameter +restpath instead of the actual path. This way you will be able to use dynamic rest +paths for your messages based on the template you are using. + + +checkpath +^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +The health check path you want to use. Do not include the leading slash character. If the full path looks like "localhost:5000/my/path", checkpath should be "my/path". +When this parameter is set, omhttp utilizes this path to determine if it is safe to resume (from suspend mode) and communicates this status back to rsyslog core. +This parameter defaults to none, which implies that health checks are not needed, and it is always safe to resume from suspend mode. + +**Important** - Note that it is highly recommended to set a valid health check path, as this allows omhttp to better determine whether it is safe to retry. +See the `rsyslog action queue documentation for more info <https://www.rsyslog.com/doc/v8-stable/configuration/actions.html>`_ regarding general rsyslog suspend and resume behavior. + + +batch +^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +Batch and bulkmode do the same thing, bulkmode included for backwards compatibility. See the `Message Batching`_ section for a detailed breakdown of how batching is implemented. + +This parameter activates batching mode, which queues messages and sends them as a single request. There are several related parameters that specify the format and size of the batch: they are batch.format_, batch.maxbytes_, and batch.maxsize_. + +Note that rsyslog core is the ultimate authority on when a batch must be submitted, due to the way that batching is implemented. This plugin implements the `output plugin transaction interface <https://www.rsyslog.com/doc/v8-stable/development/dev_oplugins.html#output-plugin-transaction-interface>`_. There may be multiple batches in a single transaction, but a batch will never span multiple transactions. This means that if batch.maxsize_ or batch.maxbytes_ is set very large, you may never actually see batches hit this size. Additionally, the number of messages per transaction is determined by the size of the main, action, and ruleset queues as well. + +Additionally, due to some open issues with rsyslog and the transaction interface, batching requires some nuanced retry_ configuration. + + +batch.format +^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "newline", "no", "none" + +This parameter specifies how to combine multiple messages into a single batch. Valid options are *newline* (default), *jsonarray*, *kafkarest*, and *lokirest*. + +Each message on the "Inputs" line is the templated log line that is fed into the omhttp action, and the "Output" line describes the resulting payload sent to the configured HTTP server. + +1. *newline* - Concatenates each message into a single string joined by newline ("\\n") characters. This mode is default and places no restrictions on the structure of the input messages. + +.. code-block:: text + + Inputs: "message 1" "message 2" "message 3" + Output: "message 1\nmessage2\nmessage3" + +2. *jsonarray* - Builds a JSON array containing all messages in the batch. This mode requires that each message is parseable JSON, since the plugin parses each message as JSON while building the array. + +.. code-block:: text + + Inputs: {"msg": "message 1"} {"msg"": "message 2"} {"msg": "message 3"} + Output: [{"msg": "message 1"}, {"msg"": "message 2"}, {"msg": "message 3"}] + +3. *kafkarest* - Builds a JSON object that conforms to the `Kafka Rest Proxy specification <https://docs.confluent.io/current/kafka-rest/docs/quickstart.html>`_. This mode requires that each message is parseable JSON, since the plugin parses each message as JSON while building the batch object. + +.. code-block:: text + + Inputs: {"msg": "message 1"} {"msg"": "message 2"} {"msg": "message 3"} + Output: {"records": [{"value": {"msg": "message 1"}}, {"value": {"msg": "message 2"}}, {"value": {"msg": "message 3"}}]} + +4. *lokirest* - Builds a JSON object that conforms to the `Loki Rest specification <https://github.com/grafana/loki/blob/master/docs/api.md#post-lokiapiv1push>`_. This mode requires that each message is parseable JSON, since the plugin parses each message as JSON while building the batch object. Additionally, the operator is responsible for providing index keys, and message values. + +.. code-block:: text + + Inputs: {"stream": {"tag1":"value1"}, values:[[ "%timestamp%", "message 1" ]]} {"stream": {"tag2":"value2"}, values:[[ %timestamp%, "message 2" ]]} + Output: {"streams": [{"stream": {"tag1":"value1"}, values:[[ "%timestamp%", "message 1" ]]},{"stream": {"tag2":"value2"}, values:[[ %timestamp%, "message 2" ]]}]} + +batch.maxsize +^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "Size", "100", "no", "none" + +This parameter specifies the maximum number of messages that will be sent in each batch. + +batch.maxbytes +^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "Size", "10485760 (10MB)", "no", "none" + +batch.maxbytes and maxbytes do the same thing, maxbytes included for backwards compatibility. + +This parameter specifies the maximum size in bytes for each batch. + +template +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "StdJSONFmt", "no", "none" + +The template to be used for the messages. + +Note that in batching mode, this describes the format of *each* individual message, *not* the format of the resulting batch. Some batch modes require that a template produces valid JSON. + + +retry +^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +This parameter specifies whether failed requests should be retried using the custom retry logic implemented in this plugin. Requests returning 5XX HTTP status codes are considered retriable. If retry is enabled, set retry.ruleset_ as well. + +Note that retries are generally handled in rsyslog by setting action.resumeRetryCount="-1" (or some other integer), and the plugin lets rsyslog know it should start retrying by suspending itself. This is still the recommended approach in the 2 cases enumerated below when using this plugin. In both of these cases, the output plugin transaction interface is not used. That is, from rsyslog core's point of view, each message is contained in its own transaction. + +1. Batching is off (batch="off") +2. Batching is on and the maximum batch size is 1 (batch="on" batch.maxsize="1") + +This custom retry behavior is the result of a bug in rsyslog's handling of transaction commits. See `this issue <https://github.com/rsyslog/rsyslog/issues/2420>`_ for full details. Essentially, if rsyslog hands omhttp 4 messages, and omhttp batches them up but the request fails, rsyslog will only retry the LAST message that it handed the plugin, instead of all 4, even if the plugin returns the correct "defer commit" statuses for messages 1, 2, and 3. This means that omhttp cannot rely on action.resumeRetryCount for any transaction that processes more than a single message, and explains why the 2 above cases do work correctly. + +It looks promising that issue will be resolved at some point, so this behavior can be revisited at that time. + +retry.ruleset +^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +This parameter specifies the ruleset where this plugin should requeue failed messages if retry_ is on. This ruleset generally would contain another omhttp action instance. + +**Important** - Note that the message that is queued on the retry ruleset is the templated output of the initial omhttp action. This means that no further templating should be done to messages inside this ruleset, unless retries should be templated differently than first-tries. An "echo template" does the trick here. + +.. code-block:: text + + template(name="tpl_echo" type="string" string="%msg%") + +This retry ruleset can recursively call itself as its own retry.ruleset to retry forever, but there is no timeout behavior currently implemented. + +Alternatively, the omhttp action in the retry ruleset could be configured to support action.resumeRetryCount as explained above in the retry parameter section. The benefit of this approach is that retried messages still hit the server in a batch format (though with a single message in it), and the ability to configure rsyslog to give up after some number of resume attempts so as to avoid resource exhaustion. + +Or, if some data loss or high latency is acceptable, do not configure retries with the retry ruleset itself. A single retry from the original ruleset might catch most failures, and errors from the retry ruleset could still be logged using the errorfile parameter and sent later on via some other process. + +ratelimit.interval +^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "600", "no", "none" + +This parameter sets the rate limiting behavior for the retry.ruleset_. Specifies the interval in seconds onto which rate-limiting is to be applied. If more than ratelimit.burst messages are read during that interval, further messages up to the end of the interval are discarded. The number of messages discarded is emitted at the end of the interval (if there were any discards). Setting this to value zero turns off ratelimiting. + +ratelimit.burst +^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "20000", "no", "none" + +This parameter sets the rate limiting behavior for the retry.ruleset_. Specifies the maximum number of messages that can be emitted within the ratelimit.interval interval. For further information, see description there. + + +errorfile +^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +Here you can set the name of a file where all errors will be written to. Any request that returns a 4XX or 5XX HTTP code is recorded in the error file. Each line is JSON formatted with "request" and "response" fields, example pretty-printed below. + +.. code-block:: text + + { + "request": { + "url": "https://url.com:443/path", + "postdata": "mypayload" + }, + "response" : { + "status": 400, + "message": "error string" + } + } + +It is intended that a full replay of failed data is possible by processing this file. + +compress +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +When switched to "on" each message will be compressed as GZIP using zlib's deflate compression algorithm. + +A "Content-Encoding: gzip" HTTP header is added to each request when this feature is used. Set the compress.level_ for fine-grained control. + +compress.level +^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "-1", "no", "none" + +Specify the zlib compression level if compress_ is enabled. Check the `zlib manual <https://www.zlib.net/manual.html>`_ for further documentation. + +"-1" is the default value that strikes a balance between best speed and best compression. "0" disables compression. "1" results in the fastest compression. "9" results in the best compression. + +useHttps +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +When switched to "on" you will use https instead of http. + + +tls.cacert +^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +This parameter sets the path to the Certificate Authority (CA) bundle. Expects .pem format. + +tls.mycert +^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +This parameter sets the path to the SSL client certificate. Expects .pem format. + +tls.myprivkey +^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +The parameters sets the path to the SSL private key. Expects .pem format. + +allowunsignedcerts +^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "boolean", "off", "no", "none" + +If `"on"`, this will set the curl `CURLOPT_SSL_VERIFYPEER` option to +`0`. You are strongly discouraged to set this to `"on"`. It is +primarily useful only for debugging or testing. + +skipverifyhost +^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "boolean", "off", "no", "none" + +If `"on"`, this will set the curl `CURLOPT_SSL_VERIFYHOST` option to +`0`. You are strongly discouraged to set this to `"on"`. It is +primarily useful only for debugging or testing. + +reloadonhup +^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +If this parameter is "on", the plugin will close and reopen any libcurl handles on a HUP signal. This option is primarily intended to enable reloading short-lived certificates without restarting rsyslog. + +Statistic Counter +================= + +This plugin maintains global :doc:`statistics <../rsyslog_statistic_counter>` for omhttp that +accumulates all action instances. The statistic origin is named "omhttp" with following counters: + +- **messages.submitted** - Number of messages submitted to omhttp. Messages resubmitted via a retry ruleset will be counted twice. + +- **messages.success** - Number of messages successfully sent. + +- **messages.fail** - Number of messages that omhttp failed to deliver for any reason. + +- **messages.retry** - Number of messages that omhttp resubmitted for retry via the retry ruleset. + +- **request.count** - Number of attempted HTTP requests. + +- **request.success** - Number of successful HTTP requests. A successful request can return *any* HTTP status code. + +- **request.fail** - Number of failed HTTP requests. A failed request is something like an invalid SSL handshake, or the server is not reachable. Requests returning 4XX or 5XX HTTP status codes are *not* failures. + +- **request.status.success** - Number of requests returning 1XX or 2XX HTTP status codes. + +- **request.status.fail** - Number of requests returning 3XX, 4XX, or 5XX HTTP status codes. If a requests fails (i.e. server not reachable) this counter will *not* be incremented. + +Message Batching +================ + +See the batch.format_ section for some light examples of available batching formats. + +Implementation +-------------- + +Here's the pseudocode of the batching algorithm used by omhttp. This section of code would run once per transaction. + +.. code-block:: python + + Q = Queue() + + def submit(Q): # function to submit + batch = serialize(Q) # serialize according to configured batch.format + result = post(batch) # http post serialized batch to server + checkFailureAndRetry(Q, result) # check if post failed and pushed failed messages to configured retry.ruleset + Q.empty() # reset for next batch + + + while isActive(transaction): # rsyslog manages the transaction + message = receiveMessage() # rsyslog sends us messages + if wouldTriggerSubmit(Q, message): # if this message puts us over maxbytes or maxsize + submit(Q) # submit the current batch + Q.push(message) # queue this message on the current batch + + submit(Q) # transaction is over, submit what is currently in the queue + + +Walkthrough +----------- + +This is a run through of a file tailing to omhttp scenario. Suppose we have a file called ``/var/log/my.log`` with this content.. + +.. code-block:: text + + 001 message + 002 message + 003 message + 004 message + 005 message + 006 message + 007 message + ... + +We are tailing this using imfile and defining a template to generate a JSON payload... + +.. code-block:: text + + input(type="imfile" File="/var/log/my.log" ruleset="rs_omhttp" ... ) + + # Produces JSON formatted payload + template(name="tpl_omhttp_json" type="list") { + constant(value="{") property(name="msg" outname="message" format="jsonfr") + constant(value=",") property(name="hostname" outname="host" format="jsonfr") + constant(value=",") property(name="timereported" outname="timestamp" format="jsonfr" dateFormat="rfc3339") + constant(value="}") + } + +Our omhttp ruleset is configured to batch using the *jsonarray* format with 5 messages per batch, and to use a retry ruleset. + + +.. code-block:: text + + module(load="omhttp") + + ruleset(name="rs_omhttp") { + action( + type="omhttp" + template="tpl_omhttp_json" + batch="on" + batch.format="jsonarray" + batch.maxsize="5" + retry="on" + retry.ruleset="rs_omhttp_retry" + ... + ) + } + + call rs_omhttp + +Each input message to this omhttp action is the output of ``tpl_omhttp_json`` with the following structure.. + +.. code-block:: text + + {"message": "001 message", "host": "localhost", "timestamp": "2018-12-28T21:14:13.840470+00:00"} + +After 5 messages have been queued, and a batch submit is triggered, omhttp serializes the messages as a JSON array and attempts to post the batch to the server. At this point the payload on the wire looks like this.. + +.. code-block:: text + + [ + {"message": "001 message", "host": "localhost", "timestamp": "2018-12-28T21:14:13.000000+00:00"}, + {"message": "002 message", "host": "localhost", "timestamp": "2018-12-28T21:14:14.000000+00:00"}, + {"message": "003 message", "host": "localhost", "timestamp": "2018-12-28T21:14:15.000000+00:00"}, + {"message": "004 message", "host": "localhost", "timestamp": "2018-12-28T21:14:16.000000+00:00"}, + {"message": "005 message", "host": "localhost", "timestamp": "2018-12-28T21:14:17.000000+00:00"} + ] + +If the request fails, omhttp requeues each failed message onto the retry ruleset. However, recall that the inputs to the ``rs_omhttp`` ruleset are the rendered *outputs* of ``tpl_json_omhttp``, and therefore we *cannot* use the same template (and therefore the same action instance) to produce the retry messages. At this point, the ``msg`` rsyslog property is ``{"message": "001 message", "host": "localhost", "timestamp": "2018-12-28T21:14:13.000000+00:00"}`` instead of the original ``001 message``, and ``tpl_json_omhttp`` would render incorrect payloads. + +Instead we define a simple template that echos its input.. + +.. code-block:: text + + template(name="tpl_echo" type="string" string="%msg%") + +And assign it to the retry template.. + +.. code-block:: text + + ruleset(name="rs_omhttp_retry") { + action( + type="omhttp" + template="tpl_echo" + batch="on" + batch.format="jsonarray" + batch.maxsize="5" + ... + ) + } + +And the destination is none the wiser! The *newline*, *jsonarray*, and *kafkarest* formats all behave in the same way with respect to their batching and retry behavior, and differ only in the format of the on-the-wire payload. The formats themselves are described in the batch.format_ section. + +Examples +======== + +Example 1 +--------- + +The following example is a basic usage, first the module is loaded and then +the action is used with a standard retry strategy. + + +.. code-block:: text + + module(load="omhttp") + template(name="tpl1" type="string" string="{\"type\":\"syslog\", \"host\":\"%HOSTNAME%\"}") + action( + type="omhttp" + server="127.0.0.1" + serverport="8080" + restpath="events" + template="tpl1" + action.resumeRetryCount="3" + ) + +Example 2 +--------- + +The following example is a basic batch usage with no retry processing. + + +.. code-block:: text + + module(load="omhttp") + template(name="tpl1" type="string" string="{\"type\":\"syslog\", \"host\":\"%HOSTNAME%\"}") + action( + type="omhttp" + server="127.0.0.1" + serverport="8080" + restpath="events" + template="tpl1" + batch="on" + batch.format="jsonarray" + batch.maxsize="10" + ) + + +Example 3 +--------- + +The following example is a batch usage with a retry ruleset that retries forever + + +.. code-block:: text + + module(load="omhttp") + + template(name="tpl_echo" type="string" string="%msg%") + ruleset(name="rs_retry_forever") { + action( + type="omhttp" + server="127.0.0.1" + serverport="8080" + restpath="events" + template="tpl_echo" + + batch="on" + batch.format="jsonarray" + batch.maxsize="10" + + retry="on" + retry.ruleset="rs_retry_forever" + ) + } + + template(name="tpl1" type="string" string="{\"type\":\"syslog\", \"host\":\"%HOSTNAME%\"}") + action( + type="omhttp" + server="127.0.0.1" + serverport="8080" + restpath="events" + template="tpl1" + + batch="on" + batch.format="jsonarray" + batch.maxsize="10" + + retry="on" + retry.ruleset="rs_retry_forever" + ) + +Example 4 +--------- + +The following example is a batch usage with a couple retry options + +.. code-block:: text + + module(load="omhttp") + + template(name="tpl_echo" type="string" string="%msg%") + + # This retry ruleset tries to send batches once then logs failures. + # Error log could be tailed by rsyslog itself or processed by some + # other program. + ruleset(name="rs_retry_once_errorfile") { + action( + type="omhttp" + server="127.0.0.1" + serverport="8080" + restpath="events" + template="tpl_echo" + + batch="on" + batch.format="jsonarray" + batch.maxsize="10" + + retry="off" + errorfile="/var/log/rsyslog/omhttp_errors.log" + ) + } + + # This retry ruleset gives up trying to batch messages and instead always + # uses a batch size of 1, relying on the suspend/resume mechanism to do + # further retries if needed. + ruleset(name="rs_retry_batchsize_1") { + action( + type="omhttp" + server="127.0.0.1" + serverport="8080" + restpath="events" + template="tpl_echo" + + batch="on" + batch.format="jsonarray" + batch.maxsize="1" + action.resumeRetryCount="-1" + ) + } + + template(name="tpl1" type="string" string="{\"type\":\"syslog\", \"host\":\"%HOSTNAME%\"}") + action( + type="omhttp" + template="tpl1" + + ... + + retry="on" + retry.ruleset="<some_retry_ruleset>" + ) + +Example 5 +--------- + +The following example is a batch action for pushing logs with checking, and queues to Loki. + +.. code-block:: text + + module(load="omhttp") + + template(name="loki" type="string" string="{\"stream\":{\"host\":\"%HOSTNAME%\",\"facility\":\"%syslogfacility-text%\",\"priority\":\"%syslogpriority-text%\",\"syslogtag\":\"%syslogtag%\"},\"values\": [[ \"%timegenerated:::date-unixtimestamp%000000000\", \"%msg%\" ]]}") + + + action( + name="loki" + type="omhttp" + useHttps="off" + server="localhost" + serverport="3100" + checkpath="ready" + + restpath="loki/api/v1/push" + template="loki" + batch.format="lokirest" + batch="on" + batch.maxsize="10" + + queue.size="10000" queue.type="linkedList" + queue.workerthreads="3" + queue.workerthreadMinimumMessages="1000" + queue.timeoutWorkerthreadShutdown="500" + queue.timeoutEnqueue="10000" + ) diff --git a/source/configuration/modules/omhttpfs.rst b/source/configuration/modules/omhttpfs.rst new file mode 100644 index 0000000..acaf369 --- /dev/null +++ b/source/configuration/modules/omhttpfs.rst @@ -0,0 +1,149 @@ +************************************* +omhttpfs: Hadoop HTTPFS Output Module +************************************* + +=========================== =========================================================================== +**Module Name:** **omhttpfs** +**Available Since:** **8.10.0** +**Author:** `sskaje <https://sskaje.me/2014/12/omhttpfs-rsyslog-hdfs-output-plugin/>`_ <sskaje@gmail.com> +=========================== =========================================================================== + + +Purpose +======= + +This module is an alternative to omhdfs via `Hadoop HDFS over HTTP <http://hadoop.apache.org/docs/current/hadoop-hdfs-httpfs/index.html>`_. + + +Dependencies +============ + +* libcurl + + +Configuration Parameters +======================== + +.. note:: + + Parameter names are case-insensitive. + + +Action Parameters +----------------- + +Host +^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "127.0.0.1", "no", "none" + +HttpFS server host. + + +Port +^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "14000", "no", "none" + +HttpFS server port. + + +User +^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "hdfs", "no", "none" + +HttpFS auth user. + + +https +^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +Turn on if your HttpFS runs on HTTPS. + + +File +^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "yes", "none" + +File to write, or a template name. + + +isDynFile +^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +Turn this on if your **file** is a template name. +See examples below. + + +Template +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "RSYSLOG_FileFormat", "no", "none" + +Format your message when writing to **file**. Default: *RSYSLOG_FileFormat* + + +Configure +========= + +.. code-block:: none + + ./configure --enable-omhttpfs + + +Examples +======== + +Example 1 +--------- + +.. code-block:: none + + module(load="omhttpfs") + template(name="hdfs_tmp_file" type="string" string="/tmp/%$YEAR%/test.log") + template(name="hdfs_tmp_filecontent" type="string" string="%$YEAR%-%$MONTH%-%$DAY% %MSG% ==\n") + local4.* action(type="omhttpfs" host="10.1.1.161" port="14000" https="off" file="hdfs_tmp_file" isDynFile="on") + local5.* action(type="omhttpfs" host="10.1.1.161" port="14000" https="off" file="hdfs_tmp_file" isDynFile="on" template="hdfs_tmp_filecontent") + + diff --git a/source/configuration/modules/omjournal.rst b/source/configuration/modules/omjournal.rst new file mode 100644 index 0000000..fd250a4 --- /dev/null +++ b/source/configuration/modules/omjournal.rst @@ -0,0 +1,71 @@ +********************************* +omjournal: Systemd Journal Output +********************************* + +=========================== =========================================================================== +**Module Name:** **omjournal** +**Author:** `Rainer Gerhards <https://rainer.gerhards.net/>`_ <rgerhards@adiscon.com> +=========================== =========================================================================== + + +Purpose +======= + +This module provides native support for logging to the systemd journal. + + +Configuration Parameters +======================== + +.. note:: + + Parameter names are case-insensitive. + + +Action Parameters +----------------- + +Template +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +Template to use when submitting messages. + +By default, rsyslog will use the incoming %msg% as the MESSAGE field +of the journald entry, and include the syslog tag and priority. + +You can override the default formatting of the message, and include +custom fields with a template. Complex fields in the template +(eg. json entries) will be added to the journal as json text. Other +fields will be coerced to strings. + +Journald requires that you include a template parameter named MESSAGE. + + +Examples +======== + +Example 1 +--------- + +The following sample writes all syslog messages to the journal with a +custom EVENT_TYPE field. + +.. code-block:: none + + module(load="omjournal") + + template(name="journal" type="list") { + constant(value="Something happened" outname="MESSAGE") + property(name="$!event-type" outname="EVENT_TYPE") + } + + action(type="omjournal" template="journal") + + diff --git a/source/configuration/modules/omkafka.rst b/source/configuration/modules/omkafka.rst new file mode 100644 index 0000000..ece8cd6 --- /dev/null +++ b/source/configuration/modules/omkafka.rst @@ -0,0 +1,478 @@ +****************************** +omkafka: write to Apache Kafka +****************************** + +=========================== =========================================================================== +**Module Name:** **omkafka** +**Author:** `Rainer Gerhards <https://rainer.gerhards.net/>`_ <rgerhards@adiscon.com> +**Available since:** v8.7.0 +=========================== =========================================================================== + + +Purpose +======= + +The omkafka plug-in implements an Apache Kafka producer, permitting +rsyslog to write data to Kafka. + + +Configuration Parameters +======================== + +.. note:: + + Parameter names are case-insensitive. + + +Action Parameters +----------------- + +Note that omkafka supports some *Array*-type parameters. While the parameter +name can only be set once, it is possible to set multiple values with that +single parameter. See the :ref:`omkafka-examples-label` section for details. + + +Broker +^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "array", "localhost:9092", "no", "none" + +Specifies the broker(s) to use. + + +Topic +^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "none", "yes", "none" + +Specifies the topic to produce to. + + +Key +^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +Kafka key to be used for all messages. + +If a key is provided and partitions.auto="on" is set, then all messages will +be assigned to a partition based on the key. + + +DynaKey +^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive", "Available since" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none", v8.1903 + +If set, the key parameter becomes a template for the key to base the +partitioning on. + + +DynaTopic +^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +If set, the topic parameter becomes a template for which topic to +produce messages to. The cache is cleared on HUP. + + +DynaTopic.Cachesize +^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "50", "no", "none" + +If set, defines the number of topics that will be kept in the dynatopic +cache. + + +Partitions.Auto +^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +Librdkafka provides an automatic partitioning function that will +automatically distribute the produced messages into all partitions +configured for that topic. + +To use, set partitions.auto="on". This is instead of specifying the +number of partitions on the producer side, where it would be easier +to change the kafka configuration on the cluster for number of +partitions/topic vs on every machine talking to Kafka via rsyslog. + +If no key is set, messages will be distributed randomly across partitions. +This results in a very even load on all partitions, but does not preserve +ordering between the messages. + +If a key is set, a partition will be chosen automatically based on it. All +messages with the same key will be sorted into the same partition, +preserving their ordering. For example, by setting the key to the hostname, +messages from a specific host will be written to one partition and ordered, +but messages from different nodes will be distributed across different +partitions. This distribution is essentially random, but stable. If the +number of different keys is much larger than the number of partitions on the +topic, load will be distributed fairly evenly. + +If set, it will override any other partitioning scheme configured. + + +Partitions.number +^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "none", "no", "none" + +If set, specifies how many partitions exists **and** activates +load-balancing among them. Messages are distributed more or +less evenly between the partitions. Note that the number specified +must be correct. Otherwise, some errors may occur or some partitions +may never receive data. + + +Partitions.useFixed +^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "none", "no", "none" + +If set, specifies the partition to which data is produced. All +data goes to this partition, no other partition is ever involved +for this action. + + +errorFile +^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +If set, messages that could not be sent and caused an error +messages are written to the file specified. This file is in JSON +format, with a single record being written for each message in +error. The entry contains the full message, as well as Kafka +error number and reason string. + +The idea behind the error file is that the admin can periodically +run a script that reads the error file and reacts on it. Note that +the error file is kept open from when the first error occurred up +until rsyslog is terminated or received a HUP signal. + + +statsFile +^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +If set, the contents of the JSON object containing the full librdkafka +statistics will be written to the file specified. The file will be +updated based on the statistics.interval.ms confparam value, which must +also be set. + + +ConfParam +^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "array", "none", "no", "none" + +Permits to specify Kafka options. Rather than offering a myriad of +config settings to match the Kafka parameters, we provide this setting +here as a vehicle to set any Kafka parameter. This has the big advantage +that Kafka parameters that come up in new releases can immediately be used. + +Note that we use librdkafka for the Kafka connection, so the parameters +are actually those that librdkafka supports. As of our understanding, this +is a superset of the native Kafka parameters. + + +TopicConfParam +^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "array", "none", "no", "none" + +In essence the same as *confParam*, but for the Kafka topic. + + +Template +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "template set via template module parameter", "no", "none" + +Sets the template to be used for this action. + + +closeTimeout +^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "2000", "no", "none" + +Sets the time to wait in ms (milliseconds) for draining messages submitted to kafka-handle +(provided by librdkafka) before closing it. + +The maximum value of closeTimeout used across all omkafka action instances +is used as librdkafka unload-timeout while unloading the module +(for shutdown, for instance). + + +resubmitOnFailure +^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +.. versionadded:: 8.28.0 + +If enabled, failed messages will be resubmit automatically when kafka is able to send +messages again. To prevent message loss, this option should be enabled. + +**Note:** Messages that are rejected by kafka due to exceeding the maximum configured +message size, are automatically dropped. These errors are not retriable. + +KeepFailedMessages +^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +If enabled, failed messages will be saved and loaded on shutdown/startup and resend after startup if +the kafka server is able to receive messages again. This setting requires resubmitOnFailure to be enabled as well. + + +failedMsgFile +^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +.. versionadded:: 8.28.0 + +Filename where the failed messages should be stored into. +Needs to be set when keepFailedMessages is enabled, otherwise failed messages won't be saved. + + +statsName +^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +.. versionadded:: 8.2108.0 + +The name assigned to statistics specific to this action instance. The supported set of +statistics tracked for this action instance are **submitted**, **acked**, **failures**. +See the :ref:`statistics-counter_label` section for more details. + + +.. _statistics-counter_label: + +Statistic Counter +================= + +This plugin maintains global :doc:`statistics <../rsyslog_statistic_counter>` for omkafka that +accumulate all action instances. The statistic origin is named "omafka" with following counters: + +- **submitted** - number of messages submitted to omkafka for processing (with both acknowledged + deliveries to broker as well as failed or re-submitted from omkafka to librdkafka). + +- **maxoutqsize** - high water mark of output queue size. + +- **failures** - number of messages that librdkafka failed to deliver. This number is + broken down into counts of various types of failures. + +- **topicdynacache.skipped** - count of dynamic topic cache lookups that find an existing topic and + skip creating a new one. + +- **topicdynacache.miss** - count of dynamic topic cache lookups that fail to find an existing topic + and end up creating new ones. + +- **topicdynacache.evicted** - count of dynamic topic cache entry evictions. + +- **acked** - count of messages that were acknowledged by kafka broker. Note that + kafka broker provides two levels of delivery acknowledgements depending on topicConfParam: + default (acks=1) implies delivery to the leader only while acks=-1 implies delivery to leader + as well as replication to all brokers. + +- **failures_msg_too_large** - count of messages dropped by librdkafka when it failed to + deliver to the broker because broker considers message to be too large. Note that + omkafka may still resubmit to librdkafka depending on resubmitOnFailure option. + +- **failures_unknown_topic** - count of messages dropped by librdkafka when it failed to + deliver to the broker because broker does not recognize the topic. + +- **failures_queue_full** - count of messages dropped by librdkafka when its queue becomes + full. Note that default size of librdkafka queue is 100,000 messages. + +- **failures_unknown_partition** - count of messages that librdkafka failed to deliver because + broker does not recognize a partition. + +- **failures_other** - count of all of the rest of the failures that do not fall in any of + the above failure categories. + +- **errors_timed_out** - count of messages that librdkafka could not deliver within timeout. These + errors will cause action to be suspended but messages can be retried depending on retry options. + +- **errors_transport** - count of messages that librdkafka could not deliver due to transport errors. + These messages can be retried depending on retry options. + +- **errors_broker_down** - count of messages that librdkafka could not deliver because it thinks that + broker is not accessible. These messages can be retried depending on options. + +- **errors_auth** - count of messages that librdkafka could not deliver due to authentication errors. + These messages can be retried depending on the options. + +- **errors_ssl** - count of messages that librdkafka could not deliver due to ssl errors. + These messages can be retried depending on the options. + +- **errors_other** - count of rest of librdkafka errors. + +- **rtt_avg_usec** - broker round trip time in microseconds averaged over all brokers. It is based + on the statistics callback window specified through statistics.interval.ms parameter to librdkafka. + Average exclude brokers with less than 100 microseconds rtt. + +- **throttle_avg_msec** - broker throttling time in milliseconds averaged over all brokers. This is + also a part of window statistics delivered by librdkakfka. Average excludes brokers with zero throttling time. + +- **int_latency_avg_usec** - internal librdkafka producer queue latency in microseconds averaged other + all brokers. This is also part of window statistics and average excludes brokers with zero internal latency. + +Note that three window statics counters are not safe with multiple clients. When statistics callback is +enabled, for example, by using statics.callback.ms=60000, omkafa will generate an internal log message every +minute for the corresponding omkafka action: + +.. code-block:: none + + 2018-03-31T01:51:59.368491+00:00 app1-1.example.com rsyslogd: statscb_window_stats: + handler_name=collections.rsyslog.core#producer-1 replyq=0 msg_cnt=30 msg_size=37986 msg_max=100000 + msg_size_max=1073741824 rtt_avg_usec=41475 throttle_avg_msec=0 int_latency_avg_usec=2943224 [v8.32.0] + +For multiple actions using statistics callback, there will be one such record for each action after specified +window period. See https://github.com/edenhill/librdkafka/wiki/Statistics for more details on statistics +callback values. + +Examples +======== + +.. _omkafka-examples-label: + +Using Array Type Parameter +-------------------------- + +Set a single value +^^^^^^^^^^^^^^^^^^ + +For example, to select "snappy" compression, you can use: + +.. code-block:: none + + action(type="omkafka" topic="mytopic" confParam="compression.codec=snappy") + + +which is equivalent to: + +.. code-block:: none + + action(type="omkafka" topic="mytopic" confParam=["compression.codec=snappy"]) + + +Set multiple values +^^^^^^^^^^^^^^^^^^^ + +To specify multiple values, just use the bracket notation and create a +comma-delimited list of values as shown here: + +.. code-block:: none + + action(type="omkafka" topic="mytopic" + confParam=["compression.codec=snappy", + "socket.timeout.ms=5", + "socket.keepalive.enable=true"] + ) + + diff --git a/source/configuration/modules/omlibdbi.rst b/source/configuration/modules/omlibdbi.rst new file mode 100644 index 0000000..01d1e95 --- /dev/null +++ b/source/configuration/modules/omlibdbi.rst @@ -0,0 +1,238 @@ +**************************************** +omlibdbi: Generic Database Output Module +**************************************** + +=========================== =========================================================================== +**Module Name:** **omlibdbi** +**Author:** `Rainer Gerhards <https://rainer.gerhards.net/>`_ <rgerhards@adiscon.com> +=========================== =========================================================================== + + +Purpose +======= + +This modules supports a large number of database systems via +`libdbi <http://libdbi.sourceforge.net/>`_. Libdbi abstracts the +database layer and provides drivers for many systems. Drivers are +available via the +`libdbi-drivers <http://libdbi-drivers.sourceforge.net/>`_ project. As +of this writing, the following drivers are available: + +- `Firebird/Interbase <http://www.firebird.sourceforge.net/>`_ +- `FreeTDS <http://www.freetds.org/>`_ (provides access to `MS SQL + Server <http://www.microsoft.com/sql>`_ and + `Sybase <http://www.sybase.com/products/informationmanagement/adaptiveserverenterprise>`_) +- `MySQL <http://www.mysql.com/>`_ (also supported via the native + `ommysql <ommysql.html>`_ plugin in rsyslog) +- `PostgreSQL <http://www.postgresql.org/>`_\ (also supported via the + native `ommysql <ommysql.html>`_ plugin in rsyslog) +- `SQLite/SQLite3 <http://www.sqlite.org/>`_ + +The following drivers are in various stages of completion: + +- `Ingres <http://ingres.com/>`_ +- `mSQL <http://www.hughes.com.au/>`_ +- `Oracle <http://www.oracle.com/>`_ + +These drivers seem to be quite usable, at least from an rsyslog point of +view. + +Libdbi provides a slim layer between rsyslog and the actual database +engine. We have not yet done any performance testing (e.g. omlibdbi vs. +:doc:`ommysql`) but honestly believe that the performance impact should be +irrelevant, if at all measurable. Part of that assumption is that +rsyslog just does the "insert" and most of the time is spent either in +the database engine or rsyslog itself. It's hard to think of any +considerable time spent in the libdbi abstraction layer. + + +Setup +===== + +In order for this plugin to work, you need to have libdbi, the libdbi +driver for your database backend and the client software for your +database backend installed. There are libdbi packages for many +distributions. Please note that rsyslogd requires a quite recent version +(0.8.3) of libdbi. It may work with older versions, but these need some +special ./configure options to support being called from a dlopen()ed +plugin (as omlibdbi is). So in short, you probably save you a lot of +headache if you make sure you have at least libdbi version 0.8.3 on your +system. + + +Configuration Parameters +======================== + +.. note:: + + Parameter names are case-insensitive. + + +Module Parameters +----------------- + +DriverDirectory +^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "``$ActionLibdbiDriverDirectory``" + +This is a global setting. It points libdbi to its driver directory. +Usually, you do not need to set it. If you installed libdbi-driver's +at a non-standard location, you may need to specify the directory +here. If you are unsure, do not use this configuration parameter. +Usually, everything works just fine. + + +Template +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +Standard template used for the actions. + + +Action Parameters +----------------- + +Driver +^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "yes", "``$ActionLibdbiDriver``" + +Name of the dbidriver to use, see libdbi-drivers documentation. As a +quick excerpt, at least those were available at the time of this +writing: + +- ``mysql`` (:doc:`ommysql` is recommended instead) +- ``firebird`` (Firebird and InterBase) +- ``ingres`` +- ``msql`` +- ``Oracle`` +- ``sqlite`` +- ``sqlite3`` +- ``freetds`` (for Microsoft SQL and Sybase) +- ``pgsql`` (:doc:`ompgsql` is recommended instead) + + +Server +^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "yes", "``$ActionLibdbiHost``" + +The host to connect to. + + +UID +^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "yes", "``$ActionLibdbiUserName``" + +The user used to connect to the database. + + +PWD +^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "yes", "``$ActionlibdbiPassword``" + +That user's password. + + +DB +^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "yes", "``$ActionlibdbiDBName``" + +The database that shall be written to. + + +Template +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +Template used for this action. + + +Caveats/Known Bugs: +=================== + +You must make sure that any templates used for omlibdbi properly escape +strings. This is usually done by supplying the SQL (or STDSQL) option to +the template. Omlibdbi rejects templates without this option for +security reasons. However, omlibdbi does not detect if you used the +right option for your backend. Future versions of rsyslog (with +full expression support) will provide advanced ways of handling this +situation. So far, you must be careful. The default template provided by +rsyslog is suitable for MySQL, but not necessarily for your database +backend. Be careful! + +If you receive the rsyslog error message "libdbi or libdbi drivers not +present on this system" you may either not have libdbi and its drivers +installed or (very probably) the version is earlier than 0.8.3. In this +case, you need to make sure you have at least 0.8.3 and the libdbi +driver for your database backend present on your system. + +I do not have most of the database supported by omlibdbi in my lab. So +it received limited cross-platform tests. If you run into troubles, be +sure the let us know at +`http://www.rsyslog.com <http://www.rsyslog.com>`_. + + +Examples +======== + +Example 1 +--------- + +The following sample writes all syslog messages to the database +"syslog_db" on mysqlserver.example.com. The server is MySQL and being +accessed under the account of "user" with password "pwd". + +.. code-block:: none + + module(load="omlibdbi") + action(type="omlibdbi" driver="mysql" server="mysqlserver.example.com" + uid="user" pwd="pwd" db="syslog_db") + + diff --git a/source/configuration/modules/ommail.rst b/source/configuration/modules/ommail.rst new file mode 100644 index 0000000..e28cd17 --- /dev/null +++ b/source/configuration/modules/ommail.rst @@ -0,0 +1,306 @@ +************************** +ommail: Mail Output Module +************************** + +.. index:: ! imudp + +=========================== =========================================================================== +**Module Name:** **ommail** +**Available Since:** **3.17.0** +**Author:** `Rainer Gerhards <https://rainer.gerhards.net/>`_ <rgerhards@adiscon.com> +=========================== =========================================================================== + + +Purpose +======= + +This module supports sending syslog messages via mail. Each syslog +message is sent via its own mail. Obviously, you will want to apply +rigorous filtering, otherwise your mailbox (and mail server) will be +heavily spammed. The ommail plugin is primarily meant for alerting +users. As such, it is assumed that mails will only be sent in an +extremely limited number of cases. + +Ommail uses up to two templates, one for the mail body and optionally +one for the subject line. Note that the subject line can also be set to +a constant text. +If neither the subject nor the mail body is provided, a quite meaningless +subject line is used +and the mail body will be a syslog message just as if it were written to +a file. It is expected that the users customizes both messages. In an +effort to support cell phones (including SMS gateways), there is an +option to turn off the body part at all. This is considered to be useful +to send a short alert to a pager-like device. +It is highly recommended to use the + +.. code-block:: none + + action.execonlyonceeveryinterval="<seconds>" + +parameter to limit the amount of mails that potentially be +generated. With it, mails are sent at most in a <seconds> interval. This +may be your life safer. And remember that an hour has 3,600 seconds, so +if you would like to receive mails at most once every two hours, include +a + +.. code-block:: none + + action.execonlyonceeveryinterval="7200" + +in the action definition. Messages sent more frequently are simply discarded. + + +Configuration Parameters +======================== + +Configuration parameters are supported starting with v8.5.0. Earlier +v7 and v8 versions did only support legacy parameters. + +.. note:: + + Parameter names are case-insensitive. + + +Action Parameters +----------------- + +Server +^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "yes", "``$ActionMailSMTPServer``" + +Name or IP address of the SMTP server to be used. Must currently be +set. The default is 127.0.0.1, the SMTP server on the local machine. +Obviously it is not good to expect one to be present on each machine, +so this value should be specified. + + +Port +^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "yes", "``$ActionMailSMTPPort``" + +Port number or name of the SMTP port to be used. The default is 25, +the standard SMTP port. + + +MailFrom +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "yes", "``$ActionMailFrom``" + +The email address used as the senders address. + + +MailTo +^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "array", "none", "yes", "``$ActionMailTo``" + +The recipient email address(es). Note that this is an array parameter. See +samples below on how to specify multiple recipients. + + +Subject.Template +^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "``$ActionMailSubject``" + +The name of the template to be used as the mail subject. + +If you want to include some information from the message inside the +template, you need to use *subject.template* with an appropriate template. +If you just need a constant text, you can simply use *subject.text* +instead, which doesn't require a template definition. + + +Subject.Text +^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "none", "no", "none" + +This is used to set a **constant** subject text. + + +Body.Enable +^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "on", "no", "``$ActionMailEnableBody``" + +Setting this to "off" permits to exclude the actual message body. +This may be useful for pager-like devices or cell phone SMS messages. +The default is "on", which is appropriate for almost all cases. Turn +it off only if you know exactly what you do! + + +Template +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "RSYSLOG_FileFormat", "no", "none" + +Template to be used for the mail body (if enabled). + +The *template.subject* and *template.text* parameters cannot be given together +inside a single action definition. Use either one of them. If none is used, +a more or less meaningless mail subject is generated (we don't tell you the exact +text because that can change - if you want to have something specific, configure it!). + + +Caveats/Known Bugs +================== + +The current ommail implementation supports SMTP-direct mode only. In +that mode, the plugin talks to the mail server via SMTP protocol. No +other process is involved. This mode offers best reliability as it is +not depending on any external entity except the mail server. Mail server +downtime is acceptable if the action is put onto its own action queue, +so that it may wait for the SMTP server to come back online. However, +the module implements only the bare SMTP essentials. Most importantly, +it does not provide any authentication capabilities. So your mail server +must be configured to accept incoming mail from ommail without any +authentication needs (this may be change in the future as need arises, +but you may also be referred to sendmail-mode). + +In theory, ommail should also offer a mode where it uses the sendmail +utility to send its mail (sendmail-mode). This is somewhat less reliable +(because we depend on an entity we do not have close control over - +sendmail). It also requires dramatically more system resources, as we +need to load the external process (but that should be no problem given +the expected infrequent number of calls into this plugin). The big +advantage of sendmail mode is that it supports all the bells and +whistles of a full-blown SMTP implementation and may even work for local +delivery without a SMTP server being present. Sendmail mode will be +implemented as need arises. So if you need it, please drop us a line (If +nobody does, sendmail mode will probably never be implemented). + + +Examples +======== + +Example 1 +--------- + +The following example alerts the operator if the string "hard disk fatal +failure" is present inside a syslog message. The mail server at +mail.example.net is used and the subject shall be "disk problem on +<hostname>". Note how \\r\\n is included inside the body text to create +line breaks. A message is sent at most once every 6 hours (21600 seconds), +any other messages are silently discarded (or, to be precise, not being +forwarded - they are still being processed by the rest of the configuration +file). + +.. code-block:: none + + module(load="ommail") + + template (name="mailBody" type="string" string="RSYSLOG Alert\\r\\nmsg='%msg%'") + template (name="mailSubject" type="string" string="disk problem on %hostname%") + + if $msg contains "hard disk fatal failure" then { + action(type="ommail" server="mail.example.net" port="25" + mailfrom="rsyslog@example.net" + mailto="operator@example.net" + subject.template="mailSubject" + action.execonlyonceeveryinterval="21600") + } + + +Example 2 +--------- + +The following example is exactly like the first one, but it sends the mails +to two different email addresses: + +.. code-block:: none + + module(load="ommail") + + template (name="mailBody" type="string" string="RSYSLOG Alert\\r\\nmsg='%msg%'") + template (name="mailSubject" type="string" string="disk problem on %hostname%") + + if $msg contains "hard disk fatal failure" then { + action(type="ommail" server="mail.example.net" port="25" + mailfrom="rsyslog@example.net" + mailto=["operator@example.net", "admin@example.net"] + subject.template="mailSubject" + action.execonlyonceeveryinterval="21600") + } + + +Example 3 +--------- + +Note the array syntax to specify email addresses. Note that while rsyslog +permits you to specify as many recipients as you like, your mail server +may limit their number. It is usually a bad idea to use more than 50 +recipients, and some servers may have lower limits. If you hit such a limit, +you could either create additional actions or (recommended) create an +email distribution list. + +The next example is again mostly equivalent to the previous one, but it uses a +constant subject line, so no subject template is required: + +.. code-block:: none + + module(load="ommail") + + template (name="mailBody" type="string" string="RSYSLOG Alert\\r\\nmsg='%msg%'") + + if $msg contains "hard disk fatal failure" then { + action(type="ommail" server="mail.example.net" port="25" + mailfrom="rsyslog@example.net" + mailto=["operator@example.net", "admin@example.net"] + subject.text="rsyslog detected disk problem" + action.execonlyonceeveryinterval="21600") + } + + +Additional Resources +==================== + +A more advanced example plus a discussion on using the email feature +inside a reliable system can be found in Rainer's blogpost "`Why is +native email capability an advantage for a +syslogd? <http://rgerhards.blogspot.com/2008/04/why-is-native-email-capability.html>`_\ " + + diff --git a/source/configuration/modules/ommongodb.rst b/source/configuration/modules/ommongodb.rst new file mode 100644 index 0000000..f048aa1 --- /dev/null +++ b/source/configuration/modules/ommongodb.rst @@ -0,0 +1,247 @@ +******************************** +ommongodb: MongoDB Output Module +******************************** + +=========================== =========================================================================== +**Module Name:** **ommongodb** +**Author:** `Rainer Gerhards <https://rainer.gerhards.net/>`_ <rgerhards@adiscon.com> +=========================== =========================================================================== + + +Purpose +======= + +This module provides native support for logging to MongoDB. + + +Configuration Parameters +======================== + +.. note:: + + Parameter names are case-insensitive. + + +Action Parameters +----------------- + +UriStr +^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +MongoDB connexion string, as defined by the MongoDB String URI Format (See: https://docs.mongodb.com/manual/reference/connection-string/). If uristr is defined, following directives will be ignored: server, serverport, uid, pwd. + + +SSL_Cert +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +Absolute path to the X509 certificate you want to use for TLS client authentication. This is optional. + + +SSL_Ca +^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +Absolute path to the trusted X509 CA certificate that signed the mongoDB server certificate. This is optional. + + +db +^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "syslog", "no", "none" + +Database to use. + + +Collection +^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "log", "no", "none" + +Collection to use. + + +Allowed_Error_Codes +^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "array", "no", "no", "none" + +The list of error codes returned by MongoDB you want ommongodb to ignore. +Please use the following format: allowed_error_codes=["11000","47"]. + + +Template +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "OMSR_TPL_AS_MSG", "no", "none" + +Template to use when submitting messages. + +Note rsyslog contains a canned default template to write to the MongoDB. +It will be used automatically if no other template is specified to be +used. This template is: + +.. code-block:: none + + template(name="BSON" type="string" string="\\"sys\\" : \\"%hostname%\\", + \\"time\\" : \\"%timereported:::rfc3339%\\", \\"time\_rcvd\\" : + \\"%timegenerated:::rfc3339%\\", \\"msg\\" : \\"%msg%\\", + \\"syslog\_fac\\" : \\"%syslogfacility%\\", \\"syslog\_server\\" : + \\"%syslogseverity%\\", \\"syslog\_tag\\" : \\"%syslogtag%\\", + \\"procid\\" : \\"%programname%\\", \\"pid\\" : \\"%procid%\\", + \\"level\\" : \\"%syslogpriority-text%\\"") + + +This creates the BSON document needed for MongoDB if no template is +specified. The default schema is aligned to CEE and project lumberjack. +As such, the field names are standard lumberjack field names, and +**not** `rsyslog property names <property_replacer.html>`_. When +specifying templates, be sure to use rsyslog property names as given in +the table. If you would like to use lumberjack-based field names inside +MongoDB (which probably is useful depending on the use case), you need +to select fields names based on the lumberjack schema. If you just want +to use a subset of the fields, but with lumberjack names, you can look +up the mapping in the default template. For example, the lumberjack +field "level" contains the rsyslog property "syslogpriority-text". + + +Examples +======== + + +Write to Database +----------------- + +The following sample writes all syslog messages to the database "syslog" +and into the collection "log" on mongoserver.example.com. The server is +being accessed under the account of "user" with password "pwd". Please note +that this syntax is deprecated by the "uristr" directive, as shown below. + +.. code-block:: none + + module(load="ommongodb") + action(type="ommongodb" + server="mongoserver.example.com" db="syslog" collection="log" + uid="user" pwd="pwd") + + +Write to mongoDB server with TLS and client authentication +---------------------------------------------------------- + +Another sample that uses the new "uristr" directives to connect to a TLS mongoDB server with TLS and client authentication. + +.. code-block:: none + + module(load="ommongodb") + action(type="ommongodb" + uristr="mongodb://vulture:9091,vulture2:9091/?replicaset=Vulture&ssl=true" + ssl_cert="/var/db/mongodb/mongod.pem" + ssl_ca="/var/db/mongodb/ca.pem" + db="logs" collection="syslog") + + +Deprecated Parameters +===================== + +.. note:: + + While these parameters are still accepted, they should no longer be used for newly created configurations. + + +Action Parameters +----------------- + +Server +^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "127.0.0.1", "no", "none" + +Name or address of the MongoDB server. + + +ServerPorted +^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "27017", "no", "none" + +Permits to select a non-standard port for the MongoDB server. The +default is 0, which means the system default port is used. There is +no need to specify this parameter unless you know the server is +running on a non-standard listen port. + + +UID +^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +Logon userid used to connect to server. Must have proper permissions. + + +PWD +^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +The user's password. + + diff --git a/source/configuration/modules/ommysql.rst b/source/configuration/modules/ommysql.rst new file mode 100644 index 0000000..12d2787 --- /dev/null +++ b/source/configuration/modules/ommysql.rst @@ -0,0 +1,201 @@ +************************************* +ommysql: MySQL Database Output Module +************************************* + +=========================== =========================================================================== +**Module Name:** **ommysql** +**Author:** Michael Meckelein (Initial Author) / `Rainer Gerhards <https://rainer.gerhards.net/>`_ <rgerhards@adiscon.com> +=========================== =========================================================================== + + +Purpose +======= + +This module provides native support for logging to MySQL databases. It +offers superior performance over the more generic +`omlibdbi <omlibdbi.html>`_ module. + + +Configuration Parameters +======================== + +.. note:: + + Parameter names are case-insensitive. + + +Action Parameters +----------------- + +Server +^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "yes", "none" + +This is the address of the MySQL-Server. + + +Socket +^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +This is the unix socket path of the MySQL-Server. When the server +address is set localhost, the mysql client library connects using +the default unix socket specified at build time. +If you run mysql server and run the unix socket path differently +than the default, you can set the socket path with this option. + + +db +^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "yes", "none" + +This is the name of the database used. + + +UID +^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "yes", "none" + +This is the user who is used. + + +PWD +^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "yes", "none" + +This is the password for the user specified in UID. + + +ServerPort +^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "none", "no", "``$ActionOmmysqlServerPort``" + +Permits to select a non-standard port for the MySQL server. The +default is 0, which means the system default port is used. There is +no need to specify this parameter unless you know the server is +running on a non-standard listen port. + + +MySQLConfig.File +^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "``$OmMySQLConfigFile``" + +Permits the selection of an optional MySQL Client Library +configuration file (my.cnf) for extended configuration functionality. +The use of this configuration parameter is necessary only if you have +a non-standard environment or if fine-grained control over the +database connection is desired. + + +MySQLConfig.Section +^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "``$OmMySQLConfigSection``" + +Permits the selection of the section within the configuration file +specified by the **$OmMySQLConfigFile** parameter. +This will likely only be used where the database administrator +provides a single configuration file with multiple profiles. +This configuration parameter is ignored unless **$OmMySQLConfigFile** +is also used in the rsyslog configuration file. +If omitted, the MySQL Client Library default of "client" will be +used. + + +Template +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "StdDBFmt", "no", "none" + +Rsyslog contains a canned default template to write to the MySQL +database. It works on the MonitorWare schema. This template is: + +.. code-block:: none + + $template tpl,"insert into SystemEvents (Message, Facility, FromHost, + Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values + ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, + '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', %iut%, + '%syslogtag%')",SQL + + +As you can see, the template is an actual SQL statement. Note the ",SQL" +option: it tells the template processor that the template is used for +SQL processing, thus quote characters are quoted to prevent security +issues. You can not assign a template without ",SQL" to a MySQL output +action. + +If you would like to change fields contents or add or delete your own +fields, you can simply do so by modifying the schema (if required) and +creating your own custom template. + + +Examples +======== + +Example 1 +--------- + +The following sample writes all syslog messages to the database +"syslog_db" on mysqlserver.example.com. The server is being accessed +under the account of "user" with password "pwd". + +.. code-block:: none + + module(load="ommysql") + action(type="ommysql" server="mysqlserver.example.com" serverport="1234" + db="syslog_db" uid="user" pwd="pwd") + + diff --git a/source/configuration/modules/omoracle.rst b/source/configuration/modules/omoracle.rst new file mode 100644 index 0000000..4133eb7 --- /dev/null +++ b/source/configuration/modules/omoracle.rst @@ -0,0 +1,200 @@ +omoracle: Oracle Database Output Module +======================================= + +**Module Name: omoracle** + +**Author:**\ Luis Fernando Muñoz Mejías +<Luis.Fernando.Munoz.Mejias@cern.ch> - this module is currently +orphaned, the original author does no longer support it. + +**Available since:**: 4.3.0, **does not work with recent rsyslog +versions (v7 and up). Use** :doc:`omlibdbi <omlibdbi>` **instead.** +An upgrade to the new interfaces is needed. If you would like +to contribute, please send us a patch or open a github pull request. + +**Status:**: contributed module, not maintained by rsyslog core authors + +**Description**: + +This module provides native support for logging to Oracle databases. It +offers superior performance over the more generic +`omlibdbi <omlibdbi.html>`_ module. It also includes a number of +enhancements, most importantly prepared statements and batching, what +provides a big performance improvement. + +Note that this module is maintained by its original author. If you need +assistance with it, it is suggested to post questions to the `rsyslog +mailing list <http://lists.adiscon.net/mailman/listinfo/rsyslog>`_. + +From the header comments of this module: + +:: + + + This is an output module feeding directly to an Oracle + database. It uses Oracle Call Interface, a propietary module + provided by Oracle. + + Selector lines to be used are of this form: + + :omoracle:;TemplateName + + The module gets its configuration via rsyslog $... directives, + namely: + + $OmoracleDBUser: user name to log in on the database. + + $OmoracleDBPassword: password to log in on the database. + + $OmoracleDB: connection string (an Oracle easy connect or a db + name as specified by tnsnames.ora) + + $OmoracleBatchSize: Number of elements to send to the DB on each + transaction. + + $OmoracleStatement: Statement to be prepared and executed in + batches. Please note that Oracle's prepared statements have their + placeholders as ':identifier', and this module uses the colon to + guess how many placeholders there will be. + + All these directives are mandatory. The dbstring can be an Oracle + easystring or a DB name, as present in the tnsnames.ora file. + + The form of the template is just a list of strings you want + inserted to the DB, for instance: + + $template TestStmt,"%hostname%%msg%" + + Will provide the arguments to a statement like + + $OmoracleStatement \ + insert into foo(hostname,message)values(:host,:message) + + Also note that identifiers to placeholders are arbitrary. You + need to define the properties on the template in the correct order + you want them passed to the statement! + +Some additional documentation contributed by Ronny Egner: + +:: + + REQUIREMENTS: + -------------- + + - Oracle Instantclient 10g (NOT 11g) Base + Devel + (if you´re on 64-bit linux you should choose the 64-bit libs!) + - JDK 1.6 (not neccessary for oracle plugin but "make" didd not finsished successfully without it) + + - "oracle-instantclient-config" script + (seems to shipped with instantclient 10g Release 1 but i was unable to find it for 10g Release 2 so here it is) + + + ====================== /usr/local/bin/oracle-instantclient-config ===================== + #!/bin/sh + # + # Oracle InstantClient SDK config file + # Jean-Christophe Duberga - Bordeaux 2 University + # + + # just adapt it to your environment + incdirs="-I/usr/include/oracle/10.2.0.4/client64" + libdirs="-L/usr/lib/oracle/10.2.0.4/client64/lib" + + usage="\ + Usage: oracle-instantclient-config [--prefix[=DIR]] [--exec-prefix[=DIR]] [--version] [--cflags] [--libs] [--static-libs]" + + if test $# -eq 0; then + echo "${usage}" 1>&2 + exit 1 + fi + + while test $# -gt 0; do + case "$1" in + -*=*) optarg=`echo "$1" | sed 's/[-_a-zA-Z0-9]*=//'` ;; + *) optarg= ;; + esac + + case $1 in + --prefix=*) + prefix=$optarg + if test $exec_prefix_set = no ; then + exec_prefix=$optarg + fi + ;; + --prefix) + echo $prefix + ;; + --exec-prefix=*) + exec_prefix=$optarg + exec_prefix_set=yes + ;; + --exec-prefix) + echo ${exec_prefix} + ;; + --version) + echo ${version} + ;; + --cflags) + echo ${incdirs} + ;; + --libs) + echo $libdirs -lclntsh -lnnz10 -locci -lociei -locijdbc10 + ;; + --static-libs) + echo "No static libs" 1>&2 + exit 1 + ;; + *) + echo "${usage}" 1>&2 + exit 1 + ;; + esac + shift + done + + =============== END ============== + + + + + COMPILING RSYSLOGD + ------------------- + + + ./configure --enable-oracle + + + + + RUNNING + ------- + + - make sure rsyslogd is able to locate the oracle libs (either via LD_LIBRARY_PATH or /etc/ld.so.conf) + - set TNS_ADMIN to point to your tnsnames.ora + - create a tnsnames.ora and test you are able to connect to the database + + - create user in oracle as shown in the following example: + create user syslog identified by syslog default tablespace users quota unlimited on users; + grant create session to syslog; + create role syslog_role; + grant syslog_role to syslog; + grant create table to syslog_role; + grant create sequence to syslog_role; + + - create tables as needed + + - configure rsyslog as shown in the following example + $ModLoad omoracle + + $OmoracleDBUser syslog + $OmoracleDBPassword syslog + $OmoracleDB syslog + $OmoracleBatchSize 1 + $OmoracleBatchItemSize 4096 + + $OmoracleStatementTemplate OmoracleStatement + $template OmoracleStatement,"insert into foo(hostname,message) values (:host,:message)" + $template TestStmt,"%hostname%%msg%" + *.* :omoracle:;TestStmt + (you guess it: username = password = database = "syslog".... see $rsyslogd_source/plugins/omoracle/omoracle.c for me info) + diff --git a/source/configuration/modules/ompgsql.rst b/source/configuration/modules/ompgsql.rst new file mode 100644 index 0000000..4df4745 --- /dev/null +++ b/source/configuration/modules/ompgsql.rst @@ -0,0 +1,239 @@ +.. index:: ! ompgsql + +******************************************* +PostgreSQL Database Output Module (ompgsql) +******************************************* + +================ ========================================================================== +**Module Name:** ompgsql +**Author:** `Rainer Gerhards <rgerhards@adiscon.com>`__ and `Dan Molik <dan@danmolik.com>`__ +**Available:** 8.32+ +================ ========================================================================== + + +Purpose +======= + +This module provides native support for logging to PostgreSQL databases. +It's an alternative (with potentially superior performance) to the more +generic :doc:`omlibdbi <omlibdbi>` module. + + +Configuration Parameters +======================== + +.. note:: + + Parameter names are case-insensitive. + + +Action Parameters +----------------- + +Conninfo +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +The URI or set of key-value pairs that describe how to connect to the PostgreSQL +server. This takes precedence over ``server``, ``port``, ``db``, and ``pass`` +parameters. Required if ``server`` and ``db`` are not specified. + +The format corresponds to `standard PostgreSQL connection string format +<https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING>`_. + +Server +^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +The hostname or address of the PostgreSQL server. Required if ``conninfo`` is +not specified. + + +Port/Serverport +^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "5432", "no", "none" + +The IP port of the PostgreSQL server. + + +db +^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +The multi-tenant database name to ``INSERT`` rows into. Required if ``conninfo`` +is not specified. + + +User/UID +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "postgres", "no", "none" + +The username to connect to the PostgreSQL server with. + + +Pass/PWD +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "postgres", "no", "none" + +The password to connect to the PostgreSQL server with. + + +Template +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +The template name to use to ``INSERT`` rows into the database with. Valid SQL +syntax is required, as the module does not perform any insertion statement +checking. + + +Examples +======== + +Example 1 +--------- + +A Basic Example using the internal PostgreSQL template. + +.. code-block:: none + + # load module + module(load="ompgsql") + + action(type="ompgsql" server="localhost" + user="rsyslog" pass="test1234" + db="syslog") + + +Example 2 +--------- + +A Basic Example using the internal PostgreSQL template and connection using URI. + +.. code-block:: none + + # load module + module(load="ompgsql") + + action(type="ompgsql" + conninfo="postgresql://rsyslog:test1234@localhost/syslog") + + +Example 3 +--------- + +A Basic Example using the internal PostgreSQL template and connection with TLS using URI. + +.. code-block:: none + + # load module + module(load="ompgsql") + + action(type="ompgsql" + conninfo="postgresql://rsyslog:test1234@postgres.example.com/syslog?sslmode=verify-full&sslrootcert=/path/to/cert") + + +Example 4 +--------- + +A Templated example. + +.. code-block:: none + + template(name="sql-syslog" type="list" option.stdsql="on") { + constant(value="INSERT INTO SystemEvents (message, timereported) values ('") + property(name="msg") + constant(value="','") + property(name="timereported" dateformat="pgsql" date.inUTC="on") + constant(value="')") + } + + # load module + module(load="ompgsql") + + action(type="ompgsql" server="localhost" + user="rsyslog" pass="test1234" + db="syslog" + template="sql-syslog" ) + + +Example 5 +--------- + +An action queue and templated example. + +.. code-block:: none + + template(name="sql-syslog" type="list" option.stdsql="on") { + constant(value="INSERT INTO SystemEvents (message, timereported) values ('") + property(name="msg") + constant(value="','") + property(name="timereported" dateformat="pgsql" date.inUTC="on") + constant(value="')") + } + + # load module + module(load="ompgsql") + + action(type="ompgsql" server="localhost" + user="rsyslog" pass="test1234" + db="syslog" + template="sql-syslog" + queue.size="10000" queue.type="linkedList" + queue.workerthreads="5" + queue.workerthreadMinimumMessages="500" + queue.timeoutWorkerthreadShutdown="1000" + queue.timeoutEnqueue="10000") + + +Building +======== + +To compile Rsyslog with PostgreSQL support you will need to: + +* install *libpq* and *libpq-dev* packages, check your package manager for the correct name. +* set *--enable-pgsql* switch on configure. + + diff --git a/source/configuration/modules/ompipe.rst b/source/configuration/modules/ompipe.rst new file mode 100644 index 0000000..1bf581e --- /dev/null +++ b/source/configuration/modules/ompipe.rst @@ -0,0 +1,48 @@ +ompipe: Pipe Output Module +========================== + +**Module Name: ompipe** + +**Author:**\ Rainer Gerhards <rgerhards@adiscon.com> + +**Description**: + +The ompipe plug-in provides the core functionality for logging output to named pipes (fifos). It is a built-in module that does not need to be loaded. + +**Global Configuration Parameters:** + +Note: parameter names are case-insensitive. + +- Template: [templateName] sets a new default template for file actions. + +**Action specific Configuration Parameters:** + +Note: parameter names are case-insensitive. + +- Pipe: string a fifo or named pipe can be used as a destination for log messages. +- tryResumeReopen: Sometimes we need to reopen a pipe after an ompipe action gets suspended. Sending an HUP signal does the job but requires an interaction with rsyslog. When set to "on" and a resume action fails, the file descriptor is closed, causing a new open in the next resume. Default: "off" to preserve existing behavior before introduction of this option. + +**Caveats/Known Bugs:** +None + +**Sample:** +The following command sends all syslog messages to a pipe named "NameofPipe". + +:: + + Module (path="builtin:ompipe") + *.* action(type="ompipe" Pipe="NameofPipe") + +**Legacy Configuration Parameters:** + +rsyslog has support for logging output to named pipes (fifos). A fifo or named pipe can be used as a destination for log messages by prepending a pipe symbol ("|") to the name of the file. This is handy for debugging. Note that the fifo must be created with the mkfifo(1) command before rsyslogd is started. + +**Legacy Sample:** + +The following command sends all syslog messages to a pipe named /var/log/pipe. + +:: + + $ModLoad ompipe + *.* |/var/log/pipe + diff --git a/source/configuration/modules/omprog.rst b/source/configuration/modules/omprog.rst new file mode 100644 index 0000000..d96fec1 --- /dev/null +++ b/source/configuration/modules/omprog.rst @@ -0,0 +1,530 @@ +***************************************** +omprog: Program integration Output module +***************************************** + +=========================== =========================================================================== +**Module Name:** **omprog** +**Author:** `Rainer Gerhards <https://rainer.gerhards.net/>`_ <rgerhards@adiscon.com> +=========================== =========================================================================== + + +Purpose +======= + +This module permits to integrate arbitrary external programs into +rsyslog's logging. It is similar to the "execute program (^)" action, +but offers better security and much higher performance. While "execute +program (^)" can be a useful tool for executing programs if rare events +occur, omprog can be used to provide massive amounts of log data to a +program. + +Executes the configured program and feeds log messages to that binary +via stdin. The binary is free to do whatever it wants with the supplied +data. If the program terminates, it is re-started. If rsyslog +terminates, the program's stdin will see EOF. The program must then +terminate. The message format passed to the program can, as usual, be +modified by defining rsyslog templates. + +Note that in order to execute the given program, rsyslog needs to have +sufficient permissions on the binary file. This is especially true if +not running as root. Also, keep in mind that default SELinux policies +most probably do not permit rsyslogd to execute arbitrary binaries. As +such, permissions must be appropriately added. Note that SELinux +restrictions also apply if rsyslogd runs under root. To check if a +problem is SELinux-related, you can temporarily disable SELinux and +retry. If it then works, you know for sure you have a SELinux issue. + +Starting with 8.4.0, rsyslogd emits an error message via the ``syslog()`` +API call when there is a problem executing the binary. This can be +extremely valuable in troubleshooting. For those technically savvy: +when we execute a binary, we need to fork, and we do not have +full access to rsyslog's usual error-reporting capabilities after the +fork. As the actual execution must happen after the fork, we cannot +use the default error logger to emit the error message. As such, +we use ``syslog()``. In most cases, there is no real difference +between both methods. However, if you run multiple rsyslog instances, +the message shows up in that instance that processes the default +log socket, which may be different from the one where the error occurred. +Also, if you redirected the log destination, that redirection may +not work as expected. + + +Configuration Parameters +======================== + +.. note:: + + Parameter names are case-insensitive. + + +Action Parameters +----------------- + +template +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "RSYSLOG_FileFormat", "no", "none" + +Name of the :doc:`template <../templates>` to use to format the log messages +passed to the external program. + + +binary +^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "", "yes", "``$ActionOMProgBinary``" + +Full path and command line parameters of the external program to execute. +Arbitrary external programs should be placed under the /usr/libexec/rsyslog directory. +That is, the binaries put in this namespaced directory are meant for the consumption +of rsyslog, and are not intended to be executed by users. +In legacy config, it is **not possible** to specify command line parameters. + + +.. _confirmMessages: + +confirmMessages +^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +.. versionadded:: 8.31.0 + +Specifies whether the external program provides feedback to rsyslog via stdout. +When this switch is set to "on", rsyslog will wait for the program to confirm +each received message. This feature facilitates error handling: instead of +having to implement a retry logic, the external program can rely on the rsyslog +queueing capabilities. + +To confirm a message, the program must write a line with the word ``OK`` to its +standard output. If it writes a line containing anything else, rsyslog considers +that the message could not be processed, keeps it in the action queue, and +re-sends it to the program later (after the period specified by the +:doc:`action.resumeInterval <../actions>` parameter). + +In addition, when a new instance of the program is started, rsyslog will also +wait for the program to confirm it is ready to start consuming logs. This +prevents rsyslog from starting to send logs to a program that could not +complete its initialization properly. + +.. seealso:: + + `Interface between rsyslog and external output plugins + <https://github.com/rsyslog/rsyslog/blob/master/plugins/external/INTERFACE.md>`_ + + +.. _confirmTimeout: + +confirmTimeout +^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "10000", "no", "none" + +.. versionadded:: 8.38.0 + +Specifies how long rsyslog must wait for the external program to confirm +each message when confirmMessages_ is set to "on". If the program does not +send a response within this timeout, it will be restarted (see signalOnClose_, +closeTimeout_ and killUnresponsive_ for details on the cleanup sequence). +The value must be expressed in milliseconds and must be greater than zero. + +.. seealso:: + + `Interface between rsyslog and external output plugins + <https://github.com/rsyslog/rsyslog/blob/master/plugins/external/INTERFACE.md>`_ + + +.. _reportFailures: + +reportFailures +^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +.. versionadded:: 8.38.0 + +Specifies whether rsyslog must internally log a warning message whenever the +program returns an error when confirming a message. The logged message will +include the error line returned by the program. This parameter is ignored when +confirmMessages_ is set to "off". + +Enabling this flag can be useful to log the problems detected by the program. +However, the information that can be logged is limited to a short error line, +and the logs will be tagged as originated by the 'syslog' facility (like the +rest of rsyslog logs). To avoid these shortcomings, consider the use of the +output_ parameter to capture the stderr of the program. + + +.. _useTransactions: + +useTransactions +^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +.. versionadded:: 8.31.0 + +Specifies whether the external program processes the messages in +:doc:`batches <../../development/dev_oplugins>` (transactions). When this +switch is enabled, the logs sent to the program are grouped in transactions. +At the start of a transaction, rsyslog sends a special mark message to the +program (see beginTransactionMark_). At the end of the transaction, rsyslog +sends another mark message (see commitTransactionMark_). + +If confirmMessages_ is also set to "on", the program must confirm both the +mark messages and the logs within the transaction. The mark messages must be +confirmed by returning ``OK``, and the individual messages by returning +``DEFER_COMMIT`` (instead of ``OK``). Refer to the link below for details. + +.. seealso:: + + `Interface between rsyslog and external output plugins + <https://github.com/rsyslog/rsyslog/blob/master/plugins/external/INTERFACE.md>`_ + +.. warning:: + + This feature is currently **experimental**. It could change in future releases + without keeping backwards compatibility with existing configurations or the + specified interface. There is also a `known issue + <https://github.com/rsyslog/rsyslog/issues/2420>`_ with the use of + transactions together with ``confirmMessages=on``. + + +.. _beginTransactionMark: + +beginTransactionMark +^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "BEGIN TRANSACTION", "no", "none" + +.. versionadded:: 8.31.0 + +Allows specifying the mark message that rsyslog will send to the external +program to indicate the start of a transaction (batch). This parameter is +ignored if useTransactions_ is disabled. + + +.. _commitTransactionMark: + +commitTransactionMark +^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "COMMIT TRANSACTION", "no", "none" + +.. versionadded:: 8.31.0 + +Allows specifying the mark message that rsyslog will send to the external +program to indicate the end of a transaction (batch). This parameter is +ignored if useTransactions_ is disabled. + + +.. _output: + +output +^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "none", "no", "none" + +.. versionadded:: v8.1.6 + +Full path of a file where the output of the external program will be saved. +If the file already exists, the output is appended to it. If the file does +not exist, it is created with the permissions specified by fileCreateMode_. + +If confirmMessages_ is set to "off" (the default), both the stdout and +stderr of the child process are written to the specified file. + +If confirmMessages_ is set to "on", only the stderr of the child is +written to the specified file (since stdout is used for confirming the +messages). + +Rsyslog will reopen the file whenever it receives a HUP signal. This allows +the file to be externally rotated (using a tool like *logrotate*): after +each rotation of the file, make sure a HUP signal is sent to rsyslogd. + +If the omprog action is configured to use multiple worker threads +(:doc:`queue.workerThreads <../../rainerscript/queue_parameters>` is +set to a value greater than 1), the lines written by the various program +instances will not appear intermingled in the output file, as long as the +lines do not exceed a certain length and the program writes them to +stdout/stderr in line-buffered mode. For details, refer to `Interface between +rsyslog and external output plugins +<https://github.com/rsyslog/rsyslog/blob/master/plugins/external/INTERFACE.md>`_. + +If this parameter is not specified, the output of the program will be +redirected to ``/dev/null``. + +.. note:: + + Before version v8.38.0, this parameter was intended for debugging purposes + only. Since v8.38.0, the parameter can be used for production. + + +.. _fileCreateMode: + +fileCreateMode +^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "0600", "no", "none" + +.. versionadded:: v8.38.0 + +Permissions the output_ file will be created with, in case the file does not +exist. The value must be a 4-digit octal number, with the initial digit being +zero. Please note that the actual permission depends on the rsyslogd process +umask. If in doubt, use ``$umask 0000`` right at the beginning of the +configuration file to remove any restrictions. + + +hup.signal +^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +.. versionadded:: 8.9.0 + +Specifies which signal, if any, is to be forwarded to the external program +when rsyslog receives a HUP signal. Currently, HUP, USR1, USR2, INT, and +TERM are supported. If unset, no signal is sent on HUP. This is the default +and what pre 8.9.0 versions did. + + +.. _signalOnClose: + +signalOnClose +^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +.. versionadded:: 8.23.0 + +Specifies whether a TERM signal must be sent to the external program before +closing it (when either the worker thread has been unscheduled, a restart +of the program is being forced, or rsyslog is about to shutdown). + +If this switch is set to "on", rsyslog will send a TERM signal to the child +process before closing the pipe. That is, the process will first receive a +TERM signal, and then an EOF on stdin. + +No signal is issued if this switch is set to "off" (default). The child +process can still detect it must terminate because reading from stdin will +return EOF. + +See the killUnresponsive_ parameter for more details. + + +.. _closeTimeout: + +closeTimeout +^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "5000", "no", "none" + +.. versionadded:: 8.35.0 + +Specifies how long rsyslog must wait for the external program to terminate +(when either the worker thread has been unscheduled, a restart of the program +is being forced, or rsyslog is about to shutdown) after closing the pipe, +that is, after sending EOF to the stdin of the child process. The value must +be expressed in milliseconds and must be greater than or equal to zero. + +See the killUnresponsive_ parameter for more details. + + +.. _killUnresponsive: + +killUnresponsive +^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "the value of 'signalOnClose'", "no", "none" + +.. versionadded:: 8.35.0 + +Specifies whether a KILL signal must be sent to the external program in case +it does not terminate within the timeout indicated by closeTimeout_ +(when either the worker thread has been unscheduled, a restart of the program +is being forced, or rsyslog is about to shutdown). + +If signalOnClose_ is set to "on", the default value of ``killUnresponsive`` +is also "on". In this case, the cleanup sequence of the child process is as +follows: (1) a TERM signal is sent to the child, (2) the pipe with the child +process is closed (the child will receive EOF on stdin), (3) rsyslog waits +for the child process to terminate during closeTimeout_, (4) if the child +has not terminated within the timeout, a KILL signal is sent to it. + +If signalOnClose_ is set to "off", the default value of ``killUnresponsive`` +is also "off". In this case, the child cleanup sequence is as follows: (1) the +pipe with the child process is closed (the child will receive EOF on stdin), +(2) rsyslog waits for the child process to terminate during closeTimeout_, +(3) if the child has not terminated within the timeout, rsyslog ignores it. + +This parameter can be set to a different value than signalOnClose_, obtaining +the corresponding variations of cleanup sequences described above. + + +forceSingleInstance +^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +.. versionadded:: v8.1.6 + +By default, the omprog action will start an instance (process) of the +external program per worker thread (the maximum number of worker threads +can be specified with the +:doc:`queue.workerThreads <../../rainerscript/queue_parameters>` +parameter). Moreover, if the action is associated to a +:doc:`disk-assisted queue <../../concepts/queues>`, an additional instance +will be started when the queue is persisted, to process the items stored +on disk. + +If you want to force a single instance of the program to be executed, +regardless of the number of worker threads or the queue type, set this +flag to "on". This is useful when the external program uses or accesses +some kind of shared resource that does not allow concurrent access from +multiple processes. + +.. note:: + + Before version v8.38.0, this parameter had no effect. + + +Examples +======== + +Example: command line arguments +------------------------------- + +In the following example, logs will be sent to a program ``log.sh`` located +in ``/usr/libexec/rsyslog``. The program will receive the command line arguments +``p1``, ``p2`` and ``--param3="value 3"``. + +.. code-block:: none + + module(load="omprog") + + action(type="omprog" + binary="/usr/libexec/rsyslog/log.sh p1 p2 --param3=\"value 3\"" + template="RSYSLOG_TraditionalFileFormat") + + +Example: external program that writes logs to a database +-------------------------------------------------------- + +In this example, logs are sent to the stdin of a Python program that +(let's assume) writes them to a database. A dedicated disk-assisted +queue with (a maximum of) 5 worker threads is used, to avoid affecting +other log destinations in moments of high load. The ``confirmMessages`` +flag is enabled, which tells rsyslog to wait for the program to confirm +its initialization and each message received. The purpose of this setup +is preventing logs from being lost because of database connection +failures. + +If the program cannot write a log to the database, it will return a +negative confirmation to rsyslog via stdout. Rsyslog will then keep the +failed log in the queue, and send it again to the program after 5 +seconds. The program can also write error details to stderr, which will +be captured by rsyslog and written to ``/var/log/db_forward.log``. If +no response is received from the program within a 30-second timeout, +rsyslog will kill and restart it. + +.. code-block:: none + + module(load="omprog") + + action(type="omprog" + name="db_forward" + binary="/usr/libexec/rsyslog/db_forward.py" + confirmMessages="on" + confirmTimeout="30000" + queue.type="LinkedList" + queue.saveOnShutdown="on" + queue.workerThreads="5" + action.resumeInterval="5" + killUnresponsive="on" + output="/var/log/db_forward.log") + +Note that the ``useTransactions`` flag is not used in this example. The +program stores and confirms each log individually. + + +|FmtObsoleteName| directives +============================ + +- **$ActionOMProgBinary** <binary> + The binary program to be executed. diff --git a/source/configuration/modules/omrabbitmq.rst b/source/configuration/modules/omrabbitmq.rst new file mode 100644 index 0000000..433abd4 --- /dev/null +++ b/source/configuration/modules/omrabbitmq.rst @@ -0,0 +1,404 @@ +********************************** +omrabbitmq: RabbitMQ output module +********************************** + +=========================== =========================================================================== +**Module Name:** **omrabbitmq** +**Authors:** Jean-Philippe Hilaire <jean-philippe.hilaire@pmu.fr> / Philippe Duveau <philippe.duveau@free.fr> / Hamid Maadani <hamid@dexo.tech> +=========================== =========================================================================== + + +Purpose +======= + +This module sends syslog messages into RabbitMQ server. +Only v6 configuration syntax is supported. + +**omrabbitmq is tested and is running in production with 8.x version of rsyslog.** + +Compile +======= + +To successfully compile omrabbitmq module you need `rabbitmq-c <https://github.com/alanxz/rabbitmq-c>`_ library version >= 0.4. + + ./configure --enable-omrabbitmq ... + +Configuration Parameters +======================== + +Action Parameters +----------------- + +host +^^^^ + +.. csv-table:: + :header: "type", "mandatory", "format", "default" + :widths: auto + :class: parameter-table + + "string", "yes", "hostname\[:port\]\[ hostname2\[:port2\]\]", + +rabbitmq server(s). See HA configuration + +port +^^^^ + +.. csv-table:: + :header: "type", "mandatory", "format", "default" + :widths: auto + :class: parameter-table + + "string", "no", "port", "5672" + +virtual\_host +^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "mandatory", "format", "default" + :widths: auto + :class: parameter-table + + "string", "yes", "path", + +virtual message broker + +user +^^^^ + +.. csv-table:: + :header: "type", "mandatory", "format", "default" + :widths: auto + :class: parameter-table + + "string", "yes", "user", + +user name + +password +^^^^^^^^ + +.. csv-table:: + :header: "type", "mandatory", "format", "default" + :widths: auto + :class: parameter-table + + "string", "yes", "password", + +user password + +ssl +^^^ + +.. csv-table:: + :header: "type", "mandatory", "format", "default" + :widths: auto + :class: parameter-table + + "binary", "no", , "off" + +enable TLS for AMQP connection + +init_openssl +^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "mandatory", "format", "default" + :widths: auto + :class: parameter-table + + "binary", "no", , "off" + +should rabbitmq-c initialize OpenSSL? This is included to prevent crashes caused by OpenSSL double initialization. Should stay off in most cases. ONLY turn on if SSL does not work due to OpenSSL not being initialized. + +verify_peer +^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "mandatory", "format", "default" + :widths: auto + :class: parameter-table + + "binary", "no", , "off" + +SSL peer verification + +verify_hostname +^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "mandatory", "format", "default" + :widths: auto + :class: parameter-table + + "binary", "no", , "off" + +SSL certificate hostname verification + +ca_cert +^^^^^^^ + +.. csv-table:: + :header: "type", "mandatory", "format", "default" + :widths: auto + :class: parameter-table + + "string", "no", "file path", + +CA certificate to be used for the SSL connection + +heartbeat_interval +^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "mandatory", "format", "default" + :widths: auto + :class: parameter-table + + "integer", "no", , "0" + +AMQP heartbeat interval in seconds. 0 means disabled, which is default. + +exchange +^^^^^^^^ + +.. csv-table:: + :header: "type", "mandatory", "format", "default" + :widths: auto + :class: parameter-table + + "string", "no", "name", + +exchange name + +routing\_key +^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "mandatory", "format", "default" + :widths: auto + :class: parameter-table + + "string", "no", "name", + +value of routing key + +routing\_key\_template +^^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "mandatory", "format", "default" + :widths: auto + :class: parameter-table + + "string", "no", "template_name", + +template used to compute the routing key + +body\_template +^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "mandatory", "format", "default" + :widths: auto + :class: parameter-table + + "string", "no", "template", "StdJSONFmt" + +template used to compute the message body. If the template is an empty string the sent message will be %rawmsg% + +delivery\_mode +^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "mandatory", "format", "default" + :widths: auto + :class: parameter-table + + "string", "no", "TRANSIENT\|PERSISTANT", "TRANSIENT" + +persistence of the message in the broker + +expiration +^^^^^^^^^^ + +.. csv-table:: + :header: "type", "mandatory", "format", "default" + :widths: auto + :class: parameter-table + + "string", "no", "milliseconds", no expiration + +ttl of the amqp message + +populate\_properties +^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "mandatory", "format", "default" + :widths: auto + :class: parameter-table + + "binary", "no", , "off" + +fill timestamp, appid, msgid, hostname (custom header) with message informations + +content\_type +^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "mandatory", "format", "default" + :widths: auto + :class: parameter-table + + "string", "no", "value", + +content type as a MIME value + +declare\_exchange +^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "mandatory", "format", "default" + :widths: auto + :class: parameter-table + + "binary", "no", "off", + +Rsyslog tries to declare the exchange on startup. Declaration failure (already exists with different parameters or insufficient rights) is warned but does not cancel the module instance. + +recover\_policy +^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "mandatory", "format", "default" + :widths: auto + :class: parameter-table + + "string", "no", "check\_interval;short\_failure_interval; short\_failure\_nb\_max;graceful\_interval", "60;6;3;600" + +See HA configuration + +HA configuration +================ + +The module can use two rabbitmq server in a fail-over mode. To configure this mode, the host parameter has to reference the two rabbitmq servers separated by space. +Each server can be optionally completed with the port (useful when they are different). +One of the servers is chosen on startup as a preferred one. The module connects to this server with a fail-over policy which can be defined through the action parameter "recover_policy". + +The module launch a back-ground thread to monitor the connection. As soon as the connection fails, the thread retries to reestablish the connection and switch to the back-up server if needed to recover the service. While connected to backup server, the thread tries to reconnect to the preferred server using a "recover_policy". This behaviour allow to load balance the client across the two rabbitmq servers on normal conditions, switch to the running server in case of failure and rebalanced on the two server as soon as the failed server is recovered without restarting clients. + +The recover policy is based on 4 parameters : + +- `check_interval` is the base duration between connection retries (default is 60 seconds) + +- `short_failure_interval` is a duration under which two successive failures are considered as abnormal for the rabbitmq server (default is `check_interval/10`) + +- `short_failure_nb_max` is the number of successive short failure are detected before to apply the graceful interval (default is 3) + +- `graceful_interval` is a longer duration used if the rabbitmq server is unstable (default is `check_interval*10`). + +The short failures detection is applied in case of unstable network or server and force to switch to back-up server for at least 'graceful-interval' avoiding heavy load on the unstable server. This can avoid dramatic scenarios in a multisites deployment. + +Examples +======== + +Example 1 +--------- + +This is the simplest action : + +- No High Availability + +- The routing-key is constant + +- The sent message use JSON format + +.. code-block:: none + + module(load='omrabbitmq') + action(type="omrabbitmq" + host="localhost" + virtual_host="/" + user="guest" + password="guest" + exchange="syslog" + routing_key="syslog.all") + +Example 2 +--------- + +Action characteristics : + +- No High Availability + +- The routing-key is computed + +- The sent message is a raw message + +.. code-block:: none + + module(load='omrabbitmq') + template(name="rkTpl" type="string" string="%syslogtag%.%syslogfacility-text%.%syslogpriority-text%") + + action(type="omrabbitmq" + host="localhost" + virtual_host="/" + user="guest" + password="guest" + exchange="syslog" + routing_key_template="rkTpl" + template_body="") + +Example 3 +--------- + +HA action : + +- High Availability between `server1:5672` and `server2:1234` + +- The routing-key is computed + +- The sent message is formatted using RSYSLOG_ForwardFormat standard template + +.. code-block:: none + + module(load='omrabbitmq') + template(name="rkTpl" type="string" string="%syslogtag%.%syslogfacility-text%.%syslogpriority-text%") + + action(type="omrabbitmq" + host="server1 server2:1234" + virtual_host="production" + user="guest" + password="guest" + exchange="syslog" + routing_key_template="rkTpl" + template_body="RSYSLOG_ForwardFormat") + +Example 4 +--------- + +SSL enabled connection, with Heartbeat : + +- No High Availability + +- The routing-key is constant + +- The sent message use JSON format + +- Heartbeat is set to 20 seconds + +.. code-block:: none + + module(load='omrabbitmq') + action(type="omrabbitmq" + host="localhost" + virtual_host="/" + user="guest" + password="guest" + ssl="on" + verify_peer="off" + verify_hostname="off" + heartbeat_interval="20" + exchange="syslog" + routing_key="syslog.all") diff --git a/source/configuration/modules/omrelp.rst b/source/configuration/modules/omrelp.rst new file mode 100644 index 0000000..11a3fd9 --- /dev/null +++ b/source/configuration/modules/omrelp.rst @@ -0,0 +1,482 @@ +************************** +omrelp: RELP Output Module +************************** + +=========================== =========================================================================== +**Module Name:** **omrelp** +**Author:** `Rainer Gerhards <https://rainer.gerhards.net/>`_ <rgerhards@adiscon.com> +=========================== =========================================================================== + + +Purpose +======= + +This module supports sending syslog messages over the reliable RELP +protocol. For RELP's advantages over plain tcp syslog, please see the +documentation for :doc:`imrelp <imrelp>` (the server counterpart). + +Setup + +Please note that `librelp <http://www.librelp.com>`__ is required for +imrelp (it provides the core relp protocol implementation). + + +Configuration Parameters +======================== + +.. note:: + + Parameter names are case-insensitive. + +Module Parameters +----------------- + +tls.tlslib +^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +.. versionadded:: 8.1903.0 + +Permits to specify the TLS library used by librelp. +All relp protocol operations or actually performed by librelp and +not rsyslog itself. This value specified is directly passed down to +librelp. Depending on librelp version and build parameters, supported +tls libraries differ (or TLS may not be supported at all). In this case +rsyslog emits an error message. + +Usually, the following options should be available: "openssl", "gnutls". + +Note that "gnutls" is the current default for historic reasons. We actually +recommend to use "openssl". It provides better error messages and accepts +a wider range of certificate types. + +If you have problems with the default setting, we recommend to switch to +"openssl". + + +Action Parameters +----------------- + +Target +^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "yes", "none" + +The target server to connect to. + + +Port +^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "514", "no", "none" + +Name or numerical value of TCP port to use when connecting to target. + + +Template +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "RSYSLOG_ForwardFormat", "no", "none" + +Defines the template to be used for the output. + + +Timeout +^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "int", "90", "no", "none" + +Timeout for relp sessions. If set too low, valid sessions may be +considered dead and tried to recover. + + +Conn.Timeout +^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "int", "10", "no", "none" + +Timeout for the socket connection. + + +RebindInterval +^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "int", "0", "no", "none" + +Permits to specify an interval at which the current connection is +broken and re-established. This setting is primarily an aid to load +balancers. After the configured number of messages has been +transmitted, the current connection is terminated and a new one +started. This usually is perceived as a \`\`new connection'' by load +balancers, which in turn forward messages to another physical target +system. + + +WindowSize +^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "int", "0", "no", "none" + +This is an **expert parameter**. It permits to override the RELP +window size being used by the client. Changing the window size has +both an effect on performance as well as potential message +duplication in failure case. A larger window size means more +performance, but also potentially more duplicated messages - and vice +versa. The default 0 means that librelp's default window size is +being used, which is considered a compromise between goals reached. +For your information: at the time of this writing, the librelp +default window size is 128 messages, but this may change at any time. +Note that there is no equivalent server parameter, as the client +proposes and manages the window size in RELP protocol. + + +TLS +^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +If set to "on", the RELP connection will be encrypted by TLS, so +that the data is protected against observers. Please note that both +the client and the server must have set TLS to either "on" or "off". +Other combinations lead to unpredictable results. + +*Attention when using GnuTLS 2.10.x or older* + +Versions older than GnuTLS 2.10.x may cause a crash (Segfault) under +certain circumstances. Most likely when an imrelp inputs and an +omrelp output is configured. The crash may happen when you are +receiving/sending messages at the same time. Upgrade to a newer +version like GnuTLS 2.12.21 to solve the problem. + + +TLS.Compression +^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +The controls if the TLS stream should be compressed (zipped). While +this increases CPU use, the network bandwidth should be reduced. Note +that typical text-based log records usually compress rather well. + + +TLS.PermittedPeer +^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "array", "none", "no", "none" + +Note: this parameter is mandatory depending on the value of +`TLS.AuthMode` but the code does currently not check this. + +Peer Places access restrictions on this forwarder. Only peers which +have been listed in this parameter may be connected to. This guards +against rouge servers and man-in-the-middle attacks. The validation +bases on the certificate the remote peer presents. + +This contains either remote system names or fingerprints, depending +on the value of parameter `TLS.AuthMode`. One or more values may be +entered. + +When a non-permitted peer is connected to, the refusal is logged +together with the given remote peer identify. This is especially +useful in *fingerprint* authentication mode: if the +administrator knows this was a valid request, he can simply add the +fingerprint by copy and paste from the logfile to rsyslog.conf. It +must be noted, though, that this situation should usually not happen +after initial client setup and administrators should be alert in this +case. + +Note that usually a single remote peer should be all that is ever +needed. Support for multiple peers is primarily included in support +of load balancing scenarios. If the connection goes to a specific +server, only one specific certificate is ever expected (just like +when connecting to a specific ssh server). +To specify multiple fingerprints, just enclose them in braces like +this: + +.. code-block:: none + + tls.permittedPeer=["SHA1:...1", "SHA1:....2"] + +To specify just a single peer, you can either specify the string +directly or enclose it in braces. + +Note that in *name* authentication mode wildcards are supported. +This can be done as follows: + +.. code-block:: none + + tls.permittedPeer="*.example.com" + +Of course, there can also be multiple names used, some with and +some without wildcards: + +.. code-block:: none + + tls.permittedPeer=["*.example.com", "srv1.example.net", "srv2.example.net"] + + +TLS.AuthMode +^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "none", "no", "none" + +Sets the mode used for mutual authentication. Supported values are +either "*fingerprint*" or "*name*". Fingerprint mode basically is +what SSH does. It does not require a full PKI to be present, instead +self-signed certs can be used on all peers. Even if a CA certificate +is given, the validity of the peer cert is NOT verified against it. +Only the certificate fingerprint counts. + +In "name" mode, certificate validation happens. Here, the matching is +done against the certificate's subjectAltName and, as a fallback, the +subject common name. If the certificate contains multiple names, a +match on any one of these names is considered good and permits the +peer to talk to rsyslog. + +The permitted names or fingerprints are configured via +`TLS.PermittedPeer`. + + +About Chained Certificates +-------------------------- + +.. versionadded:: 8.2008.0 + +With librelp 1.7.0, you can use chained certificates. +If using "openssl" as tls.tlslib, we recommend at least OpenSSL Version 1.1 +or higher. Chained certificates will also work with OpenSSL Version 1.0.2, but +they will be loaded into the main OpenSSL context object making them available +to all librelp instances (omrelp/imrelp) within the same process. + +If this is not desired, you will require to run rsyslog in multiple instances +with different omrelp configurations and certificates. + + +TLS.CaCert +^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "none", "no", "none" + +The CA certificate that can verify the machine certs. + + +TLS.MyCert +^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "none", "no", "none" + +The machine public certificate. + + +TLS.MyPrivKey +^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "none", "no", "none" + +The machine private key. + + +TLS.PriorityString +^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "none", "no", "none" + +This parameter permits to specify the so-called "priority string" to +GnuTLS. This string gives complete control over all crypto +parameters, including compression setting. For this reason, when the +prioritystring is specified, the "tls.compression" parameter has no +effect and is ignored. +Full information about how to construct a priority string can be +found in the GnuTLS manual. At the time of this writing, this +information was contained in `section 6.10 of the GnuTLS +manual <http://gnutls.org/manual/html_node/Priority-Strings.html>`__. +**Note: this is an expert parameter.** Do not use if you do not +exactly know what you are doing. + +tls.tlscfgcmd +^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "none", "no", "none" + +.. versionadded:: 8.2001.0 + +The setting can be used if tls.tlslib is set to "openssl" to pass configuration commands to +the openssl libray. +OpenSSL Version 1.0.2 or higher is required for this feature. +A list of possible commands and their valid values can be found in the documentation: +https://www.openssl.org/docs/man1.0.2/man3/SSL_CONF_cmd.html + +The setting can be single or multiline, each configuration command is separated by linefeed (\n). +Command and value are separated by equal sign (=). Here are a few samples: + +Example 1 +--------- + +This will allow all protocols except for SSLv2 and SSLv3: + +.. code-block:: none + + tls.tlscfgcmd="Protocol=ALL,-SSLv2,-SSLv3" + + +Example 2 +--------- + +This will allow all protocols except for SSLv2, SSLv3 and TLSv1. +It will also set the minimum protocol to TLSv1.2 + +.. code-block:: none + + tls.tlscfgcmd="Protocol=ALL,-SSLv2,-SSLv3,-TLSv1 + MinProtocol=TLSv1.2" + +LocalClientIp +^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +Omrelp uses ip_address as local client address while connecting +to remote logserver. + + +Examples +======== + +Sending msgs with omrelp +------------------------ + +The following sample sends all messages to the central server +"centralserv" at port 2514 (note that that server must run imrelp on +port 2514). + +.. code-block:: none + + module(load="omrelp") + action(type="omrelp" target="centralserv" port="2514") + + +Sending msgs with omrelp via TLS +------------------------------------ + +This is the same as the previous example but uses TLS (via OpenSSL) for +operations. + +Certificate files must exist at configured locations. Note that authmode +"certvalid" is not very strong - you may want to use a different one for +actual deployments. For details, see parameter descriptions. + +.. code-block:: none + + module(load="omrelp" tls.tlslib="openssl") + action(type="omrelp" + target="centralserv" port="2514" tls="on" + tls.cacert="tls-certs/ca.pem" + tls.mycert="tls-certs/cert.pem" + tls.myprivkey="tls-certs/key.pem" + tls.authmode="certvalid" + tls.permittedpeer="rsyslog") + + +|FmtObsoleteName| directives +============================ + +This module uses old-style action configuration to keep consistent with +the forwarding rule. So far, no additional configuration directives can +be specified. To send a message via RELP, use + +.. code-block:: none + + *.* :omrelp:<server>:<port>;<template> + + diff --git a/source/configuration/modules/omruleset.rst b/source/configuration/modules/omruleset.rst new file mode 100644 index 0000000..abba0b2 --- /dev/null +++ b/source/configuration/modules/omruleset.rst @@ -0,0 +1,184 @@ +****************************************** +omruleset: ruleset output/including module +****************************************** + +=========================== =========================================================================== +**Module Name:** **omruleset** +**Author:** `Rainer Gerhards <https://rainer.gerhards.net/>`_ <rgerhards@adiscon.com> +=========================== =========================================================================== + +.. warning:: + + This module is outdated and only provided to support configurations that + already use it. **Do no longer use it in new configurations.** It has + been replaced by the much more efficient `"call" RainerScript + statement <rainerscript_call.html>`_. The "call" statement supports + everything omruleset does, but in an easier to use way. + + +**Available Since**: 5.3.4 + +**Deprecated in**: 7.2.0+ + + +Purpose +======= + +This is a very special "output" module. It permits to pass a message +object to another rule set. While this is a very simple action, it +enables very complex configurations, e.g. it supports high-speed "and" +conditions, sending data to the same file in a non-racy way, +include-ruleset functionality as well as some high-performance +optimizations (in case the rule sets have the necessary queue +definitions). + +While it leads to a lot of power, this output module offers seemingly +easy functionality. The complexity (and capabilities) arise from how +everything can be combined. + +With this module, a message can be sent to processing to another +ruleset. This is somewhat similar to a "#include" in the C programming +language. However, one needs to keep on the mind that a ruleset can +contain its own queue and that a queue can run in various modes. + +Note that if no queue is defined in the ruleset, the message is enqueued +into the main message queue. This most often is not optimal and means +that message processing may be severely deferred. Also note that when the +ruleset's target queue is full and no free space can be acquired within +the usual timeout, the message object may actually be lost. This is an +extreme scenario, but users building an audit-grade system need to know +this restriction. For regular installations, it should not really be +relevant. + +**At minimum, be sure you understand the** +:doc:`$RulesetCreateMainQueue <../ruleset/rsconf1_rulesetcreatemainqueue>` +**directive as well as the importance of statement order in rsyslog.conf +before using omruleset!** + +**Recommended Use:** + +- create rulesets specifically for omruleset +- create these rulesets with their own main queue +- decent queueing parameters (sizes, threads, etc) should be used for + the ruleset main queue. If in doubt, use the same parameters as for + the overall main queue. +- if you use multiple levels of ruleset nesting, double check for + endless loops - the rsyslog engine does not detect these + + +|FmtObsoleteName| directives +============================ + +.. note:: + + Parameter names are case-insensitive. + +- **$ActionOmrulesetRulesetName** ruleset-to-submit-to + This directive specifies the name of the ruleset that the message + provided to omruleset should be submitted to. This ruleset must + already have been defined. Note that the directive is automatically + reset after each :omruleset: action and there is no default. This is + done to prevent accidental loops in ruleset definition, what can + happen very quickly. The :omruleset: action will NOT be honored if no + ruleset name has been defined. As usual, the ruleset name must be + specified in front of the action that it modifies. + + +Examples +======== + +Ruleset for Write-to-file action +-------------------------------- + +This example creates a ruleset for a write-to-file action. The idea here +is that the same file is written based on multiple filters, problems +occur if the file is used together with a buffer. That is because file +buffers are action-specific, and so some partial buffers would be +written. With omruleset, we create a single action inside its own +ruleset and then pass all messages to it whenever we need to do so. Of +course, such a simple situation could also be solved by a more complex +filter, but the method used here can also be utilized in more complex +scenarios (e.g. with multiple listeners). The example tries to keep it +simple. Note that we create a ruleset-specific main queue (for +simplicity with the default main queue parameters) in order to avoid +re-queueing messages back into the main queue. + +.. code-block:: none + + $ModLoad omruleset # define ruleset for commonly written file + $RuleSet CommonAction + $RulesetCreateMainQueue on + *.* /path/to/file.log + + #switch back to default ruleset + $ruleset RSYSLOG_DefaultRuleset + + # begin first action + # note that we must first specify which ruleset to use for omruleset: + $ActionOmrulesetRulesetName CommonAction + mail.info :omruleset: + # end first action + + # begin second action + # note that we must first specify which ruleset to use for omruleset: + $ActionOmrulesetRulesetName CommonAction + :FROMHOST, isequal, "myhost.example.com" :omruleset: + #end second action + + # of course, we can have "regular" actions alongside :omrulset: actions + *.* /path/to/general-message-file.log + + +High-performance filter condition +--------------------------------- + +The next example is used to create a high-performance nested and filter +condition. Here, it is first checked if the message contains a string +"error". If so, the message is forwarded to another ruleset which then +applies some filters. The advantage of this is that we can use +high-performance filters where we otherwise would need to use the (much +slower) expression-based filters. Also, this enables pipeline +processing, in that second ruleset is executed in parallel to the first +one. + +.. code-block:: none + + $ModLoad omruleset + # define "second" ruleset + $RuleSet nested + $RulesetCreateMainQueue on + # again, we use our own queue + mail.* /path/to/mailerr.log + kernel.* /path/to/kernelerr.log + auth.* /path/to/autherr.log + + #switch back to default ruleset + $ruleset RSYSLOG_DefaultRuleset + + # begin first action - here we filter on "error" + # note that we must first specify which ruleset to use for omruleset: + $ActionOmrulesetRulesetName nested + :msg, contains, "error" :omruleset: + #end first action + + # begin second action - as an example we can do anything else in + # this processing. Note that these actions are processed concurrently + # to the ruleset "nested" + :FROMHOST, isequal, "myhost.example.com" /path/to/host.log + #end second action + + # of course, we can have "regular" actions alongside :omrulset: actions + *.* /path/to/general-message-file.log + + +Caveats/Known Bugs +================== + +The current configuration file language is not really adequate for a +complex construct like omruleset. Unfortunately, more important work is +currently preventing me from redoing the config language. So use extreme +care when nesting rulesets and be sure to test-run your config before +putting it into production, ensuring you have a sufficiently large probe +of the traffic run over it. If problems arise, the `rsyslog debug +log <troubleshoot.html>`_ is your friend. + diff --git a/source/configuration/modules/omsnmp.rst b/source/configuration/modules/omsnmp.rst new file mode 100644 index 0000000..ba283f5 --- /dev/null +++ b/source/configuration/modules/omsnmp.rst @@ -0,0 +1,265 @@ +******************************* +omsnmp: SNMP Trap Output Module +******************************* + +=========================== =========================================================================== +**Module Name:** **omsnmp** +**Author:** Andre Lorbach <alorbach@adiscon.com> +=========================== =========================================================================== + + +Purpose +======= + +Provides the ability to send syslog messages as an SNMPv1 & v2c traps. +By default, SNMPv2c is preferred. The syslog message is wrapped into a +OCTED STRING variable. This module uses the +`NET-SNMP <http://net-snmp.sourceforge.net/>`_ library. In order to +compile this module, you will need to have the +`NET-SNMP <http://net-snmp.sourceforge.net/>`_ developer (headers) +package installed. + + +Configuration Parameters +======================== + +.. note:: + + Parameter names are case-insensitive. + +Action Parameters +----------------- + +Server +^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "none", "yes", "``$actionsnmptarget``" + +This can be a hostname or ip address, and is our snmp target host. +This parameter is required, if the snmptarget is not defined, nothing +will be send. + + +Port +^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "162", "no", "``$actionsnmptargetport``" + +The port which will be used, common values are port 162 or 161. + + +Transport +^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "udp", "no", "``$actionsnmptransport``" + +Defines the transport type you wish to use. Technically we can +support all transport types which are supported by NET-SNMP. +To name a few possible values: +udp, tcp, udp6, tcp6, icmp, icmp6 ... + + +Version +^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "1", "no", "``$actionsnmpversion``" + +There can only be two choices for this parameter for now. +0 means SNMPv1 will be used. +1 means SNMPv2c will be used. +Any other value will default to 1. + + +Community +^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "public", "no", "``$actionsnmpcommunity``" + +This sets the used SNMP Community. + + +TrapOID +^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "1.3.6.1.4.1.19406.1.2.1", "no", "``$actionsnmptrapoid``" + +The default value means "ADISCON-MONITORWARE-MIB::syslogtrap". + +This configuration parameter is used for **SNMPv2** only. +This is the OID which defines the trap-type, or notification-type +rsyslog uses to send the trap. +In order to decode this OID, you will need to have the +ADISCON-MONITORWARE-MIB and ADISCON-MIB mibs installed on the +receiver side. Downloads of these mib files can be found here: + +`http://www.adiscon.org/download/ADISCON-MIB.txt <http://www.adiscon.org/download/ADISCON-MIB.txt>`_ + +`http://www.adiscon.org/download/ADISCON-MONITORWARE-MIB.txt <http://www.adiscon.org/download/ADISCON-MONITORWARE-MIB.txt>`_ +Thanks to the net-snmp mailinglist for the help and the +recommendations ;). + + +MessageOID +^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "1.3.6.1.4.1.19406.1.2.1", "no", "``$actionsnmpsyslogmessageoid``" + +This OID will be used as a variable, type "OCTET STRING". This +variable will contain up to 255 characters of the original syslog +message including syslog header. It is recommend to use the default +OID. +In order to decode this OID, you will need to have the +ADISCON-MONITORWARE-MIB and ADISCON-MIB mibs installed on the +receiver side. To download these custom mibs, see the description of +**TrapOID**. + + +EnterpriseOID +^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "1.3.6.1.4.1.3.1.1", "no", "``$actionsnmpenterpriseoid``" + +The default value means "enterprises.cmu.1.1" + +Customize this value if needed. I recommend to use the default value +unless you require to use a different OID. +This configuration parameter is used for **SNMPv1** only. It has no +effect if **SNMPv2** is used. + + +SpecificType +^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "0", "no", "``$actionsnmpspecifictype``" + +This is the specific trap number. This configuration parameter is +used for **SNMPv1** only. It has no effect if **SNMPv2** is used. + + +Snmpv1DynSource +^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "", "no", "none" + +.. versionadded:: 8.2001 + +If set, the source field of the SNMP trap can be overwritten with the a +template. The internal default is "%fromhost-ip%". The result should be a +valid IPv4 Address. Otherwise setting the source will fail. + +Below is a sample template called "dynsource" which you canm use to set the +source to a custom property: + +.. code-block:: none + + set $!custom_host = $fromhost; + template(name="dynsource" type="list") { + property(name="$!custom_host") + } + + +This configuration parameter is used for **SNMPv1** only. +It has no effect if **SNMPv2** is used. + + +TrapType +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "6", "no", "``$actionsnmptraptype``" + +There are only 7 Possible trap types defined which can be used here. +These trap types are: + +.. code-block:: none + + 0 = SNMP_TRAP_COLDSTART + 1 = SNMP_TRAP_WARMSTART + 2 = SNMP_TRAP_LINKDOWN + 3 = SNMP_TRAP_LINKUP + 4 = SNMP_TRAP_AUTHFAIL + 5 = SNMP_TRAP_EGPNEIGHBORLOSS + 6 = SNMP_TRAP_ENTERPRISESPECIFIC + +.. note:: + + Any other value will default to 6 automatically. This configuration + parameter is used for **SNMPv1** only. It has no effect if **SNMPv2** + is used. + + +Caveats/Known Bugs +================== + +- In order to decode the custom OIDs, you will need to have the adiscon + mibs installed. + + +Examples +======== + +Sending messages as snmp traps +------------------------------ + +The following commands send every message as a snmp trap. + +.. code-block:: none + + module(load="omsnmp") + action(type="omsnmp" server="localhost" port="162" transport="udp" + version="1" community="public") + diff --git a/source/configuration/modules/omstdout.rst b/source/configuration/modules/omstdout.rst new file mode 100644 index 0000000..ee2ddf3 --- /dev/null +++ b/source/configuration/modules/omstdout.rst @@ -0,0 +1,113 @@ +*********************************************** +omstdout: stdout output module (testbench tool) +*********************************************** + +=========================== =========================================================================== +**Module Name:** **omstdout** +**Author:** `Rainer Gerhards <https://rainer.gerhards.net/>`_ <rgerhards@adiscon.com> +**Available Since:** 4.1.6 +=========================== =========================================================================== + + +Purpose +======= + +This module writes any messages that are passed to it to stdout. It +was developed for the rsyslog test suite. However, there may (limited) +exist some other usages. Please note we do not put too much effort on +the quality of this module as we do not expect it to be used in real +deployments. If you do, please drop us a note so that we can enhance +its priority! + + +Configuration +============= + +.. note:: + + Parameter names are case-insensitive. + + +Module Parameters +----------------- + +none + + +Action Parameters +----------------- + +template +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "RSYSLOG_FileFormat", "no", "none" + +Set the template which will be used for the output. If none is specified +the default will be used. + + +EnsureLFEnding +^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "on", "no", "``$ActionOMStdoutEnsureLFEnding``" + +Makes sure, that each message is written with a terminating LF. If the +message contains a trailing LF, none is added. This is needed for the +automated tests. + + +Configure statement +------------------- + +This is used when building rsyslog from source. + +./configure --enable-omstdout + + +Legacy parameter not adopted in the new style +--------------------------------------------- + +- **$ActionOMStdoutArrayInterface** + [Default: off] + This setting instructs omstdout to use the alternate array based + method of parameter passing. If used, the values will be output with + commas between the values but no other padding bytes. This is a test + aid for the alternate calling interface. + + +Examples +======== + +Minimum setup +------------- + +The following sample is the minimum setup required to have syslog messages +written to stdout. + +.. code-block:: none + + module(load="omstdout") + action(type="omstdout") + + +Example 2 +--------- + +The following sample will write syslog messages to stdout, using a template. + +.. code-block:: none + + module(load="omstdout") + action(type="omstdout" template="outfmt") + + diff --git a/source/configuration/modules/omudpspoof.rst b/source/configuration/modules/omudpspoof.rst new file mode 100644 index 0000000..2edb106 --- /dev/null +++ b/source/configuration/modules/omudpspoof.rst @@ -0,0 +1,209 @@ +************************************** +omudpspoof: UDP spoofing output module +************************************** + +=========================== =========================================================================== +**Module Name:** **omudpspoof** +**Author:** David Lang <david@lang.hm> and `Rainer Gerhards <https://rainer.gerhards.net/>`_ <rgerhards@adiscon.com> +**Available Since:** 5.1.3 +=========================== =========================================================================== + + +Purpose +======= + +This module is similar to the regular UDP forwarder, but permits to +spoof the sender address. Also, it enables to circle through a number of +source ports. + +**Important**: This module **requires root permissions**. This is a hard +requirement because raw socket access is necessary to fake UDP sender +addresses. As such, rsyslog cannot drop privileges if this module is +to be used. Ensure that you do **not** use `$PrivDropToUser` or +`$PrivDropToGroup`. Many distro default configurations (notably Ubuntu) +contain these statements. You need to remove or comment them out if you +want to use `omudpspoof`. + + +Configuration Parameters +======================== + +.. note:: + + Parameter names are case-insensitive. + +Module Parameters +----------------- + +Template +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "RSYSLOG_TraditionalForwardFormat", "no", "none" + +This setting instructs omudpspoof to use a template different from +the default template for all of its actions that do not have a +template specified explicitly. + + +Action Parameters +----------------- + +Target +^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "yes", "``$ActionOMUDPSpoofTargetHost``" + +Host that the messages shall be sent to. + + +Port +^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "514", "no", "``$ActionOMUDPSpoofTargetPort``" + +Remote port that the messages shall be sent to. Default is 514. + + +SourceTemplate +^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "RSYSLOG_omudpspoofDfltSourceTpl", "no", "``$ActionOMOMUDPSpoofSourceNameTemplate``" + +This is the name of the template that contains a numerical IP +address that is to be used as the source system IP address. While it +may often be a constant value, it can be generated as usual via the +property replacer, as long as it is a valid IPv4 address. If not +specified, the build-in default template +RSYSLOG\_omudpspoofDfltSourceTpl is used. This template is defined as +follows: +$template RSYSLOG\_omudpspoofDfltSourceTpl,"%fromhost-ip%" +So in essence, the default template spoofs the address of the system +the message was received from. This is considered the most important +use case. + + +SourcePort.start +^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "32000", "no", "``$ActionOMUDPSpoofSourcePortStart``" + +Specify the start value for circling the source ports. Start must be +less than or equal to sourcePort.End. + + +SourcePort.End +^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "42000", "no", "``$ActionOMUDPSpoofSourcePortEnd``" + +Specify the end value for circling the source ports. End must be +equal to or more than sourcePort.Start. + + +MTU +^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "1500", "no", "none" + +Maximum packet length to send. + + +Template +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "RSYSLOG_TraditionalForwardFormat", "no", "``$ActionOMUDPSpoofDefaultTemplate``" + +This setting instructs omudpspoof to use a template different from +the default template for all of its actions that do not have a +template specified explicitly. + + +Caveats/Known Bugs +================== + +- **IPv6** is currently not supported. If you need this capability, + please let us know via the rsyslog mailing list. + +- Throughput is MUCH smaller than when using omfwd module. + + +Examples +======== + +Forwarding message through multiple ports +----------------------------------------- + +Forward the message to 192.168.1.1, using original source and port between 10000 and 19999. + +.. code-block:: none + + Action ( + type="omudpspoof" + target="192.168.1.1" + sourceport.start="10000" + sourceport.end="19999" + ) + + +Forwarding message using another source address +----------------------------------------------- + +Forward the message to 192.168.1.1, using source address 192.168.111.111 and default ports. + +.. code-block:: none + + Module ( + load="omudpspoof" + ) + Template ( + name="spoofaddr" + type="string" + string="192.168.111.111" + ) + Action ( + type="omudpspoof" + target="192.168.1.1" + sourcetemplate="spoofaddr" + ) + + diff --git a/source/configuration/modules/omusrmsg.rst b/source/configuration/modules/omusrmsg.rst new file mode 100644 index 0000000..ec00956 --- /dev/null +++ b/source/configuration/modules/omusrmsg.rst @@ -0,0 +1,67 @@ +********************** +omusrmsg: notify users +********************** + +=========================== =========================================================================== +**Module Name:** **omusrmsg** +**Author:** `Rainer Gerhards <https://rainer.gerhards.net/>`_ <rgerhards@adiscon.com> +=========================== =========================================================================== + + +Purpose +======= + +This module permits to send log messages to the user terminal. This is a +built-in module so it doesn't need to be loaded. + + +Configuration Parameters +======================== + +.. note:: + + Parameter names are case-insensitive. + + +Action Parameters +----------------- + +Users +^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "none", "yes", "none" + +The name of the users to send data to. + + +Template +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "WallFmt/StdUsrMsgFmt", "no", "none" + +Template to user for the message. Default is WallFmt when parameter users is +"*" and StdUsrMsgFmt otherwise. + + +Examples +======== + +Write emergency messages to all users +------------------------------------- + +The following command writes emergency messages to all users + +.. code-block:: none + + action(type="omusrmsg" users="*") + diff --git a/source/configuration/modules/omuxsock.rst b/source/configuration/modules/omuxsock.rst new file mode 100644 index 0000000..a9ba8cd --- /dev/null +++ b/source/configuration/modules/omuxsock.rst @@ -0,0 +1,61 @@ +************************************ +omuxsock: Unix sockets Output Module +************************************ + +=========================== =========================================================================== +**Module Name:** **omuxsock** +**Author:** `Rainer Gerhards <https://rainer.gerhards.net/>`_ <rgerhards@adiscon.com> +**Available since:** 4.7.3, 5.5.7 +=========================== =========================================================================== + + +Purpose +======= + +This module supports sending syslog messages to local Unix sockets. Thus +it provided a fast message-passing interface between different rsyslog +instances. The counterpart to omuxsock is `imuxsock <imuxsock.html>`_. +Note that the template used together with omuxsock must be suitable to +be processed by the receiver. + + +Configuration Parameters +======================== + +.. note:: + + Parameter names are case-insensitive. + +|FmtObsoleteName| directives +---------------------------- + +- **$OMUxSockSocket** + Name of the socket to send data to. This has no default and **must** + be set. +- **$OMUxSockDefaultTemplate** + This can be used to override the default template to be used + together with omuxsock. This is primarily useful if there are many + forwarding actions and each of them should use the same template. + + +Caveats/Known Bugs +================== + +Currently, only datagram sockets are supported. + + +Examples +======== + +Write all messages to socket +---------------------------- + +The following sample writes all messages to the "/tmp/socksample" +socket. + +.. code-block:: none + + $ModLoad omuxsock + $OMUxSockSocket /tmp/socksample + *.* :omuxsock: + diff --git a/source/configuration/modules/pmciscoios.rst b/source/configuration/modules/pmciscoios.rst new file mode 100644 index 0000000..dc82e43 --- /dev/null +++ b/source/configuration/modules/pmciscoios.rst @@ -0,0 +1,183 @@ +********** +pmciscoios +********** + +=========================== =========================================================================== +**Module Name:** **pmciscoios** +**Author:** `Rainer Gerhards <https://rainer.gerhards.net/>`_ <rgerhards@adiscon.com> +**Available since:** 8.3.4+ +=========================== =========================================================================== + + +Purpose +======= + +This is a parser that understands Cisco IOS "syslog" format. Note +that this format is quite different from RFC syslog format, and +so the default parser chain cannot deal with it. + +Note that due to large differences in IOS logging format, pmciscoios +may currently not be able to handle all possible format variations. +Nevertheless, it should be fairly easy to adapt it to additional +requirements. So be sure to ask if you run into problems with +format issues. + +Note that if your Cisco system emits timezone information in a supported +format, rsyslog will pick it up. In order to apply proper timezone offsets, +the timezone ids (e.g. "EST") must be configured via the +:doc:`timezone object <../timezone>`. + +Note if the clock on the Cisco device has not been set and cannot be +verified the Cisco will prepend the timestamp field with an asterisk (*). +If the clock has gone out of sync with its configured NTP server the +timestamp field will be prepended with a dot (.). In both of these cases +parsing the timestamp would fail, therefore any preceding asterisks (*) or +dots (.) are ignored. This may lead to "incorrect" timestamps being logged. + + +Configuration Parameters +======================== + +.. note:: + + Parameter names are case-insensitive. + + +Parser Parameters +----------------- + +present.origin +^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +This setting tell the parser if the origin field is present inside +the message. Due to the nature of Cisco's logging format, the parser +cannot sufficiently correctly deduce if the origin field is present +or not (at least not with reasonable performance). As such, the parser +must be provided with that information. If the origin is present, +its value is stored inside the HOSTNAME message property. + + +present.xr +^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +If syslog is received from an IOSXR device the syslog format will usually +start with the RSP/LC/etc that produced the log, then the timestamp. +It will also contain an additional syslog tag before the standard Cisco +%TAG, this tag references the process that produced the log. +In order to use this Cisco IOS parser module with XR format messages both +of these additional fields must be ignored. + + +Examples +======== + +Listening multiple devices, some emitting origin information and some not +------------------------------------------------------------------------- + +We assume a scenario where we have some devices configured to emit origin +information whereas some others do not. In order to differentiate between +the two classes, rsyslog accepts input on different ports, one per class. +For each port, an input() object is defined, which binds the port to a +ruleset that uses the appropriately-configured parser. Except for the +different parsers, processing shall be identical for both classes. In our +first example we do this via a common ruleset which carries out the +actual processing: + +.. code-block:: none + + module(load="imtcp") + module(load="pmciscoios") + + input(type="imtcp" port="10514" ruleset="withoutOrigin") + input(type="imtcp" port="10515" ruleset="withOrigin") + + ruleset(name="common") { + ... do processing here ... + } + + ruleset(name="withoutOrigin" parser="rsyslog.ciscoios") { + /* this ruleset uses the default parser which was + * created during module load + */ + call common + } + + parser(name="custom.ciscoios.withOrigin" type="pmciscoios" + present.origin="on") + ruleset(name="withOrigin" parser="custom.ciscoios.withOrigin") { + /* this ruleset uses the parser defined immediately above */ + call common + } + + +Date stamp immediately following the origin +------------------------------------------- + +The example configuration above is a good solution. However, it is possible +to do the same thing in a somewhat condensed way, but if and only if the date +stamp immediately follows the origin. In that case, the parser has a chance to +detect if the origin is present or not. The key point here is to make sure +the parser checking for the origin is given before the default one, in which +case the first on will detect it does not match an pass on to the next +one inside the parser chain. However, this comes at the expense of additional +runtime overhead. The example below is **not** good practice -- it is given +as a purely educational sample to show some fine details of how parser +definitions interact. In this case, we can use a single listener. + +.. code-block:: none + + module(load="imtcp") + module(load="pmciscoios") + + input(type="imtcp" port="10514" ruleset="ciscoBoth") + + parser(name="custom.ciscoios.withOrigin" type="pmciscoios" + present.origin="on") + ruleset(name="ciscoBoth" + parser=["custom.ciscoios.withOrigin", "rsyslog.ciscoios"]) { + ... do processing here ... + } + + +Handling Cisco IOS and IOSXR formats +------------------------------------ + +The following sample demonstrates how to handle Cisco IOS and IOSXR formats + +.. code-block:: none + + module(load="imudp") + module(load="pmciscoios") + + input(type="imudp" port="10514" ruleset="ios") + input(type="imudp" port="10515" ruleset="iosxr") + + ruleset(name="common") { + ... do processing here ... + } + + ruleset(name="ios" parser="rsyslog.ciscoios") { + call common + } + + parser(name="custom.ciscoios.withXr" type="pmciscoios" + present.xr="on") + ruleset(name="iosxr" parser="custom.ciscoios.withXr"] { + call common + } + + diff --git a/source/configuration/modules/pmdb2diag.rst b/source/configuration/modules/pmdb2diag.rst new file mode 100644 index 0000000..dc36977 --- /dev/null +++ b/source/configuration/modules/pmdb2diag.rst @@ -0,0 +1,146 @@ +************************************** +pmdb2diag: DB2 Diag file parser module +************************************** + +=========================== =========================================================================== +**Module Name:** **pmdb2diag** +**Authors:** Jean-Philippe Hilaire <jean-philippe.hilaire@pmu.fr> & Philippe Duveau <philippe.duveau@free.fr> +=========================== =========================================================================== + + +Purpose +======= + +The objective of this module is to extract timestamp, procid and appname form the log +lines without altering it + +The parser is acting after an imfile input. This implies that imfile must be configured +with needParse to setted to on. + +Compile +======= + +To successfully compile pmdb2diag module you need to add it via configure. + + ./configure --enable-pmdb2diag ... + +Configuration Parameters +======================== + +**Parser Name:** "db2.diag" + +The default value of parameter are defined with escapeLF on in imfile. + +timeformat +^^^^^^^^^^ + +.. csv-table:: + :header: "type", "mandatory", "format", "default" + :widths: auto + :class: parameter-table + + "string", "no", "see strptime manual","%Y-%m-%d-%H.%M.%S." + +Format of the timestamp in d2diag log included decimal separator between seconds and second fractions. + +timepos +^^^^^^^ + +.. csv-table:: + :header: "type", "mandatory", "format", "default" + :widths: auto + :class: parameter-table + + "integer", "no", ,"0" + +Position of the timestamp in the db2 diag log. + +levelpos +^^^^^^^^ + +.. csv-table:: + :header: "type", "mandatory", "format", "default" + :widths: auto + :class: parameter-table + + "integer", "no", ,"59" + +Position of the severity (level) in the db2 diag log. + +pidstarttoprogstartshift +^^^^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "mandatory", "format", "default" + :widths: auto + :class: parameter-table + + "integer", "no", ,"49" + +Position of the prog related to the pid (form beginning to beginning) in the db2 diag log. + +Examples +======== + +Example 1 +^^^^^^^^^ + +This is the simplest parsing with default values + +.. code-block:: none + + module(load="pmdb2diag") + ruleset(name="ruleDB2" parser="db2.diag") { + ... do something + } + input(type="imfile" file="db2diag.log" ruleset="ruleDB2" tag="db2diag" + startmsg.regex="^[0-9]{4}-[0-9]{2}-[0-9]{2}" escapelf="on" needparse="on") + + +Example 2 +^^^^^^^^^ + +Parsing with custom values + +.. code-block:: none + + module(load="pmdb2diag") + parser(type="pmdb2diag" name="custom.db2.diag" levelpos="57" + timeformat=""%y-%m-%d:%H.%M.%S.") + ruleset(name="ruleDB2" parser="custom.db2.diag") { + ... do something + } + input(type="imfile" file="db2diag.log" ruleset="ruleDB2" tag="db2diag" + startmsg.regex="^[0-9]{4}-[0-9]{2}-[0-9]{2}" escapelf="on" needparse="on") + +DB2 Log sample +^^^^^^^^^^^^^^ + +.. code-block:: none + + 2015-05-06-16.53.26.989402+120 E1876227378A1702 LEVEL: Info + PID : 4390948 TID : 89500 PROC : db2sysc 0 + INSTANCE: db2itst NODE : 000 DB : DBTEST + APPHDL : 0-14410 APPID: 10.42.2.166.36261.150506120149 + AUTHID : DBUSR HOSTNAME: dev-dbm1 + EDUID : 89500 EDUNAME: db2agent (DBTEST) 0 + FUNCTION: DB2 UDB, relation data serv, sqlrr_dispatch_xa_request, probe:703 + MESSAGE : ZRC=0x80100024=-2146435036=SQLP_NOTA "Transaction was not found" + DIA8036C XA error with request type of "". Transaction was not found. + DATA #1 : String, 27 bytes + XA Dispatcher received NOTA + CALLSTCK: (Static functions may not be resolved correctly, as they are resolved to the nearest symbol) + [0] 0x090000000A496B70 sqlrr_xrollback__FP14db2UCinterface + 0x11E0 + [1] 0x090000000A356764 sqljsSyncRollback__FP14db2UCinterface + 0x6E4 + [2] 0x090000000C1FAAA8 sqljsParseRdbAccessed__FP13sqljsDrdaAsCbP13sqljDDMObjectP14db2UCinterface + 0x529C + [3] 0x0000000000000000 ?unknown + 0x0 + [4] 0x090000000C23D260 @72@sqljsSqlam__FP14db2UCinterfaceP8sqeAgentb + 0x1174 + [5] 0x090000000C23CE54 @72@sqljsSqlam__FP14db2UCinterfaceP8sqeAgentb + 0xD68 + [6] 0x090000000D74AB90 @72@sqljsDriveRequests__FP8sqeAgentP14db2UCconHandle + 0xA8 + [7] 0x090000000D74B6A0 @72@sqljsDrdaAsInnerDriver__FP18SQLCC_INITSTRUCT_Tb + 0x5F8 + [8] 0x090000000B8F85AC RunEDU__8sqeAgentFv + 0x48C38 + [9] 0x090000000B876240 RunEDU__8sqeAgentFv + 0x124 + [10] 0x090000000CD90DFC EDUDriver__9sqzEDUObjFv + 0x130 + [11] 0x090000000BE01664 sqloEDUEntry + 0x390 + [12] 0x09000000004F5E10 _pthread_body + 0xF0 + [13] 0xFFFFFFFFFFFFFFFC ?unknown + 0xFFFFFFFF diff --git a/source/configuration/modules/pmlastmsg.rst b/source/configuration/modules/pmlastmsg.rst new file mode 100644 index 0000000..711612f --- /dev/null +++ b/source/configuration/modules/pmlastmsg.rst @@ -0,0 +1,68 @@ +**************************************** +pmlastmsg: last message repeated n times +**************************************** + +=========================== =========================================================================== +**Module Name:** **pmlastmsg** +**Author:** `Rainer Gerhards <https://rainer.gerhards.net/>`_ <rgerhards@adiscon.com> +**Available Since:** 5.5.6 +=========================== =========================================================================== + + +Purpose +======= + +Some syslogds are known to emit severity malformed messages with content +"last message repeated n times". These messages can mess up message +reception, as they lead to wrong interpretation with the standard +RFC3164 parser. Rather than trying to fix this issue in pmrfc3164, we +have created a new parser module specifically for these messages. The +reason is that some processing overhead is involved in processing these +messages (they must be recognized) and we would not like to place this +toll on every user but only on those actually in need of the feature. +Note that the performance toll is not large -- but if you expect a very +high message rate with tenthousands of messages per second, you will +notice a difference. + +This module should be loaded first inside :doc:`rsyslog's parser +chain </concepts/messageparser>`. It processes all those messages that +contain a PRI, then none or some spaces and then the exact text +(case-insensitive) "last message repeated n times" where n must be an +integer. All other messages are left untouched. + + +Configuration Parameters +======================== + +.. note:: + + Parameter names are case-insensitive. + + +There do not currently exist any configuration parameters for this +module. + + +Examples +======== + +Systems emitting malformed "repeated msg" messages +-------------------------------------------------- + +This example is the typical use case, where some systems emit malformed +"repeated msg" messages. Other than that, the default :rfc:`5424` and +:rfc:`3164` parsers should be used. Note that when a parser is specified, +the default parser chain is removed, so we need to specify all three +parsers. We use this together with the default ruleset. + +.. code-block:: none + + module(load="pmlastmsg") + + parser(type="pmlastmsg" name="custom.pmlastmsg") + + ruleset(name="ruleset" parser=["custom.pmlastmsg", "rsyslog.rfc5424", + "rsyslog.rfc3164"]) { + ... do processing here ... + } + diff --git a/source/configuration/modules/pmnormalize.rst b/source/configuration/modules/pmnormalize.rst new file mode 100644 index 0000000..2a01dd5 --- /dev/null +++ b/source/configuration/modules/pmnormalize.rst @@ -0,0 +1,121 @@ +***************************************************** +Log Message Normalization Parser Module (pmnormalize) +***************************************************** + +=========================== =========================================================================== +**Module Name:** **pmnormalize** +**Author:** Pascal Withopf <pascalwithopf1@gmail.com> +**Available since:** 8.27.0 +=========================== =========================================================================== + + +Purpose +======= + +This parser normalizes messages with the specified rules and populates the +properties for further use. + + +Configuration Parameters +======================== + +.. note:: + + Parameter names are case-insensitive. + + +Action Parameters +----------------- + +Rulebase +^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +Specifies which rulebase file is to use. If there are multiple +pmnormalize instances, each one can use a different file. However, a +single instance can use only a single file. This parameter or **rule** +MUST be given, because normalization can only happen based on a rulebase. +It is recommended that an absolute path name is given. Information on +how to create the rulebase can be found in the `liblognorm +manual <http://www.liblognorm.com/files/manual/index.html>`_. + + +Rule +^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "array", "none", "no", "none" + +Contains an array of strings which will be put together as the rulebase. +This parameter or **rulebase** MUST be given, because normalization can +only happen based on a rulebase. + + +UndefinedPropertyError +^^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +With this parameter an error message is controlled, which will be put out +every time pmnormalize can't normalize a message. + + +Examples +======== + +Normalize msgs received via imtcp +--------------------------------- + +In this sample messages are received via imtcp. Then they are normalized with +the given rulebase and written to a file. + +.. code-block:: none + + module(load="imtcp") + module(load="pmnormalize") + + input(type="imtcp" port="13514" ruleset="ruleset") + + parser(name="custom.pmnormalize" type="pmnormalize" rulebase="/tmp/rules.rulebase") + + ruleset(name="ruleset" parser="custom.pmnormalize") { + action(type="omfile" file="/tmp/output") + } + + +Write normalized messages to file +--------------------------------- + +In this sample messages are received via imtcp. Then they are normalized with +the given rule array. After that they are written in a file. + +.. code-block:: none + + module(load="imtcp") + module(load="pmnormalize") + + input(type="imtcp" port="10514" ruleset="outp") + + parser(name="custom.pmnormalize" type="pmnormalize" rule=[ + "rule=:<%pri:number%> %fromhost-ip:ipv4% %hostname:word% %syslogtag:char-to:\\x3a%: %msg:rest%", + "rule=:<%pri:number%> %hostname:word% %fromhost-ip:ipv4% %syslogtag:char-to:\\x3a%: %msg:rest%"]) + + ruleset(name="outp" parser="custom.pmnormalize") { + action(type="omfile" File="/tmp/output") + } + diff --git a/source/configuration/modules/pmnull.rst b/source/configuration/modules/pmnull.rst new file mode 100644 index 0000000..3348cbf --- /dev/null +++ b/source/configuration/modules/pmnull.rst @@ -0,0 +1,123 @@ +********************************* +pmnull: Syslog Null Parser Module +********************************* + +=========================== =========================================================================== +**Module Name:** **pmnull** +**Author:** `Rainer Gerhards <https://rainer.gerhards.net/>`_ <rgerhards@adiscon.com> +=========================== =========================================================================== + + +Purpose +======= + +When a message is received it is tried to match a set of parsers to get +properties populated. This parser module sets all attributes to "" but rawmsg. +There usually should be no need to use this module. It may be useful to +process certain known-non-syslog messages. + +The pmnull module was originally written as some people thought it would +be nice to save 0.05% of time by not unnecessarily parsing the message. +We even doubt it is that amount of performance enhancement as the properties +need to be populated in any case, so the saving is really minimal (but exists). + +**If you just want to transmit or store messages exactly in the format that +they arrived in you do not need pmnull!** You can use the `rawmsg` property:: + + template(name="asReceived" type="string" string="%rawmsg%") + action(type="omfwd" target="server.example.net" template="asReceived") + +Configuration Parameters +======================== + +.. note:: + + Parameter names are case-insensitive. + + +Parser Parameters +----------------- + +Tag +^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "string", "", "no", "none" + +This setting sets the tag value to the message. + + +SyslogFacility +^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "Facility", "1", "no", "none" + +This setting sets the syslog facility value. The default comes from the +rfc3164 standard. + + +SyslogSeverity +^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "Severity", "5", "no", "none" + +This setting sets the syslog severity value. The default comes from the +rfc3164 standard. + + +Examples +======== + +Process messages received via imtcp +----------------------------------- + +In this example messages are received through imtcp on port 13514. The +ruleset uses the parser pmnull which has the parameters tag, syslogfacility +and syslogseverity given. + +.. code-block:: none + + module(load="imtcp") + module(load="pmnull") + + input(type="imtcp" port="13514" ruleset="ruleset") + parser(name="custom.pmnull" type="pmnull" tag="mytag" syslogfacility="3" + syslogseverity="1") + + ruleset(name="ruleset" parser=["custom.pmnull", "rsyslog.pmnull"]) { + action(type="omfile" file="rsyslog.out.log") + } + + +Process messages with default parameters +---------------------------------------- + +In this example the ruleset uses the parser pmnull with the default parameters +because no specifics were given. + +.. code-block:: none + + module(load="imtcp") + module(load="pmnull") + + input(type="imtcp" port="13514" ruleset="ruleset") + parser(name="custom.pmnull" type="pmnull") + + ruleset(name="ruleset" parser="custom.pmnull") { + action(type="omfile" file="rsyslog.out.log") + } + diff --git a/source/configuration/modules/pmrfc3164.rst b/source/configuration/modules/pmrfc3164.rst new file mode 100644 index 0000000..46cff38 --- /dev/null +++ b/source/configuration/modules/pmrfc3164.rst @@ -0,0 +1,161 @@ +******************************************* +pmrfc3164: Parse RFC3164-formatted messages +******************************************* + +=========================== =========================================================================== +**Module Name:** **pmrfc3164** +**Author:** `Rainer Gerhards <https://rainer.gerhards.net/>`_ <rgerhards@adiscon.com> +=========================== =========================================================================== + + +Purpose +======= + +This parser module is for parsing messages according to the traditional/legacy +syslog standard :rfc:`3164` + +It is part of the default parser chain. + +The parser can also be customized to allow the parsing of specific formats, +if they occur. + + +Configuration Parameters +======================== + +.. note:: + + Parameter names are case-insensitive. + + +Parser Parameters +----------------- + +permit.squareBracketsInHostname +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +This setting tells the parser that hostnames that are enclosed by brackets +should omit the brackets. + + +permit.slashesInHostname +^^^^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +.. versionadded:: 8.20.0 + +This setting tells the parser that hostnames may contain slashes. This +is useful when messages e.g. from a syslog-ng relay chain are received. +Syslog-ng puts the various relay hosts via slashes into the hostname +field. + + +permit.AtSignsInHostname +^^^^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +.. versionadded:: 8.25.0 + +This setting tells the parser that hostnames may contain at-signs. This +is useful when messages are relayed from a syslog-ng server in rfc3164 +format. The hostname field sent by syslog-ng may be prefixed by the source +name followed by an at-sign character. + + +force.tagEndingByColon +^^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +.. versionadded:: 8.25.0 + +This setting tells the parser that tag need to be ending by colon to be +valid. In others case, the tag is set to dash ("-") without changing +message. + + +remove.msgFirstSpace +^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +.. versionadded:: 8.25.0 + +Rfc3164 tell message is directly after tag including first white space. +This option tell to remove the first white space in message just after +reading. It make rfc3164 & rfc5424 syslog messages working in a better way. + + +detect.YearAfterTimestamp +^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "none" + +Some devices send syslog messages in a format that is similar to RFC3164, +but they also attach the year to the timestamp (which is not compliant to +the RFC). With regular parsing, the year would be recognized to be the +hostname and the hostname would become the syslogtag. This setting should +prevent this. It is also limited to years between 2000 and 2099, so +hostnames with numbers as their name can still be recognized correctly. But +everything in this range will be detected as a year. + + +Examples +======== + +Receiving malformed RFC3164 messages +------------------------------------ + +We assume a scenario where some of the devices send malformed RFC3164 +messages. The parser module will automatically detect the malformed +sections and parse them accordingly. + +.. code-block:: none + + module(load="imtcp") + + input(type="imtcp" port="514" ruleset="customparser") + + parser(name="custom.rfc3164" + type="pmrfc3164" + permit.squareBracketsInHostname="on" + detect.YearAfterTimestamp="on") + + ruleset(name="customparser" parser="custom.rfc3164") { + ... do processing here ... + } + diff --git a/source/configuration/modules/pmrfc3164sd.rst b/source/configuration/modules/pmrfc3164sd.rst new file mode 100644 index 0000000..aeb1517 --- /dev/null +++ b/source/configuration/modules/pmrfc3164sd.rst @@ -0,0 +1,5 @@ +pmrfc3164sd: Parse RFC5424 structured data inside RFC3164 messages +================================================================== + +A contributed module for supporting RFC5424 structured data inside +RFC3164 messages (not supported by the rsyslog team) diff --git a/source/configuration/modules/pmrfc5424.rst b/source/configuration/modules/pmrfc5424.rst new file mode 100644 index 0000000..21554b7 --- /dev/null +++ b/source/configuration/modules/pmrfc5424.rst @@ -0,0 +1,6 @@ +pmrfc5424: Parse RFC5424-formatted messages +=========================================== + +This is the new Syslog Standard. + +:rfc:`5424` diff --git a/source/configuration/modules/sigprov_gt.rst b/source/configuration/modules/sigprov_gt.rst new file mode 100644 index 0000000..b3cd092 --- /dev/null +++ b/source/configuration/modules/sigprov_gt.rst @@ -0,0 +1,94 @@ +GuardTime Log Signature Provider (gt) +===================================== + +**Signature Provider Name: gt** + +**Author:** Rainer Gerhards <rgerhards@adiscon.com> + +**Supported:** from 7.3.9 to 8.26.0 + +**Description**: + +Provides the ability to sign syslog messages via the GuardTime signature +services. + +**Configuration Parameters**: + +Note: parameter names are case-insensitive. + +Signature providers are loaded by omfile, when the provider is selected +in its "sig.providerName" parameter. Parameters for the provider are +given in the omfile action instance line. + +This provider creates a signature file with the same base name but the +extension ".gtsig" for each log file (both for fixed-name files as well +as dynafiles). Both files together form a set. So you need to archive +both in order to prove integrity. + +- **sig.hashFunction** <Hash Algorithm> + The following hash algorithms are currently supported: + + - SHA1 + - RIPEMD-160 + - SHA2-224 + - SHA2-256 + - SHA2-384 + - SHA2-512 + +- **sig.timestampService** <timestamper URL> + This provides the URL of the timestamper service. If not selected, a + default server is selected. This may not necessarily be a good one + for your region. + + *Note:* If you need to supply user credentials, you can add them to + the timestamper URL. If, for example, you have a user "user" with + password "pass", you can do so as follows: + + http://user:pass@timestamper.example.net + +- **sig.block.sizeLimit** <nbr-records> + The maximum number of records inside a single signature block. By + default, there is no size limit, so the signature is only written on + file closure. Note that a signature request typically takes between + one and two seconds. So signing to frequently is probably not a good + idea. + +- **sig.keepRecordHashes** <on/**off**> + Controls if record hashes are written to the .gtsig file. This + enhances the ability to spot the location of a signature breach, but + costs considerable disk space (65 bytes for each log record for + SHA2-512 hashes, for example). + +- **sig.keepTreeHashes** <on/**off**> + Controls if tree (intermediate) hashes are written to the .gtsig + file. This enhances the ability to spot the location of a signature + breach, but costs considerable disk space (a bit mire than the amount + sig.keepRecordHashes requries). Note that both Tree and Record hashes + can be kept inside the signature file. + +**See Also** + +- `How to sign log messages through signature provider + Guardtime <http://www.rsyslog.com/how-to-sign-log-messages-through-signature-provider-guardtime/>`_ + +**Caveats/Known Bugs:** + +- currently none known + +**Samples:** + +This writes a log file with it's associated signature file. Default +parameters are used. + +:: + + action(type="omfile" file="/var/log/somelog" sig.provider="gt") + +In the next sample, we use the more secure SHA2-512 hash function, sign +every 10,000 records and Tree and Record hashes are kept. + +:: + + action(type="omfile" file="/var/log/somelog" sig.provider="gt" + sig.hashfunction="SHA2-512" sig.block.sizelimit="10000" + sig.keepTreeHashes="on" sig.keepRecordHashes="on") diff --git a/source/configuration/modules/sigprov_ksi.rst b/source/configuration/modules/sigprov_ksi.rst new file mode 100644 index 0000000..99193cb --- /dev/null +++ b/source/configuration/modules/sigprov_ksi.rst @@ -0,0 +1,99 @@ +Keyless Signature Infrastructure Provider (ksi) +=============================================== + +**Signature Provider Name: ksi** + +**Author:** Rainer Gerhards <rgerhards@adiscon.com> + +**Supported:** from 8.11.0 to 8.26.0 + +**Description**: + +Provides the ability to sign syslog messages via the GuardTime KSI +signature services. + +**Configuration Parameters**: + +Note: parameter names are case-insensitive. + +Signature providers are loaded by omfile, when the provider is selected +in its "sig.providerName" parameter. Parameters for the provider are +given in the omfile action instance line. + +This provider creates a signature file with the same base name but the +extension ".ksisig" for each log file (both for fixed-name files as well +as dynafiles). Both files together form a set. So you need to archive +both in order to prove integrity. + +- **sig.hashFunction** <Hash Algorithm> + The following hash algorithms are currently supported: + + - SHA1 + - SHA2-256 + - RIPEMD-160 + - SHA2-224 + - SHA2-384 + - SHA2-512 + - RIPEMD-256 + - SHA3-244 + - SHA3-256 + - SHA3-384 + - SHA3-512 + - SM3 + +- **sig.aggregator.uri** <KSI Aggregator URL> + This provides the URL of the KSI Aggregator service provided by + guardtime and looks like this: + + ksi+tcp://[ip/dnsname]:3332 + +- **sig.aggregator.user** <KSI UserID> + Set your username provided by Guardtime here. + +- **sig.aggregator.key** <KSI Key / Password> + Set your key provided by Guardtime here. + +- **sig.block.sizeLimit** <nbr-records> + The maximum number of records inside a single signature block. By + default, there is no size limit, so the signature is only written on + file closure. Note that a signature request typically takes between + one and two seconds. So signing to frequently is probably not a good + idea. + +- **sig.keepRecordHashes** <on/**off**> + Controls if record hashes are written to the .gtsig file. This + enhances the ability to spot the location of a signature breach, but + costs considerable disk space (65 bytes for each log record for + SHA2-512 hashes, for example). + +- **sig.keepTreeHashes** <on/**off**> + Controls if tree (intermediate) hashes are written to the .gtsig + file. This enhances the ability to spot the location of a signature + breach, but costs considerable disk space (a bit mire than the amount + sig.keepRecordHashes requries). Note that both Tree and Record hashes + can be kept inside the signature file. + +**See Also** + + +**Caveats/Known Bugs:** + +- currently none known + +**Samples:** + +This writes a log file with it's associated signature file. Default +parameters are used. + +:: + + action(type="omfile" file="/var/log/somelog" sig.provider="ksi") + +In the next sample, we use the more secure SHA2-512 hash function, sign +every 10,000 records and Tree and Record hashes are kept. + +:: + + action(type="omfile" file="/var/log/somelog" sig.provider="ksi" + sig.hashfunction="SHA2-512" sig.block.sizelimit="10000" + sig.keepTreeHashes="on" sig.keepRecordHashes="on") diff --git a/source/configuration/modules/sigprov_ksi12.rst b/source/configuration/modules/sigprov_ksi12.rst new file mode 100644 index 0000000..d5b6ab7 --- /dev/null +++ b/source/configuration/modules/sigprov_ksi12.rst @@ -0,0 +1,135 @@ +KSI Signature Provider (rsyslog-ksi-ls12) +============================================================ + +**Module Name: rsyslog-ksi-ls12** + +**Available Since:** 8.27 + +**Author:** Guardtime & Adiscon + +Description +########### + +The ``rsyslog-ksi-ls12`` module enables record level log signing with Guardtime KSI Blockchain. KSI signatures provide long-term log integrity and prove the time of log records cryptographically using independent verification. + +Main features of the ``rsyslog-ksi-ls12`` module are: + +* Automated online signing of file output log. +* Efficient block-based signing with record-level verification. +* Log records removal detection. + +For best results use the ``rsyslog-ksi-ls12`` module together with Guardtime ``logksi`` tool, which will become handy in: + +* Signing recovery. +* Extension of KSI signatures inside the log signature file. +* Verification of the log using log signatures. +* Extraction of record-level signatures. +* Integration of log signature files (necessary when signing in async mode). + +Getting Started +############### + +To get started with log signing: + +- Sign up to the Guardtime tryout service to be able to connect to KSI blockchain: + `guardtime.com/technology/blockchain-developers <https://guardtime.com/technology/blockchain-developers>`_ +- Install the ``libksi`` library (v3.20 or later) + `(libksi install) <https://github.com/guardtime/libksi#installation>`_ +- Install the ``rsyslog-ksi-ls12`` module (same version as rsyslog) from Adiscon repository. +- Install the accompanying ``logksi`` tool (recommended v1.5 or later) + `(logksi install) <https://github.com/guardtime/logksi#installation>`_ + +The format of the output depends on signing mode enabled (synchronous (``sync``) or asynchronous (``async``)). + +- In ``sync`` mode, log signature file is written directly into file ``<logfile>.logsig``. This mode is blocking as issuing KSI signatures one at a time will halt actual writing of log lines into log files. This mode suits for a system where signatures are issued rarely and delay caused by signing process is acceptable. Advantage compared to ``async`` mode is that the user has no need to integrate intermediate files to get actual log signature. + +- In ``async`` mode, log signature intermediate files are written into directory ``<logfile>.logsig.parts``. This mode is not blocking enabling high availability and concurrent signing of several blocks at the same time. Log signature is divided into two files, where one contains info about log records and blocks, and the other contains KSI signatures issued asynchronously. To create ``<logfile>.logsig`` from ``<logfile>.logsig.parts``, use ``logksi integrate <logfile>``. Advantage compared to ``sync`` mode is much better operational stability and speed. + +Currently the log signing is only supported by the file output module, thus the action type must be ``omfile``. To activate signing, add the following parameters to the action of interest in your rsyslog configuration file: + +Mandatory parameters (no default value defined): + +- **sig.provider** specifies the signature provider; in case of ``rsyslog-ksi-ls12`` package this is ``"ksi_ls12"``. +- **sig.block.levelLimit** defines the maximum level of the root of the local aggregation tree per one block. The maximum number of log lines in one block is calculated as ``2^(levelLimit - 1)``. +- **sig.aggregator.url** defines the endpoint of the KSI signing service in KSI Gateway. In ``async`` mode it is possible to specify up to 3 endpoints for high availability service, where user credentials are integrated into URL. Supported URI schemes are: + + - *ksi+http://* + - *ksi+tcp://* + + Examples: + + - sig.aggregator.url="ksi+tcp://signingservice1.example.com" + + sig.aggregator.user="rsmith" + + sig.aggregator.key= "secret" + + - sig.aggregator.url="ksi+tcp://rsmith:secret@signingservice1.example.com|ksi+tcp://jsmith:terces@signingservice2.example.com" + +- **sig.aggregator.user** specifies the login name for the KSI signing service. For high availability service, credentials are specified in URI. +- **sig.aggregator.key** specifies the key for the login name. For high availability service, credentials are specified in URI. + +Optional parameters (if not defined, default value is used): + +- **sig.syncmode** defines the signing mode: ``"sync"`` (default) or ``"async"``. +- **sig.hashFunction** defines the hash function to be used for hashing, default is ``"SHA2-256"``. + Other SHA-2, as well as RIPEMED-160 functions are supported. +- **sig.block.timeLimit** defines the maximum duration of one block in seconds. + Default value ``"0"`` indicates that no time limit is set. +- **sig.block.signTimeout** specifies a time window within the block signatures + have to be issued, default is ``10``. For example, issuing 4 signatures in a + second with sign timeout 10s, it is possible to handle 4 x 10 signatures + request created at the same time. More than that, will close last blocks with + signature failure as signature requests were not sent out within 10 seconds. +- **sig.aggregator.hmacAlg** defines the HMAC algorithm to be used in communication with the KSI Gateway. + This must be agreed on with the KSI service provider, default is ``"SHA2-256"``. +- **sig.keepTreeHashes** turns on/off the storing of the hashes that were used as leaves + for building the Merkle tree, default is ``"off"``. +- **sig.keepRecordHashes** turns on/off the storing of the hashes of the log records, default is ``"on"``. +- **sig.confInterval** defines interval of periodic request for aggregator configuration in seconds, default is ``3600``. +- **sig.randomSource** defines source of random as file, default is ``"/dev/urandom"``. +- **sig.debugFile** enables libksi log and redirects it into file specified. Note that logger level has to be specified (see ``sig.debugLevel``). +- **sig.debugLevel** specifies libksi log level. Note that log file has to be specified (see ``sig.debugFile``). + + - *0* None (default). + - *1* Error. + - *2* Warning. + - *3* Notice. + - *4* Info. + - *5* Debug. + +The log signature file, which stores the KSI signatures and information about the signed blocks, appears in the same directory as the log file itself. + +Sample +###### + +To sign the logs in ``/var/log/secure`` with KSI: +:: + + # The authpriv file has restricted access and is signed with KSI + authpriv.* action(type="omfile" file="/var/log/secure" + sig.provider="ksi_ls12" + sig.syncmode="sync" + sig.hashFunction="SHA2-256" + sig.block.levelLimit="8" + sig.block.timeLimit="0" + sig.aggregator.url= + "http://tryout.guardtime.net:8080/gt-signingservice" + sig.aggregator.user="rsmith" + sig.aggregator.key="secret" + sig.aggregator.hmacAlg="SHA2-256" + sig.keepTreeHashes="off" + sig.keepRecordHashes="on") + + +Note that all parameter values must be between quotation marks! + +See Also +######## + +To better understand the log signing mechanism and the module's possibilities it is advised to consult with: + +- `KSI Rsyslog Integration User Guide <https://docs.guardtime.net/ksi-rsyslog-guide/>`_ +- `KSI Developer Guide <https://docs.guardtime.net/ksi-dev-guide/>`_ + +Access for both of these documents requires Guardtime tryout service credentials, available from `<https://guardtime.com/technology/blockchain-developers>`_ diff --git a/source/configuration/modules/workflow.rst b/source/configuration/modules/workflow.rst new file mode 100644 index 0000000..9df5f74 --- /dev/null +++ b/source/configuration/modules/workflow.rst @@ -0,0 +1,30 @@ +Where are the modules integrated into the Message Flow? +======================================================= + +Depending on their module type, modules may access and/or modify +messages at various stages during rsyslog's processing. Note that only +the "core type" (e.g. input, output) but not any type derived from it +(message modification module) specifies when a module is called. + +The simplified workflow is as follows: + +.. figure:: module_workflow.png + :align: center + :alt: module_workflow + +As can be seen, messages are received by input modules, then passed to +one or many parser modules, which generate the in-memory representation +of the message and may also modify the message itself. The internal +representation is passed to output modules, which may output a message +and (with the interfaces introduced in v5) may also modify +message object content. + +String generator modules are not included inside this picture, because +they are not a required part of the workflow. If used, they operate "in +front of" the output modules, because they are called during template +generation. + +Note that the actual flow is much more complex and depends a lot on +queue and filter settings. This graphic above is a high-level message +flow diagram. + diff --git a/source/configuration/nomatch.rst b/source/configuration/nomatch.rst new file mode 100644 index 0000000..9cb84b7 --- /dev/null +++ b/source/configuration/nomatch.rst @@ -0,0 +1,47 @@ +Property Replacer nomatch mode +------------------------------ + +**The "nomatch-Mode" specifies which string the property replacer shall +return if a regular expression did not find the search string.**. +Traditionally, the string "\*\*NO MATCH\*\*" was returned, but many +people complained this was almost never useful. Still, this mode is +support as "**DFLT**\ " for legacy configurations. + +Three additional and potentially useful modes exist: in one (**BLANK**) +a blank string is returned. This is probably useful for inserting values +into databases where no value shall be inserted if the expression could +not be found. + +A similar mode is "**ZERO**\ " where the string "0" is returned. This is +suitable for numerical values. A use case may be that you record a +traffic log based on firewall rules and the "bytes transmitted" counter +is extracted via a regular expression. If no "bytes transmitted" counter +is available in the current message, it is probably a good idea to +return an empty string, which the database layer can turn into a zero. + +The other mode is "**FIELD**\ ", in which the complete field is +returned. This may be useful in cases where absence of a match is +considered a failure and the message that triggered it shall be logged. + +If in doubt, **it is highly suggested to use the** `rsyslog online regular +expression checker and generator <http://www.rsyslog.com/tool-regex>`_ +**to see these options in action**. With that online tool, you can craft +regular expressions based on samples and try out the different modes. + +Summary of nomatch Modes +~~~~~~~~~~~~~~~~~~~~~~~~ + ++------------+-----------------------------------------------------------+ +| **Mode** | **Returned** | ++------------+-----------------------------------------------------------+ +| DFLT | "\*\*NO MATCH\*\*" | ++------------+-----------------------------------------------------------+ +| BLANK | "" (empty string) | ++------------+-----------------------------------------------------------+ +| ZERO | "0" | ++------------+-----------------------------------------------------------+ +| FIELD | full content of original field | ++------------+-----------------------------------------------------------+ +| | `Interactive Tool <http://www.rsyslog.com/tool-regex>`_ | ++------------+-----------------------------------------------------------+ + diff --git a/source/configuration/output_channels.rst b/source/configuration/output_channels.rst new file mode 100644 index 0000000..24e26c7 --- /dev/null +++ b/source/configuration/output_channels.rst @@ -0,0 +1,61 @@ +Output Channels +--------------- + +Output Channels are a new concept first introduced in rsyslog 0.9.0. +**As of this writing, it is most likely that they will be replaced by +something different in the future.** So if you use them, be prepared to +change you configuration file syntax when you upgrade to a later +release. +The idea behind output channel definitions is that it shall provide an +umbrella for any type of output that the user might want. In essence, +this is the "file" part of selector lines (and this is why we are not +sure output channel syntax will stay after the next review). There is a +difference, though: selector channels both have filter conditions +(currently facility and severity) as well as the output destination. +they can only be used to write to files - not pipes, ttys or whatever +Output channels define the output definition, only. As of this build, +else. If we stick with output channels, this will change over time. + +In concept, an output channel includes everything needed to know about +an output actions. In practice, the current implementation only carries +a filename, a maximum file size and a command to be issued when this +file size is reached. More things might be present in future version, +which might also change the syntax of the directive. + +Output channels are defined via an $outchannel directive. It's syntax is +as follows: +$outchannel name,file-name,max-size,action-on-max-size +name is the name of the output channel (not the file), file-name is the +file name to be written to, max-size the maximum allowed size and +action-on-max-size a command to be issued when the max size is reached. +This command always has exactly one parameter. The binary is that part +of action-on-max-size before the first space, its parameter is +everything behind that space. +Please note that max-size is queried BEFORE writing the log message to +the file. So be sure to set this limit reasonably low so that any +message might fit. For the current release, setting it 1k lower than you +expected is helpful. The max-size must always be specified in bytes - +there are no special symbols (like 1k, 1m,...) at this point of +development. +Keep in mind that $outchannel just defines a channel with "name". It +does not activate it. To do so, you must use a selector line (see +below). That selector line includes the channel name plus an $ sign in +front of it. A sample might be: +\*.\* :omfile:$mychannel +In its current form, output channels primarily provide the ability to +size-limit an output file. To do so, specify a maximum size. When this +size is reached, rsyslogd will execute the action-on-max-size command +and then reopen the file and retry. The command should be something like +a `log rotation script <log_rotation_fix_size.html>`_ or a similar +thing. + +If there is no action-on-max-size command or the command did not resolve +the situation, the file is closed and never reopened by rsyslogd +(except, of course, by huping it). This logic was integrated when we +first experienced severe issues with files larger 2gb, which could lead +to rsyslogd dumping core. In such cases, it is more appropriate to stop +writing to a single file. Meanwhile, rsyslogd has been fixed to support +files larger 2gb, but obviously only on file systems and operating +system versions that do so. So it can still make sense to enforce a 2gb +file size limit. + diff --git a/source/configuration/parser.rst b/source/configuration/parser.rst new file mode 100644 index 0000000..d54e0df --- /dev/null +++ b/source/configuration/parser.rst @@ -0,0 +1,87 @@ +Parser +====== + +.. index:: ! parser +.. _cfgobj_input: + +The ``parser`` object, as its name suggests, describes message parsers. +Message parsers have a standard parser name, which can be used by simply +loading the parser module. Only when specific parameters need to be set +the parser object is needed. + +In that case, it is used to define a new parser name (aka "parser definition") +which configures this name to use the parser module with set parameters. +This is important as the ruleset() object does not support to set parser +parameters. Instead, if parameters are needed, a proper parser name must +be defined using the parser() object. A parser name defined via the +parser() object can be used wherever a parser name can occur. + +Note that not all message parser modules are supported in the parser() +object. The reason is that many do not have any user-selectable +parameters and as such, there is no point in issuing a parser() object +for them. + +The parser object has different parameters: + +- those that apply to all parser and are generally available for + all of them. These are documented below. +- parser-specific parameters. These are specific to a certain parser + module. They are documented by the :doc:`parser module <modules/idx_parser>` + in question. + + +General Parser Parameters +------------------------- + +Note: parameter names are case-insensitive. + +.. function:: name <name-string> + + *Mandatory* + + This names the parser. Names starting with "rsyslog." are reserved for + rsyslog use and must not be used. It is suggested to replace "rsyslog." + with "custom." and keep the rest of the name descriptive. However, this + is not enforced and just good practice. + +.. function:: type <type-string> + + *Mandatory* + + The ``<type-string>`` is a string identifying the parser module as given + it each module's documentation. Do not mistake the parser module name + with its default parser name. + For example, the + :doc:`Cisco IOS message parser module <modules/pmciscoios>` parser module + name is "pmciscoios", whereas it's default parser name is + "rsyslog.pmciscoios". + +Samples +------- +The following example creates a custom parser definition and uses it within a ruleset: + +:: + + module(load="pmciscoios") + parser(name="custom.pmciscoios.with_origin" type="pmciscoios") + + ruleset(name="myRuleset" parser="custom.pmciscoios.with_origin") { + ... do something here ... + } + +The following example uses multiple parsers within a ruleset without a parser object (the order is important): + +:: + + module(load="pmaixforwardedfrom") + module(load="pmlastmsg") + + ruleset(name="myRuleset" parser=["rsyslog.lastline","rsyslog.aixforwardedfrom","rsyslog.rfc5424","rsyslog.rfc3164"]) { + ... do something here ... + } + + + +A more elaborate example can also be found in the +:doc:`Cisco IOS message parser module <modules/pmciscoios>` documentation. + diff --git a/source/configuration/percentile_stats.rst b/source/configuration/percentile_stats.rst new file mode 100644 index 0000000..cd9781d --- /dev/null +++ b/source/configuration/percentile_stats.rst @@ -0,0 +1,151 @@ +Percentile Stats +================ + +The Percentile Stats component allows user to configure statistical namespaces +(called stats-buckets), which can then be used to record statistical values for +publishing periodic percentiles and summaries. + + +Percentile Stats Configuration +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Percentile-stats configuration involves a **two part setup**. First, define a ``percentile-stats`` bucket and +its attributes. Then, add statistics and observation values to the namespace, by making ``percentile_observation`` calls. + +percentile_stats(name="<bucket>", percentiles=[...], windowsize="<count>"...) (object) +----------------------------------------------------------------------------------------------------------- + +**Defines** the statistics bucket(identified by the bucket-name) and allows user to set some properties that control behavior of the bucket. + +.. code-block:: + + percentile_stats(name="bucket_name" percentiles=["95"] windowsize=1000) + +Parameters: + **name** <string literal, mandatory> : Bucket name of a set of statistics to sample. + + **percentiles** <array, mandatory> : A list of strings, with percentile statistic value between 1, 100 inclusive, user would like to publish to impstats. + This list of percentiles would apply for all statistics tracked under this bucket. + + **windowSize** <number, mandatory> : A max *sliding* window (FIFO) size - rounded *up* to the nearest power of 2, that is larger than the given window size. + Specifies the maximum number of observations stored over an impstats reporting interval. + This attribute would apply to all statistics tracked under this bucket. + + **delimiter** <string literal, default: "."> : A single character delimiter used in the published fully qualified statname. + This delimiter would apply to all statistics tracked under this bucket. + + +A definition setting all the parameters looks like: + +.. code-block:: + + percentile_stats(name="host_statistics" percentiles=["50", "95", "99"] windowsize="1000" delimiter="|") + + +percentile_observe("<bucket>", "<statname>", <value>) (function) +---------------------------------------------------------------- + +**Adds** a statistical sample to the statistic set identified by the <bucket>, and <statname>. +If the number of values exceed the defined **windowSize**, the earliest recorded value is dropped +and the new value is recorded. + +.. note:: + Sums and counts only include values found in the sliding window. + Min and Max summary values will include all values seen over an observation interval. + See :ref:`Reporting` below for more info on published summary statistics. + + + +Parameters: + **bucket name** <string literal, mandatory> : Name of the statistic bucket + + **stat name** <string literal, mandatory> : Name of the statistic to record (this name will be combined with a percentile and reported by impstats to identify the percentile) + + + **value** <expression resulting in an integer value, mandatory> : Value to record + +A ``percentile_observe`` call looks like: + +.. code-block:: + + set $.ret = percentile_observe("host_statistics", "msg_per_host", $.mycount); + + if ($.ret != 0) then { + .... + } + +``$.ret`` captures the error-code. It has value ``0`` when operation is successful and non-zero when it fails. It uses Rsyslog error-codes. + + +.. _Reporting: + +Reporting +^^^^^^^^^ + +The following example metrics would be reported by **impstats** + +Legacy format: + +.. code-block:: + + ... + global: origin=percentile host_statistics.new_metric_add=1 host_statistics.ops_overflow=0 + ... + host_statistics: origin=percentile.bucket msg_per_host|p95=1950 msg_per_host|p50=1500 msg_per_host|p99=1990 msg_per_host|window_min=1001 msg_per_host|window_max=2000 msg_per_host|window_sum=1500500 msg_per_host|window_count=1000 + ... + +Json(variants with the same structure are used in other Json based formats such as ``cee`` and ``json-elasticsearch``) format: + +.. code-block:: + + ... + { "name": "global", "origin": "percentile", "values": { "host_statistics.new_metric_add": 1, "host_statistics.ops_overflow": 0 } } + ... + { "name": "host_statistics", "origin": "percentile.bucket", "values": { "msg_per_host|p95": 1950, "msg_per_host|p50": 1500, "msg_per_host|p99": 1990, "msg_per_host|window_min": 1001, "msg_per_host|window_max": 2000, "msg_per_host|window_sum": 1500500, "msg_per_host|window_count": 1000 } } + ... + +In this case counters are encapsulated inside an object hanging off top-level-key ``values``. + +Fields +------ + +**global: origin=percentile.bucket**: + **new_metric_add**: Number of "new" metrics added (new counters created). + + **ops_overflow**: Number of operations ignored because number-of-counters-tracked has hit configured max-cardinality. + +**host_statistic: origin=percentile.bucket**: + **msg_per_host|<pXX>**: percentile value, identified by <stat name><delimiter><pXX>, where pXX is one of the requested ``percentiles`` requested. + + + **msg_per_host|window_min**: minimum recorded value in the statistical population window. Identified by ``<stat-name><delimiter>window_min`` + + + **msg_per_host|window_max**: maximum recorded value in the statistical population window. Identified by ``<stat-name><delimiter>window_max`` + + + **msg_per_host|window_sum**: sum of recorded values in the statistical population window. Identified by ``<stat-name>delimiter>window_sum`` + + + **msg_per_host|window_count**: count of recorded values in the statistical population window. Identified by ``<stat-name><delimiter>window_count`` + + +Implementation Details +---------------------- + +Percentile stats module uses a sliding window, sized according to the **windowSize** configuration parameter. +The percentile values are calculated, once every **impstats** interval. +Percentiles are calculated according to the standard “nearest-rank” method: + + + n = CEILING((P/100) x N) where: + +* n - index to percentile value +* P - percentile [1, 100] +* N - number of values recorded + + +.. note:: + In order to sort the values, a standard implementation of quicksort is used, which performs pretty well + on average. However quicksort quickly degrades when there are many repeated elements, thus it is best + to avoid repeated values if possible. diff --git a/source/configuration/properties.rst b/source/configuration/properties.rst new file mode 100644 index 0000000..863984f --- /dev/null +++ b/source/configuration/properties.rst @@ -0,0 +1,322 @@ +rsyslog Properties +================== + +Data items in rsyslog are called "properties". They can have different +origin. The most important ones are those that stem from received +messages. But there are also others. Whenever you want to access data items, +you need to access the respective property. + +Properties are used in + +- :doc:`templates <templates>` +- conditional statements + +The property name is case-insensitive (prior to 3.17.0, they were case-sensitive). + +Note: many users refer to "rsyslog properties" as "rsyslog variables". You can treat +them as synonymous. +Read how `rsyslog lead author Rainer Gerhards explains the naming +difference <https://rainer.gerhards.net/2020/08/rsyslog-template-variables-where-to-find-them.html">`_. + +Message Properties +------------------ +These are extracted by rsyslog parsers from the original message. All message +properties start with a letter. + +The following message properties exist: + +**msg** + the MSG part of the message (aka "the message" ;)) + +**rawmsg** + the message "as is". Should be useful for debugging and also if a message + should be forwarded totally unaltered. + Please notice *EscapecontrolCharactersOnReceive* is enabled by default, so + it may be different from what was received in the socket. + +**rawmsg-after-pri** + Almost the same as **rawmsg**, but the syslog PRI is removed. + If no PRI was present, **rawmsg-after-pri** is identical to + **rawmsg**. Note that the syslog PRI is header field that + contains information on syslog facility and severity. It is + enclosed in greater-than and less-than characters, e.g. + "<191>". This field is often not written to log files, but + usually needs to be present for the receiver to properly + classify the message. There are some rare cases where one + wants the raw message, but not the PRI. You can use this + property to obtain that. In general, you should know that you + need this format, otherwise stay away from the property. + +**hostname** + hostname from the message + +**source** + alias for HOSTNAME + +**fromhost** + hostname of the system the message was received from (in a relay chain, + this is the system immediately in front of us and not necessarily the + original sender). This is a DNS-resolved name, except if that is not + possible or DNS resolution has been disabled. + +**fromhost-ip** + The same as fromhost, but always as an IP address. Local inputs (like + imklog) use 127.0.0.1 in this property. + +**syslogtag** + TAG from the message + +**programname** + the "static" part of the tag, as defined by BSD syslogd. For example, + when TAG is "named[12345]", programname is "named". + + Precisely, the programname is terminated by either (whichever occurs first): + + - end of tag + - nonprintable character + - ':' + - '[' + - '/' + + The above definition has been taken from the FreeBSD syslogd sources. + + Please note that some applications include slashes in the static part + of the tag, e.g. "app/foo[1234]". In this case, programname is "app". + If they store an absolute path name like in "/app/foo[1234]", programname + will become empty (""). If you need to actually store slashes as + part of the programname, you can use the global option + + global(parser.permitSlashInProgramName="on") + + to permit this. Then, a syslogtag of "/app/foo[1234]" will result in + programname being "/app/foo". Note: this option is available starting at + rsyslogd version 8.25.0. + +**pri** + PRI part of the message - undecoded (single value) + +**pri-text** + the PRI part of the message in a textual form with the numerical PRI + appended in brackets (e.g. "local0.err<133>") + +**iut** + the monitorware InfoUnitType - used when talking to a + `MonitorWare <https://www.monitorware.com>`_ backend (also for + `Adiscon LogAnalyzer <https://loganalyzer.adiscon.com/>`_) + +**syslogfacility** + the facility from the message - in numerical form + +**syslogfacility-text** + the facility from the message - in text form + +**syslogseverity** + severity from the message - in numerical form + +**syslogseverity-text** + severity from the message - in text form + +**syslogpriority** + an alias for syslogseverity - included for historical reasons (be + careful: it still is the severity, not PRI!) + +**syslogpriority-text** + an alias for syslogseverity-text + +**timegenerated** + timestamp when the message was RECEIVED. Always in high resolution + +**timereported** + timestamp from the message. Resolution depends on what was provided in + the message (in most cases, only seconds) + +**timestamp** + alias for timereported + +**protocol-version** + The contents of the PROTOCOL-VERSION field from IETF draft + draft-ietf-syslog-protocol + +**structured-data** + The contents of the STRUCTURED-DATA field from IETF draft + draft-ietf-syslog-protocol + +**app-name** + The contents of the APP-NAME field from IETF draft + draft-ietf-syslog-protocol + +**procid** + The contents of the PROCID field from IETF draft + draft-ietf-syslog-protocol + +**msgid** + The contents of the MSGID field from IETF draft + draft-ietf-syslog-protocol + +**inputname** + The name of the input module that generated the message (e.g. + "imuxsock", "imudp"). Note that not all modules necessarily provide this + property. If not provided, it is an empty string. Also note that the + input module may provide any value of its liking. Most importantly, it + is **not** necessarily the module input name. Internal sources can also + provide inputnames. Currently, "rsyslogd" is defined as inputname for + messages internally generated by rsyslogd, for example startup and + shutdown and error messages. This property is considered useful when + trying to filter messages based on where they originated - e.g. locally + generated messages ("rsyslogd", "imuxsock", "imklog") should go to a + different place than messages generated somewhere else. + +**uuid** + + *Only Available if rsyslog is build with --enable-uuid* + + A UUID for the message. It is not present by default, but will be created + on first read of the uuid property. Thereafter, in the local rsyslog + instance, it will always be the same value. This is also true if rsyslog + is restarted and messages stayed in an on-disk queue. + + Note well: the UUID is **not** automatically transmitted to remote + syslog servers when forwarding. If that is needed, a special template + needs to be created that contains the uuid. Likewise, the receiver must + parse that UUID from that template. + + The uuid property is most useful if you would like to track a single + message across multiple local destination. An example is messages being + written to a database as well as to local files. + +**jsonmesg** + + *Available since rsyslog 8.3.0* + + The whole message object as JSON representation. Note that the JSON + string will *not* include an LF and it will contain *all other message + properties* specified here as respective JSON containers. It also includes + all message variables in the "$!" subtree (this may be null if none are + present). + + This property is primarily meant as an interface to other systems and + tools that want access to the full property set (namely external + plugins). Note that it contains the same data items potentially multiple + times. For example, parts of the syslog tag will by contained in the + rawmsg, syslogtag, and programname properties. As such, this property + has some additional overhead. Thus, it is suggested to be used only + when there is actual need for it. + +System Properties +----------------- +These properties are provided by the rsyslog core engine. They are **not** +related to the message. All system properties start with a dollar-sign. + +Special care needs to be taken in regard to time-related system variables: + +* ``timereported`` contains the timestamp that is contained within the + message header. Ideally, it resembles the time when the message was + created at the original sender. + Depending on how long the message was in the relay chain, this + can be quite old. +* ``timegenerated`` contains the timestamp when the message was received + by the local system. Here "received" actually means the point in time + when the message was handed over from the OS to rsyslog's reception + buffers, but before any actual processing takes place. This also means + a message is "received" before it is placed into any queue. Note that + depending on the input, some minimal processing like extraction of the + actual message content from the receive buffer can happen. If multiple + messages are received via the same receive buffer (a common scenario + for example with TCP-based syslog), they bear the same ``timegenerated`` + stamp because they actually were received at the same time. +* ``$now`` is **not** from the message. It is the system time when the + message is being **processed**. There is always a small difference + between ``timegenerated`` and ``$now`` because processing always + happens after reception. If the message is sitting inside a queue + on the local system, the time difference between the two can be some + seconds (e.g. due to a message burst and in-memory queueing) up to + several hours in extreme cases where a message is sitting inside a + disk queue (e.g. due to a database outage). The ``timereported`` + property is usually older than ``timegenerated``, but may be totally + different due to differences in time and time zone configuration + between systems. + +The following system properties exist: + +**$bom** + The UTF-8 encoded Unicode byte-order mask (BOM). This may be useful in + templates for RFC5424 support, when the character set is know to be + Unicode. + +**$myhostname** + The name of the current host as it knows itself (probably useful for + filtering in a generic way) + +Time-Related System Properties +.............................. + +All of these system properties exist in a local time variant (e.g. \$now) +and a variant that emits UTC (e.g. \$now-utc). The UTC variant is always +available by appending "-utc". Note that within a single template, only +the localtime or UTC variant should be used. While it is possible to mix +both variants within a single template, it is **not** guaranteed that +they will provide exactly the same time. The technical reason is that +rsyslog needs to re-query system time when the variant is changed. Because +of this, we strongly recommend not mixing both variants in the same +template. + +Note that use in different templates will generate a consistent timestamp +within each template. However, as $now always provides local system time +at time of using it, time may advance and consequently different templates +may have different time stamp. To avoid this, use *timegenerated* instead. + +**$now** + The current date stamp in the format YYYY-MM-DD + +**$year** + The current year (4-digit) + +**$month** + The current month (2-digit) + +**$day** + The current day of the month (2-digit) + +**$wday** + The current week day as defined by 'gmtime()'. 0=Sunday, ..., 6=Saturday + +**$hour** + The current hour in military (24 hour) time (2-digit) + +**$hhour** + The current half hour we are in. From minute 0 to 29, this is always 0 + while from 30 to 59 it is always 1. + +**$qhour** + The current quarter hour we are in. Much like $HHOUR, but values range + from 0 to 3 (for the four quarter hours that are in each hour) + +**$minute** + The current minute (2-digit) + +**$now-unixtimestamp** + The current time as a unix timestamp (seconds since epoch). This actually + is a monotonically increasing counter and as such can also be used for any + other use cases that require such counters. This is an example of how + to use it for rate-limiting:: + + # Get Unix timestamp of current message + set $.tnow = $$now-unixtimestamp + + # Rate limit info to 5 every 60 seconds + if ($!severity == 6 and $!facility == 17) then { + if (($.tnow - $/trate) > 60) then { + # 5 seconds window expired, allow more messages + set $/trate = $.tnow; + set $/ratecount = 0; + } + if ($/ratecount > 5) then { + # discard message + stop + } else { + set $/ratecount = $/ratecount + 1; + } + } + + NOTE: by definition, there is no "UTC equivalent" of the + $now-unixtimestamp property. diff --git a/source/configuration/property_replacer.rst b/source/configuration/property_replacer.rst new file mode 100644 index 0000000..3d99f5f --- /dev/null +++ b/source/configuration/property_replacer.rst @@ -0,0 +1,374 @@ +The Property Replacer +===================== + +**The property replacer is a core component in rsyslogd's** `string template +system <templates.html>`_. A syslog message has a number of well-defined properties. +Each of these properties can be accessed **and** manipulated by +the property replacer. With it, it is easy to use only part of a +property value or manipulate the value, e.g. by converting all +characters to lower case. + +Accessing Properties +-------------------- + +Syslog message properties are used inside templates. They are accessed +by putting them between percent signs. Properties can be modified by the +property replacer. The full syntax is as follows: + +:: + + %property:fromChar:toChar:options% + +Available Properties +^^^^^^^^^^^^^^^^^^^^ + +The property replacer can use all :doc:`rsyslog properties <properties>`. + +Character Positions +^^^^^^^^^^^^^^^^^^^^ + +**FromChar** and **toChar** are used to build substrings. They +specify the offset within the string that should be copied. Offset +counting starts at 1, so if you need to obtain the first 2 characters of +the message text, you can use this syntax: "%msg:1:2%". If you do not +wish to specify from and to, but you want to specify options, you still +need to include the colons. For example, if you would like to convert +the full message text to lower case, use "%msg:::lowercase%". If you +would like to extract from a position until the end of the string, you +can place a dollar-sign ("$") in toChar (e.g. %msg:10:$%, which will +extract from position 10 to the end of the string). + +There is also support for **regular expressions**. To use them, you need +to place a "R" into FromChar. This tells rsyslog that a regular +expression instead of position-based extraction is desired. The actual +regular expression must then be provided in toChar. The regular +expression **must** be followed by the string "--end". It denotes the +end of the regular expression and will not become part of it. If you are +using regular expressions, the property replacer will return the part of +the property text that matches the regular expression. An example for a +property replacer sequence with a regular expression is: +"%msg:R:.\*Sev:. \\(.\*\\) \\[.\*--end%" + +It is possible to specify some parameters after the "R". These are +comma-separated. They are: + +R,<regexp-type>,<submatch>,<:doc:`nomatch <nomatch>`\ >,<match-number> + +regexp-type is either "BRE" for Posix basic regular expressions or "ERE" +for extended ones. The string must be given in upper case. The default +is "BRE" to be consistent with earlier versions of rsyslog that did not +support ERE. The submatch identifies the submatch to be used with the +result. A single digit is supported. Match 0 is the full match, while 1 +to 9 are the actual submatches. The match-number identifies which match +to use, if the expression occurs more than once inside the string. +Please note that the first match is number 0, the second 1 and so on. Up +to 10 matches (up to number 9) are supported. Please note that it would +be more natural to have the match-number in front of submatch, but this +would break backward-compatibility. So the match-number must be +specified after "nomatch". + +:doc:`nomatch <nomatch>` specifies what should be used in +case no match is found. + +The following is a sample of an ERE expression that takes the first +submatch from the message string and replaces the expression with the +full field if no match is found: + +:: + +%msg:R,ERE,1,FIELD:for (vlan[0-9]\*):--end% + +and this takes the first submatch of the second match of said +expression: + +:: + +%msg:R,ERE,1,FIELD,1:for (vlan[0-9]\*):--end% + +**Please note: there is also a** `rsyslog regular expression +checker/generator <http://www.rsyslog.com/tool-regex>`_ **online tool +available.** With that tool, you can check your regular expressions and +also generate a valid property replacer sequence. Usage of this tool is +recommended. Depending on the version offered, the tool may not cover +all subtleties that can be done with the property replacer. It +concentrates on the most often used cases. So it is still useful to +hand-craft expressions for demanding environments. + +**Also, extraction can be done based on so-called "fields"**. To do so, +place a "F" into FromChar. A field in its current definition is anything +that is delimited by a delimiter character. The delimiter by default is +TAB (US-ASCII value 9). However, if can be changed to any other US-ASCII +character by specifying a comma and the **decimal** US-ASCII value of +the delimiter immediately after the "F". For example, to use comma (",") +as a delimiter, use this field specifier: "F,44". If your syslog data +is delimited, this is a quicker way to extract than via regular +expressions (actually, a *much* quicker way). Field counting starts at +1. Field zero is accepted, but will always lead to a "field not found" +error. The same happens if a field number higher than the number of +fields in the property is requested. The field number must be placed in +the "ToChar" parameter. An example where the 3rd field (delimited by +TAB) from the msg property is extracted is as follows: "%msg:F:3%". The +same example with semicolon as delimiter is "%msg:F,59:3%". + +The use of fields does not permit to select substrings, what is rather +unfortunate. To solve this issue, starting with 6.3.9, fromPos and toPos +can be specified for strings as well. However, the syntax is quite ugly, +but it was the only way to integrate this functionality into the +already-existing system. To do so, use ",fromPos" and ",toPos" during +field extraction. Let's assume you want to extract the substring from +position 5 to 9 in the previous example. Then, the syntax is as follows: +"%msg:F,59,5:3,9%". As you can see, "F,59" means field-mode, with +semicolon delimiter and ",5" means starting at position 5. Then "3,9" +means field 3 and string extraction to position 9. + +Please note that the special characters "F" and "R" are case-sensitive. +Only upper case works, lower case will return an error. There are no +white spaces permitted inside the sequence (that will lead to error +messages and will NOT provide the intended result). + +Each occurrence of the field delimiter starts a new field. However, if +you add a plus sign ("+") after the field delimiter, multiple +delimiters, one immediately after the others, are treated as separate +fields. This can be useful in cases where the syslog message contains +such sequences. A frequent case may be with code that is written as +follows: + +```` + +:: + + int n, m; + ... + syslog(LOG_ERR, "%d test %6d", n, m); + +This will result into things like this in syslog messages: "1 +test 2", "1 test 23", "1 test 234567" + +As you can see, the fields are delimited by space characters, but their +exact number is unknown. They can properly be extracted as follows: + +:: + + "%msg:F,32:2%" to "%msg:F,32+:2%". + +This feature was suggested by Zhuang Yuyao and implemented by him. It is +modeled after perl compatible regular expressions. + +Property Options +^^^^^^^^^^^^^^^^ + +**Property options** are case-insensitive. Currently, the following +options are defined: + +**uppercase** + convert property to uppercase only + +**lowercase** + convert property text to lowercase only + +**fixed-width** + changes behaviour of toChar so that it pads the source string with spaces + up to the value of toChar if the source string is shorter. + *This feature was introduced in rsyslog 8.13.0* + +**json** + encode the value so that it can be used inside a JSON field. This means + that several characters (according to the JSON spec) are being escaped, for + example US-ASCII LF is replaced by "\\n". + The json option cannot be used together with either jsonf or csv options. + +**jsonf**\[:outname\] + (available in 6.3.9+) + This signifies that the property should be expressed as a JSON field. + That means not only the property is written, but rather a complete JSON field in + the format + + ``"fieldname"="value"`` + + where "fieldname" is given in the *outname* property (or the property name + if none was assigned) and value is the end result of property replacer operation. + Note that value supports all property replacer options, like substrings, case + conversion and the like. Values are properly JSON-escaped, however field names are + (currently) not, so it is expected that proper field names are configured. + The jsonf option cannot be used together with either json or csv options. + + For more information you can read `this article from Rainer's blog + <https://rainer.gerhards.net/2012/04/rsyslog-templates-json.html>`_. + +**csv** + formats the resulting field (after all modifications) in CSV format as + specified in `RFC 4180 <http://www.ietf.org/rfc/rfc4180.txt>`_. Rsyslog + will always use double quotes. Note that in order to have full + CSV-formatted text, you need to define a proper template. An example is + this one: + $template csvline,"%syslogtag:::csv%,%msg:::csv%" + Most importantly, you need to provide the commas between the fields + inside the template. + *This feature was introduced in rsyslog 4.1.6.* + +**drop-last-lf** + The last LF in the message (if any), is dropped. Especially useful for + PIX. + +**date-utc** + convert data to UTC prior to outputting it (available since 8.18.0) + +**date-mysql** + format as mysql date + +**date-rfc3164** + format as RFC 3164 date + +**date-rfc3164-buggyday** + similar to date-rfc3164, but emulates a common coding error: RFC 3164 + demands that a space is written for single-digit days. With this option, + a zero is written instead. This format seems to be used by syslog-ng and + the date-rfc3164-buggyday option can be used in migration scenarios + where otherwise lots of scripts would need to be adjusted. It is + recommended *not* to use this option when forwarding to remote hosts - + they may treat the date as invalid (especially when parsing strictly + according to RFC 3164). + + *This feature was introduced in rsyslog 4.6.2 and v4 versions above and + 5.5.3 and all versions above.* + +**date-rfc3339** + format as RFC 3339 date + +**date-unixtimestamp** + Format as a unix timestamp (seconds since epoch) + +**date-year** + just the year part (4-digit) of a timestamp + +**date-month** + just the month part (2-digit) of a timestamp + +**date-day** + just the day part (2-digit) of a timestamp + +**date-hour** + just the hour part (2-digit, 24-hour clock) of a timestamp + +**date-minute** + just the minute part (2-digit) of a timestamp + +**date-second** + just the second part (2-digit) of a timestamp + +**date-subseconds** + just the subseconds of a timestamp (always 0 for a low precision + timestamp) + +**date-tzoffshour** + just the timezone offset hour part (2-digit) of a timestamp + +**date-tzoffsmin** + just the timezone offset minute part (2-digit) of a timestamp. Note + that this is usually 0, but there are some time zones that have + offsets which are not hourly-granular. If so, this is the minute + offset. + +**date-tzoffsdirection** + just the timezone offset direction part of a timestamp. This + specifies if the offsets needs to be added ("+") or subtracted ("-") + to the timestamp in order to get UTC. + +**date-ordinal** + returns the ordinal for the given day, e.g. it is 2 for January, 2nd + +**date-iso-week** and **date-iso-week-year** + return the ISO week number adn week-numbering year, which should be used together. See `ISO week date <https://en.wikipedia.org/wiki/ISO_week_date>`_ for more details + +**date-week** + returns the week number + +**date-wday** + just the weekday number of the timstamp. This is a single digit, + with 0=Sunday, 1=Monday, ..., 6=Saturday. + +**date-wdayname** + just the abbreviated english name of the weekday (e.g. "Mon", "Sat") of + the timestamp. + +**escape-cc** + replace control characters (ASCII value 127 and values less then 32) + with an escape sequence. The sequence is "#<charval>" where charval is + the 3-digit decimal value of the control character. For example, a + tabulator would be replaced by "#009". + Note: using this option requires that + `$EscapeControlCharactersOnReceive <rsconf1_escapecontrolcharactersonreceive.html>`_ + is set to off. + +**space-cc** + replace control characters by spaces + Note: using this option requires that + `$EscapeControlCharactersOnReceive <rsconf1_escapecontrolcharactersonreceive.html>`_ + is set to off. + +**drop-cc** + drop control characters - the resulting string will neither contain + control characters, escape sequences nor any other replacement character + like space. + Note: using this option requires that + `$EscapeControlCharactersOnReceive <rsconf1_escapecontrolcharactersonreceive.html>`_ + is set to off. + +**compressspace** + compresses multiple spaces (US-ASCII SP character) inside the + string to a single one. This compression happens at a very late + stage in processing. Most importantly, it happens after substring + extraction, so the **FromChar** and **ToChar** positions are **NOT** + affected by this option. (available since v8.18.0) + +**sp-if-no-1st-sp** + This option looks scary and should probably not be used by a user. For + any field given, it returns either a single space character or no + character at all. Field content is never returned. A space is returned + if (and only if) the first character of the field's content is NOT a + space. This option is kind of a hack to solve a problem rooted in RFC + 3164: 3164 specifies no delimiter between the syslog tag sequence and + the actual message text. Almost all implementation in fact delimit the + two by a space. As of RFC 3164, this space is part of the message text + itself. This leads to a problem when building the message (e.g. when + writing to disk or forwarding). Should a delimiting space be included if + the message does not start with one? If not, the tag is immediately + followed by another non-space character, which can lead some log parsers + to misinterpret what is the tag and what the message. The problem + finally surfaced when the klog module was restructured and the tag + correctly written. It exists with other message sources, too. The + solution was the introduction of this special property replacer option. + Now, the default template can contain a conditional space, which exists + only if the message does not start with one. While this does not solve + all issues, it should work good enough in the far majority of all cases. + If you read this text and have no idea of what it is talking about - + relax: this is a good indication you will never need this option. Simply + forget about it ;) + +**secpath-drop** + Drops slashes inside the field (e.g. "a/b" becomes "ab"). Useful for + secure pathname generation (with dynafiles). + +**secpath-replace** + Replace slashes inside the field by an underscore. (e.g. "a/b" becomes + "a\_b"). Useful for secure pathname generation (with dynafiles). + +To use multiple options, simply place them one after each other with a +comma delimiting them. For example "escape-cc,sp-if-no-1st-sp". If you +use conflicting options together, the last one will override the +previous one. For example, using "escape-cc,drop-cc" will use drop-cc +and "drop-cc,escape-cc" will use escape-cc mode. + +Further Links +------------- + +- Article on ":doc:`Recording the Priority of Syslog + Messages <../tutorials/recording_pri>`" (describes use of + templates to record severity and facility of a message) +- `Configuration file syntax <rsyslog_conf.html>`_, this is where you + actually use the property replacer. + +.. toctree:: + :maxdepth: 2 + + nomatch diff --git a/source/configuration/rsyslog-example.conf b/source/configuration/rsyslog-example.conf new file mode 100644 index 0000000..a3ec2f1 --- /dev/null +++ b/source/configuration/rsyslog-example.conf @@ -0,0 +1,163 @@ +# A commented quick reference and sample configuration +# WARNING: This is not a manual, the full manual of rsyslog configuration is in +# rsyslog.conf (5) manpage +# +# "$" starts lines that contain new directives. The full list of directives +# can be found in /usr/share/doc/rsyslog-1.19.6/doc/rsyslog_conf.html or online +# at http://www.rsyslog.com/doc if you do not have (or find) a local copy. +# +# Set syslogd options + +# Some global directives +# ---------------------- + +# $AllowedSender - specifies which remote systems are allowed to send syslog messages to rsyslogd +# -------------- +$AllowedSender UDP, 127.0.0.1, 192.0.2.0/24, [::1]/128, *.example.net, somehost.example.com + +# $UMASK - specifies the rsyslogd processes' umask +# ------ +$umask 0000 + +# $FileGroup - Set the group for dynaFiles newly created +# ---------- +$FileGroup loggroup + +# $FileOwner - Set the file owner for dynaFiles newly created. +# ---------- +$FileOwner loguser + +# $IncludeConfig - include other files into the main configuration file +# -------------- +$IncludeConfig /etc/some-included-file.conf # one file +$IncludeConfig /etc/rsyslog.d/ # whole directory (must contain the final slash) + +# $ModLoad - Dynamically loads a plug-in and activates it +# -------- +$ModLoad ommysql # load MySQL functionality +$ModLoad /rsyslog/modules/somemodule.so # load a module via absolute path + + + +# Templates +# --------- + +# Templates allow to specify any format a user might want. +# They MUST be defined BEFORE they are used. + +# A template consists of a template directive, a name, the actual template text +# and optional options. A sample is: +# +$template MyTemplateName,"\7Text %property% some more text\n", + +# where: +# * $template - tells rsyslog that this line contains a template. +# * MyTemplateName - template name. All other config lines refer to this name. +# * "\7Text %property% some more text\n" - templage text + +# The backslash is an escape character, i.e. \7 rings the bell, \n is a new line. +# To escape: +# % = \% +# \ = \\ + +# Template options are case-insensitive. Currently defined are: +# sql format the string suitable for a SQL statement. This will replace single +# quotes ("'") by two single quotes ("''") to prevent the SQL injection +# (NO_BACKSLASH_ESCAPES turned off) +# stdsql - format the string suitable for a SQL statement that is to +# be sent to a standards-compliant sql server. +# (NO_BACKSLASH_ESCAPES turned on) + + + +# Properties inside templates +# --------------------------- + +# Properties can be modified by the property replacer. They are accessed +# inside the template by putting them between percent signs. The full syntax is as follows: + +# %propname:fromChar:toChar:options% + +# FromChar and toChar are used to build substrings. +# If you need to obtain the first 2 characters of the +# message text, you can use this syntax: +"%msg:1:2%". +# If you do not whish to specify from and to, but you want to +# specify options, you still need to include the colons. + +# For example, to convert the full message text to lower case only, use +# "%msg:::lowercase%". + +# The full list of property options can be found in rsyslog.conf(5) manpage + + + +# Samples of template definitions +# ------------------------------- + +# A template that resambles traditional syslogd file output: +$template TraditionalFormat,"%timegenerated% %HOSTNAME% %syslogtag%%msg:::drop-last-lf%\n" + +# A more verbose template: +$template precise,"%syslogpriority%,%syslogfacility%,%timegenerated::fulltime%,%HOSTNAME%,%syslogtag%,%msg%\n" + +# A template that resembles RFC 3164 on-the-wire format: +# (yes, there is NO space betwen syslogtag and msg! that's important!) +$template RFC3164fmt,"<%PRI%>%TIMESTAMP% %HOSTNAME% %syslogtag%%msg%" + +# a template resembling traditional wallmessage format: +$template wallmsg,"\r\n\7Message from syslogd@%HOSTNAME% at %timegenerated% ...\r\n %syslogtag%%msg%\n\r" + +# The template below emulates winsyslog format, but we need to check the time +# stamps used. It is also a good sampleof the property replacer in action. +$template WinSyslogFmt,"%HOSTNAME%,%timegenerated:1:10:date-rfc3339%,%timegenerated:12:19:date-rfc3339%,%timegenerated:1:10:date-rfc3339%,%timegenerated:12:19:date-rfc3339%,%syslogfacility%,%syslogpriority%,%syslogtag%%msg%\n" + +# A template used for database writing (notice it *is* an actual +# sql-statement): +$template dbFormat,"insert into SystemEvents (Message, Facility,FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%',%syslogpriority%, '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",sql + + + +# Samples of rules +# ---------------- +# Regular file +# ------------ +*.* /var/log/traditionalfile.log;TraditionalFormat # log to a file in the traditional format + +# Forwarding to remote machine +# ---------------------------- +*.* @172.19.2.16 # udp (standard for syslog) +*.* @@172.19.2.17 # tcp + +# Database action +# --------------- +# (you must have rsyslog-mysql package installed) +# !!! Don't forget to set permission of rsyslog.conf to 600 !!! +*.* >hostname,dbname,userid,password # (default Monitorware schema, can be created by /usr/share/doc/rsyslog-mysql-1.19.6/createDB.sql) + +# And this one uses the template defined above: +*.* >hostname,dbname,userid,password;dbFormat + +# Program to execute +# ------------------ +*.* ^alsaunmute # set default volume to soundcard + +# Filter using regex +# ------------------ +# if the user logges word rulez or rulezz or rulezzz or..., then we will shut down his pc +# (note, that + have to be double backslashed...) +:msg, regex, "rulez\\+" ^poweroff + +# A more complex example +# ---------------------- +$template bla_logged,"%timegenerated% the BLA was logged" +:msg, contains, "bla" ^logger;bla_logged + +# Pipes +# ----- +# first we need to create pipe by # mkfifo /a_big_pipe +*.* |/a_big_pipe + +# Discarding +# ---------- +*.* ~ # discards everything diff --git a/source/configuration/rsyslog_statistic_counter.rst b/source/configuration/rsyslog_statistic_counter.rst new file mode 100644 index 0000000..231b32c --- /dev/null +++ b/source/configuration/rsyslog_statistic_counter.rst @@ -0,0 +1,83 @@ +rsyslog statistic counter +========================= + +Rsyslog supports statistic counters via the :doc:`impstats <modules/impstats>` module. +It is important to know that impstats and friends only provides an infrastructure +where core components and plugins can register statistics counter. This FAQ entry +tries to describe all counters available, but please keep in mind that there may exist +that we do not know about. + +When interpreting rsyslog statistics, please keep in mind that statistics records are +processed as regular syslog messages. As such, the statistics messages themselves +increment counters when they are emitted via the regular syslog stream, which is the +default (and so counters keep slowly increasing even if there is absolutely no other +traffic). Also keep in mind that a busy rsyslog system is very dynamic. Most +importantly, this means that the counters may not be 100% consistent, but some slight +differences may exist. Avoiding such inconsistencies would be possible only at the +price of a very tight locking discipline, which would cause serious performance +bottlenecks. Thus, this is not done. Finally, though extremely unlikely, some counters +may experience an overflow and restart at 0 for that reasons. However, most counters +are 64-bit, so this is extremely unlikely. Those which are not 64 bit are typically +taken from some internal data structure that uses lower bits for performance reasons +and guards against overflow. + +The listing starts with the core component or plugin that creates the counters and +than specifies various counters that exist for the sub-entities. The listing below is +extended as new counters are added. Some counters probably do not exist in older +releases of rsyslog. + +Queue +----- + +For each queue inside the system its own set of statistics counters is created. +If there are multiple action (or main) queues, this can become a rather lengthy list. +The stats record begins with the queue name (e.g. "main Q" for the main queue; +ruleset queues have the name of the ruleset they are associated to, action queues +the name of the action). + +- **size** - currently active messages in queue + +- **enqueued** - total number of messages enqueued into this queue since startup + +- **maxsize** - maximum number of active messages the queue ever held + +- **full** - number of times the queue was actually full and could not accept additional messages + +- **discarded.full** - number of messages discarded because the queue was full + +- **discarded.nf** - number of messages discarded because the queue was nearly full. Starting at this point, messages of lower-than-configured severity are discarded to save space for higher severity ones. + +Actions +------- + +- **processed** - total number of messages processed by this action. This includes those messages that failed actual execution (so it is a total count of messages ever seen, but not necessarily successfully processed) + +- **failed** - total number of messages that failed during processing. These are actually lost if they have not been processed by some other action. Most importantly in a failover chain the messages are flagged as "failed" in the failing actions even though they are forwarded to the failover action (the failover action’s "processed" count should equal to failing actions "fail" count in this scenario)a + +- **suspended** - (7.5.8+) – total number of times this action suspended itself. Note that this counts the number of times the action transitioned from active to suspended state. The counter is no indication of how long the action was suspended or how often it was retried. This is intentional, as the counter as it currently is permits to tell how often the action ran into a failure condition. + +- **suspended.duration** - (7.5.8+) – the total number of seconds this action was disabled. Note that the second count is incremented as soon as the action is suspended for another interval. As such, the time may be higher than it should be at the reporting point of time. If the pstats interval is shorter than the suspension timeout, the same suspended.duration value may be reported for successive pstats outputs. For a long-running system, this is considered a minimal difference. In general, note that this setting is not totally accurate, especially when running with multiple worker threads. In rsyslog v8, this is the total suspended time for all worker instances of this action. + +- **resumed** - (7.5.8+) – total number of times this action resumed itself. A resumption occurs after the action has detected that a failure condition does no longer exist. + +Plugins +------- + +:doc:`imuxsock <modules/imuxsock>` + +:doc:`imudp <modules/imudp>` + +:doc:`imtcp <modules/imtcp>` + +:doc:`imptcp <modules/imptcp>` + +:doc:`imrelp <modules/imrelp>` + +:doc:`impstats <modules/impstats>` + +:doc:`omfile <modules/omfile>` + +:doc:`omelasticsearch <modules/omelasticsearch>` + +:doc:`omkafka <modules/omkafka>` + diff --git a/source/configuration/ruleset/index.rst b/source/configuration/ruleset/index.rst new file mode 100644 index 0000000..429e1a2 --- /dev/null +++ b/source/configuration/ruleset/index.rst @@ -0,0 +1,20 @@ +Ruleset-Specific Legacy Configuration Statements +================================================ +These statements can be used to set ruleset parameters. To set +these parameters, first use *$Ruleset*, **then** use the other +configuration directives. Please keep in mind +that each ruleset has a *main* queue. To specify parameter for these +ruleset (main) queues, use the main queue configuration directives. + +.. toctree:: + :glob: + + *rule* + +- **$Ruleset** *name* - starts a new ruleset or switches back to one + already defined. All following actions belong to that new rule set. + the *name* does not yet exist, it is created. To switch back to + rsyslog's default ruleset, specify "RSYSLOG\_DefaultRuleset") as the + name. All following actions belong to that new rule set. It is + advised to also read our paper on + :doc:`using multiple rule sets in rsyslog <../../concepts/multi_ruleset>`. diff --git a/source/configuration/ruleset/rsconf1_rulesetcreatemainqueue.rst b/source/configuration/ruleset/rsconf1_rulesetcreatemainqueue.rst new file mode 100644 index 0000000..16f73a6 --- /dev/null +++ b/source/configuration/ruleset/rsconf1_rulesetcreatemainqueue.rst @@ -0,0 +1,91 @@ +`rsyslog.conf configuration directive <rsyslog_conf_global.html>`_ + +$RulesetCreateMainQueue +----------------------- + +**Type:** ruleset-specific configuration directive + +**Parameter Values:** boolean (on/off, yes/no) + +**Available since:** 5.3.5+ + +**Default:** off + +**Description:** + +Rulesets may use their own "main" message queue for message submission. +Specifying this directive, **inside a ruleset definition**, turns this +on. This is both a performance enhancement and also permits different +rulesets (and thus different inputs within the same rsyslogd instance) +to use different types of main message queues. + +The ruleset queue is created with the parameters that are specified for +the main message queue at the time the directive is given. If different +queue configurations are desired, different main message queue +directives must be used **in front of** the $RulesetCreateMainQueue +directive. Note that this directive may only be given once per ruleset. +If multiple statements are specified, only the first is used and for the +others error messages are emitted. + +Note that the final set of ruleset configuration directives specifies +the parameters for the default main message queue. + +To learn more about this feature, please be sure to read about +`multi-ruleset support in rsyslog <multi_ruleset.html>`_. + +**Caveats:** + +The configuration statement "$RulesetCreateMainQueue off" has no effect +at all. The capability to specify this is an artifact of the legacy +configuration language. + +**Example:** + +This example sets up a tcp server with three listeners. Each of these +three listener is bound to a specific ruleset. As a performance +optimization, the rulesets all receive their own private queue. The +result is that received messages can be independently processed. With +only a single main message queue, we would have some lock contention +between the messages. This does not happen here. Note that in this +example, we use different processing. Of course, all messages could also +have been processed in the same way ($IncludeConfig may be useful in +that case!). + +:: + + $ModLoad imtcp + # at first, this is a copy of the unmodified rsyslog.conf + #define rulesets first + $RuleSet remote10514 + $RulesetCreateMainQueue on # create ruleset-specific queue + *.* /var/log/remote10514 + + $RuleSet remote10515 + $RulesetCreateMainQueue on # create ruleset-specific queue + *.* /var/log/remote10515 + + $RuleSet remote10516 + $RulesetCreateMainQueue on # create ruleset-specific queue + mail.* /var/log/mail10516 + & ~ + # note that the discard-action will prevent this messag from + # being written to the remote10516 file - as usual... + *.* /var/log/remote10516 + + # and now define listeners bound to the relevant ruleset + $InputTCPServerBindRuleset remote10514 + $InputTCPServerRun 10514 + + $InputTCPServerBindRuleset remote10515 + $InputTCPServerRun 10515 + + $InputTCPServerBindRuleset remote10516 + $InputTCPServerRun 10516 + + +Note the positions of the directives. With the legacy language, +position is very important. It is highly suggested to use +the *ruleset()* object in RainerScript config language if you intend +to use ruleset queues. The configuration is much more straightforward in +that language and less error-prone. + diff --git a/source/configuration/ruleset/rsconf1_rulesetparser.rst b/source/configuration/ruleset/rsconf1_rulesetparser.rst new file mode 100644 index 0000000..0fb9925 --- /dev/null +++ b/source/configuration/ruleset/rsconf1_rulesetparser.rst @@ -0,0 +1,127 @@ +$RulesetParser +-------------- + +**Type:** ruleset-specific configuration directive + +**Parameter Values:** string + +**Available since:** 5.3.4+ + +**Default:** rsyslog.rfc5424 followed by rsyslog.rfc3164 + +**Description:** + +This directive permits to specify which `message +parsers <../../concepts/messageparser.html>`_ should be used for the ruleset in +question. It no ruleset is explicitly specified, the default ruleset is +used. Message parsers are contained in (loadable) parser modules with +the most common cases (RFC3164 and RFC5424) being build-in into +rsyslogd. + +When this directive is specified the first time for a ruleset, it will +not only add the parser to the ruleset's parser chain, it will also wipe +out the default parser chain. So if you need to have them in addition to +the custom parser, you need to specify those as well. + +Order of directives is important. Parsers are tried one after another, +in the order they are specified inside the config. As soon as a parser +is able to parse the message, it will do so and no other parsers will be +executed. If no matching parser can be found, the message will be +discarded and a warning message be issued (but only for the first 1,000 +instances of this problem, to prevent message generation loops). + +Note that the rfc3164 parser will **always** be able to parse a message +- it may just not be the format that you like. This has two important +implications: 1) always place that parser at the END of the parser list, +or the other parsers after it will never be tried and 2) if you would +like to make sure no message is lost, placing the rfc3164 parser at the +end of the parser list ensures that. + +Multiple parser modules are very useful if you have various devices that +emit messages that are malformed in various ways. The route to take then +is + +- make sure you find a custom parser for that device; if there is no + one, you may consider writing one yourself (it is not that hard) or + getting one written as part of `Adiscon's professional services for + rsyslog <http://www.rsyslog.com/professional-services>`_. +- load your custom parsers via $ModLoad +- create a ruleset for each malformed format; assign the custom parser + to it +- create a specific listening port for all devices that emit the same + malformed format +- bind the listener to the ruleset with the required parser + +Note that it may be cumbersome to add all rules to all rulesets. To +avoid this, you can either use $Include or `omruleset <omruleset.html>`_ +(what probably provides the best solution). + +More information about rulesets in general can be found in +:doc:`multi-ruleset support in rsyslog <../../concepts/multi_ruleset>`. + +**Caveats:** + +currently none known + +**Example:** + +This example assumes there are two devices emitting malformed messages +via UDP. We have two custom parsers for them, named "device1.parser" and +"device2.parser". In addition to that, we have a number of other devices +sending well-formed messages, also via UDP. + +The solution is to listen for data from the two devices on two special +ports (10514 and 10515 in this example), create a ruleset for each and +assign the custom parsers to them. The rest of the messages are received +via port 514 using the regular parsers. Processing shall be equal for +all messages. So we simply forward the malformed messages to the regular +queue once they are parsed (keep in mind that a message is never again +parsed once any parser properly processed it). + +:: + + $ModLoad imudp + $ModLoad pmdevice1 # load parser "device1.parser" for device 1 + $ModLoad pmdevice2 # load parser "device2.parser" for device 2 + + # define ruleset for the first device sending malformed data + $Ruleset maldev1 + $RulesetCreateMainQueue on # create ruleset-specific queue + $RulesetParser "device1.parser" # note: this deactivates the default parsers + # forward all messages to default ruleset: + $ActionOmrulesetRulesetName RSYSLOG\_DefaultRuleset + \*.\* :omruleset: + + # define ruleset for the second device sending malformed data + $Ruleset maldev2 $RulesetCreateMainQueue on # create ruleset-specific queue + $RulesetParser "device2.parser" # note: this deactivates the default parsers + # forward all messages to default ruleset: + $ActionOmrulesetRulesetName RSYSLOG\_DefaultRuleset + \*.\* :omruleset: + + # switch back to default ruleset + $Ruleset RSYSLOG\_DefaultRuleset + \*.\* /path/to/file + auth.info @authlogger.example.net + # whatever else you usually do... + + # now define the inputs and bind them to the rulesets + # first the default listener (utilizing the default ruleset) + $UDPServerRun 514 + + # now the one with the parser for device type 1: + $InputUDPServerBindRuleset maldev1 + $UDPServerRun 10514 + + # and finally the one for device type 2: + $InputUDPServerBindRuleset maldev2 + $UDPServerRun 10515 + +For an example of how multiple parser can be chained (and an actual use +case), please see the example section on the +`pmlastmsg <pmlastmsg.html>`_ parser module. + +Note the positions of the directives. With the current config language, +**sequence of statements is very important**. This is ugly, but +unfortunately the way it currently works. + diff --git a/source/configuration/sysklogd_format.rst b/source/configuration/sysklogd_format.rst new file mode 100644 index 0000000..3fd9b8a --- /dev/null +++ b/source/configuration/sysklogd_format.rst @@ -0,0 +1,311 @@ + +*************** +sysklogd format +*************** + +This is the format in use since the beginning of syslogging. It still +is an excellent choice to do very simple things. + +For more advanced things, use the |FmtAdvancedName| format. + +DESCRIPTION +=========== + +The syslog.conf file is the main configuration file for :manpage:`syslogd(8)` +which logs system messages on \*nix systems. This file specifies rules for +logging. For special features see the sysklogd(8) manpage. + +Every rule consists of two fields, a selector field and an action field. +These two fields are separated by one or more spaces or tabs. The selector +field specifies a pattern of facilities and priorities belonging to the +specified action. + +Lines starting with a hash mark ("#") and empty lines are ignored. + +This variant of syslogd is able to understand a slightly extended syntax +compared to the original BSD syslogd. One rule may be divided into several +lines if the leading line is terminated with an backslash ("\\"). + +SELECTORS +========= + +The selector field consists of two parts, a facility and a priority, separated +by a period ("."). Both parts are case insensitive and can also be specified +as decimal numbers corresponding to the definitions in +``/usr/include/syslog.h``. It is safer to use symbolic names rather than +decimal numbers. Both facilities and priorities are described in +:manpage:`syslog(3)`. The names mentioned below +correspond to the similar ``LOG_`` values in ``/usr/include/syslog.h``. + +The facility is one of the following keywords: auth, authpriv, cron, daemon, +ftp, kern, lpr, mail, mark, news, security (same as auth), syslog, user, uucp +and local0 through local7. The keyword security is deprecated and mark is only +for internal use and therefore should not be used in applications. The facility +specifies the subsystem that produced the message, e.g. all mail programs log +with the mail facility (LOG_MAIL) if they log using syslog. + +In most cases anyone can log to any facility, so we rely on convention for the +correct facility to be chosen. However, generally only the kernel can log to +the "kern" facility. This is because the implementation of ``openlog()`` and +``syslog()`` in glibc does not allow logging to the "kern" facility. Klogd +circumvents this restriction when logging to syslogd by reimplementing those +functions itself. + +The priority is one of the following keywords, in ascending order: debug, +info, notice, warning, warn (same as warning), err, error (same as err), crit, +alert, emerg, panic (same as emerg). The keywords warn, error and panic are +deprecated and should not be used anymore. The priority defines the severity of +the message. + +The behavior of the original BSD syslogd is that all messages of the specified +priority and higher are logged according to the given action. This +:manpage:`syslogd(8)` behaves the same, but has some extensions. + +In addition to the above mentioned names the :manpage:`syslogd(8)` understands +the following extensions: +An asterisk ("\*") stands for all facilities or all priorities, depending on +where it is used (before or after the period). The keyword none stands for no +priority of the given facility. + +Multiple facilities may be specified for a single priority pattern in one +statement using the comma (",") operator to separate the facilities. You may +specify as many facilities as you want. Please note that only the facility +part from such a statement is taken, a priority part would be ignored. + +Multiple selectors may be specified for a single action using the semicolon +(";") separator. Selectors are processed from left to right, with each selector +being able to overwrite preceding ones. Using this behavior you are able to +exclude some priorities from the pattern. + +This :manpage:`syslogd(8)` has a syntax extension to the original BSD source, +which makes its use more intuitive. You may precede every priority with an +equation sign ("=") to specify that syslogd should only refer to this single +priority and not this priority and all higher priorities. + +You may also precede the priority with an exclamation mark ("!") if you want +syslogd to ignore this priority and all higher priorities. You may even use +both, the exclamation mark and the equation sign if you want syslogd to ignore +only this single priority. If you use both extensions then the exclamation +mark must occur before the equation sign, just use it intuitively. + +ACTIONS +======= + +The action field of a rule describes the abstract term "logfile". A "logfile" +need not to be a real file, btw. The :manpage:`syslogd(8)` provides the +following actions. + +Regular File +------------ + +Typically messages are logged to real files. The filename is specified with an +absolute pathname. It may be specified as a file name relative to rsyslog's +working directory if the filename starts with "." or "..". However, this is +dangerous and should be avoided. + +Named Pipes +----------- + +This version of :manpage:`syslogd(8)` has support for logging output to named +pipes (fifos). A fifo or named pipe can be used as a destination for log +messages by prepending a pipe symbol ("|") to the name of the file. This is +handy for debugging. Note that the fifo must be created with the +:manpage:`mkfifo(1)` command before :manpage:`syslogd(8)` is started. + +Terminal and Console +-------------------- + +If the file you specified is a tty, special tty-handling is done, same with +``/dev/console``. + +Remote Machine +-------------- + +This :manpage:`syslogd(8)` provides full remote logging, i.e. is able to send +messages to a remote host running :manpage:`syslogd(8)` and to receive messages +from remote hosts. The remote host won't forward the message again, it will +just log them locally. To forward messages to another host, prepend the +hostname with the at sign ("@"). + +Using this feature you are able to collect all syslog messages on a central +host, if all other machines log remotely to that one. This reduces +administration needs. + +Using a named pipe log method, messages from remote hosts can be sent to a +log program. By reading log messages line by line such a program is able to +sort log messages by host name or program name on the central log host. This +way it is possible to split the log into separate files. + +List of Users +------------- + +Usually critical messages are also directed to "root" on that machine. You can +specify a list of users that ought to receive the log message on the terminal +by writing their usernames. You may specify more than one user by separating +the usernames with commas (","). If they're logged in they will receive the +log messages. + +Everyone logged on +------------------ + +Emergency messages often go to all users currently online to notify them that +something strange is happening with the system. To specify this wall(1)-feature +use an asterisk ("*"). + +EXAMPLES +======== + +Here are some examples, partially taken from a real existing site and +configuration. Hopefully they answer all questions about configuring this +:manpage:`syslogd(8)`. If not, don't hesitate to contact the mailing list. + +:: + + # Store critical stuff in critical + # + *.=crit;kern.none /var/adm/critical + +This will store all messages of priority crit in the file +``/var/adm/critical``, with the exception of any kernel messages. + +:: + + # Kernel messages are stored in the kernel file, + # critical messages and higher ones also go + # to another host and to the console + # + kern.* /var/adm/kernel + kern.crit @finlandia + kern.crit /dev/console + kern.info;kern.!err /var/adm/kernel-info + +The first rule directs any message that has the kernel facility to the file +``/var/adm/kernel``. (But recall that only the kernel itself can log to this +facility.) + +The second statement directs all kernel messages of priority crit and higher +to the remote host finlandia. This is useful, because if the host crashes +and the disks get irreparable errors you might not be able to read the stored +messages. If they're on a remote host, too, you still can try to find out the +reason for the crash. + +The third rule directs kernel messages of priority crit and higher to the +actual console, so the person who works on the machine will get them, too. + +The fourth line tells the syslogd to save all kernel messages that come with +priorities from info up to warning in the file ``/var/adm/kernel-info``. + +This is an example of the 2nd selector overwriting part of the first one. +The first selector selects kernel messages of priority info and higher. The +second selector filters out kernel messages of priority error and higher. +This leaves just priorities info, notice and warning to get logged. + +:: + + # The tcp wrapper logs with mail.info, we display + # all the connections on tty12 + # + mail.=info /dev/tty12 + +This directs all messages that use ``mail.info`` +(in source ``LOG_MAIL | LOG_INFO``) to ``/dev/tty12``, the 12th console. +For example the tcpwrapper :manpage:`tcpd(8)` uses this as its default. + +:: + + # Write all mail related logs to a file + # + mail.*;mail.!=info /var/adm/mail + +This pattern matches all messages that come with the mail facility, +except for the info priority. These will be stored in the file +``/var/adm/mail``. + +:: + + # Log all mail.info and news.info messages to info + # + mail,news.=info /var/adm/info + +This will extract all messages that come either with mail.info or with +news.info and store them in the file ``/var/adm/info``. + +:: + + # Log info and notice messages to messages file + # + *.=info;*.=notice;\ + mail.none /var/log/messages + +This lets the syslogd log all messages that come with either the info or the +notice priority into the file ``/var/log/messages``, except for all messages +that use the mail facility. + +:: + + # Log info messages to messages file + # + *.=info;\ + mail,news.none /var/log/messages + +This statement causes the syslogd to log all messages that come with the info +priority to the file ``/var/log/messages``. But any message coming either with +the mail or the news facility will not be stored. + +:: + + # Emergency messages will be displayed using wall + # + *.=emerg * + +This rule tells the syslogd to write all emergency messages to all currently +logged in users. This is the wall action. + +:: + + # Messages of the priority alert will be directed + # to the operator + # + *.alert root,joey + +This rule directs all messages of priority alert or higher to the terminals +of the operator, i.e. of the users "root" and "joey" if they're logged in. + +:: + + *.* @finlandia + +This rule would redirect all messages to a remote host called finlandia. +This is useful especially in a cluster of machines where all syslog messages +will be stored on only one machine. + +CONFIGURATION FILE SYNTAX DIFFERENCES +===================================== + +Syslogd uses a slightly different syntax for its configuration file than the +original BSD sources. Originally all messages of a specific priority and above +were forwarded to the log file. The modifiers "=", "!" and "-" were added to +make the syslogd more flexible and to use it in a more intuitive manner. + +The original BSD syslogd doesn't understand spaces as separators between the +selector and the action field. + +BUGS +==== + +The effects of multiple selectors are sometimes not intuitive. For example +"mail.crit,\*.err" will select "mail" facility messages at the level of +"err" or higher, not at the level of "crit" or higher. + +Also, if you specify a selector with an exclamation mark in it which is not +preceded by a corresponding selector without an exclamation mark, nothing +will be logged. Intuitively, the selector "ftp.!alert" on its own will select +all ftp messages with priorities less than alert. In fact it selects nothing. +Similarly "ftp.!=alert" might reasonably be expected to select all ftp messages +other than those with priority alert, but again it selects nothing. It seems +the selectors with exclamation marks in them should only be used as "filters" +following selectors without exclamation marks. + +Finally, using a backslash to divide a line into two doesn't work if the +backslash is used immediately after the end of the selector, without +intermediate whitespace. diff --git a/source/configuration/templates.rst b/source/configuration/templates.rst new file mode 100644 index 0000000..e602241 --- /dev/null +++ b/source/configuration/templates.rst @@ -0,0 +1,908 @@ +********* +Templates +********* + +Description +=========== + +Templates are a key feature of rsyslog. They allow to specify any format +a user might want. They are also used for dynamic file name generation. +Every output in rsyslog uses templates - this holds true for files, user +messages and so on. The database writer expects its template to be a +proper SQL statement - so this is highly customizable too. You might ask +how does all of this work when no templates at all are specified. Good +question ;). The answer is simple, though. Templates compatible with the +stock syslogd formats are hardcoded into rsyslogd. So if no template is +specified, we use one of those hardcoded templates. Search for +"template\_" in rsconf.c and you will find the hardcoded ones. + +Templates are specified by template() objects. They can also be +specified via $template legacy statements. + +.. note:: + + Note: key elements of templates are rsyslog properties. See the + :doc:`rsyslog properties reference <properties>` for a list of which + are available. + + +Template processing +=================== + +Due to lack of standardization regarding logs formats, when a template is +specified it's supposed to include HEADER, as defined in `RFC5424 <https://tools.ietf.org/html/rfc5424>`_ + +It's very important to have this in mind, and also to understand how +`rsyslog parsing <http://www.rsyslog.com/doc/syslog_parsing.html>`_ works. + +For example, if the MSG field is set to "this:is a message" and neither HOSTNAME +nor TAG are specified, the outgoing parser will split the message as: + +.. code-block:: none + + TAG:this: + MSG:is a message + + +The template() object +===================== + +The template() object is used to define templates. Note that it is a +**static** object, that means all templates are defined when rsyslog +reads the config file. As such, templates are not affected by +if-statements or config nesting. + +The basic structure of the template object is as follows: + +.. code-block:: none + + template(parameters) + +In addition to this simpler syntax, list templates (to be described +below) support an extended syntax: + +.. code-block:: none + + template(parameters) { list-descriptions } + +Each template has a parameter **name**, which specifies the template +name, and a parameter **type**, which specifies the template type. The +name parameter must be unique, and behaviour is unpredictable if it is +not. The **type** parameter specifies different template types. +Different types simply enable different ways to specify the template +content. The template type **does not** affect what an (output) plugin +can do with it. So use the type that best fits your needs (from a config +writing point of view!). The following types are available: + +- list +- subtree +- string +- plugin + +The various types are described below. + +List +---- + +In this case, the template is generated by a list of constant and +variable statements. These follow the template spec in curly braces. +This type is also primarily meant to be used with structure-aware outputs, +like ommongodb. However, it also works perfectly with text-based +outputs. We recommend to use this mode if more complex property +substitutions need to be done. In that case, the list-based template +syntax is much clearer than the simple string-based one. + +The list template contains the template header (with **type="list"**) +and is followed by **constant** and **property** statements, given in +curly braces to signify the template statement they belong to. As the +name says, **constant** statements describe constant text and +**property** describes property access. There are many options to +**property**, described further below. Most of these options are used to +extract only partial property contents or to modify the text obtained +(for instance to change its case to upper or lower case). + +To grasp the idea, an actual sample is: + +.. code-block:: none + + template(name="tpl1" type="list") { + constant(value="Syslog MSG is: '") + property(name="msg") + constant(value="', ") + property(name="timereported" dateFormat="rfc3339" caseConversion="lower") + constant(value="\n") + } + +This sample is probably primarily targeted at the usual file-based +output. + +Constant statement +^^^^^^^^^^^^^^^^^^ + +This provides a way to specify constant text. The text is used +literally. It is primarily intended for text-based output, so that some +constant text can be included. For example, if a complex template is +built for file output, one usually needs to finish it by a newline, +which can be introduced by a constant statement. Here is an actual +sample of that use case from the rsyslog testbench: + +:: + + template(name="outfmt" type="list") { + property(name="$!usr!msgnum") + constant(value="\n") + } + +The following escape sequences are recognized inside the constant text: + +- \\\\ - single backslash +- \\n - LF +- \\ooo - (three octal digits) - represents a character with this + octal numerical value (e.g. \\101 equals "A"). Note that three octal digits + must be given (in contrast to some languages, where between one and + three are valid). While we support octal notation, we recommend to + use hex notation as this is better known. +- \\xhh - (where h is a hex digit) - represents a character with this + hexadecimal numerical value (e.g. \\x41 equals "A"). Note that two hexadecimal + digits must be given (in contrast to some languages where either one or two + are valid). +- ... some others ... list needs to be extended + +Note: if an unsupported character follows a backslash, this is treated +as an error. Behaviour is unpredictable in this case. + +To aid usage of the same template both for text-based outputs and +structured ones, constant text without an "outname" parameter will be +ignored when creating the name/value tree for structured outputs. So if +you want to supply some constant text e.g. to mongodb, you must include +an outname, as can be seen here: + +.. code-block:: none + + template(name="outfmt" type="list") { + property(name="$!usr!msgnum") + constant(value="\n" outname="IWantThisInMyDB") + } + +To generate a constant json field, the `format` parameter can be used, as +in this example + +.. code-block:: none + + template(name="outfmt" type="list" option.jsonf="on") { + property(outname="message" name="msg" format="jsonf") + constant(outname="@version" value="1" format="jsonf") + } + +The constant statement in this case will generate `"@version":"1"`. Note that to do +this, both the `value` and the `format` parameters must be given. + +The "constant" statement supports the following parameters: + +- value - the constant value to use +- outname - the output field name (for structured outputs) +- format - can be either empty or `jsonf` + + +Property statement +^^^^^^^^^^^^^^^^^^ + +This statement is used to include property values. It can access all +properties. Also, options permit to specify picking only part of a +property or modifying it. It supports the following parameters: + +- **name** - the name of the property to access + +- **outname** - the output field name (for structured outputs) + +- **dateformat** - the date format to use (only for date-related properties). + `Here <property_replacer.html#property-options>`_ you can find a list of all + property options. **TODO:** right now, the property replacer documentation + contains property format options for string templates, only. The formats for + non-string templates differ. For example, date format options in string + templates start with "date-" whereas those in property statements do not + (e.g. "date-year" vs. just "year"). The technical reason behind this is + that inside string templates, the option must include what it applies + to whereas with the explicit format that is part of the parameter name. + + To create a customised format you can use multiple property options + together. The following example would result in **YYYY-MM-DD**: + +.. code-block:: none + + property(name="timereported" dateformat="year") + constant(value="-") + property(name="timereported" dateformat="month") + constant(value="-") + property(name="timereported" dateformat="day") + +- **date.inUTC** - date shall be shown in UTC (please note that this + requires a bit more performance due to the necessary conversions) + Available since 8.18.0. + +- **caseconversion** - permits to convert case of the text. Supported + values are "lower" and "upper" + +- **controlcharacters** - specifies how to handle control characters. + Supported values are "escape", which escapes them, "space", which + replaces them by a single space, and "drop", which simply removes + them from the string. + +- **securepath** - used for creating pathnames suitable for use in dynafile + templates. Supported values are "drop" and "replace". + +- **format** - specify format on a field basis. Supported values are: + + - "`csv <property_replacer.html#csv>`_\ " for use when csv-data is + generated + - "`json <property_replacer.html#json>`_\ " which formats proper + json content (but without a field header) + - "`jsonf <property_replacer.html#jsonf>`_\ " which formats as a + complete json field + - "`jsonr <property_replacer.html#jsonr>`_\ " which avoids double + escaping the value but makes it safe for a json field + - "`jsonfr <property_replacer.html#jsonfr>`_\ " which is the + combination of "jsonf" and "jsonr". + +- **position.from** - obtain substring starting from this position (1 is + the first position) + +- **position.to** - obtain substring up to this position. As a special + extension available since 8.2302.0. the `position.to` value may be + negative (-n). In this case, all characters from `position.from` to the + end of the actual property string except the last n characters are extracted. + This makes it easy to strip the beginning and end of a string with changing + legth. Let us assume you have a string `"[abc]"` and want to remove the + braces. In this case, use `position.from="2" position.to="-1"` in the + `property()` object. This will result in the string `"abc"` after template + processing. This is especially useful if you want to drop the braces around + the `STRUCTURED-DATA` message property. + +- **position.relativeToEnd** - the from and to position is relative to the + end of the string instead of the usual start of string. (available + since rsyslog v7.3.10) + +- **fixedwidth** - changes behaviour of position.to so that it pads the + source string with spaces up to the value of position.to if the source + string is shorter. "on" or "off" (default) (available since rsyslog + v8.13.0) + +- **compressspace** - compresses multiple spaces (US-ASCII SP character) + inside the string to a single one. This compression happens at a very + late stage in processing. Most importantly, it happens after substring + extraction, so the **position.from** and **position.to** positions + are **NOT** affected by this option. (available since v8.18.0). + +- **field.number** - obtain this field match + +- **field.delimiter** - decimal value of delimiter character for field + extraction + +- **regex.expression** - expression to use + +- **regex.type** - either ERE or BRE + +- **regex.nomatchmode** - what to do if we have no match + +- **regex.match** - match to use + +- **regex.submatch** - submatch to use + +- **droplastlf** - drop a trailing LF, if it is present + +- **mandatory** - signifies a field as mandatory. If set to "on", this + field will always be present in data passed to structured outputs, + even if it is empty. If "off" (the default) empty fields will not be + passed to structured outputs. This is especially useful for outputs + that support dynamic schemas (like ommongodb). + +- **spifno1stsp** - expert options for RFC3164 template processing + +- **datatype** - for "jsonf" format ONLY; permits to set a datatype + Log messages as string data types natively. Thus every property inside + rsyslog is string based. However, in some end systems you need different + data types like numbers of boolean. This setting, in jsonf mode, permits + to configure a desired data type. Supported data types are: + + - number - value is treated as a JSON number and not enclosed in quotes. + If the property is empty, the value 0 is generated. + - string - value is a string and enclosed in quotes + - auto - value is treated as number if numeric and as string otherwise. + The current implementation treats only integers as numeric to avoid + confusion. + - bool - the value is treated as boolean. If it is empty or 0, it will + generate "false", else "true". + + If not specified, 'string' datatype is assumed. + This is a feature of rsyslog 8.1905.0 or later. + +- **onEmpty** - for "jsonf" format ONLY; specifies how empty values + shall be handled. Possible values are: + + - keep - emit the empty element + - skip - completely ignore the element, do not emit anything + - null - emit a JSON 'null' value + + If not specified, 'keep' is assumed. + This is a feature of rsyslog 8.1905.0 or later. + + +Subtree +------- + +Available since rsyslog 7.1.4 + +In this case, the template is generated based on a complete (CEE) +subtree. This type of template is most useful for outputs that know how +to process hierarchical structure, like ommongodb. With that type, the +parameter **subtree** must be specified, which tells which subtree to +use. For example template(name="tpl1" type="subtree" subtree="$!") +includes all CEE data, while template(name="tpl2" type="subtree" +subtree="$!usr!tpl2") includes only the subtree starting at $!usr!tpl2. +The core idea when using this type of template is that the actual data +is prefabricated via set and unset script statements, and the resulting +structure is then used inside the template. This method MUST be used if +a complete subtree needs to be placed *directly* into the object's root. +With all other template types, only subcontainers can be generated. Note +that subtree type can also be used with text-based outputs, like omfile. +HOWEVER, you do not have any capability to specify constant text, and as +such cannot include line breaks. As a consequence, using this template +type for text outputs is usually only useful for debugging or very +special cases (e.g. where the text is interpreted by a JSON parser later +on). + + +Use case +^^^^^^^^ + +A typical use case is to first create a custom subtree and then include +it into the template, like in this small example: + +.. code-block:: none + + set $!usr!tpl2!msg = $msg; + set $!usr!tpl2!dataflow = field($msg, 58, 2); + template(name="tpl2" type="subtree" subtree="$!usr!tpl2") + + +Here, we assume that $msg contains various fields, and the data from a +field is to be extracted and stored - together with the message - as +field content. + + +String +------ + +This closely resembles the legacy template statement. It has a mandatory +parameter **string**, which holds the template string to be applied. A +template string is a mix of constant text and replacement variables (see +property replacer). These variables are taken from message or other +dynamic content when the final string to be passed to a plugin is +generated. String-based templates are a great way to specify textual +content, especially if no complex manipulation to properties is +necessary. + +This is a sample for a string-based template: + +.. code-block:: none + + template(name="tpl3" type="string" + string="%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n" + ) + + +The text between percent signs ('%') is interpreted by the rsyslog +:doc:`property replacer <property_replacer>`. In a nutshell, +it contains the property to use as well as options for formatting +and further processing. This is very similar to what the ``property`` +object in list templates does (it actually is just a different language to +express most of the same things). + +Everything outside of the percent signs is constant text. In the +above case, we have mostly spaces between the property values. At the +end of the string, an escape sequence is used. + +Escape sequences permit to specify nonprintable characters. They work +very similar to escape sequences in C and many other languages. They +are initiated by the backslash characters and followed by one or more +characters that specify the actual character. For example \\7 is the +US-ASCII BEL character and \\n is a newline. The set is similar to +what C and perl support, but a bit more limited. + + +Plugin +------ + +In this case, the template is generated by a plugin (which is then +called a "strgen" or "string generator"). The format is fixed as it is +coded. While this is inflexible, it provides superior performance, and +is often used for that reason (not that "regular" templates are slow - +but in very demanding environments that "last bit" can make a +difference). Refer to the plugin's documentation for further details. +For this type, the parameter **plugin** must be specified and must +contain the name of the plugin as it identifies itself. Note that the +plugin must be loaded prior to being used inside a template. +Config example: + + ``template(name="tpl4" type="plugin" plugin="mystrgen")`` + + +Options +^^^^^^^ + +The <options> part is optional. It carries options influencing the +template as a whole and is a part of the template parameters. See details +below. Be sure NOT to mistake template options with property options - +the latter ones are processed by the property replacer and apply to a +SINGLE property, only (and not the whole template). +Template options are case-insensitive. Currently defined are: + +**option.sql** - format the string suitable for a SQL statement in MySQL +format. This will replace single quotes ("'") and the backslash +character by their backslash-escaped counterpart ("\\'" and "\\\\") +inside each field. Please note that in MySQL configuration, the +``NO_BACKSLASH_ESCAPES`` mode must be turned off for this format to work +(this is the default). + +**option.stdsql** - format the string suitable for a SQL statement that +is to be sent to a standards-compliant sql server. This will replace +single quotes ("'") by two single quotes ("''") inside each field. You +must use stdsql together with MySQL if in MySQL configuration the +``NO_BACKSLASH_ESCAPES`` is turned on. + +**option.json** - format the string suitable for a json statement. This +will replace single quotes ("'") by two single quotes ("''") inside each +field. + +**option.jsonf** - format the string as JSON object. This means a leading +and trailing curly brace "{" will be added as well as a comma between all +non-terminal properties and constants. + +**option.casesensitive** - treat property name references as case +sensitive. The default is "off", where all property name references are +first converted to lowercase during template definition. With this +option turned "on", property names are looked up as defined in the +template. Use this option if you have JSON (``$!*``), local (``!.*``), +or global (``$!\\*``) properties which contain uppercase letters. The +normal Rsyslog properties are case-insensitive, so this option is not +needed for properly referencing those properties. + +Use of the options **option.sql**, **option.stdsql**, and +**option.json** are mutually exclusive. Using more than one at the same +time can cause unpredictable behaviour. + +Either the **sql** or **stdsql** option **must** be specified when a +template is used for writing to a database, otherwise injection might +occur. Please note that due to the unfortunate fact that several vendors +have violated the sql standard and introduced their own escape methods, +it is impossible to have a single option doing all the work. So you +yourself must make sure you are using the right format. **If you choose +the wrong one, you are still vulnerable to sql injection.** +Please note that the database writer *checks* that the sql option is +present in the template. If it is not present, the write database action +is disabled. This is to guard you against accidentally forgetting it and +then becoming vulnerable to SQL injection. The sql option can also be +useful with files - especially if you want to import them into a +database on another machine for performance reasons. However, do NOT use +it if you do not have a real need for it - among others, it takes some +toll on the processing time. Not much, but on a really busy system you +might notice it. + +The default template for the write to database action has the sql option +set. As we currently support only MySQL and the sql option matches the +default MySQL configuration, this is a good choice. However, if you have +turned on ``NO_BACKSLASH_ESCAPES`` in your MySQL config, you need to +supply a template with the stdsql option. Otherwise you will become +vulnerable to SQL injection. + +.. code-block:: none + + template (name="TraditionalFormat" type="string" + string="%timegenerated% %HOSTNAME% %syslogtag%%msg%\\n") + + +Examples +======== + +Standard Template for Writing to Files +-------------------------------------- + +.. code-block:: none + + template(name="FileFormat" type="list") { + property(name="timestamp" dateFormat="rfc3339") + constant(value=" ") + property(name="hostname") + constant(value=" ") + property(name="syslogtag") + property(name="msg" spifno1stsp="on" ) + property(name="msg" droplastlf="on" ) + constant(value="\n") + } + +The equivalent string template looks like this: + +.. code-block:: none + + template(name="FileFormat" type="string" + string= "%TIMESTAMP% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n" + ) + + +.. note:: + + The template string itself must be on a single line. + + +Standard Template for Forwarding to a Remote Host (RFC3164 mode) +---------------------------------------------------------------- + +.. code-block:: none + + template(name="ForwardFormat" type="list") { + constant(value="<") + property(name="pri") + constant(value=">") + property(name="timestamp" dateFormat="rfc3339") + constant(value=" ") + property(name="hostname") + constant(value=" ") + property(name="syslogtag" position.from="1" position.to="32") + property(name="msg" spifno1stsp="on" ) + property(name="msg") + } + +The equivalent string template looks like this: + +.. code-block:: none + + template(name="forwardFormat" type="string" + string="<%PRI%>%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag:1:32%%msg:::sp-if-no-1st-sp%%msg%" + ) + +.. note:: + + The template string itself must be on a single line. + + +Standard Template for writing to the MySQL database +--------------------------------------------------- + +.. code-block:: none + + template(name="StdSQLformat" type="list" option.sql="on") { + constant(value="insert into SystemEvents (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag)") + constant(value=" values ('") + property(name="msg") + constant(value="', ") + property(name="syslogfacility") + constant(value=", '") + property(name="hostname") + constant(value="', ") + property(name="syslogpriority") + constant(value=", '") + property(name="timereported" dateFormat="mysql") + constant(value="', '") + property(name="timegenerated" dateFormat="mysql") + constant(value="', ") + property(name="iut") + constant(value=", '") + property(name="syslogtag") + constant(value="')") + } + +The equivalent string template looks like this: + +.. code-block:: none + + template(name="stdSQLformat" type="string" option.sql="on" + string="insert into SystemEvents (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')" + ) + +.. note:: + + The template string itself must be on a single line. + + +Generating JSON +--------------- + +This is especially useful for RESTful APIs, like for example ElasticSearch provides. + +This template + +.. code-block:: none + + template(name="outfmt" type="list" option.jsonf="on") { + property(outname="@timestamp" name="timereported" dateFormat="rfc3339" format="jsonf") + property(outname="host" name="hostname" format="jsonf") + property(outname="severity" name="syslogseverity" caseConversion="upper" format="jsonf" datatype="number") + property(outname="facility" name="syslogfacility" format="jsonf" datatype="number") + property(outname="syslog-tag" name="syslogtag" format="jsonf") + property(outname="source" name="app-name" format="jsonf" onEmpty="null") + property(outname="message" name="msg" format="jsonf") + + } + +Generates output similar to this + +.. code-block:: none + + {"@timestamp":"2018-03-01T01:00:00+00:00", "host":"172.20.245.8", "severity":7, "facility":20, "syslog-tag":"tag", "source":"tag", "message":" msgnum:00000000:"} + +Pretty-printed this looks like + +.. code-block:: none + + { + "@timestamp": "2018-03-01T01:00:00+00:00", + "host": "172.20.245.8", + "severity": 7, + "facility": 20, + "syslog-tag": "tag", + "source": "tag", + "message": " msgnum:00000000:" + } + + +.. note:: + + The output is **not** pretty-printed as this is just waste of resources when + used in RESTful APIs. + +If the "app-name" property is empty, a JSON null value is generated as the `onEmpty="null"` +parameter is used + +.. code-block:: none + + {"@timestamp":"2018-03-01T01:00:00+00:00", "host":"172.20.245.8", "severity":7, "facility":20, "syslog-tag":"tag", "source":null, "message":" msgnum:00000000:"} + + +Creating Dynamic File Names for omfile +-------------------------------------- + +Templates can be used to generate actions with dynamic file names. +For example, if you would like to split syslog messages from different hosts +to different files (one per host), you can define the following template: + +.. code-block:: none + + template (name="DynFile" type="string" string="/var/log/system-%HOSTNAME%.log") + + +Reserved Template Names +----------------------- + +Template names beginning with "RSYSLOG\_" are reserved for rsyslog use. +Do NOT use them, otherwise you may cause conflicts in the future +(and quite unpredictable behaviour). There is a small set of pre-defined +templates that you can use without the need to define them: + +**RSYSLOG_TraditionalFileFormat** - The "old style" default log file +format with low-precision timestamps. + +.. code-block:: none + + template(name="RSYSLOG_TraditionalFileFormat" type="string" + string="%TIMESTAMP% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n") + +**RSYSLOG_FileFormat** - A modern-style logfile format similar to +TraditionalFileFormat, both with high-precision timestamps and +timezone information. + +.. code-block:: none + + template(name="RSYSLOG_FileFormat" type="list") { + property(name="timereported" dateFormat="rfc3339") + constant(value=" ") + property(name="hostname") + constant(value=" ") + property(name="syslogtag") + property(name="msg" spifno1stsp="on") + property(name="msg" droplastlf="on") + constant(value="\n") + } + +**RSYSLOG_TraditionalForwardFormat** - The traditional forwarding format +with low-precision timestamps. Most useful if you send messages to +other syslogd's or rsyslogd below version 3.12.5. + +.. code-block:: none + + template(name="RSYSLOG_TraditionalForwardFormat" type="string" + string="<%PRI%>%TIMESTAMP% %HOSTNAME% %syslogtag:1:32%%msg:::sp-if-no-1st-sp%%msg%") + +**RSYSLOG_SysklogdFileFormat** - Sysklogd compatible log file format. If +used with options: ``$SpaceLFOnReceive on``, +``$EscapeControlCharactersOnReceive off``, ``$DropTrailingLFOnReception off``, +the log format will conform to sysklogd log format. + +.. code-block:: none + + template(name="RSYSLOG_SysklogdFileFormat" type="string" + string="%TIMESTAMP% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg%\n") + +**RSYSLOG_ForwardFormat** - a new high-precision forwarding format very +similar to the traditional one, but with high-precision timestamps +and timezone information. Recommended to be used when sending +messages to rsyslog 3.12.5 or above. + +.. code-block:: none + + template(name="RSYSLOG_ForwardFormat" type="string" + string="<%PRI%>%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag:1:32%%msg:::sp-if-no-1st-sp%%msg%") + +**RSYSLOG_SyslogProtocol23Format** - the format specified in IETF's +internet-draft ietf-syslog-protocol-23, which is very close to the actual +syslog standard `RFC5424 <https://tools.ietf.org/html/rfc5424>`_ (we couldn't +update this template as things were in production for quite some time when +RFC5424 was finally approved). This format includes several improvements. +You may use this format with all relatively recent versions of rsyslog or syslogd. + +.. code-block:: none + + template(name="RSYSLOG_SyslogProtocol23Format" type="string" + string="<%PRI%>1 %TIMESTAMP:::date-rfc3339% %HOSTNAME% %APP-NAME% %PROCID% %MSGID% %STRUCTURED-DATA% %msg%\n") + +**RSYSLOG_DebugFormat** - a special format used for troubleshooting +property problems. This format is meant to be written to a log file. +Do **not** use for production or remote forwarding. + +.. code-block:: none + + template(name="RSYSLOG_DebugFormat" type="list") { + constant(value="Debug line with all properties:\nFROMHOST: '") + property(name="fromhost") + constant(value="', fromhost-ip: '") + property(name="fromhost-ip") + constant(value="', HOSTNAME: '") + property(name="hostname") + constant(value="', PRI: '") + property(name="pri") + constant(value=",\nsyslogtag '") + property(name="syslogtag") + constant(value="', programname: '") + property(name="programname") + constant(value="', APP-NAME: '") + property(name="app-name") + constant(value="', PROCID: '") + property(name="procid") + constant(value="', MSGID: '") + property(name="msgid") + constant(value="',\nTIMESTAMP: '") + property(name="timereported") + constant(value="', STRUCTURED-DATA: '") + property(name="structured-data") + constant(value="',\nmsg: '") + property(name="msg") + constant(value="'\nescaped msg: '") + property(name="msg" controlcharacters="drop") + constant(value="'\ninputname: ") + property(name="inputname") + constant(value=" rawmsg: '") + property(name="rawmsg") + constant(value="'\n$!:") + property(name="$!") + constant(value="\n$.:") + property(name="$.") + constant(value="\n$/:") + property(name="$/") + constant(value="\n\n") + } + +**RSYSLOG_WallFmt** - Contains information about the host and the time the +message was generated and at the end the syslogtag and message itself. + +.. code-block:: none + + template(name="RSYSLOG_WallFmt" type="string" + string="\r\n\7Message from syslogd@%HOSTNAME% at %timegenerated% ...\r\n%syslogtag%%msg%\n\r") + +**RSYSLOG_StdUsrMsgFmt** - The syslogtag followed by the message is returned. + +.. code-block:: none + + template(name="RSYSLOG_StdUsrMsgFmt" type="string" + string=" %syslogtag%%msg%\n\r") + +**RSYSLOG_StdDBFmt** - Generates a insert command with the message +properties, into table SystemEvents for a mysql database. + +.. code-block:: none + + template(name="RSYSLOG_StdDBFmt" type="list") { + constant(value="insert into SystemEvents (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag)") + constant(value=" values ('") + property(name="msg") + constant(value="', ") + property(name="syslogfacility") + constant(value=", '") + property(name="hostname") + constant(value="', ") + property(name="syslogpriority") + constant(value=", '") + property(name="timereported" dateFormat="date-mysql") + constant(value="', '") + property(name="timegenerated" dateFormat="date-mysql") + constant(value="', ") + property(name="iut") + constant(value=", '") + property(name="syslogtag") + constant(value="')") + } + +**RSYSLOG_StdPgSQLFmt** - Generates a insert command with the message +properties, into table SystemEvents for a pgsql database. + +.. code-block:: none + + template(name="RSYSLOG_StdPgSQLFmt" type="string" + string="insert into SystemEvents (Message, Facility, FromHost, Priority, DeviceReportedTime, + ReceivedAt, InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', + %syslogpriority%, '%timereported:::date-pgsql%', '%timegenerated:::date-pgsql%', %iut%, + '%syslogtag%')") + +**RSYSLOG_spoofadr** - Generates a message containing nothing more than +the ip address of the sender. + +.. code-block:: none + + template(name="RSYSLOG_spoofadr" type="string" string="%fromhost-ip%") + +**RSYSLOG_StdJSONFmt** - Generates a JSON structure containing the message +properties. + +.. code-block:: none + + template(name="RSYSLOG_StdJSONFmt" type="string" + string="{\"message\":\"%msg:::json%\",\"fromhost\":\"%HOSTNAME:::json%\",\"facility\": + \"%syslogfacility-text%\",\"priority\":\"%syslogpriority-text%\",\"timereported\": + \"%timereported:::date-rfc3339%\",\"timegenerated\": + \"%timegenerated:::date-rfc3339%\"}") + + + +The LEGACY \$template statement +=============================== + +Legacy format provides limited functionality, but is still frequently used. +So you probably need to at least be able to understand them. Legacy format +is also good for fairly simple templates. + +In legacy, only string templates are supported. The full format is as follows: + +.. code-block:: none + + $template myname,<string-template> + +Here, *myname* is the name of the template (like the *name="myname"* property in +modern template format). The *<string-template>* is exactly the *string* property +like in modern template format. + +Take for example the following modern template object: + +.. code-block:: none + + template(name="tpl3" type="string" + string="%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n" + ) + + +It translates to the following legacy template statement: + +.. code-block:: none + + $template tpl3,"%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n" + + + +See Also +-------- + +- `How to bind a + template <http://www.rsyslog.com/how-to-bind-a-template/>`_ +- `Adding the BOM to a + message <http://www.rsyslog.com/adding-the-bom-to-a-message/>`_ +- `How to separate log files by host name of the sending + device <http://www.rsyslog.com/article60/>`_ + diff --git a/source/configuration/timezone.rst b/source/configuration/timezone.rst new file mode 100644 index 0000000..c297d06 --- /dev/null +++ b/source/configuration/timezone.rst @@ -0,0 +1,68 @@ +timezone +======== + +.. index:: ! timezone +.. _cfgobj_input: + +The ``timezone`` object, as its name suggests, describes timezones. +Currently, they are used by message parser modules to interpret +timestamps that contain timezone information via a timezone string +(but not an offset, e.g. "CET" but not "-01:00"). The object describes +an UTC offset for a given timezone ID. + +Each timestamp object adds the zone definition to a global table +with timezone information. Duplicate IDs are forbidden, but the +same offset may be used with multiple IDs. + +As with other configuration objects, parameters for this +object are case-insensitive. + + +Parameters +---------- + +.. function:: id <name-string> + + *Mandatory* + + This identifies the timezone. Note that this id must match the zone + name as reported within the timestamps. Different devices and vendors + use different, often non-standard, names and so it is important to use + the actual ids that messages contain. For multiple devices, this may + mean that you may need to include multiple definitions, each one with a + different id, for the same time zone. For example, it is seen that + some devices report "CEST" for central European daylight savings time + while others report "METDST" for it. + +.. function:: offset <[+/-]><hh>:<mm> + + *Mandatory* + + This defines the timezone offset over UTC. It must always be 6 characters + and start with a "+" (east of UTC) or "-" (west uf UTC) followed by a + two-digit hour offset, a colon and a two-digit minute offset. Hour offsets + can be in the range from zero to twelve, minute offsets in the range from + zero to 59. Any other format is invalid. + +Sample +------ +The following sample defines UTC time. From rsyslog PoV, it doesn't +matter if a plus or minus offset prefix is used. For consistency, +plus is suggested. + +:: + + timezone(id="UTC" offset="+00:00") + +The next sample defines some common timezones: + +:: + + timezone(id="CET" offset="+01:00") + timezone(id="CEST" offset="+02:00") + timezone(id="METDST" offset="+02:00") # duplicate to support differnt formats + timezone(id="EST" offset="-05:00") + timezone(id="EDT" offset="-04:00") + timezone(id="PST" offset="-08:00") + timezone(id="PDT" offset="-07:00") + |
