diff options
| author | Daniel Baumann <daniel.baumann@progress-linux.org> | 2024-04-15 16:27:18 +0000 |
|---|---|---|
| committer | Daniel Baumann <daniel.baumann@progress-linux.org> | 2024-04-15 16:27:18 +0000 |
| commit | f7f20c3f5e0be02585741f5f54d198689ccd7866 (patch) | |
| tree | 190d5e080f6cbcc40560b0ceaccfd883cb3faa01 /source/rainerscript | |
| parent | Initial commit. (diff) | |
| download | rsyslog-doc-f7f20c3f5e0be02585741f5f54d198689ccd7866.tar.xz rsyslog-doc-f7f20c3f5e0be02585741f5f54d198689ccd7866.zip | |
Adding upstream version 8.2402.0+dfsg.upstream/8.2402.0+dfsg
Signed-off-by: Daniel Baumann <daniel.baumann@progress-linux.org>
Diffstat (limited to '')
53 files changed, 3666 insertions, 0 deletions
diff --git a/source/rainerscript/.control_structures.rst.swp b/source/rainerscript/.control_structures.rst.swp Binary files differnew file mode 100644 index 0000000..1b9cabe --- /dev/null +++ b/source/rainerscript/.control_structures.rst.swp diff --git a/source/rainerscript/.rainerscript_call.rst.swp b/source/rainerscript/.rainerscript_call.rst.swp Binary files differnew file mode 100644 index 0000000..209c6ba --- /dev/null +++ b/source/rainerscript/.rainerscript_call.rst.swp diff --git a/source/rainerscript/configuration_objects.rst b/source/rainerscript/configuration_objects.rst new file mode 100644 index 0000000..580d641 --- /dev/null +++ b/source/rainerscript/configuration_objects.rst @@ -0,0 +1,81 @@ +configuration objects +===================== + +.. note:: + + Configuration object parameters are case-insensitive. + +Common Parameters +----------------- + +config.enabled +^^^^^^^^^^^^^^ + +.. versionadded:: 8.33.0 + +All configuration objects have a ``config.enabled`` parameter. +For auto-generated configs, it is useful to have the ability to disable some +config constructs even though they may be specified inside the config. This +can be done via the ``config.enabled`` parameter. +If set to ``on`` or not specified, the construct will be +used, if set to any other value, it will be ignored. +This can be used together with the backtick functionality to enable or +disable parts of the configuration from either a file or environment variable. + +Example: + +Let's say we want to conditionally load a module. Environment variable +``LOAD_IMPTCP`` will be either unset or ``off`` . +Then we can use this config construct: + +.. code-block:: none + :emphasize-lines: 2 + + module(load="imptcp" + config.enabled=`echo $LOAD_IMPTCP`) + +If the variable is set to ``off``, the module will **not** be loaded. + +Objects +------- + +action() +^^^^^^^^ + +The :doc:`action <../configuration/actions>` object is the primary means of +describing actions to be carried out. + +global() +^^^^^^^^ + +This is used to set global configuration parameters. For details, please +see the :doc:`rsyslog global configuration object <global>`. + +input() +^^^^^^^ + +The :doc:`input <../configuration/input>` object is the primary means of +describing inputs, which are used to gather messages for rsyslog processing. + +module() +^^^^^^^^ + +The module object is used to load plugins. + +parser() +^^^^^^^^ + +The :doc:`parser <../configuration/parser>` object is used to define +custom parser objects. + +timezone() +^^^^^^^^^^ + +The :doc:`timezone <../configuration/timezone>` object is used to define +timezone settings. + +include() +^^^^^^^^^ + +The :doc:`include <include>` object is use to include configuration snippets +stored elsewhere into the configuration. diff --git a/source/rainerscript/constant_strings.rst b/source/rainerscript/constant_strings.rst new file mode 100644 index 0000000..ee6a51d --- /dev/null +++ b/source/rainerscript/constant_strings.rst @@ -0,0 +1,83 @@ +String Constants +================ + +String constants are necessary in any scripting language. They provide +values that are evaluated at startup and never change during rsyslog's +run. + +Uses +---- +String constants are necessary in many places: comparisons, +configuration parameter values and function arguments, to name a few +important ones. + +In string constants, special characters are escaped by prepending a +backslash in front of them -- just in the same way this is done in the C +programming language or PHP. + +If in doubt how to properly escape, use the `RainerScript String Escape +Online +Tool <http://www.rsyslog.com/rainerscript-constant-string-escaper/>`_. + +Types +----- + +Rsyslog provides different types of string constants, closely inspired +by the shell: + +- single quotes + + Values are used unaltered, except for escape sequences, which are + escaped. + +- double quotes + + Right now, equivalent to single quotes, but $ signs need to be escaped. + If not escaped, a syntax error will be generated and rsyslog startup + be aborted in most cases. + The idea is to support environment variables just like the shell does + in later releases. + +- backticks + + This was added in 8.33.0. The idea is to provide a useful subset of + what the shell does. Right now, only the following is supported: + + - `echo $VARNAME` - It will evaluate the environment variable and use + it as string constant. If the variable is not found, an empty string + is generated (this is **not** an error). + + Starting with 8.37.0, the `echo` case has been enhanced. It is now + more along the lines of what bash does. It supports multiple + environment variable expansions as well as constant text + between them. + + An example: + + * env SOMEPATH is set to "/var/log/custompath" + * config is: param=echo $SOMEPATH/myfile + * param than is expanded to "/var/log/custompath/myfile" + + Note, however, that some common bash features are not supported. + Most importantly, `${VAR}` does not work. Also, environment variables + are only terminated by whitespace or `/`. Neither are things like + `$(pwd)` supported. The idea of this parameter is not to provide a + full-blown bash-equivalent, but provide some functionality that is + usually considered useful for customizing rsyslog configuration with + outside data. That said, we are still interested in extending the + coverage if clear need and reasoning is provided. + + - `cat filename` - It will evaluate to the content of the given file. + Only a single file name is supported. If the file is not readable, + it will evaluate to an empty string. + + Any other construct will currently lead to an error message. + Note that there must be exactly one space between "echo" or "cat" and + the other parameter. + + Backticks are especially useful for configuration files that are + auto-generated but need to contain a small set of special functionality. + + For an example of this in action, have a look at the rsyslog docker + appliance available at + https://github.com/rsyslog/rsyslog-docker/tree/master/appliance/alpine. diff --git a/source/rainerscript/control_structures.rst b/source/rainerscript/control_structures.rst new file mode 100644 index 0000000..12ce430 --- /dev/null +++ b/source/rainerscript/control_structures.rst @@ -0,0 +1,138 @@ +****************** +Control Structures +****************** + +Control structures in RainerScript are similar in semantics to a lot +of other mainstream languages such as C, Java, Javascript, Ruby, +Bash etc. +So this section assumes the reader is familiar with semantics of such +structures, and goes about describing RainerScript implementation in +usage-example form rather than by formal-definition and +detailed semantics documentation. + +RainerScript supports following control structures: + +if +== + +.. code-block:: none + + if ($msg contains "important") then { + if ( $.foo != "" ) then set $.foo = $.bar & $.baz; + action(type="omfile" file="/var/log/important.log" template="outfmt") + } + + +if/else-if/else +=============== + +.. code-block:: none + + if ($msg contains "important") then { + set $.foo = $.bar & $.baz; + action(type="omfile" file="/var/log/important.log" template="outfmt") + } else if ($msg startswith "slow-query:") then { + action(type="omfile" file="/var/log/slow_log.log" template="outfmt") + } else { + set $.foo = $.quux; + action(type="omfile" file="/var/log/general.log" template="outfmt") + } + + +foreach +======= + +A word of caution first: there often is a misunderstanding in regard to foreach: +this construct only works on JSON structures. Actually, we should have rejected the +proposal for "foreach" at the time it was made, but now it is too late. + +So please be warned: there is no general concept of an "array" inside the script +language. This is intentional as we do not wanted to get it too complex. +Where you can use arrays is for some config objects and a select set of comparisons, +but nowhere else. + +If you parsed JSON, foreach can iterate both JSON arrays and JSON objects inside this +parsed JSON. As opposed to JSON array-iteration (which is ordered), JSON object-iteration +accesses key-values in arbitrary order (is unordered). + +For the foreach invocation below: + +.. code-block:: none + + foreach ($.i in $.collection) do { + ... + } + + +Say ``$.collection`` holds an array ``[1, "2", {"a": "b"}, 4]``, value +of ``$.i`` across invocations would be ``1``, ``"2"``, ``{"a" : "b"}`` +and ``4``. + +Note that ``$.collection`` must have been parsed from JSON (via mmjsonparse). + +When ``$.collection`` holds an object +``{"a": "b", "c" : [1, 2, 3], "d" : {"foo": "bar"}}``, value of ``$.i`` +across invocations would be ``{"key" : "a", "value" : "b"}``, +``{"key" : "c", "value" : [1, 2, 3]}`` and +``{"key" : "d", "value" : {"foo" : "bar"}}`` (not necessarily in the that +order). In this case key and value will need to be accessed as ``$.i!key`` +and ``$.i!value`` respectively. + + + +Here is an example of a nested foreach statement: + +.. code-block:: none + + foreach ($.quux in $!foo) do { + action(type="omfile" file="./rsyslog.out.log" template="quux") + foreach ($.corge in $.quux!bar) do { + reset $.grault = $.corge; + action(type="omfile" file="./rsyslog.out.log" template="grault") + if ($.garply != "") then + set $.garply = $.garply & ", "; + reset $.garply = $.garply & $.grault!baz; + } + } + +Again, the itereted items must have been created by parsing JSON. + +Please note that asynchronous-action calls in foreach-statement body should +almost always set ``action.copyMsg`` to ``on``. This is because action calls +within foreach usually want to work with the variable loop populates (in the +above example, ``$.quux`` and ``$.corge``) which causes message-mutation and +async-action must see message as it was in a certain invocation of loop-body, +so they must make a copy to keep it safe from further modification as iteration +continues. For instance, an async-action invocation with linked-list based +queue would look like: + +.. code-block:: none + + foreach ($.quux in $!foo) do { + action(type="omfile" file="./rsyslog.out.log" template="quux + queue.type="linkedlist" action.copyMsg="on") + } + +Note well where foreach does **not** work: + +.. code-block:: none + + set $.noarr = ["192.168.1.1", "192.168.2."]; + foreach ($.elt in $.noarr) do { + ... + } + +This is the case because the assignment does not create a JSON array. + + +call +==== + +Details here: :doc:`rainerscript_call` + + +continue +======== + +A NOP, useful e.g. inside the ``then`` part of an if-structure. + diff --git a/source/rainerscript/data_types.rst b/source/rainerscript/data_types.rst new file mode 100644 index 0000000..cadf722 --- /dev/null +++ b/source/rainerscript/data_types.rst @@ -0,0 +1,9 @@ +Data Types +========== + +RainerScript is a typeless language. That doesn't imply you don't need +to care about types. Of course, expressions like "A" + "B" will not +return a valid result, as you can't really add two letters (to +concatenate them, use the concatenation operator &). However, all type +conversions are automatically done by the script interpreter when there +is need to do so.
\ No newline at end of file diff --git a/source/rainerscript/expressions.rst b/source/rainerscript/expressions.rst new file mode 100644 index 0000000..a3f6060 --- /dev/null +++ b/source/rainerscript/expressions.rst @@ -0,0 +1,25 @@ +Expressions +=========== + +The language supports arbitrary complex expressions. All usual operators +are supported. The precedence of operations is as follows (with +operations being higher in the list being carried out before those lower +in the list, e.g. multiplications are done before additions. + +- expressions in parenthesis +- not, unary minus +- \*, /, % (modulus, as in C) +- +, -, & (string concatenation) +- ==, !=, <>, <, >, <=, >=, contains (strings!), startswith (strings!) +- and +- or + +For example, "not a == b" probably returns not what you intended. The +script processor will first evaluate "not a" and then compare the +resulting boolean to the value of b. What you probably intended to do is +"not (a == b)". And if you just want to test for inequality, we highly +suggest to use "!=" or "<>". Both are exactly the same and are provided +so that you can pick whichever you like best. So inequality of a and b +should be tested as "a <> b". The "not" operator should be reserved to +cases where it actually is needed to form a complex boolean expression. +In those cases, parentheses are highly recommended. diff --git a/source/rainerscript/functions/idx_built-in_functions.rst b/source/rainerscript/functions/idx_built-in_functions.rst new file mode 100644 index 0000000..d2be538 --- /dev/null +++ b/source/rainerscript/functions/idx_built-in_functions.rst @@ -0,0 +1,11 @@ +****************** +Built-in Functions +****************** + +Following functions are available in Rainerscript. + +.. toctree:: + :glob: + :maxdepth: 1 + + rs* diff --git a/source/rainerscript/functions/idx_module_functions.rst b/source/rainerscript/functions/idx_module_functions.rst new file mode 100644 index 0000000..1998b56 --- /dev/null +++ b/source/rainerscript/functions/idx_module_functions.rst @@ -0,0 +1,16 @@ +**************** +Module Functions +**************** + +You can make module functions accessible for the configuration by loading the +corresponding module. Once they are loaded, you can use them like any other +rainerscript function. If more than one function are part of the same module, +all functions will be available once the module is loaded. + + +.. toctree:: + :glob: + :maxdepth: 1 + + mo* + diff --git a/source/rainerscript/functions/index.rst b/source/rainerscript/functions/index.rst new file mode 100644 index 0000000..a40e166 --- /dev/null +++ b/source/rainerscript/functions/index.rst @@ -0,0 +1,27 @@ +********* +Functions +********* + +There are two types of RainerScript functions: built-ins and modules. Built-in +functions can always be called in the configuration. To use module functions, +you have to load the corresponding module first. To do this, add the following +line to your configuration: + +:: + + module(load="<name of module>") + +If more than one function with the same name is present, the function of the +first module loaded will be used. Also, an error message stating this will be +generated. However, the configuration will not abort. Please note that built-in +functions will always be automatically loaded before any modules. Because of this, you +are unable to override any of the built-in functions, since their names are already +in use. The name of a function module starts with fm. + + +.. toctree:: + :glob: + :maxdepth: 2 + + idx_built-in_functions + idx_module_functions diff --git a/source/rainerscript/functions/mo-ffaup.rst b/source/rainerscript/functions/mo-ffaup.rst new file mode 100644 index 0000000..5291c74 --- /dev/null +++ b/source/rainerscript/functions/mo-ffaup.rst @@ -0,0 +1,103 @@ +************ +Faup +************ + + +Description +=========== + +These new functions allow anybody to parse any variable containing URLs, hostnames, DNS queries and such to extract: + - the *scheme* + - the *credentials* (if present) + - the *tld* (with support for second-level TLDs) + - the *domain* (with and without the tld) + - the *subdomain* + - the *full hostname* + - the *port* (if present) + - the *resource* path (if present) + - the *query string* parameters (if present) + - the *fragment* (if present) + +HowTo +----- +The module functions are fairly simple to use, and are divided in 2 classes: +* `faup()` allows to parse the entire URL and return all parts in a complete JSON +* `faup_<field>()` allows to parse the entire URL, but only returns the (potential) value of the requested field as string + +Examples +^^^^^^^^ +Using the `faup()` function +""""""""""""""""""""""""""""""" +The `faup()` is the simplest function to use, simply provide a value or variable (any type) as the only parameter, and the function returns a json object containing every element of the URL. + +*example code:* + +.. code-block:: none + + set $!url = "https://user:pass@www.rsyslog.com:443/doc/v8-stable/rainerscript/functions/mo-faup.html?param=value#faup"; + set $.faup = faup($!url); + + +*$.faup will contain:* + +.. code-block:: none + + { + "scheme": "https", + "credential": "user:pass", + "subdomain": "www", + "domain": "rsyslog.com", + "domain_without_tld": "rsyslog", + "host": "www.rsyslog.com", + "tld": "com", + "port": "443", + "resource_path": "\/doc\/v8-stable\/rainerscript\/functions\/mo-ffaup.html", + "query_string": "?param=value", + "fragment": "#faup" + } + +.. note:: + + This is a classic rsyslog variable, and you can access every sub-key with `$.faup!domain`, `$.faup!resource_path`, etc... + + +Using the `faup_<field>()` functions +"""""""""""""""""""""""""""""""""""""""" +Using the field functions is even simpler: for each field returned by the `faup()` function, there exists a corresponding function to get only that one field. + +For example, if the goal is to recover the domain without the tld, the example above could be modified as follows: + +*example code:* + +.. code-block:: none + + set $!url = "https://user:pass@www.rsyslog.com:443/doc/v8-stable/rainerscript/functions/mo-faup.html?param=value#faup"; + set $.faup = faup_domain_without_tld($!url); + +*$.faup will contain:* + +.. code-block:: none + + rsyslog + +.. note:: + + The returned value is no longer a json object, but a simple string + + +Requirements +============ +This module relies on the `faup <https://github.com/stricaud/faup>`_ library. + +The library should be compiled (see link for instructions on how to compile) and installed on build and runtime machines. + +.. warning:: + + Even if faup is statically compiled to rsyslog, the library still needs an additional file to work properly: the mozilla.tlds stored by the libfaup library in /usr/local/share/faup. It permits to properly match second-level TLDs and allow URLs such as www.rsyslog.co.uk to be correctly parsed into \<rsyslog:domain\>.\<co.uk:tld\> and not \<rsyslog:subdomain\>.\<co:domain\>.\<uk:tld\> + + +Motivations +=========== +Those functions are the answer to a growing problem encountered in Rsyslog when using modules to enrich logs : some mechanics (like lookup tables or external module calls) require "strict" URL/hostname formats that are often not formatted correctly, resulting in lookup failures/misses. + +This ensures getting stable inputs to provide to lookups/modules to enrich logs.
\ No newline at end of file diff --git a/source/rainerscript/functions/mo-hashXX.rst b/source/rainerscript/functions/mo-hashXX.rst new file mode 100644 index 0000000..61bb69a --- /dev/null +++ b/source/rainerscript/functions/mo-hashXX.rst @@ -0,0 +1,42 @@ +****** +HashXX +****** + +Purpose +======= + +Generates hash for a given string. + +hash32(literal_string) / hash32(literal_string, seed) +----------------------------------------------------- + + Generates a 32 bit hash of the given string. + - Seed is an optional parameter with default = 0. + - If seed is not a valid number, then 0 is returned. + +hash64(literal_string) / hash64(literal_string, seed) +----------------------------------------------------- + + Generates a 64 bit hash of the given string. + - Seed is an optional parameter with default = 0. + - If seed is not a valid number, then 0 is returned. + +.. warning:: + + - Default hash implementation is non-crypto. + - To use xxhash enable compile time flag. + + +Example +======= + +.. code-block:: none + + module(load="fmhash") + + set $.hash = hash64($!msg!field_1 & $!msg!field_2 & $!msg!field_3) + set $!tag= $syslogtag & $.hash; + //send out + +.. seealso:: + :doc:`Hash based sampling<../../tutorials/hash_sampling>` diff --git a/source/rainerscript/functions/mo-hashXXmod.rst b/source/rainerscript/functions/mo-hashXXmod.rst new file mode 100644 index 0000000..8fc1d77 --- /dev/null +++ b/source/rainerscript/functions/mo-hashXXmod.rst @@ -0,0 +1,48 @@ +********* +HashXXmod +********* + +Purpose +======= + +Generates a number which is mod of given string's hash. + +hash32mod(literal_string, modulo) / hash32mod(literal_string, modulo, seed) +--------------------------------------------------------------------------- + + Generates a number which is calculated on (32 bit hash of the given string % modulo) + - If modulo is not a valid number, then 0 is returned. + - If modulo is 0, then 0 is returned. + - Seed is an optional parameter with default = 0. + - If seed is not a valid unsigned number, then 0 is returned. + + +hash64mod(literal_string, modulo) / hash64mod(literal_string, modulo, seed) +--------------------------------------------------------------------------- + + Generates a number which is calculated on (64 bit hash of the given string % modulo) + - If modulo is not a valid number, then 0 is returned. + - If modulo is 0, then 0 is returned. + - Seed is an optional parameter with default = 0. + - If seed is not a valid unsigned number, then 0 is returned. + + +.. warning:: + + - Default hash implementation is non-crypto. + - To use xxhash enable compile time flag. + + +Example +======= + +.. code-block:: none + + module(load="fmhash") + + if (hash64mod($!msg!request_id, 100) <= 30) then { + //send out + } + +.. seealso:: + :doc:`Hash based sampling<../../tutorials/hash_sampling>` diff --git a/source/rainerscript/functions/mo-http_request.rst b/source/rainerscript/functions/mo-http_request.rst new file mode 100644 index 0000000..a0b0f22 --- /dev/null +++ b/source/rainerscript/functions/mo-http_request.rst @@ -0,0 +1,39 @@ +************ +HTTP-Request +************ + +Purpose +======= + +http_request(str) + +Performs a http request to target and returns the result of said request. + +.. note:: + + this function is very slow and therefore we suggest using it only seldomly + to insure adequate performance. + + +Example +======= + +The following example performs a http request and writes the returned value +to a file. + + +.. code-block:: none + + module(load="../plugins/imtcp/.libs/imtcp") + module(load="../plugins/fmhttp/.libs/fmhttp") + input(type="imtcp" port="13514") + + template(name="outfmt" type="string" string="%$!%\n") + + if $msg contains "msgnum:" then { + set $.url = "http://www.rsyslog.com/testbench/echo-get.php?content=" & ltrim($msg); + set $!reply = http_request($.url); + action(type="omfile" file="rsyslog.out.log" template="outfmt") + } + + diff --git a/source/rainerscript/functions/mo-unflatten.rst b/source/rainerscript/functions/mo-unflatten.rst new file mode 100644 index 0000000..136251e --- /dev/null +++ b/source/rainerscript/functions/mo-unflatten.rst @@ -0,0 +1,61 @@ +************ +Unflatten +************ + +Purpose +======= + +``<result> = unflatten(<source-tree>, <key-separator-character>);`` + +This function unflattens keys in a JSON object. It provides a way to expand dot-separated fields. + +It allows for instance to produce this: + ``{ "source": { "ip": "1.2.3.4", "port": 443 } }`` + +from this source data: + ``{ "source.ip": "1.2.3.4", "source.port": 443 }`` + + +Example +======= + +Here is a sample use case: + +.. code-block:: none + + module(load="fmunflatten") + + # Say you have the following tree, obtained for instance with mmnormalize. + set $!source.ip = "1.2.3.4"; + set $!source.bytes = 3258; + set $!source.geo.country_iso_code = "FR"; + set $!destination.ip = "4.3.2.1"; + + # Now unflatten the keys in the $! tree. + set $.unflatten = unflatten($!, "."); + + # You may do this to set back safely the result in $! because the function could + # return a default dummy value of 0 (rsyslog number) if $! was not touched (it + # would evaluate to an empty rsyslog string, which is not a JSON datatype). + if (script_error() == 0) then { + unset $!; + set $! = $.unflatten; + unset $.unflatten; + } + +An output of ``$!`` would give this, in pretty-print: + +.. code-block:: none + + { + "source": { + "ip": "1.2.3.4", + "bytes": 3258, + "geo": { + "country_iso_code": "FR" + } + }, + "destination": { + "ip": "4.3.2.1" + } + } diff --git a/source/rainerscript/functions/rs-cnum.rst b/source/rainerscript/functions/rs-cnum.rst new file mode 100644 index 0000000..055a343 --- /dev/null +++ b/source/rainerscript/functions/rs-cnum.rst @@ -0,0 +1,36 @@ +****** +cnum() +****** + +Purpose +======= + +cnum(expr) + +Converts expr to a number (integer). + +.. note:: + + If the expression does not contain a numerical value, the following + rule applies: the best match as the number is returned. For example + "1x234" will return the number 1 and "Test123" will return 0. Zero is + always returned if the there is no number at the start of the string. + This also is the case for empty strings. + + +Example +======= + + + +.. code-block:: none + + cnum(3+2); + +produces + +.. code-block:: none + + 5 + + diff --git a/source/rainerscript/functions/rs-cstr.rst b/source/rainerscript/functions/rs-cstr.rst new file mode 100644 index 0000000..3cf007e --- /dev/null +++ b/source/rainerscript/functions/rs-cstr.rst @@ -0,0 +1,27 @@ +****** +cstr() +****** + +Purpose +======= + +cstr(expr) + +Converts expr to a string value. + + +Example +======= + +In the following example a string is created from the expression. + +.. code-block:: none + + cstr(5*4); + +produces + +.. code-block:: none + + "20" + diff --git a/source/rainerscript/functions/rs-dyn_inc.rst b/source/rainerscript/functions/rs-dyn_inc.rst new file mode 100644 index 0000000..67769cc --- /dev/null +++ b/source/rainerscript/functions/rs-dyn_inc.rst @@ -0,0 +1,35 @@ +********* +dyn_inc() +********* + +Purpose +======= + +dyn_inc(bucket_name_literal_string, str) + +Increments counter identified by ``str`` in dyn-stats bucket identified +by ``bucket_name_literal_string``. Returns 0 when increment is successful, +any other return value indicates increment failed. + +Counters updated here are reported by **impstats**. + +Except for special circumstances (such as memory allocation failing etc), +increment may fail due to metric-name cardinality being under-estimated. +Bucket is configured to support a maximum cardinality (to prevent abuse) +and it rejects increment-operation if it encounters a new(previously unseen) +metric-name(``str``) when full. + +**Read more about it here** :doc:`Dynamic Stats<../../configuration/dyn_stats>` + + +Example +======= + +The following example shows the counter $hostname incremented in the bucket +msg_per_host. + +.. code-block:: none + + dyn_inc("msg_per_host", $hostname) + + diff --git a/source/rainerscript/functions/rs-exec_template.rst b/source/rainerscript/functions/rs-exec_template.rst new file mode 100644 index 0000000..76b07cc --- /dev/null +++ b/source/rainerscript/functions/rs-exec_template.rst @@ -0,0 +1,25 @@ +*************** +exec_template() +*************** + +Purpose +======= + +exec_template(str) + +Sets a variable through the execution of a template. Basically this permits to easily +extract some part of a property and use it later as any other variable. + +**Read more about it here :** `<http://www.rsyslog.com/how-to-use-set-variable-and-exec_template>`_ + +Example +======= + +The following example shows the template being used to extract a part of the message. + +.. code-block:: none + + template(name="extract" type="string" string="%msg:F:5%") + set $!xyz = exec_template("extract"); + + diff --git a/source/rainerscript/functions/rs-exists.rst b/source/rainerscript/functions/rs-exists.rst new file mode 100644 index 0000000..eabffd8 --- /dev/null +++ b/source/rainerscript/functions/rs-exists.rst @@ -0,0 +1,30 @@ +******** +exists() +******** + +Purpose +======= + +exists($!path!varname) + +This function checks if the specified variable exists, in other +words: contains a value. A variable that once was set and later +on unset does also not exist. + +The function accepts a single argument, which needs to be a variable. +It returns 0 if the variable does not exist and 1 otherwise. The +function can be combined with any other expression to form more +complec expressions. + +.. versionadded:: 8.2010.10 + + +Example +======= + +.. code-block:: none + + if exists(!$path!varname) then ... + if not exists($.local!var) then ... + if exists($!triggervar) and $msg contains "something" then ... + diff --git a/source/rainerscript/functions/rs-field.rst b/source/rainerscript/functions/rs-field.rst new file mode 100644 index 0000000..bc3c08b --- /dev/null +++ b/source/rainerscript/functions/rs-field.rst @@ -0,0 +1,64 @@ +******* +field() +******* + +Purpose +======= + +field(str, delim, matchnbr) + +Returns a field-based substring. str is +the string to search, delim is the delimiter and matchnbr is the +match to search for (the first match starts at 1). This works similar +as the field based property-replacer option. Versions prior to 7.3.7 +only support a single character as delimiter character. Starting with +version 7.3.7, a full string can be used as delimiter. If a single +character is being used as delimiter, delim is the numerical ascii +value of the field delimiter character (so that non-printable +characters can by specified). If a string is used as delimiter, a +multi-character string (e.g. "#011") is to be specified. + +.. note:: + + When a single character is specified as string + ``field($msg, ",", 3)`` a string-based extraction is done, which is + more performance intensive than the equivalent single-character + ``field($msg, 44 ,3)`` extraction. + + +Example +======= + +With ascii value of the field delimiter +--------------------------------------- + +Following example returns the third field delimited by space. +In this example $msg will be "field1 field2 field3 field4". + +.. code-block:: none + + set $!usr!field = field($msg, 32, 3); + +produces + +.. code-block:: none + + "field3" + + +With a string as the field delimiter +------------------------------------ + +This example returns the second field delimited by "#011". +In this example $msg will be "field1#011field2#011field3#011field4". + +.. code-block:: none + + set $!usr!field = field($msg, "#011", 2); -- the second field, delimited by "#011" + +produces + +.. code-block:: none + + "field2" + diff --git a/source/rainerscript/functions/rs-format_time.rst b/source/rainerscript/functions/rs-format_time.rst new file mode 100644 index 0000000..aac628e --- /dev/null +++ b/source/rainerscript/functions/rs-format_time.rst @@ -0,0 +1,64 @@ +************* +format_time() +************* + +Purpose +======= + +.. note:: + + This is EXPERIMENTAL code - it may be removed or altered in + later versions than 8.30.0. Please watch the ChangeLog closely for + updates. + +Converts a UNIX timestamp to a formatted RFC 3164 or RFC 3339 date/time string. +The first parameter is expected to be an integer value representing the number of +seconds since 1970-01-01T00:00:0Z (UNIX epoch). The second parameter can be one of +``"date-rfc3164"`` or ``"date-rfc3339"``. The output is a string containing +the formatted date/time. Date/time strings are expressed in **UTC** (no time zone +conversion is provided). + +.. note:: + + If the input to the function is NOT a proper UNIX timestamp, a string + containing the original value of the parameter will be returned instead of a + formatted date/time string. + + +Example +======= + +RFC 3164 timestamp +------------------ + +In the following example the integer representing a UNIX timestamp is +formatted to a rfc-3164 date/time string. + +.. code-block:: none + + format_time(1507165811, "date-rfc3164") + +produces + +.. code-block:: none + + Oct 5 01:10:11 + + +Wrong input +----------- + +In the following example a wrong parameter is given which can't be +formatted and so it is returned unchanged. + +.. code-block:: none + + format_time("foo", "date-rfc3164") + +produces + +.. code-block:: none + + foo + + diff --git a/source/rainerscript/functions/rs-get_property.rst b/source/rainerscript/functions/rs-get_property.rst new file mode 100644 index 0000000..a1536fa --- /dev/null +++ b/source/rainerscript/functions/rs-get_property.rst @@ -0,0 +1,58 @@ +************** +get_property() +************** + +Purpose +======== + +get_property(rsyslog_variable, key_str) + +Provides ability to get a rsyslog variable or property using dynamically evaluated parameters. +The first parameter is a valid rsyslog variable or property, the second parameter is a key string, or index value. + + +Example +======== + +In the following example, a json string is parsed using parse_json(), and placed into the variable ``$!parsed``. +The get_property function is then used to get property fields from ``$!parsed``. + +.. code-block:: none + + set $.ret = parse_json("{\"offsets\": [ { \"a\": 9, \"b\": 0 },\ + { \"a\": 9, \"b\": 3 } ],\ + \"boolval\": true,\ + \"int64val\": 1234567890,\ + \"nullval\": null,\ + \"doubleval\": 12345.67890 }", "\$!parsed"); + + # get different fields from $!parsed here + if $.ret == 0 then { + # dynamically evaluate different fields in $!parsed + + # do dynamic indexing into array + $.index = 1; + # set $.ret = { "a": 9, "b": 3} + set $.ret = get_property($!parsed!offsets, $.index); + + # set $.ret = true; + set $.key = "boolval"; + set $.ret = get_property($!parsed, $.key); + + # null values are evaluated to empty string + # thus below statement will set $.ret to empty string + set $.ret = get_property($!parsed, "nullval"); + + # evaluates to 1234567890 + set $.key = "int64val"; + set $.ret = get_property($!parsed, $.key); + + # using a key variable, evaluates to 12345.67890 + set $key = "doubleval"; + set $.ret = get_property($!parsed, $key); + } + + # example of dynamically building key value + set $.foo!barval = 3; + # set $.bar = 3; + set $.bar = get_property($.foo, "bar" & "val"); diff --git a/source/rainerscript/functions/rs-getenv.rst b/source/rainerscript/functions/rs-getenv.rst new file mode 100644 index 0000000..0035a8b --- /dev/null +++ b/source/rainerscript/functions/rs-getenv.rst @@ -0,0 +1,23 @@ +******** +getenv() +******** + +Purpose +======= + +getenv(str) + +Like the OS call, returns the value of the environment variable, +if it exists. Returns an empty string if it does not exist. + + +Examples +======== + +The following example can be used to build a dynamic filter based on +some environment variable: + +.. code-block:: none + + if $msg contains getenv('TRIGGERVAR') then /path/to/errfile + diff --git a/source/rainerscript/functions/rs-int2hex.rst b/source/rainerscript/functions/rs-int2hex.rst new file mode 100644 index 0000000..29cbbdd --- /dev/null +++ b/source/rainerscript/functions/rs-int2hex.rst @@ -0,0 +1,56 @@ +********* +int2hex() +********* + +Purpose +======= + +int2hex(num) + +Returns a hexadecimal number string of a given positive integer num. + +.. note:: + + When a negative integer is given the return value will be the + `Two's complement <https://en.wikipedia.org/wiki/Two%27s_complement>`_ of + the input number. + + +Example +======= + + +Positive Integer given +---------------------- + +In the following example the hexadecimal number of 61 is returned +as a string. + +.. code-block:: none + + int2hex(61) + +produces + +.. code-block:: none + + 3d + + +Negative Integer given +---------------------- + +This example shows the result on a 64-bit system when a negative +integer such as -13 is put into the function. + +.. code-block:: none + + int2hex(-13) + +produces + +.. code-block:: none + + fffffffffffffff3 + + diff --git a/source/rainerscript/functions/rs-ipv4convert.rst b/source/rainerscript/functions/rs-ipv4convert.rst new file mode 100644 index 0000000..5ab9b16 --- /dev/null +++ b/source/rainerscript/functions/rs-ipv4convert.rst @@ -0,0 +1,52 @@ +*********************** +num2ipv4() / ipv42num() +*********************** + +Purpose +======= + +num2ipv4 +-------- + +num2ipv4(int) + +Converts an integer into an IPv4-address and returns the address as string. +Input is an integer with a value between 0 and 4294967295. The output format +is '>decimal<.>decimal<.>decimal<.>decimal<' and '-1' if the integer input is +invalid or if the function encounters a problem. + + +ipv42num +-------- + +ipv42num(str) + +Converts an IPv4-address into an integer and returns the integer. Input is +a string; the expected address format may include spaces in the beginning +and end, but must not contain any other characters in between (except dots). +If the format does include these, the function results in an error and returns -1. + + +Example +======= + +num2ipv4 +-------- + +This example shows an integer being converted to an ipv4 address. + +.. code-block:: none + + num2ipv4(123456) + + +ipv42num +-------- + +This example shows the parameter fromhost-ip being converted to an integer. + +.. code-block:: none + + ipv42num($fromhost-ip) + + diff --git a/source/rainerscript/functions/rs-is_time.rst b/source/rainerscript/functions/rs-is_time.rst new file mode 100644 index 0000000..42ca727 --- /dev/null +++ b/source/rainerscript/functions/rs-is_time.rst @@ -0,0 +1,66 @@ +********* +is_time() +********* + +Purpose +======= + +is_time(timestamp) +is_time(timestamp, format_str) + +Checks the given timestamp to see if it is a valid date/time string (RFC 3164, +or RFC 3339), or a UNIX timestamp. + +This function returns ``1`` for valid date/time strings and UNIX timestamps, +``0`` otherwise. Additionally, if the input cannot be parsed, or there is +an error, ``script_error()`` will be set to error state. + +The ``format_str`` parameter is optional, and can be one of ``"date-rfc3164"``, +``"date-rfc3339"`` or ``"date-unix"``. If this parameter is specified, the +function will only succeed if the input matches that format. If omitted, the +function will compare the input to all of the known formats (indicated above) +to see if one of them matches. + +.. note:: + + This function does not support unusual RFC 3164 dates/times that + contain year or time zone information. + + +Example +======= + +Only timestamp is given +----------------------- + +The following example shows the output when a valid timestamp is given. + +.. code-block:: none + + is_time("Oct 5 01:10:11") + is_time("2017-10-05T01:10:11+04:00") + is_time(1507165811) + +all produce + +.. code-block:: none + + 1 + + +Timestamp and Format given +-------------------------- + +The following example shows the output when a valid timestamp is given but +the format does not match. + +.. code-block:: none + + is_time("2017-10-05T01:10:11+04:00", "date-unix") + +all produce + +.. code-block:: none + + 0 + diff --git a/source/rainerscript/functions/rs-lookup.rst b/source/rainerscript/functions/rs-lookup.rst new file mode 100644 index 0000000..cdcd563 --- /dev/null +++ b/source/rainerscript/functions/rs-lookup.rst @@ -0,0 +1,33 @@ +******** +lookup() +******** + +Purpose +======= + +lookup(table_name_literal_string, key) + +Lookup tables are a powerful construct to obtain *class* information based +on message content. It works on top of a data-file which maps key (to be looked +up) to value (the result of lookup). + +The idea is to use a message properties (or derivatives of it) as an index +into a table which then returns another value. For example, $fromhost-ip +could be used as an index, with the table value representing the type of +server or the department or remote office it is located in. + +**Read more about it here** :doc:`Lookup Tables<../../configuration/lookup_tables>` + + +Example +======= + +In the following example the hostname is looked up in the given table and +the corresponding value is returned. + +.. code-block:: none + + lookup_table(name="host_bu" file="/var/lib/host_billing_unit_mapping.json") + set $.bu = lookup("host_bu", $hostname); + + diff --git a/source/rainerscript/functions/rs-parse_json.rst b/source/rainerscript/functions/rs-parse_json.rst new file mode 100644 index 0000000..269b587 --- /dev/null +++ b/source/rainerscript/functions/rs-parse_json.rst @@ -0,0 +1,27 @@ +************ +parse_json() +************ + +Purpose +======= + +parse_json(str, container) + +Parses the json string ``str`` and places the resulting json object +into ``container`` where container can be any valid rsyslog variable. +Returns 0 on success and something otherwise if ``str`` does **not** +contain valid json. + + +Example +======= + +In the following example the json string is placed into the variable $!parsed. +The output is placed in variable $.ret + +.. code-block:: none + + set $.ret = parse_json("{ \"c1\":\"data\" }", "\$!parsed"); + + + diff --git a/source/rainerscript/functions/rs-parse_time.rst b/source/rainerscript/functions/rs-parse_time.rst new file mode 100644 index 0000000..434df3d --- /dev/null +++ b/source/rainerscript/functions/rs-parse_time.rst @@ -0,0 +1,44 @@ +************ +parse_time() +************ + +Purpose +======= + +parse_time(timestamp) + +Converts an RFC 3164 or RFC 3339 formatted date/time string to a UNIX timestamp +(an integer value representing the number of seconds since the UNIX epoch: +1970-01-01T00:00:0Z). + +If the input to the function is not a properly formatted RFC 3164 or RFC 3339 +date/time string, or cannot be parsed, ``0`` is returned and ``script_error()`` +will be set to error state. + +.. note:: + + This function does not support unusual RFC 3164 dates/times that + contain year or time zone information. + +.. note:: + + Fractional seconds (if present) in RFC 3339 date/time strings will + be discarded. + + +Example +======= + +In the following example a timestamp is parsed into an integer. + +.. code-block:: none + + parse_time("Oct 5 01:10:11") # Assumes the current year (2017, in this example) + +produces + +.. code-block:: none + + 1507165811 + + diff --git a/source/rainerscript/functions/rs-percentile_observe.rst b/source/rainerscript/functions/rs-percentile_observe.rst new file mode 100644 index 0000000..40bc81f --- /dev/null +++ b/source/rainerscript/functions/rs-percentile_observe.rst @@ -0,0 +1,33 @@ +******************** +percentile_observe() +******************** + +Purpose +======= + +percentile_observe(bucket_name_literal_string, stats_name_literal_string, value) + +Adds an observation, identified by the ``bucket_name_literal_string`` +and ``stats_name_literal_string``, to the observation set. + +Periodically, on an **impstats** reporting interval, percentile and summary metrics are generated +based on the statistics collected. + +Except for special circumstances (such as memory allocation failing etc), +observe calls may fail due to stat-name cardinality being under-estimated. +Bucket is configured to support a maximum cardinality (to prevent abuse) +and it rejects observe-operation if it encounters a new(previously unseen) +stat-name(``str``) when full. + +**Read more about it here** :doc:`Percentile Stats<../../configuration/percentile_stats>` + + +Example +======= + +In the following example, the ``$msg_count`` value is being recorded as a ``msg_per_host`` statistic in the ``host_statistics`` bucket. + + +.. code-block:: none + + percentile_observe("host_statistics", "msg_per_host", $msg_count) diff --git a/source/rainerscript/functions/rs-previous_action_suspended.rst b/source/rainerscript/functions/rs-previous_action_suspended.rst new file mode 100644 index 0000000..b78e448 --- /dev/null +++ b/source/rainerscript/functions/rs-previous_action_suspended.rst @@ -0,0 +1,30 @@ +*************************** +previous_action_suspended() +*************************** + +Purpose +======= + +previous_action_suspended() + +This boolean function returns 1 (true) if the previous action is suspended, +0 (false) otherwise. It can be used to initiate action that shall happen if +a function failed. Please note that an action failure may not be immediately +detected, so the function return value is a bit fuzzy. It is guaranteed, however +that a suspension will be detected with the next batch of messages that is +being processed. + + +Example +======= + +In the following example the if-clause is executed if the previous action +is suspended. + +.. code-block:: none + + action(type="omfwd" protocol="tcp" target="10.1.1.1") + + if previous_action_suspended() then {} + + diff --git a/source/rainerscript/functions/rs-prifilt.rst b/source/rainerscript/functions/rs-prifilt.rst new file mode 100644 index 0000000..c8a88b7 --- /dev/null +++ b/source/rainerscript/functions/rs-prifilt.rst @@ -0,0 +1,23 @@ +********* +prifilt() +********* + +Purpose +======= + +prifilt(constant) + +Mimics a traditional PRI-based filter (like "\*.\*" or "mail.info"). +The traditional filter string must be given as a **constant string**. +Dynamic string evaluation is not permitted (for performance reasons). + + +Example +======= + +In this example true would be returned on an auth event. + +.. code-block:: none + + prifilt("auth,authpriv.*") + diff --git a/source/rainerscript/functions/rs-random.rst b/source/rainerscript/functions/rs-random.rst new file mode 100644 index 0000000..dbf1c15 --- /dev/null +++ b/source/rainerscript/functions/rs-random.rst @@ -0,0 +1,36 @@ +******** +random() +******** + +Purpose +======= + +random(max) + +Generates a random number between 0 and the number specified, though +the maximum value supported is platform specific. + +- If a number is not specified then 0 is returned. +- If 0 is provided as the maximum value, then 0 is returned. +- If the specified value is greater than the maximum supported + for the current platform, then rsyslog will log this in + the debug output and use the maximum value supported instead. + +While the original intent of this function was for load balancing, it +is generic enough to be used for other purposes as well. + +.. warning:: + The random number must not be assumed to be crypto-grade. + +.. versionadded:: 8.12.0 + + +Example +======= + +In the following example a random number between 0 and 123456 is generated. + +.. code-block:: none + + random(123456) + diff --git a/source/rainerscript/functions/rs-re_extract.rst b/source/rainerscript/functions/rs-re_extract.rst new file mode 100644 index 0000000..cd37910 --- /dev/null +++ b/source/rainerscript/functions/rs-re_extract.rst @@ -0,0 +1,38 @@ +************ +re_extract() +************ + +Purpose +======= + +re_extract(expr, re, match, submatch, no-found) + +Extracts data from a string (property) via a regular expression match. +POSIX ERE regular expressions are used. The variable "match" contains +the number of the match to use. This permits to pick up more than the +first expression match. Submatch is the submatch to match (max 50 supported). +The "no-found" parameter specifies which string is to be returned in case +when the regular expression is not found. Note that match and +submatch start with zero. It currently is not possible to extract +more than one submatch with a single call. + +This function performs case-sensitive matching. Use the otherwise-equivalent +:doc:`re_extract_i <rs-re_extract_i>` function to perform case-insensitive +matches. + +.. note:: + + Functions using regular expressions tend to be slow and other options + may be faster. + + +Example +======= + +In the following example the msg object is checked for the regex string. +Only the first match is used and if no match was found an empty string is returned. + +.. code-block:: none + + re_extract($msg,'(5[1-5][0-9]{14})',0,1,"") + diff --git a/source/rainerscript/functions/rs-re_extract_i.rst b/source/rainerscript/functions/rs-re_extract_i.rst new file mode 100644 index 0000000..1eb96c2 --- /dev/null +++ b/source/rainerscript/functions/rs-re_extract_i.rst @@ -0,0 +1,12 @@ +************** +re_extract_i() +************** + +Purpose +======= + +re_extract_i(expr, re, match, submatch, no-found) + +This function is equivalent to `re_extract()` except that +it performs case-insensitive matches. See :doc:`re_extract <rs-re_extract>` +for further details. diff --git a/source/rainerscript/functions/rs-re_match.rst b/source/rainerscript/functions/rs-re_match.rst new file mode 100644 index 0000000..f7036bd --- /dev/null +++ b/source/rainerscript/functions/rs-re_match.rst @@ -0,0 +1,28 @@ +********** +re_match() +********** + +Purpose +======= + +re_match(expr, re) + +Returns 1, if expr matches re, 0 otherwise. Uses POSIX ERE. In contrast to +`re_match_i()` the matching is case-sensitive. + +.. note:: + + Functions using regular expressions tend to be slow and other options + may be faster. + + +Example +======= + +In the following example it is checked if the msg object matches the regex string. + +.. code-block:: none + + re_match($msg,'(5[1-5][0-9]{14})') + + diff --git a/source/rainerscript/functions/rs-re_match_i.rst b/source/rainerscript/functions/rs-re_match_i.rst new file mode 100644 index 0000000..60d8cd2 --- /dev/null +++ b/source/rainerscript/functions/rs-re_match_i.rst @@ -0,0 +1,29 @@ +************ +re_match_i() +************ + +Purpose +======= + +re_match_i(expr, re) + +Returns 1, if expr matches re, 0 otherwise. Uses POSIX ERE. In contrast to +`re_match()` the matching is case-insensitive. + +.. note:: + + Functions using regular expressions tend to be slow and other options + may be faster. + + +Example +======= + +In the following example it is checked if the msg object matches the regex string. + +.. code-block:: none + + re_match($msg,'TesT') + +It matches it the message "Test", in any case ("TEST", "tEst", ...) +is contained in msg property. diff --git a/source/rainerscript/functions/rs-replace.rst b/source/rainerscript/functions/rs-replace.rst new file mode 100644 index 0000000..2d98e75 --- /dev/null +++ b/source/rainerscript/functions/rs-replace.rst @@ -0,0 +1,26 @@ +********* +replace() +********* + +Purpose +======= + +replace(str, substr_to_replace, replace_with) + +Returns new string with all instances of substr_to_replace replaced +by replace_with. + + +Example +======= + +.. code-block:: none + + replace("foo bar baz", " b", ", B") + +produces + +.. code-block:: none + + "foo, Bar, Baz" + diff --git a/source/rainerscript/functions/rs-script_error.rst b/source/rainerscript/functions/rs-script_error.rst new file mode 100644 index 0000000..88c5bff --- /dev/null +++ b/source/rainerscript/functions/rs-script_error.rst @@ -0,0 +1,30 @@ +************** +script_error() +************** + +Purpose +======= + +script_error() + +Returns the error state of functions that support it. C-Developers note that this +is similar to ``errno`` under Linux. The error state corresponds to the function +immediately called before. The next function call overrides it. + +Right now, the value 0 means that that the previous functions succeeded, any other +value that it failed. In the future, we may have more fine-grain error codes. + +Function descriptions mention if a function supports error state information. If not, +the function call will always set ``script_error()`` to 0. + + +Example +======= + +The following example shows that script_error() only needs to be called and does +not need any parameters. + +.. code-block:: none + + script_error() + diff --git a/source/rainerscript/functions/rs-strlen.rst b/source/rainerscript/functions/rs-strlen.rst new file mode 100644 index 0000000..569a955 --- /dev/null +++ b/source/rainerscript/functions/rs-strlen.rst @@ -0,0 +1,22 @@ +******** +strlen() +******** + +Purpose +======= + +strlen(str) + +Returns the length of the provided string. + + +Examples +======== + +In the following example returns the length of the message string. + +.. code-block:: none + + strlen($msg) + + diff --git a/source/rainerscript/functions/rs-substring.rst b/source/rainerscript/functions/rs-substring.rst new file mode 100644 index 0000000..b295abf --- /dev/null +++ b/source/rainerscript/functions/rs-substring.rst @@ -0,0 +1,75 @@ +*********** +substring() +*********** + +Purpose +======= + +`substring(str, start, subStringLength)` + +Creates a substring from `str`. The substring begins at `start` and is +at most `subStringLength` characters long. If `start` is higher than the +length of the string, the result will always be an empty string. The +same is the case if `subStringLength` is zero. + +.. note:: + + The first character of the string has the value 0. So if you set + start to '2', the substring will start with the third character. + +If you want to drop only some characters at the beginning of the string, +select an overlay large `subStringLength`, e.g. 1000000. As the length is +then always larger as any normal string, the function will return all but +the first part of the string. + +In order to aid removing `n` characters from the end of the string, you +can specify a **negative** `subStringLength`. This will result in an +actual `subStringLength` that is the string-length minus the specified +`subStringLength`. This is actually a shortcut for + +.. code-block:: none + + subStringLength = strlen(str) - n + +Specifying a negative `subStringLength` does not only offer a more +compact expression, it also has some performance benefits over the +use of `strlen()`. + + +Examples +======== + +In the following example a substring from the string is created, +starting with the 4th character and being 5 characters long. + +.. code-block:: none + + substring("123456789", 3, 5) + + +This example will result in the substring "45678". + +In the following example the first and the last character +of the string will be removed. + +.. code-block:: none + + substring("123456789", 1, -2) + + +This example will result in the substring "2345678". + +Note the `-2` value for `subStringLength`. As the extraction skips the +first character, you only need a substring of `strlen("123456789") - 2` +characters, as you want to remove the last character as well. In other +words: if you want to remove the first **and** last character from the +string, the substring in question is **two** characters shorter than the +original string! We know this is a bit counter-intuitive, this we +wanted to make you aware. + +As another example let us assume you want to drop the first two and the +last character. In this case you would use `subStringLength` of three: + +.. code-block:: none + + substring("123456789", 2, -3) diff --git a/source/rainerscript/functions/rs-tolower.rst b/source/rainerscript/functions/rs-tolower.rst new file mode 100644 index 0000000..904770e --- /dev/null +++ b/source/rainerscript/functions/rs-tolower.rst @@ -0,0 +1,22 @@ +********* +tolower() +********* + +Purpose +======= + +tolower(str) + +Converts the provided string into lowercase. + + +Example +======= + +The following example shows the syslogtag being converted to lower case. + +.. code-block:: none + + tolower($syslogtag) + + diff --git a/source/rainerscript/functions/rs-trim.rst b/source/rainerscript/functions/rs-trim.rst new file mode 100644 index 0000000..835b17b --- /dev/null +++ b/source/rainerscript/functions/rs-trim.rst @@ -0,0 +1,59 @@ +***************** +ltrim() / rtrim() +***************** + +Purpose +======= + +ltrim +----- + +ltrim(str) + +Removes any spaces at the start of a given string. Input is a string, output +is the same string starting with the first non-space character. + + +rtrim +----- + +rtrim(str) + +Removes any spaces at the end of a given string. Input is a string, output +is the same string ending with the last non-space character. + + +Example +======= + +ltrim +----- + +In the following example the spaces at the beginning of the string are removed. + +.. code-block:: none + + ltrim(" str ") + +produces + +.. code-block:: none + + "str " + + +rtrim +----- + +In the following example the spaces at the end of the string are removed. + +.. code-block:: none + + rtrim(" str ") + +produces + +.. code-block:: none + + " str" + diff --git a/source/rainerscript/functions/rs-wrap.rst b/source/rainerscript/functions/rs-wrap.rst new file mode 100644 index 0000000..d1dbd1c --- /dev/null +++ b/source/rainerscript/functions/rs-wrap.rst @@ -0,0 +1,51 @@ +****** +wrap() +****** + +Purpose +======= + +wrap(str, wrapper_str) +wrap(str, wrapper_str, escaper_str) + +Returns the str wrapped with wrapper_str. +In the second syntax additionally, any instances of wrapper_str appearing +in str would be replaced by the escaper_str. + + +Example +======= + +Syntax 1 +-------- + +The following example shows how the string is wrapped into +the wrapper_string. + +.. code-block:: none + + wrap("foo bar", "##") + +produces + +.. code-block:: none + + "##foo bar##" + + +Syntax 2 +-------- + +This example shows how parts of the string are being replaced. + +.. code-block:: none + + wrap("foo'bar", "'", "_") + +produces + +.. code-block:: none + + "'foo_bar'" + + diff --git a/source/rainerscript/global.rst b/source/rainerscript/global.rst new file mode 100644 index 0000000..a55f54a --- /dev/null +++ b/source/rainerscript/global.rst @@ -0,0 +1,811 @@ +global() configuration object +============================= + +The global configuration object permits to set global parameters. Note +that each parameter can only be set once and cannot be re-set +thereafter. If a parameter is set multiple times, the behaviour is +unpredictable. As with other configuration objects, parameters for this +object are case-insensitive. + +The following parameters can be set: + +- **action.reportSuspension** - binary, default "on", v7.5.8+ + + If enabled ("on") action will log message under `*syslog.\**` when an + action suspends or resumes itself. This usually happens when there are + problems connecting to backend systems. If disabled ("off"), these + messages are not generated. These messages can be useful in detecting + problems with backend systems. Most importantly, frequent suspension + and resumption points to a problem area. + +- **action.reportSuspensionContinuation** - binary, default "off", v7.6.1+, v8.2.0+ + + If enabled ("on") the action will not only report the first suspension but + each time the suspension is prolonged. Otherwise, the follow-up messages + are not logged. If this setting is set to "on", action.reportSuspension is + also automatically turned "on". + +- **workDirectory** + + Sets the directory that rsyslog uses for work files, e.g. imfile state + or queue spool files. + +- **umask** available 8.26.0+ + + Sets the rsyslogd process' umask. If not specified, the system-provided default + is used. The value given must always be a 4-digit octal number, with the initial + digit being zero. + +- **dropMsgsWithMaliciousDNSPtrRecords** + +- **localHostname** + Permits to overwrite the local host hostname. + +- **preserveFQDN** +- **defaultNetstreamDriverCAFile** + + For `TLS syslog <http://www.rsyslog.com/doc/rsyslog_secure_tls.html>`_, + the CA certificate that can verify the machine keys and certs (see below) + +- **defaultNetstreamDriverCRLFile** + + For `TLS syslog <http://www.rsyslog.com/doc/rsyslog_secure_tls.html>`_, + the CRL File contains a List contains a list of revoked certrificates. + +- **defaultNetstreamDriverKeyFile** + + Machine private key + +- **defaultNetstreamDriverCertFile** + + Machine public key (certificate) + +- **debug.gnutls** (0-10; default:0) + + Any other parameter than 0 enables the debug messages of GnuTLS. The + amount of messages given depends on the height of the parameter, 0 + being nothing and 10 being very much. Caution! higher parameters may + give out way more information than needed. We advise you to first use + small parameters to prevent that from happening. + **This parameter only has an effect if general debugging is enabled.** + +- **netstreamDriverCaExtraFiles** + + This directive allows to configure multiple additional extra CA files. + This is intended for SSL certificate chains to work appropriately, + as the different CA files in the chain need to be specified. + It must be remarked that this parameter only works with the OpenSSL driver. + +- **processInternalMessages** binary (on/off) + + This tells rsyslog if it shall process internal messages itself. The + default mode of operations ("off") makes rsyslog send messages to the + system log sink (and if it is the only instance, receive them back from there). + This also works with systemd journal and will make rsyslog messages show up in the + systemd status control information. + + If this (instance) of rsyslog is not the main instance and there is another + main logging system, rsyslog internal messages will be inserted into + the main instance's syslog stream. In this case, setting to ("on") will + let you receive the internal messages in the instance they originate from. + + Note that earlier versions of rsyslog worked the opposite way. More + information about the change can be found in `rsyslog-error-reporting-improved <http://www.rsyslog.com/rsyslog-error-reporting-improved>`_. + + + +- **stdlog.channelspec** + + Permits to set the liblogging-stdlog channel specifier string. This + in turn permits to send rsyslog log messages to a destination different + from the system default. Note that this parameter has only effect if + *processInternalMessages* is set to "off". Otherwise it is silently + ignored. + +- **shutdown.enable.ctlc** + + If set to "on", rsyslogd can be terminated by pressing ctl-c. This is + most useful for containers. If set to "off" (the default), this is not + possible. + +- **defaultNetstreamDriver** + + Set it to "ossl" or "gtls" to enable TLS. + This `guide <http://www.rsyslog.com/doc/rsyslog_secure_tls.html>`_ + shows how to use TLS. + +- **maxMessageSize** + + Configures the maximum message size allowed for all inputs. Default is 8K. + Anything above the maximum size will be truncated. + + Note: some modules provide separate parameters that allow overriding this + setting (e.g., :doc:`imrelp's MaxDataSize parameter <../../configuration/modules/imrelp>`). + +.. _global_janitorInterval: + +- **janitor.interval** [minutes], available 8.3.3+ + + Sets the interval at which the + :doc:`janitor process <../concepts/janitor>` + runs. + +- **debug.onShutdown** available 7.5.8+ + + If enabled ("on"), rsyslog will log debug messages when a system + shutdown is requested. This can be used to track issues that happen + only during shutdown. During normal operations, system performance is + NOT affected. + Note that for this option to be useful, the debug.logFile parameter + must also be set (or the respective environment variable). + +- **debug.logFile** available 7.5.8+ + + This is used to specify the debug log file name. It is used for all + debug output. Please note that the RSYSLOG\_DEBUGLOG environment + variable always **overrides** the value of debug.logFile. + +- **net.ipprotocol** available 8.6.0+ + + This permits to instruct rsyslog to use IPv4 or IPv6 only. Possible + values are "unspecified", in which case both protocols are used, + "ipv4-only", and "ipv6-only", which restrict usage to the specified + protocol. The default is "unspecified". + + Note: this replaces the former *-4* and *-6* rsyslogd command line + options. + +- **net.aclAddHostnameOnFail** available 8.6.0+ + + If "on", during ACL processing, hostnames are resolved to IP addresses for + performance reasons. If DNS fails during that process, the hostname + is added as wildcard text, which results in proper, but somewhat + slower operation once DNS is up again. + + The default is "off". + +- **net.aclResolveHostname** available 8.6.0+ + + If "off", do not resolve hostnames to IP addresses during ACL processing. + + The default is "on". + +- **net.enableDNS** [on/off] available 8.6.0+ + + **Default:** on + + Can be used to turn DNS name resolution on or off. + +- **net.permitACLWarning** [on/off] available 8.6.0+ + + **Default:** on + + If "off", suppress warnings issued when messages are received + from non-authorized machines (those, that are in no AllowedSender list). + +- **parser.parseHostnameAndTag** [on/off] available 8.6.0+ + + **Default:** on + + This controls whether the parsers try to parse HOSTNAME and TAG fields + from messages. The default is "on", in which case parsing occurs. If + set to "off", the fields are not parsed. Note that this usually is + **not** what you want to have. + + It is highly suggested to change this setting to "off" only if you + know exactly why you are doing this. + +- **parser.permitSlashInProgramName** [on/off] available 8.25.0+ + + **Default:** off + + This controls whether slashes in the "programname" property + (the static part of the tag) are permitted or not. By default + this is not permitted, but some Linux tools (including most + importantly the journal) store slashes as part of the program + name inside the syslogtag. In those cases, the ``programname`` + is truncated at the first slash. + + In other words, if the setting is off, a value of ``app/foo[1234]`` + in the tag will result in a programname of ``app``, and if an + application stores an absolute path name like ``/app/foo[1234]``, + the ``programname`` property will be empty (""). + If set to ``on``, a syslogtag of ``/app/foo[1234]`` will result + in a ``programname`` value of ``/app/foo`` and a syslogtag of + ``app/foo[1234]`` will result in a ``programname`` value of + ``app/foo``. + +- **parser.escapeControlCharacterTab** [on/off] available since 8.7.0 + + **Default:** on + + If set to "off", the TAB control character (US-ASCII HT) will not be + escaped. If set to "on", it will be escaped to the sequence "#011". + Note that escaping is the traditional behavior and existing scripts + may get into trouble if this is changed to "off". + +- **parser.controlCharacterEscapePrefix** [char] + + **Default:** '#' + + This option specifies the prefix character to be used for control + character escaping (see option + *parser.escapeControlCharactersOnReceive*). + +- **parser.escape8BitCharactersOnReceive** [on/off] + + **Default:** off + + This parameter instructs rsyslogd to replace non US-ASCII characters + (those that have the 8th bit set) during reception of the message. + This may be useful for some systems. Please note that this escaping + breaks Unicode and many other encodings. Most importantly, it can be + assumed that Asian and European characters will be rendered hardly + readable by this settings. However, it may still be useful when the + logs themselves are primarily in English and only occasionally contain + local script. If this option is turned on, all control-characters are + converted to a 3-digit octal number and be prefixed with the + *parser.controlCharacterEscapePrefix* character (being '#' by default). + + **Warning:** + + - turning on this option most probably destroys non-western character + sets (like Japanese, Chinese and Korean) as well as European + character sets. + - turning on this option destroys digital signatures if such exists + inside the message + - if turned on, the drop-cc, space-cc and escape-cc `property + replacer <property_replacer.html>`_ options do not work as expected + because control characters are already removed upon message + reception. If you intend to use these property replacer options, you + must turn off *parser.escape8BitCharactersOnReceive*. + +- **parser.escapeControlCharactersOnReceive** [on/off] + + **Default:** on + + This parameter instructs rsyslogd to replace control characters during + reception of the message. The intent is to provide a way to stop + non-printable messages from entering the syslog system as whole. If this + option is turned on, all control-characters are converted to a 3-digit + octal number and be prefixed with the *parser.controlCharacterEscapePrefix* + character (being '#' by default). For example, if the BEL character + (ctrl-g) is included in the message, it would be converted to '#007'. + To be compatible to sysklogd, this option must be turned on. + + **Warning:** + + - turning on this option most probably destroys non-western character + sets (like Japanese, Chinese and Korean) + - turning on this option destroys digital signatures if such exists + inside the message + - if turned on, the drop-cc, space-cc and escape-cc `property + replacer <property_replacer.html>`_ options do not work as expected + because control characters are already removed upon message + reception. If you intend to use these property replacer options, you + must turn off *parser.escapeControlCharactersOnReceive*. + + +- **senders.keepTrack** [on/off] available 8.17.0+ + + **Default:** off + + If turned on, rsyslog keeps track of known senders and also reports + statistical data for them via the impstats mechanism. + + A list of active senders is kept. When a new sender is detected, an + informational message is emitted. Senders are purged from the list + only after a timeout (see *senders.timeoutAfter* parameter). Note + that we do not intentionally remove a sender when a connection is + closed. The whole point of this sender-tracking is to have the ability + to provide longer-duration data. As such, we would not like to drop + information just because the sender has disconnected for a short period + of time (e.g. for a reboot). + + Senders are tracked by their hostname (taken at connection establishment). + + Note: currently only imptcp and imtcp support sender tracking. + +- **senders.timeoutAfter** [seconds] available 8.17.0+ + + **Default:** 12 hours (12*60*60 seconds) + + Specifies after which period a sender is considered to "have gone + away". For each sender, rsyslog keeps track of the time it least + received messages from it. When it has not received a message during + that interval, rsyslog considers the sender to be no longer present. + It will then a) emit a warning message (if configured) and b) purge + it from the active senders list. As such, the sender will no longer + be reported in impstats data once it has timed out. + +- **senders.reportGoneAway** [on/off] available 8.17.0+ + + **Default:** off + + Emit a warning message when now data has been received from a sender + within the *senders.timeoutAfter* interval. + +- **senders.reportNew** [on/off] available 8.17.0+ + + **Default:** off + + If sender tracking is active, report a sender that is not yet inside + the cache. Note that this means that senders which have been timed out + due to prolonged inactivity are also reported once they connect again. + +- **debug.unloadModules** [on/off] available 8.17.0+ + + **Default:** on + + This is primarily a debug setting. If set to "off", rsyslog will never + unload any modules (including plugins). This usually causes no operational + problems, but may in extreme cases. The core benefit of this setting is + that it makes valgrind stack traces readable. In previous versions, the + same functionality was only available via a special build option. + +- **debug.files** [ARRAY of filenames] available 8.29.0+ + + **Default:** none + + This can be used to configure rsyslog to only show debug-output generated in + certain files. If the option is set, but no filename is given, the + debug-output will behave as if the option is turned off. + + Do note however that due to the way the configuration works, this might not + effect the first few debug-outputs, while rsyslog is reading in the configuration. + For optimal results we recommend to put this parameter at the very start of + your configuration to minimize unwanted output. + + See debug.whitelist for more information. + +- **debug.whitelist** [on/off] available 8.29.0+ + + **Default:** on + + This parameter is an assisting parameter of debug.files. If debug.files + is used in the configuration, debug.whitelist is a switch for the files named + to be either white- or blacklisted from displaying debug-output. If it is set to + on, the listed files will generate debug-output, but no other files will. + The reverse principle applies if the parameter is set to off. + + See debug.files for more information. + +- **environment** [ARRAY of environment variable=value strings] available 8.23.0+ + + **Default:** none + + This permits to set environment variables via rsyslog.conf. The prime + motivation for having this is that for many libraries, defaults can be + set via environment variables, **but** setting them via operating system + service startup files is cumbersome and different on different platforms. + So the *environment* parameter provides a handy way to set those + variables. + + A common example is to set the *http_proxy* variable, e.g. for use with + KSI signing or ElasticSearch. This can be done as follows:: + + global(environment="http_proxy=http://myproxy.example.net") + + Note that an environment variable set this way must contain an equal sign, + and the variable name must not be longer than 127 characters. + + It is possible to set multiple environment variables in a single + global statement. This is done in regular array syntax as follows:: + + global(environment=["http_proxy=http://myproxy.example.net", + "another_one=this string is=ok!"] + ) + + As usual, whitespace is irrelevant in regard to parameter placing. So + the above sample could also have been written on a single line. + +- **internalmsg.ratelimit.interval** [positive integer] available 8.29.0+ + + **Default:** 5 + + Specifies the interval in seconds onto which rate-limiting is to be + applied to internal messages generated by rsyslog(i.e. error messages). + If more than internalmsg.ratelimit.burst messages are read during + that interval, further messages up to the end of the interval are + discarded. + +- **internalmsg.ratelimit.burst** [positive integer] available 8.29.0+ + + **Default:** 500 + + Specifies the maximum number of internal messages that can be emitted within + the ratelimit.interval interval. For further information, see + description there. + + + **Caution:** Environment variables are set immediately when the + corresponding statement is encountered. Likewise, modules are loaded when + the module load statement is encountered. This may create **sequence + dependencies** inside rsyslog.conf. To avoid this, it is highly suggested + that environment variables are set **right at the top of rsyslog.conf**. + Also, rsyslog-related environment variables may not apply even when set + right at the top. It is safest to still set them in operating system + start files. Note that rsyslog environment variables are usually intended + only for developers so there should hardly be a need to set them for a + regular user. Also, many settings (e.g. debug) are also available as + configuration objects. + +- **internalmsg.severity** [syslog severity value] available 8.1905.0+ + + **Default:** info + + This permits to limit which internal messages are emitted by rsyslog. This + is especially useful if internal messages are reported to systemd journal, + which is the default on journal systems. In that case there is no other + ability to filter out messages before they are logged by the journal. + + While any syslog severity value can be used, the most useful ones are + + * `error`, to see only error messages but ignore anything else + * `warn`, to also see warning messages (highly recommended) + * `info`, to also see informational messages like events generated + by DA queues status checks. This is the default as the informational + messages often provide valuable information. + * `debug`, to see all messages, including only those interesting for + debugging. While this is still considerably lower volume than a + rsyslog developer debug log, this can be quite verbose. Selecting + `debug` without hard need thus is **not** recommended. + + We expect that users are most often interested in limiting verboseness + to warning messages. This can be done e.g. via:: + + global(internalmsg.severity="warn") + +- **errorMessagesToStderr.maxNumber** [positive integer] available 8.30.0+ + + **Default:** unlimited + + This permits to put a hard limit on the number of messages that can + go to stderr. If for nothing else, this capability is helpful for the + testbench. It permits to reduce spamming the test log while still + providing the ability to see initial error messages. Might also be + useful for some practical deployments. + +- **variables.caseSensitive** [boolean (on/off)] available 8.30.0+ + + **Default:** off + + This permits to make variables case-sensitive, what might be required + for some exotic input data where case is the only difference in + field names. Note that in rsyslog versions prior to 8.30, the default was + "on", which very often led to user confusion. There normally should be no + need to switch it back to "on", except for the case to be mentioned. + This is also the reason why we switched the default. + +- **internal.developeronly.options** + + This is NOT to be used by end users. It provides rsyslog developers the + ability to do some (possibly strange) things inside rsyslog, e.g. for + testing. This parameter should never be set, except if instructed by + a developer. If it is set, rsyslog may misbehave, segfault, or cause + other strange things. Note that option values are not guaranteed to + stay the same between releases, so do not be "smart" and apply settings + that you found via a web search. + + Once again: **users must NOT set this parameter!** + +- **oversizemsg.errorfile** [file name] available 8.35.0+ + + This parameter is used to specify the name of the oversize message log file. + Here messages that are longer than maxMessageSize will be gathered. + +- **oversizemsg.input.mode** [mode] available 8.35.0+ + + With this parameter the behavior for oversized messages can be specified. + Available modes are: + + - truncate: Oversized messages will be truncated. + - split: Oversized messages will be split and the rest of the message will + be sent in another message. + - accept: Oversized messages will still be accepted. + +- **oversizemsg.report** [boolean (on/off)] available 8.35.0+ + + This parameter specifies if an error shall be reported when an oversized + message is seen. The default is "on". + +- **abortOnUncleanConfig** [boolean (on/off)] available 8.37.0+ + + This parameter permits to prevent rsyslog from running when the + configuration file is not clean. "Not Clean" means there are errors or + some other annoyances that rsyslogd reports on startup. This is a + user-requested feature to have a strict startup mode. Note that with the + current code base it is not always possible to differentiate between an + real error and a warning-like condition. As such, the startup will also + prevented if warnings are present. I consider this a good thing in being + "strict", but I admit there also currently is no other way of doing it. + +- **abortOnFailedQueueStartup** [boolean (on/off)] available 8.2210.0+ + + This parameter is similiar to *abortOnUncleanConfig* but makes rsyslog + abort when there are any problems with queue startup. This is usually + caused by disk queue settings or disk queue file corruption. Normally, + rsyslog ignores disk queue definitions in this case and switches the + queue to emergency mode, which permits in-memory operations. This is + desired by the fast majority of users, because it permits rsyslog to + remain operational and process all remaining actions as well as handle + actions associated with the failed queue decently. + When this setting is "on", rsyslog aborts immediately when a queue + problem is detected during startup. If you use this mode, ensure that + your startup scripts monitor for these type of errors and handle them + appropriately. + In our opinion, it is much safer to let rsyslog start and monitor queue + error messages. + + The **default** for this setting is "off" + +- **inputs.timeout.shutdown** [numeric, ms] available 8.37.0+ + + This parameter specifies how long input modules are given time to terminate + when rsyslog is shutdown. The default is 1000ms (1 second). If the input + requires longer to terminate, it will be cancelled. This is necessary if + the input is inside a lengthy operation, but should generally be tried to + avoid. On busy systems it may make sense to increase that timeout. This + especially seems to be the case with containers. + +- **default.action.queue.timeoutshutdown** [numeric] available 8.1901.0+ +- **default.action.queue.timeoutactioncompletion** [numeric] available 8.1901.0+ +- **default.action.queue.timeoutenqueue** [numeric] available 8.1901.0+ +- **default.action.queue.timeoutworkerthreadshutdown** [numeric] available 8.1901.0+ + + These parameters set global queue defaults for the respective queue settings. + +- **reverselookup.cache.ttl.default** [numeric, seconds] available 8.1904.0+ + + Rsyslog includes a cache for ip-address-to-hostname lookups. This is most + useful for inputs without a connection. imudp is the prime example. + This settings permits to specify after which period (in seconds) an + entry expires. Upon expiration the entry will be discarded and re-queried. + The **default** value is 24 hours. + To never cache entries, set the parameter to 0, which will make cache + entries expire immediately. Note that especially with imudp this can + cause huge performance degradation and potentially also message loss. + + Note: for many years rsyslog did **not** timeout cache entries at all. This + only occasionally caused issues. We assume that the once-every-24-hrs + default value is a very good compromise between performance and + keeping reverse lookup information current. + +- **reverselookup.cache.ttl.enable** [boolean (on/off)] available 8.1904.0+ + + This configures whether rsyslog expires DNS cache entries (setting "on") or + not (setting "off", the default). If configured to "off", + *reverselookup.cache.default.ttl* is not in effect. Note that this is the + **default**. + +- **security.abortOnIDResolutionFail** [boolean (on/off)], default "on", available 8.2002.0+ + + This setting controls if rsyslog should error-terminate when an security ID cannot + be resolved during config file processing at startup. If set to "on" and + a name ID lookup fails (for user and group names) rsyslog does not start but + terminate with an error message. This is necessary as a security + measure, as otherwise the wrong permissions can be assigned or privileges + are not dropped. This setting is applied whereever security IDs are resolved, + e.g. when dropping privileges or assigning file permissions or owners. + + The setting should be at the top of the configuration parameters to make sure its + behavior is correctly applied on all other configuration parameters. + + **CHANGE OF BEHAVIOR** + + The default for this parameter is "on". In versions prior to 8.2002.0, the default + was "off" (by virtue of this parameter not existing). As such, existing + configurations may now error out. + + We have decided to accept this change of behavior because of the potential + security implications. + +- **operatingStateFile** [string, filename], default unset, available 8.39.0+ + + The operatingStateFile, as the name says, provides information about rsyslog + operating state. It can be useful for troubleshooting. + + If this parameter is not set, an operating state file will not be written. If + it is set, the file will be written **and** used to detect unclean shutdown. + Upon startup, rsyslog checks if the last recorded line contains the "clean + shutdown notification". If so, the file is deleted and re-written with new + operating state. If the notification cannot be found, rsyslog assumes unclean + shutdown and complains about this state. In this case the operating state file + is renamed to "<configured-name>.previous" and a new file is started under the + configured name for the current run. This permits the administrator to check the + previous operating state file for helpful information on why the system shut + down unclean. + +- **reportChildProcessExits** [none|errors|all], default "errors", available + 8.1901.0+ + + Tells rsyslog whether and when to log a message (under *syslog.\**) when a + child process terminates. The available modes are: + + - none: Do not report any child process termination. + - errors: Only report the termination of child processes that have exited with + a non-zero exit code, or that have been terminated by a signal. + - all: Report all child process terminations. + + The logged message will be one of the following: + + - "program 'x' (pid n) exited with status s" (with "info" severity if the + status is zero, and "warning" severity otherwise) + - "program 'x' (pid n) terminated by signal s" (with "warning" severity) + + In some cases, the program name is not included in the message (but only the PID). + + Normally, if a child process terminates prematurely for some reason, rsyslog will + also report some specific error message the next time it interacts with the process + (for example, in the case of a process started by omprog, if omprog cannot send a + message to the process because the pipe is broken, it will report an error + indicating this). This specific error message (if any) is not affected by this + global setting. + + +- **default.ruleset.queue.timeoutshutdown** +- **default.ruleset.queue.timeoutactioncompletion** +- **default.ruleset.queue.timeoutenqueue** +- **default.ruleset.queue.timeoutworkerthreadshutdown** + + Sets default parameters for ruleset queues. See queue doc for the meaning of + the individual settings. + + +- **default.action.queue.timeoutshutdown** +- **default.action.queue.timeoutactioncompletion** +- **default.action.queue.timeoutenqueue** +- **default.action.queue.timeoutworkerthreadshutdown** + + Sets default parameters for action queues. See queue doc for the meaning of + the individual settings. + + +- **shutdown.queue.doublesize** + + This setting (default "off") permits to temporarily increase the maximum queue + size during shutdown processing. This is useful when rsyslog needs to re-enqueue + some messages at shutdown *and* the queue is already full. Note that the need to + re-enqueue messages stems back to some failed operations. Note that the maximum + permitted queue size is doubled, as this ensures in all cases that re-enqueuing + can be completed. Note also that the increase of the max size is temporary during + shutdown and also does not requiere any more storage. Except, of course, for + re-enqueued message. + + The situation addressed by this setting is unlikely to happen, but it could happen. + To enable the funtionality, set it to "on". + +- **parser.supportCompressionExtension** [boolean (on/off)] available 8.2106.0+ + + This parameter permits to disable rsyslog's single-message-compression extension on + reception ("off"). The default is to keep it activated ("on"). + + The single-message-compression extension permits senders to zip-compress single + syslog messages. Such messages start with the letter "z" instead of the usual + syslog PRI value. For well-formed syslog messages, the extension works as designed. + However, some users transport non-syslog data via rsyslog, and such messages may + validly start with "z" for non-compressed data. To support such non-standard + cases, this option can be used to globally disable support for compression on + all inputs. + +privdrop.group.name +^^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "", "no", "``$PrivDropToGroup``" + +.. versionadded:: 8.2110.0 + +Name of the group rsyslog should run under after startup. Please +note that this group is looked up in the system tables. If the lookup +fails, privileges are NOT dropped. Thus it is advisable to use the +less convenient `privdrop.group.id` parameter. Note that all +supplementary groups are removed by default from the process if the +`privdrop.group.keepsupplemental` parameter is not specified. +If the group id can be looked up, but can not be set, +rsyslog aborts. + +Note: See the :doc:`privilege drop documentation<../configuration/droppriv>` +for more details on dropping privileges on startup. + +privdrop.group.id +^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "", "no", "``$PrivDropToGroupID``" + +.. versionadded:: 8.2110.0 + +Numerical user ID of the group rsyslog should run under after startup. +This is more reliable than the `privdrop.group.name` parameter, which +relies on presence of the group name in system tables. The change to +the ID will always happen if the ID is valid. + +Note: See the :doc:`privilege drop documentation<../configuration/droppriv>` +for more details on dropping privileges on startup. + +privdrop.user.name +^^^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "", "no", "``$PrivDropToUser``" + +.. versionadded:: 8.2110.0 + + +Name of the user rsyslog should run under after startup. Please note +that this user is looked up in the system tables. If the lookup +fails, privileges are NOT dropped. Thus it is advisable to use the +less convenient `privdrop.user.id` parameter. If the user id can be +looked up, but can not be set, rsyslog aborts. + +Note: See the :doc:`privilege drop documentation<../configuration/droppriv>` +for more details on dropping privileges on startup. + +privdrop.user.id +^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "", "no", "``$PrivDropToUserID``" + +.. versionadded:: 8.2110.0 + +Numerical user ID of the user rsyslog should run under after startup. +This is more reliable than the `privdrop.user.name` parameter, which +relies on presence of the user name in system tables. The change to +the ID will always happen if the ID is valid. + +Note: See the :doc:`privilege drop documentation<../configuration/droppriv>` +for more details on dropping privileges on startup. + +libcapng.default +^^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "on", "no", "none" + +.. versionadded:: 8.2306.0 + +The `libcapng.default` global option defines how rsyslog should behave +in case something went wrong when capabilities were to be dropped. +The default value is "on", in which case rsyslog exits on a libcapng +related error. If set to "off", an error message describing the problem +appears at startup, nothing more. Default value is preserved for backwards +compatibility. + +libcapng.enable +^^^^^^^^^^^^^^^ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "on", "no", "none" + +.. versionadded:: 8.2310.0 + +The `libcapng.enable` global option defines whether rsyslog should +drop capabilities at startup or not. By default, it is set to "on". +Until this point, if the project was compiled with --enable-libcap-ng option, +capabilities were automatically dropped. This is configurable now. diff --git a/source/rainerscript/include.rst b/source/rainerscript/include.rst new file mode 100644 index 0000000..f2a01c2 --- /dev/null +++ b/source/rainerscript/include.rst @@ -0,0 +1,153 @@ +**************************** +The rsyslog include() object +**************************** + +The ``include()`` object is used to include configuration snippets +stored elsewhere into the configuration. + +.. versionadded:: 8.33.0 + +.. note:: + + This configuration option deprecates the older ``$IncludeConfig`` + |FmtObsoleteName| format directive. + +How it Works +============ + +The rsyslog include object is modelled after the usual "include" directive +in programming and script languages (e.g. \#include in C). + +If you are not familiar with this, compare it to copy and paste: whenever +rsyslog finds an include object, in copies the text from that include file +at the exact position where the include is specified and removes the include +text. + +Now remember that rsyslog's configuration statements are depending on the +position inside the configuration. It is important if a statement occurs +before or after another one. This is similar how other configuration files +work and also the same concept that almost all programming and script +languages have. + +If you use includes in rsyslog, you must think about the position at which +the include text is inserted. This is especially important if you use +statements like `stop`. If given at the wrong spot, they will not work as +intended. + +If in doubt, or if you have issues, it probably is best NOT to use includes. +This makes it far more obvious to understand what happens. Once solved, you +may then change back to includes again. + + +Parameters +========== + +.. note:: + + Parameter names are case-insensitive. + +.. warning:: + + Only one of the ``file`` or ``text`` parameters may be specified for each + ``include()`` object. + + +file +---- + +Name of file to be included. May include wildcards, in which case all +matching files are included (in order of file name sort order). + + +text +---- + +Text to be included. This is most useful when using backtick string +constants. + + +mode +---- + +Affects how missing files are to be handled: + +- ``abort-if-missing``, with rsyslog aborting when the file is not present +- ``required`` *(default)*, with rsyslog emitting an error message but otherwise + continuing when the file is not present +- ``optional``, which means non-present files will be skipped without notice + +Examples +======== + +Include a required file +----------------------- + +.. code-block:: none + + include(file="/path/to/include.conf") + +.. note:: + + Unless otherwise specified, files referenced by an ``include()`` object + must be present, otherwise an error will be generated. + + +Include an optional file +------------------------ + +The referenced file will be used if found, otherwise no errors or warnings +will be generated regarding its absence. + +.. code-block:: none + :emphasize-lines: 3 + + include( + file="/path/to/include.conf" + mode="optional" + ) + + +Include multiple files +---------------------- + +.. code-block:: none + + include(file="/etc/rsyslog.d/*.conf") + +.. note:: + + Unless otherwise specified, files referenced by an ``include()`` object + must be present, otherwise an error will be generated. + + +Include an environment variable as configuration +------------------------------------------------ + +.. code-block:: none + + include(text=`echo $ENV_VAR`) + + +Include a file specified via an environment variable +---------------------------------------------------- + +.. code-block:: none + + include(file=`echo $ENV_VAR`) + +.. note:: + + Unless otherwise specified, files referenced by an ``include()`` object + must be present, otherwise an error will be generated. + + +Include an optional file specified via an environment variable +-------------------------------------------------------------- + +.. code-block:: none + :emphasize-lines: 3 + + include( + file=`echo $ENV_VAR` + mode="optional" + ) diff --git a/source/rainerscript/index.rst b/source/rainerscript/index.rst new file mode 100644 index 0000000..4b277f7 --- /dev/null +++ b/source/rainerscript/index.rst @@ -0,0 +1,30 @@ +RainerScript +============ + +**RainerScript is a scripting language specifically designed and +well-suited for processing network events and configuring event +processors.** +It is the prime configuration language used for rsyslog. +Please note that RainerScript may not be abbreviated as rscript, +because that's somebody else's trademark. + +Some limited RainerScript support is available since rsyslog 3.12.0 +(for expression support). In v5, "if .. then" statements are supported. +The first full implementation is available since rsyslog v6. + +.. toctree:: + :maxdepth: 2 + + data_types + expressions + functions/index + control_structures + configuration_objects + constant_strings + variable_property_types + lookup_tables + queue_parameters + rainerscript_call + rainerscript_call_indirect + global + include diff --git a/source/rainerscript/lookup_tables.rst b/source/rainerscript/lookup_tables.rst new file mode 100644 index 0000000..29178d3 --- /dev/null +++ b/source/rainerscript/lookup_tables.rst @@ -0,0 +1,7 @@ +Lookup Tables +============= + +:doc:`Lookup tables <../configuration/lookup_tables>` are a powerful construct to obtain +"class" information based on message content (e.g. to build log file +names for different server types, departments or remote offices). + diff --git a/source/rainerscript/queue_parameters.rst b/source/rainerscript/queue_parameters.rst new file mode 100644 index 0000000..19e850c --- /dev/null +++ b/source/rainerscript/queue_parameters.rst @@ -0,0 +1,607 @@ +************************ +General Queue Parameters +************************ + +=========================== =========================================================================== +**Authors:** `Rainer Gerhards <https://rainer.gerhards.net/>`_ <rgerhards@adiscon.com>; +=========================== =========================================================================== + + +Usage +===== + +Queue parameters can be used together with the following statements: + +- :doc:`action() <../configuration/actions>` +- ruleset() +- main\_queue() + +Queues need to be configured in the action or ruleset it should affect. +If nothing is configured, default values will be used. Thus, the default +ruleset has only the default main queue. Specific Action queues are not +set up by default. + +To fully understand queue parameters and how they interact, be sure to +read the :doc:`queues <../concepts/queues>` documentation. + + +Configuration Parameters +======================== + +.. note:: + + As with other configuration objects, parameters for this + object are case-insensitive. + + +queue.filename +-------------- + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "``$ActionQueueFileName``" + +File name to be used for the queue files. If specified, this parameter +enables disk-assisted queue functionality. If *not* specified, +the queue will operate without saving the queue to disk, either +during its operation or when shut down. See the separate +``queue.saveonshutdown`` parameter to configure that option. +Please note that this is actually just the file name. A directory +can NOT be specified in this parameter. If the files shall be +created in a specific directory, specify ``queue.spoolDirectory`` +for this. The filename is used to build to complete path for queue +files. + + +queue.spoolDirectory +-------------------- + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "none", "no", "none" + +This is the directory into which queue files will be stored. Note +that the directory must exist, it is NOT automatically created by +rsyslog. If no spoolDirectory is specified, the work directory is +used. + + +queue.size +---------- + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "1000/50000", "no", "``$ActionQueueSize``" + +This is the maximum size of the queue in number of messages. Note +that setting the queue size to very small values (roughly below 100 +messages) is not supported and can lead to unpredictable results. +For more information on the current status of this restriction see +the `rsyslog FAQ: "lower bound for queue +sizes" <http://www.rsyslog.com/lower-bound-for-queue-sizes/>`_. + +The default depends on queue type and rsyslog version, if you need +a specific value, please specify it. Otherwise rsyslog selects what +it considers appropriate for the version in question. In rsyslog +rsyslog 8.30.0, for example, ruleset queues have a default size +of 50000 and action queues which are configured to be non-direct +have a size of 1000. + + +queue.dequeueBatchSize +---------------------- + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "128/1024", "no", "``$ActionQueueDequeueBatchSize``" + +Specifies the maximum batch size for dequeue operations. This setting affects performance. +As a rule of thumb, larger batch sizes (up to a environment-induced upper limit) +provide better performance. For the average system, there usually should be no need +to adjust batch sizes as the defaults are sufficient. The default for ruleset queues +is 1024 and for action queues 128. + +Note that this only specifies the **maximum** batch size. Batches will be slower if +rsyslog does not have as many messages inside the queue at time of dequeuing it. +If you want to set a minimum Batch size as well, you can use `queue.minDequeueBatchSize`. + + +queue.minDequeueBatchSize +------------------------- + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "0", "no", "none" + +Specifies the **minimum** batch size for dequeue operations. This setting is especially +useful with outputs like ElasticSearch or ClickHouse, where you want to limit the +number of HTTP requests. With this setting, the queue engine waits up to +`queue.minDequeueBatchSize.timeout` milliseconds when there are fewer messages +currently queued. Note that the minimum batch size cannot be larger than the +configured maximum batch size. If so, it is automatically adjusted to +match the maximum. So if in doubt, you need to specify both parameters. + + +queue.minDequeueBatchSize.timeout +--------------------------------- + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "1000", "no", "none" + +This parameter is only meaningful if use together with `queue.minDequeueBatchSize`, +otherwise it is ignored. It specifies the amount of time (in milliseconds) rsyslogs +waits for new +messages so that the minimum batch size can be reached. After this period, the +batch is processed, *even if it is below minimum size*. This capability exists to +prevent having messages stalled in an incomplete batch just because no new +messages arrive. We would expect that it usually makes little sense to set it +to higher than 60.000 (60 seconds), but this is permitted. Just be warned that +this potentially delays log processing for that long. + + +queue.maxDiskSpace +------------------ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "0", "no", "``$ActionQueueMaxDiskSpace``" + +The maximum size that all queue files together will use on disk. Note +that the actual size may be slightly larger than the configured max, +as rsyslog never writes partial queue records. + + +queue.highWatermark +------------------- + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "90% of queue.size", "no", "``$ActionQueueHighWaterMark``" + +This applies to disk-assisted queues, only. When the queue fills up +to this number of messages, the queue begins to spool messages to +disk. Please note that this should not happen as part of usual +processing, because disk queue mode is very considerably slower than +in-memory queue mode. Going to disk should be reserved for cases +where an output action destination is offline for some period. + + +queue.lowWatermark +------------------ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "70% of queue.size", "no", "``$ActionQueueLowWaterMark``" + +This applies to disk-assisted queues, only. When the high watermark is +reached, the queue will write data to disk. It does so until the low +watermark is reached, then the queue reverts back to in-memory mode. + + +queue.fullDelaymark +------------------- + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "97% of queue.size", "no", "none" + +Number of messages when the queue should block delayable messages. +Messages are NO LONGER PROCESSED until the queue has sufficient space +again. If a message is delayable depends on the input. For example, +messages received via imtcp are delayable (because TCP can push back), +but those received via imudp are not (as UDP does not permit a push back). +The intent behind this setting is to leave some space in an almost-full +queue for non-delayable messages, which would be lost if the queue runs +out of space. Please note that if you use a DA queue, setting the +fulldelaymark BELOW the highwatermark makes the queue never activate +disk mode for delayable inputs. So this is probably not what you want. + + +queue.lightDelayMark +-------------------- + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "70% of queue.size", "no", "none" + +If this mark is reached the sender will be throttled if possible. The +main idea to do this is leave some space inside the queue for inputs +like UDP which cannot be throttled - and so any data arriving at +"queue full" would be discarded. + +If the special value `0` is used, `queue.LightDelayMark` will be set +to the value of `queue.size`. This effectively **disables** light delay +functionality. This is useful if a queue is not used by non-delayable +inputs like UDP. The special value was introduced in rsyslog 8.1904.0 +and is **not** available in earlier versions. There, you can achieve the +same result by setting `queue.LightDelayMark` to a very large value. + + +queue.discardMark +----------------- + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "80% of queue.size", "no", "``$ActionQueueDiscardMark``" + +Specifies the threshold at which rsyslog begins to discard less important +messages. To define which messages should be discarded use the +queue.discardseverity parameter. + + +queue.discardSeverity +--------------------- + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "8", "no", "``$ActionQueueDiscardSeverity``" + +As soon as the threshold of the parameter queue.discardMark is reached +incoming as well as queued messages with a priority equal or lower than +specified will be erased. With the default no messages will be erased. +You have to specify a numeric severity value for this parameter. + + +queue.checkpointInterval +------------------------ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "0", "no", "``$ActionQueueCheckpointInterval``" + +Disk queues by default do not update housekeeping structures every time +the queue writes to disk. This is for performance reasons. In the event of failure, +data will still be lost (except when data is mangled via the file structures). +However, disk queues can be set to write bookkeeping information on checkpoints +(every n records), so that this can be made ultra-reliable, too. If the +checkpoint interval is set to one, no data can be lost, but the queue is +exceptionally slow. + + +queue.syncqueuefiles +-------------------- + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "``$ActionQueueSyncQueueFiles``" + +Disk-based queues can be made very reliable by issuing a (f)sync after each +write operation. This happens when you set the parameter to "on". +Activating this option has a performance penalty, so it should not +be turned on without a good reason. Note that the penalty also depends on +*queue.checkpointInterval* frequency. + + +queue.samplingInterval +---------------------- + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "0", "no", "none" + +.. versionadded:: 8.23.0 + +This option allows queues to be populated by events produced at a specific interval. +It provides a way to sample data each N events, instead of processing all, in order +to reduce resources usage (disk, bandwidth...) + + +queue.type +---------- + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "word", "Direct", "no", "``$ActionQueueType``" + +Specifies the type of queue that will be used. Possible options are "FixedArray", +"LinkedList", "Direct" or "Disk". For more information read the documentation +for :doc:`queues <../concepts/queues>`. + + +queue.workerThreads +------------------- + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "1", "no", "``$ActionQueueWorkerThreads``" + +Specifies the maximum number of worker threads that can be run parallel. + + +queue.workerThreadMinimumMessages +--------------------------------- + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "queue.size/queue.workerthreads", "no", "``$ActionQueueWorkerThreadMinimumMessages``" + +Specify the number of messages a worker thread is processing before another +worker thread is created. This number is limited by parameter queue.workerThreads. +For example if this parameter is set to 200 and in the queue are 201 messages a +second worker thread will be created. + + +queue.timeoutWorkerthreadShutdown +--------------------------------- + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "60000", "no", "``$ActionQueueTimeoutWorkerthreadShutdown``" + +After starting a worker thread, it will process messages until there are no +messages for him to process. This parameter specifies the time the worker +thread has to be inactive before it times out. +The parameter must be specified in milliseconds. Which means the default of +60000 is 1 minute. + + +queue.timeoutshutdown +--------------------- + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "10/1500", "no", "``$ActionQueueTimeoutShutdown``" + +If a queue that still contains messages is terminated it will wait the +specified time interval for the worker thread to finish. +The time is specified in milliseconds (1000ms is 1sec). +Default for action queues is 10, for ruleset queues it is 1500. + + +queue.timeoutActionCompletion +----------------------------- + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "1000", "no", "``$ActionQueueTimeoutActionCompletion``" + +When a queue is terminated, the timeout shutdown is over and there is +still data in the queue, the queue will finish the current data element +and then terminate. This parameter specifies the timeout for processing +this last element. +Parameter is specified in milliseconds (1000ms is 1sec). + + +queue.timeoutEnqueue +-------------------- + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "2000", "no", "``$ActionQueueTimeoutEnqueue``" + +This timeout value is used when the queue is full. If rsyslog cannot +enqueue a message within the timeout period, the message is discarded. +Note that this is setting of last resort (assuming defaults are used +for the queue settings or proper parameters are set): all delayable +inputs (like imtcp or imfile) have already been pushed back at this +stage. Also, discarding of lower priority messages (if configured) has +already happened. So we run into one of these situations if we do not +timeout quickly enough: + +* if using imuxsock and no systemd journal is involved, the system + would become unresponsive and most probably a hard reset would be + required. +* if using imuxsock with imjournal forwarding is active, messages are + lost because the journal discards them (more aggressive than rsyslog does) +* if using imjournal, the journal will buffer messages. If journal + runs out of configured space, messages will be discarded. So in this + mode discarding is moved to a bit later place. +* other non-delayable sources like imudp will also loose messages + +So this setting is provided in order to guard against problematic situations, +which always will result either in message loss or system hang. For +action queues, one may debate if it would be better to overflow rapidly +to the main queue. If so desired, this is easy to accomplish by setting +a very large timeout value. The same, of course, is true for the main +queue, but you have been warned if you do so! + +In some other words, you can consider this scenario, using default values. +With all progress blocked (unable to deliver a message): + +* all delayable inputs (tcp, relp, imfile, imjournal, etc) will block + indefinantly (assuming queue.lightdelaymark and queue.fulldelaymark + are set sensible, which they are by default). +* imudp will be loosing messages because the OS will be dropping them +* messages arriving via UDP or imuxsock that do make it to rsyslog, + and that are a severity high enough to not be filtered by + discardseverity, will block for 2 seconds trying to put the message in + the queue (in the hope that something happens to make space in the + queue) and then be dropped to avoid blocking the machine permanently. + + Then the next message to be processed will also be tried for 2 seconds, etc. + +* If this is going into an action queue, the log message will remain + in the main queue during these 2 seconds, and additional logs that + arrive will accumulate behind this in the main queue. + + +queue.maxFileSize +----------------- + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "1m/16m", "no", "``$ActionQueueMaxFileSize``" + +Specifies the maximum size for the disk-assisted queue file. +Parameter can be specified in Mebibyte or Gibibyte, default for action +queues is 1m and for ruleset queues 16m (1m = 1024*1024). + + +queue.saveOnShutdown +-------------------- + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "binary", "off", "no", "``$ActionQueueSaveOnShutdown``" + +This parameter specifies if data should be saved at shutdown. + + +queue.dequeueSlowDown +--------------------- + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "0", "no", "``$ActionQueueDequeueSlowDown``" + +Regulates how long dequeueing should be delayed. This value must be specified +in microseconds (1000000us is 1sec). It can be used to slow down rsyslog so +it won't send things to fast. +For example if this parameter is set to 10000 on a UDP send action, the action +won't be able to put out more than 100 messages per second. + + +queue.dequeueTimeBegin +---------------------- + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "0", "no", "``$ActionQueueDequeueTimeBegin``" + +With this parameter you can specify rsyslog to process queues during specific +time periods. To define a time frame use the 24-hour format without minutes. +This parameter specifies the begin and "queue.dequeuetimeend" the end of the +time frame. + + +queue.dequeueTimeEnd +-------------------- + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "integer", "25", "no", "``$ActionQueueDequeueTimeEnd``" + +With this parameter you can specify rsyslog to process queues during specific +time periods. To define a time frame use the 24-hour format without minutes. +This parameter specifies the end and "queue.dequeuetimebegin" the begin of the +time frame. The default 25 disables the time-window. + + +queue.takeFlowCtlFromMsg +------------------------ + +.. csv-table:: + :header: "type", "default", "mandatory", "|FmtObsoleteName| directive" + :widths: auto + :class: parameter-table + + "boolean", "off", "no", "none" + +.. versionadded:: 8.1911.0 + +This is a fine-tuning parameter which permits to control whether or not +rsyslog shall always take the flow control setting from the message. If +so, non-primary queues may also **block** when reaching high water mark. + +This permits to add some synchronous processing to rsyslog core engine. +However, **this involves some risk**: Improper use may make the core engine +stall. As such, **enabling this parameter requires very careful planning +of the rsyslog configuration and deep understanding of the consequences**. + +Note that the parameter is applied to individual queues, so a configuration +with a large number of queues can (and must if used) be fine-tuned to +the exact use case. + +**The rsyslog team strongly recommends to let this parameter turned off.** + + + +Examples +======== + +Example 1 +--------- + +The following is a sample of a TCP forwarding action with its own queue. + +.. code-block:: none + + action(type="omfwd" target="192.168.2.11" port="10514" protocol="tcp" + queue.filename="forwarding" queue.size="1000000" queue.type="LinkedList" + ) + diff --git a/source/rainerscript/rainerscript_call.rst b/source/rainerscript/rainerscript_call.rst new file mode 100644 index 0000000..1dc3fa3 --- /dev/null +++ b/source/rainerscript/rainerscript_call.rst @@ -0,0 +1,56 @@ +The rsyslog "call" statement +============================ + +The rsyslog "call" statement is used to tie rulesets together. It is +modelled after the usual programming language "call" statement. Think of +a ruleset as a subroutine (what it really is!) and you get the picture. + +The "call" statement can be used to call into any type of rulesets. If a +rule set has a queue assigned, the message will be posted to that queue +and processed asynchronously. Otherwise, the ruleset will be executed +synchronously and control returns to right after the call when the rule +set has finished execution. + +Note that there is an important difference between asynchronous and +synchronous execution in regard to the "stop" statement. It will not +affect processing of the original message when run asynchronously. + +The "call" statement replaces the deprecated omruleset module. It offers +all capabilities omruleset has, but works in a much more efficient way. +Note that omruleset was a hack that made calling rulesets possible +within the constraints of the pre-v7 engine. "call" is the clean +solution for the new engine. Especially for rulesets without associated +queues (synchronous operation), it has zero overhead (really!). +omruleset always needs to duplicate messages, which usually means at +least ~250 bytes of memory writes, some allocs and frees - and even more +performance-intense operations. + + +syntax +------ + +``call rulesetname`` + +Where "rulesetname" is the name of a ruleset that is defined elsewhere +inside the configuration. If the call is synchronous or asynchronous +depends on the ruleset parameters. This cannot be overridden by the +"call" statement. + +compatibility notes +------------------- +Note that versions prior to 8.2110.0 had a bug where an explicit +'queue="direct"' setting in the ruleset definition lead call to treat +this as if a real queue existed. This could lead to some unexpected +behaviour. Beginning with 8.2110.0 this is handled consistently and +correctly. + +Under some exotic circumstances, this may look like a change of +behavior. If so, consider adding a small array-based queue to the +ruleset in question. + +related links +------------- + +- `Blog posting announcing "call" statement (with + sample) <https://rainer.gerhards.net/2012/10/how-to-use-rsyslogs-ruleset-and-call.html>`_ + diff --git a/source/rainerscript/rainerscript_call_indirect.rst b/source/rainerscript/rainerscript_call_indirect.rst new file mode 100644 index 0000000..5bedbe7 --- /dev/null +++ b/source/rainerscript/rainerscript_call_indirect.rst @@ -0,0 +1,58 @@ +The rsyslog "call_indirect" statement +===================================== + +The rsyslog "call_indirect" statement is equivalent to +:doc:`"call" statement <rainerscript_call>` +except that the name of the to be called ruleset is not constant but an +expression and so can be computed at runtime. + +If the ruleset name cannot be found when call_indirect is used, an error +message as emitted and the call_indirect statement is ignored. Execution +continues with the next statement. + +syntax +------ + +``call_indirect expression;`` + +Where "expression" is any valid expression. See +:doc:`expressions <expressions>` +for more information. Note that the trailing semicolon is needed to +indicate the end of expression. If it is not given, config load will +fail with a syntax error message. + +examples +-------- + +The potentially most useful use-case for "call_indirect" is calling a +ruleset based on a message variable. Let us assume that you have named +your rulesets according to syslog tags expected. Then you can use + +``call_indirect $syslogtag;`` + +To call these rulesets. Note, however, that this may be misused by a +malicious attacker, who injects invalid syslog tags. This could especially +be used to redirect message flow to known standard rulesets. To somewhat +mitigate against this, the ruleset name can be slightly mangled by creating +a **unique** prefix (do **not** use the one from this sample). Let us assume +the prefix "changeme-" is used, then all your rulesets should start with that +string. Then, the following call can be used: + +``call_indirect "changeme-" & $syslogtag;`` + +While it is possible to call a ruleset via a constant name: + +``call_indirect "my_ruleset";`` + +It is advised to use the "call" statement for this, as it offers superior +performance in this case. + +additional information +---------------------- + +We need to have two different statements, "call" and "call_indirect" because +"call" already existed at the time "call_indirect" was added. We could not +extend "call" to support expressions, as that would have broken existing +configs. In that case ``call ruleset`` would have become invalid and +``call "ruleset"`` would have to be used instead. Thus we decided to +add the additional "call_indirect" statement for this use case. diff --git a/source/rainerscript/variable_property_types.rst b/source/rainerscript/variable_property_types.rst new file mode 100644 index 0000000..3d0fb0a --- /dev/null +++ b/source/rainerscript/variable_property_types.rst @@ -0,0 +1,107 @@ +Variable (Property) types +========================= + +All rsyslog properties (see the :doc:`properties +<../configuration/properties>` page for a list) can be used in +RainerScript by prefixing them with "$", for example : +:: + + set $.x!host = $hostname; + +In addition, it also supports local variables. Local +variables are local to the current message, but are NOT message +properties (e.g. the "$!" all JSON property does not contain them). + +Only message json (CEE/Lumberjack) properties can be modified by the +**set**, **unset** and **reset** statements, not any other message property. Obviously, +local variables are also modifiable. + +Message JSON property names start with "$!" where the bang character +represents the root. + +Local variables names start with "$.", where the dot denotes the root. + +Both JSON properties as well as local variables may contain an arbitrary +deep path before the final element. The bang character is always used as +path separator, no matter if it is a message property or a local +variable. For example "$!path1!path2!varname" is a three-level deep +message property where as the very similar looking +"$.path1!path2!varname" specifies a three-level deep local variable. The +bang or dot character immediately following the dollar sign is used by +rsyslog to separate the different types. + +Note that the trailing semicolon is needed to indicate the end of expression. +If it is not given, config load will fail with a syntax error message. + +Check the following usage examples to understand how these statements behave: + +**set** +------- +sets the value of a local-variable or json property, but if the addressed +variable already contains a value its behaviour differs as follows: + +**merges** the value if both existing and new value are objects, +but merges the new value to *root* rather than with value of the given key. Eg. + +:: + + set $.x!one = "val_1"; + # results in $. = { "x": { "one": "val_1" } } + set $.y!two = "val_2"; + # results in $. = { "x": { "one": "val_1" }, "y": { "two": "val_2" } } + + set $.z!var = $.x; + # results in $. = { "x": { "one": "val_1" }, "y": { "two": "val_2" }, "z": { "var": { "one": "val_1" } } } + + set $.z!var = $.y; + # results in $. = { "x": { "one": "val_1" }, "y": { "two": "val_2" }, "z": { "var": { "one": "val_1" } }, "two": "val_2" } + # note that the key *two* is at root level and not under *$.z!var*. + +**ignores** the new value if old value was an object, but new value is a not an object (Eg. string, number etc). Eg: + +:: + + set $.x!one = "val_1"; + set $.x = "quux"; + # results in $. = { "x": { "one": "val_1" } } + # note that "quux" was ignored + +**resets** variable, if old value was not an object. + +:: + + set $.x!val = "val_1"; + set $.x!val = "quux"; + # results in $. = { "x": { "val": "quux" } } + +**unset** +--------- +removes the key. Eg: + +:: + + set $.x!val = "val_1"; + unset $.x!val; + # results in $. = { "x": { } } + +**reset** +--------- +force sets the new value regardless of what the variable +originally contained or if it was even set. Eg. + +:: + + # to contrast with the set example above, here is how results would look with reset + set $.x!one = "val_1"; + set $.y!two = "val_2"; + set $.z!var = $.x; + # results in $. = { "x": { "one": "val_1" }, "y": { "two": "val_2" }, "z": { "var": { "one": "val_1" } } } + # 'set' or 'reset' can be used interchangeably above(3 lines), they both have the same behaviour, as variable doesn't have an existing value + + reset $.z!var = $.y; + # results in $. = { "x": { "one": "val_1" }, "y": { "two": "val_2" }, "z": { "var": { "two": "val_2" } } } + # note how the value of $.z!var was replaced + + reset $.x = "quux"; + # results in $. = { "x": "quux", "y": { "two": "val_2" }, "z": { "var": { "two": "val_2" } } } + |