diff options
Diffstat (limited to '')
37 files changed, 1034 insertions, 0 deletions
diff --git a/source/proposals/big_restructuring/book/extending.rst b/source/proposals/big_restructuring/book/extending.rst new file mode 100644 index 0000000..0797206 --- /dev/null +++ b/source/proposals/big_restructuring/book/extending.rst @@ -0,0 +1,9 @@ +Extending rsyslog +================= + +Native plugins +-------------- + +External plugins +---------------- + diff --git a/source/proposals/big_restructuring/book/first_setup.rst b/source/proposals/big_restructuring/book/first_setup.rst new file mode 100644 index 0000000..9102c66 --- /dev/null +++ b/source/proposals/big_restructuring/book/first_setup.rst @@ -0,0 +1,6 @@ +Create your first Rsyslog setup +=============================== + +Teach how to get log messages from `logger` command and write to files conditionally. + + diff --git a/source/proposals/big_restructuring/book/index.rst b/source/proposals/big_restructuring/book/index.rst new file mode 100644 index 0000000..5b5260f --- /dev/null +++ b/source/proposals/big_restructuring/book/index.rst @@ -0,0 +1,14 @@ +The Book +======== + +.. toctree:: + + overview + installing + first_setup + language + input + output + queues + security + extending diff --git a/source/proposals/big_restructuring/book/input.rst b/source/proposals/big_restructuring/book/input.rst new file mode 100644 index 0000000..3938971 --- /dev/null +++ b/source/proposals/big_restructuring/book/input.rst @@ -0,0 +1,3 @@ +Input: from where come the logs +=============================== + diff --git a/source/proposals/big_restructuring/book/installing.rst b/source/proposals/big_restructuring/book/installing.rst new file mode 100644 index 0000000..9f1c6e8 --- /dev/null +++ b/source/proposals/big_restructuring/book/installing.rst @@ -0,0 +1,148 @@ +Installing and configuring Rsyslog +================================== + +General procedures to install and configure. + + +Installing from packages +------------------------ + +How to install using apt-get, yum, etc. + + +Installing from sources +----------------------- + +How to compile the sources into your system. + +Testing configuration blocks + + + .. code-block:: bash + + #### MODULES #### + + # Load (i)nput and (o)utput (m)odules + module(load="imuxsock") + module(load="imklog") + module(load="imudp") + module(load="imtcp") + module(load="imrelp") + module(load="omrelp") + module(load="impstats" interval="3600" severity="7" log.syslog="off" log.file="/var/log/rsyslog-stats.log") + + # Module parameters + input(type="imrelp" port="1514" ruleset="remote") + input(type="imtcp" port="514" ruleset="remote") + input(type="imudp" port="514" ruleset="remote") + + #### GLOBAL DIRECTIVES #### + + # Use default timestamp format + $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat + + # Spool files + $WorkDirectory /var/spool/rsyslog + + # Filter duplicate messages + $RepeatedMsgReduction on + + #### RULES #### + + #...cut out standard log rules for brevity...# + + ruleset(name="remote"){ + + action(Name="storage" + Type="omrelp" + Target="10.1.1.100" + Port="514" + Action.ExecOnlyWhenPreviousIsSuspended="on" + queue.FileName="storage-buffer" + queue.SaveOnShutdown="on" + queue.Type="LinkedList" + Action.ResumeInterval="30" + Action.ResumeRetryCount="-1" + Timeout="5") + + action(Name="analysis" + Type="omrelp" + Target="10.1.1.101" + Port="514" + Action.ExecOnlyWhenPreviousIsSuspended="on" + queue.FileName="analysis-buffer" + queue.SaveOnShutdown="on" + queue.Type="LinkedList" + Action.ResumeInterval="30" + Action.ResumeRetryCount="-1" + Timeout="5") + + action(Name="indexer" + Type="omfwd" + Target="10.1.1.102" + Protocol="tcp" + Port="514" + Action.ExecOnlyWhenPreviousIsSuspended="on" + queue.FileName="indexer-buffer" + queue.SaveOnShutdown="on" + queue.Type="LinkedList" + Action.ResumeInterval="30" + Action.ResumeRetryCount="-1" + Timeout="5") + } + + #### INCLUDES #### + + # Includes config files (Do these last) + $IncludeConfig /etc/rsyslog.d/*.conf + + +.. note:: + + You'll learn exactly how to load each file/format in the next section. + +.. option:: dest_dir + + Destination directory. + +.. option:: -m <module>, --module <module> + + Run a module as a script. + +.. envvar:: nome_envvar + +Descrevendo um programa. + +.. program:: rm + +.. option:: -r + + Work recursively. + +.. program:: svn + +.. option:: -r revision + + Specify the revision to work upon. + +------------------------------------------------- + +.. describe:: PAPER + + You can set this variable to select a paper size. + +------------------------------------------------- + + todo:: + + Este item é do TO DO. + +------------------------------------------------- + + todolist:: + + none + +------------------------------------------------- + +FIM diff --git a/source/proposals/big_restructuring/book/language.rst b/source/proposals/big_restructuring/book/language.rst new file mode 100644 index 0000000..c75a42f --- /dev/null +++ b/source/proposals/big_restructuring/book/language.rst @@ -0,0 +1,29 @@ +Configuration format +==================== + +Currently there are two ways of creating your configuration +script: RainerScript and the legacy format. + + +RainerScript format +------------------- + +General guidelines. + + +Legacy configuration format (deprecated) +---------------------------------------- + +General guidelines. + + +Why is there two different formats? +----------------------------------- + +Explain! + + + todo:: + + Este item é do TO DO in language-rst. + diff --git a/source/proposals/big_restructuring/book/output.rst b/source/proposals/big_restructuring/book/output.rst new file mode 100644 index 0000000..45fcc50 --- /dev/null +++ b/source/proposals/big_restructuring/book/output.rst @@ -0,0 +1,63 @@ +Output +====== + +What should be logged +--------------------- + +Message properties +^^^^^^^^^^^^^^^^^^ + + +System properties +^^^^^^^^^^^^^^^^^ + + +Variables +^^^^^^^^^ + + +Functions +^^^^^^^^^ + + +Transformation Modules +^^^^^^^^^^^^^^^^^^^^^^ + + +When should be logged +--------------------- + +Filter Conditions +^^^^^^^^^^^^^^^^^ + + * “traditional” severity and facility based selectors + * property-based filters + * expression-based filters + * BSD-style blocks (not upward compatible) + +Rulesets +^^^^^^^^ + + +How should be logged +-------------------- + + +RainerScript templates +^^^^^^^^^^^^^^^^^^^^^^ + + +Legacy format templates +^^^^^^^^^^^^^^^^^^^^^^^ + + +Properties in templates +^^^^^^^^^^^^^^^^^^^^^^^ + + +Conditionally choosing a template +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + + +Where should be send: Output Modules +------------------------------------ diff --git a/source/proposals/big_restructuring/book/overview.rst b/source/proposals/big_restructuring/book/overview.rst new file mode 100644 index 0000000..44c47d8 --- /dev/null +++ b/source/proposals/big_restructuring/book/overview.rst @@ -0,0 +1,38 @@ +Overview +======== + +Write a bit about the logging challenge. + +What is Rsyslog? +---------------- + +`Rsyslog <http://www.rsyslog.com/>`_ is a **r**\ ocket-fast **sys**\ tem for **log** processing. +It offers high-performance, great security features and a modular design. +While it started as a regular syslogd, rsyslog has evolved into a kind of +**swiss army knife of logging**, being able to + +- accept inputs from a wide variety of sources, +- transform them, +- and output the results to diverse destinations. + +Rsyslog has a strong enterprise focus but also scales down to small +systems. + +Message flow in rsyslog +----------------------- + +From where to where and when... describes the flow. + +Input +^^^^^ + +Message Transformation +^^^^^^^^^^^^^^^^^^^^^^ + +Output +^^^^^^ + +Output format: Templates +"""""""""""""""""""""""" + + diff --git a/source/proposals/big_restructuring/book/queues.rst b/source/proposals/big_restructuring/book/queues.rst new file mode 100644 index 0000000..cb35d74 --- /dev/null +++ b/source/proposals/big_restructuring/book/queues.rst @@ -0,0 +1,23 @@ +Queues: prepare for the worst +============================= + + +What are queues? +---------------- + +In-memory queues +^^^^^^^^^^^^^^^^ + +Disk queues +^^^^^^^^^^^ + +Disk-assisted queues +^^^^^^^^^^^^^^^^^^^^ + + +Main message queue +------------------ + +Action queues +------------- + diff --git a/source/proposals/big_restructuring/book/security.rst b/source/proposals/big_restructuring/book/security.rst new file mode 100644 index 0000000..573086e --- /dev/null +++ b/source/proposals/big_restructuring/book/security.rst @@ -0,0 +1,9 @@ +Security +======== + +Securing your setup +------------------- + +Dropping privileges +------------------- + diff --git a/source/proposals/big_restructuring/contributing/code/git.rst b/source/proposals/big_restructuring/contributing/code/git.rst new file mode 100644 index 0000000..e2a7176 --- /dev/null +++ b/source/proposals/big_restructuring/contributing/code/git.rst @@ -0,0 +1,41 @@ +Git +=== + +This document explains some conventions and specificities in the way we manage +the Rsyslog code with Git. + +Pull Requests +------------- + +Whenever a pull request is merged, all the information contained in the pull +request (including comments) is saved in the repository. + +You can easily spot pull request merges as the commit message always follows +this pattern: + +.. code-block:: text + + merged branch USER_NAME/BRANCH_NAME (PR #1111) + +The PR reference allows you to have a look at the original pull request on +GitHub: https://github.com/rsyslog/rsyslog/pull/1111. But all the information +you can get on GitHub is also available from the repository itself. + +The merge commit message contains the original message from the author of the +changes. Often, this can help understand what the changes were about and the +reasoning behind the changes. + +Moreover, the full discussion that might have occurred back then is also +stored as a Git note. To get access to these notes, add this line to +your ``.git/config`` file: + +.. code-block:: ini + + fetch = +refs/notes/*:refs/notes/* + +After a fetch, getting the GitHub discussion for a commit is then a matter of +adding ``--show-notes=github-comments`` to the ``git show`` command: + +.. code-block:: bash + + $ git show HEAD --show-notes=github-comments diff --git a/source/proposals/big_restructuring/contributing/code/index.rst b/source/proposals/big_restructuring/contributing/code/index.rst new file mode 100644 index 0000000..734fa33 --- /dev/null +++ b/source/proposals/big_restructuring/contributing/code/index.rst @@ -0,0 +1,8 @@ +Contributing Code +================= + +.. toctree:: + :maxdepth: 2 + + standards + git diff --git a/source/proposals/big_restructuring/contributing/code/standards.rst b/source/proposals/big_restructuring/contributing/code/standards.rst new file mode 100644 index 0000000..5528af0 --- /dev/null +++ b/source/proposals/big_restructuring/contributing/code/standards.rst @@ -0,0 +1,10 @@ +Coding Standards +================ + +When contributing code to Rsyslog, you must follow its coding standards. To +make a long story short, here is the golden rule: **Imitate the existing +Rsyslog code**. Most open-source modules and libraries recommended by Rsyslog also +follow the same guidelines, and you should too. + +Remember that the main advantage of standards is that every piece of code +looks and feels familiar, it's not about this or that being more readable. diff --git a/source/proposals/big_restructuring/contributing/community/index.rst b/source/proposals/big_restructuring/contributing/community/index.rst new file mode 100644 index 0000000..43b7b52 --- /dev/null +++ b/source/proposals/big_restructuring/contributing/community/index.rst @@ -0,0 +1,8 @@ +Community +========= + +.. toctree:: + :maxdepth: 2 + + releases + other diff --git a/source/proposals/big_restructuring/contributing/community/other.rst b/source/proposals/big_restructuring/contributing/community/other.rst new file mode 100644 index 0000000..26b2cce --- /dev/null +++ b/source/proposals/big_restructuring/contributing/community/other.rst @@ -0,0 +1,13 @@ +Other Resources +=============== + +In order to follow what is happening in the community you might find helpful +these additional resources: + +* List of open pull requests `pull requests`_ +* List of recent `commits`_ +* List of open bugs and enhancements `bugs and enhancements`_ + +.. _pull requests: https://github.com/rsyslog/rsyslog/pulls +.. _commits: https://github.com/rsyslog/rsyslog/commits/master +.. _bugs and enhancements: https://github.com/rsyslog/rsyslog/issues diff --git a/source/proposals/big_restructuring/contributing/community/releases.rst b/source/proposals/big_restructuring/contributing/community/releases.rst new file mode 100644 index 0000000..2486eda --- /dev/null +++ b/source/proposals/big_restructuring/contributing/community/releases.rst @@ -0,0 +1,43 @@ +The Release Process +=================== + +This document explains the Rsyslog release process (Rsyslog being the code +hosted on the main ``rsyslog/rsyslog`` `Git repository`_). + +Rsyslog manages its releases through a *time-based model*; a new Rsyslog minor +version comes out every *six weeks*. + +.. tip:: + + The meaning of "minor" comes from the `Semantic Versioning`_ strategy. + +Each minor version sticks to the same very well-defined process where we start +with a development period, followed by a maintenance period. + +.. note:: + + This release process has been adopted as of Rsyslog 8.2, and all the + "rules" explained in this document must be strictly followed as of Rsyslog + 8.3. + +.. _contributing-release-development: + +Development +----------- + +The full development period lasts six weeks and is divided into two phases: + +* *Development*: *Four weeks* to add new features and to enhance existing + ones; + +* *Stabilisation*: *Two weeks* to fix bugs, prepare the release, and wait + for the whole Rsyslog ecosystem (third-party libraries, bundles, and + projects using Rsyslog) to catch up. + +During the development phase, any new feature can be reverted if it won't be +finished in time or if it won't be stable enough to be included in the current +final release. + + +.. _Semantic Versioning: http://semver.org/ +.. _Git repository: https://github.com/rsyslog/rsyslog diff --git a/source/proposals/big_restructuring/contributing/documentation/index.rst b/source/proposals/big_restructuring/contributing/documentation/index.rst new file mode 100644 index 0000000..e684599 --- /dev/null +++ b/source/proposals/big_restructuring/contributing/documentation/index.rst @@ -0,0 +1,3 @@ +How to contribute to the documentation +====================================== + diff --git a/source/proposals/big_restructuring/contributing/index.rst b/source/proposals/big_restructuring/contributing/index.rst new file mode 100644 index 0000000..d4d99b5 --- /dev/null +++ b/source/proposals/big_restructuring/contributing/index.rst @@ -0,0 +1,9 @@ +Contributing +============ + +.. toctree:: + + code/index + documentation/index + community/index + diff --git a/source/proposals/big_restructuring/cookbook/index.rst b/source/proposals/big_restructuring/cookbook/index.rst new file mode 100644 index 0000000..8291e44 --- /dev/null +++ b/source/proposals/big_restructuring/cookbook/index.rst @@ -0,0 +1,7 @@ +The Cookbook +============ + +.. toctree:: + + templates/index + setup/index diff --git a/source/proposals/big_restructuring/cookbook/setup/centralised_logging_logstash.rst b/source/proposals/big_restructuring/cookbook/setup/centralised_logging_logstash.rst new file mode 100644 index 0000000..b33e9e8 --- /dev/null +++ b/source/proposals/big_restructuring/cookbook/setup/centralised_logging_logstash.rst @@ -0,0 +1,3 @@ +Centralised logging with Logstash/ElasticSearch/Kibana +====================================================== + diff --git a/source/proposals/big_restructuring/cookbook/setup/index.rst b/source/proposals/big_restructuring/cookbook/setup/index.rst new file mode 100644 index 0000000..def615e --- /dev/null +++ b/source/proposals/big_restructuring/cookbook/setup/index.rst @@ -0,0 +1,7 @@ +Setup Cookbooks +=============== + + +.. toctree:: + + centralised_logging_logstash diff --git a/source/proposals/big_restructuring/cookbook/templates/index.rst b/source/proposals/big_restructuring/cookbook/templates/index.rst new file mode 100644 index 0000000..1b3e14e --- /dev/null +++ b/source/proposals/big_restructuring/cookbook/templates/index.rst @@ -0,0 +1,8 @@ +Templates +========= + +.. toctree:: + + rfc3164 + rfc5424 + diff --git a/source/proposals/big_restructuring/cookbook/templates/rfc3164.rst b/source/proposals/big_restructuring/cookbook/templates/rfc3164.rst new file mode 100644 index 0000000..7045fb1 --- /dev/null +++ b/source/proposals/big_restructuring/cookbook/templates/rfc3164.rst @@ -0,0 +1,3 @@ +Configuring an RFC 3164 Template with Json message +================================================== + diff --git a/source/proposals/big_restructuring/cookbook/templates/rfc5424.rst b/source/proposals/big_restructuring/cookbook/templates/rfc5424.rst new file mode 100644 index 0000000..1b60645 --- /dev/null +++ b/source/proposals/big_restructuring/cookbook/templates/rfc5424.rst @@ -0,0 +1,3 @@ +Configuring an RFC 5424 Template with Json message +================================================== + diff --git a/source/proposals/big_restructuring/documentation_review.rst b/source/proposals/big_restructuring/documentation_review.rst new file mode 100644 index 0000000..b257bfd --- /dev/null +++ b/source/proposals/big_restructuring/documentation_review.rst @@ -0,0 +1,83 @@ +Rsyslog Documentation Review Proposal +===================================== + +Currently the Rsyslog documentation is spread over many places. It's not logical +and well-organized as well. The objective of this proposal is to address those issues +and establish a procedure for further development of the documentation. + +.. note:: + + We SHOULD NOT write examples using mixed formats, RainerScripts and legacy. It confused + the readers. We can provide them in both formats, but should not mix them. + + +Compile information from current sources of information +------------------------------------------------------- + +Below are listed the official locations where documentation about Rsyslog can be found. + + * Wiki: http://wiki.rsyslog.com/index.php/Main_Page + * Rainer's blog: https://rainer.gerhards.net/2012/10/how-to-use-rsyslogs-ruleset-and-call.html + * Issues: https://github.com/rsyslog/rsyslog + * docs: https://github.com/rsyslog/rsyslog-doc + * Forum: http://kb.monitorware.com/configuration-f36.html + * https://www.youtube.com/user/rainergerhards + + +Add a Cookbook Section +---------------------- + +We should create some cookbooks to help people get started with Rsyslog. +Some candidates to be a cookbook are below. + + * http://sickbits.net/log-storage-and-analysis-infrastructure-reliable-logging-and-analysis-with-rsyslog-and-relp/ + * http://kb.monitorware.com/omfile-with-dynfile-syslogfacility-text-t12515.html + * http://www.freeipa.org/page/Howto/Centralised_Logging_with_Logstash/ElasticSearch/Kibana + +Add a subsection called "processing logs from". We'd place articles that'd would help people with specific +common scenarios for a specific log sender application. + +Add a Reference Section +----------------------- + +This section would have all the reference configuration of all possible tags, in both formats, RainerScript +and legacy. + + +Write articles that address common problems +------------------------------------------- + +Some of the common are following. + + * https://github.com/rsyslog/rsyslog/issues/160 + * http://kb.monitorware.com/nginx-logging-rsyslog-t12359.html + * http://trac.nginx.org/nginx/ticket/677 + + +Extra +----- + +Some resources worth taking a look. + + * https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/System_Administrators_Guide/ch-Viewing_and_Managing_Log_Files.html + * https://www.usenix.org/system/files/login/articles/06_lang-online.pdf + * https://media.readthedocs.org/pdf/rsyslog/latest/rsyslog.pdf + * http://download.rsyslog.com/rainerscript2_rsyslog.conf + * http://people.redhat.com/pvrabec/rpms/rsyslog/rsyslog-example.conf + * http://www.rsyslog.com/doc/syslog_parsing.html + + +Initial Summary +--------------- + + - Troubleshooting + + * http://www.rsyslog.com/how-can-i-check-the-config/ + +Proposed Documentation Structure +-------------------------------- + + toctree:: + + new_documentation/index + diff --git a/source/proposals/big_restructuring/index.rst b/source/proposals/big_restructuring/index.rst new file mode 100644 index 0000000..a75c9f5 --- /dev/null +++ b/source/proposals/big_restructuring/index.rst @@ -0,0 +1,15 @@ +Rsyslog documentation +===================== + +Below is the proposed documentation structure. It's initial content is the +current documentation available, just organized differently. + +.. toctree:: + + book/index + cookbook/index + reference/index + contributing/index + documentation_review.rst + + diff --git a/source/proposals/big_restructuring/reference/action.rst b/source/proposals/big_restructuring/reference/action.rst new file mode 100644 index 0000000..7ad1aaf --- /dev/null +++ b/source/proposals/big_restructuring/reference/action.rst @@ -0,0 +1,3 @@ +Action Configuration Reference +============================== + diff --git a/source/proposals/big_restructuring/reference/global.rst b/source/proposals/big_restructuring/reference/global.rst new file mode 100644 index 0000000..cc464fd --- /dev/null +++ b/source/proposals/big_restructuring/reference/global.rst @@ -0,0 +1,3 @@ +Global Configuration Reference +============================== + diff --git a/source/proposals/big_restructuring/reference/index.rst b/source/proposals/big_restructuring/reference/index.rst new file mode 100644 index 0000000..4583a93 --- /dev/null +++ b/source/proposals/big_restructuring/reference/index.rst @@ -0,0 +1,11 @@ +Configuration Reference +======================= + +.. toctree:: + + module + input + action + parser + global + timezone diff --git a/source/proposals/big_restructuring/reference/input.rst b/source/proposals/big_restructuring/reference/input.rst new file mode 100644 index 0000000..f1dd84d --- /dev/null +++ b/source/proposals/big_restructuring/reference/input.rst @@ -0,0 +1,3 @@ +Input Configuration Reference +============================= + diff --git a/source/proposals/big_restructuring/reference/module.rst b/source/proposals/big_restructuring/reference/module.rst new file mode 100644 index 0000000..3876f37 --- /dev/null +++ b/source/proposals/big_restructuring/reference/module.rst @@ -0,0 +1,3 @@ +Module Configuration Reference +============================== + diff --git a/source/proposals/big_restructuring/reference/parser.rst b/source/proposals/big_restructuring/reference/parser.rst new file mode 100644 index 0000000..c9e6dc6 --- /dev/null +++ b/source/proposals/big_restructuring/reference/parser.rst @@ -0,0 +1,3 @@ +Parser Configuration Reference +============================== + diff --git a/source/proposals/big_restructuring/reference/timezone.rst b/source/proposals/big_restructuring/reference/timezone.rst new file mode 100644 index 0000000..34db014 --- /dev/null +++ b/source/proposals/big_restructuring/reference/timezone.rst @@ -0,0 +1,3 @@ +Timezone Configuration Reference +================================ + diff --git a/source/proposals/big_restructuring/toc_screenshot.png b/source/proposals/big_restructuring/toc_screenshot.png Binary files differnew file mode 100644 index 0000000..b474c49 --- /dev/null +++ b/source/proposals/big_restructuring/toc_screenshot.png diff --git a/source/proposals/index.rst b/source/proposals/index.rst new file mode 100644 index 0000000..41c106a --- /dev/null +++ b/source/proposals/index.rst @@ -0,0 +1,9 @@ +Proposals +========= + +.. toctree:: + :maxdepth: 2 + + version_naming + lookup_tables + big_restructuring/index diff --git a/source/proposals/lookup_tables.rst b/source/proposals/lookup_tables.rst new file mode 100644 index 0000000..f61d673 --- /dev/null +++ b/source/proposals/lookup_tables.rst @@ -0,0 +1,240 @@ +Lookup Tables +============= + +**NOTE: this is proposed functionality, which is NOT YET IMPLEMENTED!** + +**Lookup tables are a powerful construct to obtain "class" information +based on message content (e.g. to build log file names for different +server types, departments or remote offices).** + +The base idea is to use a message variable as an index into a table +which then returns another value. For example, $fromhost-ip could be +used as an index, with the table value representing the type of server +or the department or remote office it is located in. A main point with +lookup tables is that the lookup is very fast. So while lookup tables +can be emulated with if-elseif constructs, they are generally much +faster. Also, it is possible to reload lookup tables during rsyslog +runtime without the need for a full restart. + +The lookup tables itself exists in a separate configuration file (one +per table). This file is loaded on rsyslog startup and when a reload is +requested. + +There are different types of lookup tables: + +- **string** - the value to be looked up is an arbitrary string. Only + exact some strings match. +- **array** - the value to be looked up is an integer number from a + consecutive set. The set does not need to start at zero or one, but + there must be no number missing. So, for example 5,6,7,8,9 would be a + valid set of index values, while 1,2,4,5 would not be (due to missing + 2). A match happens if the requested number is present. +- **sparseArray** - the value to be looked up is an integer value, but + there may be gaps inside the set of values (usually there are large + gaps). A typical use case would be the matching of IPv4 address + information. A match happens on the first value that is less than or + equal to the requested value. + +Note that index integer numbers are represented by unsigned 32 bits. + +Lookup tables can be access via the lookup() built-in function. The core +idea is to set a local variable to the lookup result and later on use +that local variable in templates. + +More details on usage now follow. + +Lookup Table File Format +------------------------ + +Lookup table files contain a single JSON object. This object contains of +a header and a table part. + +Header +~~~~~~ + +The header is the top-level json. It has parameters "version", "nomatch", +and "type". The version parameter must be given and must always be one +for this version of rsyslog. The nomatch parameter is optional. If +specified, it contains the value to be used if lookup() is provided an +index value for which no entry exists. The default for "nomatch" is the +empty string. Type specifies the type of lookup to be done. + +Table +~~~~~ + +This must be an array of elements, even if only a single value exists +(for obvious reasons, we do not expect this to occur often). Each array +element must contain two fields "index" and "value". + +Example +~~~~~~~ + +This is a sample of how an ip-to-office mapping may look like: + +:: + + { "version":1, "nomatch":"unk", "type":"string", + "table":[ {"index":"10.0.1.1", "value":"A" }, + {"index":"10.0.1.2", "value":"A" }, + {"index":"10.0.1.3", "value":"A" }, + {"index":"10.0.2.1", "value":"B" }, + {"index":"10.0.2.2", "value":"B" }, + {"index":"10.0.2.3", "value":"B" } + ] + } + +Note: if a different IP comes in, the value "unk" is returned thanks to +the nomatch parameter in the first line. + +RainerScript Statements +----------------------- + +lookup\_table() Object +~~~~~~~~~~~~~~~~~~~~~~ + +This statement defines and initially loads a lookup table. Its format is +as follows: + +:: + + lookup_table(name="name" file="/path/to/file" reloadOnHUP="on|off") + +Parameters +^^^^^^^^^^ + +- **name** (mandatory) + + Defines the name of lookup table for further reference inside the + configuration. Names must be unique. Note that it is possible, though + not advisible, to have different names for the same file. +- **file** (mandatory) + + Specifies the full path for the lookup table file. This file must be + readable for the user rsyslog is run under (important when dropping + privileges). It must point to a valid lookup table file as described + above. +- **reloadOnHUP** (optional, default "on") + + Specifies if the table shall automatically be reloaded as part of + HUP processing. For static tables, the default is "off" and + specifying "on" triggers an error message. Note that the default of + "on" may be somewhat suboptimal performance-wise, but probably is + what the user intuitively expects. Turn it off if you know that you + do not need the automatic reload capability. + +lookup() Function +~~~~~~~~~~~~~~~~~ + +This function is used to actually do the table lookup. Format: + +:: + + lookup("name", indexvalue) + +Parameters +^^^^^^^^^^ + +- **return value** + + The function returns the string that is associated with the given + indexvalue. If the indexvalue is not present inside the lookup table, + the "nomatch" string is returned (or an empty string if it is not + defined). +- **name** (constant string) + + The lookup table to be used. Note that this must be specified as a + constant. In theory, variable table names could be made possible, but + their runtime behaviour is not as good as for static names, and we do + not (yet) see good use cases where dynamic table names could be + useful. +- **indexvalue** (expression) + + The value to be looked up. While this is an arbitrary RainerScript + expression, it's final value is always converted to a string in order + to conduct the lookup. For example, "lookup(table, 3+4)" would be + exactly the same as "lookup(table, "7")". In most cases, indexvalue + will probably be a single variable, but it could also be the result + of all RainerScript-supported expression types (like string + concatenation or substring extraction). Valid samples are + "lookup(name, $fromhost-ip & $hostname)" or "lookup(name, + substr($fromhost-ip, 0, 5))" as well as of course the usual + "lookup(table, $fromhost-ip)". + +load\_lookup\_table Statement +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +**Note: in the final implementation, this MAY be implemented as an +action. This is a low-level decesion that must be made during the detail +development process. Parameters and semantics will remain the same of +this happens.** + +This statement is used to reload a lookup table. It will fail if the +table is static. While this statement is executed, lookups to this table +are temporarily blocked. So for large tables, there may be a slight +performance hit during the load phase. It is assume that always a +triggering condition is used to load the table. + +:: + + load_lookup_table(name="name" errOnFail="on|off" valueOnFail="value") + +Parameters +^^^^^^^^^^ + +- **name** (string) + + The lookup table to be used. +- **errOnFail** (boolean, default "on") + + Specifies whether or not an error message is to be emitted if there + are any problems reloading the lookup table. +- **valueOnFail** (optional, string) + + This parameter affects processing if the lookup table cannot be + loaded for some reason: If the parameter is not present, the previous + table will be kept in use. If the parameter is given, the previous + table will no longer be used, and instead an empty table be with + nomath=valueOnFail be generated. In short, that means when the + parameter is set and the reload fails, all matches will always return + what is specified in valueOnFail. + +Usage example +~~~~~~~~~~~~~ + +For clarity, we show only those parts of rsyslog.conf that affect lookup +tables. We use the remote office example that an example lookup table +file is given above for. + +:: + + lookup_table(name="ip2office" file="/path/to/ipoffice.lu" + reloadOnHUP="off") + + + template(name="depfile" type="string" + string="/var/log/%$usr.dep%/messages") + + set $usr.dep = lookup("ip2office", $fromhost-ip); + action(type="omfile" dynfile="depfile") + + # support for reload "commands" + if $fromhost-ip == "10.0.1.123" + and $msg contains "reload office lookup table" + then + load_lookup_table(name="ip2office" errOnFail="on") + +Note: for performance reasons, it makes sense to put the reload command +into a dedicated ruleset, bound to a specific listener - which than +should also be sufficiently secured, e.g. via TLS mutual auth. + +Implementation Details +---------------------- + +The lookup table functionality is implemented via highly efficient +algorithms. The string lookup has O(log n) time complexity. The array +lookup is O(1). In case of sparseArray, we have O(log n). + +To preserve space and, more important, increase cache hit performance, +equal data values are only stored once, no matter how often a lookup +index points to them. + diff --git a/source/proposals/version_naming.rst b/source/proposals/version_naming.rst new file mode 100644 index 0000000..7a0e7f2 --- /dev/null +++ b/source/proposals/version_naming.rst @@ -0,0 +1,150 @@ +Version Naming +============== + +This is the proposal on how versions should be named in the future: + +Rsyslog version naming has undergone a number of changes in the past. +Our sincere hopes is that the scheme outlined here will serve us well +for the future. In general, a three-number versioning scheme with a +potential development state indication is used. It follows this pattern: + +major.minor.patchlevel[-devstate] + +where devstate has some further structure: +-<releaseReason><releaseNumber> + +All stable builds come without the devstate part. All unstable +development version come with it. + +The major version is incremented whenever something really important +happens. A single new feature, even if important, does not justify an +increase in the major version. There is no hard rule when the major +version needs an increment. It mostly is a soft factor, when the +developers and/or the community think there has been sufficient change +to justify that. Major version increments are expected to happen quite +infrequently, maybe around once a year. A major version increment has +important implications from the support side: without support contracts, +the current major version's last stable release and the last stable +release of the version immediately below it are supported (Adiscon, the +rsyslog sponsor, offers `support contracts <http://www.rsyslog.com/professional-services/>`_ +covering all other versions). + +The minor version is incremented whenever a non-trivial new feature is +planned to be added. Triviality of a feature is simply determined by +time estimated to implement a feature. If that's more than a few days, +it is considered a non-trivial feature. Whenever a new minor version is +begun, the desired feature is identified and will be the primary focus +of that major.minor version. Trivial features may justify a new minor +version if they either do not look trivial from the user's point of view +or change something quite considerable (so we need to alert users). A +minor version increment may also be done for some other good reasons +that the developers have. + +The patchlevel is incremented whenever there is a bugfix or very minor +feature added to a (stable or development) release. + +The devstate is important during development of a feature. It helps the +developers to release versions with new features to the general public +and in the hope that this will result in some testing. To understand how +it works, we need to look at the release cycle: As already said, at the +start of a new minor version, a new non-trivial feature to be +implemented in that version is selected. Development on this feature +begins. At the current pace of development, getting initial support for +such a non-trivial feature typically takes between two and four weeks. +During this time, new feature requests come in. Also, we may find out +that it may be just the right time to implement some not yet targeted +feature requests. A reason for this is that the minor release's feature +focus is easier to implement if the other feature is implemented first. +This is a quite common thing to happen. So development on the primary +focus may hold for a short period while we implement something else. +Even unrelated, but very trivial feature requests (maybe an hour's worth +of time to implement), may be done in between. Once we have implemented +these things, we would like to release as quickly as possible (even more +if someone has asked for the feature). So we do not like to wait for the +original focus feature to be ready (what could take maybe three more +weeks). As a result, we release the new features. But that version will +also include partial code of the focus feature. Typically this doesn't +hurt as long as noone tries to use it (what of course would miserably +fail). But still, part of the new code is already in it. When we release +such a "minor-feature enhanced" but "focus-feature not yet completed" +version, we need a way to flag it. In current thinking, that is using a +"-mf<version>" devstate in the version number ("mf" stands for "minor +feature"). Version numbers for -mf releases start at 0 for the first +release and are monotonically incremented. Once the focus feature has +been fully implemented, a new version now actually supporting that +feature will be released. Now, the release reason is changed to the +well-know "-rc<version>" where "rc" stands for release candidate. For +the first release candidate, the version starts at 0 again and is +incremented monotonically for each subsequent release. Please note that +a -rc0 may only have bare functionality but later -rc's have a richer +one. If new minor features are implemented and released once we have +reached rc stage, still a new rc version is issued. The difference +between "mf" and "rc" is simply the presence of the desired feature. No +support is provided for -mf versions once the first -rc version has been +released. And only the most current -rc version is supported. + +The -rc is removed and the version declared stable when we think it has +undergone sufficient testing and look sufficiently well. Then, it'll +turn into a stable release. Stable minor releases never receive +non-trivial new features. There may be more than one -rc releases +without a stable release present at the same time. In fact, most often +we will work on the next minor development version while the previous +minor version is still a -rc because it is not yet considered +sufficiently stable. + +Note: the absence of the -devstate part indicates that a release is +stable. Following the same logic, any release with a -devstate part is +unstable. + +A quick sample: + +4.0.0 is the stable release. We begin to implement relp, moving to +major.minor to 4.1. While we develop it, someone requests a trivial +feature, which we implement. We need to release, so we will have +4.1.0-mf0. Another new feature is requested, move to 4.1.0-mf2. A first +version of RELP is implemented: 4.1.0-rc0. A new trivial feature is +implemented: 4.1.0-rc1. Relp is being enhanced: 4.1.0-rc2. We now feel +RELP is good enough for the time being and begin to implement TLS on +plain /Tcp syslog: logical increment to 4.2. Now another new feature in +that tree: 4.2.0-mf0. Note that we now have 4.0.0 (stable) and 4.1.0-rc2 +and 4.1.0-mf0 (both devel). We find a big bug in RELP coding. Two new +releases: 4.1.0-rc3, 4.2.0-mf1 (the bug fix acts like a non-focus +feature change). We release TLS: 4.2.0-rc0. Another RELP bug fix +4.1.0-rc4, 4.2.0-rc1. After a while, RELP is matured: 4.1.0 (stable). +Now support for 4.0.x stable ends. It, however, is still provided for +3.x.x (in the actual case 2.x.x, because v3 was under the old naming +scheme and now stable v3 was ever released). + +This is how it is done so far: + +This document briefly outlines the strategy for naming versions. It +applies to versions 1.0.0 and above. Versions below that are all +unstable and have a different naming schema. + +**Please note that version naming is currently being changed. There is a +`blog post about future rsyslog +versions <https://rainer.gerhards.net/2007/08/on-rsyslog-versions.html>`_.** + +The major version is incremented whenever a considerate, major features +have been added. This is expected to happen quite infrequently. + +The minor version number is incremented whenever there is "sufficient +need" (at the discretion of the developers). There is a notable +difference between stable and unstable branches. The **stable branch** +always has a minor version number in the range from 0 to 9. It is +expected that the stable branch will receive bug and security fixes +only. So the range of minor version numbers should be quite sufficient. + +For the **unstable branch**, minor version numbers always start at 10 +and are incremented as needed (again, at the discretion of the +developers). Here, new minor versions include both fixes as well as new +features (hopefully most of the time). They are expected to be released +quite often. + +The patch level (third number) is incremented whenever a really minor +thing must be added to an existing version. This is expected to happen +quite infrequently. + +In general, the unstable branch carries all new development. Once it +concludes with a sufficiently-enhanced, quite stable version, a new +major stable version is assigned. |
