1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
|
Log Message Normalization Module (mmnormalize)
==============================================
**Module Name: mmnormalize**
**Available since:** 6.1.2+
**Author:** Rainer Gerhards <rgerhards@adiscon.com>
**Description**:
This module provides the capability to normalize log messages via
`liblognorm <http://www.liblognorm.com>`_. Thanks to liblognorm,
unstructured text, like usually found in log messages, can very quickly
be parsed and put into a normal form. This is done so quickly, that it
should be possible to normalize events in realtime.
This module is implemented via the output module interface. This means
that mmnormalize should be called just like an action. After it has been
called, the normalized message properties are available and can be
accessed. These properties are called the "CEE/lumberjack" properties,
because liblognorm creates a format that is inspired by the
CEE/lumberjack approach.
**Please note:** CEE/lumberjack properties are different from regular
properties. They have always "$!" prepended to the property name given
in the rulebase. Such a property needs to be called with
**%$!propertyname%**.
Note that from a performance point of view mmnormalize should only be called
once on each message, if possible. To do so, place all rules into a single
rule base. If that is not possible, you can safely call mmnormalize multiple
times. This incurs a small performance drawback.
Module Parameters
~~~~~~~~~~~~~~~~~
Note: parameter names are case-insensitive.
.. function:: allow_regex <boolean>
**Default**: off
Specifies if regex field-type should be allowed. Regex field-type has
significantly higher computational overhead compared to other fields,
so it should be avoided when another field-type can achieve the desired
effect. Needs to be "on" for regex field-type to work.
Action Parameters
~~~~~~~~~~~~~~~~~
Note: parameter names are case-insensitive.
.. function:: ruleBase <word>
Specifies which rulebase file is to use. If there are multiple
mmnormalize instances, each one can use a different file. However, a
single instance can use only a single file. This parameter or **rule** MUST be
given, because normalization can only happen based on a rulebase. It
is recommended that an absolute path name is given. Information on
how to create the rulebase can be found in the `liblognorm
manual <http://www.liblognorm.com/files/manual/index.html>`_.
.. function:: rule <array>
*(Available since: 8.26.0)*
Contains an array of strings which will be put together as the rulebase. This parameter
or **rulebase** MUST be given, because normalization can only happen based on a rulebase.
.. function:: useRawMsg <boolean>
**Default**: off
Specifies if the raw message should be used for normalization (on)
or just the MSG part of the message (off).
.. function:: path <word>
**Default**: $!
Specifies the JSON path under which parsed elements should be
placed. By default, all parsed properties are merged into root of
message properties. You can place them under a subtree, instead. You
can place them in local variables, also, by setting path="$.".
.. function:: variable <word>
*(Available since: 8.5.1)*
Specifies if a variable insteed of property 'msg' should be used for
normalization. A variable can be property, local variable, json-path etc.
Please note that **useRawMsg** overrides this parameter, so if **useRawMsg**
is set, **variable** will be ignored and raw message will be used.
Legacy Configuration Parameters
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Note: parameter names are case-insensitive.
- $mmnormalizeRuleBase <rulebase-file> - equivalent to the "ruleBase"
parameter.
- $mmnormalizeUseRawMsg <on/off> - equivalent to the "useRawMsg"
parameter.
See Also
~~~~~~~~
- `First steps for
mmnormalize <http://www.rsyslog.com/normalizer-first-steps-for-mmnormalize/>`_
- `Log normalization and special
characters <http://www.rsyslog.com/log-normalization-and-special-characters/>`_
- `Log normalization and the leading
space <http://www.rsyslog.com/log-normalization-and-the-leading-space/>`_
- `Using mmnormalize effectively with Adiscon
LogAnalyzer <http://www.rsyslog.com/using-rsyslog-mmnormalize-module-effectively-with-adiscon-loganalyzer/>`_
Caveats/Known Bugs
~~~~~~~~~~~~~~~~~~
None known at this time.
Example
~~~~~~~
**Sample 1:**
In this sample messages are received via imtcp. Then they are normalized with the given rulebase.
After that they are written in a file.
::
module(load="mmnormalize")
module(load="imtcp")
input(type="imtcp" port="10514" ruleset="outp")
ruleset(name="outp") {
action(type="mmnormalize" rulebase="/tmp/rules.rulebase")
action(type="omfile" File="/tmp/output")
}
**Sample 2:**
In this sample messages are received via imtcp. Then they are normalized based on the given rules.
The strings from **rule** are put together and are equal to a rulebase with the same content.
::
module(load="mmnormalize")
module(load="imtcp")
input(type="imtcp" port="10514" ruleset="outp")
ruleset(name="outp") {
action(type="mmnormalize" rule=["rule=:%host:word% %tag:char-to:\\x3a%: no longer listening on %ip:ipv4%#%port:number%", "rule=:%host:word% %ip:ipv4% user was logged out"])
action(type="omfile" File="/tmp/output")
}
**Sample 3:**
This activates the module and applies normalization to all messages:
::
module(load="mmnormalize")
action(type="mmnormalize" ruleBase="/path/to/rulebase.rb")
The same in legacy format:
::
$ModLoad mmnormalize
$mmnormalizeRuleBase /path/to/rulebase.rb
*.* :mmnormalize:
|
