1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
|
*******************************
imkafka: read from Apache Kafka
*******************************
=========================== ===========================================================================
**Module Name:** **imkafka**
**Author:** Andre Lorbach <alorbach@adiscon.com>
**Available since:** 8.27.0
=========================== ===========================================================================
Purpose
=======
The imkafka plug-in implements an Apache Kafka consumer, permitting
rsyslog to receive data from Kafka.
Configuration Parameters
========================
Note that imkafka supports some *Array*-type parameters. While the parameter
name can only be set once, it is possible to set multiple values with that
single parameter.
For example, to select a broker, you can use
.. code-block:: none
input(type="imkafka" topic="mytopic" broker="localhost:9092" consumergroup="default")
which is equivalent to
.. code-block:: none
input(type="imkafka" topic="mytopic" broker=["localhost:9092"] consumergroup="default")
To specify multiple values, just use the bracket notation and create a
comma-delimited list of values as shown here:
.. code-block:: none
input(type="imkafka" topic="mytopic"
broker=["localhost:9092",
"localhost:9093",
"localhost:9094"]
)
.. note::
Parameter names are case-insensitive.
Module Parameters
-----------------
Currently none.
Action Parameters
-----------------
Broker
^^^^^^
.. csv-table::
:header: "type", "default", "mandatory", "|FmtObsoleteName| directive"
:widths: auto
:class: parameter-table
"array", "localhost:9092", "no", "none"
Specifies the broker(s) to use.
Topic
^^^^^
.. csv-table::
:header: "type", "default", "mandatory", "|FmtObsoleteName| directive"
:widths: auto
:class: parameter-table
"string", "none", "yes", "none"
Specifies the topic to produce to.
ConfParam
^^^^^^^^^
.. csv-table::
:header: "type", "default", "mandatory", "|FmtObsoleteName| directive"
:widths: auto
:class: parameter-table
"array", "none", "no", "none"
Permits to specify Kafka options. Rather than offering a myriad of
config settings to match the Kafka parameters, we provide this setting
here as a vehicle to set any Kafka parameter. This has the big advantage
that Kafka parameters that come up in new releases can immediately be used.
Note that we use librdkafka for the Kafka connection, so the parameters
are actually those that librdkafka supports. As of our understanding, this
is a superset of the native Kafka parameters.
ConsumerGroup
^^^^^^^^^^^^^
.. csv-table::
:header: "type", "default", "mandatory", "|FmtObsoleteName| directive"
:widths: auto
:class: parameter-table
"string", "none", "no", "none"
With this parameter the group.id for the consumer is set. All consumers
sharing the same group.id belong to the same group.
Ruleset
^^^^^^^
.. csv-table::
:header: "type", "default", "mandatory", "|FmtObsoleteName| directive"
:widths: auto
:class: parameter-table
"string", "none", "no", "none"
Specifies the ruleset to be used.
ParseHostname
^^^^^^^^^^^^^
.. csv-table::
:header: "type", "default", "mandatory", "|FmtObsoleteName| directive"
:widths: auto
:class: parameter-table
"binary", "off", "no", "none"
.. versionadded:: 8.38.0
If this parameter is set to on, imkafka will parse the hostname in log
if it exists. The result can be retrieved from $hostname. If it's off,
for compatibility reasons, the local hostname is used, same as the previous
version.
Caveats/Known Bugs
==================
- currently none
Examples
========
Example 1
---------
In this sample a consumer for the topic static is created and will forward the messages to the omfile action.
.. code-block:: none
module(load="imkafka")
input(type="imkafka" topic="static" broker="localhost:9092"
consumergroup="default" ruleset="pRuleset")
ruleset(name="pRuleset") {
action(type="omfile" file="path/to/file")
}
|
