summaryrefslogtreecommitdiffstats
path: root/libnetdata/log/systemd-cat-native.md
diff options
context:
space:
mode:
authorDaniel Baumann <daniel.baumann@progress-linux.org>2024-04-19 02:57:58 +0000
committerDaniel Baumann <daniel.baumann@progress-linux.org>2024-04-19 02:57:58 +0000
commitbe1c7e50e1e8809ea56f2c9d472eccd8ffd73a97 (patch)
tree9754ff1ca740f6346cf8483ec915d4054bc5da2d /libnetdata/log/systemd-cat-native.md
parentInitial commit. (diff)
downloadnetdata-upstream/1.44.3.tar.xz
netdata-upstream/1.44.3.zip
Adding upstream version 1.44.3.upstream/1.44.3upstream
Signed-off-by: Daniel Baumann <daniel.baumann@progress-linux.org>
Diffstat (limited to '')
-rw-r--r--libnetdata/log/systemd-cat-native.md209
1 files changed, 209 insertions, 0 deletions
diff --git a/libnetdata/log/systemd-cat-native.md b/libnetdata/log/systemd-cat-native.md
new file mode 100644
index 00000000..b0b15f40
--- /dev/null
+++ b/libnetdata/log/systemd-cat-native.md
@@ -0,0 +1,209 @@
+# systemd-cat-native
+
+`systemd` includes a utility called `systemd-cat`. This utility reads log lines from its standard input and sends them
+to the local systemd journal. Its key limitation is that despite the fact that systemd journals support structured logs,
+this command does not support sending structured logs to it.
+
+`systemd-cat-native` is a Netdata supplied utility to push structured logs to systemd journals. Key features:
+
+- reads [Journal Export Format](https://systemd.io/JOURNAL_EXPORT_FORMATS/) formatted log entries
+- converts text fields into binary journal multiline log fields
+- sends logs to any of these:
+ - local default `systemd-journald`,
+ - local namespace `systemd-journald`,
+ - remote `systemd-journal-remote` using HTTP or HTTPS, the same way `systemd-journal-upload` does.
+- is the standard external logger of Netdata shell scripts
+
+## Simple use:
+
+```bash
+printf "MESSAGE=hello world\nPRIORITY=6\n\n" | systemd-cat-native
+```
+
+The result:
+
+![image](https://github.com/netdata/netdata/assets/2662304/689d5e03-97ee-40a8-a690-82b7710cef7c)
+
+
+Sending `PRIORITY=3` (error):
+
+```bash
+printf "MESSAGE=hey, this is error\nPRIORITY=3\n\n" | systemd-cat-native
+```
+
+The result:
+![image](https://github.com/netdata/netdata/assets/2662304/faf3eaa5-ac56-415b-9de8-16e6ceed9280)
+
+Sending multi-line log entries (in this example we replace the text `--NEWLINE--` with a newline in the log entry):
+
+```bash
+printf "MESSAGE=hello--NEWLINE--world\nPRIORITY=6\n\n" | systemd-cat-native --newline='--NEWLINE--'
+```
+
+The result:
+
+![image](https://github.com/netdata/netdata/assets/2662304/d6037b4a-87da-4693-ae67-e07df0decdd9)
+
+
+Processing the standard `\n` string can be tricky due to shell escaping. This works, but note that
+we have to add a lot of backslashes to printf.
+
+```bash
+printf "MESSAGE=hello\\\\nworld\nPRIORITY=6\n\n" | systemd-cat-native --newline='\n'
+```
+
+`systemd-cat-native` needs to receive it like this for newline processing to work:
+
+```bash
+# printf "MESSAGE=hello\\\\nworld\nPRIORITY=6\n\n"
+MESSAGE=hello\nworld
+PRIORITY=6
+
+```
+
+## Best practices
+
+These are the rules about fields, enforced by `systemd-journald`:
+
+- field names can be up to **64 characters**,
+- field values can be up to **48k characters**,
+- the only allowed field characters are **A-Z**, **0-9** and **underscore**,
+- the **first** character of fields cannot be a **digit**
+- **protected** journal fields start with underscore:
+ * they are accepted by `systemd-journal-remote`,
+ * they are **NOT** accepted by a local `systemd-journald`.
+
+For best results, always include these fields:
+
+- `MESSAGE=TEXT`<br/>
+ The `MESSAGE` is the body of the log entry.
+ This field is what we usually see in our logs.
+
+- `PRIORITY=NUMBER`<br/>
+ `PRIORITY` sets the severity of the log entry.<br/>
+ `0=emerg, 1=alert, 2=crit, 3=err, 4=warn, 5=notice, 6=info, 7=debug`
+ - Emergency events (0) are usually broadcast to all terminals.
+ - Emergency, alert, critical, and error (0-3) are usually colored red.
+ - Warning (4) entries are usually colored yellow.
+ - Notice (5) entries are usually bold or have a brighter white color.
+ - Info (6) entries are the default.
+ - Debug (7) entries are usually grayed or dimmed.
+
+- `SYSLOG_IDENTIFIER=NAME`<br/>
+ `SYSLOG_IDENTIFIER` sets the name of application.
+ Use something descriptive, like: `SYSLOG_IDENTIFIER=myapp`
+
+You can find the most common fields at `man systemd.journal-fields`.
+
+
+## Usage
+
+```
+Netdata systemd-cat-native v1.43.0-333-g5af71b875
+
+This program reads from its standard input, lines in the format:
+
+KEY1=VALUE1\n
+KEY2=VALUE2\n
+KEYN=VALUEN\n
+\n
+
+and sends them to systemd-journal.
+
+ - Binary journal fields are not accepted at its input
+ - Binary journal fields can be generated after newline processing
+ - Messages have to be separated by an empty line
+ - Keys starting with underscore are not accepted (by journald)
+ - Other rules imposed by systemd-journald are imposed (by journald)
+
+Usage:
+
+ systemd-cat-native
+ [--newline=STRING]
+ [--log-as-netdata|-N]
+ [--namespace=NAMESPACE] [--socket=PATH]
+ [--url=URL [--key=FILENAME] [--cert=FILENAME] [--trust=FILENAME|all]]
+
+The program has the following modes of logging:
+
+ * Log to a local systemd-journald or stderr
+
+ This is the default mode. If systemd-journald is available, logs will be
+ sent to systemd, otherwise logs will be printed on stderr, using logfmt
+ formatting. Options --socket and --namespace are available to configure
+ the journal destination:
+
+ --socket=PATH
+ The path of a systemd-journald UNIX socket.
+ The program will use the default systemd-journald socket when this
+ option is not used.
+
+ --namespace=NAMESPACE
+ The name of a configured and running systemd-journald namespace.
+ The program will produce the socket path based on its internal
+ defaults, to send the messages to the systemd journal namespace.
+
+ * Log as Netdata, enabled with --log-as-netdata or -N
+
+ In this mode the program uses environment variables set by Netdata for
+ the log destination. Only log fields defined by Netdata are accepted.
+ If the environment variables expected by Netdata are not found, it
+ falls back to stderr logging in logfmt format.
+
+ * Log to a systemd-journal-remote TCP socket, enabled with --url=URL
+
+ In this mode, the program will directly sent logs to a remote systemd
+ journal (systemd-journal-remote expected at the destination)
+ This mode is available even when the local system does not support
+ systemd, or even it is not Linux, allowing a remote Linux systemd
+ journald to become the logs database of the local system.
+
+ Unfortunately systemd-journal-remote does not accept compressed
+ data over the network, so the stream will be uncompressed.
+
+ --url=URL
+ The destination systemd-journal-remote address and port, similarly
+ to what /etc/systemd/journal-upload.conf accepts.
+ Usually it is in the form: https://ip.address:19532
+ Both http and https URLs are accepted. When using https, the
+ following additional options are accepted:
+
+ --key=FILENAME
+ The filename of the private key of the server.
+ The default is: /etc/ssl/private/journal-upload.pem
+
+ --cert=FILENAME
+ The filename of the public key of the server.
+ The default is: /etc/ssl/certs/journal-upload.pem
+
+ --trust=FILENAME | all
+ The filename of the trusted CA public key.
+ The default is: /etc/ssl/ca/trusted.pem
+ The keyword 'all' can be used to trust all CAs.
+
+ --namespace=NAMESPACE
+ Set the namespace of the messages sent.
+
+ --keep-trying
+ Keep trying to send the message, if the remote journal is not there.
+
+ NEWLINES PROCESSING
+ systemd-journal logs entries may have newlines in them. However the
+ Journal Export Format uses binary formatted data to achieve this,
+ making it hard for text processing.
+
+ To overcome this limitation, this program allows single-line text
+ formatted values at its input, to be binary formatted multi-line Journal
+ Export Format at its output.
+
+ To achieve that it allows replacing a given string to a newline.
+ The parameter --newline=STRING allows setting the string to be replaced
+ with newlines.
+
+ For example by setting --newline='--NEWLINE--', the program will replace
+ all occurrences of --NEWLINE-- with the newline character, within each
+ VALUE of the KEY=VALUE lines. Once this this done, the program will
+ switch the field to the binary Journal Export Format before sending the
+ log event to systemd-journal.
+
+``` \ No newline at end of file