Merging upstream version 1.45.3+dfsg.

Signed-off-by: Daniel Baumann <daniel.baumann@progress-linux.org>

1 files changed, 223 insertions, 0 deletions

diff --git a/src/libnetdata/log/README.md b/src/libnetdata/log/README.md
new file mode 100644
index 000000000..ef9ca1ef3
--- /dev/null
+++ b/src/libnetdata/log/README.md
@@ -0,0 +1,223 @@
+<!--
+title: "Log"
+custom_edit_url: https://github.com/netdata/netdata/edit/master/src/libnetdata/log/README.md
+sidebar_label: "Log"
+learn_status: "Published"
+learn_topic_type: "Tasks"
+learn_rel_path: "Developers/libnetdata"
+-->
+
+# Netdata Logging
+
+This document describes how Netdata generates its own logs, not how Netdata manages and queries logs databases.
+
+## Log sources
+
+Netdata supports the following log sources:
+
+1. **daemon**, logs generated by Netdata daemon.
+2. **collector**, logs generated by Netdata collectors, including internal and external ones.
+3. **access**, API requests received by Netdata
+4. **health**, all alert transitions and notifications
+
+## Log outputs
+
+For each log source, Netdata supports the following output methods:
+
+- **off**, to disable this log source
+- **journal**, to send the logs to systemd-journal.
+- **syslog**, to send the logs to syslog.
+- **system**, to send the output to `stderr` or `stdout` depending on the log source.
+- **stdout**, to write the logs to Netdata's `stdout`.
+- **stderr**, to write the logs to Netdata's `stderr`.
+- **filename**, to send the logs to a file.
+
+For `daemon` and `collector` the default is `journal` when systemd-journal is available.
+To decide if systemd-journal is available, Netdata checks:
+
+1. `stderr` is connected to systemd-journald
+2. `/run/systemd/journal/socket` exists
+3. `/host/run/systemd/journal/socket` exists (`/host` is configurable in containers)
+
+If any of the above is detected, Netdata will select `journal` for `daemon` and `collector` sources.
+
+All other sources default to a file.
+
+## Log formats
+
+| Format  | Description                                                                                            |
+|---------|--------------------------------------------------------------------------------------------------------|
+| journal | journald-specific log format. Automatically selected when logging to systemd-journal.                  |
+| logfmt  | logs data as a series of key/value pairs. The default when logging to any output other than `journal`. |
+| json    | logs data in JSON format.                                                                              |
+
+## Log levels
+
+Each time Netdata logs, it assigns a priority to the log. It can be one of this (in order of importance):
+
+| Level     | Description                                                                            |
+|-----------|----------------------------------------------------------------------------------------|
+| emergency | a fatal condition, Netdata will most likely exit immediately after.                    |
+| alert     | a very important issue that may affect how Netdata operates.                           |
+| critical  | a very important issue the user should know which, Netdata thinks it can survive.      |
+| error     | an error condition indicating that Netdata is trying to do something, but it fails.    |
+| warning   | something unexpected has happened that may or may not affect the operation of Netdata. |
+| notice    | something that does not affect the operation of Netdata, but the user should notice.   |
+| info      | the default log level about information the user should know.                          |
+| debug     | these are more verbose logs that can be ignored.                                       |
+
+## Logs Configuration
+
+In `netdata.conf`, there are the following settings:
+
+```
+[logs]
+	# logs to trigger flood protection = 1000
+	# logs flood protection period = 60
+	# facility = daemon
+	# level = info
+	# daemon = journal
+	# collector = journal
+	# access = /var/log/netdata/access.log
+	# health = /var/log/netdata/health.log
+```
+
+- `logs to trigger flood protection` and `logs flood protection period` enable logs flood protection for `daemon` and `collector` sources. It can also be configured per log source.
+- `facility` is used only when Netdata logs to syslog.
+- `level` defines the minimum [log level](#log-levels) of logs that will be logged. This setting is applied only to `daemon` and `collector` sources. It can also be configured per source.
+
+### Configuring log sources
+
+Each for the sources (`daemon`, `collector`, `access`, `health`), accepts the following: 
+
+```
+source = {FORMAT},level={LEVEL},protection={LOG}/{PERIOD}@{OUTPUT}
+```
+
+Where:
+
+- `{FORMAT}`, is one of the [log formats](#log-formats),
+- `{LEVEL}`, is the minimum [log level](#log-levels) to be logged,
+- `{LOGS}` is the number of `logs to trigger flood protection` configured per output,
+- `{PERIOD}` is the equivalent of `logs flood protection period` configured per output,
+- `{OUTPUT}` is one of the `[log outputs](#log-outputs),
+
+All parameters can be omitted, except `{OUTPUT}`. If `{OUTPUT}` is the only given parameter, `@` can be omitted.
+
+### Logs rotation
+
+Netdata comes with `logrotate` configuration to rotate its log files periodically.
+
+The default is usually found in `/etc/logrotate.d/netdata`.
+
+Sending a `SIGHUP` to Netdata, will instruct it to re-open all its log files.
+
+## Log Fields
+
+<details>
+<summary>All fields exposed by Netdata</summary>
+
+|                journal                 |             logfmt             |              json              |                                                Description                                                |
+|:--------------------------------------:|:------------------------------:|:------------------------------:|:---------------------------------------------------------------------------------------------------------:|
+|      `_SOURCE_REALTIME_TIMESTAMP`      |             `time`             |             `time`             |                                        the timestamp of the event                                         |
+|          `SYSLOG_IDENTIFIER`           |             `comm`             |             `comm`             |                                       the program logging the event                                       |
+|            `ND_LOG_SOURCE`             |            `source`            |            `source`            |                                  one of the [log sources](#log-sources)                                   |
+|         `PRIORITY`<br/>numeric         |        `level`<br/>text        |      `level`<br/>numeric       |                                   one of the [log levels](#log-levels)                                    |
+|                `ERRNO`                 |            `errno`             |            `errno`             |                                       the numeric value of `errno`                                        |
+|            `INVOCATION_ID`             |               -                |               -                | a unique UUID of the Netdata session, reset on every Netdata restart, inherited by systemd when available |
+|              `CODE_LINE`               |               -                |               -                |                         the line number of of the source code logging this event                          |
+|              `CODE_FILE`               |               -                |               -                |                            the filename of the source code logging this event                             |
+|            `CODE_FUNCTION`             |               -                |               -                |                          the function name of the source code logging this event                          |
+|                 `TID`                  |             `tid`              |             `tid`              |                              the thread id of the thread logging this event                               |
+|              `THREAD_TAG`              |            `thread`            |            `thread`            |                                 the name of the thread logging this event                                 |
+|              `MESSAGE_ID`              |            `msg_id`            |            `msg_id`            |                                      see [message IDs](#message-ids)                                      |
+|              `ND_MODULE`               |            `module`            |            `module`            |                                   the Netdata module logging this event                                   |
+|             `ND_NIDL_NODE`             |             `node`             |             `node`             |                             the hostname of the node the event is related to                              |
+|           `ND_NIDL_INSTANCE`           |           `instance`           |           `instance`           |                             the instance of the node the event is related to                              |
+|           `ND_NIDL_CONTEXT`            |           `context`            |           `context`            |    the context the event is related to (this is usually the chart name, as shown on netdata dashboards    |
+|          `ND_NIDL_DIMENSION`           |          `dimension`           |          `dimension`           |                                   the dimension the event is related to                                   |
+|           `ND_SRC_TRANSPORT`           |        `src_transport`         |        `src_transport`         |                  when the event happened during a request, this is the request transport                  |
+|              `ND_SRC_IP`               |            `src_ip`            |            `src_ip`            |          when the event happened during an inbound request, this is the IP the request came from          |
+|             `ND_SRC_PORT`              |           `src_port`           |           `src_port`           |         when the event happened during an inbound request, this is the port the request came from         |
+|        `ND_SRC_FORWARDED_HOST`         |      `src_forwarded_host`      |      `src_forwarded_host`      |                            the contents of the HTTP header `X-Forwarded-Host`                             |
+|         `ND_SRC_FORWARDED_FOR`         |      `src_forwarded_for`       |      `src_forwarded_for`       |                             the contents of the HTTP header `X-Forwarded-For`                             |
+|         `ND_SRC_CAPABILITIES`          |       `src_capabilities`       |       `src_capabilities`       |          when the request came from a child, this is the communication capabilities of the child          |
+|           `ND_DST_TRANSPORT`           |        `dst_transport`         |        `dst_transport`         |        when the event happened during an outbound request, this is the outbound request transport         |
+|              `ND_DST_IP`               |            `dst_ip`            |            `dst_ip`            |        when the event happened during an outbound request, this is the IP the request destination         |
+|             `ND_DST_PORT`              |           `dst_port`           |           `dst_port`           |       when the event happened during an outbound request, this is the port the request destination        |
+|         `ND_DST_CAPABILITIES`          |       `dst_capabilities`       |       `dst_capabilities`       |          when the request goes to a parent, this is the communication capabilities of the parent          |
+|          `ND_REQUEST_METHOD`           |          `req_method`          |          `req_method`          |      when the event happened during an inbound request, this is the method the request was received       |
+|           `ND_RESPONSE_CODE`           |             `code`             |             `code`             |                         when responding to a request, this this the response code                         |
+|           `ND_CONNECTION_ID`           |             `conn`             |             `conn`             |            when there is a connection id for an inbound connection, this is the connection id             |
+|          `ND_TRANSACTION_ID`           |         `transaction`          |         `transaction`          |                               the transaction id (UUID) of all API requests                               |
+|        `ND_RESPONSE_SENT_BYTES`        |          `sent_bytes`          |          `sent_bytes`          |                                    the bytes we sent to API responses                                     |
+|        `ND_RESPONSE_SIZE_BYTES`        |          `size_bytes`          |          `size_bytes`          |                                the uncompressed bytes of the API responses                                |
+|      `ND_RESPONSE_PREP_TIME_USEC`      |           `prep_ut`            |           `prep_ut`            |                                   the time needed to prepare a response                                   |
+|      `ND_RESPONSE_SENT_TIME_USEC`      |           `sent_ut`            |           `sent_ut`            |                                    the time needed to send a response                                     |
+|     `ND_RESPONSE_TOTAL_TIME_USEC`      |           `total_ut`           |           `total_ut`           |                               the total time needed to complete a response                                |
+|             `ND_ALERT_ID`              |           `alert_id`           |           `alert_id`           |                                   the alert id this event is related to                                   |
+|          `ND_ALERT_EVENT_ID`           |        `alert_event_id`        |        `alert_event_id`        |                          a sequential number of the alert transition (per host)                           |
+|          `ND_ALERT_UNIQUE_ID`          |       `alert_unique_id`        |       `alert_unique_id`        |                          a sequential number of the alert transition (per alert)                          |
+|        `ND_ALERT_TRANSITION_ID`        |     `alert_transition_id`      |     `alert_transition_id`      |                                 the unique UUID of this alert transition                                  |
+|           `ND_ALERT_CONFIG`            |         `alert_config`         |         `alert_config`         |                                    the alert configuration hash (UUID)                                    |
+|            `ND_ALERT_NAME`             |            `alert`             |            `alert`             |                                              the alert name                                               |
+|            `ND_ALERT_CLASS`            |         `alert_class`          |         `alert_class`          |                                         the alert classification                                          |
+|          `ND_ALERT_COMPONENT`          |       `alert_component`        |       `alert_component`        |                                            the alert component                                            |
+|            `ND_ALERT_TYPE`             |          `alert_type`          |          `alert_type`          |                                              the alert type                                               |
+|            `ND_ALERT_EXEC`             |          `alert_exec`          |          `alert_exec`          |                                      the alert notification program                                       |
+|          `ND_ALERT_RECIPIENT`          |       `alert_recipient`        |       `alert_recipient`        |                                          the alert recipient(s)                                           |
+|            `ND_ALERT_VALUE`            |         `alert_value`          |         `alert_value`          |                                          the current alert value                                          |
+|          `ND_ALERT_VALUE_OLD`          |       `alert_value_old`        |       `alert_value_old`        |                                         the previous alert value                                          |
+|           `ND_ALERT_STATUS`            |         `alert_status`         |         `alert_status`         |                                         the current alert status                                          |
+|         `ND_ALERT_STATUS_OLD`          |       `alert_value_old`        |       `alert_value_old`        |                                         the previous alert value                                          |
+|            `ND_ALERT_UNITS`            |         `alert_units`          |         `alert_units`          |                                          the units of the alert                                           |
+|           `ND_ALERT_SUMMARY`           |        `alert_summary`         |        `alert_summary`         |                                       the summary text of the alert                                       |
+|            `ND_ALERT_INFO`             |          `alert_info`          |          `alert_info`          |                                        the info text of the alert                                         |
+|          `ND_ALERT_DURATION`           |        `alert_duration`        |        `alert_duration`        |                             the duration the alert was in its previous state                              |
+| `ND_ALERT_NOTIFICATION_TIMESTAMP_USEC` | `alert_notification_timestamp` | `alert_notification_timestamp` |                           the timestamp the notification delivery is scheduled                            |
+|              `ND_REQUEST`              |           `request`            |           `request`            |                             the full request during which the event happened                              |
+|               `MESSAGE`                |             `msg`              |             `msg`              |                                             the event message                                             |
+
+</details>
+
+### Message IDs
+
+Netdata assigns specific message IDs to certain events:
+
+- `ed4cdb8f1beb4ad3b57cb3cae2d162fa` when a Netdata child connects to this Netdata
+- `6e2e3839067648968b646045dbf28d66` when this Netdata connects to a Netdata parent
+- `9ce0cb58ab8b44df82c4bf1ad9ee22de` when alerts change state
+- `6db0018e83e34320ae2a659d78019fb7` when notifications are sent
+
+You can view these events using the Netdata systemd-journal.plugin at the `MESSAGE_ID` filter,
+or using `journalctl` like this:
+
+```bash
+# query children connection
+journalctl MESSAGE_ID=ed4cdb8f1beb4ad3b57cb3cae2d162fa
+
+# query parent connection
+journalctl MESSAGE_ID=6e2e3839067648968b646045dbf28d66
+
+# query alert transitions
+journalctl MESSAGE_ID=9ce0cb58ab8b44df82c4bf1ad9ee22de
+
+# query alert notifications
+journalctl MESSAGE_ID=6db0018e83e34320ae2a659d78019fb7
+```
+
+## Using journalctl to query Netdata logs
+
+The Netdata service's processes execute within the `netdata` journal namespace. To view the Netdata logs, you should
+specify the `--namespace=netdata` option.
+
+```bash
+# Netdata logs since the last time the service was started
+journalctl _SYSTEMD_INVOCATION_ID="$(systemctl show --value --property=InvocationID netdata)" --namespace=netdata
+
+# All netdata logs, the oldest entries are displayed first  
+journalctl -u netdata --namespace=netdata
+
+# All netdata logs, the newest entries are displayed first  
+journalctl -u netdata --namespace=netdata -r
+```