diff options
Diffstat (limited to '')
| -rw-r--r-- | docs/JOURNAL_FILE_FORMAT.md | 90 |
1 files changed, 44 insertions, 46 deletions
diff --git a/docs/JOURNAL_FILE_FORMAT.md b/docs/JOURNAL_FILE_FORMAT.md index e0737c5..7d3b039 100644 --- a/docs/JOURNAL_FILE_FORMAT.md +++ b/docs/JOURNAL_FILE_FORMAT.md @@ -8,8 +8,9 @@ SPDX-License-Identifier: LGPL-2.1-or-later # Journal File Format _Note that this document describes the binary on-disk format of journals only. -For interfacing with web technologies there's the [Journal JSON Format](JOURNAL_EXPORT_FORMATS.md#journal-json-format). -For transfer of journal data across the network there's the [Journal Export Format](JOURNAL_EXPORT_FORMATS.md#journal-export-format)._ +For interfacing with web technologies there's the [Journal JSON Format](JOURNAL_EXPORT_FORMATS#journal-json-format). +For transfer of journal data across the network there's the +[Journal Export Format](JOURNAL_EXPORT_FORMATS#journal-export-format)._ The systemd journal stores log data in a binary format with several features: @@ -20,65 +21,62 @@ The systemd journal stores log data in a binary format with several features: * Support for in-line compression * Support for in-line Forward Secure Sealing -This document explains the basic structure of the file format on disk. We are -making this available primarily to allow review and provide documentation. Note -that the actual implementation in the [systemd -codebase](https://github.com/systemd/systemd/blob/main/src/libsystemd/sd-journal/) is the -only ultimately authoritative description of the format, so if this document -and the code disagree, the code is right. That said we'll of course try hard to -keep this document up-to-date and accurate. - -Instead of implementing your own reader or writer for journal files we ask you -to use the [Journal's native C -API](https://www.freedesktop.org/software/systemd/man/sd-journal.html) to access -these files. It provides you with full access to the files, and will not -withhold any data. If you find a limitation, please ping us and we might add -some additional interfaces for you. - -If you need access to the raw journal data in serialized stream form without C -API our recommendation is to make use of the [Journal Export -Format](https://systemd.io/JOURNAL_EXPORT_FORMATS#journal-export-format), which you can -get via `journalctl -o export` or via `systemd-journal-gatewayd`. The export -format is much simpler to parse, but complete and accurate. Due to its -stream-based nature it is not indexed. - -_Or, to put this in other words: this low-level document is probably not what -you want to use as base of your project. You want our [C -API](https://www.freedesktop.org/software/systemd/man/sd-journal.html) instead! +This document explains the basic structure of the file format on disk. +We are making this available primarily to allow review and provide documentation. +Note that the actual implementation in the +[systemd codebase](https://github.com/systemd/systemd/blob/main/src/libsystemd/sd-journal/) +is the only ultimately authoritative description of the format, +so if this document and the code disagree, the code is right. +That said we'll of course try hard to keep this document up-to-date and accurate. + +Instead of implementing your own reader or writer for journal files we ask you to use the +[Journal's native CAPI](https://www.freedesktop.org/software/systemd/man/sd-journal.html) +to access these files. +It provides you with full access to the files, and will not withhold any data. +If you find a limitation, please ping us and we might add some additional interfaces for you. + +If you need access to the raw journal data in serialized stream form without C API our recommendation is to make use of the +[Journal Export Format](JOURNAL_EXPORT_FORMATS#journal-export-format), +which you can get via `journalctl -o export` or via `systemd-journal-gatewayd`. +The export format is much simpler to parse, but complete and accurate. +Due to its stream-based nature it is not indexed. + +_Or, to put this in other words: this low-level document is probably not what you want to use as base of your project. +You want our [C API](https://www.freedesktop.org/software/systemd/man/sd-journal.html) instead! And if you really don't want the C API, then you want the -[Journal Export Format or Journal JSON Format](JOURNAL_EXPORT_FORMATS) -instead! This document is primarily for your entertainment and education. +[Journal Export Format or Journal JSON Format](/JOURNAL_EXPORT_FORMATS) instead! +This document is primarily for your entertainment and education. Thank you!_ -This document assumes you have a basic understanding of the journal concepts, -the properties of a journal entry and so on. If not, please go and read up, -then come back! This is a good opportunity to read about the [basic properties -of journal -entries](https://www.freedesktop.org/software/systemd/man/systemd.journal-fields.html), -in particular realize that they may include binary non-text data (though -usually don't), and the same field might have multiple values assigned within -the same entry. - -This document describes the current format of systemd 246. The documented -format is compatible with the format used in the first versions of the journal, +This document assumes you have a basic understanding of the journal concepts, the properties of a journal entry and so on. +If not, please go and read up, then come back! +This is a good opportunity to read about the +[basic properties of journal entries](https://www.freedesktop.org/software/systemd/man/systemd.journal-fields.html), +in particular realize that they may include binary non-text data (though usually don't), +and the same field might have multiple values assigned within the same entry. + +This document describes the current format of systemd 246. +The documented format is compatible with the format used in the first versions of the journal, but received various compatible and incompatible additions since. -If you are wondering why the journal file format has been created in the first -place instead of adopting an existing database implementation, please have a -look [at this -thread](https://lists.freedesktop.org/archives/systemd-devel/2012-October/007054.html). +If you are wondering why the journal file format has been created in the first place instead of adopting an existing database implementation, +please have a look [at this thread](https://lists.freedesktop.org/archives/systemd-devel/2012-October/007054.html). ## Basics * All offsets, sizes, time values, hashes (and most other numeric values) are 32-bit/64-bit unsigned integers in LE format. * Offsets are always relative to the beginning of the file. -* The 64-bit hash function siphash24 is used for newer journal files. For older files [Jenkins lookup3](https://en.wikipedia.org/wiki/Jenkins_hash_function) is used, more specifically `jenkins_hashlittle2()` with the first 32-bit integer it returns as higher 32-bit part of the 64-bit value, and the second one uses as lower 32-bit part. +* The 64-bit hash function siphash24 is used for newer journal files. + For older files [Jenkins lookup3](https://en.wikipedia.org/wiki/Jenkins_hash_function) is used, + more specifically `jenkins_hashlittle2()` with the first 32-bit integer it returns as higher 32-bit part of the 64-bit value, + and the second one uses as lower 32-bit part. * All structures are aligned to 64-bit boundaries and padded to multiples of 64-bit * The format is designed to be read and written via memory mapping using multiple mapped windows. * All time values are stored in usec since the respective epoch. * Wall clock time values are relative to the Unix time epoch, i.e. January 1st, 1970. (`CLOCK_REALTIME`) -* Monotonic time values are always stored jointly with the kernel boot ID value (i.e. `/proc/sys/kernel/random/boot_id`) they belong to. They tend to be relative to the start of the boot, but aren't for containers. (`CLOCK_MONOTONIC`) +* Monotonic time values are always stored jointly with the kernel boot ID value (i.e. `/proc/sys/kernel/random/boot_id`) they belong to. + They tend to be relative to the start of the boot, but aren't for containers. (`CLOCK_MONOTONIC`) * Randomized, unique 128-bit IDs are used in various locations. These are generally UUID v4 compatible, but this is not a requirement. ## General Rules |