1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
|
# Architecture
This document covers the internal architecture of the Logfile Navigator (lnav),
a terminal-based tool for viewing and analyzing log files.
## Goals
The following goals drive the design and implementation of lnav:
- Don't make the user do something that can be done automatically.
Example: Automatically detect log formats for files instead of making them
specify the format for each file.
- Be performant on low-spec hardware.
Example: Prefer single-threaded optimizations over trying to parallelize
- Operations should be "live" and not block the user from continuing to work.
Example: Searches are run in the background.
- Provide context-sensitive help.
Example: When the cursor is over a SQL keyword/function, the help text for
that is shown above.
- Show a preview of operations so the user knows what is going to happen.
Example: When entering a `:filter-out` command, the matched parts of the
lines are highlighted in red.
## Overview
The whole of lnav consists of a
[log file parser](https://docs.lnav.org/en/latest/formats.html),
[text UI](https://docs.lnav.org/en/latest/ui.html),
[integrations with SQLite](https://docs.lnav.org/en/latest/sqlext.html),
[command-line interface](https://docs.lnav.org/en/latest/cli.html), and
[commands for operating on logs](https://docs.lnav.org/en/latest/commands.html).
Since the majority of lnav's operations center around logs, the core
data-structure is the combined log message index. The message index is populated
when new messages are read from log files. The text UI displays a subset of
messages from the index. The SQLite virtual-tables allow for programmatic access
to the messages and lnav's internal state.
[](https://whimsical.com/lnav-architecture-UM594Qo4G3nt2XWaSZA1mh)
## File Monitoring
Each file being monitored by lnav has an associated [`logfile`](src/logfile.hh)
object, be they plaintext files or files with a recognized format. These
objects are periodically polled by the main event loop to check if the file
was deleted, truncated, or new lines added. While reading new lines, if no
log format has matched yet, each line will be passed through the log format
regular expressions to try and find a match. Each line that is read is added
to an index
#### Why is `mmap()` not used?
Note that file contents are consumed using `pread(2)`/`read(2)` and not
`mmap(2)` since `mmap(2)` does not react well to files changing out from
underneath it. For example, a truncated file would likely result in a
`SIGBUS`.
## Log Messages
As files are being indexed, if a matching format is found, the file is
"promoted" from a plaintext file to a log file. When the file is promoted,
it is added to the [logfile_sub_source](src/logfile_sub_source.hh), which
collates all log messages together into a single index.
### Timestamp Parsing
Since all log messages need to have a timestamp, timestamp parsing needs to be
very efficient. The standard `strptime()` function is quite expensive, so lnav
includes an optimized custom parser and code-generator in the
[ptimec](src/ptimec.hh) component. The code-generator is used at compile-time
to generate parsers for several [common formats](src/time_formats.am).
## Log Formats
[log_format](src/log_format.hh) instances are used to parse lines from files
into `logline` objects. The majority of log formats are
[external_log_format](src/log_format_ext.hh) objects that are create from
[JSON format definitions](https://docs.lnav.org/en/latest/formats.html). The
built-in definitions are located in the [formats](src/formats) directory. Log
formats that cannot be handled through a simple regular expression are
implemented in the [log_format_impls.cc](src/log_format_impls.cc) file.
## User Interface
The lnav text-user-interface is built on top of
[ncurses](https://invisible-island.net/ncurses/announce.html).
However, the higher-level functionality of panels, widgets, and such is not
used. Instead, the following custom components are built on top of the ncurses
primitives:
- [view_curses](src/view_curses.hh) - Provides the basics for text roles, which
allows for themes to color and style text. The `mvwattrline()` function does
all the heavy lifting of drawing ["attributed" lines](src/base/attr_line.hh),
which are strings that have attributes associated with a given range of
characters.
- [listview_curses](src/listview_curses.hh) - Displays a list of items that are
provided by a source.
- [textview_curses](src/textview_curses.hh) - Builds on the list view by adding
support for searching, filtering, bookmarks, etc... The main panel that
displays the logs/plaintext/help is a textview.
- [statusview_curses](src/state-extension-functions.cc) - Draws the status bars
at the top and bottom of the TUI.
- [vt52_curses](src/vt52_curses.hh) - Adapts vt52 escape codes to the ncurses
API.
- [readline_curses](src/readline_curses.hh) - Provides access to the readline
library. The readline code is executed in a child process since readline
does not get along with ncurses. The child process and readline is set to
use a vt52 terminal and the vt52_curses view is uses to translate those
escape codes to ncurses.
The following diagram shows the underlying components that make up the TUI:
[](https://whimsical.com/lnav-tui-MQjXc7Vx23BxQTHrnuNp5F)
|
