diff options
Diffstat (limited to '')
| -rw-r--r-- | ARCHITECTURE.md | 121 |
1 files changed, 121 insertions, 0 deletions
diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md new file mode 100644 index 0000000..3ef11cb --- /dev/null +++ b/ARCHITECTURE.md @@ -0,0 +1,121 @@ +# Architecture + +This document covers the internal architecture of the Logfile Navigator (lnav), +a terminal-based tool for viewing and analyzing log files. + +## Goals + +The following goals drive the design and implementation of lnav: + +- Don't make the user do something that can be done automatically. + + Example: Automatically detect log formats for files instead of making them + specify the format for each file. + +- Be performant on low-spec hardware. + + Example: Prefer single-threaded optimizations over trying to parallelize + +- Operations should be "live" and not block the user from continuing to work. + + Example: Searches are run in the background. + +- Provide context-sensitive help. + + Example: When the cursor is over a SQL keyword/function, the help text for + that is shown above. + +- Show a preview of operations so the user knows what is going to happen. + + Example: When entering a `:filter-out` command, the matched parts of the + lines are highlighted in red. + +## Overview + +The whole of lnav consists of a +[log file parser](https://docs.lnav.org/en/latest/formats.html), +[text UI](https://docs.lnav.org/en/latest/ui.html), +[integrations with SQLite](https://docs.lnav.org/en/latest/sqlext.html), +[command-line interface](https://docs.lnav.org/en/latest/cli.html), and +[commands for operating on logs](https://docs.lnav.org/en/latest/commands.html). +Since the majority of lnav's operations center around logs, the core +data-structure is the combined log message index. The message index is populated +when new messages are read from log files. The text UI displays a subset of +messages from the index. The SQLite virtual-tables allow for programmatic access +to the messages and lnav's internal state. + +[](https://whimsical.com/lnav-architecture-UM594Qo4G3nt2XWaSZA1mh) + +## File Monitoring + +Each file being monitored by lnav has an associated [`logfile`](src/logfile.hh) +object, be they plaintext files or files with a recognized format. These +objects are periodically polled by the main event loop to check if the file +was deleted, truncated, or new lines added. While reading new lines, if no +log format has matched yet, each line will be passed through the log format +regular expressions to try and find a match. Each line that is read is added +to an index + +#### Why is `mmap()` not used? + +Note that file contents are consumed using `pread(2)`/`read(2)` and not +`mmap(2)` since `mmap(2)` does not react well to files changing out from +underneath it. For example, a truncated file would likely result in a +`SIGBUS`. + +## Log Messages + +As files are being indexed, if a matching format is found, the file is +"promoted" from a plaintext file to a log file. When the file is promoted, +it is added to the [logfile_sub_source](src/logfile_sub_source.hh), which +collates all log messages together into a single index. + +### Timestamp Parsing + +Since all log messages need to have a timestamp, timestamp parsing needs to be +very efficient. The standard `strptime()` function is quite expensive, so lnav +includes an optimized custom parser and code-generator in the +[ptimec](src/ptimec.hh) component. The code-generator is used at compile-time +to generate parsers for several [common formats](src/time_formats.am). + +## Log Formats + +[log_format](src/log_format.hh) instances are used to parse lines from files +into `logline` objects. The majority of log formats are +[external_log_format](src/log_format_ext.hh) objects that are create from +[JSON format definitions](https://docs.lnav.org/en/latest/formats.html). The +built-in definitions are located in the [formats](src/formats) directory. Log +formats that cannot be handled through a simple regular expression are +implemented in the [log_format_impls.cc](src/log_format_impls.cc) file. + +## User Interface + +The lnav text-user-interface is built on top of +[ncurses](https://invisible-island.net/ncurses/announce.html). +However, the higher-level functionality of panels, widgets, and such is not +used. Instead, the following custom components are built on top of the ncurses +primitives: + +- [view_curses](src/view_curses.hh) - Provides the basics for text roles, which + allows for themes to color and style text. The `mvwattrline()` function does + all the heavy lifting of drawing ["attributed" lines](src/base/attr_line.hh), + which are strings that have attributes associated with a given range of + characters. +- [listview_curses](src/listview_curses.hh) - Displays a list of items that are + provided by a source. +- [textview_curses](src/textview_curses.hh) - Builds on the list view by adding + support for searching, filtering, bookmarks, etc... The main panel that + displays the logs/plaintext/help is a textview. +- [statusview_curses](src/state-extension-functions.cc) - Draws the status bars + at the top and bottom of the TUI. +- [vt52_curses](src/vt52_curses.hh) - Adapts vt52 escape codes to the ncurses + API. +- [readline_curses](src/readline_curses.hh) - Provides access to the readline + library. The readline code is executed in a child process since readline + does not get along with ncurses. The child process and readline is set to + use a vt52 terminal and the vt52_curses view is uses to translate those + escape codes to ncurses. + +The following diagram shows the underlying components that make up the TUI: + +[](https://whimsical.com/lnav-tui-MQjXc7Vx23BxQTHrnuNp5F) |