diff options
Diffstat (limited to '')
| -rw-r--r-- | NEWS.md | 1000 |
1 files changed, 1000 insertions, 0 deletions
@@ -0,0 +1,1000 @@ +## lnav v0.11.1 + +Features: +* Additional validation checks for log formats have been + added and will result in warnings. Pass `-W` on the + command-line to view the warnings. The following new + check have been added: + - Each regex must have a corresponding sample log message + that it matches. + - Each sample must be matched by only one regex. +* Added built-in support for anonymizing content. The + `:write-*` commands now accept an `--anonymize` option + and there is an `anonymize()` SQL function. The + anonymization process will try to replace identifying + information with random data. For example, IPv4 addresses + are replaced with addresses in the 10.0.0.0/8 range. + (This feature is mainly intended to help with providing + information to lnav support that does not have sensitive + values.) +* Added `parse_url()` and `unparse_url()` SQL functions for + parsing URLs into a JSON object and then back again. Note + that the implementation relies on libcurl which has some + limitations, like not supporting all types of schemes + (e.g. `mailto:`). +* Added the `subsecond-field` and `subsecond-units` log + format properties to allow for specifying a separate + field for the sub-second portion of a timestamp. +* Added a keymap for Swedish keyboards. + +Breaking changes: +* The `regexp_capture()` table-valued-function now returns NULL + instead of an empty string for the `capture_name` column if + the capture is not named. + +Fixes: +* Reduce the "no patterns have a capture" error to a warning + so that it doesn't block lnav from starting up. +* Some ANSI escape sequences will now be removed before testing + regexes against a log message. +* If a line in a JSON-lines log file does not start with a + `{`, it will now be shown as-is and will not have the JSON + parse error. + +Cost of Doing Business: +* Migrated from pcre to pcre2. + +## lnav v0.11.0 + +Features: +* Redesigned the top status area to allow for user-specified + messages and added a second line that displays an interactive + breadcrumb bar. The top status line now shows the clock and + the remaining area displays whatever messages are inserted + into the lnav_user_notifications table. The information that + was originally on top is now in a second line and organized + as breadcrumbs. Pressing `ENTER` will activate the breadcrumb bar + and the left/right cursor keys can be used to select a particular + crumb while the up/down keys can select a value to switch to. + While a crumb is selected, you can also type in some text to do + a fuzzy search on the possibilities or, if the crumb represents + an array of values, enter the index to jump to. +* The pretty-print view will now show breadcrumbs that indicate the + location of the top line in the view with the prettified structure. +* Markdown files (those with a .md extension) are now rendered in the + TEXT view. The breadcrumb bar at the top will also be updated + depending on the section of the document that you are in and you + can use it to jump to different parts of the doc. +* The `:goto` command will now accept anchor links (i.e. `#section-id`) + as an argument when the text file being viewed has sections. You + can also specify an anchor when opening a file by appending + `#<link-name>`. For example, `README.md#screenshot`. +* Log message comments are now treated as markdown and rendered + accordingly in the overlay. Multi-line comments are now supported + as well. +* Metadata embedded in files can now be accessed by the + `lnav_file_metadata` table. Currently, only the front-matter in + Markdown files is supported. +* Added an integration with regex101.com to make it easier to edit + log message regular expressions. Using the new "management CLI" + (activated by the `-m` option), a log format can be created from + a regular expression entry on regex101.com and existing patterns + can be edited. +* In the spectrogram view, the selected value range is now shown by + an overlay that includes a summary of the range and the number of + values that fall in that range. There is also a detail panel at + the bottom that shows the log-messages/DB-rows whose values are in + that range. You can then press TAB to focus on the detail view + and scroll around. +* Add initial support for pcap(3) files using tshark(1). +* SQL statement execution can now be canceled by pressing `CTRL+]` + (same as canceling out of a prompt). +* To make it possible to automate some operations, there is now an + `lnav_events` table that is updated when internal events occur + within lnav (e.g. opening a file, format is detected). You + can then add SQLite `TRIGGER`s to this table that can perform a + task by updating other tables. +* Tags can automatically be added to messages by defining a pattern + in a log format. Under a format definition, add the tag name + into the "tags" object in a format definition. The "pattern" + property specifies the regular expression to match against a line + in a file that matches the format. If a match is found, the tag + will be applied to the log message. To restrict matches to + certain files, you can add a "paths" array whose object elements + contain a "glob" property that will be matched against file names. +* Log messages can now be detected automatically via "watch + expressions". These are SQL expressions that are executed for + each log message. If the expressions evaluates to true, an + event is published to the `lnav_events` table that includes the + message contents. +* Added the `regexp_capture_into_json()` table-valued-function that + is similar to `regexp_capture()`, but returns a single row with a + JSON value for each match instead of a row for each capture. +* Added a `top_meta` column to the lnav_views table that contains + metadata related to the top line in the view. +* Added a `log_opid` hidden column to all log tables that contains + the "operation ID" as specified in the log format. +* Moved the `log_format` column from the all_logs table to a hidden + column on all tables. +* Add format for UniFi gateway. +* Added a `glob` property to search tables defined in log formats + to constrain searches to log messages from files that have a + matching log_path value. +* Initial indexing of large files should be faster. Decompression + and searching for line-endings are now pipelined, so they happen + in a thread that is separate from the regular expression matcher. +* Writing to the clipboard now falls back to OSC 52 escape sequence + if none of the clipboard commands could be detected. Your + terminal software will need to support the sequence and you may + need to explicitly enable it in the terminal. +* Added the `:export-session-to <path>` command that writes the + current session state to a file as a list of commands/SQL + statements. This script file can be executed to restore the + majority of the current state. +* Added the `echoln()` SQL function that behaves similarly to the + `:echo` command, writing its first argument to the current + output. +* Added `encode()` and `decode()` SQL functions for transcoding + blobs or text values using one of the following algorithms: + base64, hex, or uri. +* In regular expressions, capture group names are now semantically + highlighted (e.g. in the capture, `(?<name>\w+)`, "name" would + have a unique color). Also, operations or previews that use + that regular expression will highlight the matched data with + the same color. +* Added an lnav_views_echo table that is a real SQLite table that + you can create TRIGGERs on in order to perform actions when + scrolling in a view. +* Added a `yaml_to_json()` SQL function that converts a YAML + document to the equivalent JSON. + +Breaking Changes: +* Formats definitions are now checked to ensure that values have a + corresponding capture in at least one pattern. +* Added a 'language' column to the lnav_view_filters table that + specifies the language of the 'pattern' column, either 'regex' + or 'sql'. +* Timestamps that do not have a day or month are rewritten to a + full timestamp like YYYY-MM-DD HH:MM:SS. +* Removed the summary overlay at the bottom of the log view that + displayed things like "Error rate" and the time span. It doesn't + seem like anyone used it. +* Removed the `log_msg_instance` column from the logline and search + tables since it causes problems with performance. +* Search tables now search for multiple matches within a message + instead of stopping at the first hit. Each additional match is + returned as a separate row. A `match_index` column has been + added to capture the index of the match within the message. + The table regex is also compiled with the "multiline" flag enabled + so the meaning of the `^` and `$` metacharacters are changed + to match the start/end of a line instead of the start/end of + the entire message string. +* Search tables defined in formats are now constrained to only + match log messages that are in that log format instead of all + log messages. As a benefit, the search table now includes + the columns that are defined as part of the format. +* The lnav_view_filters table will treats the tuple of + (view_name, type, language, pattern) as a `UNIQUE` index and + will raise a conflict error on an `INSERT`. Use `REPLACE INTO` + instead of `INSERT INTO` to ignore conflict error. +* The types of SQL values stored as local variables in scripts + is now preserved when used as bound variables at a later point + in the script. + +Fixes: +* Toggling enabled/disabled filters when there is a SQL expression + no longer causes a crash. +* Fix a crash related to long lines that are word wrapped. +* Multiple SQL statements in a SQL block of a script are now + executed instead of just the first one. +* In cases where there were many colors on screen, some text would + be colored incorrectly. +* The pretty-print view now handles ANSI escape sequences. +* The "overstrike" convention for doing bold and underline is now + supported. (Overstrike is a character followed by a backspace + and then the same character for bold or an underscore for + underline.) +* The `:eval` command now works with searching (using the '/' + prefix). + +## lnav v0.10.1 + +Features: +* Added `:show-only-this-file` command that hides all files except the + one for the top line in the view. +* The `:write-raw-to` command now accepts a `--view` flag that specifies + the source view for the data to write. For example, to write the + results of a SQL query, you would pass `--view=db` to the command. +* The commands used to access the clipboard are now configured through + the "tuning" section of the configuration. +* Added an `lnav_version()` SQL function that returns the current + version string. +* Added basic support for the logfmt file format. Currently, only files + whose lines are entirely logfmt-encoded are supported. The lines + must also contain either a field named `time` or `ts` that contains + the timestamp. +* Added the `logfmt2json()` SQL function to convert a string containing + a logfmt-encoded message into a JSON object that can be operated on + more easily. +* Added the `gzip()` and `gunzip()` SQL functions to compress values + into a blob and decompress a blob into a string. +Interface changes: +* The xclip implementation for accessing the system clipboard now writes + to the "clipboard" selection instead of the "primary" selection. +* The 'query' bookmark type and `y`/`Y` hotkeys have been removed due to + performance issues and the functionality is probably rarely used. +Bug Fixes: +* The text "send-input" would show up on some terminals instead of + ignoring the escape sequence. This control sequence was only + intended to be used in the test suite. +* Remote file synchronization has been optimized a bit. +* Configuration values loaded from the `~/.lnav/configs` directory + are now included in the default configuration, so they won't be + saved into the `~/.lnav/config.json` user configuration file. +* Key handling in the visual filter editor will no longer swallow + certain key-presses when editing a filter. +* Scrolling performance restored in the SQL view. +* The `:redirect-to` command now works with `/dev/clipboard` +* The field overlay (opened by pressing 'p') now shows `log_time` + for the timestamp field instead of the name defined in the format. +* The search term in the bottom status bar will now update properly + when switching views. +* The "Out-Of-Time-Order Message" overlay will be shown again. +* The tab for the "Files" panel will be highlighted in red if there + is an issue opening a file. +* Overwritten files should be reloaded again. +* The `jget()` SQL function now returns numbers with the correct type. +* The `json_contains()` SQL function now returns false if the first + argument is NULL instead of NULL. +* The local copies of remote files are now cleaned up after a couple + days of the host not being accessed. +* The initial loading and indexing phase has been optimized. + +## lnav v0.10.0 + +Features: +* Files on remote machines can be viewed/tailed if they are accessible + via SSH. The syntax for specifying the host and path is similar to + scp. For example, to view the files in the /var/log directory on the + machine `host1.example.org`: + ```console + $ lnav user@host1.example.org:/var/log + ``` + Note that you must be able to log into the machine without any + interaction. +* Added the `:filter-expr` command to filter log messages based on an SQL + expression. This command allows much greater control over filtering. +* Added the `:mark-expr` command to mark log messages based on an SQL + expression. This command makes it easier to programmatically mark + log messages compared to using SQL. +* Added support for archive files, like zip, and other compression formats, + like xz, when compiled with libarchive. When one of these types of + files is detected, they are unpacked into a temporary directory and + all the files are loaded into lnav. +* Added an `xpath()` table-valued function for extracting values from + strings containing XML snippets. +* Added the `:prompt` command to allow for more customization of prompts. + Combined with a custom keymapping, you can now open a prompt and prefill + it with a given value. For example, a key could be bound to the + following command to open the command prompt with `:filter-in ` + already filled in: + ```lnav + :prompt command : 'filter-in ' + ``` +* Added support for the W3C Extended Log File Format with the name + `w3c_log`. Similarly to the bro log format, the header is used to + determine the columns in a particular file. However, since the columns + can be different between files, the SQL table only has a well-known set + of columns and the remainder are accessible through JSON-objects stored + in columns like `cs_headers` and `sc_headers`. +* Added support for the S3 Access File Format. +* To jump to the first search hit above the top line in a view, you can + press `CTRL+J` instead of `ENTER` in the search prompt. Pressing `ENTER` + will jump to the first hit below the current window. +* Filtering, as a whole, can be now disabled/enabled without affecting + the state of individual filters. This includes text and time-filters + (i.e. `:hide-lines-before`). You can enable/disable filtering by: + pressing `f` in the filter editor UI; executing the `:toggle-filtering` + command; or by doing an `UPDATE` on the "filtering" column of the + `lnav_views` SQLite table. +* Themes can now include definitions for text highlights under: + `/ui/theme-defs/<theme_name>/highlights` +* Added a "grayscale" theme that isn't so colorful. +* Added the `humanize_file_size()` SQL function that converts a numeric size + to a human-friendly string. +* Added the `sparkline()` SQL function that returns a "sparkline" bar made + out of unicode characters. It can be used with a single value or as + an aggregator. +* Added a `log_time_msecs` hidden column to the log tables that returns + the timestamp as the number of milliseconds from the epoch. +* Added an `lnav_top_file()` SQL function that can be used to get the + name of the top line in the top view or NULL if the line did not come + from a file. +* Added a `mimetype` column to the lnav_file table that returns a guess as + to the MIME type of the file contents. +* Added a `content` hidden column to the lnav_file table that can be used + to read the contents of the file. The contents can then be passed to + functions that operate on XML/JSON data, like `xpath()` or `json_tree()`. +* Added an `lnav_top_view` SQL VIEW that returns the row for the top view + in the lnav_views table. +* The `generate_series()` SQLite extension is now included by default. + One change from the standard implementation is that both the start and + stop are required parameters. +* Added the `;.read` SQL command for executing a plain SQL file. +* Added the `-N` flag so that lnav will run without opening the default + syslog file. + +Interface Changes: +* When copying log lines, the file name and time offset will be included + in the copy if they are enabled. +* Log messages that cannot be parsed properly will be given an "invalid" + log level and the invalid portions colored yellow. +* The range_start and range_stop values of the `regexp_capture()` results + now start at 1 instead of zero to match with what the other SQL string + functions expect. +* The `:write-cols-to` command has been renamed to `:write-table-to`. +* The DB view will limit the maximum column width to 120 characters. +* The `:echo` command now evaluates its message to do variable + substitution. +* The `:write-raw-to` command has been changed to write the original + log file content of marked lines. For example, when viewing a JSON + log, the JSON-Line values from the log file will be written to the + output file. The `:write-view-to` command has been added to perform + the previous work of `:write-raw-to` where the raw content of the view + is written to the file. + +Fixes: +* Unicode text can now be entered in prompts. +* The `replicate()` SQL function would cause a crash if the number of + replications was zero. +* Many internal improvements. + +## lnav v0.9.0 + +Features: +* Added support for themes and included a few as well: default, eldar, + monocai, night-owl, solarized-light, and solarized-dark. The theme + can be changed using the `:config` command, like so: + ```lnav + :config /ui/theme night-owl + ``` + Consult the online documentation for defining a new theme at: + https://lnav.readthedocs.io/en/latest/config.html#theme-definitions +* Added support for custom keymaps and included the following: de, fr, + uk, us. The keymap can be changed using the `:config` command, like so: + ```lnav + :config /ui/keymap uk + ``` + Consult the online documentation for defining a new keymap at: + https://lnav.readthedocs.io/en/latest/config.html#keymap-definitions +* The following JSON-Schemas have been published for the log format and + configuration JSON files: + - https://lnav.org/schemas/format-v1.schema.json + - https://lnav.org/schemas/config-v1.schema.json + + Formats should be updated to reference the schema using the `$schema` + property. +* Indexing of new data in log files can now be paused by pressing `=` + and unpaused by pressing it again. The bottom status bar will display + 'Paused' in the right corner while paused. +* CMake is now a supported way to build. +* When viewing data from the standard-input, a symbolic name can be used + to preserve session state. The name can be changed using the + `|rename-stdin` lnav script or by doing an `UPDATE` to the filepath + column of the lnav_file table. For example, to assign the name + "journald", the following SQL statement can be executed in lnav: + ```lnav + ;UPDATE lnav_file SET filepath='journald' WHERE filepath='stdin' + ``` +* The size of the terminal can be accessed in SQL using the `$LINES` and + `$COLS` variables. +* The `raise_error(msg)` SQL function has been added to make it easier to + raise an error in an lnav script to stop execution and notify the user. +* Added the `json_concat()` function to make it easier to append/concatenate + values onto arrays. +* Added the `:write-jsonlines-to` command that writes the result of a SQL + query to a file in the JSON Lines format. + +Interface Changes: +* Data piped into lnav is no longer dumped to the console after exit. + Instead, a file containing the data is left in `.lnav/stdin-captures/` + and a message is printed to the console indicating the file name. +* In time-offset mode, the deltas for messages before the first mark + are now negative instead of relative to the start of the log. +* The $XDG_CONFIG_HOME environment variable (or `~/.config` directory) are + now respected for storing lnav's configuration. If you have an existing + `~/.lnav` directory, that will continue to be used until you move it to + `$XDG_CONFIG_HOME/lnav` or `~/.config/lnav`. +* Removed the `:save-config` command. Changes to the configuration are now + immediately saved. + +Fixes: +* Added 'notice' log level. +* If a `timestamp-format` is used in an element of a `line-format`, the + field name is ignored and a formatted timestamp is always used. +* Ignore stdin when it is connected to `/dev/null`. + +## lnav v0.8.5 + +Features: +* Added a visual filter editor to make it easier to update existing + filters. The editor can be opened by pressing `TAB`. Once the editor + is opened, you can create/delete, enable/disable, and edit the patterns + with hotkeys. +* Added an `lnav_view_filters` SQL table that can be used to + programmatically manipulate filters. +* Added an `lnav_view_filter_stats` SQL table that contains the number of + times a given filter matched a line in the view. +* Added a `log_filters` column to log tables that can be used to see what + filters matched the log message. +* A history of locations in a view is now kept so that you can jump back + to where you were previously using the `{` and `}` keys. The location + history can also be accessed through the `:prev-location` and + `:next-location` commands. +* The `:write-*` commands will now accept `/dev/clipboard` as a file name + that writes to the system clipboard. +* The `:write-to` and `:write-raw-to` commands will now print out comments + and tags attached to the lines. +* Added a `:redirect-to <path>` command to redirect command output to the + given file. This command is mostly useful in scripts where one might + want to redirect all output from commands like `:echo` and `:write-to -` + to a single file. +* If a log file format has multiple patterns for matching log messages, + each pattern is now tried to match a message in a file. Previously, + only one pattern was ever used for an entire file. +* Added haproxy log format from Peter Hoffmann. +* Added `spooky_hash()` and `group_spooky_hash()` SQL functions to + generate a hash of their parameters. +* Added `time_offset` to the `lnav_file` table so that the timestamps in + a file can be adjusted programmatically. + +Interface Changes: +* The auto-complete behavior in the prompt has been modified to fall back + to a fuzzy search if the prefix search finds no matches. For example, + typing in `:fin` and pressing TAB would previously not do anything. + Now, the `:fin` will be completed to `:filter-in ` since that is a + strong fuzzy match. If there are multiple matches, as would happen + with `:dfil`, readline's menu-complete behavior will be engaged and + you can press `TAB` cycle through the options. +* Added `CTRL+F` to toggle the enabled/disabled state of all filters for the + current view. +* The `-r` flag is now for recursively loading files. The functionality + for loading rotated files is now under the `-R` flag. +* The current search term is now shown in the bottom status bar. +* Some initial help text is now shown for the search and SQL prompts to + refresh the memory. +* When entering the `:comment` command for a line with a comment, the + command prompt will be filled in with the existing comment to make + editing easier. +* Hidden fields now show up as a unicode vertical ellipsis (⋮) instead of + three-dot ellipsis to save space. +* Pressing 7/8 will now move to the previous/next minute. +* The `:write-raw-to` command has been changed to write the entire + contents of the current view and a `:write-screen-to` command has been + added to write only the current screen contents. +* Disabled filters are now saved in sessions. +* The `:adjust-log-time` command now accepts relative times as input. + +Fixes: +* The `:write-json-to` command will now pass through JSON cells as their + JSON values instead of a JSON-encoded string. + +## lnav v0.8.4 + +Features: +* Added the `:comment` command that can be used to attach a comment to a + log line. The comment will be displayed below the line, like so: + ``` + 2017-01-01T15:30:00 error: computer is on fire + + This is where it all went wrong + ``` + The `:clear-comment` command will remove the attached comment. Comments + are searchable with the standard search mechanism and they are available + in SQL through the `log_comment` column. +* Added the `:tag`, `:untag`, and `:delete-tags` commands that can be used + to attach/detach tags on the top log line and delete all instances of + a tag. Tags are also searchable and are available in SQL as a JSON + array in the `log_tags` column. +* Pressing left-arrow while viewing log messages will reveal the source + file name for each line and the unique parts of the source path. + Pressing again will reveal the full path. +* The file name section of the top status line will show only the unique + parts of the log file path if there is not enough room to show the full + path. +* Added the `:hide-unmarked-lines` and `:show-unmarked-lines` commands + that hide/show lines based on whether they are bookmarked. +* Added the `json_contains()` SQL function to check if a JSON value + contains a number of a string. +* The relative time parser recognizes "next" at the beginning of the + input, for example, "next hour" or "next day". Handy for use in the + `:goto` command. +* Added a "text-transform" option for formatting JSON log messages. The + supported options are: none, uppercase, lowercase, and capitalize. +* Added a special `__level__` field name for formatting JSON messages so + that the lnav level name can be used instead of the internal value in + the JSON object. +* Added a log format for journald JSON logs. + +Interface Changes: +* When typing in a search, instead of moving the view to the first match + that was found, the first ten matches will be displayed in the preview + window. +* The pretty-print view maintains highlighting from the log view. +* The pretty-print view no longer tries to reverse lookup IP addresses. +* The online help for commands and SQL functions now includes a 'See Also' + section that lists related commands/functions. + +Fixes: +* The HOME key should now work in the command-prompt and move the cursor + to the beginning of the line. +* The `:delete-filter` command should now tab-complete existing filters. +* Milliseconds can now be used in relative times (e.g. 10:00:00.123) +* The `J`/`K` hotkeys were not marking lines correctly when the bottom of + the view was reached. +* The level field in JSON logs should now be recognized by the level + patterns in the format. + +## lnav v0.8.3 + +Features: +* Support for the Bro Network Security Monitor (https://www.bro.org) log + file format. +* Added an `fstat()` table-valued function for querying the local + filesystem. +* Added `readlink()` and `realpath()` SQL functions. +* Highlights specified in log formats can now specify the colors to use + for the highlighted parts of the log message. +* Added a `:quit` command. +* Added a `/ui/default-colors` configuration option to specify that the + terminal's default background and foreground colors should be used + instead of black and white. + +Interface Changes: +* Pressing delete at a command-prompt will exit the prompt if there is no + other input. + +Fixes: +* The help view now includes all the command-help that would pop up as + you entered commands and SQL queries. +* Hidden fields and lines hidden before/after times are now saved in the + current session and restored. +* Unicode characters should now be displayed correctly (make sure you + have LANG set to a UTF-8 locale). + +## lnav v0.8.2 + +Features: +* The timestamp format for JSON log files can be specified with the + `timestamp-format` option in the `line-format` array. +* Added "min-width", "max-width", "align", and "overflow" options to the + "line-format" in format definitions for JSON log files. These options + give you more control over how the displayed line looks. +* Added a "hidden" option to log format values so that you can hide JSON + log fields from being displayed if they are not in the line format. +* Added a `rewriter` field to log format value definitions that is a + command used to rewrite the field in the pretty-printed version of a + log message. For example, the HTTP access log format will rewrite the + status code field to include the textual version (e.g. 200 (OK)). +* Log message fields can now be hidden using the `:hide-fields` command or + by setting the 'hidden' property in the log format. When hidden, the + fields will be replaced with a yellow ellipsis when displayed. Hiding + large fields that contain extra details can make the log easier to read. + The `x` hotkey can be used to quickly toggle whether these fields are + displayed or not. +* Added a `:mark` command to bookmark the top line in the current view. +* Added an `:alt-msg` command that can be used to set the text to be + displayed in the bottom right of the command line. This command is + mostly intended for use by hotkey maps to set the help text. +* In lnav scripts, the first row of a SQL query result will now be turned + into local variables that can be referenced in other commands or + queries. For example, the following script will print the number one: + ```lnav + ;SELECT 1 as foobar + :eval :echo ${foobar} + ``` +* Added an `lnav_view_stack` SQL table that gives access to the view + stack. +* Added a `top_time` column to the lnav_views table so that you can get + the timestamp for the top line in views that are time-based as well as + allowing you to move the view to a given time with an UPDATE statement. +* Added a 'search' column to the lnav_views table so that you can perform + a text search programmatically. +* Added a `regexp_capture(<string>, <pattern>)` table-valued function for + getting detailed results from matching a regular expression against a + string. +* Added a `timediff(<time1>, <time2>)` SQL function for computing the + difference between two relative or absolute timestamps. +* Log formats can now define a default set of highlights with the + "highlights" property. +* Added a `|search-for <pattern>` built-in script that can be used to + start a search from the command-line. +* Log format definitions can now specify the expected log level for a + sample line. This check should make it easier to validate the + definition. + +Interface Changes: +* Command and SQL documentation is now displayed in a section at the + bottom of the screen when a command or query is being entered. Some + commands will also display a preview of the command results. For + example, the `:open` command will display the first ten lines of the + file to be opened and the `:filter-out` command will highlight text + that matches in the current view. The preview pane can be shown/hidden + by pressing `CTRL-P`. +* The color used for text colored via `:highlight` is now based on the + the regex instead of randomly picked so that colors are consistent + across invocations. +* The "graph" view has been removed since it's functionality has been + obsoleted by other features, like `:create-search-table`. +* When doing a search, if a hit is found within a second after hitting + `<ENTER>`, the view will move to the matched line. The previous behavior + was to stay on the current line, which tended to be a surprise to new + users. +* Pressing `n`/`N` to move through the next/previous search hit will now + skip adjacent lines, up to the vertical size of the view. This should + make scanning through clusters of hits much faster. Repeatedly + pressing these keys within a short time will also accelerate scanning + by moving the view at least a full page at a time. + +Breaking Changes: +* The captured timestamp text in log files must fully match a known format + or an error will be reported. The previous behavior was to ignore any + text at the end of the line. + +Fixes: +* You can now execute commands from the standard input by using a dash (`-`) + with the `-f` command-line argument. Reading commands from a file + descriptor should also work, for example, with the following bash + syntax: + ```console + $ lnav -f <(echo :open the-file-to-open) + ``` +* Programming language syntax highlighting should now only be applied to + source code files instead of everywhere. + +## lnav v0.8.1 + +Features: +* Added a spectrogram command and view that displays the values of a + numeric field over time. The view works for log message fields or + for database result columns. +* Log formats can now create SQL views and execute other statements + by adding `.sql` files to their format directories. The SQL scripts + will be executed on startup. +* Added `json_group_object` and `json_group_array` aggregate SQL + functions that collects values from a GROUP BY query into a JSON + object or array, respectively. +* The SQL view will now graph values found in JSON objects/arrays in + addition to the regular columns in the result. +* Added an `regexp_match(<re>, <str>)` SQL function that can be used to + extract values from a string using a regular expression. +* Added an `extract(<str>)` SQL function that extracts values using the + same data discover/extraction parser used in the `logline` table. +* Added a "summary" overlay line to the bottom of the log view that + displays how long ago the last message was received, along with the + total number of files and the error rate over the past five minutes. +* Pressing `V` in the DB view will now check for a column with a + timestamp and move to the corresponding time in the log view. +* Added `a`/`A` hotkeys to restore a view previously popped with `q`/`Q`. +* Added `:hide-lines-before`, `:hide-lines-after`, and + `:show-lines-before-and-after` commands so that you can filter out + log lines based on time. +* Scripts containing lnav commands/queries can now be executed using + the pipe (`|`) hotkey. See the documentation for more information. +* Added an `:eval` command that can be used to execute a command or + query after performing environment variable substitution. +* Added an `:echo` command that can be useful for scripts to message + the user. +* The `log_part` column can now be set with an SQL `UPDATE` statement. +* Added a `log_body` hidden column that returns the body of the log + message. +* Added `:config`, `:reset-config`, and `:save-config` commands to change + configuration options, reset to default, and save them for future + executions. +* Added a `/ui/clock-format` configuration option that controls the time + format in the top-left corner. +* Added a `/ui/dim-text` configuration option that controls the brightness + of text in the UI. +* Added support for TAI64 timestamps (http://cr.yp.to/libtai/tai64.html). +* Added a safe execution mode. If the `LNAVSECURE` environment variable is + set before executing lnav, the following commands are disabled: + - `:open` + - `:pipe-to` + - `:pipe-line-to` + - `:write-*-to` + + This makes it easier to run lnav with escalated privileges in restricted + environments, without the risk of users being able to use the above + mentioned commands to gain privileged access. + +Interface Changes: +* The `o`/`O` hotkeys have been reassigned to navigate through log + messages that have a matching "opid" field. The old action of + moving forward and backward by 60 minutes can be simulated by + using the `:goto` command with a relative time and the `r`/`R` + hotkeys. +* Log messages with timestamps that pre-date previous log messages will + have the timestamp highlighted in yellow and underlined. These out- + of-time-order messages will be assigned the time of the previous + message for sorting purposes. You can press the 'p' hotkey to examine + the 'Received Time' of the message as well as the time parsed from the + original message. A `log_actual_time` hidden field has also been + added to the SQLite virtual table so you can operate on the original + message time from the file. +* The `A`/`B` hotkeys for moving forward/backward by 10% line increments + have been reassigned to `[` and `]`. The `a` and `A` hotkeys are now + used to return to the previously popped view while trying to preserve + the time range. For example, after leaving the spectrogram view with + 'q', you can press 'A' return to the view with the top time in the + spectrogram matching the top time in the log view. +* The 'Q' hotkey now pops the current view off of the stack while + maintaining the top time between views. + +Fixes: +* Issues with tailing JSON logs have been fixed. +* The `jget()` SQL function should now work for objects nested in arrays. + +## lnav v0.8.0 + +Features: +* Integration with "papertrailapp.com" for querying and tailing + server log and syslog messages. See the Papertrail section in + the online help for more details. +* Remote files can be opened when lnav is built with libcurl v7.23.0+ +* SQL queries can now be done on lines that match a regular expression + using the `log_search` table or by creating custom tables with the + `:create-search-table` command. +* Log formats that are "containers" for other log formats, like + syslog, are now supported. See the online help for more + information. +* Formats can be installed from git repositories using the `-i` option. + A standard set of extra formats can be installed by doing + `lnav -i extra`. (You must have git installed for this to work.) +* Added support for 'VMware vSphere Auto Deploy' log format. +* Added a 'sudo' log format. +* Added hotkeys to move left/right by a smaller increment (H/L or + Shift+Left/Shift+Right). +* A color-coded bar has been added to the left side to show where + messages from one file stop and messages from another file start. +* The `-C` option will now try to check any specified log files to + make sure the format(s) match all of the lines. +* Added an `all_logs` SQLite table that contains the message format + extracted from each log line. Also added a `;.msgformat` SQL command + that executes a query that returns the counts for each format and the + first line where the format was seen. +* Added an `lnav_views` SQLite table that can be used to query and + change the lnav view state. +* When typing in a command, the status bar will display a short + summary of the currently entered command. +* Added a `:delete-filter` command. +* Added a `log_msg_instance` column to the logline and log_search + tables to make it easier to join tables that are matching log + messages that are ordered. +* Added a `timeslice()` function to SQLite so that it is easier to + group log messages by time buckets. +* The `:goto` command now supports relative time values like + `a minute ago`, `an hour later`, and many more. + +Interface Changes: +* The `r`/`R` hotkeys have been reassigned to navigate through the log + messages by the relative time value that was last used with the + `:goto` command. + +Fixes: +* The pretty-print view should now work for text files. +* Nested fields in JSON logs are now supported for levels, bodies, etc... +* Tab-completion should work for quoted SQL identifiers. +* 'lo-fi' mode key shortcut changed to `CTRL+L`. +* 'redraw' shortcut removed. Relegated to just a command. +* Fixed lnav hang in pretty-print mode while doing a dns lookup. +* The generic log message parser used to extract data has been + optimized and should be a bit faster. + +## lnav v0.7.3 + +Features: +* Add `:pipe-to` and `:pipe-line-to` commands that pipe the currently + marked lines or the current log message to a shell command, + respectively. +* Added a "pretty-print" view (P hotkey) that tries to reformat log + messages so that they are easier to read. +* Added a `:redraw` command (CTRL+L hotkey) to redraw the window in + case it has been corrupted. +* Added a `:relative-goto` command to move the current view relative + to its current position. +* Experimental support for linking with jemalloc. +* The plain text view now supports filtering. +* Added `:next-mark` and `:prev-mark` commands to jump to the next or + previous bookmarked line (e.g. error, warning, ...) +* Added a `:zoom-to` command to change the zoom level of the histogram + view. +* Log formats can now define their own timestamp formats with the + `timestamp-format` field. + +Fixes: +* Autotools scripts overhaul. +* Added a configure option to disable linking with libtinfo. The newer + versions of ncurses don't require it, however the build silently pulls + it in as a dependency, if it is available on the system. This can be + explicitly disabled using the `--disable-tinfo` option during configure. +* Fixed the configure script behavior to ignore the values specified using + the CFLAGS and LDFLAGS environment variables while searching for sqlite3 + when `--with-sqlite3` switch was specified without the prefix. +* The configure script now recognizes libeditline symlink'ed to masquerade + as libreadline. This previously used to cause problems at compile time, + specially on OS X. If you come across this error, use the + `--with-readline=prefix` switch to specify the path to the correct + location of libreadline. +* The order that log formats are tried against a log file is now + automatically determined so that more specific formats are tested + before more general ones. The order is determined on startup based on + how each format matches each other formats sample lines. +* Command files (i.e. those executed via the `-f` flag) now support + commands/queries that span more than one line. +* Added more log levels: stats, debug2 - debug5. + +## lnav v0.7.2 + +* Added log formats for vdsm, openstack, and the vmkernel. +* Added a "lo-fi" mode (L hotkey) that dumps the displayed log lines + to the terminal without any decorations. The `:write-to`, `:write-json-to`, + and `:write-csv-to` commands will also write their output to the terminal + when passed `-` as the file name. This mode can be useful for copying + plain text lines to the clipboard. +* (OS X) Text search strings are copied to the system's "find" clipboard. + Also, when starting a new search, the current value in the "find" + clipboard can be tab-completed. + +## lnav v0.7.1 + +Features: +* Added an `environ` SQL table that reflects lnav's environment + variables. The table can be read and written to using SQL + `SELECT`, `INSERT`, `UPDATE`, and `DELETE` statements. Setting + variables can be a way to use SQL query results in lnav commands. +* Added a `jget()` SQLite function that can extract fields from a JSON- + encoded value. +* Added log formats for the OpenAM identity provider. +* Added a `:clear-highlight` command to clear previous calls to the + `:highlight` command. +* Fixed some performance bugs in indexing JSON log formats. Loading + times should be at least five times faster. +* Filtering performance should be improved so that enabling/disabling + filters should be almost instantaneous. +* The `:filter-in`, `:filter-out`, and `:highlight` commands now support + tab-completion of text in the current view. +* Add a `-i` flag that installs format files in: `~/.lnav/formats/installed` + +## lnav v0.7.0 + +Features: +* Add the '.schema' SQL command to open a view that displays the schema + for the internal tables and any attached databases. If lnav was only + executed with a SQLite database and no text files, this view will open + by default. +* The scroll bar now indicates the location of errors/warnings, search + hits, and bookmarks. +* The xterm title is update to reflect the file name for the top line + in the view. +* Added a "headless" mode so that you can execute commands and run SQL + queries from the command-line without having to do it from the curses + UI. +* When doing a search or SQL query, any text that is currently being + displayed can be tab-completed. +* The `-H` option was added so you can view the internal help text. +* Added the 'g/G' hotkeys to move to the top/bottom of the file. +* Added a `log_mark` column to the log tables that indicates whether or + not a log message is bookmarked. The field is writable, so you can + bookmark lines using an SQL UPDATE query. +* Added syntax-highlighting when editing SQL queries or search regexes. +* Added a `:write-json-to` command that writes the result of a SQL query + to a JSON-formatted file. +* The "elapsed time" column now uses red/green coloring to indicate + sharp changes in the message rate. +* Added a `:set-min-log-level` command to filter out log messages that + are below a given level. + +Fixes: +* Performance improvements. +* Multi-line filtering has been fixed. +* A collator has been added to the log_level column in the log tables + so that you can write expressions like `log_level > 'warning'`. +* The log_time datetime format now matches what is returned by + `datetime('now')` so that collating works correctly. +* If a search string is not valid PCRE syntax, a search is done for + the exact string instead of just returning an error. +* Static-linking has been cleaned up. +* OpenSSL is no longer a requirement. +* Alpha support for Windows/cygwin. +* Environment variables can now be accessed in SQL queries using + the syntax: `$VAR_NAME` +* An internal log is kept and written out on a crash. +* Partition bookmarks are now tracked separately from regular user + bookmarks. You can start a partition with the 'partition-name' + command and remove it with the 'clear-partition' command. +* Improved display of possible matches during tab-completion in the + command-prompt. The matches are now shown in a separate view and + pressing tab repeatedly will scroll through the view. +* The "open" command now does shell word expansion for file names. +* More config directory paths have been added: `/etc/lnav`, + `$prefix/etc/lnav`, and directories passed on the command-line + with `-I`. + +## lnav v0.6.2 + +Features: +* Word-wrap support. + +Fixes: +* Fix some OS X Mavericks build/runtime issues. + +## lnav v0.6.1 +Features: +* Support for JSON-encoded log files. + +Fixes: +* Some minor fixes and performance improvements. + +## lnav v0.6.0 + +Features: +* Custom log formats and more builtin formats +* Automatic extraction of data from logs +* UI improvements, support for 256 color terminals + +## lnav v0.5.1 + +Features: +* Added the `-t` and `-w` options which can be used to prepend a + timestamp to any data piped in on stdin and to specify a file to + write the contents of stdin to. + +Fixes: +* Cleanup for packaging. + +## lnav v0.5.0 + +Features: +* Files can be specified on the command-line using wildcards so that + new files are automatically loaded. Directories can also be passed + as command-line arguments to read all of the files in the directory. +* Builds on cygwin again. +* Added the `C` hotkey to clear any existing user bookmarks. +* Added experimental support for accepting input from mice. + +Fixes: +* Internal cleanup. +* Copying to the clipboard on OS X is now supported. +* Many bug fixes. + +## lnav v0.4.0 + +Features: +* Files that are not recognized as containing log messages have been + broken out to a separate text files view. You can flip between the + log view and the text file view with the `t` hotkey. When viewing + text files, the `f` hotkey will switch between files. +* Files compressed with bzip2 are recognized and decompressed on the + fly. +* Added a "session" file and command for storing commands that should + be executed on startup. For example, if you always want some + highlighting to be done, you can add that command to the session + file. + +Fixes: +* Add some more log file formats for generic log files. +* Performance improvements for compressed files. +* Works on OS X now. + +## lnav v0.3.0 + +Changes: +* The hotkey for the SQL view was changed to `v` and `V` from '.'. + +Features: +* You can now switch between the SQL result view and the log view while + keeping the top of the views in sync with the `log_line` column. + +Fixes: +* The `log_line` column is no longer included in the SQL result view's + stacked bar graph. +* Added a "warnings" count to the histogram view. |