diff options
Diffstat (limited to '')
32 files changed, 1629 insertions, 221 deletions
diff --git a/docs/02_downloads.md b/docs/02_downloads.md index 9c44192..9a82d48 100644 --- a/docs/02_downloads.md +++ b/docs/02_downloads.md @@ -11,7 +11,7 @@ The following options are available for installing **lnav**: ## Linux <!-- markdown-link-check-disable-next-line --> -Download a [statically linked 64-bit binary](https://github.com/tstack/lnav/releases/download/v{{site.version}}/lnav-{{site.version}}-x86_64-linux-musl.zip). +Download a [statically linked 64-bit binary](https://github.com/tstack/lnav/releases/download/v{{site.version}}/lnav-{{site.version}}-linux-musl-x86_64.zip). Install from the [Snap Store](https://snapcraft.io/lnav): diff --git a/docs/Gemfile.lock b/docs/Gemfile.lock index 275c580..d31a8c6 100644 --- a/docs/Gemfile.lock +++ b/docs/Gemfile.lock @@ -1,11 +1,12 @@ GEM remote: https://rubygems.org/ specs: - activesupport (7.0.5) + activesupport (6.1.7.6) concurrent-ruby (~> 1.0, >= 1.0.2) i18n (>= 1.6, < 2) minitest (>= 5.1) tzinfo (~> 2.0) + zeitwerk (~> 2.3) addressable (2.8.4) public_suffix (>= 2.0.2, < 6.0) coffee-script (2.4.1) @@ -13,7 +14,7 @@ GEM execjs coffee-script-source (1.11.1) colorator (1.1.0) - commonmarker (0.23.9) + commonmarker (0.23.10) concurrent-ruby (1.2.2) dnsruby (1.70.0) simpleidn (~> 0.2.1) @@ -210,9 +211,11 @@ GEM jekyll-feed (~> 0.9) jekyll-seo-tag (~> 2.1) minitest (5.18.1) - nokogiri (1.15.2-arm64-darwin) + nokogiri (1.16.3-arm64-darwin) racc (~> 1.4) - nokogiri (1.15.2-x86_64-linux) + nokogiri (1.16.3-x86_64-darwin) + racc (~> 1.4) + nokogiri (1.16.3-x86_64-linux) racc (~> 1.4) octokit (4.25.1) faraday (>= 1, < 3) @@ -220,7 +223,7 @@ GEM pathutil (0.16.2) forwardable-extended (~> 2.6) public_suffix (4.0.7) - racc (1.7.1) + racc (1.7.3) rb-fsevent (0.11.2) rb-inotify (0.10.1) ffi (~> 1.0) @@ -250,10 +253,12 @@ GEM unf_ext (0.0.8.2) unicode-display_width (1.8.0) webrick (1.8.1) + zeitwerk (2.6.11) PLATFORMS arm64-darwin-21 arm64-darwin-22 + universal-darwin-23 x86_64-linux DEPENDENCIES diff --git a/docs/_config.yml b/docs/_config.yml index 6d116ee..b258464 100644 --- a/docs/_config.yml +++ b/docs/_config.yml @@ -19,7 +19,7 @@ # in the templates via {{ site.myvariable }}. title: The Logfile Navigator -version: 0.11.1 +version: 0.12.1 email: support@lnav.org description: >- # this means to ignore newlines until "baseurl:" The Logfile Navigator, lnav for short, is an advanced log file viewer @@ -37,6 +37,9 @@ plugins: show_excerpts: true +kramdown: + syntax_highlighter: rouge + exclude: - source - tutorials diff --git a/docs/_posts/2018-04-05-linux-magazine-tutorial.md b/docs/_posts/2018-04-05-linux-magazine-tutorial.md index 9d868b5..2a749b2 100644 --- a/docs/_posts/2018-04-05-linux-magazine-tutorial.md +++ b/docs/_posts/2018-04-05-linux-magazine-tutorial.md @@ -5,5 +5,5 @@ date: 2018-04-05 00:00:00 --- Looks like there was an in-depth -[tutorial on lnav in Linux Magazine](http://www.linux-magazine.com/Issues/2017/196/Tutorials-lnav). +[tutorial on lnav in Linux Magazine](https://web.archive.org/web/20230130154739/http://www.linux-magazine.com/Issues/2017/196/Tutorials-lnav). Unfortunately, I didn't notice until now and missed out on a hardcopy. diff --git a/docs/_posts/2023-08-04-config-dump.md b/docs/_posts/2023-08-04-config-dump.md new file mode 100644 index 0000000..cfa9471 --- /dev/null +++ b/docs/_posts/2023-08-04-config-dump.md @@ -0,0 +1,56 @@ +--- +layout: post +title: Tooling for troubleshooting configuration +excerpt: Getting the final configuration and the sources of values +--- + +*(This change is in **v0.12.0+**)* + +Inspired by [this blog post about reporting configuration file locations](https://utcc.utoronto.ca/~cks/space/blog/sysadmin/ReportConfigFileLocations) +and the [ensuing HackerNews commentary](https://news.ycombinator.com/item?id=36465886). +I've added the `config get` and `config blame` management commands for getting the +final configuration and the source of each property in the configuration, respectively. +I had previously added the file locations used by **lnav** in the `lnav -h` output as +recommended by the blog post. But, the HN comments made a good case for adding the +the other troubleshooting tooling as well. + +If you would like to try out these new commands, you need to run lnav with the `-m` +option to switch to the "management" mode. For example, just running lnav with this +flag will print out the available operations: + +```console +$ lnav -m +✘ error: expecting an operation to perform + = help: the available operations are: + • config: perform operations on the lnav configuration + • format: perform operations on log file formats + • piper: perform operations on piper storage + • regex101: create and edit log message regular expressions using regex101.com +``` + +Executing `config get` will print out the final configuration that lnav is operating +with as JSON: + +```console +$ lnav -m config get +``` + +If you would like to know the source of the value for each property, you can use +the `config blame` command, like so: + +```console +$ lnav -m config blame | tail +/ui/theme-defs/solarized-light/vars/black -> solarized-light.json:15 +/ui/theme-defs/solarized-light/vars/blue -> solarized-light.json:21 +/ui/theme-defs/solarized-light/vars/cyan -> solarized-light.json:22 +/ui/theme-defs/solarized-light/vars/green -> solarized-light.json:23 +/ui/theme-defs/solarized-light/vars/magenta -> solarized-light.json:19 +/ui/theme-defs/solarized-light/vars/orange -> solarized-light.json:17 +/ui/theme-defs/solarized-light/vars/red -> solarized-light.json:18 +/ui/theme-defs/solarized-light/vars/semantic_highlight_color -> solarized-light.json:24 +/ui/theme-defs/solarized-light/vars/violet -> solarized-light.json:20 +/ui/theme-defs/solarized-light/vars/yellow -> solarized-light.json:16 +``` + +In the above output, "solarized-light.json" file is built into the lnav +executable and is not from the file system. diff --git a/docs/_posts/2024-01-09-hammered.md b/docs/_posts/2024-01-09-hammered.md new file mode 100644 index 0000000..bb94577 --- /dev/null +++ b/docs/_posts/2024-01-09-hammered.md @@ -0,0 +1,324 @@ +--- +layout: post +title: Using lnav to solve the CyberDefenders Hammered Challenge +excerpt: >- + A walkthrough that uses lnav's analysis functionality + to answer questions about a collection of logs +--- + +I recently stumbled on this nice [review of lnav](https://lopes.id/2023-lnav-test/) +by José Lopes. They use this [Hammered](https://cyberdefenders.org/blueteam-ctf-challenges/42) +challenge by [cyberdefenders.org](https://cyberdefenders.org) as a way to get to +know how to use lnav. I thought I would do the same and document the commands +I would use to give folks some practical examples of using lnav. + +(Since I'm not well-versed in forensic work, I followed this great +[walkthrough](https://forensicskween.com/ctf/cyberdefenders/hammered/).) + +#### Q1: Which service did the attackers use to gain access to the system? + +We can probably figure this out by looking for common failure messages +in the logs. But, first, we need to load the logs into lnav. You +can load all of the logs by passing the path to the `Hammered` directory +along with the `-r` option to recurse through any subdirectories: + +```console +lnav -r Hammered +``` + +Now that the logs are loaded, you can use the `.msgformats` SQL command +to execute a canned query that finds log messages with a common text +format. (Unfortunately, this command has suffered from bitrot and is +broken in the current release. It will be fixed in the next release. +In the meantime, you can copy the [snippet](#msgformatlnav) below +to a file and execute it using the `|` prompt.) You can enter the +SQL prompt by pressing `;` and then entering the command or statement: + +``` +;.msgformats +``` + +The top results I get for this batch of logs look like the following. + +``` +┏━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ +┃total┃log_line┃ log_time ┃ duration ┃ log_formats ┃ log_msg_format ┃ +┡━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ +│15179│ 798│2010-03-16 08:12:09.000│47d14h59m04s│syslog_log │#): session closed for user root │ +│14500│ 817│2010-03-16 08:17:01.000│47d14h54m00s│syslog_log │#): session opened for user root by (#) │ +│14480│ 29380│2010-04-19 04:36:49.000│7d04h03m45s │syslog_log │pam_unix(sshd:auth): #; # │ +│14478│ 29381│2010-04-19 04:36:49.000│7d04h03m45s │syslog_log │#): #; logname= # │ +│ 6300│ 74477│2010-04-20 06:57:11.000│6d03h00m42s │syslog_log │: [#]: IN=# OUT=# MAC=# SRC=# DST=# LEN=# TOS=# PREC=# TTL=# ID=# PROTO=# SPT=# DPT=# LEN=# │ +│ 5848│ 4695│2010-03-18 11:38:04.000│38d21h13m39s│syslog_log │#): #; logname= # │ +│ 5479│ 16164│2010-03-29 13:23:46.000│27d19h27m58s│syslog_log │Failed password for root from # port # # │ +... +``` + +The `#` in the `log_msg_format` column are the parts of the text +that vary between log messages. For example, the most interesting +message is "Failed password for root from # port # #". In that case, +the first `#` would be the IP address and then the port number. The +first column indicates how many times a message like this was found, +so 5,479 failed password attempts is probably a good sign of a breakin +attempt. + +To find out the service that logged this message, you can scroll down +to focus on the message and then press `Shift` + `Q` to return to the +LOG view at the line mentioned in the `log_line` column. In this case, +line 16,164, which contains: + +``` +Mar 29 13:23:46 app-1 sshd[21492]: Failed password for root from 10.0.1.2 port 51771 ssh2 +``` + +So, the attack vector is `sshd`. + + +##### msgformat.lnav + +The `;.msgformats` command has been broken for a few releases, but +its functionality can be replicated using the script below. +Copy the following to a file named `msgformat.lnav` and place it in the +`formats/installed` lnav configuration directory. + +``` +;SELECT count(*) AS total, + min(log_line) AS log_line, + min(log_time) AS log_time, + humanize_duration(timediff(max(log_time), min(log_time))) AS duration, + group_concat(DISTINCT log_format) AS log_formats, + log_msg_format + FROM all_logs + GROUP BY log_msg_format + HAVING total > 1 + ORDER BY total DESC +:switch-to-view db +``` + +#### Q2: What is the operating system version of the targeted system? (one word) + +The answer to this question has the form `4.*.*.u3` as given in the +challenge. You can do a search in lnav by pressing `/` and then +entering a PCRE-compatible regular expression. In this case, +entering `4\.[^ ]+u3` will locate lines with the desired version +number of `4.2.4-1ubuntu3`. + +#### Q3: What is the name of the compromised account? + +Using the findings of our initial analysis, the compromised account +is `root`. + +#### Q4: Consider that each unique IP represents a different attacker. How many attackers were able to get access to the system? + +Answering this question will require analyzing messages in the `auth.log` +file. Specifically, we will need to find failed password attempts, like +the following one and extract the user ID and IP address: + +``` +Apr 18 18:22:07 app-1 sshd[5266]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=61.151.246.140 user=root +``` + +The failed attempts will give us the attacker IP addresses. However, we +don't want to confuse attacker IPs with legitimate logins. So, we'll +need to look for successful login messages like this one: + +``` +Mar 16 08:26:06 app-1 sshd[4894]: Accepted password for user3 from 192.168.126.1 port 61474 ssh2 +``` + +Analyzing log data in lnav is done through the SQL interface. The +log messages can be accessed through SQL tables that are automatically +defined for each log format. However, that is pretty cumbersome +since there would be a lot of regex SQL function calls cluttering up +the queries. Instead, we can use the [`:create-search-table`](https://docs.lnav.org/en/v0.11.2/usage.html#search-tables) +command to create a SQL table that matches a regular expression +against the log messages and extracts data into column(s). We can +then write much simpler SQL queries to get the data we're interested +in. + +First, lets create an `auth_failures` table for the authentication +failure log messages: + +``` +:create-search-table auth_failures authentication failure; .* rhost=(?<ip>\d+\.\d+\.\d+\.\d+)\s+user=(?<user>[^ ]+) +``` + +Now, let's try it out by finding the IPs of failed auth attempts: + +```sql +;SELECT DISTINCT ip FROM auth_failures +``` + +Next, lets create an `auth_accepted` table for the successful +authentications: + +``` +:create-search-table auth_accepted Accepted password for (?<user>[^ ]+) from (?<ip>\d+\.\d+\.\d+\.\d+) +``` + +Now that we have these two tables, we can write a query that +gets the IPs of failed auth attempts that eventually +succeeded. We further filter out low failure counts to +eliminate human error. The full query is as follows: + +```sql +;SELECT ip, count(*) AS co FROM auth_failures WHERE user = 'root' AND ip IN (SELECT DISTINCT ip FROM auth_accepted) GROUP BY ip HAVING co > 10 +``` + +The results are the following six IPs: + +``` +┏━━━━━━━━━━━━━━━┳━━━━┓ +┃ ip ┃ co ┃ +┡━━━━━━━━━━━━━━━╇━━━━┩ +│61.168.227.12 │ 386│ +│121.11.66.70 │2858│ +│122.226.202.12 │ 626│ +│219.150.161.20 │3120│ +│222.66.204.246 │1016│ +│222.169.224.197│ 358│ +└━━━━━━━━━━━━━━━┴━━━━┘ +``` + +#### Q5: Which attacker's IP address successfully logged into the system the most number of times? + +The attacker IPs were found using the query in the previous +question, but the counts are for the number of failed auth +attempts. Probably the easiest thing to do is create a SQL +view with the previous query. That can be done quickly by +pressing `;` and then pressing the up arrow to go back in +the command history. Then, go to the start of the line and +prepend `CREATE VIEW attackers AS ` before the `SELECT`. +That will create an `attackers` SQL view that we can use +to answer this question. + +Now that we can easily get the list of attacker IPs, we +can write a query for the `auth_accepted` table that +finds all the successful auth messages. We then group +by IP and count to get the data we want: + +```sql +;SELECT ip, count(*) AS co FROM auth_accepted WHERE ip IN (SELECT ip FROM attackers) GROUP BY ip ORDER co DESC +``` + +The results are: + +``` +┏━━━━━━━━━━━━━━━┳━━┓ +┃ ip ┃co┃ +┡━━━━━━━━━━━━━━━╇━━┩ +│219.150.161.20 │ 4│ +│122.226.202.12 │ 2│ +│121.11.66.70 │ 2│ +│222.169.224.197│ 1│ +│222.66.204.246 │ 1│ +│61.168.227.12 │ 1│ +└━━━━━━━━━━━━━━━┴━━┘ +``` + +The top IP there is `219.150.161.20`. + +#### Q6: How many requests were sent to the Apache Server? + +Logs that follow the Apache log format can be accessed by the +`access_log` SQL table. The following query will count the +log messages in each access log file: + +```sql +;SELECT log_path, count(*) FROM access_log GROUP BY log_path +``` + +The results I get are: + +``` +┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┓ +┃ log_path ┃count(*)┃ +┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━┩ +│/Users/tstack/Downloads/Hammered/apache2/www-access.log│ 365│ +│/Users/tstack/Downloads/Hammered/apache2/www-media.log │ 229│ +└━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┴━━━━━━━━┘ +``` + +It seems like they want just what is in the `www-access.log` +file, so the answer is 365. + +#### Q7: How many rules have been added to the firewall? + +Rules are added by the `iptables -A` command, so we can do a +search for that command and the status bar will show +"6 hits for “iptables -A”". + +#### Q9: When was the last login from the attacker with IP 219.150.161.20? Format: MM/DD/YYYY HH:MM:SS AM + +Using the `auth_accepted` table we created previously, this is +a pretty simple query for `max(log_time)`: + +```sql +;SELECT max(log_time) FROM auth_accepted WHERE ip = '219.150.161.20' +``` + +The result I get is: + +``` +✔ SQL Result: 2010-04-19 05:56:05.000 +``` + +#### Q10: The database displayed two warning messages, provide the most important and dangerous one. + +The database log messages come out in the syslog with a procname +of `/etc/mysql/debian-start` and are recognized as warnings. +Using this, we can write a [filter expression](https://docs.lnav.org/en/v0.11.2/commands.html#filter-expr-expr) +that filters the log based on SQL expression. For the syslog +file format, the procname is accessible via the `:log_procname` +variable and the log level is in the `:log_level` variable. +The following command puts this together: + +``` +:filter-expr :log_procname = '/etc/mysql/debian-start' AND :log_level = 'warning' +``` + +After running this command, you should only see about 15 lines +of the 100+k that was originally shown. Taking a look at these +lines, the following line seems pretty bad: + +``` +Mar 18 10:18:42 app-1 /etc/mysql/debian-start[7566]: WARNING: mysql.user contains 2 root accounts without password! +``` + +To clear the filter, you can press `CTRL` + `R` to reset the +state of the session. + +#### Q12: Few attackers were using a proxy to run their scans. What is the corresponding user-agent used by this proxy? + +The user-agent can be retrieved from the `cs_user_agent` +column in the `access_log` table. The following query +will get the unique user-agent names: + +```sql +;SELECT DISTINCT cs_user_agent FROM access_log +``` + +The results I get are: + +``` +┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ +┃ cs_user_agent ┃ +┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ +│Apple-PubSub/65.12.1 │ +│Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1) │ +│Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0) │ +│iearthworm/1.0, iearthworm@yahoo.com.cn │ +│Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/532.5 (KHTML, like Gecko) Chrome/4.1.249.1045 Safari/532.5 │ +│WordPress/2.9.2; http://www.domain.org │ +│Mozilla/5.0 (Windows; U; Windows NT 5.1; es-ES; rv:1.9.0.19) Gecko/2010031422 Firefox/3.0.19 │ +│Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_2; en-us) AppleWebKit/531.21.8 (KHTML, like Gecko) Version/4.0.4 Safari/531.21.10│ +│Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3 │ +│pxyscand/2.1 │ +│- │ +│Mozilla/4.0 (compatible; NaverBot/1.0; http://help.naver.com/customer_webtxt_02.jsp) │ +│Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_2; en-us) AppleWebKit/531.22.7 (KHTML, like Gecko) Version/4.0.5 Safari/531.22.7 │ +│Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/532.5 (KHTML, like Gecko) Chrome/4.1.249.1059 Safari/532.5 │ +└━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┘ +``` + +The `pxyscand/2.1` name seems to be the one they want. diff --git a/docs/_posts/2024-03-29-prql-support.md b/docs/_posts/2024-03-29-prql-support.md new file mode 100644 index 0000000..98df8e6 --- /dev/null +++ b/docs/_posts/2024-03-29-prql-support.md @@ -0,0 +1,51 @@ +--- +layout: post +title: Support for the PRQL in the database query prompt +excerpt: >- + PRQL is a database query language that is pipeline-oriented + and easier to use interactively +--- + +The v0.12.1 release of lnav includes support for +[PRQL](https://prql-lang.org). PRQL is a database query language +that has a pipeline-oriented syntax. The main advantage of PRQL, +in the context of lnav, is that it is easier to work with +interactively compared to SQL. For example, lnav can provide +previews of different stages of the pipeline and provide more +accurate tab-completions for the columns in the result set. I'm +hoping that the ease-of-use will make doing log analysis in lnav +much easier. + +You can execute a PRQL query using the existing database prompt +(press `;`). A query is interpreted as PRQL if it starts with +the [`from`](https://prql-lang.org/book/reference/data/from.html) +keyword. After `from`, the database table should be provided. +The table for the focused log message will be suggested by default. +You can accept the suggestion by pressing TAB. To add a new stage +to the pipeline, enter a pipe symbol (`|`), followed by a +[PRQL transform](https://prql-lang.org/book/reference/stdlib/transforms/index.html) +and its arguments. In addition to the standard set of transforms, +lnav provides some convenience transforms in the `stats` and `utils` +namespaces. For example, `stats.count_by` can be passed one or more +column names to group by and count, with the result sorted by most +to least. + +As you enter a query, lnav will update various panels on the display +to show help, preview data, and errors. The following is a +screenshot of lnav viewing a web access log with a query in progress: + +![Screenshot of PRQL in action](/assets/images/lnav-prql-preview.png) + +The top half is the usual log message view. Below that is the online +help panel showing the documentation for the `stats.count_by` PRQL +function. lnav will show the help for what is currently under the +cursor. The next panel shows the preview data for the pipeline stage +that precedes the stage where the cursor is. In this case, the +results of `from access_log`, which is the contents of the access +log table. The second preview window shows the result of the +pipeline stage where the cursor is located. + +There is still a lot of work to be done on the integration and PRQL +itself, but I'm very hopeful this will work out well in the long +term. Many thanks to the PRQL team for starting the project and +keeping it going, it's not easy competing with SQL. diff --git a/docs/assets/images/lnav-prql-preview.png b/docs/assets/images/lnav-prql-preview.png Binary files differnew file mode 100644 index 0000000..1c484d7 --- /dev/null +++ b/docs/assets/images/lnav-prql-preview.png diff --git a/docs/index.markdown b/docs/index.markdown index eadb501..68ea500 100644 --- a/docs/index.markdown +++ b/docs/index.markdown @@ -11,11 +11,11 @@ layout: top style="float: right; max-width: 50%"} <div id="top-description"> -An advanced log file viewer for the small-scale +A log file viewer for the terminal </div> <div id="intro"> -<p>Watch and analyze your log files from a terminal.</p> +<p>Merge, tail, search, filter, and query log files with ease.</p> <p>No server. No setup. Still featureful.</p> diff --git a/docs/requirements.txt b/docs/requirements.txt index 1d29e34..4567bf2 100644 --- a/docs/requirements.txt +++ b/docs/requirements.txt @@ -1,3 +1,4 @@ +furo sphinx>=4.3.0 sphinx-copybutton sphinx-jsonschema diff --git a/docs/schemas/config-v1.schema.json b/docs/schemas/config-v1.schema.json index e2285cc..9ca49b4 100644 --- a/docs/schemas/config-v1.schema.json +++ b/docs/schemas/config-v1.schema.json @@ -39,6 +39,35 @@ }, "additionalProperties": false }, + "piper": { + "description": "Settings related to capturing piped data", + "title": "/tuning/piper", + "type": "object", + "properties": { + "max-size": { + "title": "/tuning/piper/max-size", + "description": "The maximum size of a capture file", + "type": "integer", + "minimum": 128 + }, + "rotations": { + "title": "/tuning/piper/rotations", + "description": "The number of rotated files to keep", + "type": "integer", + "minimum": 2 + }, + "ttl": { + "title": "/tuning/piper/ttl", + "description": "The time-to-live for captured data, expressed as a duration (e.g. '3d' for three days)", + "type": "string", + "examples": [ + "3d", + "12h" + ] + } + }, + "additionalProperties": false + }, "file-vtab": { "description": "Settings related to the lnav_file virtual-table", "title": "/tuning/file-vtab", @@ -111,7 +140,7 @@ "title": "/tuning/remote/ssh/options", "type": "object", "patternProperties": { - "(\\w+)": { + "^(\\w+)$": { "title": "/tuning/remote/ssh/options/<option_name>", "description": "Set an option to be passed to the SSH command", "type": "string" @@ -124,7 +153,7 @@ "title": "/tuning/remote/ssh/config", "type": "object", "patternProperties": { - "(\\w+)": { + "^(\\w+)$": { "title": "/tuning/remote/ssh/config/<config_name>", "description": "Set an SSH configuration value", "type": "string" @@ -148,7 +177,7 @@ "title": "/tuning/clipboard/impls", "type": "object", "patternProperties": { - "([\\w\\-]+)": { + "^([\\w\\-]+)$": { "description": "Clipboard implementation", "title": "/tuning/clipboard/impls/<clipboard_impl_name>", "type": "object", @@ -179,6 +208,28 @@ } }, "additionalProperties": false + }, + "url-scheme": { + "description": "Settings related to custom URL handling", + "title": "/tuning/url-scheme", + "type": "object", + "patternProperties": { + "^([a-z][\\w\\-\\+\\.]+)$": { + "description": "Definition of a custom URL scheme", + "title": "/tuning/url-scheme/<url_scheme>", + "type": "object", + "properties": { + "handler": { + "title": "/tuning/url-scheme/<url_scheme>/handler", + "description": "The name of the lnav script that can handle URLs with of this scheme. This should not include the '.lnav' suffix.", + "type": "string", + "pattern": "^[\\w\\-]+(?!\\.lnav)$" + } + }, + "additionalProperties": false + } + }, + "additionalProperties": false } }, "additionalProperties": false @@ -221,7 +272,7 @@ "title": "/ui/theme-defs", "type": "object", "patternProperties": { - "([\\w\\-]+)": { + "^([\\w\\-]+)$": { "description": "Theme definitions", "title": "/ui/theme-defs/<theme_name>", "type": "object", @@ -231,7 +282,7 @@ "title": "/ui/theme-defs/<theme_name>/vars", "type": "object", "patternProperties": { - "(\\w+)": { + "^(\\w+)$": { "title": "/ui/theme-defs/<theme_name>/vars/<var_name>", "description": "A theme variable definition", "type": "string" @@ -254,6 +305,11 @@ "title": "/ui/theme-defs/<theme_name>/styles/text", "$ref": "#/definitions/style" }, + "selected-text": { + "description": "Styling for text selected in a view", + "title": "/ui/theme-defs/<theme_name>/styles/selected-text", + "$ref": "#/definitions/style" + }, "alt-text": { "description": "Styling for plain text when alternating", "title": "/ui/theme-defs/<theme_name>/styles/alt-text", @@ -289,6 +345,11 @@ "title": "/ui/theme-defs/<theme_name>/styles/cursor-line", "$ref": "#/definitions/style" }, + "disabled-cursor-line": { + "description": "Styling for the cursor line when it is disabled", + "title": "/ui/theme-defs/<theme_name>/styles/disabled-cursor-line", + "$ref": "#/definitions/style" + }, "adjusted-time": { "description": "Styling for timestamps that have been adjusted", "title": "/ui/theme-defs/<theme_name>/styles/adjusted-time", @@ -299,8 +360,13 @@ "title": "/ui/theme-defs/<theme_name>/styles/skewed-time", "$ref": "#/definitions/style" }, + "file-offset": { + "description": "Styling for a file offset", + "title": "/ui/theme-defs/<theme_name>/styles/file-offset", + "$ref": "#/definitions/style" + }, "offset-time": { - "description": "Styling for hidden fields", + "description": "Styling for the elapsed time column", "title": "/ui/theme-defs/<theme_name>/styles/offset-time", "$ref": "#/definitions/style" }, @@ -413,6 +479,11 @@ "description": "Styling for snippet borders", "title": "/ui/theme-defs/<theme_name>/styles/snippet-border", "$ref": "#/definitions/style" + }, + "indent-guide": { + "description": "Styling for indent guide lines", + "title": "/ui/theme-defs/<theme_name>/styles/indent-guide", + "$ref": "#/definitions/style" } }, "additionalProperties": false @@ -422,6 +493,11 @@ "title": "/ui/theme-defs/<theme_name>/syntax-styles", "type": "object", "properties": { + "inline-code": { + "description": "Styling for inline code blocks", + "title": "/ui/theme-defs/<theme_name>/syntax-styles/inline-code", + "$ref": "#/definitions/style" + }, "quoted-code": { "description": "Styling for quoted code blocks", "title": "/ui/theme-defs/<theme_name>/syntax-styles/quoted-code", @@ -462,11 +538,41 @@ "title": "/ui/theme-defs/<theme_name>/syntax-styles/symbol", "$ref": "#/definitions/style" }, + "null": { + "description": "Styling for nulls in source files", + "title": "/ui/theme-defs/<theme_name>/syntax-styles/null", + "$ref": "#/definitions/style" + }, + "ascii-control": { + "description": "Styling for ASCII control characters in source files", + "title": "/ui/theme-defs/<theme_name>/syntax-styles/ascii-control", + "$ref": "#/definitions/style" + }, + "non-ascii": { + "description": "Styling for non-ASCII characters in source files", + "title": "/ui/theme-defs/<theme_name>/syntax-styles/non-ascii", + "$ref": "#/definitions/style" + }, "number": { "description": "Styling for numbers in source files", "title": "/ui/theme-defs/<theme_name>/syntax-styles/number", "$ref": "#/definitions/style" }, + "type": { + "description": "Styling for types in source files", + "title": "/ui/theme-defs/<theme_name>/syntax-styles/type", + "$ref": "#/definitions/style" + }, + "function": { + "description": "Styling for functions in source files", + "title": "/ui/theme-defs/<theme_name>/syntax-styles/function", + "$ref": "#/definitions/style" + }, + "separators-references-accessors": { + "description": "Styling for sigils in source files", + "title": "/ui/theme-defs/<theme_name>/syntax-styles/separators-references-accessors", + "$ref": "#/definitions/style" + }, "re-special": { "description": "Styling for special characters in regular expressions", "title": "/ui/theme-defs/<theme_name>/syntax-styles/re-special", @@ -579,6 +685,11 @@ "description": "Styling for hotkey highlights of status bars", "title": "/ui/theme-defs/<theme_name>/status-styles/hotkey", "$ref": "#/definitions/style" + }, + "suggestion": { + "description": "Styling for suggested values", + "title": "/ui/theme-defs/<theme_name>/status-styles/suggestion", + "$ref": "#/definitions/style" } }, "additionalProperties": false @@ -588,7 +699,7 @@ "title": "/ui/theme-defs/<theme_name>/log-level-styles", "type": "object", "patternProperties": { - "(trace|debug5|debug4|debug3|debug2|debug|info|stats|notice|warning|error|critical|fatal|invalid)": { + "^(trace|debug5|debug4|debug3|debug2|debug|info|stats|notice|warning|error|critical|fatal|invalid)$": { "title": "/ui/theme-defs/<theme_name>/log-level-styles/<level>", "$ref": "#/definitions/style" } @@ -600,7 +711,7 @@ "title": "/ui/theme-defs/<theme_name>/highlights", "type": "object", "patternProperties": { - "([\\w\\-]+)": { + "^([\\w\\-]+)$": { "title": "/ui/theme-defs/<theme_name>/highlights/<highlight_name>", "type": "object", "properties": { @@ -626,6 +737,27 @@ }, "additionalProperties": false }, + "mouse": { + "description": "Mouse-related settings", + "title": "/ui/mouse", + "type": "object", + "properties": { + "mode": { + "title": "/ui/mouse/mode", + "description": "Overall control for mouse support", + "type": "string", + "enum": [ + "disabled", + "enabled" + ], + "examples": [ + "enabled", + "disabled" + ] + } + }, + "additionalProperties": false + }, "movement": { "description": "Log file cursor movement mode settings", "title": "/ui/movement", @@ -652,21 +784,26 @@ "title": "/ui/keymap-defs", "type": "object", "patternProperties": { - "([\\w\\-]+)": { + "^([\\w\\-]+)$": { "description": "The keymap definitions", "title": "/ui/keymap-defs/<keymap_name>", "type": "object", "patternProperties": { - "((?:x[0-9a-f]{2})+)": { + "^((?:x[0-9a-f]{2})+)$": { "description": "Map of key codes to commands to execute. The field names are the keys to be mapped using as a hexadecimal representation of the UTF-8 encoding. Each byte of the UTF-8 should start with an 'x' followed by the hexadecimal representation of the byte.", "title": "/ui/keymap-defs/<keymap_name>/<key_seq>", "type": "object", "properties": { + "id": { + "title": "/ui/keymap-defs/<keymap_name>/<key_seq>/id", + "description": "The identifier that can be used to refer to this key", + "type": "string" + }, "command": { "title": "/ui/keymap-defs/<keymap_name>/<key_seq>/command", "description": "The command to execute for the given key sequence. Use a script to execute more complicated operations.", "type": "string", - "pattern": "^[:|;].*", + "pattern": "^$|^[:|;].*", "examples": [ ":goto next hour" ] @@ -693,12 +830,25 @@ "title": "/log", "type": "object", "properties": { + "date-time": { + "description": "Settings related to log message dates and times", + "title": "/log/date-time", + "type": "object", + "properties": { + "convert-zoned-to-local": { + "title": "/log/date-time/convert-zoned-to-local", + "description": "Convert timestamps with ", + "type": "boolean" + } + }, + "additionalProperties": false + }, "watch-expressions": { "description": "Log message watch expressions", "title": "/log/watch-expressions", "type": "object", "patternProperties": { - "([\\w\\-]+)": { + "^([\\w\\.\\-]+)$": { "description": "A log message watch expression", "title": "/log/watch-expressions/<watch_name>", "type": "object", @@ -718,6 +868,37 @@ } }, "additionalProperties": false + }, + "annotations": { + "title": "/log/annotations", + "type": "object", + "patternProperties": { + "^([\\w\\.\\-]+)$": { + "title": "/log/annotations/<annotation_name>", + "type": "object", + "properties": { + "description": { + "title": "/log/annotations/<annotation_name>/description", + "description": "A description of this annotation", + "type": "string" + }, + "condition": { + "title": "/log/annotations/<annotation_name>/condition", + "description": "The SQLite expression to execute for a log message that determines whether or not this annotation is applicable. The expression is evaluated the same way as a filter expression", + "type": "string", + "minLength": 1 + }, + "handler": { + "title": "/log/annotations/<annotation_name>/handler", + "description": "The script to execute to generate the annotation content. A JSON object with the log message content will be sent to the script on the standard input", + "type": "string", + "minLength": 1 + } + }, + "additionalProperties": false + } + }, + "additionalProperties": false } }, "additionalProperties": false @@ -727,7 +908,7 @@ "title": "/global", "type": "object", "patternProperties": { - "(\\w+)": { + "^(\\w+)$": { "title": "/global/<var_name>", "description": "A global variable definition. Global variables can be referenced in scripts, SQL statements, or commands.", "type": "string" diff --git a/docs/schemas/event-log-msg-detected-v1.schema.json b/docs/schemas/event-log-msg-detected-v1.schema.json index 30ec3a2..a82b01f 100644 --- a/docs/schemas/event-log-msg-detected-v1.schema.json +++ b/docs/schemas/event-log-msg-detected-v1.schema.json @@ -41,7 +41,7 @@ "title": "/values", "type": "object", "patternProperties": { - "([\\w\\-]+)": { + "^([\\w\\-]+)$": { "title": "/values/<name>", "type": [ "boolean", diff --git a/docs/schemas/format-v1.schema.json b/docs/schemas/format-v1.schema.json index fe781f9..d96a285 100644 --- a/docs/schemas/format-v1.schema.json +++ b/docs/schemas/format-v1.schema.json @@ -10,7 +10,7 @@ } }, "patternProperties": { - "(\\w+)": { + "^(\\w+)$": { "description": "The definition of a log file format.", "title": "/<format_name>", "type": "object", @@ -20,7 +20,7 @@ "title": "/<format_name>/regex", "type": "object", "patternProperties": { - "([^/]+)": { + "^(.+)$": { "description": "The set of patterns used to match log messages", "title": "/<format_name>/regex/<pattern_name>", "type": "object", @@ -75,16 +75,50 @@ "description": "A regular expression that restricts this format to log files with a matching name", "type": "string" }, - "mime-types": { - "title": "/<format_name>/mime-types", - "description": "A list of mime-types this format should be used for", - "type": "array", - "items": { - "type": "string", - "enum": [ - "application/vnd.tcpdump.pcap" - ] - } + "converter": { + "description": "Describes how the file format can be detected and converted to a log that can be understood by lnav", + "title": "/<format_name>/converter", + "type": "object", + "properties": { + "type": { + "title": "/<format_name>/converter/type", + "description": "The MIME type", + "type": "string" + }, + "header": { + "description": "File header detection definitions", + "title": "/<format_name>/converter/header", + "type": "object", + "properties": { + "expr": { + "description": "The expressions used to check if a file header matches this file format", + "title": "/<format_name>/converter/header/expr", + "type": "object", + "patternProperties": { + "^(\\w+)$": { + "title": "/<format_name>/converter/header/expr/<header_expr_name>", + "description": "SQLite expression", + "type": "string" + } + }, + "additionalProperties": false + }, + "size": { + "title": "/<format_name>/converter/header/size", + "description": "The minimum size required for this header type", + "type": "integer" + } + }, + "additionalProperties": false + }, + "command": { + "title": "/<format_name>/converter/command", + "description": "The script used to convert the file", + "type": "string", + "pattern": "[\\w\\.\\-]+" + } + }, + "additionalProperties": false }, "level-field": { "title": "/<format_name>/level-field", @@ -165,6 +199,123 @@ "description": "The name of the operation-id field in the log message pattern", "type": "string" }, + "opid": { + "description": "Definitions related to operations found in logs", + "title": "/<format_name>/opid", + "type": "object", + "properties": { + "subid": { + "title": "/<format_name>/opid/subid", + "description": "The field that holds the ID for a sub-operation", + "type": "string" + }, + "description": { + "description": "Define how to construct a description of an operation", + "title": "/<format_name>/opid/description", + "type": "object", + "patternProperties": { + "^([\\w\\.\\-]+)$": { + "description": "A type of description for this operation", + "title": "/<format_name>/opid/description/<opid_descriptor>", + "type": "object", + "properties": { + "format": { + "description": "Defines the elements of this operation description", + "title": "/<format_name>/opid/description/<opid_descriptor>/format", + "type": "array", + "items": { + "type": "object", + "properties": { + "field": { + "title": "/<format_name>/opid/description/<opid_descriptor>/format/field", + "description": "The field to include in the operation description", + "type": "string" + }, + "extractor": { + "title": "/<format_name>/opid/description/<opid_descriptor>/format/extractor", + "description": "The regex used to extract content for the operation description", + "type": "string" + }, + "prefix": { + "title": "/<format_name>/opid/description/<opid_descriptor>/format/prefix", + "description": "A string to prepend to this field in the description", + "type": "string" + }, + "suffix": { + "title": "/<format_name>/opid/description/<opid_descriptor>/format/suffix", + "description": "A string to append to this field in the description", + "type": "string" + }, + "joiner": { + "title": "/<format_name>/opid/description/<opid_descriptor>/format/joiner", + "description": "A string to insert between instances of this field when the field is found more than once", + "type": "string" + } + }, + "additionalProperties": false + } + } + }, + "additionalProperties": false + } + }, + "additionalProperties": false + }, + "sub-description": { + "description": "Define how to construct a description of a sub-operation", + "title": "/<format_name>/opid/sub-description", + "type": "object", + "patternProperties": { + "^([\\w\\.\\-]+)$": { + "description": "A type of description for this sub-operation", + "title": "/<format_name>/opid/sub-description/<subid_descriptor>", + "type": "object", + "properties": { + "format": { + "description": "Defines the elements of this operation description", + "title": "/<format_name>/opid/sub-description/<subid_descriptor>/format", + "type": "array", + "items": { + "type": "object", + "properties": { + "field": { + "title": "/<format_name>/opid/sub-description/<subid_descriptor>/format/field", + "description": "The field to include in the operation description", + "type": "string" + }, + "extractor": { + "title": "/<format_name>/opid/sub-description/<subid_descriptor>/format/extractor", + "description": "The regex used to extract content for the operation description", + "type": "string" + }, + "prefix": { + "title": "/<format_name>/opid/sub-description/<subid_descriptor>/format/prefix", + "description": "A string to prepend to this field in the description", + "type": "string" + }, + "suffix": { + "title": "/<format_name>/opid/sub-description/<subid_descriptor>/format/suffix", + "description": "A string to append to this field in the description", + "type": "string" + }, + "joiner": { + "title": "/<format_name>/opid/sub-description/<subid_descriptor>/format/joiner", + "description": "A string to insert between instances of this field when the field is found more than once", + "type": "string" + } + }, + "additionalProperties": false + } + } + }, + "additionalProperties": false + } + }, + "additionalProperties": false + } + }, + "additionalProperties": false + }, "ordered-by-time": { "title": "/<format_name>/ordered-by-time", "description": "Indicates that the order of messages in the file is time-based.", @@ -175,7 +326,7 @@ "title": "/<format_name>/level", "type": "object", "patternProperties": { - "(trace|debug[2345]?|info|stats|notice|warning|error|critical|fatal)": { + "^(trace|debug[2345]?|info|stats|notice|warning|error|critical|fatal)$": { "title": "/<format_name>/level/<level>", "description": "The regular expression used to match the log text for this level. For JSON logs with numeric levels, this should be the number for the corresponding level.", "type": [ @@ -191,7 +342,7 @@ "title": "/<format_name>/value", "type": "object", "patternProperties": { - "([^/]+)": { + "^(.+)$": { "description": "The set of values captured by the log message patterns", "title": "/<format_name>/value/<value_name>", "type": "object", @@ -231,7 +382,7 @@ "title": "/<format_name>/value/<value_name>/unit/scaling-factor", "type": "object", "patternProperties": { - "([^/]+)": { + "^(.+)$": { "title": "/<format_name>/value/<value_name>/unit/scaling-factor/<scale>", "type": "object", "properties": { @@ -304,7 +455,7 @@ "title": "/<format_name>/tags", "type": "object", "patternProperties": { - "([\\w:;\\._\\-]+)": { + "^([\\w:;\\._\\-]+)$": { "description": "The name of the tag to apply", "title": "/<format_name>/tags/<tag_name>", "type": "object", @@ -367,11 +518,79 @@ }, "additionalProperties": false }, + "partitions": { + "description": "The partitions to automatically apply to log messages", + "title": "/<format_name>/partitions", + "type": "object", + "patternProperties": { + "^([\\w:;\\._\\-]+)$": { + "description": "The type of partition to apply", + "title": "/<format_name>/partitions/<partition_type>", + "type": "object", + "properties": { + "paths": { + "description": "Restrict partitioning to the given paths", + "title": "/<format_name>/partitions/<partition_type>/paths", + "type": "array", + "items": { + "type": "object", + "properties": { + "glob": { + "title": "/<format_name>/partitions/<partition_type>/paths/glob", + "description": "The glob to match against file paths", + "type": "string", + "examples": [ + "*/system.log*" + ] + } + }, + "additionalProperties": false + } + }, + "pattern": { + "title": "/<format_name>/partitions/<partition_type>/pattern", + "description": "The regular expression to match against the body of the log message", + "type": "string", + "examples": [ + "\\w+ is down" + ] + }, + "description": { + "title": "/<format_name>/partitions/<partition_type>/description", + "description": "A description of this partition", + "type": "string" + }, + "level": { + "title": "/<format_name>/partitions/<partition_type>/level", + "description": "Constrain hits to log messages with this level", + "type": "string", + "enum": [ + "trace", + "debug5", + "debug4", + "debug3", + "debug2", + "debug", + "info", + "stats", + "notice", + "warning", + "error", + "critical", + "fatal" + ] + } + }, + "additionalProperties": false + } + }, + "additionalProperties": false + }, "action": { "title": "/<format_name>/action", "type": "object", "patternProperties": { - "(\\w+)": { + "^(\\w+)$": { "title": "/<format_name>/action/<action_name>", "type": [ "string", @@ -499,7 +718,8 @@ "enum": [ "abbrev", "truncate", - "dot-dot" + "dot-dot", + "last-word" ] }, "text-transform": { @@ -532,7 +752,7 @@ "title": "/<format_name>/search-table", "type": "object", "patternProperties": { - "(\\w+)": { + "^(\\w+)$": { "description": "The set of search tables to be automatically defined", "title": "/<format_name>/search-table/<table_name>", "type": "object", @@ -578,7 +798,7 @@ "title": "/<format_name>/highlights", "type": "object", "patternProperties": { - "([^/]+)": { + "^(.+)$": { "description": "The definition of a highlight", "title": "/<format_name>/highlights/<highlight_name>", "type": "object", diff --git a/docs/source/_static/theme_overrides.css b/docs/source/_static/theme_overrides.css index c25d308..39a5db1 100644 --- a/docs/source/_static/theme_overrides.css +++ b/docs/source/_static/theme_overrides.css @@ -22,36 +22,7 @@ table.query-results p { font-size: 0.9em !important; } -DL DT:target, :target > H2, :target > H3, span:target + H2, span:target + H3 { - border: 0 !important; - border-bottom: 1px solid #d3d381 !important; - background: #ffc !important; - /* padding: 1em !important;*/ -} DL DT { - border: 0 !important; - background: inherit !important; - border-bottom: 1px solid #c3c0ee !important; - /* display: block !important; */ -} - -kbd { - padding: 0.1em 0.6em; - border: 1px solid #ccc; - font-size: 11px; - font-family: Arial, Helvetica, sans-serif; - background-color: #f7f7f7; - color: #333; - -moz-box-shadow: 0 1px 0px rgba(0, 0, 0, 0.2), 0 0 0 2px #ffffff inset; - -webkit-box-shadow: 0 1px 0px rgba(0, 0, 0, 0.2), 0 0 0 2px #ffffff inset; - box-shadow: 0 1px 0px rgba(0, 0, 0, 0.2), 0 0 0 2px #ffffff inset; - -moz-border-radius: 3px; - -webkit-border-radius: 3px; - border-radius: 3px; - display: inline-block; - margin: 0 0.1em 0.1em 0.1em; - text-shadow: 0 1px 0 #fff; - line-height: 1.4; - white-space: nowrap; + border-bottom: 1px solid #333 !important; } diff --git a/docs/source/cli.rst b/docs/source/cli.rst index 565e54f..93119e5 100644 --- a/docs/source/cli.rst +++ b/docs/source/cli.rst @@ -38,13 +38,19 @@ Options Execute the given lnav command, SQL query, or lnav script. The argument must be prefixed with the character used to enter the prompt - to distinguish between the different types (i.e. ':', ';', '|'). - This option can be given multiple times. + to distinguish between the different types (i.e. :code:`:`, :code:`;`, + :code:`|`, :code:`/`). This option can be given multiple times. .. option:: -f <path> Execute the given command file. This option can be given multiple times. +.. option:: -e <command-line> + + Execute the given shell command-line and display its output. This is + equivalent to executing the :code:`:sh` command and passing the + :option:`-N` flag. This option can be given multiple times. + .. option:: -I <path> Add a configuration directory. @@ -76,23 +82,21 @@ Options Recursively load files from the given base directories. -.. option:: -t - - Prepend timestamps to the lines of data being read in on the standard input. - -.. option:: -w <path> - - Write the contents of the standard input to this file. - .. option:: -V Print the version of lnav. +.. option:: -v + + Print extra information during operations. + .. option:: -q - Do not print the log messages after executing all of the commands. + Do not print informational messages. +.. _management_cli: + Management Mode (v0.11.0+) -------------------------- @@ -107,12 +111,41 @@ Options Switch to management mode. This must be the first option passed on the command-line. +.. option:: -I <path> + + Add a configuration directory. + Subcommands ^^^^^^^^^^^ -.. option:: regex101 import <regex101-url> <format-name> [<regex-name>] +.. option:: config get - Convert a regex101.com entry into a skeleton log format file. + Print out the current configuration as JSON on the standard output. + +.. option:: config blame + + Print out the configuration options as JSON-Pointers and the + file/line-number where the configuration is sourced from. + +.. option:: config file-options <path> + + Print out the options that will be applied to the given file. The + options are stored in the :file:`file-options.json` file in the + **lnav** configuration directory. The only option available at + the moment is the timezone to be used for log message timestamps + that do not include a zone. The timezone for a file can be set + using the :ref:`:set-file-timezone<set_file_timezone>` command + and cleared with the :ref:`:clear-file-timezone<clear_file_timezone>` + command. + +.. option:: format <format-name> get + + Print information about the given log format. + +.. option:: format <format-name> source + + Print the name of the first file that contained this log format + definition. .. option:: format <format-name> regex <regex-name> push @@ -122,6 +155,22 @@ Subcommands Pull changes to a regex that was previously pushed to regex101.com . +.. option:: piper clean + + Remove all of the files that stored data that was piped into **lnav**. + +.. option:: piper list + + List all of the data that was piped into **lnav** from oldest to newest. + The listing will show the creation time, the URL you can use to reopen + the data, and a description of the data. Passing the :option:`-v` + option will print out additional metadata that was captured, such as + the current working directory of **lnav** and the environment variables. + +.. option:: regex101 import <regex101-url> <format-name> [<regex-name>] + + Convert a regex101.com entry into a skeleton log format file. + Environment Variables --------------------- @@ -142,27 +191,27 @@ Environment Variables .. envvar:: TZ - The timezone setting is used in some log formats to convert UTC timestamps - to the local timezone. + The timezone setting is used in some log formats to convert timestamps + with a timezone to the local timezone. Examples -------- - To load and follow the system syslog file: +To load and follow the system syslog file: - .. prompt:: bash +.. prompt:: bash - lnav + lnav - To load all of the files in :file:`/var/log`: +To load all of the files in :file:`/var/log`: - .. prompt:: bash +.. prompt:: bash - lnav /var/log + lnav /var/log - To watch the output of make with timestamps prepended: +To watch the output of make: - .. prompt:: bash +.. prompt:: bash - make 2>&1 | lnav -t + lnav -e 'make -j4' diff --git a/docs/source/commands.rst b/docs/source/commands.rst index d83a07e..98de0a2 100644 --- a/docs/source/commands.rst +++ b/docs/source/commands.rst @@ -68,9 +68,13 @@ an error message in the status bar, like so: .. note:: The following commands can be disabled by setting the ``LNAVSECURE`` environment variable before executing the **lnav** binary: + - :code:`:cd` + - :code:`:export-session-to` - :code:`:open` - :code:`:pipe-to` - :code:`:pipe-line-to` + - :code:`:redirecto-to` + - :code:`:sh` - :code:`:write-*-to` This makes it easier to run **lnav** in restricted environments without the diff --git a/docs/source/conf.py b/docs/source/conf.py index 3a2d8f2..4f94dba 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -229,7 +229,6 @@ lexers['custsqlite'] = CustSqliteLexer(startinline=True) # coming with Sphinx (named 'sphinx.ext.*') or your custom ones. extensions = [ 'sphinx.ext.coverage', - "sphinx_rtd_theme", 'sphinx_copybutton', 'sphinx-jsonschema', 'sphinx-prompt', @@ -257,9 +256,9 @@ copyright = u'2023, Tim Stack' # built documents. # # The short X.Y version. -version = '0.11' +version = '0.12' # The full version, including alpha/beta/rc tags. -release = '0.11.1' +release = '0.12.1' # The language for content autogenerated by Sphinx. Refer to documentation # for a list of supported languages. @@ -299,7 +298,7 @@ pygments_style = 'sphinx' # The theme to use for HTML and HTML Help pages. See the documentation for # a list of builtin themes. -html_theme = 'sphinx_rtd_theme' +html_theme = 'furo' # Theme options are theme-specific and customize the look and feel of a theme # further. For a list of options available for each theme, see the @@ -312,7 +311,7 @@ html_theme_options = { # The name for this set of Sphinx documents. If None, it defaults to # "<project> v<release> documentation". -# html_title = None +html_title = f"lnav v{release}" # A shorter title for the navigation bar. Default is the same as html_title. # html_short_title = None diff --git a/docs/source/config.rst b/docs/source/config.rst index 5294e52..d4fc9d5 100644 --- a/docs/source/config.rst +++ b/docs/source/config.rst @@ -1,23 +1,34 @@ - .. _Configuration: Configuration ============= -The configuration for **lnav** is stored in the following JSON files in -:file:`~/.lnav`: - -* :file:`config.json` -- Contains local customizations that were done using the - :code:`:config` command. -* :file:`configs/default/*.json` -- The default configuration files that are - built into lnav are written to this directory with :file:`.sample` appended. - Removing the :file:`.sample` extension and editing the file will allow you to - do basic customizations. -* :file:`configs/installed/*.json` -- Contains configuration files installed - using the :option:`-i` flag (e.g. :code:`$ lnav -i /path/to/config.json`). -* :file:`configs/*/*.json` -- Other directories that contain :file:`*.json` - files will be loaded on startup. This structure is convenient for installing - **lnav** configurations, like from a git repository. +The configuration for **lnav** is stored in the following JSON files +where :file:`<lnav-home>` refers to the location in the :envvar:`HOME` +directory where files are stored, either (:file:`~/.lnav` or +:file:`~/.config/lnav`): + +#. Builtin -- The default configuration is shipped inside the **lnav** binary. + +#. :file:`/etc/lnav/configs/*/*.json` -- System-wide configuration files can + be installed here to make it available to all users. + +#. :file:`<lnav-home>/configs/default/*.json` -- The default configuration + files that are built into lnav are written to this directory with :file:`.sample` + appended. Removing the :file:`.sample` extension and editing the file will + allow you to do basic customizations. + +#. :file:`<lnav-home>/configs/*/*.json` -- Other directories that contain :file:`*.json` + files will be loaded on startup. This structure is convenient for installing + **lnav** configurations, like from a git repository. The :file:`configs/installed` + directory is reserved for files that are installed using the :option:`-i` + flag (e.g. :code:`$ lnav -i /path/to/config.json`). + +#. :file:`-I <path>/configs/*/*.json` -- Include directories passed on the + command-line can have a :file:`configs` directory that will also be searched. + +#. :file:`<lnav-home>/config.json` -- Contains local customizations that were + done using the :code:`:config` command. A valid **lnav** configuration file must contain an object with the :code:`$schema` property, like so: @@ -34,6 +45,14 @@ A valid **lnav** configuration file must contain an object with the directly. See the :ref:`Log Formats<log_formats>` chapter for more information. +.. note:: + + Configuration files are read in the above directory order and sorted + by path name. The internal configuration is updated as files are + parsed, so one file can overwrite the settings from another. You can + use the :ref:`Management CLI<management_cli>` to get the final + configuration and where the value came from for a particular + configuration option. Options ------- @@ -115,39 +134,39 @@ You can copy the code block, save it to a file in definition, see one of the definitions built into **lnav**, like `monocai <https://github.com/tstack/lnav/blob/master/src/themes/monocai.json>`_. - .. code-block:: json - - { - "$schema": "https://lnav.org/schemas/config-v1.schema.json", - "ui": { - "theme-defs": { - "example1": { - "vars": { - "black": "#2d2a2e" - }, - "styles": { - "text": { - "color": "#f6f6f6", - "background-color": "$black" - } - } - } - } - } - } +.. code-block:: json + + { + "$schema": "https://lnav.org/schemas/config-v1.schema.json", + "ui": { + "theme-defs": { + "example1": { + "vars": { + "black": "#2d2a2e" + }, + "styles": { + "text": { + "color": "#f6f6f6", + "background-color": "$black" + } + } + } + } + } + } Reference ^^^^^^^^^ -.. jsonschema:: ../schemas/config-v1.schema.json#/properties/ui/properties/theme-defs/patternProperties/([\w\-]+)/properties/vars +.. jsonschema:: ../schemas/config-v1.schema.json#/properties/ui/properties/theme-defs/patternProperties/^([\w\-]+)$/properties/vars -.. jsonschema:: ../schemas/config-v1.schema.json#/properties/ui/properties/theme-defs/patternProperties/([\w\-]+)/properties/styles +.. jsonschema:: ../schemas/config-v1.schema.json#/properties/ui/properties/theme-defs/patternProperties/^([\w\-]+)$/properties/styles -.. jsonschema:: ../schemas/config-v1.schema.json#/properties/ui/properties/theme-defs/patternProperties/([\w\-]+)/properties/syntax-styles +.. jsonschema:: ../schemas/config-v1.schema.json#/properties/ui/properties/theme-defs/patternProperties/^([\w\-]+)$/properties/syntax-styles -.. jsonschema:: ../schemas/config-v1.schema.json#/properties/ui/properties/theme-defs/patternProperties/([\w\-]+)/properties/status-styles +.. jsonschema:: ../schemas/config-v1.schema.json#/properties/ui/properties/theme-defs/patternProperties/^([\w\-]+)$/properties/status-styles -.. jsonschema:: ../schemas/config-v1.schema.json#/properties/ui/properties/theme-defs/patternProperties/([\w\-]+)/properties/log-level-styles +.. jsonschema:: ../schemas/config-v1.schema.json#/properties/ui/properties/theme-defs/patternProperties/^([\w\-]+)$/properties/log-level-styles .. _theme_style: @@ -198,7 +217,7 @@ to the key. Reference ^^^^^^^^^ -.. jsonschema:: ../schemas/config-v1.schema.json#/properties/ui/properties/keymap-defs/patternProperties/([\w\-]+) +.. jsonschema:: ../schemas/config-v1.schema.json#/properties/ui/properties/keymap-defs/patternProperties/^([\w\-]+)$ Log Handling @@ -208,6 +227,18 @@ The handling of logs is largely determined by the :ref:`log file formats<log_formats>`, this section covers options that are not specific to a particular format. +Timezone Conversion (v0.12.0+) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Log messages that have a numeric timezone, like :code:`-03:00` or :code:`Z` +for UTC, will be converted to the local timezone as given by the :envvar:`TZ` +environment variable. For example, a timestamp ending in `-03:00` will be treated +as three hours behind UTC and then adjusted to the local timezone. + +This behavior can be disabled by setting the +:code:`/log/date-time/convert-zoned-to-local` configuration property to +:code:`false`. + Watch Expressions (v0.11.0+) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -243,10 +274,44 @@ From there, you can create a SQLite trigger on the :code:`lnav_events` table that will examine the event contents and perform an action. See the :ref:`Events` section for more information on handling events. +Annotations (v0.12.0+) +^^^^^^^^^^^^^^^^^^^^^^ + +Annotations are content generated by a script for a given log message and +displayed along with the message, like comments and tags. Since the script +is run asynchronously, it can do complex analysis without delaying loading +or interrupting the viewing experience. An annotation is defined by a +condition and a handler in the **lnav** configuration. The condition is +tested against a log message to determine if the annotation is applicable. +If it is, the handler script will be executed for that log message when +the user runs the :ref:`:annotate<annotate>` command. + +Conditions are SQLite expressions like the ones passed to +:ref:`:filter-expr<filter_expr>` where the expression is appended to +:code:`SELECT 1 WHERE`. The expression can use bound variables that +correspond to the columns that would be in the format table and are +prefixed by a colon (:code:`:`). For example, the standard +:code:`log_opid` table column can be access by using :code:`:log_opid`. + +.. note:: The expression is executed with bound variables because it + can be applied to log messages from multiple formats. Writing an + expression that could handle different formats would be more + challenging. In this approach, variables for log message fields + that are not part of a format will evaluate to :code:`NULL`. + +Handlers are executable script files that should be co-located with +the configuration file that defined the annotation. The handler will +be executed and a JSON object with log message data fed in on the +standard input. The handler should then generate the annotation +content on the standard output. The output is treated as Markdown, +so the content can be styled as desired. + Reference ^^^^^^^^^ -.. jsonschema:: ../schemas/config-v1.schema.json#/properties/log/properties/watch-expressions/patternProperties/([\w\-]+) +.. jsonschema:: ../schemas/config-v1.schema.json#/properties/log/properties/watch-expressions/patternProperties/^([\w\.\-]+)$ +.. jsonschema:: ../schemas/config-v1.schema.json#/properties/log/properties/annotations/patternProperties/^([\w\.\-]+)$ + .. _tuning: @@ -261,6 +326,8 @@ command. .. jsonschema:: ../schemas/config-v1.schema.json#/properties/tuning/properties/clipboard +.. jsonschema:: ../schemas/config-v1.schema.json#/properties/tuning/properties/piper + .. jsonschema:: ../schemas/config-v1.schema.json#/definitions/clip-commands .. jsonschema:: ../schemas/config-v1.schema.json#/properties/tuning/properties/file-vtab @@ -268,3 +335,7 @@ command. .. jsonschema:: ../schemas/config-v1.schema.json#/properties/tuning/properties/logfile .. jsonschema:: ../schemas/config-v1.schema.json#/properties/tuning/properties/remote/properties/ssh + +.. _url_scheme: + +.. jsonschema:: ../schemas/config-v1.schema.json#/properties/tuning/properties/url-scheme diff --git a/docs/source/cookbook.rst b/docs/source/cookbook.rst index 3eb0ffd..9523235 100644 --- a/docs/source/cookbook.rst +++ b/docs/source/cookbook.rst @@ -1,4 +1,3 @@ - .. _Cookbook: Cookbook @@ -69,9 +68,9 @@ Count client IPs in web access logs To count the occurrences of an IP in web access logs and order the results from highest to lowest: - .. code-block:: custsqlite +.. code-block:: custsqlite - ;SELECT c_ip, count(*) as hits FROM access_log GROUP BY c_ip ORDER BY hits DESC + ;SELECT c_ip, count(*) as hits FROM access_log GROUP BY c_ip ORDER BY hits DESC Show only lines where a numeric field is in a range @@ -81,9 +80,9 @@ The :ref:`:filter-expr<filter_expr>` command can be used to filter web access logs to only show lines where the number of bytes transferred to the client is between 10,000 and 40,000 bytes like so: - .. code-block:: custsqlite +.. code-block:: custsqlite - :filter-expr :sc_bytes BETWEEN 10000 AND 40000 + :filter-expr :sc_bytes BETWEEN 10000 AND 40000 Generating a Report diff --git a/docs/source/formats.rst b/docs/source/formats.rst index b30713d..d57add1 100644 --- a/docs/source/formats.rst +++ b/docs/source/formats.rst @@ -1,7 +1,11 @@ .. _log_formats: +*********** Log Formats -=========== +*********** + +Built-in Formats +================ Log files loaded into **lnav** are parsed based on formats defined in configuration files. Many @@ -19,6 +23,9 @@ The following log formats are built into **lnav**: :widths: 8 5 20 :file: format-table.csv +XSV Formats +----------- + In addition to the above formats, the following self-describing formats are supported: @@ -33,6 +40,24 @@ supported: self-describing, so **lnav** will read the header to determine the shape of the file. +JSON-lines +---------- + +Logs encoded as `JSON-lines <https://jsonlines.org>`_ can be parsed and +pretty-printed in lnav by creating a log format file. The format file +is a bit simpler to create since it doesn't require a regular expression +to match plain text. Instead, the format defines the relevant fields +and provides a :code:`line-format` array that specifies how the fields +in the JSON object should be displayed. + +See the following formats that are built into lnav as examples: + +* `cloudflare_log.json <https://github.com/tstack/lnav/blob/master/src/formats/cloudflare_log.json>`_ +* `github_events_log.json <https://github.com/tstack/lnav/blob/master/src/formats/github_events_log.json>`_ + +logfmt +------ + There is also basic support for the `logfmt <https://brandur.org/logfmt>`_ convention for formatting log messages. Files that use this format must have the entire line be key/value pairs and the timestamp contained in a @@ -41,9 +66,8 @@ quite follow this formatting, but wraps logfmt data with another recognized format, you can use the :ref:`logfmt2json` SQL function to convert the data into JSON for further analysis. - Defining a New Format ---------------------- +===================== New log formats can be defined by placing JSON configuration files in subdirectories of the :file:`/etc/lnav/formats` and :file:`~/.lnav/formats/` @@ -55,7 +79,7 @@ modify existing ones. Format directories can also contain '.sql' and '.lnav' script files that can be used automate log file analysis. Creating a Format Using Regex101.com (v0.11.0+) -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +----------------------------------------------- For plain-text log files, the easiest way to create a log format definition is to create the regular expression that recognizes log messages using @@ -81,7 +105,7 @@ default to :code:`string`, but you'll want to change them to the appropriate type. Format File Reference -^^^^^^^^^^^^^^^^^^^^^ +--------------------- An **lnav** format file must contain a single JSON object, preferably with a :code:`$schema` property that refers to the @@ -133,6 +157,30 @@ object with the following fields: :json: True if each log line is JSON-encoded. +:converter: An object that describes how an input file can be detected and + then converted to a form that can be interpreted by **lnav**. For + example, a PCAP file is in a binary format that cannot be handled natively + by **lnav**. However, a PCAP file can be converted by :file:`tshark` + into JSON-lines that can be handled by **lnav**. So, this configuration + describes how the input file format can be detected and converted. See + `Automatic File Conversion`_ for more information. + + :header: An object that describes how to match the header of the input + file. + + :expr: An object that contains SQLite expressions that can be used to + check if the input file's header is of this type. The property + name is the name of the expression and the value is the expression. + The expression is evaluated with the following variables: + + :\:header: The hex-encoded version of the header content. + + :\:filepath: The path to the input file. + + :size: The minimum size of header that is needed to do the match. + + :command: The command to execute to convert the input file. + :line-format: An array that specifies the text format for JSON-encoded log messages. Log files that are JSON-encoded will have each message converted from the raw JSON encoding into this format. Each element @@ -179,6 +227,9 @@ object with the following fields: :truncate: Truncates any text past the maximum width. :dot-dot: Cuts out the middle of the text and replaces it with two dots (i.e. '..'). + :last-word: Removes all but the last word in text with dot, dash, + forward-slash, or colon separators. For example, "com.example.foo" + would be shortened to "foo". (v0.8.2+) :timestamp-format: The timestamp format to use when displaying the time @@ -319,6 +370,19 @@ object with the following fields: :glob: A glob pattern to check against the log files read by lnav. +:partitions: This object contains a description of partitions that should + automatically be created in the log view. + + :pattern: The regular expression evaluated over a line in the log file as + it is read in. If there is a match, the log message the line is a part + of will be used as the start of the partition. The name of the + partition will be taken from any captures in the regex. + :paths: This array contains objects that define restrictions on the file + paths in which partitions will be created. The objects in this array + can contain: + + :glob: A glob pattern to check against the log files read by lnav. + .. _format_sample: :sample: A list of objects that contain sample log messages. All formats @@ -455,6 +519,15 @@ hash are treated as comments. The following variables are defined in a script: The arguments passed to the script. +.. envvar:: LNAV_HOME_DIR + + The path to the directory where the user's **lnav** configuration is stored. + +.. envvar:: LNAV_WORK_DIR + + The path to the directory where **lnav** caches files, like archives that + have been unpacked or piper captures. + Remember that you need to use the :ref:`:eval<eval>` command when referencing variables in most **lnav** commands. Scripts can provide help text to be displayed during interactive usage by adding the following tags in a comment @@ -482,7 +555,7 @@ header: :eval :filter-out ${pattern} VSCode Extension -^^^^^^^^^^^^^^^^ +---------------- The `lnav VSCode Extension <https://marketplace.visualstudio.com/items?itemName=lnav.lnav>`_ can be installed to add syntax highlighting to lnav scripts. @@ -533,7 +606,7 @@ Executing the format file should then install it automatically: .. _format_order: Format Order When Scanning a File ---------------------------------- +================================= When **lnav** loads a file, it tries each log format against the first 15,000 lines [#]_ of the file trying to find a match. When a match is found, that log @@ -559,3 +632,31 @@ will win. .. [#] The maximum number of lines to check can be configured. See the :ref:`tuning` section for more details. + +Automatic File Conversion +========================= + +File formats that are not naturally understood by **lnav** can be +automatically detected and converted to a usable form using the +:code:`converter` property. For example, PCAP files can be +detected and converted to a JSON-lines form using :code:`tshark`. +The conversion process works as follows: + +#. The first 1024 bytes of the file are read, if available. +#. This header is converted into a hex string. +#. For each log format that has defined a :code:`converter`, + every "header expression" is evaluated to see if there is a + match. The header expressions are SQLite expressions where + the following variables are defined: + + :\:header: A string containing the header as a hex string. + + :\:filepath: The path to the file. + +#. If a match is found, the converter script defined in the + log format will be invoked and passed the format name and + path to the file as arguments. The script should write + the converted form of the input file on its standard output. + Any errors should be written to the standard error. +#. The log format will be associated with the original file will + be used to interpret the converted file. diff --git a/docs/source/hotkeys.rst b/docs/source/hotkeys.rst index 4bad1d9..7b065e7 100644 --- a/docs/source/hotkeys.rst +++ b/docs/source/hotkeys.rst @@ -7,6 +7,23 @@ This reference covers the keys used to control **lnav**. Consult the `built-in help <https://github.com/tstack/lnav/blob/master/src/help.txt>`_ in **lnav** for a more detailed explanation of each key. +Global +------ + +.. list-table:: + :header-rows: 1 + :widths: 6 20 + + * - Keypress + - Command + + * - :kbd:`Ctrl` + :kbd:`C` + - If the focused line is from a file connected to an open pipe (the "↻" + icon will be next to the file name), a SIGINT will be sent to the + child process. Otherwise, **lnav** will quickly exit. If **lnav** + seems to be stuck in a loop, pressing :kbd:`Ctrl` + :kbd:`C` three + times will trigger an abort exit. + Spatial Navigation ------------------ @@ -148,9 +165,10 @@ status bar. * - Keypress - Description + * - :kbd:`\`` + - Focus on the breadcrumb bar. * - :kbd:`ENTER` - - Focus on the breadcrumb bar. Or, if the bar is currently focused, - accept the selected value and drop focus. + - If the bar is currently focused, accept the selected value and drop focus. * - :kbd:`Escape` - Drop focus on the breadcrumb bar. * - :kbd:`←` @@ -223,7 +241,9 @@ Display * - :kbd:`Shift` + :kbd:`p` - Switch to/from the pretty-printed view of the displayed log or text files * - :kbd:`Shift` + :kbd:`t` - - Display elapsed time between lines + - Display the elapsed time from a bookmark to a given line. In the TEXT view, + this only works for content that was captured from stdin or a :code:`:sh` + command. * - :kbd:`t` - Switch to/from the text file view * - :kbd:`i` diff --git a/docs/source/index.rst b/docs/source/index.rst index d94c0eb..29d2be1 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -2,7 +2,8 @@ Welcome to lnav's documentation! ================================ The `Log File Navigator <http://lnav.org>`_ (**lnav**) is an advanced log file -viewer for the console. +viewer for the console. If you have a bunch of log files that you need to +look through to find issues, **lnav** is the tool for you. Contents: diff --git a/docs/source/intro.rst b/docs/source/intro.rst index 96b0612..28172a1 100644 --- a/docs/source/intro.rst +++ b/docs/source/intro.rst @@ -11,16 +11,12 @@ navigate through your logs using a variety of :ref:`hotkeys<hotkeys>`. for doing things like applying filters, tagging messages, and more. You can then analyze your log messages using the :ref:`sql-ext`. -Development ------------ +.. figure:: lnav-intro-screen.png + :align: center + :alt: Screenshot of lnav showing a mix of syslog and web access_log messages. -Development of lnav is hosted on `GitHub <https://github.com/tstack/lnav/>`_. + Screenshot of **lnav** viewing syslog and web access_log messages. -`Issues <https://github.com/tstack/lnav/issues>`_ should be used for bugs -and feature requests. - -`Discussions <https://github.com/tstack/lnav/discussions>`_ should be used -for asking questions and sharing tips. Downloads --------- @@ -28,34 +24,7 @@ Downloads Binaries and source code for lnav can be downloaded from the `releases page <https://github.com/tstack/lnav/releases>`_. -When building from source, follow the steps below. - -Dependencies -^^^^^^^^^^^^ - -When compiling from source, the following dependencies are required: - -* `NCurses <http://www.gnu.org/s/ncurses/>`_ -* `PCRE2 <http://www.pcre.org>`_ -* `SQLite <http://www.sqlite.org>`_ -* `ZLib <http://wwww.zlib.net>`_ -* `Bzip2 <http://www.bzip.org>`_ -* `Readline <http://www.gnu.org/s/readline>`_ -* `libcurl <https://curl.haxx.se>`_ -* `libarchive <https://libarchive.org>`_ - -Installation -^^^^^^^^^^^^ - -Check the `downloads page <http://lnav.org/downloads>`_ to see if there are -packages for your operating system. To compile from source, use the following -commands: - -.. prompt:: bash - - ./configure - make - sudo make install +When building from source, follow the steps in :ref:`development`. Viewing Logs @@ -129,6 +98,7 @@ command can be used to the change the theme: The builtin themes are: `default <https://github.com/tstack/lnav/blob/master/src/themes/default-theme.json>`_, +`dracula <https://github.com/tstack/lnav/blob/master/src/themes/dracula.json>`_, `eldar <https://github.com/tstack/lnav/blob/master/src/themes/eldar.json>`_, `grayscale <https://github.com/tstack/lnav/blob/master/src/themes/grayscale.json>`_, `monocai <https://github.com/tstack/lnav/blob/master/src/themes/monocai.json>`_, @@ -143,16 +113,33 @@ To create or customize a theme, consult the :ref:`themes` section. Cursor Mode (v0.11.2+) ^^^^^^^^^^^^^^^^^^^^^^ -The default mode for scrolling in **lnav** is to move the contents of the -main view when the arrow keys are pressed. Any interactions, such as -jumping to a search hit, are then focused on the top line in the view. -Alternatively, you can enable "cursor" mode where there is a cursor line -in the view that is moved by the arrow keys and other interactions. You -can enable cursor mode with the following command: +The default mode for scrolling in **lnav** is "cursor" mode where +there is a cursor line in the view that is moved by the arrow keys +and other interactions. Any interactions, such as jumping to a +search hit, are then focused on that line. + +Alternatively, you can enable "top" mode where the contents of the +main view are moved when the arrow keys are pressed. Any +interactions, such as jumping to a search hit, are then focused +on the top line in the view. You can change to "top" mode with +the following command: + +.. code-block:: lnav + + :config /ui/movement/mode top + + +Mouse Support (v0.12.2+) +^^^^^^^^^^^^^^^^^^^^^^^^ + +Mouse support can be enabled temporarily by pressing :kbd:`F2` +and can be set as the default by executing the following command: .. code-block:: lnav - :config /ui/movement/mode cursor + :config /ui/mouse/mode enabled + +See :ref:`ui_mouse` for more details. Log Formats ^^^^^^^^^^^ @@ -162,3 +149,48 @@ parse the log messages using a log format definition. There are many log formats builtin and **lnav** will automatically determine the best format to use. In case your log file is not recognized, consult the :ref:`log_formats` section for information on how to create a format. + +.. _development: + +Development +----------- + +Development of lnav is hosted on `GitHub <https://github.com/tstack/lnav/>`_. + +`Issues <https://github.com/tstack/lnav/issues>`_ should be used for bugs +and feature requests. + +`Discussions <https://github.com/tstack/lnav/discussions>`_ should be used +for asking questions and sharing tips. + +Dependencies +^^^^^^^^^^^^ + +When compiling from source, the following dependencies are required: + +* `NCurses <http://www.gnu.org/s/ncurses/>`_ +* `PCRE2 <http://www.pcre.org>`_ +* `SQLite <http://www.sqlite.org>`_ +* `ZLib <http://wwww.zlib.net>`_ +* `Bzip2 <http://www.bzip.org>`_ +* `Readline <http://www.gnu.org/s/readline>`_ +* `libcurl <https://curl.haxx.se>`_ +* `libarchive <https://libarchive.org>`_ + +Installation +^^^^^^^^^^^^ + +Check the `downloads page <http://lnav.org/downloads>`_ to see if there are +packages for your operating system. To compile from source, use the following +commands: + +.. note:: + + Run :code:`./autogen.sh` if compiling from the git repository. + +.. prompt:: bash + + ./configure + make + sudo make install + diff --git a/docs/source/lnav-gantt-1.png b/docs/source/lnav-gantt-1.png Binary files differnew file mode 100644 index 0000000..306187c --- /dev/null +++ b/docs/source/lnav-gantt-1.png diff --git a/docs/source/lnav-gantt-2.png b/docs/source/lnav-gantt-2.png Binary files differnew file mode 100644 index 0000000..90bf8c1 --- /dev/null +++ b/docs/source/lnav-gantt-2.png diff --git a/docs/source/lnav-intro-screen.png b/docs/source/lnav-intro-screen.png Binary files differnew file mode 100644 index 0000000..39d8129 --- /dev/null +++ b/docs/source/lnav-intro-screen.png diff --git a/docs/source/lnav-make-check-log.png b/docs/source/lnav-make-check-log.png Binary files differnew file mode 100644 index 0000000..68ac351 --- /dev/null +++ b/docs/source/lnav-make-check-log.png diff --git a/docs/source/query-results.png b/docs/source/query-results.png Binary files differindex 14ae5b8..8343d71 100644 --- a/docs/source/query-results.png +++ b/docs/source/query-results.png diff --git a/docs/source/sqlext.rst b/docs/source/sqlext.rst index 221b100..3b0ee75 100644 --- a/docs/source/sqlext.rst +++ b/docs/source/sqlext.rst @@ -47,6 +47,14 @@ and maximum number of bytes returned by the server, grouped by IP address: ;SELECT c_ip, avg(sc_bytes), max(sc_bytes) FROM access_log GROUP BY c_ip +.. note:: + + For reference, the PRQL query would look like this: + + .. code-block:: elm + + from access_log | stats.by c_ip {average sc_bytes, max sc_bytes} + After pressing :kbd:`Enter`, SQLite will execute the query using **lnav**'s virtual table implementation to extract the data directly from the log files. Once the query has finished, the main window will switch to the DB view to @@ -71,6 +79,39 @@ The DB view has the following display features: the `jget`_ function. +PRQL Support (v0.12.1+) +----------------------- + +`PRQL <https://prql-lang.org>`_ is an alternative database query language +that compiles to SQLite. You can enter PRQL in the database query prompt +and lnav will switch accordingly. A major advantage of using PRQL is that +lnav can show previews of the results of the pipeline stages and provide +better tab completion options. + +A PRQL query starts with the :code:`from` keyword that specifies the table +to use as a data source. The next stage of a pipeline is started by +entering a pipe symbol (:code:`|`) followed by a +`PRQL transform <https://prql-lang.org/book/reference/stdlib/transforms/index.html>`_. +As you build the query in the prompt, lnav will display any relevant +help and preview for the current and previous stages of the pipeline. + +The following is a screenshot of lnav viewing a web access log with a +query in progress: + +.. figure:: ../assets/images/lnav-prql-preview.png + :align: center + + Screenshot of a PRQL query in progress + +The top half of the window is the usual log message view. Below that is +the online help panel showing the documentation for the :code:`stats.count_by` +PRQL function. lnav will show the help for what is currently under the +cursor. The next panel shows the preview data for the pipeline stage +that precedes the stage where the cursor is. In this case, the +results of :code:`from access_log`, which is the contents of the access +log table. The second preview window shows the result of the +pipeline stage where the cursor is located. + Log Tables ---------- @@ -105,6 +146,8 @@ The following columns are builtin and included in a :code:`SELECT *`: an :code:`UPDATE` or the :ref:`:comment<comment>` command. :log_tags: A JSON list of tags for the message. This column can be changed by an :code:`UPDATE` or the :ref:`:tag<tag>` command. + :log_annotations: A JSON object of annotations for this message. + This column is populated by the :ref:`:annotate<annotate>` command. :log_filters: A JSON list of filter IDs that matched this message The following columns are builtin and are hidden, so they will *not* be diff --git a/docs/source/sqltab.rst b/docs/source/sqltab.rst index 2531390..299ba2b 100644 --- a/docs/source/sqltab.rst +++ b/docs/source/sqltab.rst @@ -7,6 +7,7 @@ In addition to the tables generated for each log format, **lnav** includes the following tables/views: * `environ`_ +* `fstat(<path|pattern>)`_ * `lnav_events`_ * `lnav_file`_ * `lnav_file_metadata`_ @@ -31,7 +32,7 @@ by executing the '.schema' SQL command, like so:: environ ------- -The **environ** table gives you access to the **lnav** process' environment +The :code:`environ` table gives you access to the **lnav** process' environment variables. You can :code:`SELECT`, :code:`INSERT`, and :code:`UPDATE` environment variables, like so: @@ -55,6 +56,19 @@ named "FILENAME" and then open it in **lnav** by referencing it with :open $FILENAME +fstat(<path|pattern>) +--------------------- + +The :code:`fstat` table-valued function provides access to the local +file system. The function takes a file path or a glob pattern and +returns the results of :code:`lstat(2)` for the matching files. If +the parameter is a pattern that matches nothing, no rows will be +returned. If the parameter is a path for a non-existent file, a +row will be returned with the :code:`error` column set and the +stat columns as :code:`NULL`. To read the contents of a file, you +can :code:`SELECT` the hidden :code:`data` column. + + .. _table_lnav_events: lnav_events @@ -81,12 +95,22 @@ following columns are available in this table: :device: The device the file is stored on. :inode: The inode for the file on the device. :filepath: If this is a real file, it will be the absolute path. Otherwise, - it is a symbolic name. If it is a symbolic name, it can be UPDATEd so that - this file will be considered when saving and loading session information. + it is a symbolic name. If it is a symbolic name, it can be UPDATEd + so that this file will be considered when saving and loading session + information. + :mimetype: The detected MIME type of the file. + :content_id: The hash of some unique content in the file. :format: The log file format for the file. :lines: The number of lines in the file. :time_offset: The millisecond offset for timestamps. This column can be UPDATEd to change the offset of timestamps in the file. + :options_path: Options can be applied to files based on a path or glob + pattern. If this file matches a set of options, the matching path/pattern + is available in this column and the actual options themselves are in the + :code:`options` column. + :options: The options that are applicable to this file. Currently, the + only options available are for the timezone set by the + :ref:`:set-file-timezone<set_file_timezone>` command. lnav_file_metadata ------------------ @@ -169,6 +193,7 @@ available in this table: :top_meta: A JSON object that contains metadata related to the top line in the view. :selection: The number of the line that is focused for selection. +:options: A JSON object that contains optional settings for this view. lnav_views_echo --------------- diff --git a/docs/source/ui.rst b/docs/source/ui.rst index a718d06..b2de5b7 100644 --- a/docs/source/ui.rst +++ b/docs/source/ui.rst @@ -8,7 +8,7 @@ with status bars above and below, and the interactive prompt as the last line. .. figure:: lnav-ui.png :align: center - :alt: Screenshot of lnav showing a mix of syslog and web access_log messages. + :figwidth: 90% Screenshot of **lnav** viewing syslog and web access_log messages. @@ -26,11 +26,11 @@ times in the views. **lnav** provides many operations to work with the log/text data in the main view. For example, you can add comments and tags to log messages. -By default, the top line is used as the reference point to edit the +The highlighted cursor line is used as the reference point to edit the comment or tags. Alternatively, you can press :kbd:`Ctrl` + :kbd:`x` -to switch to "cursor" mode where the "focused" line is highlighted and -most operations now work with that line. When in "cursor" mode, the -:kbd:`↑` and :kbd:`↓` keys now move the focused line instead of scrolling +to switch to "top" mode where the "focused" line is the top line in the +view and most operations now work with that line. When in "cursor" mode, +the :kbd:`↑` and :kbd:`↓` keys now move the focused line instead of scrolling the view. Jumping to bookmarks, like errors, will also move the focused line instead of moving the next error to the top of the view. @@ -173,7 +173,9 @@ LOG ^^^ The log view displays the log messages from any loaded log files in time -order. This view will be shown by default if any log messages are available. +order. This view will be shown by default if any log files were detected. +If plain text files were also loaded, they will be available in the TEXT +view, which you can switch to by pressing :kbd:`t`. On color displays, the log messages will be highlighted as follows: @@ -194,11 +196,18 @@ On color displays, the log messages will be highlighted as follows: :config /ui/theme grayscale +Timestamps in log messages will be rewritten to the local timezone (or the +timezone specified by :envvar:`TZ`) automatically if they include a +timezone component. If a file's timestamps do not include a timezone, they +will be treated as if they are from the local zone. You can change the zone +to use for these types of files using the +:ref:`:set-file-timezone<set_file_timezone>` command. + .. note:: If a log message has a timestamp that is out-of-order with its neighboring messages, the timestamp will be highlighted in yellow. When one of these - messages is at the top of the log view, an overlay will display the + messages is focused, an overlay will display the difference between the "actual time" and the "received time". The "actual time" is the original textual timestamp. The "received time" is the time of an earlier message that is larger than this log message's time. @@ -221,6 +230,10 @@ These crumbs are interactive and can be used to navigate to different parts of the log view. For example, selecting a different value in the log format crumb will jump to the first message with that format. +The file crumb will show a "↻" icon if the file is from the output of a FIFO, +:code:`:sh` command, or data that was piped into the standard input. When +the pipe is closed, the icon will disappear. + TEXT ^^^^ @@ -230,16 +243,39 @@ Press :kbd:`t` to switch to the text view. While in the text view, you can press :kbd:`f` or :kbd:`Shift` + :kbd:`F` to switch to the next / previous text file. +The breadcrumb bar will show the name of the file and any structure that was +discovered in the content. The file crumb will show a "↻" icon if the file +is from the output of a FIFO, :code:`:sh` command, or data that was piped +into the standard input. When the pipe is closed, the icon will disappear. + +If the content is piped into lnav through standard input, a FIFO, or a +:code:`:sh` command, the time that lines are received are recorded. You +can press :kbd:`Shift` + :kbd:`T` to view the elapsed time like in the +LOG view. The breadcrumb bar will also show the received time of the +focused line after the file name crumb. If the output being shown is from +a :code:`:sh` command, you can press :kbd:`Ctrl` + :kbd:`C` to send a +SIGINT to the child process without killing **lnav** itself. + +.. figure:: lnav-make-check-log.png + :align: center + :figwidth: 90% + + Screenshot of the TEXT view showing the output of :code:`sh make check`. + Each line is timestamped internally when it was received so it's + possible to view how long each test is taking to run. The "↻" icon + next to the file name in the breadcrumb bar means that the make is + still running. + Markdown """""""" Files with an :code:`.md` (or :code:`.markdown`) extension will be treated as Markdown files and rendered separately. - .. figure:: lnav-markdown-example.png - :align: center +.. figure:: lnav-markdown-example.png + :align: center - Viewing the **lnav** :file:`README.md` file. + Viewing the **lnav** :file:`README.md` file. DB @@ -269,6 +305,61 @@ can also press :kbd:`Shift` + :kbd:`i` to toggle the histogram view while synchronizing the top time. While in the histogram view, pressing :kbd:`z` / :kbd:`Shift` + :kbd:`z` will zoom in/out. +GANTT +^^^^^ + +.. note:: This feature is available in v0.12.0+. + +.. figure:: lnav-gantt-1.png + :align: center + + Screenshot of the Gantt chart view when viewing logs from the + VMWare Update Manager. Most rows show API requests as they + are received and processed. + +The Gantt Chart view visualizes operations over time. The operations +are identified by the "opid" field defined in the log format. In the +view, there is a header that shows the overall time span, the +narrowed time span around the focused line, and the column headers. +Each row in the view shows the following: + +* The duration of the operation +* Sparklines showing the number of errors and warnings relative to the + total number of messages associated with the OPID. +* The OPID itself. +* A description of the operation as captured from the log messages. + +The rows are sorted by the start time of each operation. + +If an operation row is in the focused time span, a reverse-video +bar will show when the operation started and finished (unless it +extends outside the time span). As you move the focused line, the +focused time span will be adjusted to keep the preceding and following +five operations within the span. + +The preview panel at the bottom of the display will show the +messages associated with the focused operation. + +The following hotkeys can be useful in this view: + +* :kbd:`p` -- If the log format defined sub-operations with the + :code:`opid/subid` property, this will toggle an overlay panel + that displays the sub-operation descriptions. + + .. figure:: lnav-gantt-2.png + :align: center + + Screenshot showing the same log as above after pressing + :kbd:`p`. The overlay panel shows a breakdown of + sub-operations performed while processing the main operation. + +* :kbd:`Shift` + :kbd:`q` -- Return to the previous view and change + its focused line to match the time that was focused in the gantt + view. +* :kbd:`Shift` + :kbd:`a` -- After leaving the gantt view, pressing + these keys will return to the Gantt view while keeping the focused + time in sync. + PRETTY ^^^^^^ @@ -325,3 +416,71 @@ range of values. The panel at the bottom of the view shows the data points themselves from the original source, the log file or the SQL query results. You can press :kbd:`TAB` to focus on the details panel so you can scroll around and get a closer look at the values. + +.. _ui_mouse: + +Mouse Support (v0.12.2+) +------------------------ + +With mouse support enabled, either through the `/ui/mouse/mode` +configuration option or by pressing :kbd:`F2`, many of the UI +elements will respond to mouse inputs: + +* clicking on the main view will move the cursor to the given + row and dragging will scroll the view as needed; +* double-clicking in the main view will select the underlying + text and drag-selecting within a line will select the given + text; +* when double-clicking text: if the mouse pointer is inside + a quoted string, the contents of the string will be selected; + if the mouse pointer is on the quote, the quote will be included + in the selection; if the mouse pointer is over a bracket + (e.g. [],{},()) where the matching bracket is on the same line, + the selection will span from one bracket to the other; +* when text is selected, a menu will pop up that can be used + to filter based on the current text, search for it, or copy + it to the clipboard; +* right-clicking the start of a log message in the main view + will open the parser details overlay; +* the parser details now displays a diamond next to fields to + indicate whether they are shown/hidden and this can be + clicked to toggle the state; +* the parser details will show a bar chart icon for fields with + values which, when clicked, will open either the spectrogram + view for the given field or open the DB query prompt with a + PRQL query to generate a histogram of the field values; +* clicking in the scroll area will move the view by a page, + double-clicking will move the view to that area, and + dragging the scrollbar will move the view to the given spot; +* clicking on the breadcrumb bar will select a crumb and + selecting a possibility from the popup will move to that + location in the view; +* clicking on portions of the bottom status bar will trigger + a relevant action (e.g. clicking the line number will open + the command prompt with :code:`:goto <current-line>`); +* clicking on the configuration panel tabs (i.e. Files/Filters) + will open the selected panel and clicking parts of the + display in there will perform the relevant action (e.g. + clicking the diamond will enable/disable the file/filter); +* clicking in a prompt will move the cursor to the location; +* clicking on a column in the spectrogram view will select it. + +.. note:: + + A downside of enabling mouse support is that normal text + selection and copy will no longer work. While lnav has + some support for selection in the main view, there are + still likely to be cases where that is insufficient. + In those cases, you can press :kbd:`F2` to quickly + switch back-and-forth. Or, some terminals have support + for switching while a modifier is pressed: + + .. list-table:: + :header-rows: 1 + + * - Key + - Terminal + * - :kbd:`Option` + - iTerm, Hyper + * - :kbd:`Fn` + - Terminal.app diff --git a/docs/source/usage.rst b/docs/source/usage.rst index d43ed52..df50c7d 100644 --- a/docs/source/usage.rst +++ b/docs/source/usage.rst @@ -116,6 +116,68 @@ file. The binary file is named ``tailer.bin.XXXXXX`` where *XXXXXX* is 6 random digits. The file is, under normal circumstancies, deleted immediately. +Command Output +^^^^^^^^^^^^^^ + +The output of commands can be captured and displayed in **lnav** using +the :ref:`:sh<sh>` command or by passing the :option:`-e` option on the +command-line. The captured output will be displayed in the TEXT view. +The lines from stdout and stderr are recorded separately so that the +lines from stderr can be shown in the theme's "error" highlight. The +time that the lines were received are also recorded internally so that +the "time-offset" display (enabled by pressing :kbd:`Shift` + :kbd:`T`) +can be shown and the "jump to slow-down" hotkeys (:kbd:`s` / +:kbd:`Shift` + :kbd:`S`) work. Since the line-by-line timestamps are +recorded internally, they will not interfere with timestamps that are +in the commands output. + +Docker Logs +^^^^^^^^^^^ + +To make it easier to view +`docker logs <https://docs.docker.com/engine/reference/commandline/logs/>`_ +within **lnav**, a :code:`docker://` URL scheme is available. Passing +the container name in the authority field will run the :code:`docker logs` +command. If a path is added to the URL, then **lnav** will execute +:code:`docker exec <container> tail -F -n +0 /path/to/file` to try and +tail the file in the container. + +Custom URL Schemes +^^^^^^^^^^^^^^^^^^ + +Custom URL schemes can be defined using the :ref:`/tuning/url-schemes<url_scheme>` +configuration. By adding a scheme name to the tuning configuration along +with the name of an **lnav** handler script, you can control how the URL is +interpreted and turned into **lnav** commands. This feature is how the +`Docker Logs`_ functionality is implemented. + +Custom URLs can be passed on the command-line or to the :ref:`:open<open>` +command. When passed on the command-line, an :code:`:open` command with the +URL is added to the list of initial commands. When the :code:`:open` command +detects a custom URL, it checks for the definition in the configuration. +If found, it will call the associated handler script with the URL as the +first parameter. The script can parse the URL using the :ref:`parse_url` +SQL function, if needed. The script should then execute whatever commands +it needs to open the destination for viewing in **lnav**. For example, +the docker URL handler uses the :ref:`:sh<sh>` command to run +:code:`docker logs` with the container. + +Using as a PAGER +^^^^^^^^^^^^^^^^ + +Setting **lnav** as your :envvar:`PAGER` can have some advantages, like +basic syntax highlighting and discovering sections in a document. For +example, when viewing a man page, the current section is displayed in +the breadcrumb bar and you can jump to a section with the +:ref:`:goto<goto>` command. + +You will probably want to pass the :option:`-q` option to suppress the +message showing the path to the captured input. + +.. prompt:: bash + + export PAGER="lnav -q" + Searching --------- @@ -203,10 +265,41 @@ columns will be included in the table. Taking Notes ------------ -A few of the columns in the log tables can be updated on a row-by-row basis to -allow you to take notes. The majority of the columns in a log table are -read-only since they are backed by the log files themselves. However, the -following columns can be changed by an :code:`UPDATE` statement: +As you are looking through logs, you might find that you want to leave some +notes of your findings. **lnav** can help here by saving information in +the session without needing to modify the actual log files. Thus, when +you re-open the files in lnav, the notes will be restored. The following +types of information can be saved: + +:tags: Log messages can be tagged with the :ref:`:tag<tag>` command as a + simple way to leave a descriptive mark. The tags attached to a + message will be shown underneath the message. You can press + :kbd:`u` and :kbd:`Shift` + :kbd:`u` to jump to the next/previous + marked line. A regular search will also match tags. + +:comments: Free-form text can be attached to a log message with the + :ref:`:comment<comment>` command. The comment will be shown + underneath the message. If the text contains markdown syntax, + it will be rendered to the best of the terminal's ability. + You can press :kbd:`u` and :kbd:`Shift` + :kbd:`u` to jump to the + next/previous marked line. A regular search will also match the + comment text. + +:partitions: The log view can be partitioned to provide some context + about where you are in a collection of logs. For example, in logs + for a test run, partitions could be created with the name for each + test. The current partition is shown in the breadcrumb bar and + prefixed by the "⊑" symbol. You can select the partition breadcrumb + to jump to another partition. Pressing :kbd:`{` and :kbd:`}` will + jump to the next/previous partition. + +Accessing notes through the SQLite interface +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The note taking functionality in lnav can also be accessed through the +log tables exposed through SQLite. The majority of the columns in a log +table are read-only since they are backed by the log files themselves. +However, the following columns can be changed by an :code:`UPDATE` statement: * **log_part** - The "partition" the log message belongs to. This column can also be changed by the :ref:`:partition-name<partition_name>` command. |