diff options
Diffstat (limited to '')
122 files changed, 7769 insertions, 0 deletions
diff --git a/docs/.gitignore b/docs/.gitignore new file mode 100644 index 0000000..f40fbd8 --- /dev/null +++ b/docs/.gitignore @@ -0,0 +1,5 @@ +_site +.sass-cache +.jekyll-cache +.jekyll-metadata +vendor diff --git a/docs/02_downloads.md b/docs/02_downloads.md new file mode 100644 index 0000000..9c44192 --- /dev/null +++ b/docs/02_downloads.md @@ -0,0 +1,65 @@ +--- +layout: page +title: Downloads +permalink: /downloads +--- + +The latest **stable** release is [**v{{ site.version }}**](https://github.com/tstack/lnav/releases/latest). + +The following options are available for installing **lnav**: + +## Linux + +<!-- markdown-link-check-disable-next-line --> +Download a [statically linked 64-bit binary](https://github.com/tstack/lnav/releases/download/v{{site.version}}/lnav-{{site.version}}-x86_64-linux-musl.zip). + +Install from the [Snap Store](https://snapcraft.io/lnav): + +```console +$ sudo snap install lnav +``` + +Install RPMs from [Package Cloud](https://packagecloud.io/tstack/lnav): + +```console +$ curl -s https://packagecloud.io/install/repositories/tstack/lnav/script.rpm.sh | sudo bash +$ sudo yum install lnav +``` + +## MacOS + +<!-- markdown-link-check-disable-next-line --> +Download a [statically linked 64-bit binary](https://github.com/tstack/lnav/releases/download/v{{site.version}}/lnav-{{site.version}}-x86_64-macos.zip) + +Install using [Homebrew](https://formulae.brew.sh/formula/lnav): + +```console +$ brew install lnav +``` + +## Source + +<!-- markdown-link-check-disable-next-line --> +Download the [source](https://github.com/tstack/lnav/releases/download/v{{site.version}}/lnav-{{site.version}}.tar.gz) +and install any dependencies. The following commands will unpack the source +tar ball, configure the build for your system, build, and then install: + +```console +$ tar xvfz lnav-{{site.version}}.tar.gz +$ cd lnav-{{site.version}} +$ ./configure +$ make +$ make install +``` + +### GitHub + +If you would like to contribute to the development of lnav, visit our page on +[GitHub](https://github.com/tstack/lnav). + +# VSCode Extension + +The [lnav VSCode Extension](https://marketplace.visualstudio.com/items?itemName=lnav.lnav) +can be used to add syntax highlighting to lnav scripts. + + diff --git a/docs/03_features.md b/docs/03_features.md new file mode 100644 index 0000000..513cd2c --- /dev/null +++ b/docs/03_features.md @@ -0,0 +1,146 @@ +--- +layout: page +title: Features +permalink: /features +--- + +* TOC +{:toc} + +## Single Log View + +All log file contents are merged into a single view based on message timestamps. +You no longer need to manually correlate timestamps across multiple windows or +figure out the order in which to view rotated log files. The color bars on the +left-hand side help to show which file a message belongs to. + + + +## Automatic Log Format Detection + +The log message format is automatically determined by lnav while scanning your +files. The following are some of the [formats that are built in by default](https://docs.lnav.org/en/latest/formats.html): + +* Common Web Access Log format +* W3C Extended Log File Format +* logfmt +* CUPS page_log +* Syslog +* Glog +* VMware ESXi/vCenter Logs +* dpkg.log +* uwsgi +* "Generic" - Any message that starts with a timestamp +* Strace +* sudo + +If your log file format is JSON-lines or can be matched by a PCRE regular +expression, you can define your own format in a +[JSON file](https://docs.lnav.org/en/latest/formats.html#defining-a-new-format). + +GZIP'ed and BZIP2'ed files are also detected automatically and decompressed on-the-fly. + +## Filters + +Display only lines that match or do not match a set of regular expressions. +Useful for removing extraneous log lines that you are not interested in. + +## Timeline View + +The timeline view shows a histogram of messages over time. The number of +warnings and errors are highlighted in the display so that you can easily see +where problems have occurred. Once you have found a period of time that is of +interest, a key-press will take you back to the log message view at the +corresponding time. + + + +## Pretty-Print View + +The pretty-print view will reformat structured data, like XML or JSON, so that +it is easier to read. Simply press SHIFT+P in the log view to have all the +currently displayed lines pretty-printed. + +The following screenshot shows an XML blob with no indentation: + + + +After pressing SHIFT+P, the XML is pretty-printed for easier viewing: + + + +## Query Logs Using SQL + +Log files are directly used as the backing for SQLite virtual tables. This +means you can perform queries on messages without having to load the data into +an SQL database. For example, the screenshot below shows the result of +running the following query against an Apache access_log file: + +```sql +SELECT c_ip, count(*), sum(sc_bytes) AS total FROM access_log + GROUP BY c_ip ORDER BY total DESC; +``` + + + +## "Live" Operation + +Searches are done as you type; new log lines are automatically loaded and +searched as they are added; filters apply to lines as they are loaded; and, SQL +queries are checked for correctness as you type. + +## Themes + +The UI can be [customized through themes](https://lnav.readthedocs.io/en/latest/config.html#theme-definitions). + + + +## Syntax Highlighting + +Errors and warnings are colored in red and yellow, respectively. Highlights are +also applied to: SQL keywords, XML tags, file and line numbers in Java +backtraces, and quoted strings. The search and SQL query prompt are also +highlighted as you type, making it easier to see errors and matching brackets. + + + +## Tab-completion + +The command prompt supports tab-completion for almost all operations. For +example, when doing a search, you can tab-complete words that are displayed on +screen rather than having to do a copy & paste. + + + +## Custom Keymaps + +[Hotkeys can be customized](https://lnav.readthedocs.io/en/latest/config.html#keymap-definitions) +to run lnav commands or scripts. + +## Sessions + +Session information is saved automatically and restored when you are viewing the +same set of files. The current location in files, bookmarks, and applied filters +are all saved as part of the session. + +## Headless Mode + +The log processing features of lnav can be used in scripts if you have a canned +set of operations or queries that you want to perform regularly. You can enable +headless mode with the '-n' switch on the command-line and then use the '-c' +flag to specify the commands or queries you want to execute. For example, to get +the top 10 client IP addresses from an apache access log file and write the +results to standard out in CSV format: + +```console +$ lnav -n \ + -c ';SELECT c_ip, count(*) AS total FROM access_log GROUP BY c_ip ORDER BY total DESC LIMIT 10' \ + -c ':write-csv-to -' \ + access.log + +c_ip,total +10.208.110.176,2989570 +10.178.4.102,11183 +10.32.110.197,2020 +10.29.165.250,443 +``` diff --git a/docs/04_tutorials.md b/docs/04_tutorials.md new file mode 100644 index 0000000..1913e5a --- /dev/null +++ b/docs/04_tutorials.md @@ -0,0 +1,30 @@ +--- +layout: page +title: Tutorials +permalink: /tutorials +--- + +These tutorials are provided to help you learn how **lnav** works +without having to install anything. They are running on a shared +[fly.io](https://fly.io) instance, so please be kind. + +The tutorials are implemented using features in **lnav** and not +built in to the code itself. The tutorial text is +[markdown](https://docs.lnav.org/en/latest/ui.html#markdown), +the logic is written in [SQL](https://docs.lnav.org/en/latest/sqlext.html), +and the reactions are triggered through +[events](https://docs.lnav.org/en/latest/events.html). + +The source for the tutorials can be found [here](https://github.com/tstack/lnav/tree/master/docs/tutorials). + +# Tutorial 1 + +<div id="playground-box"> +<h4>Learn how to navigate an example log file using lnav:</h4> + +<code> +<span class="prompt">$</span> +<a href="ssh://tutorial1@demo.lnav.org">ssh tutorial1@demo.lnav.org</a> +</code> +</div> + diff --git a/docs/05_docs.md b/docs/05_docs.md new file mode 100644 index 0000000..3c19354 --- /dev/null +++ b/docs/05_docs.md @@ -0,0 +1,6 @@ +--- +title: Docs +permalink: /docs +redirect_to: + - https://docs.lnav.org +--- diff --git a/docs/06_changeblog.md b/docs/06_changeblog.md new file mode 100644 index 0000000..1f2bdc9 --- /dev/null +++ b/docs/06_changeblog.md @@ -0,0 +1,6 @@ +--- +title: ChangeBlog +layout: home +permalink: /blog/ +--- + diff --git a/docs/404.html b/docs/404.html new file mode 100644 index 0000000..086a5c9 --- /dev/null +++ b/docs/404.html @@ -0,0 +1,25 @@ +--- +permalink: /404.html +layout: default +--- + +<style type="text/css" media="screen"> + .container { + margin: 10px auto; + max-width: 600px; + text-align: center; + } + h1 { + margin: 30px 0; + font-size: 4em; + line-height: 1; + letter-spacing: -1px; + } +</style> + +<div class="container"> + <h1>404</h1> + + <p><strong>Page not found :(</strong></p> + <p>The requested page could not be found.</p> +</div> diff --git a/docs/CNAME b/docs/CNAME new file mode 100644 index 0000000..330a24a --- /dev/null +++ b/docs/CNAME @@ -0,0 +1 @@ +lnav.org
\ No newline at end of file diff --git a/docs/Doxyfile.in b/docs/Doxyfile.in new file mode 100644 index 0000000..a6da763 --- /dev/null +++ b/docs/Doxyfile.in @@ -0,0 +1,32 @@ +# Configuration for Doxygen for use with CMake +# Only options that deviate from the default are included +# To create a new Doxyfile containing all available options, call `doxygen -g` + +# Get Project name and version from CMake +PROJECT_NAME = "@PROJECT_NAME@" +PROJECT_NUMBER = "@PROJECT_VERSION@" + +# Add sources +INPUT = "@PROJECT_SOURCE_DIR@/README.md" "@PROJECT_SOURCE_DIR@/include" "@PROJECT_SOURCE_DIR@/source" "@PROJECT_SOURCE_DIR@/docs/pages" +EXTRACT_ALL = YES +RECURSIVE = YES +OUTPUT_DIRECTORY = "@DOXYGEN_OUTPUT_DIRECTORY@" + +# Use the README as a main page +USE_MDFILE_AS_MAINPAGE = "@PROJECT_SOURCE_DIR@/README.md" + +# set relative include paths +FULL_PATH_NAMES = YES +STRIP_FROM_PATH = "@PROJECT_SOURCE_DIR@/include" "@PROJECT_SOURCE_DIR@" +STRIP_FROM_INC_PATH = + +# We use m.css to generate the html documentation, so we only need XML output +GENERATE_XML = YES +GENERATE_HTML = NO +GENERATE_LATEX = NO +XML_PROGRAMLISTING = NO +CREATE_SUBDIRS = NO + +# Include all directories, files and namespaces in the documentation +# Disable to include only explicitly documented objects +M_SHOW_UNDOCUMENTED = YES diff --git a/docs/Gemfile b/docs/Gemfile new file mode 100644 index 0000000..10560d9 --- /dev/null +++ b/docs/Gemfile @@ -0,0 +1,31 @@ +source "https://rubygems.org" +# Hello! This is where you manage which Jekyll version is used to run. +# When you want to use a different version, change it below, save the +# file and run `bundle install`. Run Jekyll with `bundle exec`, like so: +# +# bundle exec jekyll serve +# +# This will help ensure the proper Jekyll version is running. +# Happy Jekylling! +# gem "jekyll", "~> 4.2.0" +# This is the default theme for new Jekyll sites. You may change this to anything you like. +gem "minima", "~> 2.5" +# If you want to use GitHub Pages, remove the "gem "jekyll"" above and +# uncomment the line below. To upgrade, run `bundle update github-pages`. +gem "github-pages", group: :jekyll_plugins +# If you have any plugins, put them here! +group :jekyll_plugins do +end + +# Windows and JRuby does not include zoneinfo files, so bundle the tzinfo-data gem +# and associated library. +platforms :mingw, :x64_mingw, :mswin, :jruby do + gem "tzinfo", "~> 1.2" + gem "tzinfo-data" +end + +# Performance-booster for watching directories on Windows +gem "wdm", "~> 0.1.1", :platforms => [:mingw, :x64_mingw, :mswin] + + +gem "webrick", "~> 1.7" diff --git a/docs/Gemfile.lock b/docs/Gemfile.lock new file mode 100644 index 0000000..275c580 --- /dev/null +++ b/docs/Gemfile.lock @@ -0,0 +1,268 @@ +GEM + remote: https://rubygems.org/ + specs: + activesupport (7.0.5) + concurrent-ruby (~> 1.0, >= 1.0.2) + i18n (>= 1.6, < 2) + minitest (>= 5.1) + tzinfo (~> 2.0) + addressable (2.8.4) + public_suffix (>= 2.0.2, < 6.0) + coffee-script (2.4.1) + coffee-script-source + execjs + coffee-script-source (1.11.1) + colorator (1.1.0) + commonmarker (0.23.9) + concurrent-ruby (1.2.2) + dnsruby (1.70.0) + simpleidn (~> 0.2.1) + em-websocket (0.5.3) + eventmachine (>= 0.12.9) + http_parser.rb (~> 0) + ethon (0.16.0) + ffi (>= 1.15.0) + eventmachine (1.2.7) + execjs (2.8.1) + faraday (2.7.7) + faraday-net_http (>= 2.0, < 3.1) + ruby2_keywords (>= 0.0.4) + faraday-net_http (3.0.2) + ffi (1.15.5) + forwardable-extended (2.6.0) + gemoji (3.0.1) + github-pages (228) + github-pages-health-check (= 1.17.9) + jekyll (= 3.9.3) + jekyll-avatar (= 0.7.0) + jekyll-coffeescript (= 1.1.1) + jekyll-commonmark-ghpages (= 0.4.0) + jekyll-default-layout (= 0.1.4) + jekyll-feed (= 0.15.1) + jekyll-gist (= 1.5.0) + jekyll-github-metadata (= 2.13.0) + jekyll-include-cache (= 0.2.1) + jekyll-mentions (= 1.6.0) + jekyll-optional-front-matter (= 0.3.2) + jekyll-paginate (= 1.1.0) + jekyll-readme-index (= 0.3.0) + jekyll-redirect-from (= 0.16.0) + jekyll-relative-links (= 0.6.1) + jekyll-remote-theme (= 0.4.3) + jekyll-sass-converter (= 1.5.2) + jekyll-seo-tag (= 2.8.0) + jekyll-sitemap (= 1.4.0) + jekyll-swiss (= 1.0.0) + jekyll-theme-architect (= 0.2.0) + jekyll-theme-cayman (= 0.2.0) + jekyll-theme-dinky (= 0.2.0) + jekyll-theme-hacker (= 0.2.0) + jekyll-theme-leap-day (= 0.2.0) + jekyll-theme-merlot (= 0.2.0) + jekyll-theme-midnight (= 0.2.0) + jekyll-theme-minimal (= 0.2.0) + jekyll-theme-modernist (= 0.2.0) + jekyll-theme-primer (= 0.6.0) + jekyll-theme-slate (= 0.2.0) + jekyll-theme-tactile (= 0.2.0) + jekyll-theme-time-machine (= 0.2.0) + jekyll-titles-from-headings (= 0.5.3) + jemoji (= 0.12.0) + kramdown (= 2.3.2) + kramdown-parser-gfm (= 1.1.0) + liquid (= 4.0.4) + mercenary (~> 0.3) + minima (= 2.5.1) + nokogiri (>= 1.13.6, < 2.0) + rouge (= 3.26.0) + terminal-table (~> 1.4) + github-pages-health-check (1.17.9) + addressable (~> 2.3) + dnsruby (~> 1.60) + octokit (~> 4.0) + public_suffix (>= 3.0, < 5.0) + typhoeus (~> 1.3) + html-pipeline (2.14.3) + activesupport (>= 2) + nokogiri (>= 1.4) + http_parser.rb (0.8.0) + i18n (1.14.1) + concurrent-ruby (~> 1.0) + jekyll (3.9.3) + addressable (~> 2.4) + colorator (~> 1.0) + em-websocket (~> 0.5) + i18n (>= 0.7, < 2) + jekyll-sass-converter (~> 1.0) + jekyll-watch (~> 2.0) + kramdown (>= 1.17, < 3) + liquid (~> 4.0) + mercenary (~> 0.3.3) + pathutil (~> 0.9) + rouge (>= 1.7, < 4) + safe_yaml (~> 1.0) + jekyll-avatar (0.7.0) + jekyll (>= 3.0, < 5.0) + jekyll-coffeescript (1.1.1) + coffee-script (~> 2.2) + coffee-script-source (~> 1.11.1) + jekyll-commonmark (1.4.0) + commonmarker (~> 0.22) + jekyll-commonmark-ghpages (0.4.0) + commonmarker (~> 0.23.7) + jekyll (~> 3.9.0) + jekyll-commonmark (~> 1.4.0) + rouge (>= 2.0, < 5.0) + jekyll-default-layout (0.1.4) + jekyll (~> 3.0) + jekyll-feed (0.15.1) + jekyll (>= 3.7, < 5.0) + jekyll-gist (1.5.0) + octokit (~> 4.2) + jekyll-github-metadata (2.13.0) + jekyll (>= 3.4, < 5.0) + octokit (~> 4.0, != 4.4.0) + jekyll-include-cache (0.2.1) + jekyll (>= 3.7, < 5.0) + jekyll-mentions (1.6.0) + html-pipeline (~> 2.3) + jekyll (>= 3.7, < 5.0) + jekyll-optional-front-matter (0.3.2) + jekyll (>= 3.0, < 5.0) + jekyll-paginate (1.1.0) + jekyll-readme-index (0.3.0) + jekyll (>= 3.0, < 5.0) + jekyll-redirect-from (0.16.0) + jekyll (>= 3.3, < 5.0) + jekyll-relative-links (0.6.1) + jekyll (>= 3.3, < 5.0) + jekyll-remote-theme (0.4.3) + addressable (~> 2.0) + jekyll (>= 3.5, < 5.0) + jekyll-sass-converter (>= 1.0, <= 3.0.0, != 2.0.0) + rubyzip (>= 1.3.0, < 3.0) + jekyll-sass-converter (1.5.2) + sass (~> 3.4) + jekyll-seo-tag (2.8.0) + jekyll (>= 3.8, < 5.0) + jekyll-sitemap (1.4.0) + jekyll (>= 3.7, < 5.0) + jekyll-swiss (1.0.0) + jekyll-theme-architect (0.2.0) + jekyll (> 3.5, < 5.0) + jekyll-seo-tag (~> 2.0) + jekyll-theme-cayman (0.2.0) + jekyll (> 3.5, < 5.0) + jekyll-seo-tag (~> 2.0) + jekyll-theme-dinky (0.2.0) + jekyll (> 3.5, < 5.0) + jekyll-seo-tag (~> 2.0) + jekyll-theme-hacker (0.2.0) + jekyll (> 3.5, < 5.0) + jekyll-seo-tag (~> 2.0) + jekyll-theme-leap-day (0.2.0) + jekyll (> 3.5, < 5.0) + jekyll-seo-tag (~> 2.0) + jekyll-theme-merlot (0.2.0) + jekyll (> 3.5, < 5.0) + jekyll-seo-tag (~> 2.0) + jekyll-theme-midnight (0.2.0) + jekyll (> 3.5, < 5.0) + jekyll-seo-tag (~> 2.0) + jekyll-theme-minimal (0.2.0) + jekyll (> 3.5, < 5.0) + jekyll-seo-tag (~> 2.0) + jekyll-theme-modernist (0.2.0) + jekyll (> 3.5, < 5.0) + jekyll-seo-tag (~> 2.0) + jekyll-theme-primer (0.6.0) + jekyll (> 3.5, < 5.0) + jekyll-github-metadata (~> 2.9) + jekyll-seo-tag (~> 2.0) + jekyll-theme-slate (0.2.0) + jekyll (> 3.5, < 5.0) + jekyll-seo-tag (~> 2.0) + jekyll-theme-tactile (0.2.0) + jekyll (> 3.5, < 5.0) + jekyll-seo-tag (~> 2.0) + jekyll-theme-time-machine (0.2.0) + jekyll (> 3.5, < 5.0) + jekyll-seo-tag (~> 2.0) + jekyll-titles-from-headings (0.5.3) + jekyll (>= 3.3, < 5.0) + jekyll-watch (2.2.1) + listen (~> 3.0) + jemoji (0.12.0) + gemoji (~> 3.0) + html-pipeline (~> 2.2) + jekyll (>= 3.0, < 5.0) + kramdown (2.3.2) + rexml + kramdown-parser-gfm (1.1.0) + kramdown (~> 2.0) + liquid (4.0.4) + listen (3.8.0) + rb-fsevent (~> 0.10, >= 0.10.3) + rb-inotify (~> 0.9, >= 0.9.10) + mercenary (0.3.6) + minima (2.5.1) + jekyll (>= 3.5, < 5.0) + jekyll-feed (~> 0.9) + jekyll-seo-tag (~> 2.1) + minitest (5.18.1) + nokogiri (1.15.2-arm64-darwin) + racc (~> 1.4) + nokogiri (1.15.2-x86_64-linux) + racc (~> 1.4) + octokit (4.25.1) + faraday (>= 1, < 3) + sawyer (~> 0.9) + pathutil (0.16.2) + forwardable-extended (~> 2.6) + public_suffix (4.0.7) + racc (1.7.1) + rb-fsevent (0.11.2) + rb-inotify (0.10.1) + ffi (~> 1.0) + rexml (3.2.5) + rouge (3.26.0) + ruby2_keywords (0.0.5) + rubyzip (2.3.2) + safe_yaml (1.0.5) + sass (3.7.4) + sass-listen (~> 4.0.0) + sass-listen (4.0.0) + rb-fsevent (~> 0.9, >= 0.9.4) + rb-inotify (~> 0.9, >= 0.9.7) + sawyer (0.9.2) + addressable (>= 2.3.5) + faraday (>= 0.17.3, < 3) + simpleidn (0.2.1) + unf (~> 0.1.4) + terminal-table (1.8.0) + unicode-display_width (~> 1.1, >= 1.1.1) + typhoeus (1.4.0) + ethon (>= 0.9.0) + tzinfo (2.0.6) + concurrent-ruby (~> 1.0) + unf (0.1.4) + unf_ext + unf_ext (0.0.8.2) + unicode-display_width (1.8.0) + webrick (1.8.1) + +PLATFORMS + arm64-darwin-21 + arm64-darwin-22 + x86_64-linux + +DEPENDENCIES + github-pages + minima (~> 2.5) + tzinfo (~> 1.2) + tzinfo-data + wdm (~> 0.1.1) + webrick (~> 1.7) + +BUNDLED WITH + 2.3.20 diff --git a/docs/Makefile b/docs/Makefile new file mode 100644 index 0000000..b16b62c --- /dev/null +++ b/docs/Makefile @@ -0,0 +1,153 @@ +# Makefile for Sphinx documentation +# + +# You can set these variables from the command line. +SPHINXOPTS = +SPHINXBUILD = sphinx-build +PAPER = +BUILDDIR = build + +# Internal variables. +PAPEROPT_a4 = -D latex_paper_size=a4 +PAPEROPT_letter = -D latex_paper_size=letter +ALLSPHINXOPTS = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) source +# the i18n builder cannot share the environment and doctrees with the others +I18NSPHINXOPTS = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) source + +.PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest gettext + +help: + @echo "Please use \`make <target>' where <target> is one of" + @echo " html to make standalone HTML files" + @echo " dirhtml to make HTML files named index.html in directories" + @echo " singlehtml to make a single large HTML file" + @echo " pickle to make pickle files" + @echo " json to make JSON files" + @echo " htmlhelp to make HTML files and a HTML help project" + @echo " qthelp to make HTML files and a qthelp project" + @echo " devhelp to make HTML files and a Devhelp project" + @echo " epub to make an epub" + @echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter" + @echo " latexpdf to make LaTeX files and run them through pdflatex" + @echo " text to make text files" + @echo " man to make manual pages" + @echo " texinfo to make Texinfo files" + @echo " info to make Texinfo files and run them through makeinfo" + @echo " gettext to make PO message catalogs" + @echo " changes to make an overview of all changed/added/deprecated items" + @echo " linkcheck to check all external links for integrity" + @echo " doctest to run all doctests embedded in the documentation (if enabled)" + +clean: + -rm -rf $(BUILDDIR)/* + +html: + $(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html + @echo + @echo "Build finished. The HTML pages are in $(BUILDDIR)/html." + +dirhtml: + $(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml + @echo + @echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml." + +singlehtml: + $(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml + @echo + @echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml." + +pickle: + $(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle + @echo + @echo "Build finished; now you can process the pickle files." + +json: + $(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json + @echo + @echo "Build finished; now you can process the JSON files." + +htmlhelp: + $(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp + @echo + @echo "Build finished; now you can run HTML Help Workshop with the" \ + ".hhp project file in $(BUILDDIR)/htmlhelp." + +qthelp: + $(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp + @echo + @echo "Build finished; now you can run "qcollectiongenerator" with the" \ + ".qhcp project file in $(BUILDDIR)/qthelp, like this:" + @echo "# qcollectiongenerator $(BUILDDIR)/qthelp/lnav.qhcp" + @echo "To view the help file:" + @echo "# assistant -collectionFile $(BUILDDIR)/qthelp/lnav.qhc" + +devhelp: + $(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp + @echo + @echo "Build finished." + @echo "To view the help file:" + @echo "# mkdir -p $$HOME/.local/share/devhelp/lnav" + @echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/lnav" + @echo "# devhelp" + +epub: + $(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub + @echo + @echo "Build finished. The epub file is in $(BUILDDIR)/epub." + +latex: + $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex + @echo + @echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex." + @echo "Run \`make' in that directory to run these through (pdf)latex" \ + "(use \`make latexpdf' here to do that automatically)." + +latexpdf: + $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex + @echo "Running LaTeX files through pdflatex..." + $(MAKE) -C $(BUILDDIR)/latex all-pdf + @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." + +text: + $(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text + @echo + @echo "Build finished. The text files are in $(BUILDDIR)/text." + +man: + $(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man + @echo + @echo "Build finished. The manual pages are in $(BUILDDIR)/man." + +texinfo: + $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo + @echo + @echo "Build finished. The Texinfo files are in $(BUILDDIR)/texinfo." + @echo "Run \`make' in that directory to run these through makeinfo" \ + "(use \`make info' here to do that automatically)." + +info: + $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo + @echo "Running Texinfo files through makeinfo..." + make -C $(BUILDDIR)/texinfo info + @echo "makeinfo finished; the Info files are in $(BUILDDIR)/texinfo." + +gettext: + $(SPHINXBUILD) -b gettext $(I18NSPHINXOPTS) $(BUILDDIR)/locale + @echo + @echo "Build finished. The message catalogs are in $(BUILDDIR)/locale." + +changes: + $(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes + @echo + @echo "The overview file is in $(BUILDDIR)/changes." + +linkcheck: + $(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck + @echo + @echo "Link check complete; look for any errors in the above output " \ + "or in $(BUILDDIR)/linkcheck/output.txt." + +doctest: + $(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest + @echo "Testing of doctests in the sources finished, look at the " \ + "results in $(BUILDDIR)/doctest/output.txt." diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000..c93d111 --- /dev/null +++ b/docs/README.md @@ -0,0 +1,4 @@ +# Docs + +This directory contains the ReST documentation that is published to +[docs.lnav.org](https://docs.lnav.org) diff --git a/docs/_config.yml b/docs/_config.yml new file mode 100644 index 0000000..6d116ee --- /dev/null +++ b/docs/_config.yml @@ -0,0 +1,62 @@ +# Welcome to Jekyll! +# +# This config file is meant for settings that affect your whole blog, values +# which you are expected to set up once and rarely edit after that. If you find +# yourself editing this file very often, consider using Jekyll's data files +# feature for the data you need to update frequently. +# +# For technical reasons, this file is *NOT* reloaded automatically when you use +# 'bundle exec jekyll serve'. If you change this file, please restart the server process. +# +# If you need help with YAML syntax, here are some quick references for you: +# https://learn-the-web.algonquindesign.ca/topics/markdown-yaml-cheat-sheet/#yaml +# https://learnxinyminutes.com/docs/yaml/ +# +# Site settings +# These are used to personalize your new site. If you look in the HTML files, +# you will see them accessed via {{ site.title }}, {{ site.email }}, and so on. +# You can create any custom variable you would like, and they will be accessible +# in the templates via {{ site.myvariable }}. + +title: The Logfile Navigator +version: 0.11.1 +email: support@lnav.org +description: >- # this means to ignore newlines until "baseurl:" + The Logfile Navigator, lnav for short, is an advanced log file viewer + for the small-scale. +# baseurl: /lnav +# url: "http://lnav.org" # the base hostname & protocol for your site, e.g. http://example.com +twitter_username: lnavapp +github: tstack/lnav + +# Build settings +theme: minima +plugins: + - jekyll-feed + - jekyll-redirect-from + +show_excerpts: true + +exclude: + - source + - tutorials + +# Exclude from processing. +# The following items will not be processed, by default. +# Any item listed under the `exclude:` key here will be automatically added to +# the internal "default list". +# +# Excluded items can be processed by explicitly listing the directories or +# their entries' file path in the `include:` list. +# +# exclude: +# - .sass-cache/ +# - .jekyll-cache/ +# - gemfiles/ +# - Gemfile +# - Gemfile.lock +# - node_modules/ +# - vendor/bundle/ +# - vendor/cache/ +# - vendor/gems/ +# - vendor/ruby/ diff --git a/docs/_includes/social.html b/docs/_includes/social.html new file mode 100644 index 0000000..15672c0 --- /dev/null +++ b/docs/_includes/social.html @@ -0,0 +1,121 @@ +<ul class="social-media-list"> + {%- if site.dribbble_username -%} + <li> + <a href="https://dribbble.com/{{ site.dribbble_username| cgi_escape | escape }}"> + <svg class="svg-icon"> + <use xlink:href="{{ '/assets/minima-social-icons.svg#dribbble' | relative_url }}"></use> + </svg> + <span class="username">{{ site.dribbble_username| escape }}</span></a> + </li> + {%- endif -%} + {%- if site.facebook_username -%} + <li> + <a href="https://www.facebook.com/{{ site.facebook_username| cgi_escape | escape }}"> + <svg class="svg-icon"> + <use xlink:href="{{ '/assets/minima-social-icons.svg#facebook' | relative_url }}"></use> + </svg> + <span class="username">{{ site.facebook_username| escape }}</span></a> + </li> + {%- endif -%} + {%- if site.flickr_username -%} + <li> + <a href="https://www.flickr.com/photos/{{ site.flickr_username| cgi_escape | escape }}"> + <svg class="svg-icon"> + <use xlink:href="{{ '/assets/minima-social-icons.svg#flickr' | relative_url }}"></use> + </svg> + <span class="username">{{ site.flickr_username| escape }}</span></a> + </li> + {%- endif -%} + {%- if site.github_username -%} + <li> + <a href="https://github.com/{{ site.github_username| cgi_escape | escape }}"> + <svg class="svg-icon"> + <use xlink:href="{{ '/assets/minima-social-icons.svg#github' | relative_url }}"></use> + </svg> + <span class="username">{{ site.github_username| escape }}</span></a> + </li> + {%- endif -%} + {%- if site.github -%} + <li><a href="https://github.com/{{ site.github }}"> + <svg class="svg-icon"> + <use xlink:href="{{ '/assets/minima-social-icons.svg#github' | relative_url }}"></use> + </svg> + <span class="username">{{ site.github | escape }}</span></a></li> + {%- endif -%} + {%- if site.instagram_username -%} + <li> + <a href="https://instagram.com/{{ site.instagram_username| cgi_escape | escape }}"> + <svg class="svg-icon"> + <use xlink:href="{{ '/assets/minima-social-icons.svg#instagram' | relative_url }}"></use> + </svg> + <span class="username">{{ site.instagram_username| escape }}</span></a> + </li> + {%- endif -%} + {%- if site.linkedin_username -%} + <li> + <a href="https://www.linkedin.com/in/{{ site.linkedin_username| cgi_escape | escape }}"> + <svg class="svg-icon"> + <use xlink:href="{{ '/assets/minima-social-icons.svg#linkedin' | relative_url }}"></use> + </svg> + <span class="username">{{ site.linkedin_username| escape }}</span></a> + </li> + {%- endif -%} + {%- if site.pinterest_username -%} + <li> + <a href="https://www.pinterest.com/{{ site.pinterest_username| cgi_escape | escape }}"> + <svg class="svg-icon"> + <use xlink:href="{{ '/assets/minima-social-icons.svg#pinterest' | relative_url }}"></use> + </svg> + <span class="username">{{ site.pinterest_username| escape }}</span></a> + </li> + {%- endif -%} + {%- for mst in site.mastodon -%}{%- if mst.username and mst.instance -%} + <li> + <a href="https://{{ mst.instance| cgi_escape | escape}}/@{{mst.username}}"> + <svg class="svg-icon"> + <use xlink:href="{{ '/assets/minima-social-icons.svg#mastodon' | relative_url }}"></use> + </svg> + <span class="username">{{ mst.username|escape }}</span></a></li> + {%- endif -%}{%- endfor -%} + {%- if site.twitter_username -%} + <li> + <a href="https://www.twitter.com/{{ site.twitter_username| cgi_escape | escape }}"> + <svg class="svg-icon"> + <use xlink:href="{{ '/assets/minima-social-icons.svg#twitter' | relative_url }}"></use> + </svg> + <span class="username">{{ site.twitter_username| escape }}</span></a> + </li> + {%- endif -%} + {%- if site.youtube_username -%} + <li> + <a href="https://youtube.com/{{ site.youtube_username| cgi_escape | escape }}"> + <svg class="svg-icon"> + <use xlink:href="{{ '/assets/minima-social-icons.svg#youtube' | relative_url }}"></use> + </svg> + <span class="username">{{ site.youtube_username| escape }}</span></a> + </li> + {%- endif -%} + {%- if site.googleplus_username -%} + <li> + <a href="https://plus.google.com/{{ site.googleplus_username| escape }}"> + <svg class="svg-icon"> + <use xlink:href="{{ '/assets/minima-social-icons.svg#googleplus' | relative_url }}"></use> + </svg> + <span class="username">{{ site.googleplus_username| escape }}</span></a> + </li> + {%- endif -%} + {%- if site.rss -%} + <li><a href="{{ 'feed.xml' | relative_url }}"> + <svg class="svg-icon"> + <use xlink:href="{{ '/assets/minima-social-icons.svg#rss' | relative_url }}"></use> + </svg> + <span>{{ site.rss | escape }}</span></a></li> + {%- endif -%} + <li><a href="https://discord.gg/erBPnKwz7R"> + <img style="height: 1.75em" + src="https://assets-global.website-files.com/6257adef93867e50d84d30e2/62594fdd4b264a31eafa1e5d_92ad040ed5143bfb541ea61f5c3bb18f.svg"/> + </a></li> +</ul> + +<script data-goatcounter="https://lnav.goatcounter.com/count" + async src="//gc.zgo.at/count.js"></script> diff --git a/docs/_layouts/post.html b/docs/_layouts/post.html new file mode 100644 index 0000000..6deec42 --- /dev/null +++ b/docs/_layouts/post.html @@ -0,0 +1,17 @@ +--- +layout: default +--- + +<div class="post"> + + <header class="post-header"> + <h1 class="post-title">{{ page.title }}</h1> + </header> + + <article class="post-content"> + {{ content }} + </article> + + <script src="/assets/js/codeblock.js"></script> + +</div> diff --git a/docs/_layouts/top.html b/docs/_layouts/top.html new file mode 100644 index 0000000..75be485 --- /dev/null +++ b/docs/_layouts/top.html @@ -0,0 +1,78 @@ +<!DOCTYPE html> +<html lang="{{ page.lang | default: site.lang | default: "en" }}"> + +{%- include head.html -%} + +<link rel="icon" type="image/svg+xml" href="/assets/images/favicon.svg"> +<link rel="icon" type="image/png" href="/assets/images/favicon.png"> + +<style> + main { + margin: 0 50px; + } + + #top-description { + font-size: xx-large; + margin-top: 3em; + margin-left: 1em; + } + + #intro { + font-size: x-large; + font-weight: lighter; + margin-top: 2em; + margin-left: 2em; + } + + .dlrow { + display: grid; + align-items: start; + grid-template-columns: repeat(auto-fit, minmax(22em, 1fr)); + width: 100%; + } + + p:before, #top-description:before { + content: ""; + width: 6em; + display: block; + overflow: hidden; + } + + dl { + display: grid; + grid-template-rows: auto auto; + grid-auto-flow: column; + } + dl dt { + font-weight: normal; + font-size: x-large; + text-align: center; + border-bottom: 1px solid #8d8; + margin: 20px 30px; + height: 2em; + } + dl dd { + margin: 0 30px 20px 30px; + min-height: 3em; + color: #828282; + font-size: medium; + } +</style> + +<body> + +<a href="https://github.com/tstack/lnav"> + <img loading="lazy" width="149" style="position: absolute; z-index: 100" height="149" src="https://github.blog/wp-content/uploads/2008/12/forkme_left_green_007200.png?resize=149%2C149" class="attachment-full size-full" alt="Fork me on GitHub" data-recalc-dims="1"> +</a> + +{%- include header.html -%} + +<main class="page-content" aria-label="Content"> + {{ content }} +</main> + +{%- include footer.html -%} + +</body> + +</html> diff --git a/docs/_posts/2013-09-10-json-encoded-logs.md b/docs/_posts/2013-09-10-json-encoded-logs.md new file mode 100644 index 0000000..1fa36a4 --- /dev/null +++ b/docs/_posts/2013-09-10-json-encoded-logs.md @@ -0,0 +1,56 @@ +--- +layout: post +title: "Support for JSON-encoded logs in v0.6.1" +date: 2013-09-10 00:00:00 +excerpt: Turning JSON barf into something readable. +--- + +Making logs easily digestible by machines is becoming a concern as tools like +elasticsearch become more popular. One of the popular strategies is to encode +the whole log message in JSON and then write that as a single line to a file. +For example: + +```json +{"time": "2013-09-04T23:55:09.274041Z", "level" : "INFO", "body" : "Hello, World!" } +{"time": "2013-09-04T23:56:00.285224Z", "level" : "ERROR", "body" : "Something terrible has happened!", "tb": " foo.c:12\n bar.y:33" } +``` + +Unfortunately, what is good for a machine is not so great for a human. To try to +improve the situation, the latest release of lnav includes support for parsing +JSON log messages and transforming them on-the-fly. The display format is +specified in a log format configuration and can specify which fields should be +displayed on the main message line. Any unused fields that are found in the +message will be displayed below the main field so you don't miss anything. + +The above log lines can be transformed using the following format configuration: + +```json +{ + "json_ex_log": { + "title": "Example JSON Log", + "description": "An example log format configuration for JSON logs", + "json": true, + "file-pattern": "test-log\\.json.*", + "level-field": "level", + "line-format": [ + { + "field": "time" + }, + " ", + { + "field": "body" + } + ] + } +} +``` + +After copying this config to `~/.lnav/formats/test/format.json`, the log messages +will look like this when viewed in lnav: + +``` +2013-09-04T23:55:09.274041Z Hello, World! +2013-09-04T23:56:00.285224Z Something terrible has happened! + tb: foo.c:12 + tb: bar.y:33 +``` diff --git a/docs/_posts/2013-09-13-four-years-on-github.md b/docs/_posts/2013-09-13-four-years-on-github.md new file mode 100644 index 0000000..875dc53 --- /dev/null +++ b/docs/_posts/2013-09-13-four-years-on-github.md @@ -0,0 +1,12 @@ +--- +layout: post +title: "Four years on GitHub" +date: 2013-09-13 00:00:00 +excerpt: Still going strong! +--- + +The [first commit](https://github.com/tstack/lnav/commit/b4ec432515e95e86ec9d711833b8cb34d0912546) +to the [lnav repository](https://github.com/tstack/lnav) was four years ago +today! I started working on lnav a couple of years before that, but this was +its first public appearance. I still use it every day and plan to keep things +going for years to come. diff --git a/docs/_posts/2013-10-05-mini-review-in-linux-magazine.md b/docs/_posts/2013-10-05-mini-review-in-linux-magazine.md new file mode 100644 index 0000000..19f9c73 --- /dev/null +++ b/docs/_posts/2013-10-05-mini-review-in-linux-magazine.md @@ -0,0 +1,8 @@ +--- +layout: post +title: "Mini review of lnav in Linux Magazine" +date: 2013-10-05 00:00:00 +--- + +The [October issue of Linux Magazine](http://www.linux-magazine.com/Issues/2013/155/Tool-Tips) +has a nice review of lnav v0.5.0, the author gave it four out of five stars! diff --git a/docs/_posts/2013-10-06-competing-with-tail.md b/docs/_posts/2013-10-06-competing-with-tail.md new file mode 100644 index 0000000..b93e7bd --- /dev/null +++ b/docs/_posts/2013-10-06-competing-with-tail.md @@ -0,0 +1,23 @@ +--- +layout: post +title: "Competing with 'tail -f'" +date: 2013-09-10 00:00:00 +excerpt: The standard utilities are tough competition. +--- + +Probably the toughest competition for lnav is the standard Unix utilities like +tail, grep, less, and emacs/vim. It can be hard trying to convince people that +these built-in commands that they've used for forever can be improved upon. The +advanced features of lnav might even work against it since folks are expecting +to have to learn a bunch of stuff to see any benefits. + +The reality is that there are quite a few "passive" features in lnav that can +provide value with no effort required by the user. For example, lnav can easily +replace 'tail -f', it's even shorter to type! Beyond the basic task of +displaying new lines appended to a log file, you also get to see log messages +from multiple files interleaved, the ability to scroll backwards, syntax +highlighting, live searching, and so on. These basic features do not have the +same "wow" factor as executing a SQL query over data automatically extracted +from a log file, but they're the features that get used 90% of the time. + +Anyways, I think I'm gaining a new appreciation for marketing/sales... diff --git a/docs/_posts/2013-11-01-mini-review-linux-user-magazine.md b/docs/_posts/2013-11-01-mini-review-linux-user-magazine.md new file mode 100644 index 0000000..02d67f6 --- /dev/null +++ b/docs/_posts/2013-11-01-mini-review-linux-user-magazine.md @@ -0,0 +1,9 @@ +--- +layout: post +title: "Mini-review of lnav in Linux User Magazine" +date: 2013-09-10 00:00:00 +--- + +The german magazine, [Linux User](http://www.linux-user.de/), has a +mini-review of lnav v0.6.0 in their +[November issue](http://www.linux-user.de/Downloads/LUCE/2013/lu-ce_2013-11.pdf)! diff --git a/docs/_posts/2014-02-22-changes-to-the-scrollbar.md b/docs/_posts/2014-02-22-changes-to-the-scrollbar.md new file mode 100644 index 0000000..2c9a21f --- /dev/null +++ b/docs/_posts/2014-02-22-changes-to-the-scrollbar.md @@ -0,0 +1,21 @@ +--- +layout: post +title: "Changes To The Scrollbar" +date: 2014-02-22 00:00:00 +excerpt: Packing more information into the right scrollbar. +--- + +I've made some changes to the scrollbar that is shown on the right side of the +display based on some feedback from users. The scroll area now has a single +vertical line extending from the top to the bottom. Previously, this area would +show log message characters and it wasn't very clear that the scroll bar +existed. The line is colored based whether there are errors or warnings in that +part of the log. The coloring should make it easier to see the distribution of +errors across the whole log. Similarly, there are notches added to the left and +right side of the line to show search hits and bookmarks, respectively. See the +following screenshot to get an idea of what it looks like: + + + +These changes are currently only in the latest code from git. I'll be playing +with things a bit more before making a release. diff --git a/docs/_posts/2014-11-11-lofi-mode.md b/docs/_posts/2014-11-11-lofi-mode.md new file mode 100644 index 0000000..b6ca628 --- /dev/null +++ b/docs/_posts/2014-11-11-lofi-mode.md @@ -0,0 +1,23 @@ +--- +layout: post +title: Added "lo-fi" mode +excerpt: An alternative way to copy displayed text. +--- + +*(This change is in v0.7.2+)* + +Copying text to the clipboard can be done by marking lines with the +[bookmark hotkeys](https://docs.lnav.org/en/latest/hotkeys.html#bookmarks), +like `m`, and then pressing `c`. Commands that write to a file, +like [`:write-csv-to`](https://docs.lnav.org/en/latest/commands.html#write-csv-to-path) +accept `/dev/clipboard` as a way to write to the clipboard. However, if the +native clipboard isn't supported, or you're on an SSH connection, you can +now switch to "lo-fi" mode. In "lo-fi" mode, lnav drops out of the curses +display and prints the raw text to the terminal. You can switch to "lo-fi" +mode in a view by pressing `CTRL-L`. For commands, you can use a dash `-` +to switch to "lo-fi" and print to standard out. + +<script id="asciicast-fH4cdgugIJVcPQnSwCmTdaA7f" + src="https://asciinema.org/a/fH4cdgugIJVcPQnSwCmTdaA7f.js" + async> +</script> diff --git a/docs/_posts/2015-04-11-pretty-print-view.md b/docs/_posts/2015-04-11-pretty-print-view.md new file mode 100644 index 0000000..6ea31b1 --- /dev/null +++ b/docs/_posts/2015-04-11-pretty-print-view.md @@ -0,0 +1,39 @@ +--- +layout: post +title: "Pretty-print view in v0.7.3" +date: 2015-04-11 00:00:00 +excerpt: Automatically reformat structured data with "SHIFT+P". +--- + +I wanted to call out the pretty-print feature in the latest release of lnav. +This idea came from a coworker of Suresh who was having a hard time trying to +read some unformatted XML in a log. They wanted the XML pretty-printed and were +hoping that could be done by just piping the message to xmlpp or the like. So, +first we implemented the 'pipe-to' and 'pipe-line-to' commands that will let you +pipe log messages to a command and then display the result inside of lnav. That +worked well enough, but pretty-printing is such a frequent operation that having +to execute a command was kind of a pain. It would also be nice if it worked for +a variety of text, like JSON or Python data. The solution we came up with was to +leverage the existing code for parsing log messages to create a simple +pretty-printer that should work for most data formats. Another benefit is that +the log message does not have to be well-formed for the printer to work, any +leading or trailing garbage shouldn't confuse things. + +As an example, here is a screenshot of the log message with the unformatted XML +text with word-wrapping turned on: + + + +That's not very easy to read and it's hard to figure out the structure of the +message. Now, here is that same message after pressing SHIFT+P to switch to the +pretty-print view of lnav: + + + +The XML text is indented nicely and the usual syntax highlighting is applied. +Also notice that lnav will automatically try to lookup the DNS name for IP +addresses. Overall, I think it's a major improvement over the raw view. + +This is a pretty simple feature but I have found it quite useful in the couple +weeks that it has been implemented. It's so useful that I'm kicking myself for +not having thought of it before. diff --git a/docs/_posts/2016-03-20-lnav-in-print.md b/docs/_posts/2016-03-20-lnav-in-print.md new file mode 100644 index 0000000..73e7722 --- /dev/null +++ b/docs/_posts/2016-03-20-lnav-in-print.md @@ -0,0 +1,10 @@ +--- +layout: post +title: "lnav in print" +date: 2016-03-20 00:00:00 +--- + +A [review of lnav](https://archive.org/details/Linux_User_Developer_162_2016_UK/page/n87/mode/2up) +is in Linux User & Developer magazine issue 162: + + diff --git a/docs/_posts/2018-03-27-reveal-file-paths.md b/docs/_posts/2018-03-27-reveal-file-paths.md new file mode 100644 index 0000000..01b31d8 --- /dev/null +++ b/docs/_posts/2018-03-27-reveal-file-paths.md @@ -0,0 +1,16 @@ +--- +layout: post +title: "Reveal log message source file paths by pressing left-arrow" +excerpt: Find out where log messages are coming from. +--- + +*(This change is in v0.8.4+)* + +If you want to know which file log messages are coming from, you can press the +left-arrow to reveal the unique name and then press it again to reveal the +full path. + +<script id="asciicast-ATHHpQiHVaTVxVRkhCv4ED7wT" + src="https://asciinema.org/a/ATHHpQiHVaTVxVRkhCv4ED7wT.js" + async> +</script> diff --git a/docs/_posts/2018-04-05-linux-magazine-tutorial.md b/docs/_posts/2018-04-05-linux-magazine-tutorial.md new file mode 100644 index 0000000..9d868b5 --- /dev/null +++ b/docs/_posts/2018-04-05-linux-magazine-tutorial.md @@ -0,0 +1,9 @@ +--- +layout: post +title: "Tutorial for lnav in Linux Magazine" +date: 2018-04-05 00:00:00 +--- + +Looks like there was an in-depth +[tutorial on lnav in Linux Magazine](http://www.linux-magazine.com/Issues/2017/196/Tutorials-lnav). +Unfortunately, I didn't notice until now and missed out on a hardcopy. diff --git a/docs/_posts/2018-05-17-tags-and-comments.md b/docs/_posts/2018-05-17-tags-and-comments.md new file mode 100644 index 0000000..c6f2fef --- /dev/null +++ b/docs/_posts/2018-05-17-tags-and-comments.md @@ -0,0 +1,22 @@ +--- +layout: post +title: "Support for tagging and commenting on log messages" +excerpt: Annotate log messages with your thoughts. +--- + +*(This change is in v0.8.4+)* + +If you have been wanting to add notes to log messages you might be interested +in, the new [`:comment`](https://docs.lnav.org/en/latest/commands.html#comment-text) +and [`:tag`](https://docs.lnav.org/en/latest/commands.html#tag-tag) commands. +These commands add a comment or tag(s) to the top message in the log view. +The comments and tags are saved in the session, so they will be restored +automatically when the file is reopened. These annotations can be searched +for using the regular search prompt and can be accessed in the +[log tables](https://docs.lnav.org/en/latest/sqlext.html#log-tables) using the +`log_tags` and `log_comment` columns. + +<script id="asciicast-yRTcQd2VMv3QZVs5597OyAAxI" + src="https://asciinema.org/a/yRTcQd2VMv3QZVs5597OyAAxI.js" + async> +</script> diff --git a/docs/_posts/2018-11-9-visual-filter-editor.md b/docs/_posts/2018-11-9-visual-filter-editor.md new file mode 100644 index 0000000..8fb59a4 --- /dev/null +++ b/docs/_posts/2018-11-9-visual-filter-editor.md @@ -0,0 +1,35 @@ +--- +layout: post +title: "Visual filter editor" +excerpt: A friendlier way to interact with filters. +--- + +*(This change is in v0.8.5+)* + +A visual filter editor has been added to make it easier to create, edit, +enable, disable, and delete filters. In the log or text views, pressing `TAB` +will open the filter editor panel. While the panel is in focus, the following +hotkeys can be used: + +- `i` - Create an IN filter that will only show lines that match the given + regular expression. +- `o` - Create an OUT filter that will hide lines that match the given regular + expression. +- `Space` - Toggle the filter between being enabled and disabled. +- `Enter` - Edit the selected filter. +- `Shift+D` - Delete the filter. +- `t` - Switch a filter from an IN to an OUT or vice-versa. +- `f` - Globally enable or disable filtering. + +When editing a filter, the main view will highlight lines that portion of the +lines that match the given regular expression: + +- Lines that match an OUT filter are highlighted with red; +- Lines that match an IN filter are highlighted with green. + +You can also press `TAB` to complete words that are visible in the main view. + +<script id="asciicast-tcHeLbqVImRVcxWTYIrm3v6bw" + src="https://asciinema.org/a/tcHeLbqVImRVcxWTYIrm3v6bw.js" + async> +</script> diff --git a/docs/_posts/2019-05-08-themes.md b/docs/_posts/2019-05-08-themes.md new file mode 100644 index 0000000..a145ca6 --- /dev/null +++ b/docs/_posts/2019-05-08-themes.md @@ -0,0 +1,28 @@ +--- +layout: post +title: Support for Themes +excerpt: Change the user-interface to your liking +--- + +*(This change is in v0.9.0+)* + +The lnav user-interface can now be customized by selecting one of the +builtin themes, like +[monocai](https://github.com/tstack/lnav/blob/master/src/themes/monocai.json) +and +[night-owl](https://github.com/tstack/lnav/blob/master/src/themes/night-owl.json), +or +[by defining your own theme](https://lnav.readthedocs.io/en/latest/config.html#theme-definitions). + +Selecting a theme can be done through the +[`:config`](https://docs.lnav.org/en/latest/commands.html#config-option-value) +command, like so: + +``` +:config /ui/theme monocai +``` + +Pressing `TAB` after the `/ui/theme ` will cycle through the available themes, +like so: + + diff --git a/docs/_posts/2020-12-23-xpath-sql-function.md b/docs/_posts/2020-12-23-xpath-sql-function.md new file mode 100644 index 0000000..b0a7657 --- /dev/null +++ b/docs/_posts/2020-12-23-xpath-sql-function.md @@ -0,0 +1,39 @@ +--- +layout: post +title: Drilling down into XML snippets +excerpt: The new "xpath()" table-valued SQL function. +--- + +*(This change is in [**v0.10.0+**](https://github.com/tstack/lnav/releases/tag/v0.10.0))* + +XML snippets in log messages can now be queried using the +[`xpath()`](https://docs.lnav.org/en/latest/sqlext.html#xpath-xpath-xmldoc) +table-valued SQL function. The function takes an +[XPath](https://developer.mozilla.org/en-US/docs/Web/XPath), the XML snippet +to be queried, and returns a table with the results of the XPath query. +For example, given following XML document: + +```xml +<msg>Hello, World!</msg> +``` + +Extracting the text value from the `msg` node can be done using the following +query: + +```sql +SELECT result FROM xpath('/msg/text()', '<msg>Hello, World!</msg>') +``` + +Of course, you won't typically be passing XML values as string literals, you +will be extracting them from log messages. Assuming your log format already +extracts the XML data, you can do a `SELECT` on the log format table and join +that with the `xpath()` call. Since it can be challenging to construct a +correct `xpath()` call, lnav will suggest calls for the nodes it finds in any +XML log message fields. The following asciicast demonstrates this flow: + +<script id="asciicast-x89mrk8JPHBmB4pTbaZvTt8Do" + src="https://asciinema.org/a/x89mrk8JPHBmB4pTbaZvTt8Do.js" + async> +</script> + +The implementation uses the [pugixml](https://pugixml.org) library. diff --git a/docs/_posts/2021-05-03-tailing-remote-files.md b/docs/_posts/2021-05-03-tailing-remote-files.md new file mode 100644 index 0000000..9d1d15c --- /dev/null +++ b/docs/_posts/2021-05-03-tailing-remote-files.md @@ -0,0 +1,32 @@ +--- +layout: post +title: Tailing files on remote hosts +excerpt: Native support for tailing logs on machines accessible via SSH +--- + +*(This change is in [**v0.10.0+**](https://github.com/tstack/lnav/releases/tag/v0.10.0))* + +One of the new features in the upcoming v0.10.0 release of lnav is support +for tailing log files on remote hosts via SSH. This feature allows you to +view local files and files from multiple remote hosts alongside each other +in the log view. The only setup required is to ensure the machines can be +accessed via SSH without any interaction, meaning the host key must have +been previously accepted and public key authentication configured. Opening +a remote file is then simply a matter of specifying the location using the +common scp syntax (i.e. `user@host:/path/to/file`). + +When lnav accesses a remote host, it transfers an agent (called the +"tailer") to the host to handle file system requests from lnav. The agent +is an [αcτµαlly pδrταblε εxεcµταblε](https://justine.lol/ape.html) that +should run on most X86 Operating Systems. The agent will monitor the +files of interest and synchronize their contents back to the host machine. +In addition, the agent can be used to satisfy interactive requests for +TAB-completion of remote file paths and previewing directory and file +contents. + +The following asciicast shows lnav opening log files on MacOS and FreeBSD: + +<script id="asciicast-fblzf1Ir5Rr0b5wMGEJBb95ye" + src="https://asciinema.org/a/fblzf1Ir5Rr0b5wMGEJBb95ye.js" + async> +</script> diff --git a/docs/_posts/2022-05-01-regex101-integration.md b/docs/_posts/2022-05-01-regex101-integration.md new file mode 100644 index 0000000..02011ce --- /dev/null +++ b/docs/_posts/2022-05-01-regex101-integration.md @@ -0,0 +1,73 @@ +--- +layout: post +title: Integration with regex101.com +excerpt: Create/edit format files using regex101.com +--- + +*(This change will be in the upcoming v0.11.0 release)* + +Creating and updating format files for **lnav** can be a bit tedious and +error-prone. To help streamline the process, an integration with regex101.com +has been added. Now, you can create regular expressions for plaintext log +files on https://regex101.com and then create a skeleton format file with a +simple command. If you already have a format file that needs to be updated, +you can push the regexes up to regex101, edit them with their interface, and +then pull the changes back down as a format patch file. + +To further improve the experience of developing with format files, there is +also work underway to improve error messages. Many messages should be clearer, +more context is provided, and they should look nicer as well. For example, the +following error is displayed when a format regex is not valid: + + + +## Management CLI + +The regex101 integration can be accessed through the new "management-mode CLI". +This mode can be accessed by passing `-m` as the first option to **lnav**. The +management CLI is organized as a series of nested commands. If you're not sure +what to do at a given level, run the command as-is and the CLI should print out +help text to guide you through the hierarchy of commands and required +parameters. + +### Create a format from a regular expression + +The `regex101 import` command can be used to import a regular expression from +regex101.com and create or patch a format file. The command takes the URL of +the regex, the format name, and the name of the regex in the log format ( +defaults to "std" if not given). For example, the following command can be used +to import the regex at "https://regex101.com/r/zpEnjV/2" into the format named " +re101_example_log": + +```console +$ lnav -m regex101 import https://regex101.com/r/zpEnjV/2 re101_example_log +``` + +If the import was successful, the path to the skeleton format file will be +printed. You will most likely need to edit the file to fill in more details +about your log format. + +### Editing an existing regular expression + +If you have a log format with a regex that needs to be updated, you can push +the regex to regex101.com for editing with a command like (replace +"myformat_log"/"std" with the name of your format and regex): + +```console +$ lnav -m format myformat_log regex std regex101 push +``` + +Along with the regex, the format's samples will be added to the entry to ensure +changes won't break existing matches. If the push was successful, the URL for +the new regex101.com entry will be printed out. You can use that URL to edit the +regex to your needs. Once you're done editing the regex, you can pull the +changes down to a "patch" file using the following command: + +```console +$ lnav -m format myformat_log regex std regex101 pull +``` + +The patch file will be evaluated after the original format file and override +the values from the original. Once you are satisfied with the changes, you +can move the contents of the patch file to the original file and delete the +patch. diff --git a/docs/_posts/2022-08-04-pretty-errors.md b/docs/_posts/2022-08-04-pretty-errors.md new file mode 100644 index 0000000..b04c9e8 --- /dev/null +++ b/docs/_posts/2022-08-04-pretty-errors.md @@ -0,0 +1,46 @@ +--- +layout: post +title: Pretty error messages +excerpt: Error message improvements +--- + +*(This change will be in the upcoming v0.11.0 release)* + +Taking a page from compilers like rustc, I've spent some time +improving error messages to make them look nicer and be more +helpful. Fortunately, SQLite has improved their error reporting +as well by adding +[sqlite3_error_offset()](http://sqlite.com/c3ref/errcode.html). +This function can point to the part of the SQL statement that +was in error. As an example of the improvement, a SQL file +that contained the following content: + +```sql + +-- This is a test +SELECT abc), +rtrim(def) +FROM mytable; +``` + +Would report an error like the following on startup in v0.10.1: + +```text +error:/Users/tstack/.config/lnav/formats/installed/test.sql:2:near ")": syntax error +``` + +Now, you will get a clearer error message with a syntax-highlighted +code snippet and a pointer to the part of the code that has the +problem: + + + +Inside the TUI, a panel has been added at the bottom to display these +long-form error messages. The panel will disappear after a short +time or when input is received. Here is an example showing an error +for an invalid regular expression: + +<script id="asciicast-lmYMLZsB02WbSO8VEz4aVLXa1" + src="https://asciinema.org/a/lmYMLZsB02WbSO8VEz4aVLXa1.js" + async> +</script> diff --git a/docs/_posts/2022-08-06-markdown-support.md b/docs/_posts/2022-08-06-markdown-support.md new file mode 100644 index 0000000..4504bbd --- /dev/null +++ b/docs/_posts/2022-08-06-markdown-support.md @@ -0,0 +1,33 @@ +--- +layout: post +title: Markdown Support +excerpt: A side effect of fancier help text +--- + +*(This change will be in the upcoming v0.11.0 release)* + +As part of the effort to polish the lnav TUI, I wanted to make the builtin +help text look a bit nicer. The current help text is a plain text file with +some ANSI escape sequences for colors. It's not easy to write or read. Since +Markdown has become a dominant way to write this type of document, I figured +I could use that and have the side benefit of allowing lnav to read Markdown +docs. Fortunately, the [MD4C](https://github.com/mity/md4c) library exists. +This library provides a nice event-driven parser for documents instead of +just converting directly to HTML. In addition, document structure is now +shown/navigable through the new breadcrumb bar at the top. I think the +result is pretty nice: + +<script id="asciicast-2hx3UiyzOHQXBQOBf31ztKvHc" + src="https://asciinema.org/a/2hx3UiyzOHQXBQOBf31ztKvHc.js" + async> +</script> + +## Viewing Markdown Files + +Files with an `.md` suffix will be considered as Markdown and will be +parsed as such. As an example, here is lnav displaying its README.md file: + +<script id="asciicast-iw4rwddZNGCe3v8DyOfItERG9" + src="https://asciinema.org/a/iw4rwddZNGCe3v8DyOfItERG9.js" + async> +</script> diff --git a/docs/_posts/2022-09-01-playground.md b/docs/_posts/2022-09-01-playground.md new file mode 100644 index 0000000..3cbaa11 --- /dev/null +++ b/docs/_posts/2022-09-01-playground.md @@ -0,0 +1,27 @@ +--- +layout: post +title: Playground and Tutorial +excerpt: Try lnav without having to install anything +--- + +To make it easier to try out **lnav**, I've deployed an ssh-based playground +and tutorial. You can SSH as follows to try them out: + +```console +$ ssh playground@demo.lnav.org +$ ssh tutorial1@demo.lnav.org +``` + +<script id="asciicast-HiiUMMmRKZh0uCVKm1Uw8WLlw" + src="https://asciinema.org/a/HiiUMMmRKZh0uCVKm1Uw8WLlw.js" + async> +</script> + +The playground has a couple of example logs to play with. The tutorial +tries to guide you through the basics of navigating log files with lnav. +The server is running on the free-tier of [fly.io](https://fly.io), so +please be kind. + +This effort was inspired by the `git.charm.sh` SSH server and by the +[fasterthanli.me](https://fasterthanli.me/articles/remote-development-with-rust-on-fly-io) +post on doing remote development on fly.io. diff --git a/docs/_posts/2022-09-24-vscode-extension.md b/docs/_posts/2022-09-24-vscode-extension.md new file mode 100644 index 0000000..6c7b78f --- /dev/null +++ b/docs/_posts/2022-09-24-vscode-extension.md @@ -0,0 +1,21 @@ +--- +layout: post +title: VSCode Extension for lnav +excerpt: Syntax highlighting for lnav scripts +--- + +I've published a simple [Visual Studio Code extension for lnav]( +https://marketplace.visualstudio.com/items?itemName=lnav.lnav) +that adds syntax highlighting for scripts. The following is a +screenshot showing the `dhclient-summary.lnav` script that is +builtin: + + + +The lnav commands, those prefixed with colons, are marked as +keywords and the SQL blocks are treated as an embedded language +and highlighted accordingly. + +If people find this useful, we can take it further and add +support for running the current script/snippet in a new lnav +process or even talking to an existing one. diff --git a/docs/_posts/2023-06-23-cursor-mode.md b/docs/_posts/2023-06-23-cursor-mode.md new file mode 100644 index 0000000..c0a9f1d --- /dev/null +++ b/docs/_posts/2023-06-23-cursor-mode.md @@ -0,0 +1,30 @@ +--- +layout: post +title: Cursor Mode +excerpt: Move around the main view using a cursor +--- + +*(This change is in [**v0.11.2+**](https://github.com/tstack/lnav/releases/tag/v0.11.2-rc3))* + +The major change in the v0.11.2 release is the addition of a "cursor mode" +for the main view. Instead of focusing on the top line for interacting +with **lnav**, a cursor line is displayed and interactions focus on that. +The arrow keys and the hotkeys that jump between bookmarks, like search +hits and errors, now move the focused line instead of scrolling the view. +To help provide context for what you're looking at, large jumps will keep +the focused line in the middle of the view. Smaller movements, like +moving the cursor above the top line, will scroll the view a small amount +so as not to be jarring. + +You can enable/disable cursor mode interactively by pressing `CTRL` + `x`. +Or, you can permanently enable cursor mode by running the following +`:config` command: + +``` +:config /ui/movement/mode cursor +``` + +<script async + id="asciicast-d94CmxlGM01I0L5HNn9qDn917" + src="https://asciinema.org/a/d94CmxlGM01I0L5HNn9qDn917.js"> +</script> diff --git a/docs/assets/images/favicon.png b/docs/assets/images/favicon.png Binary files differnew file mode 100644 index 0000000..e126db3 --- /dev/null +++ b/docs/assets/images/favicon.png diff --git a/docs/assets/images/favicon.svg b/docs/assets/images/favicon.svg new file mode 100644 index 0000000..6c62941 --- /dev/null +++ b/docs/assets/images/favicon.svg @@ -0,0 +1,3 @@ +<svg xmlns="http://www.w3.org/2000/svg" version="1.1" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:svgjs="http://svgjs.com/svgjs" width="1000" height="1000"><g transform="matrix(38.46153846153846,0,0,38.46153846153846,0,0)"><svg xmlns="http://www.w3.org/2000/svg" version="1.1" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:svgjs="http://svgjs.com/svgjs" width="26" height="26"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 26 26"> + <path d="M 13 0.1875 C 5.925781 0.1875 0.1875 5.925781 0.1875 13 C 0.1875 20.074219 5.925781 25.8125 13 25.8125 C 20.074219 25.8125 25.8125 20.074219 25.8125 13 C 25.8125 5.925781 20.074219 0.1875 13 0.1875 Z M 11.9375 3.25 C 11.785156 3.464844 11.6875 3.71875 11.6875 4 C 11.6875 4.734375 12.265625 5.3125 13 5.3125 C 13.734375 5.3125 14.3125 4.734375 14.3125 4 C 14.3125 3.71875 14.214844 3.464844 14.0625 3.25 C 18.625 3.746094 22.253906 7.375 22.75 11.9375 C 22.535156 11.785156 22.28125 11.6875 22 11.6875 C 21.265625 11.6875 20.6875 12.265625 20.6875 13 C 20.6875 13.734375 21.265625 14.3125 22 14.3125 C 22.28125 14.3125 22.535156 14.214844 22.75 14.0625 C 22.253906 18.625 18.625 22.253906 14.0625 22.75 C 14.214844 22.535156 14.3125 22.28125 14.3125 22 C 14.3125 21.265625 13.734375 20.6875 13 20.6875 C 12.265625 20.6875 11.6875 21.265625 11.6875 22 C 11.6875 22.28125 11.785156 22.535156 11.9375 22.75 C 7.375 22.253906 3.746094 18.625 3.25 14.0625 C 3.464844 14.214844 3.71875 14.3125 4 14.3125 C 4.734375 14.3125 5.3125 13.734375 5.3125 13 C 5.3125 12.265625 4.734375 11.6875 4 11.6875 C 3.71875 11.6875 3.464844 11.785156 3.25 11.9375 C 3.746094 7.375 7.375 3.746094 11.9375 3.25 Z M 18.3125 7.6875 L 11.375 11.375 L 7.6875 18.3125 L 14.625 14.625 Z M 13 11.75 C 13.695313 11.75 14.25 12.304688 14.25 13 C 14.25 13.695313 13.695313 14.25 13 14.25 C 12.304688 14.25 11.75 13.695313 11.75 13 C 11.75 12.304688 12.304688 11.75 13 11.75 Z"></path> +</svg></svg></g></svg>
\ No newline at end of file diff --git a/docs/assets/images/linux-user-and-dev-mag.jpeg b/docs/assets/images/linux-user-and-dev-mag.jpeg Binary files differnew file mode 100644 index 0000000..c61e8e0 --- /dev/null +++ b/docs/assets/images/linux-user-and-dev-mag.jpeg diff --git a/docs/assets/images/lnav-after-pretty.png b/docs/assets/images/lnav-after-pretty.png Binary files differnew file mode 100644 index 0000000..3983713 --- /dev/null +++ b/docs/assets/images/lnav-after-pretty.png diff --git a/docs/assets/images/lnav-before-pretty.png b/docs/assets/images/lnav-before-pretty.png Binary files differnew file mode 100644 index 0000000..3100091 --- /dev/null +++ b/docs/assets/images/lnav-before-pretty.png diff --git a/docs/assets/images/lnav-front-page.png b/docs/assets/images/lnav-front-page.png Binary files differnew file mode 100644 index 0000000..20483dd --- /dev/null +++ b/docs/assets/images/lnav-front-page.png diff --git a/docs/assets/images/lnav-hist.png b/docs/assets/images/lnav-hist.png Binary files differnew file mode 100644 index 0000000..f3e92eb --- /dev/null +++ b/docs/assets/images/lnav-hist.png diff --git a/docs/assets/images/lnav-invalid-regex-error.png b/docs/assets/images/lnav-invalid-regex-error.png Binary files differnew file mode 100644 index 0000000..d67d53c --- /dev/null +++ b/docs/assets/images/lnav-invalid-regex-error.png diff --git a/docs/assets/images/lnav-multi-file2.png b/docs/assets/images/lnav-multi-file2.png Binary files differnew file mode 100644 index 0000000..3498fbc --- /dev/null +++ b/docs/assets/images/lnav-multi-file2.png diff --git a/docs/assets/images/lnav-query.png b/docs/assets/images/lnav-query.png Binary files differnew file mode 100644 index 0000000..470fdc6 --- /dev/null +++ b/docs/assets/images/lnav-query.png diff --git a/docs/assets/images/lnav-sql-error-msg.png b/docs/assets/images/lnav-sql-error-msg.png Binary files differnew file mode 100644 index 0000000..526f6c3 --- /dev/null +++ b/docs/assets/images/lnav-sql-error-msg.png diff --git a/docs/assets/images/lnav-syntax-highlight.gif b/docs/assets/images/lnav-syntax-highlight.gif Binary files differnew file mode 100644 index 0000000..6f0ce13 --- /dev/null +++ b/docs/assets/images/lnav-syntax-highlight.gif diff --git a/docs/assets/images/lnav-syslog-thumb.png b/docs/assets/images/lnav-syslog-thumb.png Binary files differnew file mode 100644 index 0000000..e1f0533 --- /dev/null +++ b/docs/assets/images/lnav-syslog-thumb.png diff --git a/docs/assets/images/lnav-syslog.png b/docs/assets/images/lnav-syslog.png Binary files differnew file mode 100644 index 0000000..b676931 --- /dev/null +++ b/docs/assets/images/lnav-syslog.png diff --git a/docs/assets/images/lnav-tab-complete.gif b/docs/assets/images/lnav-tab-complete.gif Binary files differnew file mode 100644 index 0000000..de49539 --- /dev/null +++ b/docs/assets/images/lnav-tab-complete.gif diff --git a/docs/assets/images/lnav-theme-cycle.gif b/docs/assets/images/lnav-theme-cycle.gif Binary files differnew file mode 100644 index 0000000..f21281f --- /dev/null +++ b/docs/assets/images/lnav-theme-cycle.gif diff --git a/docs/assets/images/lnav-vscode-extension.png b/docs/assets/images/lnav-vscode-extension.png Binary files differnew file mode 100644 index 0000000..f41760f --- /dev/null +++ b/docs/assets/images/lnav-vscode-extension.png diff --git a/docs/assets/images/scrollbar-change-2.png b/docs/assets/images/scrollbar-change-2.png Binary files differnew file mode 100644 index 0000000..c7671c1 --- /dev/null +++ b/docs/assets/images/scrollbar-change-2.png diff --git a/docs/assets/js/codeblock.js b/docs/assets/js/codeblock.js new file mode 100644 index 0000000..b41e902 --- /dev/null +++ b/docs/assets/js/codeblock.js @@ -0,0 +1,23 @@ +var codeBlocks = document.querySelectorAll('pre.highlight'); + +codeBlocks.forEach(function (codeBlock) { + var copyButton = document.createElement('button'); + copyButton.className = 'copy'; + copyButton.type = 'button'; + copyButton.ariaLabel = 'Copy code to clipboard'; + copyButton.innerText = 'Copy'; + + codeBlock.append(copyButton); + + copyButton.addEventListener('click', function () { + var code = codeBlock.querySelector('code').innerText.trim(); + window.navigator.clipboard.writeText(code); + + copyButton.innerText = 'Copied'; + var fourSeconds = 4000; + + setTimeout(function () { + copyButton.innerText = 'Copy'; + }, fourSeconds); + }); +}); diff --git a/docs/assets/main.scss b/docs/assets/main.scss new file mode 100644 index 0000000..24d9dbd --- /dev/null +++ b/docs/assets/main.scss @@ -0,0 +1,83 @@ +--- +--- + +@import "{{ site.theme }}"; + +#playground-box { + font-size: x-large; + border-radius: 25px; + background: #8d8; + padding: 20px; + display: inline-block; +} + +#playground-box h4 { + margin-bottom: 0; +} + +#playground-box code { + padding-left: 20px; + background: #444; + border-color: #444; +} + +#playground-box code a { + padding-right: 20px; + color: white; +} + +#playground-box .prompt { + color: #4f4; +} + + +pre.highlight { + padding: 8px 12px; + position: relative; + + // override skeleton styles + > code { + border: 0; + overflow-x: auto; + padding-right: 0; + padding-left: 0; + } + + &.highlight { + border-left: 15px solid #35383c; + color: #c1c2c3; + overflow: auto; + white-space: pre; + word-wrap: normal; + + &, + code { + background-color: #222; + font-size: 14px; + } + } + + // code to clipboard + .copy { + color: #4AF626; + position: absolute; + right: 1.2rem; + top: 1.2rem; + opacity: 0; + + &:active, + &:focus, + &:hover { + background: rgba(0, 0, 0, 0.7); + opacity: 1; + } + } + + &:active .copy, + &:focus .copy, + &:hover .copy { + background: rgba(0, 0, 0, 0.7); + opacity: 1; + } +} + diff --git a/docs/conf.py.in b/docs/conf.py.in new file mode 100644 index 0000000..b81e3d9 --- /dev/null +++ b/docs/conf.py.in @@ -0,0 +1,6 @@ +DOXYFILE = 'Doxyfile' + +LINKS_NAVBAR1 = [ + (None, 'pages', [(None, 'about')]), + (None, 'namespaces', []), +] diff --git a/docs/index.markdown b/docs/index.markdown new file mode 100644 index 0000000..eadb501 --- /dev/null +++ b/docs/index.markdown @@ -0,0 +1,58 @@ +--- + +# Feel free to add content and custom Front Matter to this file. + +# To modify the layout, see https://jekyllrb.com/docs/themes/#overriding-theme-defaults + +layout: top +--- + +{: +style="float: right; max-width: 50%"} + +<div id="top-description"> +An advanced log file viewer for the small-scale +</div> + +<div id="intro"> +<p>Watch and analyze your log files from a terminal.</p> + +<p>No server. No setup. Still featureful.</p> + +<div id="playground-box"> +<h4>Try it out:</h4> + +<code> +<span class="prompt">$</span> +<a href="ssh://playground@demo.lnav.org">ssh playground@demo.lnav.org</a> +</code> +</div> +</div> + +<div class="dlrow"> +<dl> +<dt>In Your Terminal</dt> +<dd> +Many logging tools, like Splunk, provide great features but are optimized for +large-scale deployments. They require installing and configuring servers +before they can be effectively used. There is still a need for a robust log +file analyzer for the terminal. +</dd> +</dl> + +<dl> +<dt>Easy to Use</dt> +<dd> +Just point lnav to a directory and it will take care of the rest. File formats +are automatically detected and compressed files are unpacked on the fly. +</dd> +</dl> + +<dl> +<dt>Improved Presentation</dt> +<dd> +Log files are a wealth of information, lnav can help highlight the parts that +are important and filter out the noise. +</dd> +</dl> +</div> diff --git a/docs/lnav-architecture.png b/docs/lnav-architecture.png Binary files differnew file mode 100644 index 0000000..dc9fae9 --- /dev/null +++ b/docs/lnav-architecture.png diff --git a/docs/lnav-tui.png b/docs/lnav-tui.png Binary files differnew file mode 100644 index 0000000..bcb87ca --- /dev/null +++ b/docs/lnav-tui.png diff --git a/docs/pages/about.dox b/docs/pages/about.dox new file mode 100644 index 0000000..2efbda9 --- /dev/null +++ b/docs/pages/about.dox @@ -0,0 +1,7 @@ +/** + * @page about About + * @section about-doxygen Doxygen documentation + * This page is auto generated using + * <a href="https://www.doxygen.nl/">Doxygen</a>, making use of some useful + * <a href="https://www.doxygen.nl/manual/commands.html">special commands</a>. + */ diff --git a/docs/requirements.txt b/docs/requirements.txt new file mode 100644 index 0000000..1d29e34 --- /dev/null +++ b/docs/requirements.txt @@ -0,0 +1,6 @@ +sphinx>=4.3.0 +sphinx-copybutton +sphinx-jsonschema +sphinx-prompt +pathlib +sphinx-rtd-theme>=0.5.1 diff --git a/docs/schemas/config-v1.schema.json b/docs/schemas/config-v1.schema.json new file mode 100644 index 0000000..e2285cc --- /dev/null +++ b/docs/schemas/config-v1.schema.json @@ -0,0 +1,804 @@ +{ + "$id": "https://lnav.org/schemas/config-v1.schema.json", + "title": "https://lnav.org/schemas/config-v1.schema.json", + "$schema": "http://json-schema.org/draft-07/schema#", + "properties": { + "$schema": { + "title": "/$schema", + "description": "The URI that specifies the schema that describes this type of file", + "type": "string", + "examples": [ + "https://lnav.org/schemas/config-v1.schema.json" + ] + }, + "tuning": { + "description": "Internal settings", + "title": "/tuning", + "type": "object", + "properties": { + "archive-manager": { + "description": "Settings related to opening archive files", + "title": "/tuning/archive-manager", + "type": "object", + "properties": { + "min-free-space": { + "title": "/tuning/archive-manager/min-free-space", + "description": "The minimum free space, in bytes, to maintain when unpacking archives", + "type": "integer", + "minimum": 0 + }, + "cache-ttl": { + "title": "/tuning/archive-manager/cache-ttl", + "description": "The time-to-live for unpacked archives, expressed as a duration (e.g. '3d' for three days)", + "type": "string", + "examples": [ + "3d", + "12h" + ] + } + }, + "additionalProperties": false + }, + "file-vtab": { + "description": "Settings related to the lnav_file virtual-table", + "title": "/tuning/file-vtab", + "type": "object", + "properties": { + "max-content-size": { + "title": "/tuning/file-vtab/max-content-size", + "description": "The maximum allowed file size for the content column", + "type": "integer", + "minimum": 0 + } + }, + "additionalProperties": false + }, + "logfile": { + "description": "Settings related to log files", + "title": "/tuning/logfile", + "type": "object", + "properties": { + "max-unrecognized-lines": { + "title": "/tuning/logfile/max-unrecognized-lines", + "description": "The maximum number of lines in a file to use when detecting the format", + "type": "integer", + "minimum": 1 + } + }, + "additionalProperties": false + }, + "remote": { + "description": "Settings related to remote file support", + "title": "/tuning/remote", + "type": "object", + "properties": { + "cache-ttl": { + "title": "/tuning/remote/cache-ttl", + "description": "The time-to-live for files copied from remote hosts, expressed as a duration (e.g. '3d' for three days)", + "type": "string", + "examples": [ + "3d", + "12h" + ] + }, + "ssh": { + "description": "Settings related to the ssh command used to contact remote machines", + "title": "/tuning/remote/ssh", + "type": "object", + "properties": { + "command": { + "title": "/tuning/remote/ssh/command", + "description": "The SSH command to execute", + "type": "string" + }, + "transfer-command": { + "title": "/tuning/remote/ssh/transfer-command", + "description": "Command executed on the remote host when transferring the file", + "type": "string" + }, + "start-command": { + "title": "/tuning/remote/ssh/start-command", + "description": "Command executed on the remote host to start the tailer", + "type": "string" + }, + "flags": { + "title": "/tuning/remote/ssh/flags", + "description": "The flags to pass to the SSH command", + "type": "string" + }, + "options": { + "description": "The options to pass to the SSH command", + "title": "/tuning/remote/ssh/options", + "type": "object", + "patternProperties": { + "(\\w+)": { + "title": "/tuning/remote/ssh/options/<option_name>", + "description": "Set an option to be passed to the SSH command", + "type": "string" + } + }, + "additionalProperties": false + }, + "config": { + "description": "The ssh_config options to pass to SSH with the -o option", + "title": "/tuning/remote/ssh/config", + "type": "object", + "patternProperties": { + "(\\w+)": { + "title": "/tuning/remote/ssh/config/<config_name>", + "description": "Set an SSH configuration value", + "type": "string" + } + }, + "additionalProperties": false + } + }, + "additionalProperties": false + } + }, + "additionalProperties": false + }, + "clipboard": { + "description": "Settings related to the clipboard", + "title": "/tuning/clipboard", + "type": "object", + "properties": { + "impls": { + "description": "Clipboard implementations", + "title": "/tuning/clipboard/impls", + "type": "object", + "patternProperties": { + "([\\w\\-]+)": { + "description": "Clipboard implementation", + "title": "/tuning/clipboard/impls/<clipboard_impl_name>", + "type": "object", + "properties": { + "test": { + "title": "/tuning/clipboard/impls/<clipboard_impl_name>/test", + "description": "The command that checks", + "type": "string", + "examples": [ + "command -v pbcopy" + ] + }, + "general": { + "description": "Commands to work with the general clipboard", + "title": "/tuning/clipboard/impls/<clipboard_impl_name>/general", + "$ref": "#/definitions/clip-commands" + }, + "find": { + "description": "Commands to work with the find clipboard", + "title": "/tuning/clipboard/impls/<clipboard_impl_name>/find", + "$ref": "#/definitions/clip-commands" + } + }, + "additionalProperties": false + } + }, + "additionalProperties": false + } + }, + "additionalProperties": false + } + }, + "additionalProperties": false + }, + "ui": { + "description": "User-interface settings", + "title": "/ui", + "type": "object", + "properties": { + "clock-format": { + "title": "/ui/clock-format", + "description": "The format for the clock displayed in the top-left corner using strftime(3) conversions", + "type": "string", + "examples": [ + "%a %b %d %H:%M:%S %Z" + ] + }, + "dim-text": { + "title": "/ui/dim-text", + "description": "Reduce the brightness of text (useful for xterms). This setting can be useful when running in an xterm where the white color is very bright.", + "type": "boolean" + }, + "default-colors": { + "title": "/ui/default-colors", + "description": "Use default terminal background and foreground colors instead of black and white for all text coloring. This setting can be useful when transparent background or alternate color theme terminal is used.", + "type": "boolean" + }, + "keymap": { + "title": "/ui/keymap", + "description": "The name of the keymap to use.", + "type": "string" + }, + "theme": { + "title": "/ui/theme", + "description": "The name of the theme to use.", + "type": "string" + }, + "theme-defs": { + "description": "Theme definitions.", + "title": "/ui/theme-defs", + "type": "object", + "patternProperties": { + "([\\w\\-]+)": { + "description": "Theme definitions", + "title": "/ui/theme-defs/<theme_name>", + "type": "object", + "properties": { + "vars": { + "description": "Variables definitions that are used in this theme.", + "title": "/ui/theme-defs/<theme_name>/vars", + "type": "object", + "patternProperties": { + "(\\w+)": { + "title": "/ui/theme-defs/<theme_name>/vars/<var_name>", + "description": "A theme variable definition", + "type": "string" + } + }, + "additionalProperties": false + }, + "styles": { + "description": "Styles for log messages.", + "title": "/ui/theme-defs/<theme_name>/styles", + "type": "object", + "properties": { + "identifier": { + "description": "Styling for identifiers in logs", + "title": "/ui/theme-defs/<theme_name>/styles/identifier", + "$ref": "#/definitions/style" + }, + "text": { + "description": "Styling for plain text", + "title": "/ui/theme-defs/<theme_name>/styles/text", + "$ref": "#/definitions/style" + }, + "alt-text": { + "description": "Styling for plain text when alternating", + "title": "/ui/theme-defs/<theme_name>/styles/alt-text", + "$ref": "#/definitions/style" + }, + "error": { + "description": "Styling for error messages", + "title": "/ui/theme-defs/<theme_name>/styles/error", + "$ref": "#/definitions/style" + }, + "ok": { + "description": "Styling for success messages", + "title": "/ui/theme-defs/<theme_name>/styles/ok", + "$ref": "#/definitions/style" + }, + "info": { + "description": "Styling for informational messages", + "title": "/ui/theme-defs/<theme_name>/styles/info", + "$ref": "#/definitions/style" + }, + "warning": { + "description": "Styling for warning messages", + "title": "/ui/theme-defs/<theme_name>/styles/warning", + "$ref": "#/definitions/style" + }, + "hidden": { + "description": "Styling for hidden fields in logs", + "title": "/ui/theme-defs/<theme_name>/styles/hidden", + "$ref": "#/definitions/style" + }, + "cursor-line": { + "description": "Styling for the cursor line in the main view", + "title": "/ui/theme-defs/<theme_name>/styles/cursor-line", + "$ref": "#/definitions/style" + }, + "adjusted-time": { + "description": "Styling for timestamps that have been adjusted", + "title": "/ui/theme-defs/<theme_name>/styles/adjusted-time", + "$ref": "#/definitions/style" + }, + "skewed-time": { + "description": "Styling for timestamps that are different from the received time", + "title": "/ui/theme-defs/<theme_name>/styles/skewed-time", + "$ref": "#/definitions/style" + }, + "offset-time": { + "description": "Styling for hidden fields", + "title": "/ui/theme-defs/<theme_name>/styles/offset-time", + "$ref": "#/definitions/style" + }, + "invalid-msg": { + "description": "Styling for invalid log messages", + "title": "/ui/theme-defs/<theme_name>/styles/invalid-msg", + "$ref": "#/definitions/style" + }, + "popup": { + "description": "Styling for popup windows", + "title": "/ui/theme-defs/<theme_name>/styles/popup", + "$ref": "#/definitions/style" + }, + "focused": { + "description": "Styling for a focused row in a list view", + "title": "/ui/theme-defs/<theme_name>/styles/focused", + "$ref": "#/definitions/style" + }, + "disabled-focused": { + "description": "Styling for a disabled focused row in a list view", + "title": "/ui/theme-defs/<theme_name>/styles/disabled-focused", + "$ref": "#/definitions/style" + }, + "scrollbar": { + "description": "Styling for scrollbars", + "title": "/ui/theme-defs/<theme_name>/styles/scrollbar", + "$ref": "#/definitions/style" + }, + "h1": { + "description": "Styling for top-level headers", + "title": "/ui/theme-defs/<theme_name>/styles/h1", + "$ref": "#/definitions/style" + }, + "h2": { + "description": "Styling for 2nd-level headers", + "title": "/ui/theme-defs/<theme_name>/styles/h2", + "$ref": "#/definitions/style" + }, + "h3": { + "description": "Styling for 3rd-level headers", + "title": "/ui/theme-defs/<theme_name>/styles/h3", + "$ref": "#/definitions/style" + }, + "h4": { + "description": "Styling for 4th-level headers", + "title": "/ui/theme-defs/<theme_name>/styles/h4", + "$ref": "#/definitions/style" + }, + "h5": { + "description": "Styling for 5th-level headers", + "title": "/ui/theme-defs/<theme_name>/styles/h5", + "$ref": "#/definitions/style" + }, + "h6": { + "description": "Styling for 6th-level headers", + "title": "/ui/theme-defs/<theme_name>/styles/h6", + "$ref": "#/definitions/style" + }, + "hr": { + "description": "Styling for horizontal rules", + "title": "/ui/theme-defs/<theme_name>/styles/hr", + "$ref": "#/definitions/style" + }, + "hyperlink": { + "description": "Styling for hyperlinks", + "title": "/ui/theme-defs/<theme_name>/styles/hyperlink", + "$ref": "#/definitions/style" + }, + "list-glyph": { + "description": "Styling for glyphs that prefix a list item", + "title": "/ui/theme-defs/<theme_name>/styles/list-glyph", + "$ref": "#/definitions/style" + }, + "breadcrumb": { + "description": "Styling for the separator between breadcrumbs", + "title": "/ui/theme-defs/<theme_name>/styles/breadcrumb", + "$ref": "#/definitions/style" + }, + "table-border": { + "description": "Styling for table borders", + "title": "/ui/theme-defs/<theme_name>/styles/table-border", + "$ref": "#/definitions/style" + }, + "table-header": { + "description": "Styling for table headers", + "title": "/ui/theme-defs/<theme_name>/styles/table-header", + "$ref": "#/definitions/style" + }, + "quote-border": { + "description": "Styling for quoted-block borders", + "title": "/ui/theme-defs/<theme_name>/styles/quote-border", + "$ref": "#/definitions/style" + }, + "quoted-text": { + "description": "Styling for quoted text blocks", + "title": "/ui/theme-defs/<theme_name>/styles/quoted-text", + "$ref": "#/definitions/style" + }, + "footnote-border": { + "description": "Styling for footnote borders", + "title": "/ui/theme-defs/<theme_name>/styles/footnote-border", + "$ref": "#/definitions/style" + }, + "footnote-text": { + "description": "Styling for footnote text", + "title": "/ui/theme-defs/<theme_name>/styles/footnote-text", + "$ref": "#/definitions/style" + }, + "snippet-border": { + "description": "Styling for snippet borders", + "title": "/ui/theme-defs/<theme_name>/styles/snippet-border", + "$ref": "#/definitions/style" + } + }, + "additionalProperties": false + }, + "syntax-styles": { + "description": "Styles for syntax highlighting in text files.", + "title": "/ui/theme-defs/<theme_name>/syntax-styles", + "type": "object", + "properties": { + "quoted-code": { + "description": "Styling for quoted code blocks", + "title": "/ui/theme-defs/<theme_name>/syntax-styles/quoted-code", + "$ref": "#/definitions/style" + }, + "code-border": { + "description": "Styling for quoted-code borders", + "title": "/ui/theme-defs/<theme_name>/syntax-styles/code-border", + "$ref": "#/definitions/style" + }, + "keyword": { + "description": "Styling for keywords in source files", + "title": "/ui/theme-defs/<theme_name>/syntax-styles/keyword", + "$ref": "#/definitions/style" + }, + "string": { + "description": "Styling for single/double-quoted strings in text", + "title": "/ui/theme-defs/<theme_name>/syntax-styles/string", + "$ref": "#/definitions/style" + }, + "comment": { + "description": "Styling for comments in source files", + "title": "/ui/theme-defs/<theme_name>/syntax-styles/comment", + "$ref": "#/definitions/style" + }, + "doc-directive": { + "description": "Styling for documentation directives in source files", + "title": "/ui/theme-defs/<theme_name>/syntax-styles/doc-directive", + "$ref": "#/definitions/style" + }, + "variable": { + "description": "Styling for variables in text", + "title": "/ui/theme-defs/<theme_name>/syntax-styles/variable", + "$ref": "#/definitions/style" + }, + "symbol": { + "description": "Styling for symbols in source files", + "title": "/ui/theme-defs/<theme_name>/syntax-styles/symbol", + "$ref": "#/definitions/style" + }, + "number": { + "description": "Styling for numbers in source files", + "title": "/ui/theme-defs/<theme_name>/syntax-styles/number", + "$ref": "#/definitions/style" + }, + "re-special": { + "description": "Styling for special characters in regular expressions", + "title": "/ui/theme-defs/<theme_name>/syntax-styles/re-special", + "$ref": "#/definitions/style" + }, + "re-repeat": { + "description": "Styling for repeats in regular expressions", + "title": "/ui/theme-defs/<theme_name>/syntax-styles/re-repeat", + "$ref": "#/definitions/style" + }, + "diff-delete": { + "description": "Styling for deleted lines in diffs", + "title": "/ui/theme-defs/<theme_name>/syntax-styles/diff-delete", + "$ref": "#/definitions/style" + }, + "diff-add": { + "description": "Styling for added lines in diffs", + "title": "/ui/theme-defs/<theme_name>/syntax-styles/diff-add", + "$ref": "#/definitions/style" + }, + "diff-section": { + "description": "Styling for diffs", + "title": "/ui/theme-defs/<theme_name>/syntax-styles/diff-section", + "$ref": "#/definitions/style" + }, + "spectrogram-low": { + "description": "Styling for the lower threshold values in the spectrogram view", + "title": "/ui/theme-defs/<theme_name>/syntax-styles/spectrogram-low", + "$ref": "#/definitions/style" + }, + "spectrogram-medium": { + "description": "Styling for the medium threshold values in the spectrogram view", + "title": "/ui/theme-defs/<theme_name>/syntax-styles/spectrogram-medium", + "$ref": "#/definitions/style" + }, + "spectrogram-high": { + "description": "Styling for the high threshold values in the spectrogram view", + "title": "/ui/theme-defs/<theme_name>/syntax-styles/spectrogram-high", + "$ref": "#/definitions/style" + }, + "file": { + "description": "Styling for file names in source files", + "title": "/ui/theme-defs/<theme_name>/syntax-styles/file", + "$ref": "#/definitions/style" + } + }, + "additionalProperties": false + }, + "status-styles": { + "description": "Styles for the user-interface components.", + "title": "/ui/theme-defs/<theme_name>/status-styles", + "type": "object", + "properties": { + "text": { + "description": "Styling for status bars", + "title": "/ui/theme-defs/<theme_name>/status-styles/text", + "$ref": "#/definitions/style" + }, + "warn": { + "description": "Styling for warnings in status bars", + "title": "/ui/theme-defs/<theme_name>/status-styles/warn", + "$ref": "#/definitions/style" + }, + "alert": { + "description": "Styling for alerts in status bars", + "title": "/ui/theme-defs/<theme_name>/status-styles/alert", + "$ref": "#/definitions/style" + }, + "active": { + "description": "Styling for activity in status bars", + "title": "/ui/theme-defs/<theme_name>/status-styles/active", + "$ref": "#/definitions/style" + }, + "inactive-alert": { + "description": "Styling for inactive alert status bars", + "title": "/ui/theme-defs/<theme_name>/status-styles/inactive-alert", + "$ref": "#/definitions/style" + }, + "inactive": { + "description": "Styling for inactive status bars", + "title": "/ui/theme-defs/<theme_name>/status-styles/inactive", + "$ref": "#/definitions/style" + }, + "title-hotkey": { + "description": "Styling for hotkey highlights in titles", + "title": "/ui/theme-defs/<theme_name>/status-styles/title-hotkey", + "$ref": "#/definitions/style" + }, + "title": { + "description": "Styling for title sections of status bars", + "title": "/ui/theme-defs/<theme_name>/status-styles/title", + "$ref": "#/definitions/style" + }, + "disabled-title": { + "description": "Styling for title sections of status bars", + "title": "/ui/theme-defs/<theme_name>/status-styles/disabled-title", + "$ref": "#/definitions/style" + }, + "subtitle": { + "description": "Styling for subtitle sections of status bars", + "title": "/ui/theme-defs/<theme_name>/status-styles/subtitle", + "$ref": "#/definitions/style" + }, + "info": { + "description": "Styling for informational messages in status bars", + "title": "/ui/theme-defs/<theme_name>/status-styles/info", + "$ref": "#/definitions/style" + }, + "hotkey": { + "description": "Styling for hotkey highlights of status bars", + "title": "/ui/theme-defs/<theme_name>/status-styles/hotkey", + "$ref": "#/definitions/style" + } + }, + "additionalProperties": false + }, + "log-level-styles": { + "description": "Styles for each log message level.", + "title": "/ui/theme-defs/<theme_name>/log-level-styles", + "type": "object", + "patternProperties": { + "(trace|debug5|debug4|debug3|debug2|debug|info|stats|notice|warning|error|critical|fatal|invalid)": { + "title": "/ui/theme-defs/<theme_name>/log-level-styles/<level>", + "$ref": "#/definitions/style" + } + }, + "additionalProperties": false + }, + "highlights": { + "description": "Styles for text highlights.", + "title": "/ui/theme-defs/<theme_name>/highlights", + "type": "object", + "patternProperties": { + "([\\w\\-]+)": { + "title": "/ui/theme-defs/<theme_name>/highlights/<highlight_name>", + "type": "object", + "properties": { + "pattern": { + "title": "/ui/theme-defs/<theme_name>/highlights/<highlight_name>/pattern", + "description": "The regular expression to highlight", + "type": "string" + }, + "style": { + "description": "The styling for the text that matches the associated pattern", + "title": "/ui/theme-defs/<theme_name>/highlights/<highlight_name>/style", + "$ref": "#/definitions/style" + } + }, + "additionalProperties": false + } + }, + "additionalProperties": false + } + }, + "additionalProperties": false + } + }, + "additionalProperties": false + }, + "movement": { + "description": "Log file cursor movement mode settings", + "title": "/ui/movement", + "type": "object", + "properties": { + "mode": { + "title": "/ui/movement/mode", + "description": "The mode of cursor movement to use.", + "type": "string", + "enum": [ + "top", + "cursor" + ], + "examples": [ + "top", + "cursor" + ] + } + }, + "additionalProperties": false + }, + "keymap-defs": { + "description": "Keymap definitions.", + "title": "/ui/keymap-defs", + "type": "object", + "patternProperties": { + "([\\w\\-]+)": { + "description": "The keymap definitions", + "title": "/ui/keymap-defs/<keymap_name>", + "type": "object", + "patternProperties": { + "((?:x[0-9a-f]{2})+)": { + "description": "Map of key codes to commands to execute. The field names are the keys to be mapped using as a hexadecimal representation of the UTF-8 encoding. Each byte of the UTF-8 should start with an 'x' followed by the hexadecimal representation of the byte.", + "title": "/ui/keymap-defs/<keymap_name>/<key_seq>", + "type": "object", + "properties": { + "command": { + "title": "/ui/keymap-defs/<keymap_name>/<key_seq>/command", + "description": "The command to execute for the given key sequence. Use a script to execute more complicated operations.", + "type": "string", + "pattern": "^[:|;].*", + "examples": [ + ":goto next hour" + ] + }, + "alt-msg": { + "title": "/ui/keymap-defs/<keymap_name>/<key_seq>/alt-msg", + "description": "The help message to display after the key is pressed.", + "type": "string" + } + }, + "additionalProperties": false + } + }, + "additionalProperties": false + } + }, + "additionalProperties": false + } + }, + "additionalProperties": false + }, + "log": { + "description": "Log message settings", + "title": "/log", + "type": "object", + "properties": { + "watch-expressions": { + "description": "Log message watch expressions", + "title": "/log/watch-expressions", + "type": "object", + "patternProperties": { + "([\\w\\-]+)": { + "description": "A log message watch expression", + "title": "/log/watch-expressions/<watch_name>", + "type": "object", + "properties": { + "expr": { + "title": "/log/watch-expressions/<watch_name>/expr", + "description": "The SQL expression to execute for each input line. If expression evaluates to true, a 'log message detected' event will be published.", + "type": "string" + }, + "enabled": { + "title": "/log/watch-expressions/<watch_name>/enabled", + "description": "Indicates whether or not this expression should be evaluated during log processing.", + "type": "boolean" + } + }, + "additionalProperties": false + } + }, + "additionalProperties": false + } + }, + "additionalProperties": false + }, + "global": { + "description": "Global variable definitions", + "title": "/global", + "type": "object", + "patternProperties": { + "(\\w+)": { + "title": "/global/<var_name>", + "description": "A global variable definition. Global variables can be referenced in scripts, SQL statements, or commands.", + "type": "string" + } + }, + "additionalProperties": false + } + }, + "additionalProperties": false, + "definitions": { + "clip-commands": { + "title": "clip-commands", + "description": "Container for the commands used to read from and write to the system clipboard", + "type": "object", + "$$target": "#/definitions/clip-commands", + "properties": { + "write": { + "title": "/write", + "description": "The command used to write to the clipboard", + "type": "string", + "examples": [ + "pbcopy" + ] + }, + "read": { + "title": "/read", + "description": "The command used to read from the clipboard", + "type": "string", + "examples": [ + "pbpaste" + ] + } + }, + "additionalProperties": false + }, + "style": { + "title": "style", + "type": "object", + "$$target": "#/definitions/style", + "properties": { + "color": { + "title": "/color", + "description": "The foreground color value for this style. The value can be the name of an xterm color, the hexadecimal value, or a theme variable reference.", + "type": "string", + "examples": [ + "#fff", + "Green", + "$black" + ] + }, + "background-color": { + "title": "/background-color", + "description": "The background color value for this style. The value can be the name of an xterm color, the hexadecimal value, or a theme variable reference.", + "type": "string", + "examples": [ + "#2d2a2e", + "Green" + ] + }, + "underline": { + "title": "/underline", + "description": "Indicates that the text should be underlined.", + "type": "boolean" + }, + "bold": { + "title": "/bold", + "description": "Indicates that the text should be bolded.", + "type": "boolean" + } + }, + "additionalProperties": false + } + } +} diff --git a/docs/schemas/event-file-format-detected-v1.schema.json b/docs/schemas/event-file-format-detected-v1.schema.json new file mode 100644 index 0000000..edb8531 --- /dev/null +++ b/docs/schemas/event-file-format-detected-v1.schema.json @@ -0,0 +1,26 @@ +{ + "$id": "https://lnav.org/event-file-format-detected-v1.schema.json", + "title": "https://lnav.org/event-file-format-detected-v1.schema.json", + "$schema": "http://json-schema.org/draft-07/schema#", + "description": "Event fired when a log format is detected for a file.", + "properties": { + "$schema": { + "title": "/$schema", + "type": "string", + "examples": [ + "https://lnav.org/event-file-format-detected-v1.schema.json" + ] + }, + "filename": { + "title": "/filename", + "description": "The path of the file for which a matching format was found", + "type": "string" + }, + "format": { + "title": "/format", + "description": "The name of the format", + "type": "string" + } + }, + "additionalProperties": false +} diff --git a/docs/schemas/event-file-open-v1.schema.json b/docs/schemas/event-file-open-v1.schema.json new file mode 100644 index 0000000..76a6bc1 --- /dev/null +++ b/docs/schemas/event-file-open-v1.schema.json @@ -0,0 +1,21 @@ +{ + "$id": "https://lnav.org/event-file-open-v1.schema.json", + "title": "https://lnav.org/event-file-open-v1.schema.json", + "$schema": "http://json-schema.org/draft-07/schema#", + "description": "Event fired when a file is opened.", + "properties": { + "$schema": { + "title": "/$schema", + "type": "string", + "examples": [ + "https://lnav.org/event-file-open-v1.schema.json" + ] + }, + "filename": { + "title": "/filename", + "description": "The path of the file that was opened", + "type": "string" + } + }, + "additionalProperties": false +} diff --git a/docs/schemas/event-log-msg-detected-v1.schema.json b/docs/schemas/event-log-msg-detected-v1.schema.json new file mode 100644 index 0000000..30ec3a2 --- /dev/null +++ b/docs/schemas/event-log-msg-detected-v1.schema.json @@ -0,0 +1,57 @@ +{ + "$id": "https://lnav.org/event-log-msg-detected-v1.schema.json", + "title": "https://lnav.org/event-log-msg-detected-v1.schema.json", + "$schema": "http://json-schema.org/draft-07/schema#", + "description": "Event fired when a log message is detected by a watch expression.", + "properties": { + "$schema": { + "title": "/$schema", + "type": "string", + "examples": [ + "https://lnav.org/event-log-msg-detected-v1.schema.json" + ] + }, + "watch-name": { + "title": "/watch-name", + "description": "The name of the watch expression that matched this log message", + "type": "string" + }, + "filename": { + "title": "/filename", + "description": "The path of the file containing the log message", + "type": "string" + }, + "line-number": { + "title": "/line-number", + "description": "The line number in the file, starting from zero", + "type": "integer" + }, + "format": { + "title": "/format", + "description": "The name of the log format that matched this log message", + "type": "string" + }, + "timestamp": { + "title": "/timestamp", + "description": "The timestamp of the log message", + "type": "string" + }, + "values": { + "description": "The log message values captured by the log format", + "title": "/values", + "type": "object", + "patternProperties": { + "([\\w\\-]+)": { + "title": "/values/<name>", + "type": [ + "boolean", + "integer", + "string" + ] + } + }, + "additionalProperties": false + } + }, + "additionalProperties": false +} diff --git a/docs/schemas/event-session-loaded-v1.schema.json b/docs/schemas/event-session-loaded-v1.schema.json new file mode 100644 index 0000000..f64f051 --- /dev/null +++ b/docs/schemas/event-session-loaded-v1.schema.json @@ -0,0 +1,16 @@ +{ + "$id": "https://lnav.org/event-session-loaded-v1.schema.json", + "title": "https://lnav.org/event-session-loaded-v1.schema.json", + "$schema": "http://json-schema.org/draft-07/schema#", + "description": "Event fired when a session is loaded.", + "properties": { + "$schema": { + "title": "/$schema", + "type": "string", + "examples": [ + "https://lnav.org/event-session-loaded-v1.schema.json" + ] + } + }, + "additionalProperties": false +} diff --git a/docs/schemas/format-v1.schema.json b/docs/schemas/format-v1.schema.json new file mode 100644 index 0000000..fe781f9 --- /dev/null +++ b/docs/schemas/format-v1.schema.json @@ -0,0 +1,638 @@ +{ + "$id": "https://lnav.org/schemas/format-v1.schema.json", + "title": "https://lnav.org/schemas/format-v1.schema.json", + "$schema": "http://json-schema.org/draft-07/schema#", + "properties": { + "$schema": { + "title": "/$schema", + "description": "Specifies the type of this file", + "type": "string" + } + }, + "patternProperties": { + "(\\w+)": { + "description": "The definition of a log file format.", + "title": "/<format_name>", + "type": "object", + "properties": { + "regex": { + "description": "The set of regular expressions used to match log messages", + "title": "/<format_name>/regex", + "type": "object", + "patternProperties": { + "([^/]+)": { + "description": "The set of patterns used to match log messages", + "title": "/<format_name>/regex/<pattern_name>", + "type": "object", + "properties": { + "pattern": { + "title": "/<format_name>/regex/<pattern_name>/pattern", + "description": "The regular expression to match a log message and capture fields.", + "type": "string", + "minLength": 1 + }, + "module-format": { + "title": "/<format_name>/regex/<pattern_name>/module-format", + "description": "If true, this pattern will only be used to parse message bodies of container formats, like syslog", + "type": "boolean" + } + }, + "additionalProperties": false + } + }, + "additionalProperties": false + }, + "json": { + "title": "/<format_name>/json", + "description": "Indicates that log files are JSON-encoded (deprecated, use \"file-type\": \"json\")", + "type": "boolean" + }, + "convert-to-local-time": { + "title": "/<format_name>/convert-to-local-time", + "description": "Indicates that displayed timestamps should automatically be converted to local time", + "type": "boolean" + }, + "hide-extra": { + "title": "/<format_name>/hide-extra", + "description": "Specifies whether extra values in JSON logs should be displayed", + "type": "boolean" + }, + "multiline": { + "title": "/<format_name>/multiline", + "description": "Indicates that log messages can span multiple lines", + "type": "boolean" + }, + "timestamp-divisor": { + "title": "/<format_name>/timestamp-divisor", + "description": "The value to divide a numeric timestamp by in a JSON log.", + "type": [ + "integer", + "number" + ] + }, + "file-pattern": { + "title": "/<format_name>/file-pattern", + "description": "A regular expression that restricts this format to log files with a matching name", + "type": "string" + }, + "mime-types": { + "title": "/<format_name>/mime-types", + "description": "A list of mime-types this format should be used for", + "type": "array", + "items": { + "type": "string", + "enum": [ + "application/vnd.tcpdump.pcap" + ] + } + }, + "level-field": { + "title": "/<format_name>/level-field", + "description": "The name of the level field in the log message pattern", + "type": "string" + }, + "level-pointer": { + "title": "/<format_name>/level-pointer", + "description": "A regular-expression that matches the JSON-pointer of the level property", + "type": "string" + }, + "timestamp-field": { + "title": "/<format_name>/timestamp-field", + "description": "The name of the timestamp field in the log message pattern", + "type": "string" + }, + "subsecond-field": { + "title": "/<format_name>/subsecond-field", + "description": "The path to the property in a JSON-lines log message that contains the sub-second time value", + "type": "string" + }, + "subsecond-units": { + "title": "/<format_name>/subsecond-units", + "description": "The units of the subsecond-field property value", + "type": "string", + "enum": [ + "milli", + "micro", + "nano" + ] + }, + "time-field": { + "title": "/<format_name>/time-field", + "description": "The name of the time field in the log message pattern. This field should only be specified if the timestamp field only contains a date.", + "type": "string" + }, + "body-field": { + "title": "/<format_name>/body-field", + "description": "The name of the body field in the log message pattern", + "type": "string" + }, + "url": { + "title": "/<format_name>/url", + "description": "A URL with more information about this log format", + "type": [ + "array", + "string" + ], + "items": { + "type": "string" + } + }, + "title": { + "title": "/<format_name>/title", + "description": "The human-readable name for this log format", + "type": "string" + }, + "description": { + "title": "/<format_name>/description", + "description": "A longer description of this log format", + "type": "string" + }, + "timestamp-format": { + "title": "/<format_name>/timestamp-format", + "description": "An array of strptime(3)-like timestamp formats", + "type": "array", + "items": { + "type": "string" + } + }, + "module-field": { + "title": "/<format_name>/module-field", + "description": "The name of the module field in the log message pattern", + "type": "string" + }, + "opid-field": { + "title": "/<format_name>/opid-field", + "description": "The name of the operation-id field in the log message pattern", + "type": "string" + }, + "ordered-by-time": { + "title": "/<format_name>/ordered-by-time", + "description": "Indicates that the order of messages in the file is time-based.", + "type": "boolean" + }, + "level": { + "description": "The map of level names to patterns or integer values", + "title": "/<format_name>/level", + "type": "object", + "patternProperties": { + "(trace|debug[2345]?|info|stats|notice|warning|error|critical|fatal)": { + "title": "/<format_name>/level/<level>", + "description": "The regular expression used to match the log text for this level. For JSON logs with numeric levels, this should be the number for the corresponding level.", + "type": [ + "integer", + "string" + ] + } + }, + "additionalProperties": false + }, + "value": { + "description": "The set of value definitions", + "title": "/<format_name>/value", + "type": "object", + "patternProperties": { + "([^/]+)": { + "description": "The set of values captured by the log message patterns", + "title": "/<format_name>/value/<value_name>", + "type": "object", + "properties": { + "kind": { + "title": "/<format_name>/value/<value_name>/kind", + "description": "The type of data in the field", + "type": "string", + "enum": [ + "string", + "integer", + "float", + "boolean", + "json", + "struct", + "quoted", + "xml" + ] + }, + "collate": { + "title": "/<format_name>/value/<value_name>/collate", + "description": "The collating function to use for this column", + "type": "string" + }, + "unit": { + "description": "Unit definitions for this field", + "title": "/<format_name>/value/<value_name>/unit", + "type": "object", + "properties": { + "field": { + "title": "/<format_name>/value/<value_name>/unit/field", + "description": "The name of the field that contains the units for this field", + "type": "string" + }, + "scaling-factor": { + "description": "Transforms the numeric value by the given factor", + "title": "/<format_name>/value/<value_name>/unit/scaling-factor", + "type": "object", + "patternProperties": { + "([^/]+)": { + "title": "/<format_name>/value/<value_name>/unit/scaling-factor/<scale>", + "type": "object", + "properties": { + "op": { + "title": "/<format_name>/value/<value_name>/unit/scaling-factor/<scale>/op", + "type": "string", + "enum": [ + "identity", + "multiply", + "divide" + ] + }, + "value": { + "title": "/<format_name>/value/<value_name>/unit/scaling-factor/<scale>/value", + "type": "number" + } + }, + "additionalProperties": false + } + }, + "additionalProperties": false + } + }, + "additionalProperties": false + }, + "identifier": { + "title": "/<format_name>/value/<value_name>/identifier", + "description": "Indicates whether or not this field contains an identifier that should be highlighted", + "type": "boolean" + }, + "foreign-key": { + "title": "/<format_name>/value/<value_name>/foreign-key", + "description": "Indicates whether or not this field should be treated as a foreign key for row in another table", + "type": "boolean" + }, + "hidden": { + "title": "/<format_name>/value/<value_name>/hidden", + "description": "Indicates whether or not this field should be hidden", + "type": "boolean" + }, + "action-list": { + "title": "/<format_name>/value/<value_name>/action-list", + "description": "Actions to execute when this field is clicked on", + "type": "array", + "items": { + "type": "string" + } + }, + "rewriter": { + "title": "/<format_name>/value/<value_name>/rewriter", + "description": "A command that will rewrite this field when pretty-printing", + "type": "string", + "examples": [ + ";SELECT :sc_status || ' (' || (SELECT message FROM http_status_codes WHERE status = :sc_status) || ') '" + ] + }, + "description": { + "title": "/<format_name>/value/<value_name>/description", + "description": "A description of the field", + "type": "string" + } + }, + "additionalProperties": false + } + }, + "additionalProperties": false + }, + "tags": { + "description": "The tags to automatically apply to log messages", + "title": "/<format_name>/tags", + "type": "object", + "patternProperties": { + "([\\w:;\\._\\-]+)": { + "description": "The name of the tag to apply", + "title": "/<format_name>/tags/<tag_name>", + "type": "object", + "properties": { + "paths": { + "description": "Restrict tagging to the given paths", + "title": "/<format_name>/tags/<tag_name>/paths", + "type": "array", + "items": { + "type": "object", + "properties": { + "glob": { + "title": "/<format_name>/tags/<tag_name>/paths/glob", + "description": "The glob to match against file paths", + "type": "string", + "examples": [ + "*/system.log*" + ] + } + }, + "additionalProperties": false + } + }, + "pattern": { + "title": "/<format_name>/tags/<tag_name>/pattern", + "description": "The regular expression to match against the body of the log message", + "type": "string", + "examples": [ + "\\w+ is down" + ] + }, + "description": { + "title": "/<format_name>/tags/<tag_name>/description", + "description": "A description of this tag", + "type": "string" + }, + "level": { + "title": "/<format_name>/tags/<tag_name>/level", + "description": "Constrain hits to log messages with this level", + "type": "string", + "enum": [ + "trace", + "debug5", + "debug4", + "debug3", + "debug2", + "debug", + "info", + "stats", + "notice", + "warning", + "error", + "critical", + "fatal" + ] + } + }, + "additionalProperties": false + } + }, + "additionalProperties": false + }, + "action": { + "title": "/<format_name>/action", + "type": "object", + "patternProperties": { + "(\\w+)": { + "title": "/<format_name>/action/<action_name>", + "type": [ + "string", + "object" + ], + "properties": { + "label": { + "title": "/<format_name>/action/<action_name>/label", + "type": "string" + }, + "capture-output": { + "title": "/<format_name>/action/<action_name>/capture-output", + "type": "boolean" + }, + "cmd": { + "title": "/<format_name>/action/<action_name>/cmd", + "type": "array", + "items": { + "type": "string" + } + } + }, + "additionalProperties": false + } + }, + "additionalProperties": false + }, + "sample": { + "description": "An array of sample log messages to be tested against the log message patterns", + "title": "/<format_name>/sample", + "type": "array", + "items": { + "type": "object", + "properties": { + "description": { + "title": "/<format_name>/sample/description", + "description": "A description of this sample.", + "type": "string" + }, + "line": { + "title": "/<format_name>/sample/line", + "description": "A sample log line that should match a pattern in this format.", + "type": "string" + }, + "level": { + "title": "/<format_name>/sample/level", + "description": "The expected level for this sample log line.", + "type": "string", + "enum": [ + "trace", + "debug5", + "debug4", + "debug3", + "debug2", + "debug", + "info", + "stats", + "notice", + "warning", + "error", + "critical", + "fatal" + ] + } + }, + "additionalProperties": false + } + }, + "line-format": { + "description": "The display format for JSON-encoded log messages", + "title": "/<format_name>/line-format", + "type": "array", + "items": { + "type": [ + "string", + "object" + ], + "properties": { + "field": { + "title": "/<format_name>/line-format/field", + "description": "The name of the field to substitute at this position", + "type": "string" + }, + "default-value": { + "title": "/<format_name>/line-format/default-value", + "description": "The default value for this position if the field is null", + "type": "string" + }, + "timestamp-format": { + "title": "/<format_name>/line-format/timestamp-format", + "description": "The strftime(3) format for this field", + "type": "string", + "minLength": 1 + }, + "min-width": { + "title": "/<format_name>/line-format/min-width", + "description": "The minimum width of the field", + "type": "integer", + "minimum": 0 + }, + "auto-width": { + "title": "/<format_name>/line-format/auto-width", + "description": "Automatically detect the necessary width of the field based on the observed values", + "type": "boolean" + }, + "max-width": { + "title": "/<format_name>/line-format/max-width", + "description": "The maximum width of the field", + "type": "integer", + "minimum": 0 + }, + "align": { + "title": "/<format_name>/line-format/align", + "description": "Align the text in the column to the left or right side", + "type": "string", + "enum": [ + "left", + "right" + ] + }, + "overflow": { + "title": "/<format_name>/line-format/overflow", + "description": "Overflow style", + "type": "string", + "enum": [ + "abbrev", + "truncate", + "dot-dot" + ] + }, + "text-transform": { + "title": "/<format_name>/line-format/text-transform", + "description": "Text transformation", + "type": "string", + "enum": [ + "none", + "uppercase", + "lowercase", + "capitalize" + ] + }, + "prefix": { + "title": "/<format_name>/line-format/prefix", + "description": "Text to prepend to the value", + "type": "string" + }, + "suffix": { + "title": "/<format_name>/line-format/suffix", + "description": "Text to append to the value", + "type": "string" + } + }, + "additionalProperties": false + } + }, + "search-table": { + "description": "Search tables to automatically define for this log format", + "title": "/<format_name>/search-table", + "type": "object", + "patternProperties": { + "(\\w+)": { + "description": "The set of search tables to be automatically defined", + "title": "/<format_name>/search-table/<table_name>", + "type": "object", + "properties": { + "pattern": { + "title": "/<format_name>/search-table/<table_name>/pattern", + "description": "The regular expression for this search table.", + "type": "string" + }, + "glob": { + "title": "/<format_name>/search-table/<table_name>/glob", + "description": "Glob pattern used to constrain hits to messages that match the given pattern.", + "type": "string" + }, + "level": { + "title": "/<format_name>/search-table/<table_name>/level", + "description": "Constrain hits to log messages with this level", + "type": "string", + "enum": [ + "trace", + "debug5", + "debug4", + "debug3", + "debug2", + "debug", + "info", + "stats", + "notice", + "warning", + "error", + "critical", + "fatal" + ] + } + }, + "additionalProperties": false + } + }, + "additionalProperties": false + }, + "highlights": { + "description": "The set of highlight definitions", + "title": "/<format_name>/highlights", + "type": "object", + "patternProperties": { + "([^/]+)": { + "description": "The definition of a highlight", + "title": "/<format_name>/highlights/<highlight_name>", + "type": "object", + "properties": { + "pattern": { + "title": "/<format_name>/highlights/<highlight_name>/pattern", + "description": "A regular expression to highlight in logs of this format.", + "type": "string" + }, + "color": { + "title": "/<format_name>/highlights/<highlight_name>/color", + "description": "The color to use when highlighting this pattern.", + "type": "string" + }, + "background-color": { + "title": "/<format_name>/highlights/<highlight_name>/background-color", + "description": "The background color to use when highlighting this pattern.", + "type": "string" + }, + "underline": { + "title": "/<format_name>/highlights/<highlight_name>/underline", + "description": "Highlight this pattern with an underline.", + "type": "boolean" + }, + "blink": { + "title": "/<format_name>/highlights/<highlight_name>/blink", + "description": "Highlight this pattern by blinking.", + "type": "boolean" + } + }, + "additionalProperties": false + } + }, + "additionalProperties": false + }, + "file-type": { + "title": "/<format_name>/file-type", + "description": "The type of file that contains the log messages", + "type": "string", + "enum": [ + "text", + "json", + "csv" + ] + }, + "max-unrecognized-lines": { + "title": "/<format_name>/max-unrecognized-lines", + "description": "The maximum number of lines in a file to use when detecting the format", + "type": "integer", + "minimum": 1 + } + }, + "additionalProperties": false + } + }, + "additionalProperties": false +} diff --git a/docs/source/_ext/__init__.py b/docs/source/_ext/__init__.py new file mode 100644 index 0000000..e69de29 --- /dev/null +++ b/docs/source/_ext/__init__.py diff --git a/docs/source/_ext/lnavlexer.py b/docs/source/_ext/lnavlexer.py new file mode 100644 index 0000000..2509a5f --- /dev/null +++ b/docs/source/_ext/lnavlexer.py @@ -0,0 +1,26 @@ + +__all__ = ['LnavCommandLexer'] + +import re + +from pygments.token import Whitespace, Text, Keyword, Literal +from pygments.lexers._mapping import LEXERS +from pygments.lexers.python import RegexLexer + +class LnavCommandLexer(RegexLexer): + name = 'lnav' + + flags = re.IGNORECASE + tokens = { + 'root': [ + (r'\s+', Whitespace), + (r':[\w\-]+', Keyword), + (r'\<[\w\-]+\>', Literal.String.Doc), + (r'.', Text), + ] + } + +def setup(app): + LEXERS['LnavCommandLexer'] = ( + '_ext.lnavlexer', 'lnav', ('lnav',), ('*.lnav',), ('text/lnav',)) + app.add_lexer('lnav', LnavCommandLexer) diff --git a/docs/source/_static/theme_overrides.css b/docs/source/_static/theme_overrides.css new file mode 100644 index 0000000..c25d308 --- /dev/null +++ b/docs/source/_static/theme_overrides.css @@ -0,0 +1,57 @@ +/* override table width restrictions */ +@media screen and (min-width: 767px) { + + .wy-table-responsive table td { + /* !important prevents the common CSS stylesheets from overriding + this as on RTD they are loaded after this stylesheet */ + white-space: normal !important; + } + +} + +th p { + margin-bottom: 0px; +} + +.wy-nav-content { + max-width: 900px; + overflow-x: auto; +} + +table.query-results p { + font-size: 0.9em !important; +} + +DL DT:target, :target > H2, :target > H3, span:target + H2, span:target + H3 { + border: 0 !important; + border-bottom: 1px solid #d3d381 !important; + background: #ffc !important; + /* padding: 1em !important;*/ +} + +DL DT { + border: 0 !important; + background: inherit !important; + border-bottom: 1px solid #c3c0ee !important; + /* display: block !important; */ +} + +kbd { + padding: 0.1em 0.6em; + border: 1px solid #ccc; + font-size: 11px; + font-family: Arial, Helvetica, sans-serif; + background-color: #f7f7f7; + color: #333; + -moz-box-shadow: 0 1px 0px rgba(0, 0, 0, 0.2), 0 0 0 2px #ffffff inset; + -webkit-box-shadow: 0 1px 0px rgba(0, 0, 0, 0.2), 0 0 0 2px #ffffff inset; + box-shadow: 0 1px 0px rgba(0, 0, 0, 0.2), 0 0 0 2px #ffffff inset; + -moz-border-radius: 3px; + -webkit-border-radius: 3px; + border-radius: 3px; + display: inline-block; + margin: 0 0.1em 0.1em 0.1em; + text-shadow: 0 1px 0 #fff; + line-height: 1.4; + white-space: nowrap; +} diff --git a/docs/source/cli.rst b/docs/source/cli.rst new file mode 100644 index 0000000..565e54f --- /dev/null +++ b/docs/source/cli.rst @@ -0,0 +1,168 @@ +.. _cli: + +Command Line Interface +====================== + +There are two command-line interfaces provided by **lnav**, one for viewing +files and one for managing **lnav**'s configuration. The file viewing mode is +the default and is all that most people will need. The management mode can +be useful for those that are developing log file formats and is activated by +passing the :option:`-m` option as the first argument. + +File Viewing Mode +----------------- + +The following options can be used when starting **lnav**. There are not +many flags because the majority of the functionality is accessed using +the :option:`-c` option to execute :ref:`commands<commands>` or +:ref:`SQL queries<sql-ext>`. + +Options +^^^^^^^ + +.. option:: -h + + Print these command-line options and exit. + +.. option:: -H + + Start lnav and switch to the help view. + +.. option:: -C + + Check the given files against the configuration, report any errors, and + exit. This option can be helpful for validating that a log format is + well-formed. + +.. option:: -c <command> + + Execute the given lnav command, SQL query, or lnav script. The + argument must be prefixed with the character used to enter the prompt + to distinguish between the different types (i.e. ':', ';', '|'). + This option can be given multiple times. + +.. option:: -f <path> + + Execute the given command file. This option can be given multiple times. + +.. option:: -I <path> + + Add a configuration directory. + +.. option:: -i + + Install the format files in the :file:`.lnav/formats/` directory. + Individual files will be installed in the :file:`installed` + directory and git repositories will be cloned with a directory + name based on their repository URI. + +.. option:: -u + + Update formats installed from git repositories. + +.. option:: -d <path> + + Write debug messages to the given file. + +.. option:: -n + + Run without the curses UI (headless mode). + +.. option:: -N + + Do not open the default syslog file if no files are given. + +.. option:: -r + + Recursively load files from the given base directories. + +.. option:: -t + + Prepend timestamps to the lines of data being read in on the standard input. + +.. option:: -w <path> + + Write the contents of the standard input to this file. + +.. option:: -V + + Print the version of lnav. + +.. option:: -q + + Do not print the log messages after executing all of the commands. + + +Management Mode (v0.11.0+) +-------------------------- + +The management CLI mode provides functionality for query **lnav**'s log +format definitions. + +Options +^^^^^^^ + +.. option:: -m + + Switch to management mode. This must be the first option passed on the + command-line. + +Subcommands +^^^^^^^^^^^ + +.. option:: regex101 import <regex101-url> <format-name> [<regex-name>] + + Convert a regex101.com entry into a skeleton log format file. + +.. option:: format <format-name> regex <regex-name> push + + Push a log format regular expression to regex101.com . + +.. option:: format <format-name> regex <regex-name> pull + + Pull changes to a regex that was previously pushed to regex101.com . + +Environment Variables +--------------------- + +.. envvar:: XDG_CONFIG_HOME + + If this variable is set, lnav will use this directory to store its + configuration in a sub-directory named :file:`lnav`. + +.. envvar:: HOME + + If :envvar:`XDG_CONFIG_HOME` is not set, lnav will use this directory + to store its configuration in a sub-directory named :file:`.lnav`. + +.. envvar:: APPDATA + + On Windows, lnav will use this directory instead of HOME + to store its configuration in a sub-directory named :file:`.lnav`. + +.. envvar:: TZ + + The timezone setting is used in some log formats to convert UTC timestamps + to the local timezone. + + +Examples +-------- + + To load and follow the system syslog file: + + .. prompt:: bash + + lnav + + To load all of the files in :file:`/var/log`: + + .. prompt:: bash + + lnav /var/log + + To watch the output of make with timestamps prepended: + + .. prompt:: bash + + make 2>&1 | lnav -t diff --git a/docs/source/commands.rst b/docs/source/commands.rst new file mode 100644 index 0000000..d83a07e --- /dev/null +++ b/docs/source/commands.rst @@ -0,0 +1,132 @@ +.. role:: lnavcmd(code) + :language: lnav + :class: highlight + +.. _commands: + +Commands +======== + +Commands provide access to some of the more advanced features in **lnav**, like +:ref:`filtering<filtering>` and +:ref:`"search tables"<search_tables>`. You can activate the command +prompt by pressing the :kbd:`:` key. At the prompt, you can start typing +in the desired command and/or double-tap :kbd:`TAB` to activate +auto-completion and show the available commands. To guide you in the usage of +the commands, a help window will appear above the command prompt with an +explanation of the command and its parameters (if it has any). For example, +the screenshot below shows the help for the :code:`:open` command: + +.. figure:: open-help.png + :align: center + :figwidth: 50% + + Screenshot of the online help for the :code:`:open` command. + +In addition to online help, many commands provide a preview of the effects that +the command will have. This preview will activate shortly after you have +finished typing, but before you have pressed :kbd:`Enter` to execute the +command. For example, the :code:`:open` command will show a preview of the +first few lines of the file given as its argument: + +.. figure:: open-preview.png + :align: center + + Screenshot of the preview shown for the :code:`:open` command. + +The :lnavcmd:`:filter-out pattern` command is another instance where the preview behavior +can help you craft the correct command-line. This command takes a PCRE2 regular +expression that specifies the log messages that should be filtered out of the +view. The preview for this command will highlight the portion of the log +messages that match the expression in red. Thus, you can be certain that the +regular expression is matching the log messages you are interested in before +committing the filter. The following screenshot shows an example of this +preview behavior for the string "launchd": + +.. figure:: filter-out-preview.png + :align: center + + Screenshot showing the preview for the :code:`:filter-out` command. + +Any errors detected during preview will be shown in the status bar right above +the command prompt. For example, an attempt to open an unknown file will show +an error message in the status bar, like so: + +.. figure:: open-error.png + :align: center + + Screenshot of the error shown when trying to open a non-existent file. + +.. tip:: + + Note that almost all commands support TAB-completion for their arguments. + So, if you are in doubt as to what to type for an argument, you can double- + tap the :kbd:`TAB` key to get suggestions. For example, the + TAB-completion for the :code:`filter-in` command will suggest words that are + currently displayed in the view. + +.. note:: The following commands can be disabled by setting the ``LNAVSECURE`` + environment variable before executing the **lnav** binary: + + - :code:`:open` + - :code:`:pipe-to` + - :code:`:pipe-line-to` + - :code:`:write-*-to` + + This makes it easier to run **lnav** in restricted environments without the + risk of privilege escalation. + +I/O Commands +------------ + +Anonymization +^^^^^^^^^^^^^ + +Anonymization is the process of removing identifying information from content +to make it safer for sharing with others. For example, an IP address can +often be used to uniquely identify an entity. Substituting all instances of +a particular IP with the same dummy value would remove the identifying data +without losing statistical accuracy. **lnav** has built-in support for +anonymization through the :code:`--anonymize` flag on the :code:`:write-*` +collection of commands. While the anonymization process should catch most + + :IPv4 Addresses: Are replaced with addresses in the :code:`10.0.0.0/8` range. + + :IPv6 Addresses: Are replaced with addresses in the :code:`2001:db8::/32` range. + + :URL User Names: Are replaced with a random animal name. + + :URL Passwords: Are replaced with a hash of the input password. + + :URL Hosts: Are replaced with a random name under the example.com domain. + + :URL Paths: Are recursively examined for substitution. + + :URL Query Strings: Are recursively examined for substitution. + + :URL Fragments: Are recursively examined for substitution. + + :Paths: Are recursively examined for substitution. + + :Credit Card Numbers: Are replaced with a 16 digit hash of the input number. + + :MAC Addresses: Are replaced with addresses in the :code:`00:00:5E:00:53:00` range. + + :Hex Dumps: Are replaced with a hash of the input replicated to the size of input. + + :Email User Names: Are replaced with a random animal name. + + :Email Host Names: Are replaced with a random name under the example.com domain. + + :Words: Are replaced with a random word with a matching case style. + + :Quoted Strings: Are recursively examined for substitution. + + :UUID: Are replaced with a hash of the input. + + :XML Attribute Values: Are recursively examined for substitution. + +Reference +--------- + +.. include:: ../../src/internals/cmd-ref.rst diff --git a/docs/source/conf.py b/docs/source/conf.py new file mode 100644 index 0000000..3a2d8f2 --- /dev/null +++ b/docs/source/conf.py @@ -0,0 +1,461 @@ +# -*- coding: utf-8 -*- +# +# lnav documentation build configuration file, created by +# sphinx-quickstart on Fri Jul 12 21:09:39 2013. +# +# This file is execfile()d with the current directory set to its containing dir. +# +# Note that not all possible configuration values are present in this +# autogenerated file. +# +# All configuration values have a default; values that are commented out +# serve to show the default. + +import sys, os + +# If extensions (or modules to document with autodoc) are in another directory, +# add these directories to sys.path here. If the directory is relative to the +# documentation root, use os.path.abspath to make it absolute, like shown here. +this_dir = os.path.abspath('.') +src_dir = os.path.join(this_dir, "..", "..", "src") +sys.path.insert(0, this_dir) +sys.path.insert(0, src_dir) + +import format2csv + +format2csv.main(["", + os.path.join(this_dir, "format-table.csv"), + os.path.join(src_dir, "formats")]) + +import re +from pygments.lexer import RegexLexer, words +from pygments.token import Punctuation, Whitespace, Text, Comment, Operator, \ + Keyword, Name, String, Literal, Number, Generic +from sphinx.highlighting import lexers + + +class CustSqliteLexer(RegexLexer): + name = 'custsqlite' + + flags = re.IGNORECASE + tokens = { + 'root': [ + (r'\s+', Text), + (r'--.*\n?', Comment.Single), + (r'#.*\n?', Comment.Single), + (r'/\*', Comment.Multiline, 'multiline-comments'), + (words(( + 'ABORT', + 'ACTION', + 'ADD', + 'AFTER', + 'ALL', + 'ALTER', + 'ALWAYS', + 'ANALYZE', + 'AND', + 'AS', + 'ASC', + 'ATTACH', + 'AUTOINCREMENT', + 'BEFORE', + 'BEGIN', + 'BETWEEN', + 'BY', + 'CASCADE', + 'CASE', + 'CAST', + 'CHECK', + 'COLLATE', + 'COLUMN', + 'COMMIT', + 'CONFLICT', + 'CONSTRAINT', + 'CREATE', + 'CROSS', + 'CURRENT', + 'CURRENT_DATE', + 'CURRENT_TIME', + 'CURRENT_TIMESTAMP', + 'DATABASE', + 'DEFAULT', + 'DEFERRABLE', + 'DEFERRED', + 'DELETE', + 'DESC', + 'DETACH', + 'DISTINCT', + 'DO', + 'DROP', + 'EACH', + 'ELSE', + 'END', + 'ESCAPE', + 'EXCEPT', + 'EXCLUDE', + 'EXCLUSIVE', + 'EXISTS', + 'EXPLAIN', + 'FAIL', + 'FILTER', + 'FIRST', + 'FOLLOWING', + 'FOR', + 'FOREIGN', + 'FROM', + 'FULL', + 'GENERATED', + 'GLOB', + 'GROUP', + 'GROUPS', + 'HAVING', + 'IF', + 'IGNORE', + 'IMMEDIATE', + 'IN', + 'INDEX', + 'INDEXED', + 'INITIALLY', + 'INNER', + 'INSERT', + 'INSTEAD', + 'INTERSECT', + 'INTO', + 'IS', + 'ISNULL', + 'JOIN', + 'KEY', + 'LAST', + 'LEFT', + 'LIKE', + 'LIMIT', + 'MATCH', + 'NATURAL', + 'NO', + 'NOT', + 'NOTHING', + 'NOTNULL', + 'NULL', + 'NULLS', + 'OF', + 'OFFSET', + 'ON', + 'OR', + 'ORDER', + 'OTHERS', + 'OUTER', + 'OVER', + 'PARTITION', + 'PLAN', + 'PRAGMA', + 'PRECEDING', + 'PRIMARY', + 'QUERY', + 'RAISE', + 'RANGE', + 'RECURSIVE', + 'REFERENCES', + 'REGEXP', + 'REINDEX', + 'RELEASE', + 'RENAME', + 'REPLACE', + 'RESTRICT', + 'RIGHT', + 'ROLLBACK', + 'ROW', + 'ROWS', + 'SAVEPOINT', + 'SELECT', + 'SET', + 'TABLE', + 'TEMP', + 'TEMPORARY', + 'THEN', + 'TIES', + 'TO', + 'TRANSACTION', + 'TRIGGER', + 'UNBOUNDED', + 'UNION', + 'UNIQUE', + 'UPDATE', + 'USING', + 'VACUUM', + 'VALUES', + 'VIEW', + 'VIRTUAL', + 'WHEN', + 'WHERE', + 'WINDOW', + 'WITH', + 'WITHOUT'), suffix=r'\b'), + Keyword), + (words(( + 'ARRAY', 'BIGINT', 'BINARY', 'BIT', 'BLOB', 'BOOLEAN', 'CHAR', + 'CHARACTER', 'DATE', 'DEC', 'DECIMAL', 'FLOAT', 'INT', 'INTEGER', + 'INTERVAL', 'NUMBER', 'NUMERIC', 'REAL', 'SERIAL', 'SMALLINT', + 'VARCHAR', 'VARYING', 'INT8', 'SERIAL8', 'TEXT'), suffix=r'\b'), + Name.Builtin), + (r'[+*/<>=~!@#%^&|`?-]', Operator), + (r'[0-9]+', Number.Integer), + # TODO: Backslash escapes? + (r"'(''|[^'])*'", String.Single), + (r'"(""|[^"])*"', String.Symbol), # not a real string literal in ANSI SQL + (r'[a-z_][\w$]*', Name), # allow $s in strings for Oracle + (r'\$[a-z_]+', Name), + (r'[;:()\[\],.]', Punctuation) + ], + 'multiline-comments': [ + (r'/\*', Comment.Multiline, 'multiline-comments'), + (r'\*/', Comment.Multiline, '#pop'), + (r'[^/*]+', Comment.Multiline), + (r'[/*]', Comment.Multiline) + ] + } + + def analyse_text(text): + return 0.01 + + +lexers['custsqlite'] = CustSqliteLexer(startinline=True) + +# -- General configuration ----------------------------------------------------- + +# If your documentation needs a minimal Sphinx version, state it here. +# needs_sphinx = '1.0' + +# Add any Sphinx extension module names here, as strings. They can be extensions +# coming with Sphinx (named 'sphinx.ext.*') or your custom ones. +extensions = [ + 'sphinx.ext.coverage', + "sphinx_rtd_theme", + 'sphinx_copybutton', + 'sphinx-jsonschema', + 'sphinx-prompt', + '_ext.lnavlexer', +] + +# Add any paths that contain templates here, relative to this directory. +templates_path = ['_templates'] + +# The suffix of source filenames. +source_suffix = '.rst' + +# The encoding of source files. +# source_encoding = 'utf-8-sig' + +# The master toctree document. +master_doc = 'index' + +# General information about the project. +project = u'lnav' +copyright = u'2023, Tim Stack' + +# The version info for the project you're documenting, acts as replacement for +# |version| and |release|, also used in various other places throughout the +# built documents. +# +# The short X.Y version. +version = '0.11' +# The full version, including alpha/beta/rc tags. +release = '0.11.1' + +# The language for content autogenerated by Sphinx. Refer to documentation +# for a list of supported languages. +# language = None + +# There are two options for replacing |today|: either, you set today to some +# non-false value, then it is used: +# today = '' +# Else, today_fmt is used as the format for a strftime call. +# today_fmt = '%B %d, %Y' + +# List of patterns, relative to source directory, that match files and +# directories to ignore when looking for source files. +exclude_patterns = [] + +# The reST default role (used for this markup: `text`) to use for all documents. +# default_role = None + +# If true, '()' will be appended to :func: etc. cross-reference text. +# add_function_parentheses = True + +# If true, the current module name will be prepended to all description +# unit titles (such as .. function::). +# add_module_names = True + +# If true, sectionauthor and moduleauthor directives will be shown in the +# output. They are ignored by default. +# show_authors = False + +# The name of the Pygments (syntax highlighting) style to use. +pygments_style = 'sphinx' + +# A list of ignored prefixes for module index sorting. +# modindex_common_prefix = [] + +# -- Options for HTML output --------------------------------------------------- + +# The theme to use for HTML and HTML Help pages. See the documentation for +# a list of builtin themes. +html_theme = 'sphinx_rtd_theme' + +# Theme options are theme-specific and customize the look and feel of a theme +# further. For a list of options available for each theme, see the +# documentation. +html_theme_options = { +} + +# Add any paths that contain custom themes here, relative to this directory. +# html_theme_path = [] + +# The name for this set of Sphinx documents. If None, it defaults to +# "<project> v<release> documentation". +# html_title = None + +# A shorter title for the navigation bar. Default is the same as html_title. +# html_short_title = None + +# The name of an image file (relative to this directory) to place at the top +# of the sidebar. +# html_logo = None + +# The name of an image file (within the static path) to use as favicon of the +# docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32 +# pixels large. +# html_favicon = None + +# Add any paths that contain custom static files (such as style sheets) here, +# relative to this directory. They are copied after the builtin static files, +# so a file named "default.css" will overwrite the builtin "default.css". +html_static_path = ['_static'] + + +def setup(app): + app.add_css_file('theme_overrides.css') + + +# html_context = { +# 'css_files': [ +# '_static/theme_overrides.css', # override wide tables in RTD theme +# ], +# } + +# If not '', a 'Last updated on:' timestamp is inserted at every page bottom, +# using the given strftime format. +# html_last_updated_fmt = '%b %d, %Y' + +# If true, SmartyPants will be used to convert quotes and dashes to +# typographically correct entities. +# html_use_smartypants = True + +# Custom sidebar templates, maps document names to template names. +# html_sidebars = {} + +# Additional templates that should be rendered to pages, maps page names to +# template names. +# html_additional_pages = {} + +# If false, no module index is generated. +# html_domain_indices = True + +# If false, no index is generated. +# html_use_index = True + +# If true, the index is split into individual pages for each letter. +# html_split_index = False + +# If true, links to the reST sources are added to the pages. +# html_show_sourcelink = True + +# If true, "Created using Sphinx" is shown in the HTML footer. Default is True. +# html_show_sphinx = True + +# If true, "(C) Copyright ..." is shown in the HTML footer. Default is True. +# html_show_copyright = True + +# If true, an OpenSearch description file will be output, and all pages will +# contain a <link> tag referring to it. The value of this option must be the +# base URL from which the finished HTML is served. +# html_use_opensearch = '' + +# This is the file name suffix for HTML files (e.g. ".xhtml"). +# html_file_suffix = None + +# Output file base name for HTML help builder. +htmlhelp_basename = 'lnavdoc' + +# -- Options for LaTeX output -------------------------------------------------- + +latex_elements = { + # The paper size ('letterpaper' or 'a4paper'). + # 'papersize': 'letterpaper', + + # The font size ('10pt', '11pt' or '12pt'). + # 'pointsize': '10pt', + + # Additional stuff for the LaTeX preamble. + # 'preamble': '', +} + +# Grouping the document tree into LaTeX files. List of tuples +# (source start file, target name, title, author, documentclass [howto/manual]). +latex_documents = [ + ('index', 'lnav.tex', u'lnav Documentation', + u'Tim Stack', 'manual'), +] + +# The name of an image file (relative to this directory) to place at the top of +# the title page. +# latex_logo = None + +# For "manual" documents, if this is true, then toplevel headings are parts, +# not chapters. +# latex_use_parts = False + +# If true, show page references after internal links. +# latex_show_pagerefs = False + +# If true, show URL addresses after external links. +# latex_show_urls = False + +# Documents to append as an appendix to all manuals. +# latex_appendices = [] + +# If false, no module index is generated. +# latex_domain_indices = True + + +# -- Options for manual page output -------------------------------------------- + +# One entry per manual page. List of tuples +# (source start file, name, description, authors, manual section). +man_pages = [ + ('index', 'lnav', u'lnav Documentation', + [u'Tim Stack'], 1) +] + +# If true, show URL addresses after external links. +# man_show_urls = False + + +# -- Options for Texinfo output ------------------------------------------------ + +# Grouping the document tree into Texinfo files. List of tuples +# (source start file, target name, title, author, +# dir menu entry, description, category) +texinfo_documents = [ + ('index', 'lnav', u'lnav Documentation', + u'Tim Stack', 'lnav', 'One line description of project.', + 'Miscellaneous'), +] + +# Documents to append as an appendix to all manuals. +# texinfo_appendices = [] + +# If false, no module index is generated. +# texinfo_domain_indices = True + +# How to display URL addresses: 'footnote', 'no', or 'inline'. +# texinfo_show_urls = 'footnote' diff --git a/docs/source/config.rst b/docs/source/config.rst new file mode 100644 index 0000000..5294e52 --- /dev/null +++ b/docs/source/config.rst @@ -0,0 +1,270 @@ + +.. _Configuration: + +Configuration +============= + +The configuration for **lnav** is stored in the following JSON files in +:file:`~/.lnav`: + +* :file:`config.json` -- Contains local customizations that were done using the + :code:`:config` command. +* :file:`configs/default/*.json` -- The default configuration files that are + built into lnav are written to this directory with :file:`.sample` appended. + Removing the :file:`.sample` extension and editing the file will allow you to + do basic customizations. +* :file:`configs/installed/*.json` -- Contains configuration files installed + using the :option:`-i` flag (e.g. :code:`$ lnav -i /path/to/config.json`). +* :file:`configs/*/*.json` -- Other directories that contain :file:`*.json` + files will be loaded on startup. This structure is convenient for installing + **lnav** configurations, like from a git repository. + +A valid **lnav** configuration file must contain an object with the +:code:`$schema` property, like so: + +.. code-block:: json + + { + "$schema": "https://lnav.org/schemas/config-v1.schema.json" + } + +.. note:: + + Log format definitions are stored separately in the :file:`~/.lnav/formats` + directly. See the :ref:`Log Formats<log_formats>` chapter for more + information. + + +Options +------- + +The following configuration options can be used to customize the **lnav** UI to +your liking. The options can be changed using the :code:`:config` command. + +.. jsonschema:: ../schemas/config-v1.schema.json#/properties/ui/properties/keymap + +.. jsonschema:: ../schemas/config-v1.schema.json#/properties/ui/properties/theme + +.. jsonschema:: ../schemas/config-v1.schema.json#/properties/ui/properties/clock-format + +.. jsonschema:: ../schemas/config-v1.schema.json#/properties/ui/properties/dim-text + +.. jsonschema:: ../schemas/config-v1.schema.json#/properties/ui/properties/default-colors + + +.. _themes: + +Theme Definitions +----------------- + +User Interface themes are defined in a JSON configuration file. A theme is +made up of the style definitions for different types of text in the UI. A +:ref:`definition<theme_style>` can include the foreground/background colors +and the bold/underline attributes. The style definitions are broken up into +multiple categories for the sake of organization. To make it easier to write +a definition, a theme can define variables that can be referenced as color +values. + +Variables +^^^^^^^^^ + +The :code:`vars` object in a theme definition contains the mapping of variable +names to color values. These variables can be referenced in style definitions +by prefixing them with a dollar-sign (e.g. :code:`$black`). The following +variables can also be defined to control the values of the ANSI colors that +are log messages or plain text: + +.. csv-table:: ANSI colors + :header: "Variable Name", "ANSI Escape" + + "black", "ESC[30m" + "red", "ESC[31m" + "green", "ESC[32m" + "yellow", "ESC[33m" + "blue", "ESC[34m" + "magenta", "ESC[35m" + "cyan", "ESC[36m" + "white", "ESC[37m" + +Specifying Colors +^^^^^^^^^^^^^^^^^ + +Colors can be specified using hexadecimal notation by starting with a hash +(e.g. :code:`#aabbcc`) or using a color name as found at +http://jonasjacek.github.io/colors/. If colors are not specified for a style, +the values from the :code:`styles/text` definition. + +.. note:: + + When specifying colors in hexadecimal notation, you do not need to have an + exact match in the XTerm 256 color palette. A best approximation will be + picked based on the `CIEDE2000 <https://en.wikipedia.org/wiki/Color_difference#CIEDE2000>`_ + color difference algorithm. + + + +Example +^^^^^^^ + +The following example sets the black/background color for text to a dark grey +using a variable and sets the foreground to an off-white. This theme is +incomplete, but it works enough to give you an idea of how a theme is defined. +You can copy the code block, save it to a file in +:file:`~/.lnav/configs/installed/` and then activate it by executing +:code:`:config /ui/theme example` in lnav. For a more complete theme +definition, see one of the definitions built into **lnav**, like +`monocai <https://github.com/tstack/lnav/blob/master/src/themes/monocai.json>`_. + + .. code-block:: json + + { + "$schema": "https://lnav.org/schemas/config-v1.schema.json", + "ui": { + "theme-defs": { + "example1": { + "vars": { + "black": "#2d2a2e" + }, + "styles": { + "text": { + "color": "#f6f6f6", + "background-color": "$black" + } + } + } + } + } + } + +Reference +^^^^^^^^^ + +.. jsonschema:: ../schemas/config-v1.schema.json#/properties/ui/properties/theme-defs/patternProperties/([\w\-]+)/properties/vars + +.. jsonschema:: ../schemas/config-v1.schema.json#/properties/ui/properties/theme-defs/patternProperties/([\w\-]+)/properties/styles + +.. jsonschema:: ../schemas/config-v1.schema.json#/properties/ui/properties/theme-defs/patternProperties/([\w\-]+)/properties/syntax-styles + +.. jsonschema:: ../schemas/config-v1.schema.json#/properties/ui/properties/theme-defs/patternProperties/([\w\-]+)/properties/status-styles + +.. jsonschema:: ../schemas/config-v1.schema.json#/properties/ui/properties/theme-defs/patternProperties/([\w\-]+)/properties/log-level-styles + +.. _theme_style: + +.. jsonschema:: ../schemas/config-v1.schema.json#/definitions/style + + +.. _keymaps: + +Keymap Definitions +------------------ + +Keymaps in **lnav** map a key sequence to a command to execute. When a key is +pressed, it is converted into a hex-encoded string that is looked up in the +keymap. The :code:`command` value associated with the entry in the keymap is +then executed. Note that the "command" can be an **lnav** +:ref:`command<commands>`, a :ref:`SQL statement/query<sql-ext>`, or an +**lnav** script. If an :code:`alt-msg` value is included in the entry, the +bottom-right section of the UI will be updated with the help text. + +.. note:: + + Not all functionality is available via commands or SQL at the moment. Also, + some hotkeys are not implemented via keymaps. + +Key Sequence Encoding +^^^^^^^^^^^^^^^^^^^^^ + +Key presses are converted into a hex-encoded string that is used to lookup an +entry in the keymap. Each byte of the keypress value is formatted as an +:code:`x` followed by the hex-encoding in lowercase. For example, the encoding +for the £ key would be :code:`xc2xa3`. To make it easier to discover the +encoding for unassigned keys, **lnav** will print in the command prompt the +:code:`:config` command and +`JSON-Pointer <https://tools.ietf.org/html/rfc6901>`_ for assigning a command +to the key. + +.. figure:: key-encoding-prompt.png + :align: center + + Screenshot of the command prompt when an unassigned key is pressed. + +.. note:: + + Since **lnav** is a terminal application, it can only receive keypresses that + can be represented as characters or escape sequences. For example, it cannot + handle the press of a modifier key. + +Reference +^^^^^^^^^ + +.. jsonschema:: ../schemas/config-v1.schema.json#/properties/ui/properties/keymap-defs/patternProperties/([\w\-]+) + + +Log Handling +------------ + +The handling of logs is largely determined by the +:ref:`log file formats<log_formats>`, this section covers options that are not +specific to a particular format. + +Watch Expressions (v0.11.0+) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Watch expressions can be used to fire an event when a log message matches a +condition. You can then install a listener for these events and trigger an +action to be performed. For example, to automate filtering based on +identifiers, a watch expression can match messages that mention the ID and then +a trigger can install a filter for that ID. Creating a watch expression is +done by adding an entry into the :code:`/log/watch-expressions` configuration +tree. For example, to create a watch named "dhcpdiscover" that matches +DHCPDISCOVER messages from the :code:`dhclient` daemon, you would run the +following: + +.. code-block:: lnav + + :config /log/watch-expressions/dhcpdiscover/expr :log_procname = 'dhclient' AND startswith(:log_body, 'DHCPDISCOVER') + +The watch expression can refer to column names in the log message by prefixing +them with a colon. The expression is evaluated by passing the log message +fields as bound parameters and not against a table. The easiest way to test +out an expression is with the :ref:`mark_expr` command, since it will behave +similarly. After changing the configuration, you'll need to restart lnav +for the effect to take place. You can then query the :code:`lnav_events` +table to see any generated +:code:`https://lnav.org/event-log-msg-detected-v1.schema.json` events from the +logs that were loaded: + +.. code-block:: custsqlite + + ;SELECT * FROM lnav_events + +From there, you can create a SQLite trigger on the :code:`lnav_events` table +that will examine the event contents and perform an action. See the +:ref:`Events` section for more information on handling events. + +Reference +^^^^^^^^^ + +.. jsonschema:: ../schemas/config-v1.schema.json#/properties/log/properties/watch-expressions/patternProperties/([\w\-]+) + +.. _tuning: + +Tuning +------ + +The following configuration options can be used to tune the internals of +**lnav** to your liking. The options can be changed using the :code:`:config` +command. + +.. jsonschema:: ../schemas/config-v1.schema.json#/properties/tuning/properties/archive-manager + +.. jsonschema:: ../schemas/config-v1.schema.json#/properties/tuning/properties/clipboard + +.. jsonschema:: ../schemas/config-v1.schema.json#/definitions/clip-commands + +.. jsonschema:: ../schemas/config-v1.schema.json#/properties/tuning/properties/file-vtab + +.. jsonschema:: ../schemas/config-v1.schema.json#/properties/tuning/properties/logfile + +.. jsonschema:: ../schemas/config-v1.schema.json#/properties/tuning/properties/remote/properties/ssh diff --git a/docs/source/cookbook.rst b/docs/source/cookbook.rst new file mode 100644 index 0000000..3eb0ffd --- /dev/null +++ b/docs/source/cookbook.rst @@ -0,0 +1,104 @@ + +.. _Cookbook: + +Cookbook +======== + +This chapter contains recipes for common tasks that can be done in **lnav**. +These recipes can be used as a starting point for your own needs after some +adaptation. + + +Log Formats +----------- + +TBD + +Defining a New Format +^^^^^^^^^^^^^^^^^^^^^ + +TBD + + +Annotating Logs +--------------- + +Log messages can be annotated in a couple of different ways in **lnav** to help +you get organized. + +Create partitions for Linux boots +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +When digging through logs that can be broken up into multiple sections, +**lnav**'s :ref:`partitioning feature<taking_notes>` can be used to keep track +of which section you are in. For example, if a collection of Linux logs +covered multiple boots, the following script could be used to create partitions +for each boot. After the partition name is set for the log messages, the +current name will show up in the top status bar next to the current time. + +.. literalinclude:: ../../src/scripts/partition-by-boot.lnav + :language: custsqlite + :caption: partition-by-boot.lnav + :linenos: + +Tagging SSH log messages +^^^^^^^^^^^^^^^^^^^^^^^^ + +Log messages can be tagged interactively with the :ref:`:tag<tag>` command or +programmatically using the :ref:`sql-ext`. This example uses a script to +search for interesting SSH messages and automatically adds an appropriate tag. + +.. literalinclude:: ../../example-scripts/tag-ssh-msgs.lnav + :language: custsqlite + :caption: tag-ssh-msgs.lnav + :linenos: + +Log Analysis +------------ + +Most log analysis within **lnav** is done through the :ref:`sql-ext`. The +following examples should give you some ideas to start leveraging this +functionality. One thing to keep in mind is that if a query gets to be too +large or multiple statements need to be executed, you can create a +:code:`.lnav` script that contains the statements and execute it using the +:kbd:`\|` command prompt. + +Count client IPs in web access logs +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +To count the occurrences of an IP in web access logs and order the results +from highest to lowest: + + .. code-block:: custsqlite + + ;SELECT c_ip, count(*) as hits FROM access_log GROUP BY c_ip ORDER BY hits DESC + + +Show only lines where a numeric field is in a range +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The :ref:`:filter-expr<filter_expr>` command can be used to filter web access +logs to only show lines where the number of bytes transferred to the client is +between 10,000 and 40,000 bytes like so: + + .. code-block:: custsqlite + + :filter-expr :sc_bytes BETWEEN 10000 AND 40000 + + +Generating a Report +^^^^^^^^^^^^^^^^^^^ + +Reports can be generated by writing an **lnav** :ref:`script<scripts>` that +uses SQL queries and commands to format a document. A basic script can simply +execute a SQL query that is shown in the DB view. More sophisticated scripts +can use the following commands to generate customized output for a report: + +* The :ref:`:echo<echo>` command to write plain text +* :ref:`SQL queries<sql-ext>` followed by a "write" command, like + :ref:`:write-table-to<write_table_to>`. + +.. literalinclude:: ../../example-scripts/report-demo.lnav + :language: custsqlite + :caption: report-demo.lnav + :linenos: diff --git a/docs/source/data.rst b/docs/source/data.rst new file mode 100644 index 0000000..a7352e1 --- /dev/null +++ b/docs/source/data.rst @@ -0,0 +1,193 @@ + +.. _data-ext: + +Extracting Data +=============== + +**Note**: This feature is still in **BETA**, you should expect bugs and +incompatible changes in the future. + +Log messages contain a good deal of useful data, but it's not always easy to get +at. The log parser built into **lnav** is able to extract data as described by +:ref:`log_formats` as well as discovering data in plain text messages. This data +can then be queried and processed using the SQLite front-end that is also +incorporated into **lnav**. As an example, the following Syslog message from +:code:`sudo` can be processed to extract several key/value pairs:: + + Jul 31 11:42:26 Example-MacBook-Pro.local sudo[87024]: testuser : TTY=ttys004 ; PWD=/Users/testuser/github/lbuild ; USER=root ; COMMAND=/usr/bin/make install + +The data that can be extracted by the parser is viewable directly in **lnav** +by pressing the 'p' key. The results will be shown in an overlay like the +following:: + + Current Time: 2013-07-31T11:42:26.000 Original Time: 2013-07-31T11:42:26.000 Offset: +0.000 + Known message fields: + ├ log_hostname = Example-MacBook-Pro.local + ├ log_procname = sudo + ├ log_pid = 87024 + Discovered message fields: + ├ col_0 = testuser + ├ TTY = ttys004 + ├ PWD = /Users/testuser/github/lbuild + ├ USER = root + └ COMMAND = /usr/bin/make install + +Notice that the parser has detected pairs of the form '<key>=<value>'. The data +parser will also look for pairs separated by a colon. If there are no clearly +demarcated pairs, then the parser will extract anything that looks like data +values and assign them keys of the form 'col_N'. For example, two data values, +an IPv4 address and a symbol, will be extracted from the following log +message:: + + Apr 29 08:13:43 sample-centos5 avahi-daemon[2467]: Registering new address record for 10.1.10.62 on eth0. + +Since there are no keys for the values in the message, the parser will assign +'col_0' for the IP address and 'col_1' for the symbol, as seen here:: + + Current Time: 2013-04-29T08:13:43.000 Original Time: 2013-04-29T08:13:43.000 Offset: +0.000 + Known message fields: + ├ log_hostname = sample-centos5 + ├ log_procname = avahi-daemon + ├ log_pid = 2467 + Discovered message fields: + ├ col_0 = 10.1.10.62 + └ col_1 = eth0 + +Now that you have an idea of how the parser works, you can begin to perform +queries on the data that is being extracted. The SQLite database engine is +embedded into **lnav** and its `Virtual Table +<http://www.sqlite.org/vtab.html>`_ mechanism is used to provide a means to +process this log data. Each log format has its own table that can be used to +access all of the loaded messages that are in that format. For accessing log +message content that is more free-form, like the examples given here, the +**logline** table can be used. The **logline** table is recreated for each +query and is based on the format and pairs discovered in the log message at +the top of the display. + +Queries can be performed by pressing the semi-colon (;) key in **lnav**. After +pressing the key, the overlay showing any known or discovered fields will be +displayed to give you an idea of what data is available. The query can be any +`SQL query <http://sqlite.org/lang.html>`_ supported by SQLite. To make +analysis easier, **lnav** includes many extra functions for processing strings, +paths, and IP addresses. See :ref:`sql-ext` for more information. + +As an example, the simplest query to perform initially would be a "select all", +like so: + +.. code-block:: sql + + SELECT * FROM logline + +When this query is run against the second example log message given above, the +following results are received:: + + log_line log_part log_time log_idle_msecs log_level log_hostname log_procname log_pid col_0 col_1 + + 292 p.0 2013-04-11T16:42:51.000 0 info localhost avahi-daemon 2480 fe80::a00:27ff:fe98:7f6e eth0 + 293 p.0 2013-04-11T16:42:51.000 0 info localhost avahi-daemon 2480 10.0.2.15 eth0 + 330 p.0 2013-04-11T16:47:02.000 0 info localhost avahi-daemon 2480 fe80::a00:27ff:fe98:7f6e eth0 + 336 p.0 2013-04-11T16:47:02.000 0 info localhost avahi-daemon 2480 10.1.10.75 eth0 + 343 p.0 2013-04-11T16:47:02.000 0 info localhost avahi-daemon 2480 10.1.10.75 eth0 + 370 p.0 2013-04-11T16:59:39.000 0 info localhost avahi-daemon 2480 10.1.10.75 eth0 + 377 p.0 2013-04-11T16:59:39.000 0 info localhost avahi-daemon 2480 10.1.10.75 eth0 + 382 p.0 2013-04-11T16:59:41.000 0 info localhost avahi-daemon 2480 fe80::a00:27ff:fe98:7f6e eth0 + 401 p.0 2013-04-11T17:20:45.000 0 info localhost avahi-daemon 4247 fe80::a00:27ff:fe98:7f6e eth0 + 402 p.0 2013-04-11T17:20:45.000 0 info localhost avahi-daemon 4247 10.1.10.75 eth0 + + 735 p.0 2013-04-11T17:41:46.000 0 info sample-centos5 avahi-daemon 2465 fe80::a00:27ff:fe98:7f6e eth0 + 736 p.0 2013-04-11T17:41:46.000 0 info sample-centos5 avahi-daemon 2465 10.1.10.75 eth0 + 781 p.0 2013-04-12T03:32:30.000 0 info sample-centos5 avahi-daemon 2465 10.1.10.64 eth0 + 788 p.0 2013-04-12T03:32:30.000 0 info sample-centos5 avahi-daemon 2465 10.1.10.64 eth0 + 1166 p.0 2013-04-25T10:56:00.000 0 info sample-centos5 avahi-daemon 2467 fe80::a00:27ff:fe98:7f6e eth0 + 1167 p.0 2013-04-25T10:56:00.000 0 info sample-centos5 avahi-daemon 2467 10.1.10.111 eth0 + 1246 p.0 2013-04-26T06:06:25.000 0 info sample-centos5 avahi-daemon 2467 10.1.10.49 eth0 + 1253 p.0 2013-04-26T06:06:25.000 0 info sample-centos5 avahi-daemon 2467 10.1.10.49 eth0 + 1454 p.0 2013-04-28T06:53:55.000 0 info sample-centos5 avahi-daemon 2467 10.1.10.103 eth0 + 1461 p.0 2013-04-28T06:53:55.000 0 info sample-centos5 avahi-daemon 2467 10.1.10.103 eth0 + + 1497 p.0 2013-04-29T08:13:43.000 0 info sample-centos5 avahi-daemon 2467 10.1.10.62 eth0 + 1504 p.0 2013-04-29T08:13:43.000 0 info sample-centos5 avahi-daemon 2467 10.1.10.62 eth0 + +Note that **lnav** is not returning results for all messages that are in this +syslog file. Rather, it searches for messages that match the format for the +given line and returns only those messages in results. In this case, that +format is "Registering new address record for <IP> on <symbol>", which +corresponds to the parts of the message that were not recognized as data. + +More sophisticated queries can be done, of course. For example, to find out the +frequency of IP addresses mentioned in these messages, you can run: + +.. code-block:: sql + + SELECT col_0,count(*) FROM logline GROUP BY col_0 + +The results for this query are:: + + col_0 count(*) + + 10.0.2.15 1 + 10.1.10.49 2 + 10.1.10.62 2 + 10.1.10.64 2 + 10.1.10.75 6 + 10.1.10.103 2 + 10.1.10.111 1 + fe80::a00:27ff:fe98:7f6e 6 + +Since this type of query is fairly common, **lnav** includes a "summarize" +command that will compute the frequencies of identifiers as well as min, max, +average, median, and standard deviation for number columns. In this case, you +can run the following to compute the frequencies and return an ordered set of +results:: + + :summarize col_0 + + +Recognized Data Types +--------------------- + +When searching for data to extract from log messages, **lnav** looks for the +following set of patterns: + + +Strings + Single and double-quoted strings. Example: "The quick brown fox." + +URLs + URLs that contain the '://' separator. Example: http://example.com + +Paths + File system paths. Examples: /path/to/file, ./relative/path + +MAC Address + Ethernet MAC addresses. Example: c4:2c:03:0e:e4:4a + +Hex Dumps + A colon-separated string of hex numbers. Example: e8:06:88:ff + +Date/Time + Date and time stamps of the form "YYYY-mm-DD" and "HH:MM:SS". + +IP Addresses + IPv4 and IPv6 addresses. Examples: 127.0.0.1, fe80::c62c:3ff:fe0e:e44a%en0 + +UUID + The common formatting for 128-bit UUIDs. Example: + 0E305E39-F1E9-4DE4-B10B-5829E5DF54D0 + +Version Numbers + Dot-separated version numbers. Example: 3.7.17 + +Numbers + Numbers in base ten, hex, and octal formats. Examples: 1234, 0xbeef, 0777 + +E-Mail Address + Strings that look close to an e-mail address. Example: gary@example.com + +Constants + Common constants in languages, like: true, false, null, None. + +Symbols + Words that follow the common conventions for symbols in programming + languages. For example, containing all capital letters, or separated + by colons. Example: SOME_CONSTANT_VALUE, namespace::value diff --git a/docs/source/docutils.conf b/docs/source/docutils.conf new file mode 100644 index 0000000..1bf4d83 --- /dev/null +++ b/docs/source/docutils.conf @@ -0,0 +1,2 @@ +[restructuredtext parser] +syntax_highlight = short diff --git a/docs/source/events.rst b/docs/source/events.rst new file mode 100644 index 0000000..8318abf --- /dev/null +++ b/docs/source/events.rst @@ -0,0 +1,56 @@ +.. _Events: + +Events (v0.11.0+) +================= + +The events mechanism allows **lnav** to be automated based on events that +occur during processing. For example, filters could be added only when a +particular log file format is detected instead of always installing them. +Events are published through the :ref:`lnav_events<table_lnav_events>` SQLite +table. Reacting to events can be done by creating a SQLite trigger on the +table and inspecting the content of the event. + +Trigger Example +--------------- + +The following is an example of a trigger that adds an out filter when a +syslog file is loaded. You can copy the code into an :file:`.sql` file and +install it by running :code:`lnav -i my_trigger.sql`. + +.. code-block:: sql + :caption: my_trigger.sql + :linenos: + + CREATE TRIGGER IF NOT EXISTS add_format_specific_filters + AFTER INSERT ON lnav_events WHEN + -- Check the event type + jget(NEW.content, '/$schema') = + 'https://lnav.org/event-file-format-detected-v1.schema.json' AND + -- Only create the filter when a given format is seen + jget(NEW.content, '/format') = 'syslog_log' AND + -- Don't create the filter if it's already there + NOT EXISTS ( + SELECT 1 FROM lnav_view_filters WHERE pattern = 'noisy message') + BEGIN + INSERT INTO lnav_view_filters (view_name, enabled, type, pattern) VALUES + ('log', 1, 'OUT', 'noisy message'); + END; + +.. _event_reference: + +Reference +--------- + +The following tables describe the schema of the event JSON objects. + +.. jsonschema:: ../schemas/event-file-open-v1.schema.json# + :lift_description: + +.. jsonschema:: ../schemas/event-file-format-detected-v1.schema.json# + :lift_description: + +.. jsonschema:: ../schemas/event-log-msg-detected-v1.schema.json# + :lift_description: + +.. jsonschema:: ../schemas/event-session-loaded-v1.schema.json# + :lift_description: diff --git a/docs/source/faq.rst b/docs/source/faq.rst new file mode 100644 index 0000000..013d1c9 --- /dev/null +++ b/docs/source/faq.rst @@ -0,0 +1,85 @@ +.. _faq: + +Frequently Asked Questions +========================== + +Q: How can I copy & paste without decorations? +---------------------------------------------- + +:Answer: There are a couple ways to do this: + + * Use the :ref:`bookmark<hotkeys_bookmarks>` hotkeys to mark lines and then + press :kbd:`c` to copy to the local system keyboard. The system clipboard + is accessed using commands like :code:`pbcopy` and :code:`xclip`. See the + :ref:`tuning` section for more details. + + If a system clipboard is not available, + the `OSC 52 <https://www.reddit.com/r/vim/comments/k1ydpn/a_guide_on_how_to_copy_text_from_anywhere/>`_ + terminal escape sequence will be tried. If your terminal supports this + escape sequence, the selected text will be copied to the clipboard, even + if you are on an SSH connection. + + * Press :kbd:`CTRL` + :kbd:`l` to temporarily switch to "lo-fi" + mode where the contents of the current view are printed to the terminal. + This option is useful when you are logged into a remote host. + + +Q: How can I force a format for a file? +--------------------------------------- + +:Answer: The log format for a file is automatically detected and cannot be + forced. + +:Solution: Add some of the log file lines to the :ref:`sample<format_sample>` + array and then startup lnav to get a detailed explanation of where the format + patterns are not matching the sample lines. + +:Details: The first lines of the file are matched against the + :ref:`regular expressions defined in the format definitions<format_regex>`. + The order of the formats is automatically determined so that more specific + formats are tried before more generic ones. Therefore, if the expected + format is not being chosen for a file, then it means the regular expressions + defined by that format are not matching the first few lines of the file. + + See :ref:`format_order` for more information. + + +Q: How can I search backwards, like pressing :kbd:`?` in less? +-------------------------------------------------------------- + +:Answer: Searches in **lnav** runs in the background and do not block input + waiting to find the first hit. While the search prompt is open, pressing + :kbd:`CTRL` + :kbd:`j` will jump to the previous hit that was found. A + preview panel is also opened that shows the hits that have been found so + far. + + After pressing :kbd:`Enter` at the search prompt, the view will jump to + the first hit that was found. Then, you can press :kbd:`n` to move to + the next search hit and :kbd:`N` to move to the previous one. If you + would like to add a hotkey for jumping to the previous hit by default, + enter the following configuration command: + + .. code-block:: lnav + + :config /ui/keymap-defs/default/x3f/command :prompt --alt search ? + + +Q: Why isn't my log file highlighted correctly? +----------------------------------------------- + +TBD + +Q: Why isn't a file being displayed? +------------------------------------ + +:Answer: Plaintext files are displayed separately from log files in the TEXT + view. + +:Solution: Press the :kbd:`t` key to switch to the text view. Or, open the + files configuration panel by pressing :kbd:`TAB` to cycle through the + panels, and then press :kbd:`/` to search for the file you're interested in. + If the file is a log, a new :ref:`log format<log_formats>` will need to be + created or an existing one modified. + +:Details: If a file being monitored by lnav does not match a known log file + format, it is treated as plaintext and will be displayed in the TEXT view. diff --git a/docs/source/filter-out-preview.png b/docs/source/filter-out-preview.png Binary files differnew file mode 100644 index 0000000..8f9816f --- /dev/null +++ b/docs/source/filter-out-preview.png diff --git a/docs/source/formats.rst b/docs/source/formats.rst new file mode 100644 index 0000000..b30713d --- /dev/null +++ b/docs/source/formats.rst @@ -0,0 +1,561 @@ +.. _log_formats: + +Log Formats +=========== + +Log files loaded into **lnav** are parsed based on formats defined in +configuration files. Many +formats are already built in to the **lnav** binary and you can define your own +using a JSON file. When loading files, each format is checked to see if it can +parse the first few lines in the file. Once a match is found, that format will +be considered that files format and used to parse the remaining lines in the +file. If no match is found, the file is considered to be plain text and can +be viewed in the "text" view that is accessed with the **t** key. + +The following log formats are built into **lnav**: + +.. csv-table:: + :header: "Name", "Table Name", "Description" + :widths: 8 5 20 + :file: format-table.csv + +In addition to the above formats, the following self-describing formats are +supported: + +* The + `Bro Network Security Monitor <https://www.bro.org/sphinx/script-reference/log-files.html>`_ + TSV log format is supported in lnav versions v0.8.3+. The Bro log format is + self-describing, so **lnav** will read the header to determine the shape of + the file. +* The + `W3C Extended Log File Format <https://www.w3.org/TR/WD-logfile.html>`_ + is supported in lnav versions v0.10.0+. The W3C log format is + self-describing, so **lnav** will read the header to determine the shape of + the file. + +There is also basic support for the `logfmt <https://brandur.org/logfmt>`_ +convention for formatting log messages. Files that use this format must +have the entire line be key/value pairs and the timestamp contained in a +field named :code:`time` or :code:`ts`. If the file you're using does not +quite follow this formatting, but wraps logfmt data with another recognized +format, you can use the :ref:`logfmt2json` SQL function to convert the data +into JSON for further analysis. + + +Defining a New Format +--------------------- + +New log formats can be defined by placing JSON configuration files in +subdirectories of the :file:`/etc/lnav/formats` and :file:`~/.lnav/formats/` +directories. The directories and files can be named anything you like, but the +files must have the '.json' suffix. A sample file containing the builtin +configuration will be written to this directory when **lnav** starts up. +You can consult that file when writing your own formats or if you need to +modify existing ones. Format directories can also contain '.sql' and '.lnav' +script files that can be used automate log file analysis. + +Creating a Format Using Regex101.com (v0.11.0+) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +For plain-text log files, the easiest way to create a log format definition is +to create the regular expression that recognizes log messages using +https://regex101.com . Simply copy a log line into the test string input box +on the site and then start editing the regular expression. When building the +regular expression, you'll want to use named captures for the structured parts +of the log message. Any raw message text should be matched by a captured named +"body". Once you have a regex that matches the whole log message, you can use +**lnav**'s "management CLI" to create a skeleton format file. The skeleton +will be populated with the regular expression from the site and the test +string, along with any unit tests, will be added to the "samples" list. The +"regex101 import" management command is used to create the skeleton and has +the following form: + +.. prompt:: bash + + lnav -m regex101 import <regex101-url> <format-name> [<regex-name>] + +If the import was successful, the path to the new format file should be +printed out. The skeleton will most likely need some changes to make it +fully functional. For example, the :code:`kind` properties for captured values +default to :code:`string`, but you'll want to change them to the appropriate +type. + +Format File Reference +^^^^^^^^^^^^^^^^^^^^^ + +An **lnav** format file must contain a single JSON object, preferably with a +:code:`$schema` property that refers to the +`format-v1.schema <https://lnav.org/schemas/format-v1.schema.json>`_, +like so: + +.. code-block:: json + + { + "$schema": "https://lnav.org/schemas/format-v1.schema.json" + } + +Each format to be defined in the file should be a separate field in the top-level +object. The field name should be the symbolic name of the format and consist +only of alphanumeric characters and underscores. This value will also be used +as the SQL table name for the log. The value for each field should be another +object with the following fields: + +:title: The short and human-readable name for the format. +:description: A longer description of the format. +:url: A URL to the definition of the format. + +:file-pattern: A regular expression used to match log file paths. Typically, + every file format will be tried during the detection process. This field + can be used to limit which files a format is applied to in case there is + a potential for conflicts. + +.. _format_regex: + +:regex: This object contains sub-objects that describe the message formats + to match in a plain-text log file. Each :code:`regex` MUST only match one + type of log message. It must not match log messages that are matched by + other regexes in this format. This uniqueness requirement is necessary + because **lnav** will "lock-on" to a regex and use it to match against + the next line in a file. So, if the regexes do not uniquely match each + type of log message, messages can be matched by the wrong regex. The + "lock-on" behavior is needed to avoid the performance hit of having to + try too many different regexes. + + .. note:: Log files that contain JSON messages should not specify this field. + + :pattern: The regular expression that should be used to match log messages. + The `PCRE2 <http://www.pcre.org>`_ library is used by **lnav** to do all + regular expression matching. + + :module-format: If true, this regex will only be used to parse message + bodies for formats that can act as containers, such as syslog. Default: + false. + +:json: True if each log line is JSON-encoded. + +:line-format: An array that specifies the text format for JSON-encoded + log messages. Log files that are JSON-encoded will have each message + converted from the raw JSON encoding into this format. Each element + is either an object that defines which fields should be inserted into + the final message string and or a string constant that should be + inserted. For example, the following configuration will transform each + log message object into a string that contains the timestamp, followed + by a space, and then the message body: + + .. code-block:: json + + [ { "field": "ts" }, " ", { "field": "msg" } ] + + .. note:: Line-feeds at the end of a value are automatically stripped. + + :field: The name or `JSON-Pointer <https://tools.ietf.org/html/rfc6901>`_ + of the message field that should be inserted at this point in the + message. The special :code:`__timestamp__` field name can be used to + insert a human-readable timestamp. The :code:`__level__` field can be + used to insert the level name as defined by lnav. + + .. tip:: + + Use a JSON-Pointer to reference nested fields. For example, to include + a "procname" property that is nested in a "details" object, you would + write the field reference as :code:`/details/procname`. + + :min-width: The minimum width for the field. If the value for the field + in a given log message is shorter, padding will be added as needed to + meet the minimum-width requirement. (v0.8.2+) + :max-width: The maximum width for the field. If the value for the field + in a given log message is longer, the overflow algorithm will be applied + to try and shorten the field. (v0.8.2+) + :auto-width: Flag that indicates that the width of the field should + automatically be set to the widest value seen. (v0.11.2) + :align: Specifies the alignment for the field, either "left" or "right". + If "left", padding to meet the minimum-width will be added on the right. + If "right", padding will be added on the left. (v0.8.2+) + :overflow: The algorithm used to shorten a field that is longer than + "max-width". The following algorithms are supported: + + :abbrev: Removes all but the first letter in dotted text. For example, + "com.example.foo" would be shortened to "c.e.foo". + :truncate: Truncates any text past the maximum width. + :dot-dot: Cuts out the middle of the text and replaces it with two + dots (i.e. '..'). + + (v0.8.2+) + :timestamp-format: The timestamp format to use when displaying the time + for this log message. (v0.8.2+) + :default-value: The default value to use if the field could not be found + in the current log message. The built-in default is "-". + :text-transform: Transform the text in the field. Supported options are: + none, uppercase, lowercase, capitalize + :prefix: Text to prepend to the value. If the value is empty, this prefix + will not be added. + :suffix: Text to append to the value. If the value is empty, this suffix + will not be added. + +:timestamp-field: The name of the field that contains the log message + timestamp. Defaults to "timestamp". + +:timestamp-format: An array of timestamp formats using a subset of the + strftime conversion specification. The following conversions are + supported: %a, %b, %L, %M, %H, %I, %d, %e, %k, %l, %m, %p, %y, %Y, %S, %s, + %Z, %z. In addition, you can also use the following: + + :%L: Milliseconds as a decimal number (range 000 to 999). + :%f: Microseconds as a decimal number (range 000000 to 999999). + :%N: Nanoseconds as a decimal number (range 000000000 to 999999999). + :%q: Seconds from the epoch as a hexidecimal number. + :%i: Milliseconds from the epoch. + :%6: Microseconds from the epoch. + +:timestamp-divisor: For JSON logs with numeric timestamps, this value is used + to divide the timestamp by to get the number of seconds and fractional + seconds. + +:subsecond-field: (v0.11.1+) The path to the property in a JSON-lines log + message that contains the sub-second time value + +:subsecond-units: (v0.11.1+) The units of the subsecond-field property value. + The following values are supported: + + :milli: for milliseconds + :micro: for microseconds + :nano: for nanoseconds + +:ordered-by-time: (v0.8.3+) Indicates that the order of messages in the file + is time-based. Files that are not naturally ordered by time will be sorted + in order to display them in the correct order. Note that this sorting can + incur a performance penalty when tailing logs. + +:level-field: The name of the regex capture group that contains the log + message level. Defaults to "level". + +:body-field: The name of the field that contains the main body of the + message. Defaults to "body". + +:opid-field: The name of the field that contains the "operation ID" of the + message. An "operation ID" establishes a thread of messages that might + correspond to a particular operation/request/transaction. The user can + press the 'o' or 'Shift+O' hotkeys to move forward/backward through the + list of messages that have the same operation ID. Note: For JSON-encoded + logs, the opid field can be a path (e.g. "foo/bar/opid") if the field is + nested in an object and it MUST be included in the "line-format" for the + 'o' hotkeys to work. + +:module-field: The name of the field that contains the module identifier + that distinguishes messages from one log source from another. This field + should be used if this message format can act as a container for other + types of log messages. For example, an Apache access log can be sent to + syslog instead of written to a file. In this case, **lnav** will parse + the syslog message and then separately parse the body of the message to + determine the "sub" format. This module identifier is used to help + **lnav** quickly identify the format to use when parsing message bodies. + +:hide-extra: A boolean for JSON logs that indicates whether fields not + present in the line-format should be displayed on their own lines. + +:level: A mapping of error levels to regular expressions. During scanning + the contents of the capture group specified by *level-field* will be + checked against each of these regexes. Once a match is found, the log + message level will set to the corresponding level. The available levels, + in order of severity, are: **fatal**, **critical**, **error**, + **warning**, **stats**, **info**, **debug**, **debug2-5**, **trace**. + For JSON logs with exact numeric levels, the number for the corresponding + level can be supplied. If the JSON log format uses numeric ranges instead + of exact numbers, you can supply a pattern and the number found in the log + will be converted to a string for pattern-matching. + + .. note:: The regular expression is not anchored to the start of the + string by default, so an expression like :code:`1` will match + :code:`-1`. If you want to exactly match :code:`1`, you would + use :code:`^1$` as the expression. + +:multiline: If false, **lnav** will consider any log lines that do not + match one of the message patterns to be in error when checking files with + the '-C' option. This flag will not affect normal viewing operation. + Default: true. + +:value: This object contains the definitions for the values captured by the + regexes. + + :kind: The type of data that was captured **string**, **integer**, + **float**, **json**, **quoted**. + :collate: The name of the SQLite collation function for this value. + The standard SQLite collation functions can be used as well as the + ones defined by lnav, as described in :ref:`collators`. + :identifier: A boolean that indicates whether or not this field represents + an identifier and should be syntax colored. + :foreign-key: A boolean that indicates that this field is a key and should + not be graphed. This should only need to be set for integer fields. + :hidden: A boolean for log fields that indicates whether they should + be displayed. The behavior is slightly different for JSON logs and text + logs. For a JSON log, this property determines whether an extra line + will be added with the key/value pair. For text logs, this property + controls whether the value should be displayed by default or replaced + with an ellipsis. + :rewriter: A command to rewrite this field when pretty-printing log + messages containing this value. The command must start with ':', ';', + or '|' to signify whether it is a regular command, SQL query, or a script + to be executed. The other fields in the line are accessible in SQL by + using the ':' prefix. The text value of this field will then be replaced + with the result of the command when pretty-printing. For example, the + HTTP access log format will rewrite the status code field to include the + textual version (e.g. 200 (OK)) using the following SQL query: + + .. code-block:: sql + + ;SELECT :sc_status || ' (' || ( + SELECT message FROM http_status_codes + WHERE status = :sc_status) || ') ' + +:tags: This object contains the tags that should automatically be added to + log messages. + + :pattern: The regular expression evaluated over a line in the log file as + it is read in. If there is a match, the log message the line is a part + of will have this tag added to it. + :paths: This array contains objects that define restrictions on the file + paths that the tags will be applied to. The objects in this array can + contain: + + :glob: A glob pattern to check against the log files read by lnav. + +.. _format_sample: + +:sample: A list of objects that contain sample log messages. All formats + must include at least one sample and it must be matched by one of the + included regexes. Each object must contain the following field: + + :line: The sample message. + :level: The expected error level. An error will be raised if this level + does not match the level parsed by lnav for this sample message. + +:highlights: This object contains the definitions for patterns to be + highlighted in a log message. Each entry should have a name and a + definition with the following fields: + + :pattern: The regular expression to match in the log message body. + :color: The foreground color to use when highlighting the part of the + message that matched the pattern. If no color is specified, one will be + picked automatically. Colors can be specified using hexadecimal notation + by starting with a hash (e.g. #aabbcc) or using a color name as found + at http://jonasjacek.github.io/colors/. + :background-color: The background color to use when highlighting the part + of the message that matched the pattern. If no background color is + specified, black will be used. The background color is only considered + if a foreground color is specified. + :underline: If true, underline the part of the message that matched the + pattern. + :blink: If true, blink the part of the message that matched the pattern. + +Example format: + +.. code-block:: json + + { + "$schema": "https://lnav.org/schemas/format-v1.schema.json", + "example_log" : { + "title" : "Example Log Format", + "description" : "Log format used in the documentation example.", + "url" : "http://example.com/log-format.html", + "regex" : { + "basic" : { + "pattern" : "^(?<timestamp>\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}\\.\\d{3}Z)>>(?<level>\\w+)>>(?<component>\\w+)>>(?<body>.*)$" + } + }, + "level-field" : "level", + "level" : { + "error" : "ERROR", + "warning" : "WARNING" + }, + "value" : { + "component" : { + "kind" : "string", + "identifier" : true + } + }, + "sample" : [ + { + "line" : "2011-04-01T15:14:34.203Z>>ERROR>>core>>Shit's on fire yo!" + } + ] + } + } + +Patching an Existing Format +--------------------------- + +When loading log formats from files, **lnav** will overlay any new data over +previously loaded data. This feature allows you to override existing value or +append new ones to the format configurations. For example, you can separately +add a new regex to the example log format given above by creating another file +with the following contents: + +.. code-block:: json + + { + "$schema": "https://lnav.org/schemas/format-v1.schema.json", + "example_log" : { + "regex" : { + "custom1" : { + "pattern" : "^(?<timestamp>\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}\\.\\d{3}Z)<<(?<level>\\w+)--(?<component>\\w+)>>(?<body>.*)$" + } + }, + "sample" : [ + { + "line" : "2011-04-01T15:14:34.203Z<<ERROR--core>>Shit's on fire yo!" + } + ] + } + } + + +This example overrides the default `syslog_log <https://github.com/tstack/lnav/blob/master/src/formats/syslog_log.json>`_ +error detection regex to **not** match the :code:`errors=` string. + +.. code-block:: json + + { + "syslog_log": { + "level": { + "error": "(?:(?:(?<![a-zA-Z]))(?:(?i)error(?:s)?(?!=))(?:(?![a-zA-Z]))|failed|failure)" + } + } + } + + +.. _scripts: + +Scripts +------- + +Format directories may also contain :file:`.sql` and :file:`.lnav` files to help automate +log file analysis. The SQL files are executed on startup to create any helper +tables or views and the '.lnav' script files can be executed using the pipe +hotkey :kbd:`|`. For example, **lnav** includes a "partition-by-boot" script that +partitions the log view based on boot messages from the Linux kernel. A script +can have a mix of SQL and **lnav** commands, as well as include other scripts. +The type of statement to execute is determined by the leading character on a +line: a semi-colon begins a SQL statement; a colon starts an **lnav** command; +and a pipe :code:`|` denotes another script to be executed. Lines beginning with a +hash are treated as comments. The following variables are defined in a script: + +.. envvar:: # + + The number of arguments passed to the script. + +.. envvar:: __all__ + + A string containing all the arguments joined by a single space. + +.. envvar:: 0 + + The path to the script being executed. + +.. envvar:: 1-N + + The arguments passed to the script. + +Remember that you need to use the :ref:`:eval<eval>` command when referencing +variables in most **lnav** commands. Scripts can provide help text to be +displayed during interactive usage by adding the following tags in a comment +header: + + :@synopsis: The synopsis should contain the name of the script and any + parameters to be passed. For example:: + + # @synopsis: hello-world <name1> [<name2> ... <nameN>] + + :@description: A one-line description of what the script does. For example:: + + # @description: Say hello to the given names. + + + +.. tip:: + + The :ref:`:eval<eval>` command can be used to do variable substitution for + commands that do not natively support it. For example, to substitute the + variable, :code:`pattern`, in a :ref:`:filter-out<filter_out>` command: + + .. code-block:: lnav + + :eval :filter-out ${pattern} + +VSCode Extension +^^^^^^^^^^^^^^^^ + +The `lnav VSCode Extension <https://marketplace.visualstudio.com/items?itemName=lnav.lnav>`_ +can be installed to add syntax highlighting to lnav scripts. + +Installing Formats +------------------ + +File formats are loaded from subdirectories in :file:`/etc/lnav/formats` and +:file:`~/.lnav/formats/`. You can manually create these subdirectories and +copy the format files into there. Or, you can pass the '-i' option to **lnav** +to automatically install formats from the command-line. For example: + +.. code-block:: bash + + $ lnav -i myformat.json + info: installed: /home/example/.lnav/formats/installed/myformat_log.json + +Format files installed using this method will be placed in the :file:`installed` +subdirectory and named based on the first format name found in the file. + +You can also install formats from git repositories by passing the repository's +clone URL. A standard set of repositories is maintained at +(https://github.com/tstack/lnav-config) and can be installed by passing 'extra' +on the command line, like so: + +.. prompt:: bash + + lnav -i extra + +These repositories can be updated by running **lnav** with the '-u' flag. + +Format files can also be made executable by adding a shebang (#!) line to the +top of the file, like so:: + + #! /usr/bin/env lnav -i + { + "myformat_log" : ... + } + +Executing the format file should then install it automatically: + +.. code-block:: bash + + $ chmod ugo+rx myformat.json + $ ./myformat.json + info: installed: /home/example/.lnav/formats/installed/myformat_log.json + +.. _format_order: + +Format Order When Scanning a File +--------------------------------- + +When **lnav** loads a file, it tries each log format against the first 15,000 +lines [#]_ of the file trying to find a match. When a match is found, that log +format will be locked in and used for the rest of the lines in that file. +Since there may be overlap between formats, **lnav** performs a test on +startup to determine which formats match each others sample lines. Using +this information it will create an ordering of the formats so that the more +specific formats are tried before the more generic ones. For example, a +format that matches certain syslog messages will match its own sample lines, +but not the ones in the syslog samples. On the other hand, the syslog format +will match its own samples and those in the more specific format. You can +see the order of the format by enabling debugging and checking the **lnav** +log file for the "Format order" message: + +.. prompt:: bash + + lnav -d /tmp/lnav.log + +For JSON-lines log files, the log message must have the timestamp property +specified in the format in order to match. If multiple formats match a +message, the format that has the most matching :code:`line-format` elements +will win. + +.. [#] The maximum number of lines to check can be configured. See the + :ref:`tuning` section for more details. diff --git a/docs/source/group_concat-help.png b/docs/source/group_concat-help.png Binary files differnew file mode 100644 index 0000000..f7b6f08 --- /dev/null +++ b/docs/source/group_concat-help.png diff --git a/docs/source/hotkey-tips.png b/docs/source/hotkey-tips.png Binary files differnew file mode 100644 index 0000000..5b78686 --- /dev/null +++ b/docs/source/hotkey-tips.png diff --git a/docs/source/hotkeys.rst b/docs/source/hotkeys.rst new file mode 100644 index 0000000..4bad1d9 --- /dev/null +++ b/docs/source/hotkeys.rst @@ -0,0 +1,300 @@ +.. _hotkeys: + +Hotkey Reference +================ + +This reference covers the keys used to control **lnav**. Consult the `built-in +help <https://github.com/tstack/lnav/blob/master/src/help.txt>`_ in **lnav** for +a more detailed explanation of each key. + +Spatial Navigation +------------------ + +The majority of these hotkeys should be available in all views. + +.. list-table:: + :header-rows: 1 + :widths: 6 6 6 20 + + * - Keypress + - + - + - Command + * - :kbd:`Space` + - :kbd:`PgDn` + - + - Down a page + * - :kbd:`Ctrl` + :kbd:`d` + - + - + - Down by half a page + * - :kbd:`b` + - :kbd:`Backspace` + - :kbd:`PgUp` + - Up a page + * - :kbd:`Ctrl` + :kbd:`u` + - + - + - Up by half a page + * - :kbd:`j` + - :kbd:`↓` + - + - Down a line + * - :kbd:`k` + - :kbd:`↑` + - + - Up a line + * - :kbd:`h` + - :kbd:`←` + - + - Left half a page. In the log view, pressing left while at the start of + the message text will reveal the shortened source file name for each line. + Pressing again will reveal the full path. + * - :kbd:`Shift` + :kbd:`h` + - :kbd:`Shift` + :kbd:`←` + - + - Left ten columns + * - :kbd:`l` + - :kbd:`→` + - + - Right half a page + * - :kbd:`Shift` + :kbd:`l` + - :kbd:`Shift` + :kbd:`→` + - + - Right ten columns + * - :kbd:`Home` + - :kbd:`g` + - + - Top of the view + * - :kbd:`End` + - :kbd:`G` + - + - Bottom of the view + * - :kbd:`e` + - :kbd:`Shift` + :kbd:`e` + - + - Next/previous error + * - :kbd:`w` + - :kbd:`Shift` + :kbd:`w` + - + - Next/previous warning + * - :kbd:`n` + - :kbd:`Shift` + :kbd:`n` + - + - Next/previous search hit + * - :kbd:`>` + - :kbd:`<` + - + - Next/previous search hit (horizontal) + * - :kbd:`f` + - :kbd:`Shift` + :kbd:`f` + - + - Next/previous file + * - :kbd:`u` + - :kbd:`Shift` + :kbd:`u` + - + - Next/previous bookmark + * - :kbd:`o` + - :kbd:`Shift` + :kbd:`o` + - + - Forward/backward through log messages with a matching "opid" field + * - :kbd:`s` + - :kbd:`Shift` + :kbd:`s` + - + - Next/previous slow down in the log message rate + * - :kbd:`{` + - :kbd:`}` + - + - Previous/next location in history + +Chronological Navigation +------------------------ + +These hotkeys are only functional on views that are time-based, like the log +view or the histogram view. + +.. list-table:: + :header-rows: 1 + :widths: 5 5 20 + + * - Keypress + - + - Command + * - :kbd:`d` + - :kbd:`Shift` + :kbd:`d` + - Forward/backward 24 hours + * - :kbd:`1` - :kbd:`6` + - :kbd:`Shift` + :kbd:`1` - :kbd:`6` + - Next/previous n'th ten minute of the hour + * - :kbd:`7` + - :kbd:`8` + - Previous/next minute + * - :kbd:`0` + - :kbd:`Shift` + :kbd:`0` + - Next/previous day + * - :kbd:`r` + - :kbd:`Shift` + :kbd:`r` + - Forward/backward by the relative time that was last used with the goto command. + +Breadcrumb Navigation +--------------------- + +The following hotkeys are related to the breadcrumb bar that is below the top +status bar. + +.. list-table:: + :header-rows: 1 + :widths: 5 20 + + * - Keypress + - Description + * - :kbd:`ENTER` + - Focus on the breadcrumb bar. Or, if the bar is currently focused, + accept the selected value and drop focus. + * - :kbd:`Escape` + - Drop focus on the breadcrumb bar. + * - :kbd:`←` + - Select the crumb to the left. If the first crumb is selected, the + selection will wrap around to the last crumb. + * - :kbd:`→` + - Accept the current value, which might mean navigating to the value in + the view, then selecting the crumb to the right. + * - :kbd:`Ctrl` + :kbd:`a` + - Select the first crumb. + * - :kbd:`Ctrl` + :kbd:`e` + - Select the last crumb. + * - :kbd:`↓` + - Select the next value in the crumb dropdown. + * - :kbd:`↑` + - Select the previous value in the crumb dropdown. + * - :kbd:`Home` + - Select the first value in the crumb dropdown. + * - :kbd:`End` + - Select the last value in the crumb dropdown. + +While a crumb is selected, you can perform a fuzzy search on the possible +values by typing in the value you are interested in. + +.. _hotkeys_bookmarks: + +Bookmarks +--------- + +.. list-table:: + :header-rows: 1 + :widths: 5 20 + + * - Keypress + - Command + * - :kbd:`m` + - Mark/unmark the top line or focused line when in cursor mode + * - :kbd:`Shift` + :kbd:`m` + - Mark/unmark the range of lines from the last marked to the top + * - :kbd:`Shift` + :kbd:`j` + - Mark/unmark the next line after the previously marked + * - :kbd:`Shift` + :kbd:`k` + - Mark/unmark the previous line + * - :kbd:`c` + - Copy marked lines to the clipboard + * - :kbd:`Shift` + :kbd:`c` + - Clear marked lines + +.. _hotkeys_display: + +Display +------- + +.. list-table:: + :header-rows: 1 + :widths: 5 20 + + * - Keypress + - Command + * - :kbd:`?` + - View/leave builtin help + * - :kbd:`q` + - Return to the previous view/quit + * - :kbd:`Shift` + :kbd:`q` + - Return to the previous view/quit while matching the top times of the two views + * - :kbd:`a` + - Restore the view that was previously popped with 'q/Q' + * - :kbd:`Shift` + :kbd:`a` + - Restore the view that was previously popped with 'q/Q' and match the top times of the views + * - :kbd:`Shift` + :kbd:`p` + - Switch to/from the pretty-printed view of the displayed log or text files + * - :kbd:`Shift` + :kbd:`t` + - Display elapsed time between lines + * - :kbd:`t` + - Switch to/from the text file view + * - :kbd:`i` + - Switch to/from the histogram view + * - :kbd:`Shift` + :kbd:`i` + - Switch to/from the histogram view + * - :kbd:`v` + - Switch to/from the SQL result view + * - :kbd:`Shift` + :kbd:`v` + - Switch to/from the SQL result view and move to the corresponding in the + log_line column + * - :kbd:`p` + - Toggle the display of the log parser results + * - :kbd:`Tab` + - In the log/text views, focus on the configuration panel for editing + filters and examining the list of loaded files. In the SQL result view, + cycle through columns to display as bar graphs + * - :kbd:`Ctrl` + :kbd:`l` + - Switch to lo-fi mode. The displayed log lines will be dumped to the + terminal without any decorations so they can be copied easily. + * - :kbd:`Ctrl` + :kbd:`w` + - Toggle word-wrap. + * - :kbd:`Ctrl` + :kbd:`p` + - Show/hide the data preview panel that may be opened when entering + commands or SQL queries. + * - :kbd:`Ctrl` + :kbd:`f` + - Toggle the enabled/disabled state of all filters in the current view. + * - :kbd:`x` + - Toggle the hiding of log message fields. The hidden fields will be + replaced with three bullets and highlighted in yellow. + * - :kbd:`Ctrl` + :kbd:`x` + - Toggle the cursor mode. Allows moving the focused line instead of + keeping it fixed at the top of the current screen. + * - :kbd:`=` + - Pause/unpause loading of new file data. + +Session +------- + +.. list-table:: + :header-rows: 1 + :widths: 5 20 + + * - Keypress + - Command + * - :kbd:`Ctrl` + :kbd:`R` + - Reset the current :ref:`session<sessions>` state. The session state + includes things like filters, bookmarks, and hidden fields. + +Query Prompts +------------- + +.. list-table:: + :header-rows: 1 + :widths: 5 20 + + * - Keypress + - Command + * - :kbd:`/` + - Search for lines matching a regular expression + * - :kbd:`;` + - Open the :ref:`sql-ext` to execute SQL statements/queries + * - :kbd:`:` + - Execute an internal command, see :ref:`commands` for more information + * - :kbd:`\|` + - Execute an lnav script located in a format directory + * - :kbd:`Ctrl` + :kbd:`]` + - Abort the prompt + +Customizing +----------- + +You can customize the behavior of hotkeys by defining your own keymaps. +Consult the :ref:`Keymaps<keymaps>` configuration section for more information. diff --git a/docs/source/howitworks.rst b/docs/source/howitworks.rst new file mode 100644 index 0000000..d111fd0 --- /dev/null +++ b/docs/source/howitworks.rst @@ -0,0 +1,13 @@ + +.. _howitworks: + +How It Works +============ + +"Magic" + +Internal Architecture +--------------------- + +The `ARCHITECTURE.md <https://github.com/tstack/lnav/blob/master/ARCHITECTURE.md>`_ +file in the source tree contains some information about lnav's internals. diff --git a/docs/source/index.rst b/docs/source/index.rst new file mode 100644 index 0000000..d94c0eb --- /dev/null +++ b/docs/source/index.rst @@ -0,0 +1,35 @@ +Welcome to lnav's documentation! +================================ + +The `Log File Navigator <http://lnav.org>`_ (**lnav**) is an advanced log file +viewer for the console. + +Contents: + +.. toctree:: + :maxdepth: 2 + + intro + ui + hotkeys + cli + usage + cookbook + config + formats + sessions + commands + sqlext + sqltab + events + data + howitworks + faq + + +Indices and tables +================== + +* :ref:`genindex` +* :ref:`modindex` +* :ref:`search` diff --git a/docs/source/intro.rst b/docs/source/intro.rst new file mode 100644 index 0000000..96b0612 --- /dev/null +++ b/docs/source/intro.rst @@ -0,0 +1,164 @@ +Introduction +============ + +The Log File Navigator, **lnav**, is an advanced log file viewer for the +terminal. It provides an :ref:`easy-to-use interface<ui>` for monitoring and +analyzing your log files with little to no setup. Simply point **lnav** at +your log files and it will automatically detect the :ref:`log_formats`, index +their contents, and display a combined view of all log messages. You can +navigate through your logs using a variety of :ref:`hotkeys<hotkeys>`. +:ref:`Commands<commands>` give you additional control over **lnav**'s behavior +for doing things like applying filters, tagging messages, and more. You can +then analyze your log messages using the :ref:`sql-ext`. + +Development +----------- + +Development of lnav is hosted on `GitHub <https://github.com/tstack/lnav/>`_. + +`Issues <https://github.com/tstack/lnav/issues>`_ should be used for bugs +and feature requests. + +`Discussions <https://github.com/tstack/lnav/discussions>`_ should be used +for asking questions and sharing tips. + +Downloads +--------- + +Binaries and source code for lnav can be downloaded from the +`releases page <https://github.com/tstack/lnav/releases>`_. + +When building from source, follow the steps below. + +Dependencies +^^^^^^^^^^^^ + +When compiling from source, the following dependencies are required: + +* `NCurses <http://www.gnu.org/s/ncurses/>`_ +* `PCRE2 <http://www.pcre.org>`_ +* `SQLite <http://www.sqlite.org>`_ +* `ZLib <http://wwww.zlib.net>`_ +* `Bzip2 <http://www.bzip.org>`_ +* `Readline <http://www.gnu.org/s/readline>`_ +* `libcurl <https://curl.haxx.se>`_ +* `libarchive <https://libarchive.org>`_ + +Installation +^^^^^^^^^^^^ + +Check the `downloads page <http://lnav.org/downloads>`_ to see if there are +packages for your operating system. To compile from source, use the following +commands: + +.. prompt:: bash + + ./configure + make + sudo make install + + +Viewing Logs +------------ + +The arguments to **lnav** are the log files, directories, or URLs to be viewed. +For example, to view all of the CUPS logs on your system: + +.. prompt:: bash + + lnav /var/log/cups + +The formats of the logs are determined automatically and indexed on-the-fly. +See :ref:`log_formats` for a listing of the predefined formats and how to +define your own. + +If no arguments are given, **lnav** will try to open the syslog file on your +system: + +.. prompt:: bash + + lnav + + +Setup +----- + +After starting **lnav**, you might want to set the +:ref:`configuration options<Configuration>` mentioned below. Configuration in +**lnav** is done using the :code:`:config` command. To change a configuration +option, start by pressing :kbd:`:` to enter the command prompt. Then, +type "config" followed by the option name and value. + +.. note:: + + Tab-completion is available for these configuration options and, in some + cases, their values as well. + + +Keymap +^^^^^^ + +The keymap defines the mapping from :ref:`hotkeys<hotkeys>` to commands to +execute. The default mapping is for "U.S." keyboards. The following command +can be used to change the keymap: + +.. code-block:: lnav + + :config /ui/keymap <keymap-name> + +The builtin keymaps are: + + :de: `German <https://github.com/tstack/lnav/blob/master/src/keymaps/de-keymap.json>`_ + :fr: `French <https://github.com/tstack/lnav/blob/master/src/keymaps/fr-keymap.json>`_ + :sv: `Swedish <https://github.com/tstack/lnav/blob/master/src/keymaps/sv-keymap.json>`_ + :uk: `United Kingdom <https://github.com/tstack/lnav/blob/master/src/keymaps/uk-keymap.json>`_ + :us: `United States <https://github.com/tstack/lnav/blob/master/src/keymaps/us-keymap.json>`_ + +To create or customize a keymap, consult the :ref:`keymaps` section. + + +Theme +^^^^^ + +The visual styling of **lnav** can be customized using a theme. The following +command can be used to the change the theme: + +.. code-block:: lnav + + :config /ui/theme <theme-name> + +The builtin themes are: +`default <https://github.com/tstack/lnav/blob/master/src/themes/default-theme.json>`_, +`eldar <https://github.com/tstack/lnav/blob/master/src/themes/eldar.json>`_, +`grayscale <https://github.com/tstack/lnav/blob/master/src/themes/grayscale.json>`_, +`monocai <https://github.com/tstack/lnav/blob/master/src/themes/monocai.json>`_, +`night-owl <https://github.com/tstack/lnav/blob/master/src/themes/night-owl.json>`_, +`solarized-dark <https://github.com/tstack/lnav/blob/master/src/themes/solarized-dark.json>`_, +and +`solarized-light <https://github.com/tstack/lnav/blob/master/src/themes/default-theme.json>`_. + +To create or customize a theme, consult the :ref:`themes` section. + + +Cursor Mode (v0.11.2+) +^^^^^^^^^^^^^^^^^^^^^^ + +The default mode for scrolling in **lnav** is to move the contents of the +main view when the arrow keys are pressed. Any interactions, such as +jumping to a search hit, are then focused on the top line in the view. +Alternatively, you can enable "cursor" mode where there is a cursor line +in the view that is moved by the arrow keys and other interactions. You +can enable cursor mode with the following command: + +.. code-block:: lnav + + :config /ui/movement/mode cursor + +Log Formats +^^^^^^^^^^^ + +In order for **lnav** to understand your log files, it needs to told how to +parse the log messages using a log format definition. There are many log +formats builtin and **lnav** will automatically determine the best format to +use. In case your log file is not recognized, consult the :ref:`log_formats` +section for information on how to create a format. diff --git a/docs/source/key-encoding-prompt.png b/docs/source/key-encoding-prompt.png Binary files differnew file mode 100644 index 0000000..fe63412 --- /dev/null +++ b/docs/source/key-encoding-prompt.png diff --git a/docs/source/lnav-breadcrumbs-help.png b/docs/source/lnav-breadcrumbs-help.png Binary files differnew file mode 100644 index 0000000..b3b80d0 --- /dev/null +++ b/docs/source/lnav-breadcrumbs-help.png diff --git a/docs/source/lnav-config-header.png b/docs/source/lnav-config-header.png Binary files differnew file mode 100644 index 0000000..079b9c2 --- /dev/null +++ b/docs/source/lnav-config-header.png diff --git a/docs/source/lnav-files-panel.png b/docs/source/lnav-files-panel.png Binary files differnew file mode 100644 index 0000000..7aed793 --- /dev/null +++ b/docs/source/lnav-files-panel.png diff --git a/docs/source/lnav-filters-panel.png b/docs/source/lnav-filters-panel.png Binary files differnew file mode 100644 index 0000000..788e63f --- /dev/null +++ b/docs/source/lnav-filters-panel.png diff --git a/docs/source/lnav-markdown-example.png b/docs/source/lnav-markdown-example.png Binary files differnew file mode 100644 index 0000000..e51bad0 --- /dev/null +++ b/docs/source/lnav-markdown-example.png diff --git a/docs/source/lnav-pretty-view-after.png b/docs/source/lnav-pretty-view-after.png Binary files differnew file mode 100644 index 0000000..7fcee79 --- /dev/null +++ b/docs/source/lnav-pretty-view-after.png diff --git a/docs/source/lnav-pretty-view-before.png b/docs/source/lnav-pretty-view-before.png Binary files differnew file mode 100644 index 0000000..e3a5a13 --- /dev/null +++ b/docs/source/lnav-pretty-view-before.png diff --git a/docs/source/lnav-spectro-cpu-pct.png b/docs/source/lnav-spectro-cpu-pct.png Binary files differnew file mode 100644 index 0000000..a9df2b6 --- /dev/null +++ b/docs/source/lnav-spectro-cpu-pct.png diff --git a/docs/source/lnav-ui.png b/docs/source/lnav-ui.png Binary files differnew file mode 100644 index 0000000..2dcad66 --- /dev/null +++ b/docs/source/lnav-ui.png diff --git a/docs/source/open-error.png b/docs/source/open-error.png Binary files differnew file mode 100644 index 0000000..6454b2d --- /dev/null +++ b/docs/source/open-error.png diff --git a/docs/source/open-help.png b/docs/source/open-help.png Binary files differnew file mode 100644 index 0000000..a11656a --- /dev/null +++ b/docs/source/open-help.png diff --git a/docs/source/open-preview.png b/docs/source/open-preview.png Binary files differnew file mode 100644 index 0000000..b888376 --- /dev/null +++ b/docs/source/open-preview.png diff --git a/docs/source/query-results.png b/docs/source/query-results.png Binary files differnew file mode 100644 index 0000000..14ae5b8 --- /dev/null +++ b/docs/source/query-results.png diff --git a/docs/source/sessions.rst b/docs/source/sessions.rst new file mode 100644 index 0000000..e97e598 --- /dev/null +++ b/docs/source/sessions.rst @@ -0,0 +1,23 @@ + +.. _sessions: + +Sessions +======== + +Session information is stored automatically for the set of files that were +passed in on the command-line and reloaded the next time **lnav** is executed. +The information currently stored is: + +* Position within the files being viewed. +* Active searches for each view. +* :ref:`Log filters<filtering>`. +* :ref:`Highlights<highlight>`. +* :ref:`Hidden files<hide_file>`. +* :ref:`Hidden fields<hide_fields>`. + +Bookmarks and log-time adjustments are stored separately on a per-file basis. +Note that the bookmarks are associated with files based on the content of the +first line of the file so that they are preserved even if the file has been +moved from its current location. + +Session data is stored in the :file:`~/.lnav` directory. diff --git a/docs/source/sql-help.png b/docs/source/sql-help.png Binary files differnew file mode 100644 index 0000000..44901a4 --- /dev/null +++ b/docs/source/sql-help.png diff --git a/docs/source/sqlext.rst b/docs/source/sqlext.rst new file mode 100644 index 0000000..221b100 --- /dev/null +++ b/docs/source/sqlext.rst @@ -0,0 +1,184 @@ +.. _sql-ext: + +SQLite Interface +================ + +Log analysis in **lnav** can be done using the SQLite interface. Log messages +can be accessed via `virtual tables <https://www.sqlite.org/vtab.html>`_ that +are created for each file format. The tables have the same name as the log +format and each message is its own row in the table. For example, given the +following log message from an Apache access log:: + + 127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 + +These columns would be available for its row in the :code:`access_log` table: + +.. csv-table:: + :class: query-results + :header-rows: 1 + + log_line,log_part,log_time,log_idle_msecs,log_level,log_mark,log_comment,log_tags,log_filters,c_ip,cs_method,cs_referer,cs_uri_query,cs_uri_stem,cs_user_agent,cs_username,cs_version,sc_bytes,sc_status + 0,<NULL>,2000-10-10 13:55:36.000,0,info,1,<NULL>,<NULL>,<NULL>,127.0.0.1,GET,<NULL>,<NULL>,/apache_pb.gif,<NULL>,frank,HTTP/1.0,2326,200 + +.. note:: Some columns are hidden by default to reduce the amount of noise in + results, but they can still be accessed when explicitly used. The hidden + columns are: :code:`log_path`, :code:`log_text`, :code:`log_body`, and + :code:`log_raw_text`. + +You can activate the SQL prompt by pressing the :kbd:`;` key. At the +prompt, you can start typing in the desired SQL statement and/or double-tap +:kbd:`TAB` to activate auto-completion. A help window will appear above +the prompt to guide you in the usage of SQL keywords and functions. + +.. figure:: sql-help.png + :align: center + + Screenshot of the online help for the SQL prompt. + +.. figure:: group_concat-help.png + :align: center + + Screenshot of the online help for the :code:`group_concat()` function. + +A simple query to perform on an Apache access log might be to get the average +and maximum number of bytes returned by the server, grouped by IP address: + +.. code-block:: custsqlite + + ;SELECT c_ip, avg(sc_bytes), max(sc_bytes) FROM access_log GROUP BY c_ip + +After pressing :kbd:`Enter`, SQLite will execute the query using **lnav**'s +virtual table implementation to extract the data directly from the log files. +Once the query has finished, the main window will switch to the DB view to +show the results. Press :kbd:`q` to return to the log view and press :kbd:`v` +to return to the log view. If the SQL results contain a +:code:`log_line` column, you can press to :kbd:`Shift` + :kbd:`V` to +switch between the DB view and the log + +.. figure:: query-results.png + :align: center + + Screenshot of the SQL results view. + +The DB view has the following display features: + +* Column headers stick to the top of the view when scrolling. +* A stacked bar chart of the numeric column values is displayed underneath the + rows. Pressing :kbd:`TAB` will cycle through displaying no columns, each + individual column, or all columns. +* JSON columns in the top row can be pretty-printed by pressing :kbd:`p`. + The display will show the value and JSON-Pointer path that can be passed to + the `jget`_ function. + + +Log Tables +---------- + +Each log format has its own database table that can be used to access log +messages that match that format. The table name is the same as the format +name, for example, the :code:`syslog_log` format will have a table that is +also named :code:`syslog_log`. There is also an :code:`all_logs` table +that provides access to all messages from all formats. + +.. note:: Only the displayed log messages are reflected in the SQLite + interface. Any log messages that have been filtered out are not + accessible. + +The columns in the log tables are made up of several builtins along with +the values captured by the log format specification. Use the :code:`.schema` +command in the SQL prompt to examine a dump of the current database schema. + +The following columns are builtin and included in a :code:`SELECT *`: + + :log_line: The line number for the message in the log view. + :log_part: The partition the message is in. This column can be changed by + an :code:`UPDATE` or the :ref:`:parition-name<partition_name>` command. + :log_time: The adjusted timestamp for the log message. This time can differ + from the log message's time stamp if it arrived out-of-order and the log + format expects log files to be time-ordered. + :log_actual_time: The log messages original timestamp in the file. + :log_idle_msecs: The difference in time between this messages and the + previous. The unit of time is milliseconds. + :log_level: The log message level. + :log_mark: True if the log message was marked by the user. + :log_comment: The comment for the message. This column can be changed by + an :code:`UPDATE` or the :ref:`:comment<comment>` command. + :log_tags: A JSON list of tags for the message. This column can be changed + by an :code:`UPDATE` or the :ref:`:tag<tag>` command. + :log_filters: A JSON list of filter IDs that matched this message + +The following columns are builtin and are hidden, so they will *not* be +included in a :code:`SELECT *`: + + :log_time_msecs: The adjusted timestamp for the log message as the number of + milliseconds from the epoch. This column can be more efficient to use for + time-related operations, like :ref:`timeslice()<timeslice>`. + :log_path: The path to the log file this message is from. + :log_text: The full text of the log message. + :log_body: The body of the log message. + :log_raw_text: The raw text of this message from the log file. In this case + of JSON and CSV logs, this will be the exact line of JSON-Line and CSV + text from the file. + +Extensions +---------- + +To make it easier to analyze log data from within **lnav**, there are several +built-in extensions that provide extra functions and collators beyond those +`provided by SQLite <http://www.sqlite.org/lang_corefunc.html>`_. The majority +of the functions are from the +`extensions-functions.c <http://www.sqlite.org/contrib>`_ file available from +the `sqlite.org <http://sqlite.org>`_ web site. + +.. tip:: You can include a SQLite database file on the command-line and use + **lnav**'s interface to perform queries. The database will be attached with + a name based on the database file name. + +Commands +-------- + +A SQL command is an internal macro implemented by lnav. + +* :code:`.schema` - Open the schema view. This view contains a dump of the + schema for the internal tables and any tables in attached databases. +* :code:`.msgformats` - Executes a canned query that groups and counts log + messages by the format of their message bodies. This command can be useful + for quickly finding out the types of messages that are most common in a log + file. + +Variables +--------- + +The following variables are available in SQL statements: + +* :code:`$LINES` - The number of lines in the terminal window. +* :code:`$COLS` - The number of columns in the terminal window. + +Environment +----------- + +Environment variables can be accessed in queries using the usual syntax of +:code:`$VAR_NAME`. For example, to read the value of the "USER" variable, you +can write: + +.. code-block:: custsqlite + + ;SELECT $USER + +.. _collators: + +Collators +--------- + +* **naturalcase** - Compare strings "naturally" so that number values in the + string are compared based on their numeric value and not their character + values. For example, "foo10" would be considered greater than "foo2". +* **naturalnocase** - The same as naturalcase, but case-insensitive. +* **ipaddress** - Compare IPv4/IPv6 addresses. + +Reference +--------- + +The following is a reference of the SQL syntax and functions that are available: + +.. include:: ../../src/internals/sql-ref.rst diff --git a/docs/source/sqltab.rst b/docs/source/sqltab.rst new file mode 100644 index 0000000..2531390 --- /dev/null +++ b/docs/source/sqltab.rst @@ -0,0 +1,263 @@ +.. _sql-tab: + +SQLite Tables Reference +======================= + +In addition to the tables generated for each log format, **lnav** includes +the following tables/views: + +* `environ`_ +* `lnav_events`_ +* `lnav_file`_ +* `lnav_file_metadata`_ +* `lnav_user_notifications`_ +* `lnav_views`_ +* `lnav_views_echo`_ +* `lnav_view_files`_ +* `lnav_view_stack`_ +* `lnav_view_filters`_ +* `lnav_view_filter_stats`_ +* `lnav_view_filters_and_stats`_ +* `all_logs`_ +* `http_status_codes`_ +* `regexp_capture(<string>, <regex>)`_ + +These extra tables provide useful information and can let you manipulate +**lnav**'s internal state. You can get a dump of the entire database schema +by executing the '.schema' SQL command, like so:: + + ;.schema + +environ +------- + +The **environ** table gives you access to the **lnav** process' environment +variables. You can :code:`SELECT`, :code:`INSERT`, and :code:`UPDATE` +environment variables, like so: + +.. code-block:: custsqlite + + ;SELECT * FROM environ WHERE name = 'SHELL' + name value + SHELL /bin/tcsh + + ;UPDATE environ SET value = '/bin/sh' WHERE name = 'SHELL' + +Environment variables can be used to store simple values or pass values +from **lnav**'s SQL environment to **lnav**'s commands. For example, the +:code:`:open` command will do variable substitution, so you can insert a variable +named "FILENAME" and then open it in **lnav** by referencing it with +"$FILENAME": + +.. code-block:: custsqlite + + ;INSERT INTO environ VALUES ('FILENAME', '/path/to/file') + :open $FILENAME + + +.. _table_lnav_events: + +lnav_events +----------- + +The :code:`lnav_events` table allows you to react to events that occur while +**lnav** is running using SQLite triggers. For example, when a file is +opened, a row is inserted into the :code:`lnav_events` table that contains +a timestamp and a JSON object with the event ID and the path of the file. +The following columns are available in this table: + + :ts: The timestamp of the event. + :content: A JSON object that contains the event information. See the + :ref:`event_reference` for more information about the types + of events that are available. + +lnav_file +--------- + +The :code:`lnav_file` table allows you to examine and perform limited updates to +the metadata for the files that are currently loaded into **lnav**. The +following columns are available in this table: + + :device: The device the file is stored on. + :inode: The inode for the file on the device. + :filepath: If this is a real file, it will be the absolute path. Otherwise, + it is a symbolic name. If it is a symbolic name, it can be UPDATEd so that + this file will be considered when saving and loading session information. + :format: The log file format for the file. + :lines: The number of lines in the file. + :time_offset: The millisecond offset for timestamps. This column can be + UPDATEd to change the offset of timestamps in the file. + +lnav_file_metadata +------------------ + +The :code:`lnav_file_metadata` table gives access to metadata associated with a +loaded file. Currently, + +:filepath: The path to the file. +:descriptor: A descriptor that identifies the source of the metadata. The + following descriptors are supported: + + :net.zlib.gzip.header: The header on a gzipped file. The content is a + JSON object with the following properties: + + :name: The original name of the file. + :mtime: The last modified time of the file when it was compressed. + :comment: A text comment associated with the file. + :net.daringfireball.markdown.frontmatter: The frontmatter on a + markdown file. If the frontmatter is delimited by three dashes + (:code:`---`), the :code:`mimetype` will be :code:`application/yaml`. + If the frontmatter is delimited by three pluses (:code:`+++`) the + :code:`mimetype` will be :code:`application/toml`. +:mimetype: The MIME type of the metadata. +:content: The metadata itself. + +.. _table_lnav_user_notifications: + +lnav_user_notifications +----------------------- + +The :code:`lnav_user_notifications` table allows you to display a custom message +in the top-right corner of the UI. For example, to display "Hello, World!", +you can enter: + +.. code-block:: custsqlite + + ;REPLACE INTO lnav_user_notifications (message) VALUES ('Hello, World!') + +There are additional columns to have finer control of what is displayed and +when: + + :id: The unique ID for the message, defaults to "org.lnav.user". This is + the primary key for the table, so more than one type of message is not + allowed. + :priority: The priority of the message. Higher priority messages will be + displayed until they are cleared or are expired. + :created: The time the message was created. + :expiration: The time when the message should expire or NULL if it should + not automatically expire. + :views: A JSON array of view names where the message is applicable or NULL + if the message should be shown in all views. + :message: The message itself. + +This table will most likely be used in combination with :ref:`Events` and the +`lnav_views_echo`_ table. + +lnav_views +---------- + +The :code:`lnav_views` table allows you to SELECT and UPDATE information related +to **lnav**'s "views" (e.g. log, text, ...). The following columns are +available in this table: + +:name: The name of the view. +:top: The line number at the top of the view. This value can be UPDATEd to + move the view to the given line. +:left: The left-most column number to display. This value can be UPDATEd to + move the view left or right. +:height: The number of lines that are displayed on the screen. +:inner_height: The number of lines of content being displayed. +:top_time: The timestamp of the top line in the view or NULL if the view is + not time-based. This value can be UPDATEd to move the view to the given + time. +:top_file: The file the top line in the view is from. +:paused: Indicates if the view is paused and will not load new data. +:search: The search string for this view. This value can be UPDATEd to + initiate a text search in this view. +:filtering: Indicates if the view is applying filters. +:movement: The movement mode, either 'top' or 'cursor'. +:top_meta: A JSON object that contains metadata related to the top line + in the view. +:selection: The number of the line that is focused for selection. + +lnav_views_echo +--------------- + +The :code:`lnav_views_echo` table is a real SQLite table that you can create +TRIGGERs on in order to react to users moving around in a view. + +.. note:: + + The table is periodically updated to reflect the current state of the views. + The changes are *not* performed immediately after the user action. + +lnav_view_files +--------------- + +The :code:`lnav_view_files` table provides access to details about the files +displayed in a particular view. The main purpose of this table is to allow +you to programmatically control which files are shown / hidden in the view. +The following columns are available in this table: + +:view_name: The name of the view. +:filepath: The file's path. +:visible: Determines whether the file is visible in the view. This column + can be changed using an :code:`UPDATE` statement to hide or show the file. + +lnav_view_stack +--------------- + +The :code:`lnav_view_stack` table allows you to :code:`SELECT` and :code:`DELETE` +from the stack of **lnav** "views" (e.g. log, text, ...). The following columns +are available in this table: + + :name: The name of the view. + +.. _table_lnav_view_filters: + +lnav_view_filters +----------------- + +The :code:`lnav_view_filters` table allows you to manipulate the filters in the +**lnav** views. The following columns are available in this table: + + :view_name: The name of the view the filter is applied to. + :filter_id: The filter identifier. This will be assigned on insertion. + :enabled: Indicates whether this filter is enabled or disabled. + :type: The type of filter, either 'in' or 'out'. + :pattern: The regular expression to filter on. + +This table supports :code:`SELECT`, :code:`INSERT`, :code:`UPDATE`, and +:code:`DELETE` on the table rows to read, create, update, and delete +filters for the views. + +lnav_view_filter_stats +---------------------- + +The :code:`lnav_view_filter_stats` table allows you to get information about how +many lines matched a given filter. The following columns are available in +this table: + + :view_name: The name of the view. + :filter_id: The filter identifier. + :hits: The number of lines that matched this filter. + +This table is read-only. + +lnav_view_filters_and_stats +--------------------------- + +The :code:`lnav_view_filters_and_stats` view joins the :code:`lnav_view_filters` +table with the :code:`lnav_view_filter_stats` table into a single view for ease of use. + +all_logs +-------- + +.. f0:sql.tables.all_logs + +The :code:`all_logs` table lets you query the format derived from the **lnav** +log message parser that is used to automatically extract data, see +:ref:`data-ext` for more details. + +http_status_codes +----------------- + +The :code:`http_status_codes` table is a handy reference that can be used to turn +HTTP status codes into human-readable messages. + +regexp_capture(<string>, <regex>) +--------------------------------- + +The :code:`regexp_capture()` table-valued function applies the regular expression +to the given string and returns detailed results for the captured portions of +the string. diff --git a/docs/source/ui.rst b/docs/source/ui.rst new file mode 100644 index 0000000..a718d06 --- /dev/null +++ b/docs/source/ui.rst @@ -0,0 +1,327 @@ +.. _ui: + +User Interface +============== + +The **lnav** TUI displays the content of the current "view" in the middle, +with status bars above and below, and the interactive prompt as the last line. + +.. figure:: lnav-ui.png + :align: center + :alt: Screenshot of lnav showing a mix of syslog and web access_log messages. + + Screenshot of **lnav** viewing syslog and web access_log messages. + +The default view shows the log messages from the log files that have been +loaded. There are other views for displaying content like plaintext files +and SQL results. The :ref:`ui_views` section describes the characteristics of +each view in more detail. You can switch to the different views using the +hotkeys described in the :ref:`hotkeys_display` section or by pressing +:kbd:`ENTER` to activate the breadcrumb bar, moving to the first crumb, and +then selecting the desired view. You can switch back to the previous view by +pressing :kbd:`q`. You can switch forward to the new view by pressing +:kbd:`a`. If the views are time-based (e.g. log and histogram), pressing +:kbd:`Shift` + :kbd:`q` and :kbd:`Shift` + :kbd:`a` will synchronize the top +times in the views. + +**lnav** provides many operations to work with the log/text data in the +main view. For example, you can add comments and tags to log messages. +By default, the top line is used as the reference point to edit the +comment or tags. Alternatively, you can press :kbd:`Ctrl` + :kbd:`x` +to switch to "cursor" mode where the "focused" line is highlighted and +most operations now work with that line. When in "cursor" mode, the +:kbd:`↑` and :kbd:`↓` keys now move the focused line instead of scrolling +the view. Jumping to bookmarks, like errors, will also move the focused +line instead of moving the next error to the top of the view. + +The right side of the display has a proportionally sized 'scrollbar' that +shows: + +* the current position in the file; +* the locations of errors/warnings in the log files by using red or yellow + coloring; +* the locations of search hits by using a tick-mark pointing to the left; +* the locations of bookmarks by using a tick-mark pointing to the right. + +Top Status Bar +-------------- + +The top status bar shows the current time and messages stored in the +:ref:`table_lnav_user_notifications` table. + +Below the top status bar is the breadcrumb bar that displays the semantic +location of the focused line in the main view. For example, within a +pretty-printed JSON document, it will show the path to property at the top +of the view. The actual content of the bar depends on the current view and +will be updated as you navigate around the main view. The bar can also be +used to navigate around the document by focusing on it. + +Breadcrumb Bar +-------------- + +.. figure:: lnav-breadcrumbs-help.png + :align: center + :figwidth: 90% + + Screenshot of the breadcrumb bar focused and navigating the help text + +To focus on the breadcrumb bar, press :kbd:`ENTER`. The :kbd:`←`/:kbd:`→` +cursor keys can be used to select a crumb and the :kbd:`↑`/:kbd:`↓` keys can +be used select a value of that crumb. To accept a value and drop focus on the +bar, press :kbd:`ENTER`. To accept a value and move to the next crumb, press +:kbd:`→`. Using :kbd:`→` makes it quicker to drill down into a document +without having to constantly switch focus. To drop focus on the bar without +accepting anything, press :kbd:`Escape`. + +There are three types of crumbs: + +* a dropdown where one of a limited set of values can be selected; +* a combobox where a value can be entered directly or selected; +* a numeric input for entering array indexes. + +When a dropdown or combobox is selected, you can type part of the desired value +to filter the list of values. For example, the first crumb is always the +current view, typing in "hi" will filter the list down to the "HIST" value. + +Configuration Panels +-------------------- + +.. figure:: lnav-config-header.png + :align: center + :figwidth: 90% + + Screenshot of the header for the configuration panels when they are hidden. + +After the main view content, there is a header bar for two configuration +panels: Files and Filters. These panels provide visual access to parts of +lnav's configuration. To access the panels, press the :kbd:`TAB` key. +To hide the panels again, press :kbd:`q`. + +.. figure:: lnav-files-panel.png + :align: center + :figwidth: 90% + + Screenshot of the files panel showing the loaded files. + +The Files panel is open initially to display progress in loading files. +The following information can be displayed for each file: + +* the "unique" portion of the path relative to the other files; +* the amount of data that has been indexed; +* the date range of log messages contained in the file; +* the errors that were encountered while trying to index the file; +* the notes recorded for files where some automatic action was taken, + like hiding the file if it was seen as a duplicate of another file. + +.. figure:: lnav-filters-panel.png + :align: center + :figwidth: 90% + + Screenshot of the filters panel showing an OUT and a disabled IN filter. + +If the view supports filtering, there will be a status line showing the +following: + +* the number of enabled filters and the total number of filters; +* the number of lines that are **not** displayed because of filtering. + +To edit the filters, you can press TAB to change the focus from the main +view to the filter editor. The editor allows you to create, enable/disable, +and delete filters easily. + +Bottom Status Bar +----------------- + +The second to last line is the bottom status bar, which shows the following: + +* the line number of the focused line, starting from zero; +* the location within the view, as a percentage; +* the current search hit, the total number of hits, and the search term; +* the loading indicator. + +When the interactive prompt is active, this bar can show the prompt +description, help text, or error message. + +Prompt +------ + +Finally, the last line on the display is where you can enter search +patterns and execute internal commands, such as converting a +unix-timestamp into a human-readable date. The following key-presses +will activate a corresponding prompt: + +* :kbd:`/` - The search prompt. You can enter a PCRE2-flavored regular + expression to search for in the current view. +* :kbd:`:` - The command prompt. Commands are used to perform common + operations. +* :kbd:`;` - The SQL prompt. SQL queries can be used for log analysis + and manipulating **lnav**'s state. +* :kbd:`|` - The script prompt. Enter a path to the lnav script to + execute, along with the arguments to pass in. + +The command-line is by the readline library, so the usual set of keyboard +shortcuts can be used for editing and moving within the command-line. + +.. _ui_views: + +Views +----- + +The accessible content within lnav is separated into the following views. + +LOG +^^^ + +The log view displays the log messages from any loaded log files in time +order. This view will be shown by default if any log messages are available. + +On color displays, the log messages will be highlighted as follows: + +* Errors will be colored in red; +* warnings will be yellow; +* search hits are reverse video; +* various color highlights will be applied to: IP addresses, SQL keywords, + XML tags, file and line numbers in Java backtraces, and quoted strings; +* "identifiers" in the messages will be randomly assigned colors based on their + content (works best on "xterm-256color" terminals). + +.. note:: + + If the coloring is too much for your tastes, you can change to the + "grayscale" theme by entering the following command: + + .. code-block:: lnav + + :config /ui/theme grayscale + +.. note:: + + If a log message has a timestamp that is out-of-order with its neighboring + messages, the timestamp will be highlighted in yellow. When one of these + messages is at the top of the log view, an overlay will display the + difference between the "actual time" and the "received time". The "actual + time" is the original textual timestamp. The "received time" is the time + of an earlier message that is larger than this log message's time. + +The source file name for each message can be displayed by scrolling left. +Scrolling left once will show the shortened version of the file name relative +to the other files that are loaded. In the shortened version, the unique +portion of the file name will be in square brackets. Scrolling left a second +time will show the full path. + +The breadcrumb bar will show the following crumbs: + +* the timestamp for the focused line; +* the log format for the focused line; +* the name of the file the focused line was pulled from; +* the "operation ID" of the focused log message, if it is supported by the log + format. + +These crumbs are interactive and can be used to navigate to different parts +of the log view. For example, selecting a different value in the log format +crumb will jump to the first message with that format. + +TEXT +^^^^ + +The text view displays files for which lnav could not detect any log messages. + +Press :kbd:`t` to switch to the text view. While in the text view, you can +press :kbd:`f` or :kbd:`Shift` + :kbd:`F` to switch to the next / previous +text file. + +Markdown +"""""""" + +Files with an :code:`.md` (or :code:`.markdown`) extension will be treated as +Markdown files and rendered separately. + + .. figure:: lnav-markdown-example.png + :align: center + + Viewing the **lnav** :file:`README.md` file. + + +DB +^^ + +The DB view shows the results of queries done through the SQLite interface. +You can execute a query by pressing :kbd:`;` and then entering a SQL statement. + +Press :kbd:`v` to switch to the database result view. + +HELP +^^^^ + +The help view displays the builtin help text. While in the help view, the +breadcrumb bar can be used to navigate to different sections of the document. + +Press :kbd:`?` to switch to the help view. + +HIST +^^^^ + +The histogram view displays a stacked bar chart of messages over time +classified by their log level and whether they've been bookmarked. + +Press :kbd:`i` to switch back and forth to the histogram view. You +can also press :kbd:`Shift` + :kbd:`i` to toggle the histogram view +while synchronizing the top time. While in the histogram view, +pressing :kbd:`z` / :kbd:`Shift` + :kbd:`z` will zoom in/out. + +PRETTY +^^^^^^ + +The pretty-print view takes the text displayed in the current view and shows +the result of a pretty-printer run on that text. For example, if a log +message contained an XML message on a single line, the pretty-printer would +break the XML across multiple lines with appropriate indentation. + +.. figure:: lnav-pretty-view-before.png + :align: center + :figwidth: 90% + + Screenshot of a log message with a flat JSON object. + +.. figure:: lnav-pretty-view-after.png + :align: center + :figwidth: 90% + + Screenshot of the same log message in the PRETTY view. The JSON object + is now indented for easier reading. + +Press :kbd:`Shift` + :kbd:`P` to switch to the pretty-print view. + +SCHEMA +^^^^^^ + +The schema view displays the current schema of the builtin SQLite database. + +Press :kbd:`;` to enter the SQL prompt and then enter :code:`.schema` to +open the schema view. + +SPECTRO +^^^^^^^ + +The spectrogram view is a "three"-dimensional display of data points of a log +field or a SQL query column. The dimensions are time on the Y axis, the range +of data point values on the X axis, and the number of data points as a color. +For example, if you were to visualize process CPU usage over time, the range +of values on the X axis would be CPU percentages and there would be colored +blocks at each point on the line where a process had that CPU percentage, like +so + +.. figure:: lnav-spectro-cpu-pct.png + :align: center + + Screenshot of the **lnav** spectrogram view showing CPU usage of processes. + +The colors correspond to the relative number of data points in a bucket. +The legend overlaid at the top line in the view shows the counts of data +points that are in a particular color, with green having the fewest number of +data points, yellow the middle, and red the most. You can select a particular +bucket using the cursor keys to see the exact number of data points and the +range of values. The panel at the bottom of the view shows the data points +themselves from the original source, the log file or the SQL query results. +You can press :kbd:`TAB` to focus on the details panel so you can scroll +around and get a closer look at the values. diff --git a/docs/source/usage.rst b/docs/source/usage.rst new file mode 100644 index 0000000..d43ed52 --- /dev/null +++ b/docs/source/usage.rst @@ -0,0 +1,291 @@ +.. _usage: + +Usage +===== + +This chapter contains an overview of how to use **lnav**. + + +Basic Controls +-------------- + +Like most file viewers, scrolling through files can be done with the usual +:ref:`hotkeys<hotkeys>`. For non-trivial operations, you can enter the +:ref:`command<commands>` prompt by pressing :kbd:`:`. To analyze data in a +log file, you can enter the :ref:`SQL prompt<sql-ext>` by pressing :kbd:`;`. + +.. tip:: + + Check the bottom right corner of the screen for tips on hotkeys that might + be useful in the current context. + + .. figure:: hotkey-tips.png + :align: center + + When **lnav** is first open, it suggests using :kbd:`e` and + :kbd:`Shift` + :kbd:`e` to jump to error messages. + + +Viewing Files +------------- + +The files to view in **lnav** can be given on the command-line or passed to the +:ref:`:open<open>` command. A +`glob pattern <https://en.wikipedia.org/wiki/Glob_(programming)>`_ can be given +to watch for files with a common name. If the path is a directory, all of the +files in the directory will be opened and the directory will be monitored for +files to be added or removed from the view. If the path is an archive or +compressed file (and lnav was built with libarchive), the archive will be +extracted to a temporary location and the files within will be loaded. The +files that are found will be scanned to identify their file format. Files +that match a log format will be collated by time and displayed in the LOG +view. Plain text files can be viewed in the TEXT view, which can be accessed +by pressing :kbd:`t`. + + +Archive Support +^^^^^^^^^^^^^^^ + +.. f0:archive + +If **lnav** is compiled with `libarchive <https://www.libarchive.org>`_, +any files to be opened will be examined to see if they are a supported archive +type. If so, the contents of the archive will be extracted to the +:code:`$TMPDIR/lnav-user-${UID}-work/archives/` directory. Once extracted, the +files within will be loaded into lnav. To speed up opening large amounts of +files, any file that meets the following conditions will be automatically +hidden and not indexed: + +* Binary files +* Plain text files that are larger than 128KB +* Duplicate log files + +The unpacked files will be left in the temporary directory after exiting +**lnav** so that opening the same archive again will be faster. Unpacked +archives that have not been accessed in the past two days will be automatically +deleted the next time **lnav** is started. + + +.. _remote: + +Remote Files +^^^^^^^^^^^^ + +Files on remote machines can be viewed and tailed if you have access to the +machines via SSH. First, make sure you can SSH into the remote machine without +any interaction by: 1) accepting the host key as known and 2) copying your +identity's public key to the :file:`.ssh/authorized_keys` file on the remote +machine. Once the setup is complete, you can open a file on a remote host +using the same syntax as :command:`scp(1)` where the username and host are +given, followed by a colon, and then the path to the files, like so:: + + [user@]host:/path/to/logs + +For example, to open :file:`/var/log/syslog.log` on "host1.example.com" as the +user "dean", you would write: + +.. prompt:: bash + + lnav dean@host1.example.com:/var/log/syslog.log + +Remote files can also be opened using the :ref:`:open<open>` command. Opening +a remote file in the TUI has the advantage that the file path can be +:kbd:`TAB`-completed and a preview is shown of the first few lines of the +file. + +.. note:: + + If lnav is installed from the `snap <https://snapcraft.io/lnav>`_, you will + need to connect it to the + `ssh-keys plug <https://snapcraft.io/docs/ssh-keys-interface>`_ using the + following command: + + .. prompt:: bash + + sudo snap connect lnav:ssh-keys + +.. note:: + + Remote file access is implemented by transferring an + `αcτµαlly pδrταblε εxεcµταblε <https://justine.lol/ape.html>`_ to the + destination and invoking it. An APE binary can run on most any x86_64 + machine and OS (i.e. MacOS, Linux, FreeBSD, Windows). The binary is + baked into the lnav executable itself, so there is no extra setup that + needs to be done on the remote machine. + + The binary file is named ``tailer.bin.XXXXXX`` where *XXXXXX* is 6 random digits. + The file is, under normal circumstancies, deleted immediately. + +Searching +--------- + +Any log messages that are loaded into **lnav** are indexed by time and log +level (e.g. error, warning) to make searching quick and easy with +:ref:`hotkeys<hotkeys>`. For example, pressing :kbd:`e` will jump to the +next error in the file and pressing :kbd:`Shift` + :kbd:`e` will jump to +the previous error. Plain text searches can be done by pressing :kbd:`/` +to enter the search prompt. A regular expression can be entered into the +prompt to start a search through the current view. + + +.. _filtering: + +Filtering +--------- + +To reduce the amount of noise in a log file, **lnav** can hide log messages +that match certain criteria. The following sub-sections explain ways to go +about that. + + +Regular Expression Match +^^^^^^^^^^^^^^^^^^^^^^^^ + +If there are log messages that you are not interested in, you can do a +"filter out" to hide messages that match a pattern. A filter can be created +using the interactive editor, the :ref:`:filter-out<filter_out>` command, or +by doing an :code:`INSERT` into the +:ref:`lnav_view_filters<table_lnav_view_filters>` table. + +If there are log messages that you are only interested in, you can do a +"filter in" to only show messages that match a pattern. The filter can be +created using the interactive editor, the :ref:`:filter-in<filter_in>` command, +or by doing an :code:`INSERT` into the +:ref:`lnav_view_filters<table_lnav_view_filters>` table. + + +SQLite Expression +^^^^^^^^^^^^^^^^^ + +Complex filtering can be done by passing a SQLite expression to the +:ref:`:filter-expr<filter_expr>` command. The expression will be executed for +every log message and if it returns true, the line will be shown in the log +view. + + +Time +^^^^ + +To limit log messages to a given time frame, the +:ref:`:hide-lines-before<hide_lines_before>` and +:ref:`:hide-lines-after<hide_lines_after>` commands can be used to specify +the beginning and end of the time frame. + + +Log level +^^^^^^^^^ + +To hide messages below a certain log level, you can use the +:ref:`:set-min-log-level<set_min_log_level>` command. + +.. _search_tables: + +Search Tables +------------- + +Search tables allow you to access arbitrary data in log messages through +SQLite virtual tables. If there is some data in a log message that you can +match with a regular expression, you can create a search-table that matches +that data and any capture groups will be plumbed through as columns in the +search table. + +Creating a search table can be done interactively using the +:ref:`:create-search-table<create_search_table>` command or by adding it to +a :ref:`log format definition<log_formats>`. The main difference between +the two is that tables defined as part of a format will only search messages +from log files with that format and the tables will include log message +columns defined in that format. Whereas a table created with the command +will search messages from all different formats and no format-specific +columns will be included in the table. + +.. _taking_notes: + +Taking Notes +------------ + +A few of the columns in the log tables can be updated on a row-by-row basis to +allow you to take notes. The majority of the columns in a log table are +read-only since they are backed by the log files themselves. However, the +following columns can be changed by an :code:`UPDATE` statement: + +* **log_part** - The "partition" the log message belongs to. This column can + also be changed by the :ref:`:partition-name<partition_name>` command. +* **log_mark** - Indicates whether the line has been bookmarked. +* **log_comment** - A free-form text field for storing commentary. This + column can also be changed by the :ref:`:comment<comment>` command. +* **log_tags** - A JSON list of tags associated with the log message. This + column can also be changed by the :ref:`:tag<tag>` command. + +While these columns can be updated by through other means, using the SQL +interface allows you to make changes automatically and en masse. For example, +to bookmark all lines that have the text "something interesting" in the log +message body, you can execute: + +.. code-block:: custsqlite + + ;UPDATE all_logs SET log_mark = 1 WHERE log_body LIKE '%something interesting%' + +As a more advanced example of the power afforded by SQL and **lnav**'s virtual +tables, we will tag log messages where the IP address bound by dhclient has +changed. For example, if dhclient reports "bound to 10.0.0.1" initially and +then reports "bound to 10.0.0.2", we want to tag only the messages where the +IP address was different from the previous message. While this can be done +with a single SQL statement [#]_, we will break things down into a few steps for +this example. First, we will use the :ref:`:create-search-table<create_search_table>` +command to match the dhclient message and extract the IP address: + +.. code-block:: lnav + + :create-search-table dhclient_ip bound to (?<ip>[^ ]+) + +The above command will create a new table named :code:`dhclient_ip` with the +standard log columns and an :code:`ip` column that contains the IP address. +Next, we will create a view over the :code:`dhclient_ip` table that returns +the log message line number, the IP address from the current row and the IP +address from the previous row: + +.. code-block:: custsqlite + + ;CREATE VIEW IF NOT EXISTS dhclient_ip_changes AS SELECT log_line, ip, lag(ip) OVER (ORDER BY log_line) AS prev_ip FROM dhclient_ip + +Finally, the following :code:`UPDATE` statement will concatenate the tag +"#ipchanged" onto the :code:`log_tags` column for any rows in the view where +the current IP is different from the previous IP: + +.. code-block:: custsqlite + + ;UPDATE syslog_log SET log_tags = json_concat(log_tags, '#ipchanged') WHERE log_line IN (SELECT log_line FROM dhclient_ip_changes WHERE ip != prev_ip) + +Since the above can be a lot to type out interactively, you can put these +commands into a :ref:`script<scripts>` and execute that script with the +:kbd:`\|` hotkey. + +.. [#] The expression :code:`regexp_match('bound to ([^ ]+)', log_body) as ip` + can be used to extract the IP address from the log message body. + +Sharing Sessions With Others +---------------------------- + +After setting up filters, bookmarks, and making notes, you might want to share +your work with others. If they have access to the same log files, you can +use the :ref:`:export-session-to<export_session_to>` command to write an +executable **lnav** script that will recreate the current session state. The +script contains various SQL statements and **lnav** commands that capture the +current state. So, you should feel free to modify the script or use it as a +reference to learn about more advanced uses of lnav. + +The script will capture the file paths that were explicitly specified and +not the files that were actually opened. For example, if you specified +"/var/log" on the command line, the script will include +:code:`:open /var/log/*` and not an individual open for each file in that +directory. + +Also, in order to support archives of log files, lnav will try to find the +directory where the archive was unpacked and use that as the base for the +:code:`:open` command. Currently, this is done by searching for the top +"README" file in the directory hierarchy containing the files [1]_. The +consumer of the session script can then set the :code:`LOG_DIR_0` (or 1, 2, +...) environment variable to change where the log files will be loaded from. + +.. [1] It is assumed a log archive would have a descriptive README file. + Other heuristics may be added in the future. diff --git a/docs/tutorials/playground/index.md b/docs/tutorials/playground/index.md new file mode 100644 index 0000000..e1516b1 --- /dev/null +++ b/docs/tutorials/playground/index.md @@ -0,0 +1,9 @@ + +# Playground + +Welcome to the **lnav** playground! + +There are some sample files loaded into the log and text views. +Press `q` to switch back to the log view and start exploring. +You can also press `f` in this view to switch to the other +text files that are loaded, like a markdown sample. diff --git a/docs/tutorials/playground/logs/access_log.gz b/docs/tutorials/playground/logs/access_log.gz Binary files differnew file mode 100644 index 0000000..485cf4e --- /dev/null +++ b/docs/tutorials/playground/logs/access_log.gz diff --git a/docs/tutorials/playground/logs/messages.gz b/docs/tutorials/playground/logs/messages.gz Binary files differnew file mode 100644 index 0000000..e6ad012 --- /dev/null +++ b/docs/tutorials/playground/logs/messages.gz diff --git a/docs/tutorials/playground/run.sh b/docs/tutorials/playground/run.sh new file mode 100755 index 0000000..4a6723d --- /dev/null +++ b/docs/tutorials/playground/run.sh @@ -0,0 +1,16 @@ +#!/bin/bash + +export LNAVSECURE=1 +export TERM=xterm-256color + +timeout --foreground --kill-after=30s 10m lnav \ + -d "/tmp/$(echo "playground."$(date "+%Y-%m-%dT%H-%M-%S")".$$.log")" \ + /tutorials/playground/logs \ + /tutorials/playground/text \ + /tutorials/playground/index.md#playground + +if [ $? = 124 ]; then + echo "error: reached connection time limit, reconnect if you're not a bot." +else + echo "Thanks for trying out lnav! Have a nice day!" +fi diff --git a/docs/tutorials/playground/text/markdown-sample.md b/docs/tutorials/playground/text/markdown-sample.md new file mode 100644 index 0000000..4b56cae --- /dev/null +++ b/docs/tutorials/playground/text/markdown-sample.md @@ -0,0 +1,157 @@ +An h1 header +============ + +Paragraphs are separated by a blank line. + +2nd paragraph. *Italic*, **bold**, and `monospace`. Itemized lists +look like: + + * this one + * that one + * the other one + +Note that --- not considering the asterisk --- the actual text +content starts at 4-columns in. + +> Block quotes are +> written like so. +> +> They can span multiple paragraphs, +> if you like. + +Use 3 dashes for an em-dash. Use 2 dashes for ranges (ex., "it's all +in chapters 12--14"). Three dots ... will be converted to an ellipsis. +Unicode is supported. ☺ + + + +An h2 header +------------ + +Here's a numbered list: + + 1. first item + 2. second item + 3. third item + +Note again how the actual text starts at 4 columns in (4 characters +from the left side). Here's a code sample: + + # Let me re-iterate ... + for i in 1 .. 10 { do-something(i) } + +As you probably guessed, indented 4 spaces. By the way, instead of +indenting the block, you can use delimited blocks, if you like: + +~~~ +define foobar() { + print "Welcome to flavor country!"; +} +~~~ + +(which makes copying & pasting easier). You can optionally mark the +delimited block for Pandoc to syntax highlight it: + +~~~python +import time +# Quick, count to ten! +for i in range(10): + # (but not *too* quick) + time.sleep(0.5) + print i +~~~ + + + +### An h3 header ### + +Now a nested list: + + 1. First, get these ingredients: + + * carrots + * celery + * lentils + + 2. Boil some water. + + 3. Dump everything in the pot and follow + this algorithm: + + find wooden spoon + uncover pot + stir + cover pot + balance wooden spoon precariously on pot handle + wait 10 minutes + goto first step (or shut off burner when done) + + Do not bump wooden spoon or it will fall. + +Notice again how text always lines up on 4-space indents (including +that last line which continues item 3 above). + +Here's a link to [a website](https://lnav.org), to a [local +doc](../index.md), and to a [section heading in the current +doc](#an-h2-header). Here's a footnote [^1]. + +[^1]: Footnote text goes here. + +Tables can look like this: + +size material color +---- ------------ ------------ +9 leather brown +10 hemp canvas natural +11 glass transparent + +Table: Shoes, their sizes, and what they're made of + +(The above is the caption for the table.) Pandoc also supports +multi-line tables: + +-------- ----------------------- +keyword text +-------- ----------------------- +red Sunsets, apples, and + other red or reddish + things. + +green Leaves, grass, frogs + and other things it's + not easy being. +-------- ----------------------- + +A horizontal rule follows. + +*** + +Here's a definition list: + +apples + : Good for making applesauce. +oranges + : Citrus! +tomatoes + : There's no "e" in tomatoe. + +Again, text is indented 4 spaces. (Put a blank line between each +term/definition pair to spread things out more.) + +Here's a "line block": + +| Line one +| Line too +| Line tree + +and images can be specified like so: + + + +Inline math equations go in like so: $\omega = d\phi / dt$. Display +math should get its own line and be put in in double-dollarsigns: + +$$I = \int \rho R^{2} dV$$ + +And note that you can backslash-escape any punctuation characters +which you wish to be displayed literally, ex.: \`foo\`, \*bar\*, etc.
\ No newline at end of file diff --git a/docs/tutorials/tutorial-lib/configs/tutorial1/config.json b/docs/tutorials/tutorial-lib/configs/tutorial1/config.json new file mode 100644 index 0000000..4407368 --- /dev/null +++ b/docs/tutorials/tutorial-lib/configs/tutorial1/config.json @@ -0,0 +1,18 @@ +{ + "$schema": "https://lnav.org/schemas/config-v1.schema.json", + "global": { + "lnav_tutorial_name": "tutorial1" + }, + "ui": { + "keymap-defs": { + "default": { + "x79": { + "command": "|lnav-tutorial-key-handler next" + }, + "x59": { + "command": "|lnav-tutorial-key-handler prev" + } + } + } + } +}
\ No newline at end of file diff --git a/docs/tutorials/tutorial-lib/formats/tutorial-lib/lnav-tutorial-key-handler.lnav b/docs/tutorials/tutorial-lib/formats/tutorial-lib/lnav-tutorial-key-handler.lnav new file mode 100644 index 0000000..becadec --- /dev/null +++ b/docs/tutorials/tutorial-lib/formats/tutorial-lib/lnav-tutorial-key-handler.lnav @@ -0,0 +1,29 @@ + +;SELECT filepath AS tutorial_path FROM lnav_file + WHERE filepath GLOB '*/tutorial1/index.md' LIMIT 1 +;SELECT CASE + WHEN $1 = 'next' AND + step < (SELECT max(step) FROM lnav_tutorial_steps WHERE name = $lnav_tutorial_name) + THEN step + 1 + WHEN $1 = 'prev' AND step > 1 THEN step - 1 + ELSE step + END AS new_step + FROM lnav_tutorial_step WHERE name = $lnav_tutorial_name +;SELECT CASE + WHEN $1 = 'next' AND + step = (SELECT max(step) FROM lnav_tutorial_steps WHERE name = $lnav_tutorial_name) + THEN '#conclusion' + ELSE '#step-' || $new_step + END AS new_anchor + FROM lnav_tutorial_step WHERE name = $lnav_tutorial_name +;UPDATE lnav_tutorial_step SET step = $new_step WHERE name = $lnav_tutorial_name +;UPDATE lnav_views SET top_meta = json_object( + 'file', $tutorial_path, + 'anchor', $new_anchor +) + WHERE name = 'text' +:switch-to-view text +;UPDATE lnav_views SET top = 0, left = 0 + WHERE name = 'log' +;REPLACE INTO lnav_user_notifications (id, views, message) + SELECT * FROM lnav_tutorial_log_notification;
\ No newline at end of file diff --git a/docs/tutorials/tutorial-lib/formats/tutorial-lib/tutorial.sql b/docs/tutorials/tutorial-lib/formats/tutorial-lib/tutorial.sql new file mode 100644 index 0000000..2813357 --- /dev/null +++ b/docs/tutorials/tutorial-lib/formats/tutorial-lib/tutorial.sql @@ -0,0 +1,154 @@ + +-- Tracks the current step in the tutorial +CREATE TABLE lnav_tutorial_step +( + name TEXT NOT NULL PRIMARY KEY, + step INTEGER NOT NULL +); + +INSERT INTO lnav_tutorial_step + VALUES ('tutorial1', 1); + +-- A description of each step in the tutorial with its achievements +CREATE TABLE lnav_tutorial_steps +( + name TEXT NOT NULL, + step INTEGER NOT NULL, + achievements TEXT NOT NULL, + PRIMARY KEY (name, step) +); + +-- Tracks the progress through the achievements in a step of the tutorial +CREATE TABLE IF NOT EXISTS lnav_tutorial_progress +( + name TEXT NOT NULL, + step INTEGER NOT NULL, + achieved TEXT NOT NULL, + + PRIMARY KEY (name, step, achieved) +); + +CREATE TABLE IF NOT EXISTS lnav_tutorial_lines +( + name TEXT NOT NULL, + step INTEGER NOT NULL, + view_ptr TEXT NOT NULL, + view_value TEXT NOT NULL, + achievement TEXT NOT NULL, + log_comment TEXT +); + +-- Copy the tutorial data from the markdown frontmatter to +-- the appropriate tables. +CREATE TRIGGER IF NOT EXISTS add_tutorial_data + AFTER INSERT + ON lnav_events + WHEN jget(new.content, '/$schema') = 'https://lnav.org/event-file-format-detected-v1.schema.json' AND + jget(new.content, '/format') = 'text/markdown' +BEGIN + INSERT INTO lnav_tutorial_steps + SELECT jget(tutorial_meta, '/name'), + key + 1, + value + FROM (SELECT yaml_to_json(lnav_file_metadata.content) AS tutorial_meta + FROM lnav_file_metadata + WHERE filepath = jget(new.content, '/filename')) AS meta_content, + json_each(jget(meta_content.tutorial_meta, '/steps')); + + REPLACE INTO lnav_tutorial_lines + SELECT name, + step, + jget(value, '/view_ptr'), + jget(value, '/view_value'), + key, + jget(value, '/comment') + FROM lnav_tutorial_steps, + json_each(achievements) + WHERE jget(value, '/view_ptr') IS NOT NULL; + + REPLACE INTO lnav_user_notifications (id, views, message) + SELECT * + FROM lnav_tutorial_log_notification; +END; + +CREATE TRIGGER IF NOT EXISTS tutorial_move_log_after_load + AFTER INSERT + ON lnav_events + WHEN jget(new.content, '/$schema') = 'https://lnav.org/event-session-loaded-v1.schema.json' +BEGIN + UPDATE lnav_views SET top = 0 WHERE name = 'log'; +END; + +CREATE TRIGGER IF NOT EXISTS lnav_tutorial_view_listener UPDATE OF top + ON lnav_views_echo + WHEN new.name = 'log' +BEGIN + INSERT OR IGNORE INTO lnav_tutorial_progress + SELECT lnav_tutorial_lines.name, + lnav_tutorial_lines.step, + achievement + FROM lnav_tutorial_step, + lnav_tutorial_lines + WHERE lnav_tutorial_step.step = lnav_tutorial_lines.step + AND jget(json_object('top', new.top, + 'left', new.left, + 'search', new.search), + view_ptr) = view_value; + UPDATE all_logs + SET log_comment = (SELECT log_comment + FROM lnav_tutorial_step, + lnav_tutorial_lines + WHERE lnav_tutorial_step.step = lnav_tutorial_lines.step + AND lnav_tutorial_lines.log_comment IS NOT NULL + AND jget(json_object('top', new.top, + 'left', new.left, + 'search', new.search), view_ptr) = view_value) + WHERE log_line = new.top + AND log_comment IS NULL; +END; + +CREATE TABLE lnav_tutorial_message +( + msgid INTEGER PRIMARY KEY, + msg TEXT +); + +CREATE VIEW lnav_tutorial_current_achievements AS +SELECT key AS achievement, value + FROM lnav_tutorial_step, + lnav_tutorial_steps, json_each(lnav_tutorial_steps.achievements) + WHERE lnav_tutorial_step.step = lnav_tutorial_steps.step; + +CREATE VIEW lnav_tutorial_current_progress AS +SELECT achieved + FROM lnav_tutorial_step, + lnav_tutorial_progress + WHERE lnav_tutorial_step.step = lnav_tutorial_progress.step; + +CREATE VIEW lnav_tutorial_remaining_achievements AS +SELECT * + FROM lnav_tutorial_current_achievements + WHERE achievement NOT IN (SELECT * FROM lnav_tutorial_current_progress); + +CREATE VIEW lnav_tutorial_log_notification AS +SELECT * + FROM (SELECT 'org.lnav.tutorial.log' AS id, '["log"]' AS views, jget(value, '/notification') AS message + FROM lnav_tutorial_remaining_achievements + UNION ALL + SELECT 'org.lnav.tutorial.log' AS id, + '["log"]' AS views, + 'Press `y` to go to the next step in the tutorial' AS message) + LIMIT 1; + +CREATE TRIGGER IF NOT EXISTS lnav_tutorial_progress_listener + AFTER INSERT + ON lnav_tutorial_progress +BEGIN + DELETE FROM lnav_user_notifications WHERE id = 'org.lnav.tutorial.log'; + REPLACE INTO lnav_user_notifications (id, views, message) + SELECT * + FROM lnav_tutorial_log_notification; +END; + +REPLACE INTO lnav_user_notifications (id, views, message) + VALUES ('org.lnav.tutorial.text', '["text"]', 'Press `q` to go to the log view') diff --git a/docs/tutorials/tutorial1/index.md b/docs/tutorials/tutorial1/index.md new file mode 100644 index 0000000..a608ba1 --- /dev/null +++ b/docs/tutorials/tutorial1/index.md @@ -0,0 +1,145 @@ +--- +name: tutorial1 +steps: + - move-to-error: + description: "Move to an error" + view_ptr: /top + view_value: 6 + notification: | + Press `e`/`Shift+E` to move through the + <span class="-lnav_log-level-styles_error">errors</span> + comment: | + You found the error! + [Log formats](https://docs.lnav.org/en/latest/formats.html#format-file-reference) + can define the log levels for a given message. + The [theme](https://docs.lnav.org/en/latest/config.html#theme-definitions) defines + how the levels are displayed. + move-to-warning: + description: "Move to a warning" + notification: | + Press `w`/`Shift+W` to move through the + <span class="-lnav_log-level-styles_warning">warnings</span> + view_ptr: /top + view_value: 3 + comment: | + You found the warning! The scrollbar on the right is highlighted + to show the position of + <span class="-lnav_log-level-styles_warning">warnings</span> and + <span class="-lnav_log-level-styles_error">errors</span> in this + view. + - search-for-term: + description: "Search for something" + notification: "Press `/` to search for '1AF9...'" + view_ptr: /search + view_value: 1AF9293A-F42D-4318-BCDF-60234B240955 + move-to-next-hit: + description: "Move to the next hit" + notification: "Press `n`/`Shift+N` to move through the search hits" + view_ptr: /top + view_value: 53 + comment: | + The matching text in a search is highlighted in + <span class="-lnav_styles_search">reverse-video</span>. + However, the text is not always on-screen, so the bar on the + left will also be highlighted. You can then press `>` to + move right to the next (horizontal) search hit. Pressing + `<` will move left to the previous (horizontal) hit or all + the way back to the start of the line. + move-right: + description: "Move to the right" + notification: "Press `>` to move horizontally to view the search hit" + view_ptr: /left + view_value: 582 + - move-to-half-hour: + description: "Move to the next half-hour" + notification: "Press `3`/`Shift+3` to move through the half-hour marks" + view_ptr: /top + view_value: 34 + comment: | + This file is in the _glog_ format and timestamps consist of the + year, month, and day squished together. This log message's + timestamp is March 22nd, 2017. You can see the timestamp for + the top line in the view in the breadcrumb bar. Next, go to the + log messages for the following day using `:goto March 23` or the + breadcrumb bar above. + move-to-timestamp: + description: "Move to a given timestamp" + notification: "Move to '**March 23**' using `:goto` or the breadcrumb bar" + view_ptr: /top + view_value: 79 + comment: | + Many different timestamp formats are recognized as well as + relative times, like `+1h` or `-2h`. +--- +# Tutorial 1 + +Welcome to the first _interactive_ **lnav** tutorial! + +This tutorial will guide you through the basics of navigating log files. +Pressing `q` will display an example log file to try out commands on. +Pressing `y` will return you to the next step in the tutorial. + +## Step 1 + +Finding errors quickly is one of the main use-cases for **lnav**. To +make that quick and easy, **lnav** parses the log messages in log files +as they are loaded and builds indexes of the errors and warnings. You +can then use the following hotkeys to jump to them in the log view: + +| Key | Action | +|-----------|----------------------------------------------------------------------------------| +| `e` | Move to the next <span class="-lnav_log-level-styles_error">error</span> | +| `Shift+E` | Move to the previous <span class="-lnav_log-level-styles_error">error</span> | +| `w` | Move to the next <span class="-lnav_log-level-styles_warning">warning</span> | +| `Shift+W` | Move to the previous <span class="-lnav_log-level-styles_warning">warning</span> | + +To complete this step in the tutorial, you'll need to navigate to the +errors and warnings in the sample log file. You can check the upper-right +↗↗↗ status bar for tips on what you need to do next. Now, press `q` to +switch to the log view and begin navigating the sample log file. + +## Step 2 + +To search for text in files, you can press `/` to enter the search +prompt. To make it easier to search for text that is on-screen, you +can press `TAB` to complete values that are shown on screen. For +example, to search for the UUID "1AF9293A-F42D-4318-BCDF-60234B240955" +that is in one of the error messages, you can enter "1AF9" and then +press `TAB` to complete the rest of the UUID. + +Press `q` to switch to the log view and try searching for the UUID. + +## Step 3 + +To move to a particular time in the logs, you have a few options: + +* The number keys can be used to move to messages at the ten-minute + marks within an hour. For example, pressing `2` will move to the + first message after the next twenty-minute mark, pressing `3` + will move to the next half-hour mark, and so on. +* Pressing `ENTER` to focus on the breadcrumb bar, then you + can press `TAB` (or right-arrow) to move to the time crumb. + With the time crumb selected, you can then type in an absolute + or relative time. Or, you can use the up and down arrow keys + to select a preset relative time. +* Pressing `:` will activate the command prompt, then you can use + the `:goto` command to move to a given timestamp (or line number). + +Press `q` to switch to the log view and try moving to different +times. + +## Conclusion + +That's all for now, thanks for your time! Visit the +[downloads](https://lnav.org/downloads) page to find out how to +download or install **lnav** for your system. The full +documentation is available at https://docs.lnav.org + +Press `q` to switch to the log view and then press `q` again to +exit **lnav**. + +## Colophon + +The source for this tutorial is available here: + +https://github.com/tstack/lnav/tree/master/docs/tutorials/ diff --git a/docs/tutorials/tutorial1/run.sh b/docs/tutorials/tutorial1/run.sh new file mode 100755 index 0000000..6785ac4 --- /dev/null +++ b/docs/tutorials/tutorial1/run.sh @@ -0,0 +1,16 @@ +#!/bin/bash + +export LNAVSECURE=1 +export TERM=xterm-256color + +timeout --foreground --kill-after=30s 5m lnav \ + -d "/tmp/$(echo "tutorial1."$(date "+%Y-%m-%dT%H-%M-%S")".$$.log")" \ + -I /tutorials/tutorial-lib \ + /tutorials/tutorial1/tutorial1.glog \ + /tutorials/tutorial1/index.md#tutorial-1 + +if [ $? = 124 ]; then + echo "error: reached connection time limit, reconnect if you're not a bot." +else + echo "Thanks for trying out lnav! Have a nice day!" +fi diff --git a/docs/tutorials/tutorial1/tutorial1.glog b/docs/tutorials/tutorial1/tutorial1.glog new file mode 100644 index 0000000..b996d68 --- /dev/null +++ b/docs/tutorials/tutorial1/tutorial1.glog @@ -0,0 +1,100 @@ +I20170322 06:58:47.082758 61456 blaster.cc:112] Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. +I20170322 06:58:58.019562 121 demo.cc:123] Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. +I20170322 06:58:59.059175 123 blaster.cc:6782] Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. +W20170322 06:59:16.062826 61456 demo.cc:352] Ut enim ad minim veniam, quis nostrud exercitation 1AF9293A-F42D-4318-BCDF-60234B240955 ullamco laboris nisi ut aliquip ex ea commodo consequat. +I20170322 06:59:28.084062 124 blaster.cc:112] Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. +I20170322 06:59:32.053551 123 loader.cc:13] Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. +E20170322 06:59:53.084969 123 loader.cc:552] Excepteur sint occaecat cupidatat non proident 1AF9293A-F42D-4318-BCDF-60234B240955, sunt in culpa qui officia deserunt mollit anim id est laborum. +I20170322 07:00:00.096693 123 blaster.cc:112] Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. +I20170322 07:00:03.049849 123 demo.cc:352] Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. +I20170322 07:00:08.070575 123 blaster.cc:6782] Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. +I20170322 07:00:23.019849 123 blaster.cc:352] Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. +I20170322 07:00:28.022692 61457 loader.cc:552] Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. +I20170322 07:00:29.058438 61456 blaster.cc:352] Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. +I20170322 07:00:30.028483 123 loader.cc:13] Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. +I20170322 07:00:49.070676 123 demo.cc:352] Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. +I20170322 07:00:56.095214 123 loader.cc:552] Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. +W20170322 07:01:14.042785 123 blaster.cc:112] Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. +I20170322 07:01:31.083704 123 blaster.cc:112] Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. +I20170322 07:01:44.013733 121 blaster.cc:112] Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. +I20170322 07:01:55.024085 121 blaster.cc:112] Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. +I20170322 07:02:02.027811 121 blaster.cc:6782] Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. +I20170322 07:02:14.022939 61456 blaster.cc:112] Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. +I20170322 07:02:30.035925 123 loader.cc:13] Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. +I20170322 07:02:49.024985 123 loader.cc:13] Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. +I20170322 07:03:09.056478 121 blaster.cc:6782] Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. +I20170322 07:03:15.023777 123 demo.cc:352] Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. +I20170322 07:03:32.066107 123 blaster.cc:352] Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. +I20170322 07:03:48.028662 124 blaster.cc:6782] Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. +I20170322 07:03:54.027078 123 demo.cc:123] Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. +I20170322 07:04:09.041478 123 demo.cc:123] Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. +I20170322 07:04:14.068162 121 blaster.cc:6782] Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. +I20170322 07:04:28.099513 124 blaster.cc:112] Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. +I20170322 07:04:40.063473 124 loader.cc:552] Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. +I20170322 07:04:50.024030 123 loader.cc:552] Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. +I20170322 07:34:56.081415 121 blaster.cc:352] Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. +I20170322 07:35:14.096304 123 blaster.cc:352] Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. +I20170322 08:05:21.086331 123 demo.cc:352] Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. +I20170322 08:05:33.039503 123 loader.cc:13] Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. +I20170322 08:05:43.092657 124 blaster.cc:6782] Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. +I20170322 08:05:59.002644 123 demo.cc:123] Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. +I20170322 08:06:01.022102 123 demo.cc:352] Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. +I20170322 08:06:22.005675 123 blaster.cc:6782] Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. +I20170322 08:06:37.088974 123 blaster.cc:112] Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. +I20170322 08:06:44.043938 61457 demo.cc:123] Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. +I20170322 08:06:47.060703 123 loader.cc:13] Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. +I20170322 08:06:49.052185 61456 demo.cc:123] Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. +I20170322 08:06:52.074424 61457 demo.cc:352] Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. +I20170322 08:07:02.063191 123 demo.cc:123] Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. +I20170322 08:07:10.030327 61457 blaster.cc:112] Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. +I20170322 08:07:11.011338 123 loader.cc:13] Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. +I20170322 08:07:27.078391 123 blaster.cc:352] Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. +I20170322 08:07:41.061684 123 blaster.cc:112] Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. +I20170322 08:07:53.076558 121 blaster.cc:112] Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. +I20170322 08:38:04.055174 121 blaster.cc:112] Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur: 1AF9293A-F42D-4318-BCDF-60234B240955 +I20170322 08:38:18.046756 123 blaster.cc:112] Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. +I20170322 08:48:28.004198 123 loader.cc:552] Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. +I20170322 08:48:36.032193 61457 blaster.cc:352] Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. +I20170322 09:08:50.028964 61456 loader.cc:13] Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. +I20170322 09:08:56.074576 124 blaster.cc:112] Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. +I20170322 09:08:57.090258 123 loader.cc:13] Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. +I20170322 09:09:00.067690 121 blaster.cc:352] Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. +I20170322 09:09:19.036483 61457 blaster.cc:112] Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. +I20170322 09:09:40.048046 123 blaster.cc:352] Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. +I20170322 09:09:52.051526 123 loader.cc:13] Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. +I20170322 09:10:11.003845 61456 loader.cc:552] Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. +I20170322 09:10:27.094133 123 blaster.cc:6782] Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. +I20170322 09:10:43.027892 121 blaster.cc:352] Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. +I20170322 09:10:57.078489 124 demo.cc:352] Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. +I20170322 09:11:09.014685 123 demo.cc:123] Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. +I20170322 09:11:18.029203 61456 blaster.cc:352] Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. +I20170322 09:11:24.067068 121 demo.cc:123] Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. +I20170322 09:11:38.053891 61456 loader.cc:552] Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. +I20170322 09:11:59.027292 61457 blaster.cc:112] Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. +I20170322 09:12:10.069054 61457 loader.cc:13] Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. +I20170322 09:12:22.018053 123 loader.cc:552] Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. +I20170322 09:12:39.000436 123 demo.cc:352] Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. +I20170322 09:12:53.009916 123 loader.cc:13] Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. +I20170322 09:13:13.051890 121 demo.cc:123] Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. +I20170322 09:13:24.076724 123 demo.cc:123] Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. +I20170323 06:13:34.075980 123 blaster.cc:6782] Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. +I20170323 06:13:35.096130 61456 blaster.cc:6782] Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. +I20170323 06:13:49.087790 121 demo.cc:123] Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. +I20170323 06:14:08.033671 61457 blaster.cc:112] Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. +I20170323 06:14:23.091358 61456 blaster.cc:112] Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. +I20170323 06:14:35.088133 61456 demo.cc:123] Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. +I20170323 06:14:55.005577 123 blaster.cc:352] Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. +I20170323 06:14:58.008392 61457 demo.cc:123] Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. +I20170323 06:15:05.004789 123 loader.cc:552] Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. +I20170323 07:28:07.070013 123 blaster.cc:6782] Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. +I20170323 07:32:08.012805 123 blaster.cc:112] Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. +I20170323 07:33:25.042509 61456 loader.cc:552] Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. +E20170323 08:15:32.027688 123 blaster.cc:6782] Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. +I20170323 08:15:41.020299 61456 blaster.cc:6782] Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. +I20170323 08:15:42.021039 124 loader.cc:552] Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. +I20170323 08:15:59.063918 123 blaster.cc:112] Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. +I20170323 08:16:19.082250 123 loader.cc:552] Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. +I20170323 08:16:20.026445 61457 loader.cc:13] Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. +I20170323 08:16:41.048447 123 blaster.cc:6782] Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. +I20170323 08:16:52.097215 61456 demo.cc:123] Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. +I20170323 09:17:01.020663 61456 blaster.cc:112] Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. |
