diff options
Diffstat (limited to 'src/help.txt')
| -rw-r--r-- | src/help.txt | 1001 |
1 files changed, 1001 insertions, 0 deletions
diff --git a/src/help.txt b/src/help.txt new file mode 100644 index 0000000..4797e4c --- /dev/null +++ b/src/help.txt @@ -0,0 +1,1001 @@ + + lnav - A fancy log file viewer + +DESCRIPTION +=========== + +The log file navigator, lnav, is an enhanced log file viewer that +takes advantage of any semantic information that can be gleaned from +the files being viewed, such as timestamps and log levels. Using this +extra semantic information, lnav can do things like interleaving +messages from different files, generate histograms of messages over +time, and providing hotkeys for navigating through the file. It is +hoped that these features will allow the user to quickly and +efficiently zero in on problems. + + +OPENING PATHS/URLs +================== + +The main arguments to lnav are the files, directories, glob patterns, +or URLs to be viewed. If no arguments are given, the default syslog +file for your system will be opened. These arguments will be polled +periodically so that any new data or files will be automatically +loaded. If a previously loaded file is removed or replaced, it will +be closed and the replacement opened. + +Note: When opening SFTP URLs, if the password is not provided for the +host, the SSH agent can be used to do authentication. + + +OPTIONS +======= + +Lnav takes a list of files to view and/or you can use the flag +arguments to load well-known log files, such as the syslog log +files. The flag arguments are: + + -a Load all of the most recent log file types. + -r Recursively load files from the given directory hierarchies. + -R Load older rotated log files as well. + +When using the flag arguments, lnav will look for the files relative +to the current directory and its parent directories. In other words, +if you are working within a directory that has the well-known log +files, those will be preferred over any others. + +If you do not want the default syslog file to be loaded when +no files are specified, you can pass the '-N' flag. + +Any files given on the command-line are scanned to determine their log +file format and to create an index for each line in the file. You do +not have to manually specify the log file format. The currently +supported formats are: syslog, apache, strace, tcsh history, and +generic log files with timestamps. + +Lnav will also display data piped in on the standard input. The +following options are available when doing so: + + -t Prepend timestamps to the lines of data being read in + on the standard input. + -w file Write the contents of the standard input to this file. + +To automatically execute queries or lnav commands after the files +have been loaded, you can use the following options: + + -c cmd A command, query, or file to execute. The first character + determines the type of operation: a colon is used for the + built-in commands; a semi-colon for SQL queries; and a + pipe symbol (|) for executing a file containing other + commands. For example, to open the file "foo.log" and go + to the tenth line in the file, you can do: + + lnav -c ':goto 10' foo.log + + This option can be given multiple times to execute multiple + operations in sequence. + + -f file A file that contains commands, queries, or files to execute. + This option is a shortcut for "-c '|file'". You can use a + dash (-) to execute commands from the standard input. + +To execute commands/queries without the opening the interactive text UI, +you can pass the '-n' option. This combination of options allows you to +write scripts for processing logs with lnav. For example, to get a list +of IP addresses that dhclient has bound to in CSV format: + + #! /usr/bin/lnav -nf + + # Usage: dhcp_ip.lnav /var/log/messages + + # Only include lines that look like: + # Apr 29 00:31:56 example-centos5 dhclient: bound to 10.1.10.103 -- renewal in 9938 seconds. + :filter-in dhclient: bound to + + # The log message parser will extract the IP address as col_0, so we + # select that and alias it to "dhcp_ip". + ;select distinct col_0 as dhcp_ip from logline; + + # Finally, write the results of the query to stdout. + :write-csv-to - + + +DISPLAY +======= + +The main part of the display shows the log lines from the files interleaved +based on time-of-day. New lines are automatically loaded as they are appended +to the files and, if you are viewing the bottom of the files, lnav will scroll +down to display the new lines, much like 'tail -f'. + +On color displays, the lines will be highlighted as follows: + + * Errors will be colored in ${ansi_red}red${ansi_norm}; + * warnings will be ${ansi_yellow}yellow${ansi_norm}; + * boundaries between days will be ${ansi_underline}underlined${ansi_norm}; and + * various color highlights will be applied to: IP addresses, SQL keywords, + XML tags, file and line numbers in Java backtraces, and quoted strings. + +To give you an idea of where you are spatially, the right side of the +display has a proportionally sized 'scroll bar' that indicates your +current position in the files. The scroll bar will also show areas of +the file where warnings or errors are detected by coloring the bar +yellow or red, respectively. Tick marks will also be added to the +left and right hand side of the bar, for search hits and bookmarks. + +A bar on the left side is color coded and broken up to indicate which +messages are from the same file. Pressing the left-arrow or 'h' will +reveal the source file names for each message and pressing again will +show the full paths. + +When at the bottom of the log view, a summary line will be displayed on the +right-hand-side to give you some more information about your logs, including: +how long ago the last message was generated, the number of log files, the +error rate, and how much time the logs cover. The error rate display shows +the errors-per-minute over the last five minutes. A bar chart is also +overlaid on the "Error rate" label to show the error rate over the past ten +seconds. For example, if there have not been many errors in the past five +minutes and there is a sudden spike, the bar chart will fill up completely. +But, if there has been a steady stream of errors, then the chart will only +partially fill based on the recent error frequency. + +Above and below the main body are status lines that display: + + * the current time; + * the name of the file the top line was pulled from; + * the log format for the top line; + * the current view; + * the line number for the top line in the display; + * the current search hit, the total number of hits, and the search term; + +If the view supports filtering, there will be a status line showing the +following: + + * the number of enabled filters and the total number of filters; + * the number of lines not displayed because of filtering. + +To edit the filters, you can press TAB to change the focus from the main +view to the filter editor. The editor allows you to create, enable/disable, +and delete filters easily. + +Finally, the last line on the display is where you can enter search +patterns and execute internal commands, such as converting a +unix-timestamp into a human-readable date. The command-line is +implemented using the readline library, so the usual set of keyboard +shortcuts are available. Most commands and searches also support +tab-completion. + + +The body of the display is also used to display other content, such +as: the help file, histograms of the log messages over time, and +SQL results. The views are organized into a stack so that any time +you activate a new view with a key press or command, the new view +is pushed onto the stack. Pressing the same key again will pop the +view off of the stack and return you to the previous view. Note +that you can always use 'q' to pop the top view off of the stack. + + +DEFAULT KEY BINDINGS +==================== + +Views +----- + + ? View/leave this help message. + q Leave the current view or quit the program when in + the log file view. + Q Similar to 'q', except it will try to sync the top time + between the current and former views. For example, when + leaving the spectrogram view with 'Q', the top time in that + view will be matched to the top time in the log view. + + TAB Toggle focusing on the filter editor or the main view. + + a/A Restore the view that was previously popped with 'q/Q'. + The 'A' hotkey will try to match the top times between the + two views. + + X Close the current text file or log file. + +Spatial Navigation +------------------ + + g/home Move to the top of the file. + G/end Move to the end of the file. If the view is already + at the end, it will move to the last line. + space/pgdn Move down a page. + b/bs/pgup Move up a page. + j/cr/down-arrow Move down a line. + k/up-arrow Move up a line. + h/left-arrow Move to the left. In the log view, moving left will reveal + the source log file names for each line. Pressing again + will reveal the full path. + l/right-arrow Move to the right. + H/Shift+left Move to the left by a smaller increment. + L/Shift+right Move to the right by a smaller increment. + + e/E Move to the next/previous error. + w/W Move to the next/previous warning. + n/N Move to the next/previous search hit. When pressed + repeatedly within a short time, the view will move at + least a full page at a time instead of moving to the + next hit. + f/F Move to the next/previous file. In the log view, this + moves to the next line from a different file. In the + text view, this rotates the view to the next file. + + >/< Move horizontally to the next/previous search hit. + + o/O Move forward/backward to the log message with a matching + 'operation ID' (opid) field. + + u/U Move forward/backward through any user bookmarks + you have added using the 'm' key. This hotkey will + also jump to the start of any log partitions that have + been created with the 'partition-name' command. + + s/S Move to the next/previous "slow down" in the log message + rate. A slow down is detected by measuring how quickly + the message rate has changed over the previous several + messages. For example, if one message is logged every + second for five seconds and then the last message arrives + five seconds later, the last message will be highlighted + as a slow down. + + {/} Move to the previous/next location in history. Whenever + you jump to a new location in the view, the location will + be added to the history. The history is not updated when + using only the arrow keys. + + +Chronological Navigation +------------------------ + + d/D Move forward/backward 24 hours from the current + position in the log file. + + 1-6/Shift 1-6 Move to the next/previous n'th ten minute of the + hour. For example, '4' would move to the first + log line in the fortieth minute of the current + hour in the log. And, '6' would move to the next + hour boundary. + + 7/8 Move to the previous/next minute. + + 0/Shift 0 Move to the next/previous day boundary. + + r/R Move forward/backward based on the relative time that + was last used with the 'goto' command. For example, + executing ':goto a minute later' will move the log view + forward a minute and then pressing 'r' will move it + forward a minute again. Pressing 'R' will then move the + view in the opposite direction, so backwards a minute. + +Bookmarks +--------- + + m Mark/unmark the line at the top of the display. + The line will be highlighted with reverse video to + indicate that it is a user bookmark. You can use + the 'u' hotkey to iterate through marks you have + added. + + M Mark/unmark all the lines between the top of the + display and the last line marked/unmarked. + + J Mark/unmark the next line after the previously + marked line. + + K Like 'J' except it toggles the mark on the + previous line. + + c Copy the marked text to the X11 selection buffer or OS X + clipboard. + + C Clear all marked lines. + +Display options +--------------- + + P Switch to/from the pretty-printed view of the log or text + files currently displayed. In this view, structured data, + such as XML, will be reformatted to make it easier to read. + + t Switch to/from the text file view. The text file view is + for any files that are not recognized as log files. + + = Pause/unpause loading of new file data. + + Ctrl-L (Lo-fi mode) Exit screen-mode and write the + displayed log lines in plain text to the terminal + until a key is pressed. Useful for copying long lines + from the terminal without picking up any of the extra + decorations. + + T Toggle the display of the "elapsed time" column that shows + the time elapsed since the beginning of the logs or the + offset from the previous bookmark. Sharp changes in the + message rate are highlighted by coloring the separator + between the time column and the log message. A red + highlight means the message rate has slowed down and green + means it has sped up. You can use the "s/S" hotkeys to + scan through the slow downs. + + i View/leave a histogram of the log messages over + time. The histogram counts the number of + displayed log lines for each bucket of time. The + bars are layed out horizontally with colored + segments representing the different log levels. + You can use the 'z' hotkey to change the size of + the time buckets (e.g. ten minutes, one hour, one + day). + + I Switch between the log and histogram views while + keeping the time displayed at the top of each view + in sync. For example, if the top line in the log + view is "11:40", hitting 'I' will switch to the + histogram view and scrolled to display "11:00" at + the top (if the zoom level is hours). + + z/Shift Z Zoom in or out one step in the histogram view. + + v Switch to/from the SQL result view. + + V Switch between the log and SQL result views while + keeping the top line number in the log view in + sync with the log_line column in the SQL view. + For example, doing a query that selects for + "log_idle_msecs" and "log_line", you can move the + top of the SQL view to a line and hit 'V' to switch + to the log view and move to the line number that was + selected in the "log_line" column. If there is no + "log_line" column, lnav will find the first column with + a timestamp and move to corresponding time in the log + view. + + TAB/Shift+TAB In the SQL result view, cycle through the columns that + are graphed. Initially, all number values are displayed + in a stacked graph. Pressing TAB will change the display + to only graph the first column. Repeatedly pressing TAB + will cycle through the columns until they are all graphed + again. + + p In the log view: enable or disable the display of the + fields that the log message parser knows about or has + discovered. This overlay is temporarily enabled when the + semicolon key (;) is pressed so that it is easier to write + queries. + + In the DB view: enable or disable the display of values + in columns containing JSON-encoded values in the top row. + The overlay will display the JSON-Pointer reference and + value for all fields in the JSON data. + + CTRL-W Toggle word-wrapping. + + CTRL-P Show/hide the data preview panel that may be opened when + entering commands or SQL queries. + + CTRL-F Toggle the enabled/disabled state of all filters in the + current view. + + x Toggle the hiding of log message fields. The hidden fields + will be replaced with three bullets and highlighted in + yellow. + + F2 Toggle mouse support. + +Query +----- + + /<regexp> Start a search for the given regular expression. + The search is live, so when there is a pause in + typing, the currently running search will be + canceled and a new one started. The first ten lines + that match the search will be displayed in the preview + window at the bottom of the view. History is + maintained for your searches so you can rerun them + easily. Words that are currently displayed are also + available for tab-completion, so you can easily + search for values without needing to copy-and-paste + the string. If there is an error encountered while + trying to interpret the expression, the error will + be displayed in red on the status line. While the + search is active, the 'hits' field in the status + line will be green, when finished it will turn + back to black. + + Note: The regular expression format used by is PCRE + (Perl-Compatible Regular Expressions). For example, + if you wanted to search for ethernet device names, + regardless of their ID number, you can type: + + eth\\d+ + + You can find more information about Perl regular + expressions at: + + http://perldoc.perl.org/perlre.html + + If the search string is not valid PCRE, a search + is done for the exact string instead of doing a + regex search. + + :<command> Execute an internal command. The commands are + listed below. History is also supported in this + context as well as tab-completion for commands and + some arguments. The result of the command + replaces the command you typed. + + ;<sql> Execute an SQL query. Most supported log file + formats provide a sqlite virtual table backend + that can be used in queries. See the SQL section + below for more information. + + |<script> [arg1 .. argN] + Execute an lnav script contained in a format directory + (e.g. \~/.lnav/formats/default). The script can contain + lines starting with ':', ';', or '|' to execute commands, + SQL queries or execute other files in lnav. Any values + after the script name are treated as arguments can be + referenced in the script using '\$1', '\$2', and so on, like + in a shell script. + + CTRL+], ESCAPE Abort command-line entry started with '/', ':', ';', or '|'. + +Session +------- + + CTRL-R Reset the session state. This will save the current + session state (filters, highlights) and then reset the + state to the factory default. + +Filter Editor +------------- + +The following hotkeys are only available when the focus is on the filter +editor. You can change the focus by pressing TAB. + + q Switch the focus back to the main view. + j/down-arrow Select the next filter. + k/up-arrow Select the previous filter. + o Create a new "out" filter. + i Create a new "in" filter . + SPACE Toggle the enabled/disabled state of the currently + selected filter. + t Toggle the type of filter between "in" and "out". + ENTER Edit the selected filter. + D Delete the selected filter. + + +MOUSE SUPPORT (experimental) +============================ + +If you are using Xterm, or a compatible terminal, you can use the mouse to +mark lines of text and move the view by grabbing the scrollbar. + +NOTE: You need to manually enable this feature by setting the LNAV_EXP +environment variable to "mouse". F2 toggles mouse support. + + +COMMANDS +======== + + help Switch to this help text view. + + adjust-log-time <date|relative-time> + Change the time of the top log line to the given time or + adjusted by the relative time. All other log lines in the + same file will also be adjusted using the same offset. + After the adjustment, the displayed timestamp will be + rewritten to the new time and highlighted with a magenta + color. + + This command is useful for lining up log files that + have timestamps from different machines. + + unix-time <secs-or-date> + Convert a unix-timestamp in seconds to a + human-readable form or vice-versa. + BEWARE OF TIMEZONE DIFFERENCES. + + current-time Print the current time in human-readable form and + as a unix-timestamp. + + goto <line#|N%|abs-time|relative-time> + Go to the given line number, N percent into the + file, or the given timestamp in the log view. If the + line number is negative, it is considered an offset + from the last line. Relative time values, like + 'a minute ago', 'an hour later', and many other formats + are supported. When using a relative time, the 'r/R' + hotkeys can be used to move the same amount again or in + the same amount in the opposite direction. + + relative-goto <line#|N%> + Move the current view up or down by the given amount. + + comment <message> + Add a comment to the top line in the log view. The + comment will be saved in the session and will be available + the next time the file is loaded. Searches will also scan + the comment for any matches. + + clear-comment Clear the comment that is attached to the top line in the + log view. + + tag <tag1> [<tag2> [... <tagN>]] + Attach a tag to the top line in the log view. The tags are + prefixed with a '#', if they don't have one already. And, + like comments, they are saved in the session and + searchable. + + untag <tag1> [<tag2> [... <tagN>]] + Detach a tag from the top line in the log view. + + delete-tags <tag1> [<tag2> [... <tagN>]] + Detach the tags from all log lines. + + next-mark error|warning|search|user|file|meta + Move to the next bookmark of the given type in the + current view. + + prev-mark error|warning|search|user|file|meta + Move to the previous bookmark of the given type in the + current view. + + hide-lines-before <abs-time|relative-time> + Hide lines that are before the given time. The given + time can be absolute (e.g. 2015-10-11) + The hidden lines can be shown again with the + 'show-lines-before-and-after' command. + + hide-lines-after <abs-time|relative-time> + Hide lines that are after the given time. The time can + The hidden lines can be shown again with the + 'show-lines-before-and-after' command. + + show-lines-before-and-after + Show lines that were hidden by the 'hide-lines' commands. + + hide-unmarked-lines + Hide lines that have not been bookmarked. + + show-unmarked-lines + Show lines that have not been bookmarked. + + hide-fields <field-name> [<field-name2> ... <field-nameN>] + Hide large log message fields by replacing them with an + ellipsis. You can quickly switching between showing and + hiding hidden fields using the 'x' hotkey. + + show-fields <field-name> [<field-name2> ... <field-nameN>] + Show log messages fields that were previously hidden with + the ':hide-fields' command. + + highlight <regex> Highlight strings that match the given regular + expression. + + clear-highlight <regex> + Clear an existing highlight created with the 'highlight' + command. + + filter-in <regex> Only display lines that match the given regular + expression. This command can be used multiple + times to add more lines to the display. The number + of lines that are filtered out will be shown in the + bottom status bar as 'Not Shown'. Note that filtering + only works in the log and plain text views. There is also + a limit of 32 filters per-view at any one time. + + filter-out <regex> + Do not display lines that match the given regular + expression. This command can be used multiple + times to remove more lines from the display. If a + 'filter-in' expression is also active, it takes + priority and the filter-out will remove lines that + were matched by the 'filter-in'. The number + of lines that are filtered out will be shown in the + bottom status bar as 'Not Shown'. Note that filtering + only works in the log and plain text views. There is also + a limit of 32 filters per-view at any one time. While + entering the regular expression at the command-prompt, the + matches in the current text view will be highlighted in red + after a short delay. + + disable-filter <regex> + Disable an active 'filter-in' or 'filter-out' + expression. + + enable-filter <regex> + Enable a inactive 'filter-in' or 'filter-out' + expression. + + delete-filter <regex> + Permanently delete a filter. + + set-min-log-level <level> + Set the minimum level to display in the log view. + You can use TAB to view the possible values. + + disable-word-wrap Disable word wrapping in the log and text file views. + enable-word-wrap Enable word wrapping in the log and text file views. + + open <filename>[:<line>] + Open the given file within lnav and, if it is a + text file, switch to the text view and jump to + the given line number. + + close Close the current text file or log file. You can also + close the current file by pressing 'X'. + + spectrogram <field-name> + Generate a spectrogram for a numeric log message field or + SQL result column. The spectrogram view displays the range + of possible values of the field on the horizontal axis and + time on the vertical axis. The horizontal axis is split + into buckets where each bucket counts how many log messages + contained the field with a value in that range. The buckets + are colored based on the count in the bucket: green means + low, yellow means medium, and red means high. The exact + ranges for the colors is computed automatically and + displayed in the middle of the top line of the view. The + minimum and maximum values for the field are displayed in + the top left and right sides of the view, respectively. + + append-to <file> Append any marked lines to the given file. + + write-to <file> Write any marked lines to the given file. Use '-' to + write the lines to the terminal. + + write-csv-to <file> + Write the results of a SQL query to a CSV-formatted file. + When running in non-interactive mode, a dash can be used + to write to standard out. Use '-' to write the data to + the terminal. + + write-json-to <file> + Write the results of a SQL query to a JSON-formatted file. + The contents of the file will be an array of objects with + each column in the query being a field in the objects. + When running in non-interactive mode, a dash can be used + to write to standard out. Use '-' to write the data to + the terminal. + + write-jsonlines-to <file> + Write the results of a SQL query to a JSON-Lines-formatted + file. Each row in the result set will be a single line of + JSON in the output and each column will be a property in + that object. Use '-' to write the data to the terminal. + + pipe-to <shell-cmd> + Send the currently marked lines to the given shell command + for processing and open the resulting file for viewing. + + pipe-line-to <shell-cmd> + Send the top log message to the given shell command + for processing and open the resulting file for viewing. + The known and discovered message fields are available as + environment variables. For example, log_procname in a + syslog message. + + redirect-to <path> + If a path is given, all output from commands, like + ":echo" and when writing to stdout (e.g. :write-to -), will + be sent to the given file. If no path is specified, the + current redirect will be cleared and output will be + captured as it was before the redirect was done. + + session <cmd> Add the given command to the session file + (\~/.lnav/session). Any commands listed in the session file + are executed on startup. Only the highlight, word-wrap, and + filter-related commands can be added to the session file. + + create-logline-table <table-name> + Create an SQL table using the top line of the log view + as a template. See the "SQL QUERIES" and "DYNAMIC LOG + LINE TABLE" sections below for more information. + + delete-logline-table <table-name> + Delete an SQL table created by the 'create-logline-table' + command. + + create-search-table <table-name> [<regex>] + Create an SQL table that extracts information from logs + using the provided regular expression or the last search + that was done. Any captures in the expression will be + used as columns in the SQL table. If the capture is named, + that name will be used as the column name, otherwise the + column name will be of the form 'col_N'. + + delete-search-table <table-name> + Delete a table that was created with create-search-table. + + switch-to-view <view-name> + Switch the display to the given view, which can be one of: + help, log, text, histogram, db, and schema. + + zoom-to <zoom-level> + Change the histogram zoom level to the given value, which + can be one of: day, 4-hour, hour, 10-minute, minute + + redraw Force redraw the window. + + partition-name <name> + Mark the top line in the log view as the start of a new + partition with the given name. The current partition name + will be reflected in the top status bar next to the current + time as well as being available in the 'log_part' column + of the SQL log tables. Partitions can be used to make it + easier to query subsections of log messages. + + clear-partition + Clear the partition the top line is a part of. + + echo [-n] <msg> Display the given message. Useful for scripts to message + the user. The '-n' option leaves out the new line at the + end of the message. + + eval <cmd|query|file> + Execute the given command, query, or file after doing + environment variable substitution. The argument to this + command should start with a ':', ';', or '|' signify the + type of action to perform (command, SQL query, execute + script). + + pt-min-time [<date>|<relative-time>] + Set/get the minimum time range for any papertrail queries. + Absolute or relative time values can be specified. + + pt-max-time [<date>|<relative-time>] + Set/get the maximum time range for any papertrail queries. + Absolute or relative time values can be specified. + + config <option> [value] + Set/get the value of a configuration option. + + reset-config <option> + Reset a configuration option to the default value. Use + '*' to reset all options. + + quit Quit lnav. + +SQL QUERIES (experimental) +=========== + +Lnav has support for performing SQL queries on log files using the +Sqlite3 "virtual" table feature. For all supported log file types, +lnav will create tables that can be queried using the subset of SQL +that is supported by Sqlite3. For example, to get the top ten URLs +being accessed in any loaded Apache log files, you can execute: + + ;select cs_uri_stem, count(*) as total from access_log + group by cs_uri_stem order by total desc limit 10; + +The query result view shows the results and graphs any numeric +values found in the result, much like the histogram view. + +The builtin set of log tables are listed below. Note that only the +log messages that match a particular format can be queried by a +particular table. You can find the file format and table name for +the top log message by looking in the upper right hand corner of the +log file view. + +Some commonly used format tables are: + + access_log Apache common access log format + syslog_log Syslog format + strace_log Strace log format + generic_log 'Generic' log format. This table contains messages + from files that have a very simple format with a + leading timestamp followed by the message. + +NOTE: You can get a dump of the schema for the internal tables, and +any attached databases, by running the '.schema' SQL command. + +The columns available for the top log line in the view will +automatically be displayed after pressing the semicolon (;) key. +All log tables contain at least the following columns: + + log_line The line number in the file, starting at zero. + log_part The name of the partition. You can change this + column using an UPDATE SQL statement or with the + 'partition-name' command. After a value is set, + the following log messages will have the same + partition name up until another name is set. + log_time The time of the log entry. + log_idle_msecs The amount of time, in milliseconds, between the + current log message and the previous one. + log_level The log level (e.g. info, error, etc...). + log_mark The bookmark status for the line. This column + can be written to using an UPDATE query. + log_path The full path to the file. + log_text The raw line of text. Note that this column is + not included in the result of a 'select *', but + it does exist. + +The following tables include the basic columns as listed above and +include a few more columns since the log file format is more +structured. + + syslog_log + + log_hostname The hostname the message was received from. + log_procname The name of the process that sent the message. + log_pid The process ID of the process that sent the message. + + access_log (The column names are the same as those in the + Microsoft LogParser tool.) + + c_ip The client IP address. + cs_username The client user name. + cs_method The HTTP method. + cs_uri_stem The stem portion of the URI. + cs_uri_query The query portion of the URI. + cs_version The HTTP version string. + sc_status The status number returned to the client. + sc_bytes The number of bytes sent to the client. + cs_referrer The URL of the referring page. + cs_user_agent The user agent string. + + strace_log (Currently, you need to run strace with the + "-tt -T" options so there are timestamps for + each function call.) + + funcname The name of the syscall. + result The result code. + duration The amount of time spent in the syscall. + arg0 - arg9 The arguments passed to the syscall. + +These tables are created dynamically and not stored in memory or on +disk. If you would like to persist some information from the tables, +you can attach another database and create tables in that database. +For example, if you wanted to save the results from the earlier +example of a top ten query into the "/tmp/topten.db" file, you can do: + + ;attach database "/tmp/topten.db" as topten; + ;create table topten.foo as select cs_uri_stem, count(*) as total + from access_log group by cs_uri_stem order by total desc + limit 10; + + +DYNAMIC LOG LINE TABLE (experimental) +====================== + +(NOTE: This feature is still very new and not completely reliable yet, + use with care.) + +For log formats that lack message structure, lnav can parse the log +message and attempt to extract any data fields that it finds. This +feature is available through the "logline" log table. This table is +dynamically created and defined based on the message at the top of +the log view. For example, given the following log message from "sudo", +lnav will create the "logline" table with columns for "TTY", "PWD", +"USER", and "COMMAND": + + May 24 06:48:38 Tim-Stacks-iMac.local sudo[76387]: stack : TTY=ttys003 ; + PWD=/Users/stack/github/lbuild ; USER=root ; + COMMAND=/bin/echo Hello, World! + +Queries executed against this table will then only return results for +other log messages that have the same format. So, if you were to +execute the following query while viewing the above line, you might +get the following results: + + ;select USER,COMMAND from logline; + + USER | COMMAND + ---- | ------------------------- + root | /bin/echo Hello, World! + mal | /bin/echo Goodbye, World! + + +The log parser works by examining each message for key/value pairs +separated by an equal sign (=) or a colon (:). For example, in the +previous example of a "sudo" message, the parser sees the "USER=root" +string as a pair where the key is "USER" and the value is "root". +If no pairs can be found, then anything that looks like a value is +extracted and assigned a numbered column. For example, the following +line is from "dhcpd": + + Sep 16 22:35:57 drill dhcpd: DHCPDISCOVER from 00:16:ce:54:4e:f3 via hme3 + +In this case, the lnav parser recognizes that "DHCPDISCOVER", the MAC +address and the "hme3" device name are values and not normal words. So, +it builds a table with three columns for each of these values. The +regular words in the message, like "from" and "via", are then used to +find other messages with a similar format. + +If you would like to execute queries against log messages of different +formats at the same time, you can use the 'create-logline-table' command +to permanently create a table using the top line of the log view as a +template. + + +OTHER SQL FEATURES +================== + +Environment variables can be used in SQL statements by prefixing the +variable name with a dollar-sign (\$). For example, to read the value of +the HOME variable, you can do: + + ;SELECT \$HOME; + +To select the syslog messages that have a hostname field that is equal +to the HOSTNAME variable: + + ;SELECT * FROM syslog_log WHERE log_hostname = \$HOSTNAME; + +NOTE: Variable substitution is done for fields in the query and is not +a plain text substitution. For example, the following statement +WILL NOT WORK: + + ;SELECT * FROM \$TABLE_NAME; -- Syntax error + + +Access to lnav's environment variables is also available via the "environ" +table. The table has two columns (name, value) and can be read and written +to using SQL SELECT, INSERT, UPDATE, and DELETE statements. For example, +to set the "FOO" variable to the value "BAR": + + ;INSERT INTO environ SELECT 'FOO', 'BAR'; + +As a more complex example, you can set the variable "LAST" to the last +syslog line number by doing: + + ;INSERT INTO environ SELECT 'LAST', (SELECT max(log_line) FROM syslog_log); + +A delete will unset the environment variable: + + ;DELETE FROM environ WHERE name='LAST'; + +The table allows you to easily use the results of a SQL query in lnav +commands, which is especially useful when scripting lnav. + + +PAPERTRAIL INTEGRATION +====================== + +Papertrail is a log management service with free and paid plans at: + + http://papertrailapp.com + +To configure lnav to communicate with the papertrail service, you will +need to set the PAPERTRAIL_API_TOKEN environment variable. You can +get your API token from your user profile, available here: + + https://papertrailapp.com/user/edit + +Searching papertrail using lnav can be done by prefixing the search terms +with "pt:" and passing the value as a file name. For example, to search +for log messages with the string 'Critical Error' when starting lnav you +can do the following: + + \$ setenv PAPERTRAIL_API_TOKEN xxxxxxxxx + \$ lnav "pt:'Critical Error'" + +If lnav is already started, you can use the ':open' command like so: + + :open pt:'Critical Error' + +If you just want to tail your logs in papertrail, you can pass an empty +search string (i.e. "pt:"). + +Only one papertrail search can be active at a time. So, if an ':open' +is done with a new query, the previous query will be closed first. + + +CONTACT +======= + +For more information, visit the lnav website at: + + http://lnav.org + +For support questions, email: + + lnav@googlegroups.com + + +REFERENCE +========= |