Adding upstream version 1.16~pre1.upstream/1.16_pre1

Signed-off-by: Daniel Baumann <mail@daniel-baumann.ch>

1 files changed, 606 insertions, 0 deletions

diff --git a/doc/lziprecover.texi b/doc/lziprecover.texi
new file mode 100644
index 0000000..be4fc27
--- /dev/null
+++ b/doc/lziprecover.texi
@@ -0,0 +1,606 @@
+\input texinfo @c -*-texinfo-*-
+@c %**start of header
+@setfilename lziprecover.info
+@documentencoding ISO-8859-15
+@settitle Lziprecover Manual
+@finalout
+@c %**end of header
+
+@set UPDATED 5 April 2014
+@set VERSION 1.16-pre1
+
+@dircategory Data Compression
+@direntry
+* Lziprecover: (lziprecover).   Data recovery tool for lzip files
+@end direntry
+
+
+@ifnothtml
+@titlepage
+@title Lziprecover
+@subtitle Data recovery tool for lzip files
+@subtitle for Lziprecover version @value{VERSION}, @value{UPDATED}
+@author by Antonio Diaz Diaz
+
+@page
+@vskip 0pt plus 1filll
+@end titlepage
+
+@contents
+@end ifnothtml
+
+@node Top
+@top
+
+This manual is for Lziprecover (version @value{VERSION}, @value{UPDATED}).
+
+@menu
+* Introduction::           Purpose and features of lziprecover
+* Invoking lziprecover::   Command line interface
+* Repairing files::        Fixing bit-flip and similar errors
+* Merging files::          Fixing several damaged copies
+* File format::            Detailed format of the compressed file
+* Examples::               A small tutorial with examples
+* Unzcrash::               Testing the robustness of decompressors
+* Problems::               Reporting bugs
+* Concept index::          Index of concepts
+@end menu
+
+@sp 1
+Copyright @copyright{} 2009, 2010, 2011, 2012, 2013, 2014
+Antonio Diaz Diaz.
+
+This manual is free documentation: you have unlimited permission
+to copy, distribute and modify it.
+
+
+@node Introduction
+@chapter Introduction
+@cindex introduction
+
+Lziprecover is a data recovery tool and decompressor for files in the
+lzip compressed data format (.lz), able to repair slightly damaged
+files, recover badly damaged files from two or more copies, extract data
+from damaged files, decompress files and test integrity of files.
+
+The lzip file format is designed for long-term data archiving, taking
+into account both data integrity and decoder availability:
+
+@itemize @bullet
+@item
+The lzip format provides very safe integrity checking and some data
+recovery means. The lziprecover program can repair bit-flip errors (one
+of the most common forms of data corruption) in lzip files, and provides
+data recovery capabilities, including error-checked merging of damaged
+copies of a file.
+
+@item
+The lzip format is as simple as possible (but not simpler). The lzip
+manual provides the code of a simple decompressor along with a detailed
+explanation of how it works, so that with the only help of the lzip
+manual it would be possible for a digital archaeologist to extract the
+data from a lzip file long after quantum computers eventually render
+LZMA obsolete.
+
+@item
+Additionally lzip is copylefted, which guarantees that it will remain
+free forever.
+@end itemize
+
+Lziprecover is able to recover or decompress files produced by any of
+the compressors in the lzip family; lzip, plzip, minilzip/lzlib, clzip
+and pdlzip.
+
+If the cause of file corruption is damaged media, the combination
+@w{GNU ddrescue + lziprecover} is the best option for recovering data
+from multiple damaged copies. @xref{ddrescue-example}, for an example.
+
+If a file is too damaged for lziprecover to repair it, all the
+recoverable data in all members of the file can be extracted with the
+following command (the resulting file may contain errors and some
+garbage data may be produced at the end of each member):
+
+@example
+lziprecover -D0 -i -o file -q file.lz
+@end example
+
+Lziprecover is able to efficiently extract a range of bytes from a
+multi-member file, because it only decompresses the members containing
+the desired data.
+
+Lziprecover can print correct total file sizes and ratios even for
+multi-member files.
+
+When recovering data, lziprecover takes as arguments the names of the
+damaged files and writes zero or more recovered files depending on the
+operation selected and whether the recovery succeeded or not. The
+damaged files themselves are never modified.
+
+When decompressing or testing file integrity, lziprecover behaves like
+lzip or lunzip.
+
+Lziprecover is not a replacement for regular backups, but a last line of
+defense for the case where the backups are also damaged.
+
+
+@node Invoking lziprecover
+@chapter Invoking lziprecover
+@cindex invoking
+
+The format for running lziprecover is:
+
+@example
+lziprecover [@var{options}] [@var{files}]
+@end example
+
+Lziprecover supports the following options:
+
+@table @samp
+@item -h
+@itemx --help
+Print an informative help message describing the options and exit.
+
+@item -V
+@itemx --version
+Print the version number of lziprecover on the standard output and exit.
+
+@item -c
+@itemx --stdout
+Decompress to standard output. Needed when reading from a named pipe
+(fifo) or from a device. Use it to recover as much of the uncompressed
+data as possible when decompressing a corrupt file.
+
+@item -d
+@itemx --decompress
+Decompress.
+
+@item -D @var{range}
+@itemx --range-decompress=@var{range}
+Decompress only a range of bytes starting at decompressed byte position
+@samp{@var{begin}} and up to byte position @w{@samp{@var{end} - 1}}.
+Three formats of @var{range} are recognized, @samp{@var{begin}},
+@samp{@var{begin}-@var{end}}, and @samp{@var{begin},@var{size}}. If only
+@var{begin} is specified, @var{end} is taken as the end of the file. The
+produced bytes are sent to standard output unless the @samp{--output}
+option is used. In order to guarantee the correctness of the data
+produced, all members containing any part of the desired data are
+decompressed and their integrity is verified. This operation is more
+efficient in multi-member files because it only decompresses the members
+containing the desired data.
+
+@item -f
+@itemx --force
+Force overwrite of output files.
+
+@item -i
+@itemx --ignore-errors
+Make @samp{--range-decompress} ignore data errors and continue
+decompressing the remaining members in the file. For example,
+@w{@samp{lziprecover -i -D0 file.lz > file}} decompresses all the
+recoverable data in all members of @samp{file.lz} without having to
+split it first.
+
+@item -k
+@itemx --keep
+Keep (don't delete) input files during decompression.
+
+@item -l
+@itemx --list
+Print total file sizes and ratios. The values produced are correct even
+for multi-member files. Use it together with @samp{-v} to see
+information about the members in the file.
+
+@item -m
+@itemx --merge
+Try to produce a correct file merging the good parts of two or more
+damaged copies. If successful, a repaired copy is written to the file
+@samp{@var{file}_fixed.lz}. The exit status is 0 if a correct file could
+be produced, 2 otherwise. See the chapter @samp{Merging files}
+(@pxref{Merging files}) for a complete description of the merge mode.
+
+@item -o @var{file}
+@itemx --output=@var{file}
+Place the output into @samp{@var{file}} instead of into
+@samp{@var{file}_fixed.lz}. If splitting, the names of the files
+produced are in the form @samp{rec01@var{file}}, @samp{rec02@var{file}},
+etc. If decompressing from standard input and @samp{--stdout} has not
+been specified, use @samp{@var{file}} as the name of the decompressed
+file.
+
+@item -q
+@itemx --quiet
+Quiet operation. Suppress all messages.
+
+@item -R
+@itemx --repair
+Try to repair a file with small errors (up to one byte error per
+member). If successful, a repaired copy is written to the file
+@samp{@var{file}_fixed.lz}. @samp{@var{file}} is not modified at all.
+The exit status is 0 if the file could be repaired, 2 otherwise. See the
+chapter @samp{Repairing files} (@pxref{Repairing files}) for a complete
+description of the repair mode.
+
+@item -s
+@itemx --split
+Search for members in @samp{@var{file}} and write each member in its own
+@samp{.lz} file. You can then use @samp{lziprecover -t} to test the
+integrity of the resulting files, decompress those which are undamaged,
+and try to repair or partially decompress those which are damaged.
+
+The names of the files produced are in the form
+@samp{rec01@var{file}.lz}, @samp{rec02@var{file}.lz}, etc, and are
+designed so that the use of wildcards in subsequent processing, for
+example, @w{@samp{lziprecover -cd rec*@var{file}.lz > recovered_data}},
+processes the files in the correct order. The number of digits used in
+the names varies depending on the number of members in @samp{@var{file}}.
+
+@item -t
+@itemx --test
+Check integrity of the specified file(s), but don't decompress them.
+This really performs a trial decompression and throws away the result.
+Use it together with @samp{-v} to see information about the file.
+
+@item -v
+@itemx --verbose
+Verbose mode.@*
+When decompressing or testing, further -v's (up to 4) increase the
+verbosity level, showing status, compression ratio, dictionary size,
+trailer contents (CRC, data size, member size), and up to 6 bytes of
+trailing garbage (if any).
+
+@end table
+
+Numbers given as arguments to options may be followed by a multiplier
+and an optional @samp{B} for "byte".
+
+Table of SI and binary prefixes (unit multipliers):
+
+@multitable {Prefix} {kilobyte  (10^3 = 1000)} {|} {Prefix} {kibibyte (2^10 = 1024)}
+@item Prefix @tab Value               @tab | @tab Prefix @tab Value
+@item k @tab kilobyte  (10^3 = 1000)  @tab | @tab Ki @tab kibibyte (2^10 = 1024)
+@item M @tab megabyte  (10^6)         @tab | @tab Mi @tab mebibyte (2^20)
+@item G @tab gigabyte  (10^9)         @tab | @tab Gi @tab gibibyte (2^30)
+@item T @tab terabyte  (10^12)        @tab | @tab Ti @tab tebibyte (2^40)
+@item P @tab petabyte  (10^15)        @tab | @tab Pi @tab pebibyte (2^50)
+@item E @tab exabyte   (10^18)        @tab | @tab Ei @tab exbibyte (2^60)
+@item Z @tab zettabyte (10^21)        @tab | @tab Zi @tab zebibyte (2^70)
+@item Y @tab yottabyte (10^24)        @tab | @tab Yi @tab yobibyte (2^80)
+@end multitable
+
+@sp 1
+Exit status: 0 for a normal exit, 1 for environmental problems (file not
+found, invalid flags, I/O errors, etc), 2 to indicate a corrupt or
+invalid input file, 3 for an internal consistency error (eg, bug) which
+caused lziprecover to panic.
+
+
+@node Repairing files
+@chapter Repairing files
+@cindex repairing files
+
+Lziprecover is able to repair files with small errors (up to one byte
+error per member). The error may be located anywhere in the file except
+in the header (first 6 bytes of each member) or in the @samp{Member
+size} field of the trailer (last 8 bytes of each member). This makes
+lzip files resistant to bit-flip, one of the most common forms of data
+corruption.
+
+Bit-flip happens when one bit in the file is changed from 0 to 1 or vice
+versa. It may be caused by bad RAM or even by natural radiation. I have
+seen a case of bit-flip in a file stored in an USB flash drive.
+
+
+@node Merging files
+@chapter Merging files
+@cindex merging files
+
+If you have several copies of a file but all of them are too damaged to
+repair them (@pxref{Repairing files}), lziprecover can try to produce a
+correct file merging the good parts of the damaged copies.
+
+The merge may succeed even if some copies of the file have all the
+headers and trailers damaged, as long as there is at least one copy of
+every header and trailer intact, even if they are in different copies of
+the file.
+
+The merge will fail if the damaged areas overlap (at least one byte is
+damaged in all copies), or are adjacent and the boundary can't be
+determined, or if the copies have too many damaged areas.
+
+All the copies must have the same size. If some of them have been
+truncated and are therefore smaller than they should, you can extend
+them to the correct size with the following command before merging them
+with the other copies:
+
+@example
+ddrescue --extend-outfile=<correct_size> small_file.lz extended_file.lz
+@end example
+
+If some of the copies have got garbage data at the end and are therefore
+larger than they should, you can reduce their sizes to the correct value
+with the following command before merging them with the other copies:
+
+@example
+ddrescue --size=<correct_size> large_file.lz reduced_file.lz
+@end example
+
+To give you an idea of its possibilities, when merging two copies, each
+of them with one damaged area affecting 1 percent of the copy, the
+probability of obtaining a correct file is about 98 percent. With three
+such copies the probability rises to 99.97 percent. For large files (a
+few MB) with small errors (one sector damaged per copy), the probability
+approaches 100 percent even with only two copies.
+
+
+@node File format
+@chapter File format
+@cindex file format
+
+Perfection is reached, not when there is no longer anything to add, but
+when there is no longer anything to take away.@*
+--- Antoine de Saint-Exupery
+
+@sp 1
+In the diagram below, a box like this:
+@verbatim
++---+
+|   | <-- the vertical bars might be missing
++---+
+@end verbatim
+
+represents one byte; a box like this:
+@verbatim
++==============+
+|              |
++==============+
+@end verbatim
+
+represents a variable number of bytes.
+
+@sp 1
+A lzip file consists of a series of "members" (compressed data sets).
+The members simply appear one after another in the file, with no
+additional information before, between, or after them.
+
+Each member has the following structure:
+@verbatim
++--+--+--+--+----+----+=============+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+| ID string | VN | DS | Lzma stream | CRC32 |   Data size   |  Member size  |
++--+--+--+--+----+----+=============+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+@end verbatim
+
+All multibyte values are stored in little endian order.
+
+@table @samp
+@item ID string
+A four byte string, identifying the lzip format, with the value "LZIP"
+(0x4C, 0x5A, 0x49, 0x50).
+
+@item VN (version number, 1 byte)
+Just in case something needs to be modified in the future. 1 for now.
+
+@item DS (coded dictionary size, 1 byte)
+Lzip divides the distance between any two powers of 2 into 8 equally
+spaced intervals, named "wedges". The dictionary size is calculated by
+taking a power of 2 (the base size) and substracting from it a number of
+wedges between 0 and 7. The size of a wedge is (base_size / 16).@*
+Bits 4-0 contain the base 2 logarithm of the base size (12 to 29).@*
+Bits 7-5 contain the number of wedges (0 to 7) to substract from the
+base size to obtain the dictionary size.@*
+Example: 0xD3 = 2^19 - 6 * 2^15 = 512 KiB - 6 * 32 KiB = 320 KiB@*
+Valid values for dictionary size range from 4 KiB to 512 MiB.
+
+@item Lzma stream
+The lzma stream, finished by an end of stream marker. Uses default values
+for encoder properties. See the lzip manual for a full description.
+
+@item CRC32 (4 bytes)
+CRC of the uncompressed original data.
+
+@item Data size (8 bytes)
+Size of the uncompressed original data.
+
+@item Member size (8 bytes)
+Total size of the member, including header and trailer. This field acts
+as a distributed index, allows the verification of stream integrity, and
+facilitates safe recovery of undamaged members from multi-member files.
+
+@end table
+
+
+@node Examples
+@chapter A small tutorial with examples
+@cindex examples
+
+Example 1: Restore a regular file from its compressed version
+@samp{file.lz}. If the operation is successful, @samp{file.lz} is
+removed.
+
+@example
+lziprecover -d file.lz
+@end example
+
+@sp 1
+@noindent
+Example 2: Verify the integrity of the compressed file @samp{file.lz}
+and show status.
+
+@example
+lziprecover -tv file.lz
+@end example
+
+@sp 1
+@noindent
+Example 3: Decompress @samp{file.lz} partially until 10 KiB of
+decompressed data are produced.
+
+@example
+lziprecover -D 0,10KiB file.lz
+@end example
+
+@sp 1
+@noindent
+Example 4: Decompress @samp{file.lz} partially from decompressed byte
+10000 to decompressed byte 15000 (5000 bytes are produced).
+
+@example
+lziprecover -D 10000-15000 file.lz
+@end example
+
+@sp 1
+@noindent
+Example 5: Repair small errors in the file @samp{file.lz}. (Indented
+lines are abridged diagnostic messages from lziprecover).
+
+@example
+lziprecover -v -R file.lz
+  Copy of input file repaired successfully.
+mv file_fixed.lz file.lz
+@end example
+
+@sp 1
+@noindent
+Example 6: Split the multi-member file @samp{file.lz} and write each
+member in its own @samp{recXXXfile.lz} file. Then use
+@w{@samp{lziprecover -t}} to test the integrity of the resulting files.
+
+@example
+lziprecover -s file.lz
+lziprecover -tv rec*file.lz
+@end example
+
+@sp 1
+@anchor{ddrescue-example}
+@noindent
+Example 7: Recover a compressed backup from two copies on CD-ROM with
+error-checked merging of copies
+@ifnothtml
+(@xref{Top,GNU ddrescue manual,,ddrescue},
+@end ifnothtml
+@ifhtml
+(See the
+@uref{http://www.gnu.org/software/ddrescue/manual/ddrescue_manual.html,,ddrescue manual}
+@end ifhtml
+for details about ddrescue).
+
+@example
+ddrescue -b2048 /dev/cdrom cdimage1 logfile1
+mount -t iso9660 -o loop,ro cdimage1 /mnt/cdimage
+cp /mnt/cdimage/backup.tar.lz rescued1.tar.lz
+umount /mnt/cdimage
+  (insert second copy in the CD drive)
+ddrescue -b2048 /dev/cdrom cdimage2 logfile2
+mount -t iso9660 -o loop,ro cdimage2 /mnt/cdimage
+cp /mnt/cdimage/backup.tar.lz rescued2.tar.lz
+umount /mnt/cdimage
+lziprecover -m -v -o backup.tar.lz rescued1.tar.lz rescued2.tar.lz
+@end example
+
+@sp 1
+@noindent
+Example 8: Recover the first volume of those created with the command
+@w{@samp{lzip -b 32MiB -S 650MB big_db}} from two copies,
+@samp{big_db1_00001.lz} and @samp{big_db2_00001.lz}, with member 07
+damaged in the first copy, member 18 damaged in the second copy, and
+member 12 damaged in both copies. The correct file produced is saved in
+@samp{big_db_00001.lz}.
+
+@example
+lziprecover -m -v -o big_db_00001.lz big_db1_00001.lz big_db2_00001.lz
+  Input files merged successfully
+@end example
+
+
+@node Unzcrash
+@chapter Testing the robustness of decompressors
+@cindex unzcrash
+
+The lziprecover package also includes unzcrash, a program written to
+test robustness to decompression of corrupted data, inspired by
+unzcrash.c from Julian Seward's bzip2. Type @samp{make unzcrash} in the
+lziprecover source directory to build it.
+
+Unzcrash reads the specified file and then repeatedly decompresses it,
+increasing 256 times each byte of the compressed data, so as to test all
+possible one-byte errors. This should not cause any invalid memory
+accesses. If it does, please, report it as a bug.
+
+Unzcrash really executes as a subprocess the shell command specified in
+the first non-option argument, and then writes the file specified in the
+second non-option argument to the standard input of the subprocess,
+modifying the corresponding byte each time. Therefore you can use
+unzcrash to test any decompressor (not only lzip), or even other decoder
+programs with a suitable command line syntax.
+
+The format for running unzcrash is:
+
+@example
+unzcrash [@var{options}] "lzip -tv" @var{filename}.lz
+@end example
+
+Unzcrash supports the following options:
+
+@table @samp
+@item -h
+@itemx --help
+Print an informative help message describing the options and exit.
+
+@item -V
+@itemx --version
+Print the version number of unzcrash on the standard output and exit.
+
+@item -b @var{range}
+@itemx --bits=@var{range}
+Test N-bit errors only, instead of testing all the 255 wrong values for
+each byte. @samp{N-bit error} means any value differing from the
+original value in N bit positions, not a value differing from the
+original value in the bit position N.@*
+The number of N-bit errors per byte (N = 1 to 8) is: 8 28 56 70 56 28 8 1@*
+Examples of @var{range}:  1  1,2,3  1-4  1,3-5,8  1-3,5-8
+
+@item -p @var{bytes}
+@itemx --position=@var{bytes}
+First byte position to test in the file. Defaults to 0.
+
+@item -q
+@itemx --quiet
+Quiet operation. Suppress all messages.
+
+@item -s @var{bytes}
+@itemx --size=@var{bytes}
+Number of byte positions to test. If not specified, the whole file is
+tested.
+
+@item -v
+@itemx --verbose
+Verbose mode.
+
+@end table
+
+Exit status: 0 for a normal exit, 1 for environmental problems (file not
+found, invalid flags, I/O errors, etc), 2 to indicate a corrupt or
+invalid input file, 3 for an internal consistency error (eg, bug) which
+caused unzcrash to panic.
+
+
+@node Problems
+@chapter Reporting bugs
+@cindex bugs
+@cindex getting help
+
+There are probably bugs in lziprecover. There are certainly errors and
+omissions in this manual. If you report them, they will get fixed. If
+you don't, no one will ever know about them and they will remain unfixed
+for all eternity, if not longer.
+
+If you find a bug in lziprecover, please send electronic mail to
+@email{lzip-bug@@nongnu.org}. Include the version number, which you can
+find by running @w{@samp{lziprecover --version}}.
+
+
+@node Concept index
+@unnumbered Concept index
+
+@printindex cp
+
+@bye