diff options
Diffstat (limited to 'doc/lziprecover.texi')
| -rw-r--r-- | doc/lziprecover.texi | 212 |
1 files changed, 148 insertions, 64 deletions
diff --git a/doc/lziprecover.texi b/doc/lziprecover.texi index e29a59f..2702d70 100644 --- a/doc/lziprecover.texi +++ b/doc/lziprecover.texi @@ -6,8 +6,8 @@ @finalout @c %**end of header -@set UPDATED 16 September 2015 -@set VERSION 1.18-pre2 +@set UPDATED 12 May 2016 +@set VERSION 1.18 @dircategory Data Compression @direntry @@ -50,7 +50,7 @@ This manual is for Lziprecover (version @value{VERSION}, @value{UPDATED}). @end menu @sp 1 -Copyright @copyright{} 2009-2015 Antonio Diaz Diaz. +Copyright @copyright{} 2009-2016 Antonio Diaz Diaz. This manual is free documentation: you have unlimited permission to copy, distribute and modify it. @@ -66,6 +66,9 @@ files, produce a correct file by merging the good parts of two or more damaged copies, extract data from damaged files, decompress files and test integrity of files. +Lziprecover is not a replacement for regular backups, but a last line of +defense for the case where the backups are also damaged. + The lzip file format is designed for data sharing and long-term archiving, taking into account both data integrity and decoder availability: @@ -96,6 +99,10 @@ repair the nearer it is from the beginning of the file. Therefore, with the help of lziprecover, losing an entire archive just because of a corrupt byte near the beginning is a thing of the past. +For compressible data, multiple lzip-compressed copies have a better +chance of surviving intact than one uncompressed copy using the same +amount of storage space. + Lziprecover is able to recover or decompress files produced by any of the compressors in the lzip family; lzip, plzip, minilzip/lzlib, clzip and pdlzip. @@ -113,12 +120,11 @@ garbage data may be produced at the end of each member): lziprecover -D0 -i -o file -q file.lz @end example -Lziprecover is able to efficiently extract a range of bytes from a -multi-member file, because it only decompresses the members containing -the desired data. +Lziprecover provides random access to the data in multimember files; it +only decompresses the members containing the desired data. Lziprecover can print correct total file sizes and ratios even for -multi-member files. +multimember files. When recovering data, lziprecover takes as arguments the names of the damaged files and writes zero or more recovered files depending on the @@ -128,9 +134,6 @@ damaged files themselves are never modified. When decompressing or testing file integrity, lziprecover behaves like lzip or lunzip. -Lziprecover is not a replacement for regular backups, but a last line of -defense for the case where the backups are also damaged. - @node Invoking lziprecover @chapter Invoking lziprecover @@ -142,6 +145,11 @@ The format for running lziprecover is: lziprecover [@var{options}] [@var{files}] @end example +@noindent +When decompressing or testing, @samp{-} used as a @var{file} argument +means standard input. It can be mixed with other @var{files} and is read +just once, the first time it appears in the command line. + Lziprecover supports the following options: @table @code @@ -160,31 +168,54 @@ Exit with error status 2 if any remaining input is detected after decompressing the last member. Such remaining input is usually trailing garbage that can be safely ignored. @xref{concat-example}. +@item -A +@itemx --alone-to-lz +Convert lzma-alone files to lzip format without recompressing, just +adding a lzip header and trailer. The conversion minimizes the +dictionary size of the resulting file (and therefore the amount of +memory required to decompress it). Only streamed files with default LZMA +properties can be converted; non-streamed lzma-alone files lack the end +of stream marker required in lzip files. + +The name of the converted lzip file is derived from that of the original +lzma-alone file as follows: + +@multitable {filename.lzma} {becomes} {anyothername.lz} +@item filename.lzma @tab becomes @tab filename.lz +@item filename.tlz @tab becomes @tab filename.tar.lz +@item anyothername @tab becomes @tab anyothername.lz +@end multitable + @item -c @itemx --stdout -Decompress to standard output. Needed when reading from a named pipe -(fifo) or from a device. Use it to recover as much of the uncompressed -data as possible when decompressing a corrupt file. +Write decompressed data to standard output; keep input files unchanged. +This option is needed when reading from a named pipe (fifo) or from a +device. Use it also to recover as much of the uncompressed data as +possible when decompressing a corrupt file. @item -d @itemx --decompress -Decompress the specified file(s). If a file fails to decompress, -lziprecover exits immediately without decompressing the rest of the -files. +Decompress the specified file(s). If a file does not exist or can't be +opened, lziprecover continues decompressing the rest of the files. If a +file fails to decompress, lziprecover exits immediately without +decompressing the rest of the files. @item -D @var{range} @itemx --range-decompress=@var{range} Decompress only a range of bytes starting at decompressed byte position @samp{@var{begin}} and up to byte position @w{@samp{@var{end} - 1}}. -Three formats of @var{range} are recognized, @samp{@var{begin}}, -@samp{@var{begin}-@var{end}}, and @samp{@var{begin},@var{size}}. If only -@var{begin} is specified, @var{end} is taken as the end of the file. The -produced bytes are sent to standard output unless the @samp{--output} -option is used. In order to guarantee the correctness of the data -produced, all members containing any part of the desired data are -decompressed and their integrity is verified. This operation is more -efficient in multi-member files because it only decompresses the members -containing the desired data. +This option provides random access to the data in multimember files; it +only decompresses the members containing the desired data. In order to +guarantee the correctness of the data produced, all members containing +any part of the desired data are decompressed and their integrity is +verified. + +Four formats of @var{range} are recognized, @samp{@var{begin}}, +@samp{@var{begin}-@var{end}}, @samp{@var{begin},@var{size}}, and +@samp{,@var{size}}. If only @var{begin} is specified, @var{end} is taken +as the end of the file. If only @var{size} is specified, @var{begin} is +taken as the beginning of the file. The produced bytes are sent to +standard output unless the @samp{--output} option is used. @item -f @itemx --force @@ -194,7 +225,7 @@ Force overwrite of output files. @itemx --ignore-errors Make @samp{--range-decompress} ignore data errors and continue decompressing the remaining members in the file. For example, -@w{@samp{lziprecover -i -D0 file.lz > file}} decompresses all the +@w{@samp{lziprecover -D0 -i file.lz > file}} decompresses all the recoverable data in all members of @samp{file.lz} without having to split it first. @@ -205,8 +236,8 @@ Keep (don't delete) input files during decompression. @item -l @itemx --list Print total file sizes and ratios. The values produced are correct even -for multi-member files. Use it together with @samp{-v} to see -information about the members in the file. +for multimember files. Use it together with @samp{-v} to see information +about the members in the file. @item -m @itemx --merge @@ -223,7 +254,11 @@ Place the output into @samp{@var{file}} instead of into produced are in the form @samp{rec01@var{file}}, @samp{rec02@var{file}}, etc. If decompressing from standard input and @samp{--stdout} has not been specified, use @samp{@var{file}} as the name of the decompressed -file. +file. If converting a lzma-alone file from standard input and +@samp{--stdout} has not been specified, use @samp{@var{file}.lz} as the +name of the converted file. (Or plain @samp{@var{file}} if it already +ends in @samp{.lz} or @samp{.tlz}). + @item -q @itemx --quiet @@ -299,15 +334,15 @@ caused lziprecover to panic. @cindex data safety There are 3 main types of data corruption that may cause data loss: -single-byte errors, multi-byte errors (generally affecting a whole -sector in a block device), and total device failure. +single-byte errors, multibyte errors (generally affecting a whole sector +in a block device), and total device failure. Lziprecover protects natively against single-byte errors (@pxref{Repairing files}), as long as file integrity is checked frequently enough that a second single-byte error does not develop in the same member before the first one is repaired. -Lziprecover also protects against multi-byte errors (@pxref{Merging +Lziprecover also protects against multibyte errors (@pxref{Merging files}), if at least one backup copy of the file is made. The only remedy for total device failure is storing backup copies in @@ -326,11 +361,11 @@ If you used bzip2, and if the file is large enough to contain more than one compressed data block (usually larger than 900 kB uncompressed), and if no block is damaged in both files, then the data can be manually recovered by splitting the files with bzip2recover, verifying every -block and then copying the right blocks in the right order in another +block and then copying the right blocks in the right order into another file. But if you used lzip, the data can be automatically recovered as long as -no byte is damaged in both files. +the damaged areas don't overlap. Note that each error in a bzip2 file makes a whole block unusable, but each error in a lzip file only affects the damaged bytes, making it @@ -347,8 +382,8 @@ at all. If the reparation is successful, the repaired file will be identical bit for bit to the original. This makes lzip files resistant to bit-flip, one of the most common forms of data corruption. -The error may be located anywhere in the file except in the header -(first 6 bytes of each member) or in the @samp{Member size} field of the +The error may be located anywhere in the file except in the first 5 +bytes of each member header or in the @samp{Member size} field of the trailer (last 8 bytes of each member). If the error is in the header it can be easily repaired with a text editor like GNU Moe (@pxref{File format}). If the error is in the member size, it is enough to ignore the @@ -391,21 +426,14 @@ The merge will fail if the damaged areas overlap (at least one byte is damaged in all copies), or are adjacent and the boundary can't be determined, or if the copies have too many damaged areas. -All the copies must have the same size. If some of them have been -truncated and are therefore smaller than they should, they can be -extended to the correct size with the following command before merging -them with the other copies: +All the copies must have the same size. If any of them is larger or +smaller than it should, either because it has been truncated or because +it got some garbage data appended at the end, it can be brought to the +correct size with the following command before merging it with the other +copies: @example -ddrescue --extend-outfile=<correct_size> small_file.lz extended_file.lz -@end example - -If some of the copies have got garbage data at the end and are therefore -larger than they should, their sizes can be reduced to the correct value -with the following command before merging them with the other copies: - -@example -ddrescue --size=<correct_size> large_file.lz reduced_file.lz +ddrescue -s<correct_size> -x<correct_size> file.lz correct_size_file.lz @end example To give you an idea of its possibilities, when merging two copies, each @@ -461,7 +489,7 @@ additional information before, between, or after them. Each member has the following structure: @verbatim +--+--+--+--+----+----+=============+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -| ID string | VN | DS | Lzma stream | CRC32 | Data size | Member size | +| ID string | VN | DS | LZMA stream | CRC32 | Data size | Member size | +--+--+--+--+----+----+=============+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ @end verbatim @@ -485,8 +513,8 @@ from the base size to obtain the dictionary size.@* Example: 0xD3 = 2^19 - 6 * 2^15 = 512 KiB - 6 * 32 KiB = 320 KiB@* Valid values for dictionary size range from 4 KiB to 512 MiB. -@item Lzma stream -The lzma stream, finished by an end of stream marker. Uses default +@item LZMA stream +The LZMA stream, finished by an end of stream marker. Uses default values for encoder properties. @ifnothtml @xref{Stream format,,,lzip}, @@ -506,7 +534,7 @@ Size of the uncompressed original data. @item Member size (8 bytes) Total size of the member, including header and trailer. This field acts as a distributed index, allows the verification of stream integrity, and -facilitates safe recovery of undamaged members from multi-member files. +facilitates safe recovery of undamaged members from multimember files. @end table @@ -536,7 +564,7 @@ hash value (for a chosen hash) coincide with those of another file. @item In very rare cases, trailing data could be the corrupt header of another -member. In multi-member or concatenated files the probability of +member. In multimember or concatenated files the probability of corruption happening in the magic bytes is 5 times smaller than the probability of getting a false positive caused by the corruption of the integrity information itself. Therefore it can be considered to be below @@ -544,8 +572,8 @@ the noise level. @end itemize Trailing data can be safely ignored in most cases. In some cases, like -user-added data, it is expected to be ignored. In those cases where a -file containing trailing data must be rejected, the option +that of user-added data, it is expected to be ignored. In those cases +where a file containing trailing data must be rejected, the option @samp{--trailing-error} can be used. @xref{--trailing-error}. @@ -616,7 +644,7 @@ mv file_fixed.lz file.lz @sp 1 @noindent -Example 7: Split the multi-member file @samp{file.lz} and write each +Example 7: Split the multimember file @samp{file.lz} and write each member in its own @samp{recXXXfile.lz} file. Then use @w{@samp{lziprecover -t}} to test the integrity of the resulting files. @@ -681,10 +709,22 @@ test robustness to decompression of corrupted data, inspired by unzcrash.c from Julian Seward's bzip2. Type @samp{make unzcrash} in the lziprecover source directory to build it. -Unzcrash reads the specified file and then repeatedly decompresses it, -increasing 256 times each byte of the compressed data, so as to test all -possible one-byte errors. This should not cause any invalid memory -accesses. If it does, please, report it as a bug. +By default, unzcrash reads the specified file and then repeatedly +decompresses it, increasing 256 times each byte of the compressed data, +so as to test all possible one-byte errors. + +If the @code{--block} option is given, unzcrash reads the specified file +and then repeatedly decompresses it, setting all bytes in each +successive block to the value given, so as to test all possible full +sector errors. + +If the @code{--truncate} option is given, unzcrash reads the specified +file and then repeatedly decompresses it, truncating the file to +increasing lengths, so as to test all possible truncation points. + +None of the three test modes described above should cause any invalid +memory accesses. If any of them does, please, report it as a bug to the +maintainers of the decompressor being tested. Unzcrash really executes as a subprocess the shell command specified in the first non-option argument, and then writes the file specified in the @@ -693,6 +733,27 @@ modifying the corresponding byte each time. Therefore unzcrash can be used to test any decompressor (not only lzip), or even other decoder programs having a suitable command line syntax. +If the decompressor returns with zero status, unzcrash compares the +output of the decompressor for the original and corrupt files. If the +outputs differ, it means that the decompressor returned a false +negative; it failed to recognize the corruption and produced garbage +output. The only exception is when a multimember file is truncated just +after the last byte of a member, producing a shorter but valid +compressed file. Except in this latter case, please, report any false +negative as a bug. + +In order to compare the outputs, unzcrash needs a @samp{zcmp} program +able to understand the format being tested. For example the one provided +by @samp{zutils}. +@ifnothtml +@xref{Zcmp,,,zutils}, +@end ifnothtml +@ifhtml +See +@uref{http://www.nongnu.org/zutils/manual/zutils_manual.html#Zcmp,,zcmp} +@end ifhtml + + The format for running unzcrash is: @example @@ -728,9 +789,21 @@ The number of N-bit errors per byte (N = 1 to 8) is: @item 1-3,5-8 @tab 1, 2, 3, 5, 6, 7 and 8 @end multitable +@item -B[@var{size}][,@var{value}] +@itemx --block[=@var{size}][,@var{value}] +Test block errors of given @var{size} aligned to a @var{size}-byte +boundary, simulating a whole sector I/O error. Block @var{size} defaults +to 512 bytes. @var{value} defaults to 0. + +@item -d @var{n} +@itemx --delta=@var{n} +Test only one of every @var{n} bytes, blocks or truncation sizes, +instead of all of them. + @item -p @var{bytes} @itemx --position=@var{bytes} -First byte position to test in the file. Defaults to 0. +First byte position to test in the file. Defaults to 0. Negative values +are relative to the end of the file. @item -q @itemx --quiet @@ -738,13 +811,24 @@ Quiet operation. Suppress all messages. @item -s @var{bytes} @itemx --size=@var{bytes} -Number of byte positions to test. If not specified, the whole file is -tested. +Number of byte positions to test. If not specified, the rest of the file +is tested (from @code{--position} to end of file). Negative values are +relative to the rest of the file. + +@item -t +@itemx --truncate +Test all possible truncation points in the range specified by +@code{--position} and @code{--size}. @item -v @itemx --verbose Verbose mode. +@item -z +@itemx --zcmp=<command> +Set zcmp command name and options. Defaults to @code{zcmp}. Use +@code{--zcmp=false} to disable comparisons. + @end table Exit status: 0 for a normal exit, 1 for environmental problems (file not |