diff options
Diffstat (limited to 'doc/lziprecover.texi')
| -rw-r--r-- | doc/lziprecover.texi | 144 |
1 files changed, 99 insertions, 45 deletions
diff --git a/doc/lziprecover.texi b/doc/lziprecover.texi index 2702d70..ae3be14 100644 --- a/doc/lziprecover.texi +++ b/doc/lziprecover.texi @@ -6,8 +6,8 @@ @finalout @c %**end of header -@set UPDATED 12 May 2016 -@set VERSION 1.18 +@set UPDATED 10 April 2017 +@set VERSION 1.19 @dircategory Data Compression @direntry @@ -50,7 +50,7 @@ This manual is for Lziprecover (version @value{VERSION}, @value{UPDATED}). @end menu @sp 1 -Copyright @copyright{} 2009-2016 Antonio Diaz Diaz. +Copyright @copyright{} 2009-2017 Antonio Diaz Diaz. This manual is free documentation: you have unlimited permission to copy, distribute and modify it. @@ -61,10 +61,13 @@ to copy, distribute and modify it. @cindex introduction Lziprecover is a data recovery tool and decompressor for files in the -lzip compressed data format (.lz), able to repair slightly damaged -files, produce a correct file by merging the good parts of two or more -damaged copies, extract data from damaged files, decompress files and -test integrity of files. +lzip compressed data format (.lz). Lziprecover is able to repair +slightly damaged files, produce a correct file by merging the good parts +of two or more damaged copies, extract data from damaged files, +decompress files and test integrity of files. + +Lziprecover provides random access to the data in multimember files; it +only decompresses the members containing the desired data. Lziprecover is not a replacement for regular backups, but a last line of defense for the case where the backups are also damaged. @@ -83,10 +86,10 @@ copies of a file. @xref{Data safety}. @item The lzip format is as simple as possible (but not simpler). The lzip -manual provides the code of a simple decompressor along with a detailed -explanation of how it works, so that with the only help of the lzip -manual it would be possible for a digital archaeologist to extract the -data from a lzip file long after quantum computers eventually render +manual provides the source code of a simple decompressor along with a +detailed explanation of how it works, so that with the only help of the +lzip manual it would be possible for a digital archaeologist to extract +the data from a lzip file long after quantum computers eventually render LZMA obsolete. @item @@ -120,12 +123,6 @@ garbage data may be produced at the end of each member): lziprecover -D0 -i -o file -q file.lz @end example -Lziprecover provides random access to the data in multimember files; it -only decompresses the members containing the desired data. - -Lziprecover can print correct total file sizes and ratios even for -multimember files. - When recovering data, lziprecover takes as arguments the names of the damaged files and writes zero or more recovered files depending on the operation selected and whether the recovery succeeded or not. The @@ -134,6 +131,10 @@ damaged files themselves are never modified. When decompressing or testing file integrity, lziprecover behaves like lzip or lunzip. +LANGUAGE NOTE: Uncompressed = not compressed = plain data; it may never +have been compressed. Decompressed is used to refer to data which have +undergone the process of decompression. + @node Invoking lziprecover @chapter Invoking lziprecover @@ -235,9 +236,17 @@ Keep (don't delete) input files during decompression. @item -l @itemx --list -Print total file sizes and ratios. The values produced are correct even -for multimember files. Use it together with @samp{-v} to see information -about the members in the file. +Print the uncompressed size, compressed size and percentage saved of the +specified file(s). Trailing data are ignored. The values produced are +correct even for multimember files. If more than one file is given, a +final line containing the cumulative sizes is printed. With @samp{-v}, +the dictionary size, the number of members in the file, and the amount +of trailing data (if any) are also printed. With @samp{-vv}, the +positions and sizes of each member in multimember files are also +printed. @samp{-lq} can be used to verify quickly (without +decompressing) the structural integrity of the specified files. (Use +@samp{--test} to verify the data integrity). @samp{-alq} additionally +verifies that none of the specified files contain trailing data. @item -m @itemx --merge @@ -259,14 +268,13 @@ file. If converting a lzma-alone file from standard input and name of the converted file. (Or plain @samp{@var{file}} if it already ends in @samp{.lz} or @samp{.tlz}). - @item -q @itemx --quiet Quiet operation. Suppress all messages. @item -R @itemx --repair -Try to repair a file with small errors (up to one byte error per +Try to repair a file with small errors (up to one single-byte error per member). If successful, a repaired copy is written to the file @samp{@var{file}_fixed.lz}. @samp{@var{file}} is not modified at all. The exit status is 0 if the file could be repaired, 2 otherwise. See the @@ -292,8 +300,8 @@ on the number of members in @samp{@var{file}}. Check integrity of the specified file(s), but don't decompress them. This really performs a trial decompression and throws away the result. Use it together with @samp{-v} to see information about the file(s). If -a file fails the test, lziprecover continues checking the rest of the -files. +a file fails the test, does not exist, can't be opened, or is a +terminal, lziprecover continues checking the rest of the files. @item -v @itemx --verbose @@ -301,7 +309,10 @@ Verbose mode.@* When decompressing or testing, further -v's (up to 4) increase the verbosity level, showing status, compression ratio, dictionary size, trailer contents (CRC, data size, member size), and up to 6 bytes of -trailing data (if any). +trailing data (if any) both in hexadecimal and as a string of printable +ASCII characters.@* +In other modes, increasing verbosity levels show final status, progress +of operations, and extra information (for example, the failed areas). @end table @@ -349,7 +360,7 @@ The only remedy for total device failure is storing backup copies in separate media. How does lzip compare with gzip and bzip2 with respect to data safety? -Lets suppose that you made a backup of your valuable scientific data, +Let's suppose that you made a backup of your valuable scientific data, compressed it, and stored two copies on separate media. Years later you notice that both copies are corrupt. @@ -393,11 +404,12 @@ Bit-flip happens when one bit in the file is changed from 0 to 1 or vice versa. It may be caused by bad RAM or even by natural radiation. I have seen a case of bit-flip in a file stored on an USB flash drive. -One byte may seem small, but most file corruptions not produced by I/O -errors just affect one byte, or even one bit, of the file. Also, unlike -magnetic media, where errors usually affect a whole sector, solid-state -storage devices tend to produce single-byte errors, making of lzip the -perfect format for data stored on such devices. +One byte may seem small, but most file corruptions not produced by +transmission errors or I/O errors just affect one byte, or even one bit, +of the file. Also, unlike magnetic media, where errors usually affect a +whole sector, solid-state storage devices tend to produce single-byte +errors, making of lzip the perfect format for data stored on such +devices. Repairing a file can take some time. Small files or files with the error located near the beginning can be repaired in a few seconds. But @@ -426,11 +438,11 @@ The merge will fail if the damaged areas overlap (at least one byte is damaged in all copies), or are adjacent and the boundary can't be determined, or if the copies have too many damaged areas. -All the copies must have the same size. If any of them is larger or -smaller than it should, either because it has been truncated or because -it got some garbage data appended at the end, it can be brought to the -correct size with the following command before merging it with the other -copies: +All the copies to be merged must have the same size. If any of them is +larger or smaller than it should, either because it has been truncated +or because it got some garbage data appended at the end, it can be +brought to the correct size with the following command before merging it +with the other copies: @example ddrescue -s<correct_size> -x<correct_size> file.lz correct_size_file.lz @@ -444,6 +456,31 @@ few MB) with small errors (one sector damaged per copy), the probability approaches 100 percent even with only two copies. (Supposing that the errors are randomly located inside each copy). +Some types of solid-state device (NAND flash, for example) can produce +bursts of scattered single-bit errors. Lziprecover is able to merge +files with thousands of such scattered errors by grouping the errors +into clusters and then merging the files as if each cluster were a +single error. + +Here is a real case of successful merging. Two copies of the file +@samp{icecat-3.5.3-x86.tar.lz} (compressed size 9 MB) became corrupt +while stored on the same NAND flash device. One of the copies had 76 +single-bit errors scattered in an area of 1020 bytes, and the other had +3028 such errors in an area of 31729 bytes. Lziprecover produced a +correct file, identical to the original, in just 5 seconds: + +@example +$ lziprecover -vvm a/icecat-3.5.3-x86.tar.lz b/icecat-3.5.3-x86.tar.lz +Merging member 1 of 1 (2552 errors) + 2552 errors have been grouped in 16 clusters. + Trying variation 2 of 2, block 2 +Input files merged successfully. +@end example + +Note that the number of errors reported by lziprecover (2552) is lower +than the number of corrupt bytes (3104) because contiguous corrupt bytes +are counted as a single multibyte error. + @node File names @chapter Names of the files produced by lziprecover @@ -543,20 +580,24 @@ facilitates safe recovery of undamaged members from multimember files. @chapter Extra data appended to the file @cindex trailing data -Sometimes extra data is found appended to a lzip file after the last +Sometimes extra data are found appended to a lzip file after the last member. Such trailing data may be: @itemize @bullet @item Padding added to make the file size a multiple of some block size, for -example when writing to a tape. +example when writing to a tape. It is safe to append any amount of +padding zero bytes to a lzip file. @item -Garbage added by some not totally successful copy operation. +Useful data added by the user; a cryptographically secure hash, a +description of file contents, etc. It is safe to append any amount of +text to a lzip file as long as the text does not begin with the string +"LZIP", and does not contain any zero bytes (null characters). Nonzero +bytes and zero bytes can't be safely mixed in trailing data. @item -Useful data added by the user; a cryptographically secure hash, a -description of file contents, etc. +Garbage added by some not totally successful copy operation. @item Malicious data added to the file in order to make its total size and @@ -571,8 +612,12 @@ integrity information itself. Therefore it can be considered to be below the noise level. @end itemize +Trailing data are in no way part of the lzip file format, but tools +reading lzip files are expected to behave as correctly and usefully as +possible in the presence of trailing data. + Trailing data can be safely ignored in most cases. In some cases, like -that of user-added data, it is expected to be ignored. In those cases +that of user-added data, they are expected to be ignored. In those cases where a file containing trailing data must be rejected, the option @samp{--trailing-error} can be used. @xref{--trailing-error}. @@ -601,8 +646,8 @@ lziprecover -tv file.lz @sp 1 @anchor{concat-example} @noindent -Example 3: The right way of concatenating compressed files. -@xref{Trailing data}. +Example 3: The right way of concatenating the decompressed output of two +or more compressed files. @xref{Trailing data}. @example Don't do this @@ -753,7 +798,6 @@ See @uref{http://www.nongnu.org/zutils/manual/zutils_manual.html#Zcmp,,zcmp} @end ifhtml - The format for running unzcrash is: @example @@ -800,6 +844,16 @@ to 512 bytes. @var{value} defaults to 0. Test only one of every @var{n} bytes, blocks or truncation sizes, instead of all of them. +@item -e @var{position},@var{value} +@itemx --set-byte=@var{position},@var{value} +Set byte at @var{position} to @var{value} in the internal buffer after +reading and testing @var{filename}.lz but before the first test call to +the decompressor. If @var{value} is preceded by @samp{+}, it is added to +the original value of the byte at @var{position}. If @var{value} is +preceded by @samp{f} (flip), it is XORed with the original value of the +byte at @var{position}. This option can be used to run tests with a +changed dictionary size, for example. + @item -p @var{bytes} @itemx --position=@var{bytes} First byte position to test in the file. Defaults to 0. Negative values |