diff options
Diffstat (limited to 'doc/lziprecover.info')
| -rw-r--r-- | doc/lziprecover.info | 223 |
1 files changed, 148 insertions, 75 deletions
diff --git a/doc/lziprecover.info b/doc/lziprecover.info index 8d7bc66..17985d2 100644 --- a/doc/lziprecover.info +++ b/doc/lziprecover.info @@ -12,7 +12,7 @@ File: lziprecover.info, Node: Top, Next: Introduction, Up: (dir) Lziprecover Manual ****************** -This manual is for Lziprecover (version 1.18-pre2, 16 September 2015). +This manual is for Lziprecover (version 1.18, 12 May 2016). * Menu: @@ -30,7 +30,7 @@ This manual is for Lziprecover (version 1.18-pre2, 16 September 2015). * Concept index:: Index of concepts - Copyright (C) 2009-2015 Antonio Diaz Diaz. + Copyright (C) 2009-2016 Antonio Diaz Diaz. This manual is free documentation: you have unlimited permission to copy, distribute and modify it. @@ -47,6 +47,9 @@ files, produce a correct file by merging the good parts of two or more damaged copies, extract data from damaged files, decompress files and test integrity of files. + Lziprecover is not a replacement for regular backups, but a last +line of defense for the case where the backups are also damaged. + The lzip file format is designed for data sharing and long-term archiving, taking into account both data integrity and decoder availability: @@ -72,6 +75,10 @@ repair the nearer it is from the beginning of the file. Therefore, with the help of lziprecover, losing an entire archive just because of a corrupt byte near the beginning is a thing of the past. + For compressible data, multiple lzip-compressed copies have a better +chance of surviving intact than one uncompressed copy using the same +amount of storage space. + Lziprecover is able to recover or decompress files produced by any of the compressors in the lzip family; lzip, plzip, minilzip/lzlib, clzip and pdlzip. @@ -87,12 +94,11 @@ garbage data may be produced at the end of each member): lziprecover -D0 -i -o file -q file.lz - Lziprecover is able to efficiently extract a range of bytes from a -multi-member file, because it only decompresses the members containing -the desired data. + Lziprecover provides random access to the data in multimember files; +it only decompresses the members containing the desired data. Lziprecover can print correct total file sizes and ratios even for -multi-member files. +multimember files. When recovering data, lziprecover takes as arguments the names of the damaged files and writes zero or more recovered files depending on the @@ -102,9 +108,6 @@ damaged files themselves are never modified. When decompressing or testing file integrity, lziprecover behaves like lzip or lunzip. - Lziprecover is not a replacement for regular backups, but a last -line of defense for the case where the backups are also damaged. - File: lziprecover.info, Node: Invoking lziprecover, Next: Data safety, Prev: Introduction, Up: Top @@ -115,6 +118,10 @@ The format for running lziprecover is: lziprecover [OPTIONS] [FILES] +When decompressing or testing, '-' used as a FILE argument means +standard input. It can be mixed with other FILES and is read just once, +the first time it appears in the command line. + Lziprecover supports the following options: '-h' @@ -133,30 +140,51 @@ The format for running lziprecover is: trailing garbage that can be safely ignored. *Note concat-example::. +'-A' +'--alone-to-lz' + Convert lzma-alone files to lzip format without recompressing, just + adding a lzip header and trailer. The conversion minimizes the + dictionary size of the resulting file (and therefore the amount of + memory required to decompress it). Only streamed files with + default LZMA properties can be converted; non-streamed lzma-alone + files lack the end of stream marker required in lzip files. + + The name of the converted lzip file is derived from that of the + original lzma-alone file as follows: + + filename.lzma becomes filename.lz + filename.tlz becomes filename.tar.lz + anyothername becomes anyothername.lz + '-c' '--stdout' - Decompress to standard output. Needed when reading from a named - pipe (fifo) or from a device. Use it to recover as much of the + Write decompressed data to standard output; keep input files + unchanged. This option is needed when reading from a named pipe + (fifo) or from a device. Use it also to recover as much of the uncompressed data as possible when decompressing a corrupt file. '-d' '--decompress' - Decompress the specified file(s). If a file fails to decompress, - lziprecover exits immediately without decompressing the rest of the - files. + Decompress the specified file(s). If a file does not exist or + can't be opened, lziprecover continues decompressing the rest of + the files. If a file fails to decompress, lziprecover exits + immediately without decompressing the rest of the files. '-D RANGE' '--range-decompress=RANGE' Decompress only a range of bytes starting at decompressed byte - position 'BEGIN' and up to byte position 'END - 1'. Three formats - of RANGE are recognized, 'BEGIN', 'BEGIN-END', and 'BEGIN,SIZE'. - If only BEGIN is specified, END is taken as the end of the file. - The produced bytes are sent to standard output unless the - '--output' option is used. In order to guarantee the correctness - of the data produced, all members containing any part of the - desired data are decompressed and their integrity is verified. - This operation is more efficient in multi-member files because it - only decompresses the members containing the desired data. + position 'BEGIN' and up to byte position 'END - 1'. This option + provides random access to the data in multimember files; it only + decompresses the members containing the desired data. In order to + guarantee the correctness of the data produced, all members + containing any part of the desired data are decompressed and their + integrity is verified. + + Four formats of RANGE are recognized, 'BEGIN', 'BEGIN-END', + 'BEGIN,SIZE', and ',SIZE'. If only BEGIN is specified, END is taken + as the end of the file. If only SIZE is specified, BEGIN is taken + as the beginning of the file. The produced bytes are sent to + standard output unless the '--output' option is used. '-f' '--force' @@ -166,7 +194,7 @@ The format for running lziprecover is: '--ignore-errors' Make '--range-decompress' ignore data errors and continue decompressing the remaining members in the file. For example, - 'lziprecover -i -D0 file.lz > file' decompresses all the + 'lziprecover -D0 -i file.lz > file' decompresses all the recoverable data in all members of 'file.lz' without having to split it first. @@ -177,7 +205,7 @@ The format for running lziprecover is: '-l' '--list' Print total file sizes and ratios. The values produced are correct - even for multi-member files. Use it together with '-v' to see + even for multimember files. Use it together with '-v' to see information about the members in the file. '-m' @@ -195,7 +223,10 @@ The format for running lziprecover is: splitting, the names of the files produced are in the form 'rec01FILE', 'rec02FILE', etc. If decompressing from standard input and '--stdout' has not been specified, use 'FILE' as the - name of the decompressed file. + name of the decompressed file. If converting a lzma-alone file + from standard input and '--stdout' has not been specified, use + 'FILE.lz' as the name of the converted file. (Or plain 'FILE' if + it already ends in '.lz' or '.tlz'). '-q' '--quiet' @@ -270,15 +301,15 @@ File: lziprecover.info, Node: Data safety, Next: Repairing files, Prev: Invok ************************************** There are 3 main types of data corruption that may cause data loss: -single-byte errors, multi-byte errors (generally affecting a whole -sector in a block device), and total device failure. +single-byte errors, multibyte errors (generally affecting a whole sector +in a block device), and total device failure. Lziprecover protects natively against single-byte errors (*note Repairing files::), as long as file integrity is checked frequently enough that a second single-byte error does not develop in the same member before the first one is repaired. - Lziprecover also protects against multi-byte errors (*note Merging + Lziprecover also protects against multibyte errors (*note Merging files::), if at least one backup copy of the file is made. The only remedy for total device failure is storing backup copies in @@ -298,10 +329,10 @@ than one compressed data block (usually larger than 900 kB uncompressed), and if no block is damaged in both files, then the data can be manually recovered by splitting the files with bzip2recover, verifying every block and then copying the right blocks in the right -order in another file. +order into another file. But if you used lzip, the data can be automatically recovered as -long as no byte is damaged in both files. +long as the damaged areas don't overlap. Note that each error in a bzip2 file makes a whole block unusable, but each error in a lzip file only affects the damaged bytes, making it @@ -319,8 +350,8 @@ at all. If the reparation is successful, the repaired file will be identical bit for bit to the original. This makes lzip files resistant to bit-flip, one of the most common forms of data corruption. - The error may be located anywhere in the file except in the header -(first 6 bytes of each member) or in the 'Member size' field of the + The error may be located anywhere in the file except in the first 5 +bytes of each member header or in the 'Member size' field of the trailer (last 8 bytes of each member). If the error is in the header it can be easily repaired with a text editor like GNU Moe (*note File format::). If the error is in the member size, it is enough to ignore @@ -364,19 +395,13 @@ the file. is damaged in all copies), or are adjacent and the boundary can't be determined, or if the copies have too many damaged areas. - All the copies must have the same size. If some of them have been -truncated and are therefore smaller than they should, they can be -extended to the correct size with the following command before merging -them with the other copies: - - ddrescue --extend-outfile=<correct_size> small_file.lz extended_file.lz + All the copies must have the same size. If any of them is larger or +smaller than it should, either because it has been truncated or because +it got some garbage data appended at the end, it can be brought to the +correct size with the following command before merging it with the other +copies: - If some of the copies have got garbage data at the end and are -therefore larger than they should, their sizes can be reduced to the -correct value with the following command before merging them with the -other copies: - - ddrescue --size=<correct_size> large_file.lz reduced_file.lz + ddrescue -s<correct_size> -x<correct_size> file.lz correct_size_file.lz To give you an idea of its possibilities, when merging two copies, each of them with one damaged area affecting 1 percent of the copy, the @@ -427,7 +452,7 @@ additional information before, between, or after them. Each member has the following structure: +--+--+--+--+----+----+=============+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -| ID string | VN | DS | Lzma stream | CRC32 | Data size | Member size | +| ID string | VN | DS | LZMA stream | CRC32 | Data size | Member size | +--+--+--+--+----+----+=============+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ All multibyte values are stored in little endian order. @@ -450,8 +475,8 @@ additional information before, between, or after them. Example: 0xD3 = 2^19 - 6 * 2^15 = 512 KiB - 6 * 32 KiB = 320 KiB Valid values for dictionary size range from 4 KiB to 512 MiB. -'Lzma stream' - The lzma stream, finished by an end of stream marker. Uses default +'LZMA stream' + The LZMA stream, finished by an end of stream marker. Uses default values for encoder properties. *Note Stream format: (lzip)Stream format, for a complete description. @@ -465,7 +490,7 @@ additional information before, between, or after them. Total size of the member, including header and trailer. This field acts as a distributed index, allows the verification of stream integrity, and facilitates safe recovery of undamaged members from - multi-member files. + multimember files. @@ -490,15 +515,15 @@ member. Such trailing data may be: file. * In very rare cases, trailing data could be the corrupt header of - another member. In multi-member or concatenated files the + another member. In multimember or concatenated files the probability of corruption happening in the magic bytes is 5 times smaller than the probability of getting a false positive caused by the corruption of the integrity information itself. Therefore it can be considered to be below the noise level. Trailing data can be safely ignored in most cases. In some cases, -like user-added data, it is expected to be ignored. In those cases -where a file containing trailing data must be rejected, the option +like that of user-added data, it is expected to be ignored. In those +cases where a file containing trailing data must be rejected, the option '--trailing-error' can be used. *Note --trailing-error::. @@ -550,7 +575,7 @@ are abridged diagnostic messages from lziprecover). mv file_fixed.lz file.lz -Example 7: Split the multi-member file 'file.lz' and write each member +Example 7: Split the multimember file 'file.lz' and write each member in its own 'recXXXfile.lz' file. Then use 'lziprecover -t' to test the integrity of the resulting files. @@ -599,10 +624,22 @@ test robustness to decompression of corrupted data, inspired by unzcrash.c from Julian Seward's bzip2. Type 'make unzcrash' in the lziprecover source directory to build it. - Unzcrash reads the specified file and then repeatedly decompresses -it, increasing 256 times each byte of the compressed data, so as to -test all possible one-byte errors. This should not cause any invalid -memory accesses. If it does, please, report it as a bug. + By default, unzcrash reads the specified file and then repeatedly +decompresses it, increasing 256 times each byte of the compressed data, +so as to test all possible one-byte errors. + + If the '--block' option is given, unzcrash reads the specified file +and then repeatedly decompresses it, setting all bytes in each +successive block to the value given, so as to test all possible full +sector errors. + + If the '--truncate' option is given, unzcrash reads the specified +file and then repeatedly decompresses it, truncating the file to +increasing lengths, so as to test all possible truncation points. + + None of the three test modes described above should cause any invalid +memory accesses. If any of them does, please, report it as a bug to the +maintainers of the decompressor being tested. Unzcrash really executes as a subprocess the shell command specified in the first non-option argument, and then writes the file specified in @@ -611,6 +648,19 @@ modifying the corresponding byte each time. Therefore unzcrash can be used to test any decompressor (not only lzip), or even other decoder programs having a suitable command line syntax. + If the decompressor returns with zero status, unzcrash compares the +output of the decompressor for the original and corrupt files. If the +outputs differ, it means that the decompressor returned a false +negative; it failed to recognize the corruption and produced garbage +output. The only exception is when a multimember file is truncated just +after the last byte of a member, producing a shorter but valid +compressed file. Except in this latter case, please, report any false +negative as a bug. + + In order to compare the outputs, unzcrash needs a 'zcmp' program +able to understand the format being tested. For example the one provided +by 'zutils'. *Note Zcmp: (zutils)Zcmp, + The format for running unzcrash is: unzcrash [OPTIONS] "lzip -tv" FILENAME.lz @@ -642,9 +692,21 @@ programs having a suitable command line syntax. 1,3-5,8 1, 3, 4, 5 and 8 1-3,5-8 1, 2, 3, 5, 6, 7 and 8 +'-B[SIZE][,VALUE]' +'--block[=SIZE][,VALUE]' + Test block errors of given SIZE aligned to a SIZE-byte boundary, + simulating a whole sector I/O error. Block SIZE defaults to 512 + bytes. VALUE defaults to 0. + +'-d N' +'--delta=N' + Test only one of every N bytes, blocks or truncation sizes, + instead of all of them. + '-p BYTES' '--position=BYTES' - First byte position to test in the file. Defaults to 0. + First byte position to test in the file. Defaults to 0. Negative + values are relative to the end of the file. '-q' '--quiet' @@ -652,13 +714,24 @@ programs having a suitable command line syntax. '-s BYTES' '--size=BYTES' - Number of byte positions to test. If not specified, the whole file - is tested. + Number of byte positions to test. If not specified, the rest of + the file is tested (from '--position' to end of file). Negative + values are relative to the rest of the file. + +'-t' +'--truncate' + Test all possible truncation points in the range specified by + '--position' and '--size'. '-v' '--verbose' Verbose mode. +'-z' +'--zcmp=<command>' + Set zcmp command name and options. Defaults to 'zcmp'. Use + '--zcmp=false' to disable comparisons. + Exit status: 0 for a normal exit, 1 for environmental problems (file not found, invalid flags, I/O errors, etc), 2 to indicate a corrupt or @@ -706,21 +779,21 @@ Concept index Tag Table: Node: Top231 -Node: Introduction1278 -Node: Invoking lziprecover4395 -Ref: --trailing-error4860 -Node: Data safety10294 -Node: Repairing files12218 -Node: Merging files14120 -Node: File names15961 -Node: File format16425 -Node: Trailing data18854 -Node: Examples20230 -Ref: concat-example20661 -Ref: ddrescue-example21725 -Node: Unzcrash23015 -Node: Problems25571 -Node: Concept index26123 +Node: Introduction1267 +Node: Invoking lziprecover4525 +Ref: --trailing-error5175 +Node: Data safety11779 +Node: Repairing files13702 +Node: Merging files15602 +Node: File names17217 +Node: File format17681 +Node: Trailing data20109 +Node: Examples21492 +Ref: concat-example21923 +Ref: ddrescue-example22986 +Node: Unzcrash24276 +Node: Problems28786 +Node: Concept index29338 End Tag Table |