diff options
Diffstat (limited to 'doc')
| -rw-r--r-- | doc/lziprecover.1 | 5 | ||||
| -rw-r--r-- | doc/lziprecover.info | 129 | ||||
| -rw-r--r-- | doc/lziprecover.texi | 101 |
3 files changed, 179 insertions, 56 deletions
diff --git a/doc/lziprecover.1 b/doc/lziprecover.1 index 99b61dd..87c0598 100644 --- a/doc/lziprecover.1 +++ b/doc/lziprecover.1 @@ -1,5 +1,5 @@ .\" DO NOT MODIFY THIS FILE! It was generated by help2man 1.46.1. -.TH LZIPRECOVER "1" "June 2015" "lziprecover 1.18-pre1" "User Commands" +.TH LZIPRECOVER "1" "September 2015" "lziprecover 1.18-pre2" "User Commands" .SH NAME lziprecover \- recovers data from damaged lzip files .SH SYNOPSIS @@ -23,6 +23,9 @@ display this help and exit \fB\-V\fR, \fB\-\-version\fR output version information and exit .TP +\fB\-a\fR, \fB\-\-trailing\-error\fR +exit with error status if trailing data +.TP \fB\-c\fR, \fB\-\-stdout\fR send decompressed output to standard output .TP diff --git a/doc/lziprecover.info b/doc/lziprecover.info index 654e60c..8d7bc66 100644 --- a/doc/lziprecover.info +++ b/doc/lziprecover.info @@ -12,7 +12,7 @@ File: lziprecover.info, Node: Top, Next: Introduction, Up: (dir) Lziprecover Manual ****************** -This manual is for Lziprecover (version 1.18-pre1, 30 June 2015). +This manual is for Lziprecover (version 1.18-pre2, 16 September 2015). * Menu: @@ -23,6 +23,7 @@ This manual is for Lziprecover (version 1.18-pre1, 30 June 2015). * Merging files:: Fixing several damaged copies * File names:: Names of the files produced by lziprecover * File format:: Detailed format of the compressed file +* Trailing data:: Extra data appended to the file * Examples:: A small tutorial with examples * Unzcrash:: Testing the robustness of decompressors * Problems:: Reporting bugs @@ -54,7 +55,7 @@ availability: recovery means. The lziprecover program can repair bit-flip errors (one of the most common forms of data corruption) in lzip files, and provides data recovery capabilities, including error-checked - merging of damaged copies of a file. + merging of damaged copies of a file. *Note Data safety::. * The lzip format is as simple as possible (but not simpler). The lzip manual provides the code of a simple decompressor along with @@ -125,6 +126,13 @@ The format for running lziprecover is: Print the version number of lziprecover on the standard output and exit. +'-a' +'--trailing-error' + Exit with error status 2 if any remaining input is detected after + decompressing the last member. Such remaining input is usually + trailing garbage that can be safely ignored. *Note + concat-example::. + '-c' '--stdout' Decompress to standard output. Needed when reading from a named @@ -133,7 +141,9 @@ The format for running lziprecover is: '-d' '--decompress' - Decompress. + Decompress the specified file(s). If a file fails to decompress, + lziprecover exits immediately without decompressing the rest of the + files. '-D RANGE' '--range-decompress=RANGE' @@ -220,7 +230,8 @@ The format for running lziprecover is: Check integrity of the specified file(s), but don't decompress them. This really performs a trial decompression and throws away the result. Use it together with '-v' to see information about - the file. + the file(s). If a file fails the test, lziprecover continues + checking the rest of the files. '-v' '--verbose' @@ -228,7 +239,7 @@ The format for running lziprecover is: When decompressing or testing, further -v's (up to 4) increase the verbosity level, showing status, compression ratio, dictionary size, trailer contents (CRC, data size, member size), and up to 6 - bytes of trailing garbage (if any). + bytes of trailing data (if any). Numbers given as arguments to options may be followed by a multiplier @@ -387,7 +398,7 @@ original file name ends with one of the extensions '.tar.lz', '.lz' or '.tlz', the string '_fixed' is inserted before the extension. -File: lziprecover.info, Node: File format, Next: Examples, Prev: File names, Up: Top +File: lziprecover.info, Node: File format, Next: Trailing data, Prev: File names, Up: Top 7 File format ************* @@ -421,7 +432,7 @@ additional information before, between, or after them. All multibyte values are stored in little endian order. -'ID string' +'ID string (the "magic" bytes)' A four byte string, identifying the lzip format, with the value "LZIP" (0x4C, 0x5A, 0x49, 0x50). @@ -458,9 +469,42 @@ additional information before, between, or after them. -File: lziprecover.info, Node: Examples, Next: Unzcrash, Prev: File format, Up: Top +File: lziprecover.info, Node: Trailing data, Next: Examples, Prev: File format, Up: Top + +8 Extra data appended to the file +********************************* + +Sometimes extra data is found appended to a lzip file after the last +member. Such trailing data may be: + + * Padding added to make the file size a multiple of some block size, + for example when writing to a tape. + + * Garbage added by some not totally successful copy operation. + + * Useful data added by the user; a cryptographically secure hash, a + description of file contents, etc. + + * Malicious data added to the file in order to make its total size + and hash value (for a chosen hash) coincide with those of another + file. + + * In very rare cases, trailing data could be the corrupt header of + another member. In multi-member or concatenated files the + probability of corruption happening in the magic bytes is 5 times + smaller than the probability of getting a false positive caused by + the corruption of the integrity information itself. Therefore it + can be considered to be below the noise level. + + Trailing data can be safely ignored in most cases. In some cases, +like user-added data, it is expected to be ignored. In those cases +where a file containing trailing data must be rejected, the option +'--trailing-error' can be used. *Note --trailing-error::. + + +File: lziprecover.info, Node: Examples, Next: Unzcrash, Prev: Trailing data, Up: Top -8 A small tutorial with examples +9 A small tutorial with examples ******************************** Example 1: Restore a regular file from its compressed version @@ -475,29 +519,38 @@ show status. lziprecover -tv file.lz -Example 3: Decompress 'file.lz' partially until 10 KiB of decompressed +Example 3: The right way of concatenating compressed files. *Note +Trailing data::. + + Don't do this + cat file1.lz file2.lz file3.lz | lziprecover -d + Do this instead + lziprecover -cd file1.lz file2.lz file3.lz + + +Example 4: Decompress 'file.lz' partially until 10 KiB of decompressed data are produced. lziprecover -D 0,10KiB file.lz -Example 4: Decompress 'file.lz' partially from decompressed byte 10000 +Example 5: Decompress 'file.lz' partially from decompressed byte 10000 to decompressed byte 15000 (5000 bytes are produced). lziprecover -D 10000-15000 file.lz -Example 5: Repair small errors in the file 'file.lz'. (Indented lines +Example 6: Repair small errors in the file 'file.lz'. (Indented lines are abridged diagnostic messages from lziprecover). lziprecover -v -R file.lz Copy of input file repaired successfully. lziprecover -tv file_fixed.lz - ok + file_fixed.lz: ok mv file_fixed.lz file.lz -Example 6: Split the multi-member file 'file.lz' and write each member +Example 7: Split the multi-member file 'file.lz' and write each member in its own 'recXXXfile.lz' file. Then use 'lziprecover -t' to test the integrity of the resulting files. @@ -505,26 +558,26 @@ integrity of the resulting files. lziprecover -tv rec*file.lz -Example 7: Recover a compressed backup from two copies on CD-ROM with -error-checked merging of copies (*Note GNU ddrescue manual: +Example 8: Recover a compressed backup from two copies on CD-ROM with +error-checked merging of copies. (*Note GNU ddrescue manual: (ddrescue)Top, for details about ddrescue). - ddrescue -d -r1 -b2048 /dev/cdrom cdimage1 logfile1 + ddrescue -d -r1 -b2048 /dev/cdrom cdimage1 mapfile1 mount -t iso9660 -o loop,ro cdimage1 /mnt/cdimage cp /mnt/cdimage/backup.tar.lz rescued1.tar.lz umount /mnt/cdimage (insert second copy in the CD drive) - ddrescue -d -r1 -b2048 /dev/cdrom cdimage2 logfile2 + ddrescue -d -r1 -b2048 /dev/cdrom cdimage2 mapfile2 mount -t iso9660 -o loop,ro cdimage2 /mnt/cdimage cp /mnt/cdimage/backup.tar.lz rescued2.tar.lz umount /mnt/cdimage lziprecover -m -v -o backup.tar.lz rescued1.tar.lz rescued2.tar.lz Input files merged successfully. lziprecover -tv backup.tar.lz - ok + backup.tar.lz: ok -Example 8: Recover the first volume of those created with the command +Example 9: Recover the first volume of those created with the command 'lzip -b 32MiB -S 650MB big_db' from two copies, 'big_db1_00001.lz' and 'big_db2_00001.lz', with member 07 damaged in the first copy, member 18 damaged in the second copy, and member 12 damaged in both copies. The @@ -533,13 +586,13 @@ correct file produced is saved in 'big_db_00001.lz'. lziprecover -m -v -o big_db_00001.lz big_db1_00001.lz big_db2_00001.lz Input files merged successfully. lziprecover -tv big_db_00001.lz - ok + big_db_00001.lz: ok File: lziprecover.info, Node: Unzcrash, Next: Problems, Prev: Examples, Up: Top -9 Testing the robustness of decompressors -***************************************** +10 Testing the robustness of decompressors +****************************************** The lziprecover package also includes unzcrash, a program written to test robustness to decompression of corrupted data, inspired by @@ -615,7 +668,7 @@ caused unzcrash to panic. File: lziprecover.info, Node: Problems, Next: Concept index, Prev: Unzcrash, Up: Top -10 Reporting bugs +11 Reporting bugs ***************** There are probably bugs in lziprecover. There are certainly errors and @@ -646,24 +699,28 @@ Concept index * invoking: Invoking lziprecover. (line 6) * merging files: Merging files. (line 6) * repairing files: Repairing files. (line 6) +* trailing data: Trailing data. (line 6) * unzcrash: Unzcrash. (line 6) Tag Table: Node: Top231 -Node: Introduction1214 -Node: Invoking lziprecover4310 -Node: Data safety9743 -Node: Repairing files11667 -Node: Merging files13569 -Node: File names15410 -Node: File format15874 -Node: Examples18278 -Ref: ddrescue-example19524 -Node: Unzcrash20780 -Node: Problems23334 -Node: Concept index23886 +Node: Introduction1278 +Node: Invoking lziprecover4395 +Ref: --trailing-error4860 +Node: Data safety10294 +Node: Repairing files12218 +Node: Merging files14120 +Node: File names15961 +Node: File format16425 +Node: Trailing data18854 +Node: Examples20230 +Ref: concat-example20661 +Ref: ddrescue-example21725 +Node: Unzcrash23015 +Node: Problems25571 +Node: Concept index26123 End Tag Table diff --git a/doc/lziprecover.texi b/doc/lziprecover.texi index 29045e7..e29a59f 100644 --- a/doc/lziprecover.texi +++ b/doc/lziprecover.texi @@ -6,8 +6,8 @@ @finalout @c %**end of header -@set UPDATED 30 June 2015 -@set VERSION 1.18-pre1 +@set UPDATED 16 September 2015 +@set VERSION 1.18-pre2 @dircategory Data Compression @direntry @@ -42,6 +42,7 @@ This manual is for Lziprecover (version @value{VERSION}, @value{UPDATED}). * Merging files:: Fixing several damaged copies * File names:: Names of the files produced by lziprecover * File format:: Detailed format of the compressed file +* Trailing data:: Extra data appended to the file * Examples:: A small tutorial with examples * Unzcrash:: Testing the robustness of decompressors * Problems:: Reporting bugs @@ -75,7 +76,7 @@ The lzip format provides very safe integrity checking and some data recovery means. The lziprecover program can repair bit-flip errors (one of the most common forms of data corruption) in lzip files, and provides data recovery capabilities, including error-checked merging of damaged -copies of a file. +copies of a file. @xref{Data safety}. @item The lzip format is as simple as possible (but not simpler). The lzip @@ -152,6 +153,13 @@ Print an informative help message describing the options and exit. @itemx --version Print the version number of lziprecover on the standard output and exit. +@anchor{--trailing-error} +@item -a +@itemx --trailing-error +Exit with error status 2 if any remaining input is detected after +decompressing the last member. Such remaining input is usually trailing +garbage that can be safely ignored. @xref{concat-example}. + @item -c @itemx --stdout Decompress to standard output. Needed when reading from a named pipe @@ -160,7 +168,9 @@ data as possible when decompressing a corrupt file. @item -d @itemx --decompress -Decompress. +Decompress the specified file(s). If a file fails to decompress, +lziprecover exits immediately without decompressing the rest of the +files. @item -D @var{range} @itemx --range-decompress=@var{range} @@ -246,7 +256,9 @@ on the number of members in @samp{@var{file}}. @itemx --test Check integrity of the specified file(s), but don't decompress them. This really performs a trial decompression and throws away the result. -Use it together with @samp{-v} to see information about the file. +Use it together with @samp{-v} to see information about the file(s). If +a file fails the test, lziprecover continues checking the rest of the +files. @item -v @itemx --verbose @@ -254,7 +266,7 @@ Verbose mode.@* When decompressing or testing, further -v's (up to 4) increase the verbosity level, showing status, compression ratio, dictionary size, trailer contents (CRC, data size, member size), and up to 6 bytes of -trailing garbage (if any). +trailing data (if any). @end table @@ -456,7 +468,7 @@ Each member has the following structure: All multibyte values are stored in little endian order. @table @samp -@item ID string +@item ID string (the "magic" bytes) A four byte string, identifying the lzip format, with the value "LZIP" (0x4C, 0x5A, 0x49, 0x50). @@ -499,6 +511,44 @@ facilitates safe recovery of undamaged members from multi-member files. @end table +@node Trailing data +@chapter Extra data appended to the file +@cindex trailing data + +Sometimes extra data is found appended to a lzip file after the last +member. Such trailing data may be: + +@itemize @bullet +@item +Padding added to make the file size a multiple of some block size, for +example when writing to a tape. + +@item +Garbage added by some not totally successful copy operation. + +@item +Useful data added by the user; a cryptographically secure hash, a +description of file contents, etc. + +@item +Malicious data added to the file in order to make its total size and +hash value (for a chosen hash) coincide with those of another file. + +@item +In very rare cases, trailing data could be the corrupt header of another +member. In multi-member or concatenated files the probability of +corruption happening in the magic bytes is 5 times smaller than the +probability of getting a false positive caused by the corruption of the +integrity information itself. Therefore it can be considered to be below +the noise level. +@end itemize + +Trailing data can be safely ignored in most cases. In some cases, like +user-added data, it is expected to be ignored. In those cases where a +file containing trailing data must be rejected, the option +@samp{--trailing-error} can be used. @xref{--trailing-error}. + + @node Examples @chapter A small tutorial with examples @cindex examples @@ -521,8 +571,21 @@ lziprecover -tv file.lz @end example @sp 1 +@anchor{concat-example} +@noindent +Example 3: The right way of concatenating compressed files. +@xref{Trailing data}. + +@example +Don't do this + cat file1.lz file2.lz file3.lz | lziprecover -d +Do this instead + lziprecover -cd file1.lz file2.lz file3.lz +@end example + +@sp 1 @noindent -Example 3: Decompress @samp{file.lz} partially until 10 KiB of +Example 4: Decompress @samp{file.lz} partially until 10 KiB of decompressed data are produced. @example @@ -531,7 +594,7 @@ lziprecover -D 0,10KiB file.lz @sp 1 @noindent -Example 4: Decompress @samp{file.lz} partially from decompressed byte +Example 5: Decompress @samp{file.lz} partially from decompressed byte 10000 to decompressed byte 15000 (5000 bytes are produced). @example @@ -540,20 +603,20 @@ lziprecover -D 10000-15000 file.lz @sp 1 @noindent -Example 5: Repair small errors in the file @samp{file.lz}. (Indented +Example 6: Repair small errors in the file @samp{file.lz}. (Indented lines are abridged diagnostic messages from lziprecover). @example lziprecover -v -R file.lz Copy of input file repaired successfully. lziprecover -tv file_fixed.lz - ok + file_fixed.lz: ok mv file_fixed.lz file.lz @end example @sp 1 @noindent -Example 6: Split the multi-member file @samp{file.lz} and write each +Example 7: Split the multi-member file @samp{file.lz} and write each member in its own @samp{recXXXfile.lz} file. Then use @w{@samp{lziprecover -t}} to test the integrity of the resulting files. @@ -565,8 +628,8 @@ lziprecover -tv rec*file.lz @sp 1 @anchor{ddrescue-example} @noindent -Example 7: Recover a compressed backup from two copies on CD-ROM with -error-checked merging of copies +Example 8: Recover a compressed backup from two copies on CD-ROM with +error-checked merging of copies. @ifnothtml (@xref{Top,GNU ddrescue manual,,ddrescue}, @end ifnothtml @@ -577,24 +640,24 @@ error-checked merging of copies for details about ddrescue). @example -ddrescue -d -r1 -b2048 /dev/cdrom cdimage1 logfile1 +ddrescue -d -r1 -b2048 /dev/cdrom cdimage1 mapfile1 mount -t iso9660 -o loop,ro cdimage1 /mnt/cdimage cp /mnt/cdimage/backup.tar.lz rescued1.tar.lz umount /mnt/cdimage (insert second copy in the CD drive) -ddrescue -d -r1 -b2048 /dev/cdrom cdimage2 logfile2 +ddrescue -d -r1 -b2048 /dev/cdrom cdimage2 mapfile2 mount -t iso9660 -o loop,ro cdimage2 /mnt/cdimage cp /mnt/cdimage/backup.tar.lz rescued2.tar.lz umount /mnt/cdimage lziprecover -m -v -o backup.tar.lz rescued1.tar.lz rescued2.tar.lz Input files merged successfully. lziprecover -tv backup.tar.lz - ok + backup.tar.lz: ok @end example @sp 1 @noindent -Example 8: Recover the first volume of those created with the command +Example 9: Recover the first volume of those created with the command @w{@samp{lzip -b 32MiB -S 650MB big_db}} from two copies, @samp{big_db1_00001.lz} and @samp{big_db2_00001.lz}, with member 07 damaged in the first copy, member 18 damaged in the second copy, and @@ -605,7 +668,7 @@ member 12 damaged in both copies. The correct file produced is saved in lziprecover -m -v -o big_db_00001.lz big_db1_00001.lz big_db2_00001.lz Input files merged successfully. lziprecover -tv big_db_00001.lz - ok + big_db_00001.lz: ok @end example |