diff options
Diffstat (limited to 'doc/lziprecover.info')
| -rw-r--r-- | doc/lziprecover.info | 306 |
1 files changed, 221 insertions, 85 deletions
diff --git a/doc/lziprecover.info b/doc/lziprecover.info index 55d044b..0339c15 100644 --- a/doc/lziprecover.info +++ b/doc/lziprecover.info @@ -12,7 +12,7 @@ File: lziprecover.info, Node: Top, Next: Introduction, Up: (dir) Lziprecover Manual ****************** -This manual is for Lziprecover (version 1.20, 12 February 2018). +This manual is for Lziprecover (version 1.21, 4 January 2019). * Menu: @@ -21,6 +21,7 @@ This manual is for Lziprecover (version 1.20, 12 February 2018). * Data safety:: Protecting data from accidental loss * Repairing files:: Fixing bit flips and similar errors * Merging files:: Fixing several damaged copies +* Tarlz:: Options supporting the tar.lz format * File names:: Names of the files produced by lziprecover * File format:: Detailed format of the compressed file * Trailing data:: Extra data appended to the file @@ -30,7 +31,7 @@ This manual is for Lziprecover (version 1.20, 12 February 2018). * Concept index:: Index of concepts - Copyright (C) 2009-2018 Antonio Diaz Diaz. + Copyright (C) 2009-2019 Antonio Diaz Diaz. This manual is free documentation: you have unlimited permission to copy, distribute and modify it. @@ -43,13 +44,19 @@ File: lziprecover.info, Node: Introduction, Next: Invoking lziprecover, Prev: Lziprecover is a data recovery tool and decompressor for files in the lzip compressed data format (.lz). Lziprecover is able to repair -slightly damaged files, produce a correct file by merging the good parts -of two or more damaged copies, extract data from damaged files, +slightly damaged files, produce a correct file by merging the good +parts of two or more damaged copies, extract data from damaged files, decompress files and test integrity of files. + Lziprecover can remove the damaged members from multimember files, +for example multimember tar.lz archives. + Lziprecover provides random access to the data in multimember files; it only decompresses the members containing the desired data. + Lziprecover facilitates the management of metadata stored as trailing +data in lzip files. + Lziprecover is not a replacement for regular backups, but a last line of defense for the case where the backups are also damaged. @@ -100,7 +107,7 @@ garbage data may be produced at the end of each member): When recovering data, lziprecover takes as arguments the names of the damaged files and writes zero or more recovered files depending on the operation selected and whether the recovery succeeded or not. The -damaged files themselves are never modified. +damaged files themselves are kept unchanged. When decompressing or testing file integrity, lziprecover behaves like lzip or lunzip. @@ -132,7 +139,7 @@ the first time it appears in the command line. '-V' '--version' Print the version number of lziprecover on the standard output and - exit. + exit. This version number should be included in all bug reports. '-a' '--trailing-error' @@ -194,12 +201,15 @@ the first time it appears in the command line. '-i' '--ignore-errors' - Make '--range-decompress' ignore data errors and continue - decompressing the remaining members in the file. For example, + Make '--range-decompress' ignore errors and continue decompressing + the remaining members in the file. For example, 'lziprecover -D0 -i file.lz > file' decompresses all the recoverable data in all members of 'file.lz' without having to split it first. + Make '--list', '--dump', '--remove' and '--strip' ignore format + errors. + '-k' '--keep' Keep (don't delete) input files during decompression. @@ -213,20 +223,23 @@ the first time it appears in the command line. printed. With '-v', the dictionary size, the number of members in the file, and the amount of trailing data (if any) are also printed. With '-vv', the positions and sizes of each member in - multimember files are also printed. '-lq' can be used to verify - quickly (without decompressing) the structural integrity of the - specified files. (Use '--test' to verify the data integrity). - '-alq' additionally verifies that none of the specified files - contain trailing data. + multimember files are also printed. With '-i', format errors are + ignored, and with '-ivv', gaps between members are shown. The + member numbers shown coincide with the file numbers produced by + '--split'. + + '-lq' can be used to verify quickly (without decompressing) the + structural integrity of the specified files. (Use '--test' to + verify the data integrity). '-alq' additionally verifies that none + of the specified files contain trailing data. '-m' '--merge' Try to produce a correct file by merging the good parts of two or more damaged copies. If successful, a repaired copy is written to the file 'FILE_fixed.lz'. The exit status is 0 if a correct file - could be produced, 2 otherwise. See the chapter 'Merging files' - (*note Merging files::) for a complete description of the merge - mode. + could be produced, 2 otherwise. *Note Merging files::, for a + complete description of the merge mode. '-o FILE' '--output=FILE' @@ -248,17 +261,21 @@ the first time it appears in the command line. Try to repair a file with small errors (up to one single-byte error per member). If successful, a repaired copy is written to the file 'FILE_fixed.lz'. 'FILE' is not modified at all. The exit - status is 0 if the file could be repaired, 2 otherwise. See the - chapter 'Repairing files' (*note Repairing files::) for a complete - description of the repair mode. + status is 0 if the file could be repaired, 2 otherwise. *Note + Repairing files::, for a complete description of the repair mode. '-s' '--split' Search for members in 'FILE' and write each member in its own - '.lz' file. You can then use 'lziprecover -t' to test the - integrity of the resulting files, decompress those which are - undamaged, and try to repair or partially decompress those which - are damaged. + file. Gaps between members are detected and each gap is saved in + its own file. Trailing data (if any) are saved alone in the last + file. You can then use 'lziprecover -t' to test the integrity of + the resulting files, decompress those which are undamaged, and try + to repair or partially decompress those which are damaged. Gaps + may contain garbage or may be members with corrupt headers or + trailers. If other lziprecover functions fail to work on a + multimember FILE because of damage in headers or trailers, try to + split FILE and then work on each member individually. The names of the files produced are in the form 'rec01FILE', 'rec02FILE', etc, and are designed so that the use of wildcards in @@ -297,33 +314,75 @@ the first time it appears in the command line. if a file triggers a "corrupt header" error and the cause is not indeed a corrupt header. -'--dump-tdata' - Dump the trailing data (if any) of one or more regular files to +'--dump=[MEMBER_LIST][:damaged][:tdata]' + Dump the members listed, the damaged members (if any), or the + trailing data (if any) of one or more regular multimember files to standard output, or to a file if the '--output' option is used. If - more than one file is given, the trailing data of all files are - concatenated. If a file does not exist, can't be opened, or is not - regular, lziprecover continues processing the rest of the files. - If the dump fails in one file, lziprecover exits immediately - without processing the rest of the files. - -'--remove-tdata' - Remove the trailing data from regular files in place. The date of - each file is preserved if possible. If the removal fails in one - file, lziprecover continues processing the rest of the files. This - option may be dangerous if the file is corrupt or if the trailing - data contain a forbidden combination of characters. *Note Trailing - data::. Verify that 'lzip -cd file.lz | wc -c' and the + more than one file is given, the elements dumped from all files + are concatenated. If a file does not exist, can't be opened, or + is not regular, lziprecover continues processing the rest of the + files. If the dump fails in one file, lziprecover exits + immediately without processing the rest of the files. + + The argument to '--dump' is a colon-separated list of the following + element specifiers; a member list (1,3-6), a reverse member list + (r1,3-6), and the strings "damaged" and "tdata" (which may be + shortened to 'd' and 't' respectively). A member list selects the + members (or gaps) listed, whose numbers coincide with those shown + by '--list'. A reverse member list selects the members listed + counting from the last member in the file (r1). Negated versions + of both kinds of lists exist (^1,3-6:r^1,3-6) which selects all + the members except those in the list. The strings "damaged" and + "tdata" select the damaged members and the trailing data + respectively. If the same member is selected more than once, for + example by '1:r1' in a single-member file, it is dumped just once. + See the following examples: + + '--dump' argument Elements dumped + --------------------------------------------------------------------- + '1,3-6' members 1, 3, 4, 5 and 6 + 'r1-3' last 3 members in file + '^13,15' all but 13th and 15th members in file + 'r^1' all but last member in file + 'damaged' all damaged members in file + 'tdata' trailing data + '1-5:r1:tdata' members 1 to 5, last member, trailing data + 'damaged:tdata' damaged members, trailing data + '3,12:damaged:tdata' members 3, 12, damaged members, trailing data + +'--remove=[MEMBER_LIST][:damaged][:tdata]' + Remove the members listed, the damaged members (if any), or the + trailing data (if any) from regular multimember files in place. + The date of each file is preserved if possible. If all members in + a file are selected to be removed, the file is left unchanged and + the exit status is set to 2. If a file does not exist, can't be + opened, is not regular, or is left unchanged, lziprecover + continues processing the rest of the files. In case of I/O error, + lziprecover exits immediately without processing the rest of the + files. See '--dump' above for a description of the argument. + + This option may be dangerous even if only the trailing data is + being removed because the file may be corrupt or the trailing data + may contain a forbidden combination of characters. *Note Trailing + data::. It is advisable to make a backup before attempting the + removal. At least verify that 'lzip -cd file.lz | wc -c' and the uncompressed size shown by 'lzip -l file.lz' match before - attempting the removal. - -'--strip-tdata' - Copy one or more regular files to standard output (or to a file if - the '--output' option is used), stripping the trailing data (if - any) from each file. If more than one file is given, the files are - concatenated. If a file does not exist, can't be opened, or is not + attempting the removal of trailing data. + +'--strip=[MEMBER_LIST][:damaged][:tdata]' + Copy one or more regular multimember files to standard output (or + to a file if the '--output' option is used), stripping the members + listed, the damaged members (if any), or the trailing data (if + any) from each file. If all members in a file are selected to be + stripped, the trailing data (if any) are also stripped even if + 'tdata' is not specified. If more than one file is given, the + files are concatenated. In this case the trailing data are also + stripped from all but the last file even if 'tdata' is not + specified. If a file does not exist, can't be opened, or is not regular, lziprecover continues processing the rest of the files. If a file fails to copy, lziprecover exits immediately without - processing the rest of the files. + processing the rest of the files. See '--dump' above for a + description of the argument. Numbers given as arguments to options may be followed by a multiplier @@ -431,7 +490,7 @@ cause much more loss of data than errors located near the end. So lziprecover repairs more efficiently the worst errors. -File: lziprecover.info, Node: Merging files, Next: File names, Prev: Repairing files, Up: Top +File: lziprecover.info, Node: Merging files, Next: Tarlz, Prev: Repairing files, Up: Top 5 Merging files *************** @@ -489,9 +548,74 @@ lower than the number of corrupt bytes (3104) because contiguous corrupt bytes are counted as a single multibyte error. -File: lziprecover.info, Node: File names, Next: File format, Prev: Merging files, Up: Top +File: lziprecover.info, Node: Tarlz, Next: File names, Prev: Merging files, Up: Top + +6 Options supporting the tar.lz format +************************************** + +Tarlz is an implementation of the tar archiver which by default creates +archives compressed with lzip on a per file basis. Tarlz can append +files to the end of such compressed archives because each tar member is +compressed in its own lzip member, as well as the end-of-file blocks. +Thus tarlz archives are multimember lzip files, which has some safety +advantages over solidly compressed tar.lz archives. For example, in +case of corruption, tarlz can extract all the undamaged members from +the tar.lz archive, skipping over the damaged members, just like the +standard (uncompressed) tar. In this chapter we'll explain the ways in +which lziprecover can recover and process multimember tar.lz archives. +*Note tarlz manual: (tarlz)Top. + + +6.1 Recovering damaged multimember tar.lz archives +================================================== + +If you have several copies of the damaged archive, try merging them +first because merging has a high probability of success. If the command +below prints something like 'Input files merged successfully.' you are +done and 'archive.tar.lz' now contains the recovered archive: + + lziprecover -m -v -o archive.tar.lz a/archive.tar.lz b/archive.tar.lz -6 Names of the files produced by lziprecover + If you only have one copy of the damaged archive, you may try to +repair the archive, but this has a lower probability of success. If the +command below prints something like +'Copy of input file repaired successfully.' you are done and +'archive_fixed.tar.lz' now contains the recovered archive: + + lziprecover -v -R archive.tar.lz + + If all the above fails, you may save the damaged members for later +and then copy the good members to another archive. If the two commands +below succeed, 'bad_members.tar.lz' will contain all the damaged members +and 'archive_cleaned.tar.lz' will contain a good archive with the +damaged members removed: + + lziprecover -v --dump=damaged -o bad_members.tar.lz archive.tar.lz + lziprecover -v --strip=damaged -o archive_cleaned.tar.lz archive.tar.lz + + You can then use 'tarlz --keep-damaged' to recover as much data as +possible from each damaged member in 'bad_members.tar.lz': + + mkdir tmp + cd tmp + tarlz --keep-damaged -xvf ../bad_members.tar.lz + + +6.2 Processing multimember tar.lz archives +========================================== + +Lziprecover is able to copy a list of members from a file to another. +For example the command +'lziprecover --dump=1-10:r1:tdata archive.tar.lz > subarch.tar.lz' +creates a subset archive containing the first ten members, the +end-of-file blocks, and the trailing data (if any) of 'archive.tar.lz'. +The 'r1' part selects the last member, which in an appendable tar.lz +archive contains the end-of-file blocks. + + +File: lziprecover.info, Node: File names, Next: File format, Prev: Tarlz, Up: Top + +7 Names of the files produced by lziprecover ******************************************** The name of the fixed file produced by '--merge' and '--repair' is made @@ -502,7 +626,7 @@ original file name ends with one of the extensions '.tar.lz', '.lz' or File: lziprecover.info, Node: File format, Next: Trailing data, Prev: File names, Up: Top -7 File format +8 File format ************* Perfection is reached, not when there is no longer anything to add, but @@ -544,11 +668,11 @@ additional information before, between, or after them. 'DS (coded dictionary size, 1 byte)' The dictionary size is calculated by taking a power of 2 (the base - size) and substracting from it a fraction between 0/16 and 7/16 of + size) and subtracting from it a fraction between 0/16 and 7/16 of the base size. Bits 4-0 contain the base 2 logarithm of the base size (12 to 29). - Bits 7-5 contain the numerator of the fraction (0 to 7) to - substract from the base size to obtain the dictionary size. + Bits 7-5 contain the numerator of the fraction (0 to 7) to subtract + from the base size to obtain the dictionary size. Example: 0xD3 = 2^19 - 6 * 2^15 = 512 KiB - 6 * 32 KiB = 320 KiB Valid values for dictionary size range from 4 KiB to 512 MiB. @@ -573,7 +697,7 @@ additional information before, between, or after them. File: lziprecover.info, Node: Trailing data, Next: Examples, Prev: File format, Up: Top -8 Extra data appended to the file +9 Extra data appended to the file ********************************* Sometimes extra data are found appended to a lzip file after the last @@ -626,11 +750,11 @@ Example 1: Add a comment or description to a compressed file. # First append the comment as trailing data to a lzip file echo 'This file contains this and that' >> file.lz # This command prints the comment to standard output - lziprecover --dump-tdata file.lz + lziprecover --dump=tdata file.lz # This command outputs file.lz without the comment - lziprecover --strip-tdata file.lz + lziprecover --strip=tdata file.lz # This command removes the comment from file.lz - lziprecover --remove-tdata file.lz + lziprecover --remove=tdata file.lz Example 2: Add and verify a cryptographically secure hash. (This may be @@ -639,14 +763,14 @@ to guarantee that both file and hash have not been maliciously replaced). sha256sum < file.lz >> file.lz - lziprecover --strip-tdata file.lz | sha256sum -c \ - <(lziprecover --dump-tdata file.lz) + lziprecover --strip=tdata file.lz | sha256sum -c \ + <(lziprecover --dump=tdata file.lz) File: lziprecover.info, Node: Examples, Next: Unzcrash, Prev: Trailing data, Up: Top -9 A small tutorial with examples -******************************** +10 A small tutorial with examples +********************************* Example 1: Restore a regular file from its compressed version 'file.lz'. If the operation is successful, 'file.lz' is removed. @@ -667,6 +791,10 @@ or more compressed files. *Note Trailing data::. cat file1.lz file2.lz file3.lz | lziprecover -d Do this instead lziprecover -cd file1.lz file2.lz file3.lz + You may also concatenate the compressed files like this + lziprecover --strip=tdata file1.lz file2.lz file3.lz > file123.lz + Or keeping the trailing data of the last file like this + lziprecover --strip=damaged file1.lz file2.lz file3.lz > file123.lz Example 4: Decompress 'file.lz' partially until 10 KiB of decompressed @@ -700,8 +828,8 @@ integrity of the resulting files. Example 8: Recover a compressed backup from two copies on CD-ROM with -error-checked merging of copies. (*Note GNU ddrescue manual: -(ddrescue)Top, for details about ddrescue). +error-checked merging of copies. *Note GNU ddrescue manual: +(ddrescue)Top, for details about ddrescue. ddrescue -d -r1 -b2048 /dev/cdrom cdimage1 mapfile1 mount -t iso9660 -o loop,ro cdimage1 /mnt/cdimage @@ -732,7 +860,7 @@ correct file produced is saved in 'big_db_00001.lz'. File: lziprecover.info, Node: Unzcrash, Next: Problems, Prev: Examples, Up: Top -10 Testing the robustness of decompressors +11 Testing the robustness of decompressors ****************************************** The lziprecover package also includes unzcrash, a program written to @@ -775,14 +903,17 @@ after the last byte of a member, producing a shorter but valid compressed file. Except in this latter case, please, report any false negative as a bug. - In order to compare the outputs, unzcrash needs a 'zcmp' program -able to understand the format being tested. For example the one provided -by 'zutils'. *Note Zcmp: (zutils)Zcmp, + In order to compare the outputs, unzcrash needs a 'zcmp' program able +to understand the format being tested. For example the 'zcmp' provided +by 'zutils'. *Note Zcmp: (zutils)Zcmp. The format for running unzcrash is: unzcrash [OPTIONS] 'lzip -t' FILE.lz +FILE.lz must not contain errors and must be correctly decompressed by +the decompressor being tested for the comparisons to work. + unzcrash supports the following options: '-h' @@ -792,7 +923,7 @@ by 'zutils'. *Note Zcmp: (zutils)Zcmp, '-V' '--version' Print the version number of unzcrash on the standard output and - exit. + exit. This version number should be included in all bug reports. '-b RANGE' '--bits=RANGE' @@ -868,7 +999,10 @@ by 'zutils'. *Note Zcmp: (zutils)Zcmp, '-z' '--zcmp=<command>' Set zcmp command name and options. Defaults to 'zcmp'. Use - '--zcmp=false' to disable comparisons. + '--zcmp=false' to disable comparisons. If testing a decompressor + different from the one used by default by zcmp, it is needed to + force unzcrash and zcmp to use the same decompressor with a + command like 'unzcrash --zcmp='zcmp --lz=plzip' 'plzip -t' FILE.lz' Exit status: 0 for a normal exit, 1 for environmental problems (file @@ -879,7 +1013,7 @@ caused unzcrash to panic. File: lziprecover.info, Node: Problems, Next: Concept index, Prev: Unzcrash, Up: Top -11 Reporting bugs +12 Reporting bugs ***************** There are probably bugs in lziprecover. There are certainly errors and @@ -911,6 +1045,7 @@ Concept index * merging files: Merging files. (line 6) * options: Invoking lziprecover. (line 6) * repairing files: Repairing files. (line 6) +* tarlz: Tarlz. (line 6) * trailing data: Trailing data. (line 6) * unzcrash: Unzcrash. (line 6) * usage: Invoking lziprecover. (line 6) @@ -920,21 +1055,22 @@ Concept index Tag Table: Node: Top231 -Node: Introduction1273 -Node: Invoking lziprecover4650 -Ref: --trailing-error5300 -Node: Data safety14832 -Node: Repairing files16783 -Node: Merging files18706 -Node: File names21468 -Node: File format21932 -Node: Trailing data24360 -Node: Examples27595 -Ref: concat-example28026 -Ref: ddrescue-example29127 -Node: Unzcrash30417 -Node: Problems36055 -Node: Concept index36607 +Node: Introduction1335 +Node: Invoking lziprecover4918 +Ref: --trailing-error5628 +Node: Data safety18371 +Node: Repairing files20322 +Node: Merging files22245 +Node: Tarlz25002 +Node: File names27857 +Node: File format28313 +Node: Trailing data30739 +Node: Examples33974 +Ref: concat-example34407 +Ref: ddrescue-example35778 +Node: Unzcrash37066 +Node: Problems43130 +Node: Concept index43682 End Tag Table |