From 3d3968e451576fa317143140cb93242dba81f418 Mon Sep 17 00:00:00 2001 From: Daniel Baumann Date: Sat, 7 Nov 2015 16:36:59 +0100 Subject: Merging upstream version 1.2~pre1. Signed-off-by: Daniel Baumann --- doc/plzip.1 | 6 +- doc/plzip.info | 188 +++++++++++++------------ doc/plzip.texi | 415 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ doc/plzip.texinfo | 404 ---------------------------------------------------- 4 files changed, 517 insertions(+), 496 deletions(-) create mode 100644 doc/plzip.texi delete mode 100644 doc/plzip.texinfo (limited to 'doc') diff --git a/doc/plzip.1 b/doc/plzip.1 index 291a67f..11bd052 100644 --- a/doc/plzip.1 +++ b/doc/plzip.1 @@ -1,5 +1,5 @@ .\" DO NOT MODIFY THIS FILE! It was generated by help2man 1.37.1. -.TH PLZIP "1" "September 2013" "Plzip 1.1" "User Commands" +.TH PLZIP "1" "January 2014" "Plzip 1.2-pre1" "User Commands" .SH NAME Plzip \- reduces the size of files .SH SYNOPSIS @@ -84,8 +84,8 @@ Plzip home page: http://www.nongnu.org/lzip/plzip.html .SH COPYRIGHT Copyright \(co 2009 Laszlo Ersek. .br -Copyright \(co 2013 Antonio Diaz Diaz. -Using Lzlib 1.5 +Copyright \(co 2014 Antonio Diaz Diaz. +Using Lzlib 1.6\-pre1 License GPLv3+: GNU GPL version 3 or later .br This is free software: you are free to change and redistribute it. diff --git a/doc/plzip.info b/doc/plzip.info index 7461e06..717471c 100644 --- a/doc/plzip.info +++ b/doc/plzip.info @@ -1,5 +1,4 @@ -This is plzip.info, produced by makeinfo version 4.13 from -plzip.texinfo. +This is plzip.info, produced by makeinfo version 4.13+ from plzip.texi. INFO-DIR-SECTION Data Compression START-INFO-DIR-ENTRY @@ -12,7 +11,7 @@ File: plzip.info, Node: Top, Next: Introduction, Up: (dir) Plzip Manual ************ -This manual is for Plzip (version 1.1, 17 September 2013). +This manual is for Plzip (version 1.2-pre1, 20 January 2014). * Menu: @@ -24,7 +23,7 @@ This manual is for Plzip (version 1.1, 17 September 2013). * Concept index:: Index of concepts - Copyright (C) 2009, 2010, 2011, 2012, 2013 Antonio Diaz Diaz. + Copyright (C) 2009, 2010, 2011, 2012, 2013, 2014 Antonio Diaz Diaz. This manual is free documentation: you have unlimited permission to copy, distribute and modify it. @@ -41,12 +40,9 @@ the one of lzip, bzip2 or gzip. Plzip can compress/decompress large files on multiprocessor machines much faster than lzip, at the cost of a slightly reduced compression -ratio. On files large enough (several GB), plzip can use hundreds of -processors. On files of only a few MB it is better to use lzip. - - Plzip uses the same well-defined exit status values used by lzip and -bzip2, which makes it safer when used in pipes or scripts than -compressors returning ambiguous warning values, like gzip. +ratio. Note that the number of usable threads is limited by file size, +so on files larger than a few GB plzip can use hundreds of processors, +but on files of only a few MB plzip is no faster than lzip. Plzip uses the lzip file format; the files produced by plzip are fully compatible with lzip-1.4 or newer, and can be rescued with @@ -71,12 +67,27 @@ lziprecover program. Lziprecover makes lzip files resistant to bit-flip recovery capabilities, including error-checked merging of damaged copies of a file. - Plzip replaces every file given in the command line with a compressed -version of itself, with the name "original_name.lz". Each compressed -file has the same modification date, permissions, and, when possible, -ownership as the corresponding original, so that these properties can be -correctly restored at decompression time. Plzip is able to read from -some types of non regular files if the `--stdout' option is specified. + Plzip uses the same well-defined exit status values used by lzip and +bzip2, which makes it safer than compressors returning ambiguous warning +values (like gzip) when it is used as a back end for tar or zutils. + + When compressing, plzip replaces every file given in the command line +with a compressed version of itself, with the name "original_name.lz". +When decompressing, plzip attempts to guess the name for the +decompressed file from that of the compressed file as follows: + +filename.lz becomes filename +filename.tlz becomes filename.tar +anyothername becomes anyothername.out + + (De)compressing a file is much like copying or moving it; therefore +plzip preserves the access and modification dates, permissions, and, +when possible, ownership of the file just as "cp -p" does. (If the user +ID or the group ID can't be duplicated, the file permission bits +S_ISUID and S_ISGID are cleared). + + Plzip is able to read from some types of non regular files if the +'--stdout' option is specified. If no file names are specified, plzip compresses (or decompresses) from standard input to standard output. In this case, plzip will @@ -88,19 +99,12 @@ two or more compressed files. The result is the concatenation of the corresponding uncompressed files. Integrity testing of concatenated compressed files is also supported. - When decompressing, plzip attempts to guess the name for the -decompressed file from that of the compressed file as follows: - -filename.lz becomes filename -filename.tlz becomes filename.tar -anyothername becomes anyothername.out - WARNING! Even if plzip is bug-free, other causes may result in a corrupt compressed file (bugs in the system libraries, memory errors, etc). Therefore, if the data you are going to compress is important, -give the `--keep' option to plzip and do not remove the original file +give the '--keep' option to plzip and do not remove the original file until you verify the compressed file with a command like -`plzip -cd file.lz | cmp file -'. +'plzip -cd file.lz | cmp file -'.  File: plzip.info, Node: Program design, Next: Invoking plzip, Prev: Introduction, Up: Top @@ -137,73 +141,78 @@ The format for running plzip is: Plzip supports the following options: -`-h' -`--help' +'-h' +'--help' Print an informative help message describing the options and exit. -`-V' -`--version' +'-V' +'--version' Print the version number of plzip on the standard output and exit. -`-B BYTES' -`--data-size=BYTES' +'-B BYTES' +'--data-size=BYTES' Set the input data block size in bytes. The input file will be divided in chunks of this size before compression is performed. Valid values range from 8 KiB to 1 GiB. Default value is two times the dictionary size. Plzip will reduce the dictionary size if it is larger than the chosen data size. -`-c' -`--stdout' +'-c' +'--stdout' Compress or decompress to standard output. Needed when reading from a named pipe (fifo) or from a device. -`-d' -`--decompress' +'-d' +'--decompress' Decompress. -`-f' -`--force' +'-f' +'--force' Force overwrite of output files. -`-F' -`--recompress' - Force recompression of files whose name already has the `.lz' or - `.tlz' suffix. +'-F' +'--recompress' + Force recompression of files whose name already has the '.lz' or + '.tlz' suffix. -`-k' -`--keep' +'-k' +'--keep' Keep (don't delete) input files during compression or decompression. -`-m BYTES' -`--match-length=BYTES' +'-m BYTES' +'--match-length=BYTES' Set the match length limit in bytes. After a match this long is found, the search is finished. Valid values range from 5 to 273. Larger values usually give better compression ratios but longer compression times. -`-n N' -`--threads=N' +'-n N' +'--threads=N' Set the number of worker threads. Valid values range from 1 to "as many as your system can support". If this option is not used, plzip tries to detect the number of processors in the system and - use it as default value. `plzip --help' shows the system's default + use it as default value. 'plzip --help' shows the system's default value. -`-o FILE' -`--output=FILE' - When reading from standard input and `--stdout' has not been - specified, use `FILE' as the virtual name of the uncompressed - file. This produces a file named `FILE' when decompressing, and a - file named `FILE.lz' when compressing. + Note that the number of usable threads is limited to + ceil( file_size / data_size ) during compression (*note + --data-size::), and to the number of members in the input during + decompression. + +'-o FILE' +'--output=FILE' + When reading from standard input and '--stdout' has not been + specified, use 'FILE' as the virtual name of the uncompressed + file. This produces a file named 'FILE' when decompressing, and a + file named 'FILE.lz' when compressing. -`-q' -`--quiet' +'-q' +'--quiet' Quiet operation. Suppress all messages. -`-s BYTES' -`--dictionary-size=BYTES' +'-s BYTES' +'--dictionary-size=BYTES' Set the dictionary size limit in bytes. Valid values range from 4 KiB to 512 MiB. Plzip will use the smallest possible dictionary size for each member without exceeding this limit. Note that @@ -216,33 +225,33 @@ The format for running plzip is: requirement is affected at compression time by the choice of dictionary size limit. -`-t' -`--test' +'-t' +'--test' Check integrity of the specified file(s), but don't decompress them. This really performs a trial decompression and throws away - the result. Use it together with `-v' to see information about + the result. Use it together with '-v' to see information about the file. -`-v' -`--verbose' +'-v' +'--verbose' Verbose mode. When compressing, show the compression ratio for each file - processed. A second -v shows the progress of compression. + processed. A second '-v' shows the progress of compression. When decompressing or testing, further -v's (up to 4) increase the verbosity level, showing status, compression ratio, decompressed size, and compressed size. -`-1 .. -9' +'-1 .. -9' Set the compression parameters (dictionary size and match length - limit) as shown in the table below. Note that `-9' can be much - slower than `-1'. These options have no effect when decompressing. + limit) as shown in the table below. Note that '-9' can be much + slower than '-1'. These options have no effect when decompressing. The bidimensional parameter space of LZMA can't be mapped to a linear scale optimal for all files. If your files are large, very - repetitive, etc, you may need to use the `--match-length' and - `--dictionary-size' options directly to achieve optimal - performance. For example, `-9m64' usually compresses executables - more (and faster) than `-9'. + repetitive, etc, you may need to use the '--match-length' and + '--dictionary-size' options directly to achieve optimal + performance. For example, '-9m64' usually compresses executables + more (and faster) than '-9'. Level Dictionary size Match length limit -1 1 MiB 5 bytes @@ -255,13 +264,13 @@ The format for running plzip is: -8 24 MiB 132 bytes -9 32 MiB 273 bytes -`--fast' -`--best' +'--fast' +'--best' Aliases for GNU gzip compatibility. Numbers given as arguments to options may be followed by a multiplier -and an optional `B' for "byte". +and an optional 'B' for "byte". Table of SI and binary prefixes (unit multipliers): @@ -316,15 +325,15 @@ additional information before, between, or after them. All multibyte values are stored in little endian order. -`ID string' +'ID string' A four byte string, identifying the lzip format, with the value "LZIP" (0x4C, 0x5A, 0x49, 0x50). -`VN (version number, 1 byte)' +'VN (version number, 1 byte)' Just in case something needs to be modified in the future. 1 for now. -`DS (coded dictionary size, 1 byte)' +'DS (coded dictionary size, 1 byte)' Lzip divides the distance between any two powers of 2 into 8 equally spaced intervals, named "wedges". The dictionary size is calculated by taking a power of 2 (the base size) and substracting @@ -336,18 +345,18 @@ additional information before, between, or after them. Example: 0xD3 = 2^19 - 6 * 2^15 = 512 KiB - 6 * 32 KiB = 320 KiB Valid values for dictionary size range from 4 KiB to 512 MiB. -`Lzma stream' +'Lzma stream' The lzma stream, finished by an end of stream marker. Uses default values for encoder properties. See the lzip manual for a full description. -`CRC32 (4 bytes)' +'CRC32 (4 bytes)' CRC of the uncompressed original data. -`Data size (8 bytes)' +'Data size (8 bytes)' Size of the uncompressed original data. -`Member size (8 bytes)' +'Member size (8 bytes)' Total size of the member, including header and trailer. This field acts as a distributed index, allows the verification of stream integrity, and facilitates safe recovery of undamaged members from @@ -367,7 +376,7 @@ for all eternity, if not longer. If you find a bug in plzip, please send electronic mail to . Include the version number, which you can find -by running `plzip --version'. +by running 'plzip --version'.  File: plzip.info, Node: Concept index, Prev: Problems, Up: Top @@ -391,13 +400,14 @@ Concept index  Tag Table: -Node: Top223 -Node: Introduction871 -Node: Program design4426 -Node: Invoking plzip5480 -Node: File format10864 -Node: Problems13369 -Node: Concept index13898 +Node: Top221 +Node: Introduction878 +Node: Program design4650 +Node: Invoking plzip5704 +Ref: --data-size6149 +Node: File format11300 +Node: Problems13805 +Node: Concept index14334  End Tag Table diff --git a/doc/plzip.texi b/doc/plzip.texi new file mode 100644 index 0000000..413a9e3 --- /dev/null +++ b/doc/plzip.texi @@ -0,0 +1,415 @@ +\input texinfo @c -*-texinfo-*- +@c %**start of header +@setfilename plzip.info +@documentencoding ISO-8859-15 +@settitle Plzip Manual +@finalout +@c %**end of header + +@set UPDATED 20 January 2014 +@set VERSION 1.2-pre1 + +@dircategory Data Compression +@direntry +* Plzip: (plzip). Parallel compressor compatible with lzip +@end direntry + + +@ifnothtml +@titlepage +@title Plzip +@subtitle Parallel compressor compatible with lzip +@subtitle for Plzip version @value{VERSION}, @value{UPDATED} +@author by Antonio Diaz Diaz + +@page +@vskip 0pt plus 1filll +@end titlepage + +@contents +@end ifnothtml + +@node Top +@top + +This manual is for Plzip (version @value{VERSION}, @value{UPDATED}). + +@menu +* Introduction:: Purpose and features of plzip +* Program design:: Internal structure of plzip +* Invoking plzip:: Command line interface +* File format:: Detailed format of the compressed file +* Problems:: Reporting bugs +* Concept index:: Index of concepts +@end menu + +@sp 1 +Copyright @copyright{} 2009, 2010, 2011, 2012, 2013, 2014 +Antonio Diaz Diaz. + +This manual is free documentation: you have unlimited permission +to copy, distribute and modify it. + + +@node Introduction +@chapter Introduction +@cindex introduction + +Plzip is a massively parallel (multi-threaded), lossless data compressor +based on the lzlib compression library, with a user interface similar to +the one of lzip, bzip2 or gzip. + +Plzip can compress/decompress large files on multiprocessor machines +much faster than lzip, at the cost of a slightly reduced compression +ratio. Note that the number of usable threads is limited by file size, +so on files larger than a few GB plzip can use hundreds of processors, +but on files of only a few MB plzip is no faster than lzip. + +Plzip uses the lzip file format; the files produced by plzip are fully +compatible with lzip-1.4 or newer, and can be rescued with lziprecover. + +The lzip file format is designed for long-term data archiving and +provides very safe integrity checking. The member trailer stores the +32-bit CRC of the original data, the size of the original data and the +size of the member. These values, together with the value remaining in +the range decoder and the end-of-stream marker, provide a 4 factor +integrity checking which guarantees that the decompressed version of the +data is identical to the original. This guards against corruption of the +compressed data, and against undetected bugs in plzip (hopefully very +unlikely). The chances of data corruption going undetected are +microscopic. Be aware, though, that the check occurs upon decompression, +so it can only tell you that something is wrong. It can't help you +recover the original uncompressed data. + +If you ever need to recover data from a damaged lzip file, try the +lziprecover program. Lziprecover makes lzip files resistant to bit-flip +(one of the most common forms of data corruption), and provides data +recovery capabilities, including error-checked merging of damaged copies +of a file. + +Plzip uses the same well-defined exit status values used by lzip and +bzip2, which makes it safer than compressors returning ambiguous warning +values (like gzip) when it is used as a back end for tar or zutils. + +When compressing, plzip replaces every file given in the command line +with a compressed version of itself, with the name "original_name.lz". +When decompressing, plzip attempts to guess the name for the decompressed +file from that of the compressed file as follows: + +@multitable {anyothername} {becomes} {anyothername.out} +@item filename.lz @tab becomes @tab filename +@item filename.tlz @tab becomes @tab filename.tar +@item anyothername @tab becomes @tab anyothername.out +@end multitable + +(De)compressing a file is much like copying or moving it; therefore plzip +preserves the access and modification dates, permissions, and, when +possible, ownership of the file just as "cp -p" does. (If the user ID or +the group ID can't be duplicated, the file permission bits S_ISUID and +S_ISGID are cleared). + +Plzip is able to read from some types of non regular files if the +@samp{--stdout} option is specified. + +If no file names are specified, plzip compresses (or decompresses) from +standard input to standard output. In this case, plzip will decline to +write compressed output to a terminal, as this would be entirely +incomprehensible and therefore pointless. + +Plzip will correctly decompress a file which is the concatenation of two +or more compressed files. The result is the concatenation of the +corresponding uncompressed files. Integrity testing of concatenated +compressed files is also supported. + +WARNING! Even if plzip is bug-free, other causes may result in a corrupt +compressed file (bugs in the system libraries, memory errors, etc). +Therefore, if the data you are going to compress is important, give the +@samp{--keep} option to plzip and do not remove the original file until +you verify the compressed file with a command like +@w{@samp{plzip -cd file.lz | cmp file -}}. + + +@node Program design +@chapter Program design +@cindex program design + +For each input file, a splitter thread and several worker threads are +created, acting the main thread as muxer (multiplexer) thread. A "packet +courier" takes care of data transfers among threads and limits the +maximum number of data blocks (packets) being processed simultaneously. + +The splitter reads data blocks from the input file, and distributes them +to the workers. The workers (de)compress the blocks received from the +splitter. The muxer collects processed packets from the workers, and +writes them to the output file. + +When decompressing from a regular file, the splitter is removed and the +workers read directly from the input file. If the output file is also a +regular file, the muxer is also removed, and the workers write directly +to the output file. With these optimizations, decompression speed of +large files with many members is only limited by the number of +processors available and by I/O speed. + + +@node Invoking plzip +@chapter Invoking plzip +@cindex invoking +@cindex options +@cindex usage +@cindex version + +The format for running plzip is: + +@example +plzip [@var{options}] [@var{files}] +@end example + +Plzip supports the following options: + +@table @samp +@item -h +@itemx --help +Print an informative help message describing the options and exit. + +@item -V +@itemx --version +Print the version number of plzip on the standard output and exit. + +@item -B @var{bytes} +@itemx --data-size=@var{bytes} +@anchor{--data-size} +Set the input data block size in bytes. The input file will be divided +in chunks of this size before compression is performed. Valid values +range from 8 KiB to 1 GiB. Default value is two times the dictionary +size. Plzip will reduce the dictionary size if it is larger than the +chosen data size. + +@item -c +@itemx --stdout +Compress or decompress to standard output. Needed when reading from a +named pipe (fifo) or from a device. + +@item -d +@itemx --decompress +Decompress. + +@item -f +@itemx --force +Force overwrite of output files. + +@item -F +@itemx --recompress +Force recompression of files whose name already has the @samp{.lz} or +@samp{.tlz} suffix. + +@item -k +@itemx --keep +Keep (don't delete) input files during compression or decompression. + +@item -m @var{bytes} +@itemx --match-length=@var{bytes} +Set the match length limit in bytes. After a match this long is found, +the search is finished. Valid values range from 5 to 273. Larger values +usually give better compression ratios but longer compression times. + +@item -n @var{n} +@itemx --threads=@var{n} +Set the number of worker threads. Valid values range from 1 to "as many +as your system can support". If this option is not used, plzip tries to +detect the number of processors in the system and use it as default +value. @w{@samp{plzip --help}} shows the system's default value. + +Note that the number of usable threads is limited to @w{ceil( file_size +/ data_size )} during compression (@pxref{--data-size}), and to the +number of members in the input during decompression. + +@item -o @var{file} +@itemx --output=@var{file} +When reading from standard input and @samp{--stdout} has not been +specified, use @samp{@var{file}} as the virtual name of the uncompressed +file. This produces a file named @samp{@var{file}} when decompressing, +and a file named @samp{@var{file}.lz} when compressing. + +@item -q +@itemx --quiet +Quiet operation. Suppress all messages. + +@item -s @var{bytes} +@itemx --dictionary-size=@var{bytes} +Set the dictionary size limit in bytes. Valid values range from 4 KiB to +512 MiB. Plzip will use the smallest possible dictionary size for each +member without exceeding this limit. Note that dictionary sizes are +quantized. If the specified size does not match one of the valid sizes, +it will be rounded upwards by adding up to (@var{bytes} / 16) to it. + +For maximum compression you should use a dictionary size limit as large +as possible, but keep in mind that the decompression memory requirement +is affected at compression time by the choice of dictionary size limit. + +@item -t +@itemx --test +Check integrity of the specified file(s), but don't decompress them. +This really performs a trial decompression and throws away the result. +Use it together with @samp{-v} to see information about the file. + +@item -v +@itemx --verbose +Verbose mode.@* +When compressing, show the compression ratio for each file processed. A +second @samp{-v} shows the progress of compression.@* +When decompressing or testing, further -v's (up to 4) increase the +verbosity level, showing status, compression ratio, decompressed size, +and compressed size. + +@item -1 .. -9 +Set the compression parameters (dictionary size and match length limit) +as shown in the table below. Note that @samp{-9} can be much slower than +@samp{-1}. These options have no effect when decompressing. + +The bidimensional parameter space of LZMA can't be mapped to a linear +scale optimal for all files. If your files are large, very repetitive, +etc, you may need to use the @samp{--match-length} and +@samp{--dictionary-size} options directly to achieve optimal +performance. For example, @samp{-9m64} usually compresses executables +more (and faster) than @samp{-9}. + +@multitable {Level} {Dictionary size} {Match length limit} +@item Level @tab Dictionary size @tab Match length limit +@item -1 @tab 1 MiB @tab 5 bytes +@item -2 @tab 1.5 MiB @tab 6 bytes +@item -3 @tab 2 MiB @tab 8 bytes +@item -4 @tab 3 MiB @tab 12 bytes +@item -5 @tab 4 MiB @tab 20 bytes +@item -6 @tab 8 MiB @tab 36 bytes +@item -7 @tab 16 MiB @tab 68 bytes +@item -8 @tab 24 MiB @tab 132 bytes +@item -9 @tab 32 MiB @tab 273 bytes +@end multitable + +@item --fast +@itemx --best +Aliases for GNU gzip compatibility. + +@end table + +Numbers given as arguments to options may be followed by a multiplier +and an optional @samp{B} for "byte". + +Table of SI and binary prefixes (unit multipliers): + +@multitable {Prefix} {kilobyte (10^3 = 1000)} {|} {Prefix} {kibibyte (2^10 = 1024)} +@item Prefix @tab Value @tab | @tab Prefix @tab Value +@item k @tab kilobyte (10^3 = 1000) @tab | @tab Ki @tab kibibyte (2^10 = 1024) +@item M @tab megabyte (10^6) @tab | @tab Mi @tab mebibyte (2^20) +@item G @tab gigabyte (10^9) @tab | @tab Gi @tab gibibyte (2^30) +@item T @tab terabyte (10^12) @tab | @tab Ti @tab tebibyte (2^40) +@item P @tab petabyte (10^15) @tab | @tab Pi @tab pebibyte (2^50) +@item E @tab exabyte (10^18) @tab | @tab Ei @tab exbibyte (2^60) +@item Z @tab zettabyte (10^21) @tab | @tab Zi @tab zebibyte (2^70) +@item Y @tab yottabyte (10^24) @tab | @tab Yi @tab yobibyte (2^80) +@end multitable + +@sp 1 +Exit status: 0 for a normal exit, 1 for environmental problems (file not +found, invalid flags, I/O errors, etc), 2 to indicate a corrupt or +invalid input file, 3 for an internal consistency error (eg, bug) which +caused plzip to panic. + + +@node File format +@chapter File format +@cindex file format + +Perfection is reached, not when there is no longer anything to add, but +when there is no longer anything to take away.@* +--- Antoine de Saint-Exupery + +@sp 1 +In the diagram below, a box like this: +@verbatim ++---+ +| | <-- the vertical bars might be missing ++---+ +@end verbatim + +represents one byte; a box like this: +@verbatim ++==============+ +| | ++==============+ +@end verbatim + +represents a variable number of bytes. + +@sp 1 +A lzip file consists of a series of "members" (compressed data sets). +The members simply appear one after another in the file, with no +additional information before, between, or after them. + +Each member has the following structure: +@verbatim ++--+--+--+--+----+----+=============+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +| ID string | VN | DS | Lzma stream | CRC32 | Data size | Member size | ++--+--+--+--+----+----+=============+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +@end verbatim + +All multibyte values are stored in little endian order. + +@table @samp +@item ID string +A four byte string, identifying the lzip format, with the value "LZIP" +(0x4C, 0x5A, 0x49, 0x50). + +@item VN (version number, 1 byte) +Just in case something needs to be modified in the future. 1 for now. + +@item DS (coded dictionary size, 1 byte) +Lzip divides the distance between any two powers of 2 into 8 equally +spaced intervals, named "wedges". The dictionary size is calculated by +taking a power of 2 (the base size) and substracting from it a number of +wedges between 0 and 7. The size of a wedge is (base_size / 16).@* +Bits 4-0 contain the base 2 logarithm of the base size (12 to 29).@* +Bits 7-5 contain the number of wedges (0 to 7) to substract from the +base size to obtain the dictionary size.@* +Example: 0xD3 = 2^19 - 6 * 2^15 = 512 KiB - 6 * 32 KiB = 320 KiB@* +Valid values for dictionary size range from 4 KiB to 512 MiB. + +@item Lzma stream +The lzma stream, finished by an end of stream marker. Uses default values +for encoder properties. See the lzip manual for a full description. + +@item CRC32 (4 bytes) +CRC of the uncompressed original data. + +@item Data size (8 bytes) +Size of the uncompressed original data. + +@item Member size (8 bytes) +Total size of the member, including header and trailer. This field acts +as a distributed index, allows the verification of stream integrity, and +facilitates safe recovery of undamaged members from multi-member files. + +@end table + + +@node Problems +@chapter Reporting bugs +@cindex bugs +@cindex getting help + +There are probably bugs in plzip. There are certainly errors and +omissions in this manual. If you report them, they will get fixed. If +you don't, no one will ever know about them and they will remain unfixed +for all eternity, if not longer. + +If you find a bug in plzip, please send electronic mail to +@email{lzip-bug@@nongnu.org}. Include the version number, which you can +find by running @w{@samp{plzip --version}}. + + +@node Concept index +@unnumbered Concept index + +@printindex cp + +@bye diff --git a/doc/plzip.texinfo b/doc/plzip.texinfo deleted file mode 100644 index 0370677..0000000 --- a/doc/plzip.texinfo +++ /dev/null @@ -1,404 +0,0 @@ -\input texinfo @c -*-texinfo-*- -@c %**start of header -@setfilename plzip.info -@documentencoding ISO-8859-15 -@settitle Plzip Manual -@finalout -@c %**end of header - -@set UPDATED 17 September 2013 -@set VERSION 1.1 - -@dircategory Data Compression -@direntry -* Plzip: (plzip). Parallel compressor compatible with lzip -@end direntry - - -@ifnothtml -@titlepage -@title Plzip -@subtitle Parallel compressor compatible with lzip -@subtitle for Plzip version @value{VERSION}, @value{UPDATED} -@author by Antonio Diaz Diaz - -@page -@vskip 0pt plus 1filll -@end titlepage - -@contents -@end ifnothtml - -@node Top -@top - -This manual is for Plzip (version @value{VERSION}, @value{UPDATED}). - -@menu -* Introduction:: Purpose and features of plzip -* Program design:: Internal structure of plzip -* Invoking plzip:: Command line interface -* File format:: Detailed format of the compressed file -* Problems:: Reporting bugs -* Concept index:: Index of concepts -@end menu - -@sp 1 -Copyright @copyright{} 2009, 2010, 2011, 2012, 2013 Antonio Diaz Diaz. - -This manual is free documentation: you have unlimited permission -to copy, distribute and modify it. - - -@node Introduction -@chapter Introduction -@cindex introduction - -Plzip is a massively parallel (multi-threaded), lossless data compressor -based on the lzlib compression library, with a user interface similar to -the one of lzip, bzip2 or gzip. - -Plzip can compress/decompress large files on multiprocessor machines -much faster than lzip, at the cost of a slightly reduced compression -ratio. On files large enough (several GB), plzip can use hundreds of -processors. On files of only a few MB it is better to use lzip. - -Plzip uses the same well-defined exit status values used by lzip and -bzip2, which makes it safer when used in pipes or scripts than -compressors returning ambiguous warning values, like gzip. - -Plzip uses the lzip file format; the files produced by plzip are fully -compatible with lzip-1.4 or newer, and can be rescued with lziprecover. - -The lzip file format is designed for long-term data archiving and -provides very safe integrity checking. The member trailer stores the -32-bit CRC of the original data, the size of the original data and the -size of the member. These values, together with the value remaining in -the range decoder and the end-of-stream marker, provide a 4 factor -integrity checking which guarantees that the decompressed version of the -data is identical to the original. This guards against corruption of the -compressed data, and against undetected bugs in plzip (hopefully very -unlikely). The chances of data corruption going undetected are -microscopic. Be aware, though, that the check occurs upon decompression, -so it can only tell you that something is wrong. It can't help you -recover the original uncompressed data. - -If you ever need to recover data from a damaged lzip file, try the -lziprecover program. Lziprecover makes lzip files resistant to bit-flip -(one of the most common forms of data corruption), and provides data -recovery capabilities, including error-checked merging of damaged copies -of a file. - -Plzip replaces every file given in the command line with a compressed -version of itself, with the name "original_name.lz". Each compressed -file has the same modification date, permissions, and, when possible, -ownership as the corresponding original, so that these properties can be -correctly restored at decompression time. Plzip is able to read from some -types of non regular files if the @samp{--stdout} option is specified. - -If no file names are specified, plzip compresses (or decompresses) from -standard input to standard output. In this case, plzip will decline to -write compressed output to a terminal, as this would be entirely -incomprehensible and therefore pointless. - -Plzip will correctly decompress a file which is the concatenation of two -or more compressed files. The result is the concatenation of the -corresponding uncompressed files. Integrity testing of concatenated -compressed files is also supported. - -When decompressing, plzip attempts to guess the name for the decompressed -file from that of the compressed file as follows: - -@multitable {anyothername} {becomes} {anyothername.out} -@item filename.lz @tab becomes @tab filename -@item filename.tlz @tab becomes @tab filename.tar -@item anyothername @tab becomes @tab anyothername.out -@end multitable - -WARNING! Even if plzip is bug-free, other causes may result in a corrupt -compressed file (bugs in the system libraries, memory errors, etc). -Therefore, if the data you are going to compress is important, give the -@samp{--keep} option to plzip and do not remove the original file until -you verify the compressed file with a command like -@w{@samp{plzip -cd file.lz | cmp file -}}. - - -@node Program design -@chapter Program design -@cindex program design - -For each input file, a splitter thread and several worker threads are -created, acting the main thread as muxer (multiplexer) thread. A "packet -courier" takes care of data transfers among threads and limits the -maximum number of data blocks (packets) being processed simultaneously. - -The splitter reads data blocks from the input file, and distributes them -to the workers. The workers (de)compress the blocks received from the -splitter. The muxer collects processed packets from the workers, and -writes them to the output file. - -When decompressing from a regular file, the splitter is removed and the -workers read directly from the input file. If the output file is also a -regular file, the muxer is also removed, and the workers write directly -to the output file. With these optimizations, decompression speed of -large files with many members is only limited by the number of -processors available and by I/O speed. - - -@node Invoking plzip -@chapter Invoking plzip -@cindex invoking -@cindex options -@cindex usage -@cindex version - -The format for running plzip is: - -@example -plzip [@var{options}] [@var{files}] -@end example - -Plzip supports the following options: - -@table @samp -@item -h -@itemx --help -Print an informative help message describing the options and exit. - -@item -V -@itemx --version -Print the version number of plzip on the standard output and exit. - -@item -B @var{bytes} -@itemx --data-size=@var{bytes} -Set the input data block size in bytes. The input file will be divided -in chunks of this size before compression is performed. Valid values -range from 8 KiB to 1 GiB. Default value is two times the dictionary -size. Plzip will reduce the dictionary size if it is larger than the -chosen data size. - -@item -c -@itemx --stdout -Compress or decompress to standard output. Needed when reading from a -named pipe (fifo) or from a device. - -@item -d -@itemx --decompress -Decompress. - -@item -f -@itemx --force -Force overwrite of output files. - -@item -F -@itemx --recompress -Force recompression of files whose name already has the @samp{.lz} or -@samp{.tlz} suffix. - -@item -k -@itemx --keep -Keep (don't delete) input files during compression or decompression. - -@item -m @var{bytes} -@itemx --match-length=@var{bytes} -Set the match length limit in bytes. After a match this long is found, -the search is finished. Valid values range from 5 to 273. Larger values -usually give better compression ratios but longer compression times. - -@item -n @var{n} -@itemx --threads=@var{n} -Set the number of worker threads. Valid values range from 1 to "as many -as your system can support". If this option is not used, plzip tries to -detect the number of processors in the system and use it as default -value. @w{@samp{plzip --help}} shows the system's default value. - -@item -o @var{file} -@itemx --output=@var{file} -When reading from standard input and @samp{--stdout} has not been -specified, use @samp{@var{file}} as the virtual name of the uncompressed -file. This produces a file named @samp{@var{file}} when decompressing, -and a file named @samp{@var{file}.lz} when compressing. - -@item -q -@itemx --quiet -Quiet operation. Suppress all messages. - -@item -s @var{bytes} -@itemx --dictionary-size=@var{bytes} -Set the dictionary size limit in bytes. Valid values range from 4 KiB to -512 MiB. Plzip will use the smallest possible dictionary size for each -member without exceeding this limit. Note that dictionary sizes are -quantized. If the specified size does not match one of the valid sizes, -it will be rounded upwards by adding up to (@var{bytes} / 16) to it. - -For maximum compression you should use a dictionary size limit as large -as possible, but keep in mind that the decompression memory requirement -is affected at compression time by the choice of dictionary size limit. - -@item -t -@itemx --test -Check integrity of the specified file(s), but don't decompress them. -This really performs a trial decompression and throws away the result. -Use it together with @samp{-v} to see information about the file. - -@item -v -@itemx --verbose -Verbose mode.@* -When compressing, show the compression ratio for each file processed. A -second -v shows the progress of compression.@* -When decompressing or testing, further -v's (up to 4) increase the -verbosity level, showing status, compression ratio, decompressed size, -and compressed size. - -@item -1 .. -9 -Set the compression parameters (dictionary size and match length limit) -as shown in the table below. Note that @samp{-9} can be much slower than -@samp{-1}. These options have no effect when decompressing. - -The bidimensional parameter space of LZMA can't be mapped to a linear -scale optimal for all files. If your files are large, very repetitive, -etc, you may need to use the @samp{--match-length} and -@samp{--dictionary-size} options directly to achieve optimal -performance. For example, @samp{-9m64} usually compresses executables -more (and faster) than @samp{-9}. - -@multitable {Level} {Dictionary size} {Match length limit} -@item Level @tab Dictionary size @tab Match length limit -@item -1 @tab 1 MiB @tab 5 bytes -@item -2 @tab 1.5 MiB @tab 6 bytes -@item -3 @tab 2 MiB @tab 8 bytes -@item -4 @tab 3 MiB @tab 12 bytes -@item -5 @tab 4 MiB @tab 20 bytes -@item -6 @tab 8 MiB @tab 36 bytes -@item -7 @tab 16 MiB @tab 68 bytes -@item -8 @tab 24 MiB @tab 132 bytes -@item -9 @tab 32 MiB @tab 273 bytes -@end multitable - -@item --fast -@itemx --best -Aliases for GNU gzip compatibility. - -@end table - -Numbers given as arguments to options may be followed by a multiplier -and an optional @samp{B} for "byte". - -Table of SI and binary prefixes (unit multipliers): - -@multitable {Prefix} {kilobyte (10^3 = 1000)} {|} {Prefix} {kibibyte (2^10 = 1024)} -@item Prefix @tab Value @tab | @tab Prefix @tab Value -@item k @tab kilobyte (10^3 = 1000) @tab | @tab Ki @tab kibibyte (2^10 = 1024) -@item M @tab megabyte (10^6) @tab | @tab Mi @tab mebibyte (2^20) -@item G @tab gigabyte (10^9) @tab | @tab Gi @tab gibibyte (2^30) -@item T @tab terabyte (10^12) @tab | @tab Ti @tab tebibyte (2^40) -@item P @tab petabyte (10^15) @tab | @tab Pi @tab pebibyte (2^50) -@item E @tab exabyte (10^18) @tab | @tab Ei @tab exbibyte (2^60) -@item Z @tab zettabyte (10^21) @tab | @tab Zi @tab zebibyte (2^70) -@item Y @tab yottabyte (10^24) @tab | @tab Yi @tab yobibyte (2^80) -@end multitable - -@sp 1 -Exit status: 0 for a normal exit, 1 for environmental problems (file not -found, invalid flags, I/O errors, etc), 2 to indicate a corrupt or -invalid input file, 3 for an internal consistency error (eg, bug) which -caused plzip to panic. - - -@node File format -@chapter File format -@cindex file format - -Perfection is reached, not when there is no longer anything to add, but -when there is no longer anything to take away.@* ---- Antoine de Saint-Exupery - -@sp 1 -In the diagram below, a box like this: -@verbatim -+---+ -| | <-- the vertical bars might be missing -+---+ -@end verbatim - -represents one byte; a box like this: -@verbatim -+==============+ -| | -+==============+ -@end verbatim - -represents a variable number of bytes. - -@sp 1 -A lzip file consists of a series of "members" (compressed data sets). -The members simply appear one after another in the file, with no -additional information before, between, or after them. - -Each member has the following structure: -@verbatim -+--+--+--+--+----+----+=============+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -| ID string | VN | DS | Lzma stream | CRC32 | Data size | Member size | -+--+--+--+--+----+----+=============+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -@end verbatim - -All multibyte values are stored in little endian order. - -@table @samp -@item ID string -A four byte string, identifying the lzip format, with the value "LZIP" -(0x4C, 0x5A, 0x49, 0x50). - -@item VN (version number, 1 byte) -Just in case something needs to be modified in the future. 1 for now. - -@item DS (coded dictionary size, 1 byte) -Lzip divides the distance between any two powers of 2 into 8 equally -spaced intervals, named "wedges". The dictionary size is calculated by -taking a power of 2 (the base size) and substracting from it a number of -wedges between 0 and 7. The size of a wedge is (base_size / 16).@* -Bits 4-0 contain the base 2 logarithm of the base size (12 to 29).@* -Bits 7-5 contain the number of wedges (0 to 7) to substract from the -base size to obtain the dictionary size.@* -Example: 0xD3 = 2^19 - 6 * 2^15 = 512 KiB - 6 * 32 KiB = 320 KiB@* -Valid values for dictionary size range from 4 KiB to 512 MiB. - -@item Lzma stream -The lzma stream, finished by an end of stream marker. Uses default values -for encoder properties. See the lzip manual for a full description. - -@item CRC32 (4 bytes) -CRC of the uncompressed original data. - -@item Data size (8 bytes) -Size of the uncompressed original data. - -@item Member size (8 bytes) -Total size of the member, including header and trailer. This field acts -as a distributed index, allows the verification of stream integrity, and -facilitates safe recovery of undamaged members from multi-member files. - -@end table - - -@node Problems -@chapter Reporting bugs -@cindex bugs -@cindex getting help - -There are probably bugs in plzip. There are certainly errors and -omissions in this manual. If you report them, they will get fixed. If -you don't, no one will ever know about them and they will remain unfixed -for all eternity, if not longer. - -If you find a bug in plzip, please send electronic mail to -@email{lzip-bug@@nongnu.org}. Include the version number, which you can -find by running @w{@samp{plzip --version}}. - - -@node Concept index -@unnumbered Concept index - -@printindex cp - -@bye -- cgit v1.2.3