diff options
Diffstat (limited to 'doc/plzip.texi')
| -rw-r--r-- | doc/plzip.texi | 415 |
1 files changed, 415 insertions, 0 deletions
diff --git a/doc/plzip.texi b/doc/plzip.texi new file mode 100644 index 0000000..413a9e3 --- /dev/null +++ b/doc/plzip.texi @@ -0,0 +1,415 @@ +\input texinfo @c -*-texinfo-*- +@c %**start of header +@setfilename plzip.info +@documentencoding ISO-8859-15 +@settitle Plzip Manual +@finalout +@c %**end of header + +@set UPDATED 20 January 2014 +@set VERSION 1.2-pre1 + +@dircategory Data Compression +@direntry +* Plzip: (plzip). Parallel compressor compatible with lzip +@end direntry + + +@ifnothtml +@titlepage +@title Plzip +@subtitle Parallel compressor compatible with lzip +@subtitle for Plzip version @value{VERSION}, @value{UPDATED} +@author by Antonio Diaz Diaz + +@page +@vskip 0pt plus 1filll +@end titlepage + +@contents +@end ifnothtml + +@node Top +@top + +This manual is for Plzip (version @value{VERSION}, @value{UPDATED}). + +@menu +* Introduction:: Purpose and features of plzip +* Program design:: Internal structure of plzip +* Invoking plzip:: Command line interface +* File format:: Detailed format of the compressed file +* Problems:: Reporting bugs +* Concept index:: Index of concepts +@end menu + +@sp 1 +Copyright @copyright{} 2009, 2010, 2011, 2012, 2013, 2014 +Antonio Diaz Diaz. + +This manual is free documentation: you have unlimited permission +to copy, distribute and modify it. + + +@node Introduction +@chapter Introduction +@cindex introduction + +Plzip is a massively parallel (multi-threaded), lossless data compressor +based on the lzlib compression library, with a user interface similar to +the one of lzip, bzip2 or gzip. + +Plzip can compress/decompress large files on multiprocessor machines +much faster than lzip, at the cost of a slightly reduced compression +ratio. Note that the number of usable threads is limited by file size, +so on files larger than a few GB plzip can use hundreds of processors, +but on files of only a few MB plzip is no faster than lzip. + +Plzip uses the lzip file format; the files produced by plzip are fully +compatible with lzip-1.4 or newer, and can be rescued with lziprecover. + +The lzip file format is designed for long-term data archiving and +provides very safe integrity checking. The member trailer stores the +32-bit CRC of the original data, the size of the original data and the +size of the member. These values, together with the value remaining in +the range decoder and the end-of-stream marker, provide a 4 factor +integrity checking which guarantees that the decompressed version of the +data is identical to the original. This guards against corruption of the +compressed data, and against undetected bugs in plzip (hopefully very +unlikely). The chances of data corruption going undetected are +microscopic. Be aware, though, that the check occurs upon decompression, +so it can only tell you that something is wrong. It can't help you +recover the original uncompressed data. + +If you ever need to recover data from a damaged lzip file, try the +lziprecover program. Lziprecover makes lzip files resistant to bit-flip +(one of the most common forms of data corruption), and provides data +recovery capabilities, including error-checked merging of damaged copies +of a file. + +Plzip uses the same well-defined exit status values used by lzip and +bzip2, which makes it safer than compressors returning ambiguous warning +values (like gzip) when it is used as a back end for tar or zutils. + +When compressing, plzip replaces every file given in the command line +with a compressed version of itself, with the name "original_name.lz". +When decompressing, plzip attempts to guess the name for the decompressed +file from that of the compressed file as follows: + +@multitable {anyothername} {becomes} {anyothername.out} +@item filename.lz @tab becomes @tab filename +@item filename.tlz @tab becomes @tab filename.tar +@item anyothername @tab becomes @tab anyothername.out +@end multitable + +(De)compressing a file is much like copying or moving it; therefore plzip +preserves the access and modification dates, permissions, and, when +possible, ownership of the file just as "cp -p" does. (If the user ID or +the group ID can't be duplicated, the file permission bits S_ISUID and +S_ISGID are cleared). + +Plzip is able to read from some types of non regular files if the +@samp{--stdout} option is specified. + +If no file names are specified, plzip compresses (or decompresses) from +standard input to standard output. In this case, plzip will decline to +write compressed output to a terminal, as this would be entirely +incomprehensible and therefore pointless. + +Plzip will correctly decompress a file which is the concatenation of two +or more compressed files. The result is the concatenation of the +corresponding uncompressed files. Integrity testing of concatenated +compressed files is also supported. + +WARNING! Even if plzip is bug-free, other causes may result in a corrupt +compressed file (bugs in the system libraries, memory errors, etc). +Therefore, if the data you are going to compress is important, give the +@samp{--keep} option to plzip and do not remove the original file until +you verify the compressed file with a command like +@w{@samp{plzip -cd file.lz | cmp file -}}. + + +@node Program design +@chapter Program design +@cindex program design + +For each input file, a splitter thread and several worker threads are +created, acting the main thread as muxer (multiplexer) thread. A "packet +courier" takes care of data transfers among threads and limits the +maximum number of data blocks (packets) being processed simultaneously. + +The splitter reads data blocks from the input file, and distributes them +to the workers. The workers (de)compress the blocks received from the +splitter. The muxer collects processed packets from the workers, and +writes them to the output file. + +When decompressing from a regular file, the splitter is removed and the +workers read directly from the input file. If the output file is also a +regular file, the muxer is also removed, and the workers write directly +to the output file. With these optimizations, decompression speed of +large files with many members is only limited by the number of +processors available and by I/O speed. + + +@node Invoking plzip +@chapter Invoking plzip +@cindex invoking +@cindex options +@cindex usage +@cindex version + +The format for running plzip is: + +@example +plzip [@var{options}] [@var{files}] +@end example + +Plzip supports the following options: + +@table @samp +@item -h +@itemx --help +Print an informative help message describing the options and exit. + +@item -V +@itemx --version +Print the version number of plzip on the standard output and exit. + +@item -B @var{bytes} +@itemx --data-size=@var{bytes} +@anchor{--data-size} +Set the input data block size in bytes. The input file will be divided +in chunks of this size before compression is performed. Valid values +range from 8 KiB to 1 GiB. Default value is two times the dictionary +size. Plzip will reduce the dictionary size if it is larger than the +chosen data size. + +@item -c +@itemx --stdout +Compress or decompress to standard output. Needed when reading from a +named pipe (fifo) or from a device. + +@item -d +@itemx --decompress +Decompress. + +@item -f +@itemx --force +Force overwrite of output files. + +@item -F +@itemx --recompress +Force recompression of files whose name already has the @samp{.lz} or +@samp{.tlz} suffix. + +@item -k +@itemx --keep +Keep (don't delete) input files during compression or decompression. + +@item -m @var{bytes} +@itemx --match-length=@var{bytes} +Set the match length limit in bytes. After a match this long is found, +the search is finished. Valid values range from 5 to 273. Larger values +usually give better compression ratios but longer compression times. + +@item -n @var{n} +@itemx --threads=@var{n} +Set the number of worker threads. Valid values range from 1 to "as many +as your system can support". If this option is not used, plzip tries to +detect the number of processors in the system and use it as default +value. @w{@samp{plzip --help}} shows the system's default value. + +Note that the number of usable threads is limited to @w{ceil( file_size +/ data_size )} during compression (@pxref{--data-size}), and to the +number of members in the input during decompression. + +@item -o @var{file} +@itemx --output=@var{file} +When reading from standard input and @samp{--stdout} has not been +specified, use @samp{@var{file}} as the virtual name of the uncompressed +file. This produces a file named @samp{@var{file}} when decompressing, +and a file named @samp{@var{file}.lz} when compressing. + +@item -q +@itemx --quiet +Quiet operation. Suppress all messages. + +@item -s @var{bytes} +@itemx --dictionary-size=@var{bytes} +Set the dictionary size limit in bytes. Valid values range from 4 KiB to +512 MiB. Plzip will use the smallest possible dictionary size for each +member without exceeding this limit. Note that dictionary sizes are +quantized. If the specified size does not match one of the valid sizes, +it will be rounded upwards by adding up to (@var{bytes} / 16) to it. + +For maximum compression you should use a dictionary size limit as large +as possible, but keep in mind that the decompression memory requirement +is affected at compression time by the choice of dictionary size limit. + +@item -t +@itemx --test +Check integrity of the specified file(s), but don't decompress them. +This really performs a trial decompression and throws away the result. +Use it together with @samp{-v} to see information about the file. + +@item -v +@itemx --verbose +Verbose mode.@* +When compressing, show the compression ratio for each file processed. A +second @samp{-v} shows the progress of compression.@* +When decompressing or testing, further -v's (up to 4) increase the +verbosity level, showing status, compression ratio, decompressed size, +and compressed size. + +@item -1 .. -9 +Set the compression parameters (dictionary size and match length limit) +as shown in the table below. Note that @samp{-9} can be much slower than +@samp{-1}. These options have no effect when decompressing. + +The bidimensional parameter space of LZMA can't be mapped to a linear +scale optimal for all files. If your files are large, very repetitive, +etc, you may need to use the @samp{--match-length} and +@samp{--dictionary-size} options directly to achieve optimal +performance. For example, @samp{-9m64} usually compresses executables +more (and faster) than @samp{-9}. + +@multitable {Level} {Dictionary size} {Match length limit} +@item Level @tab Dictionary size @tab Match length limit +@item -1 @tab 1 MiB @tab 5 bytes +@item -2 @tab 1.5 MiB @tab 6 bytes +@item -3 @tab 2 MiB @tab 8 bytes +@item -4 @tab 3 MiB @tab 12 bytes +@item -5 @tab 4 MiB @tab 20 bytes +@item -6 @tab 8 MiB @tab 36 bytes +@item -7 @tab 16 MiB @tab 68 bytes +@item -8 @tab 24 MiB @tab 132 bytes +@item -9 @tab 32 MiB @tab 273 bytes +@end multitable + +@item --fast +@itemx --best +Aliases for GNU gzip compatibility. + +@end table + +Numbers given as arguments to options may be followed by a multiplier +and an optional @samp{B} for "byte". + +Table of SI and binary prefixes (unit multipliers): + +@multitable {Prefix} {kilobyte (10^3 = 1000)} {|} {Prefix} {kibibyte (2^10 = 1024)} +@item Prefix @tab Value @tab | @tab Prefix @tab Value +@item k @tab kilobyte (10^3 = 1000) @tab | @tab Ki @tab kibibyte (2^10 = 1024) +@item M @tab megabyte (10^6) @tab | @tab Mi @tab mebibyte (2^20) +@item G @tab gigabyte (10^9) @tab | @tab Gi @tab gibibyte (2^30) +@item T @tab terabyte (10^12) @tab | @tab Ti @tab tebibyte (2^40) +@item P @tab petabyte (10^15) @tab | @tab Pi @tab pebibyte (2^50) +@item E @tab exabyte (10^18) @tab | @tab Ei @tab exbibyte (2^60) +@item Z @tab zettabyte (10^21) @tab | @tab Zi @tab zebibyte (2^70) +@item Y @tab yottabyte (10^24) @tab | @tab Yi @tab yobibyte (2^80) +@end multitable + +@sp 1 +Exit status: 0 for a normal exit, 1 for environmental problems (file not +found, invalid flags, I/O errors, etc), 2 to indicate a corrupt or +invalid input file, 3 for an internal consistency error (eg, bug) which +caused plzip to panic. + + +@node File format +@chapter File format +@cindex file format + +Perfection is reached, not when there is no longer anything to add, but +when there is no longer anything to take away.@* +--- Antoine de Saint-Exupery + +@sp 1 +In the diagram below, a box like this: +@verbatim ++---+ +| | <-- the vertical bars might be missing ++---+ +@end verbatim + +represents one byte; a box like this: +@verbatim ++==============+ +| | ++==============+ +@end verbatim + +represents a variable number of bytes. + +@sp 1 +A lzip file consists of a series of "members" (compressed data sets). +The members simply appear one after another in the file, with no +additional information before, between, or after them. + +Each member has the following structure: +@verbatim ++--+--+--+--+----+----+=============+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +| ID string | VN | DS | Lzma stream | CRC32 | Data size | Member size | ++--+--+--+--+----+----+=============+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +@end verbatim + +All multibyte values are stored in little endian order. + +@table @samp +@item ID string +A four byte string, identifying the lzip format, with the value "LZIP" +(0x4C, 0x5A, 0x49, 0x50). + +@item VN (version number, 1 byte) +Just in case something needs to be modified in the future. 1 for now. + +@item DS (coded dictionary size, 1 byte) +Lzip divides the distance between any two powers of 2 into 8 equally +spaced intervals, named "wedges". The dictionary size is calculated by +taking a power of 2 (the base size) and substracting from it a number of +wedges between 0 and 7. The size of a wedge is (base_size / 16).@* +Bits 4-0 contain the base 2 logarithm of the base size (12 to 29).@* +Bits 7-5 contain the number of wedges (0 to 7) to substract from the +base size to obtain the dictionary size.@* +Example: 0xD3 = 2^19 - 6 * 2^15 = 512 KiB - 6 * 32 KiB = 320 KiB@* +Valid values for dictionary size range from 4 KiB to 512 MiB. + +@item Lzma stream +The lzma stream, finished by an end of stream marker. Uses default values +for encoder properties. See the lzip manual for a full description. + +@item CRC32 (4 bytes) +CRC of the uncompressed original data. + +@item Data size (8 bytes) +Size of the uncompressed original data. + +@item Member size (8 bytes) +Total size of the member, including header and trailer. This field acts +as a distributed index, allows the verification of stream integrity, and +facilitates safe recovery of undamaged members from multi-member files. + +@end table + + +@node Problems +@chapter Reporting bugs +@cindex bugs +@cindex getting help + +There are probably bugs in plzip. There are certainly errors and +omissions in this manual. If you report them, they will get fixed. If +you don't, no one will ever know about them and they will remain unfixed +for all eternity, if not longer. + +If you find a bug in plzip, please send electronic mail to +@email{lzip-bug@@nongnu.org}. Include the version number, which you can +find by running @w{@samp{plzip --version}}. + + +@node Concept index +@unnumbered Concept index + +@printindex cp + +@bye |