diff options
Diffstat (limited to '')
| -rw-r--r-- | doc/plzip.1 | 12 | ||||
| -rw-r--r-- | doc/plzip.info | 80 | ||||
| -rw-r--r-- | doc/plzip.texi | 72 |
3 files changed, 116 insertions, 48 deletions
diff --git a/doc/plzip.1 b/doc/plzip.1 index bfe6d56..cab69f4 100644 --- a/doc/plzip.1 +++ b/doc/plzip.1 @@ -1,10 +1,10 @@ -.\" DO NOT MODIFY THIS FILE! It was generated by help2man 1.37.1. -.TH PLZIP "1" "July 2014" "plzip 1.2-rc2" "User Commands" +.\" DO NOT MODIFY THIS FILE! It was generated by help2man 1.46.1. +.TH PLZIP "1" "August 2014" "plzip 1.2" "User Commands" .SH NAME plzip \- reduces the size of files .SH SYNOPSIS .B plzip -[\fIoptions\fR] [\fIfiles\fR] +[\fI\,options\/\fR] [\fI\,files\/\fR] .SH DESCRIPTION Plzip \- Parallel compressor compatible with lzip. .SH OPTIONS @@ -16,7 +16,7 @@ display this help and exit output version information and exit .TP \fB\-B\fR, \fB\-\-data\-size=\fR<bytes> -set input data block size in bytes +set size of input data blocks, in bytes .TP \fB\-c\fR, \fB\-\-stdout\fR send output to standard output @@ -37,7 +37,7 @@ keep (don't delete) input files set match length limit in bytes [36] .TP \fB\-n\fR, \fB\-\-threads=\fR<n> -set number of (de)compression threads [1] +set number of (de)compression threads [2] .TP \fB\-o\fR, \fB\-\-output=\fR<file> if reading stdin, place the output into <file> @@ -85,7 +85,7 @@ Plzip home page: http://www.nongnu.org/lzip/plzip.html Copyright \(co 2009 Laszlo Ersek. .br Copyright \(co 2014 Antonio Diaz Diaz. -Using Lzlib 1.6\-rc2 +Using Lzlib 1.6 License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl.html> .br This is free software: you are free to change and redistribute it. diff --git a/doc/plzip.info b/doc/plzip.info index f8ca35c..f8f7e08 100644 --- a/doc/plzip.info +++ b/doc/plzip.info @@ -11,7 +11,7 @@ File: plzip.info, Node: Top, Next: Introduction, Up: (dir) Plzip Manual ************ -This manual is for Plzip (version 1.2-rc2, 1 July 2014). +This manual is for Plzip (version 1.2, 29 August 2014). * Menu: @@ -23,7 +23,7 @@ This manual is for Plzip (version 1.2-rc2, 1 July 2014). * Concept index:: Index of concepts - Copyright (C) 2009, 2010, 2011, 2012, 2013, 2014 Antonio Diaz Diaz. + Copyright (C) 2009-2014 Antonio Diaz Diaz. This manual is free documentation: you have unlimited permission to copy, distribute and modify it. @@ -34,15 +34,15 @@ File: plzip.info, Node: Introduction, Next: Program design, Prev: Top, Up: T 1 Introduction ************** -Plzip is a massively parallel (multi-threaded), lossless data compressor +Plzip is a massively parallel (multi-threaded) lossless data compressor based on the lzlib compression library, with a user interface similar to the one of lzip, bzip2 or gzip. Plzip can compress/decompress large files on multiprocessor machines much faster than lzip, at the cost of a slightly reduced compression -ratio. Note that the number of usable threads is limited by file size, -so on files larger than a few GB plzip can use hundreds of processors, -but on files of only a few MB plzip is no faster than lzip. +ratio. Note that the number of usable threads is limited by file size; +on files larger than a few GB plzip can use hundreds of processors, but +on files of only a few MB plzip is no faster than lzip. Plzip uses the lzip file format; the files produced by plzip are fully compatible with lzip-1.4 or newer, and can be rescued with @@ -67,6 +67,11 @@ into account both data integrity and decoder availability: * Additionally lzip is copylefted, which guarantees that it will remain free forever. + A nice feature of the lzip format is that a corrupt byte is easier to +repair the nearer it is from the beginning of the file. Therefore, with +the help of lziprecover, losing an entire archive just because of a +corrupt byte near the beginning is a thing of the past. + The member trailer stores the 32-bit CRC of the original data, the size of the original data and the size of the member. These values, together with the value remaining in the range decoder and the @@ -81,7 +86,23 @@ uncompressed data. Plzip uses the same well-defined exit status values used by lzip and bzip2, which makes it safer than compressors returning ambiguous warning -values (like gzip) when it is used as a back end for tar or zutils. +values (like gzip) when it is used as a back end for other programs like +tar or zutils. + + The amount of memory required *per thread* is approximately the +following: + + * For compression; 3 times the data size (*note --data-size::) plus + 11 times the dictionary size. + + * For decompression or testing of a non-seekable file or of standard + input; 2 times the dictionary size plus up to 32 MiB. + + * For decompression of a regular file to a non-seekable file or to + standard output; the dictionary size plus up to 32 MiB. + + * For decompression of a regular file to another regular file, or for + testing of a regular file; the dictionary size. Plzip will automatically use the smallest possible dictionary size for each file without exceeding the given limit. Keep in mind that the @@ -129,7 +150,17 @@ File: plzip.info, Node: Program design, Next: Invoking plzip, Prev: Introduct 2 Program design **************** -For each input file, a splitter thread and several worker threads are +When compressing, plzip divides the input file into chunks and +compresses as many chunks simultaneously as worker threads are chosen, +creating a multi-member compressed file. + + When decompressing, plzip decompresses as many members +simultaneously as worker threads are chosen. Files that were compressed +with lzip will not be decompressed faster than using lzip (unless the +'-b' option was used) because lzip usually produces single-member +files, which can't be decompressed in parallel. + + For each input file, a splitter thread and several worker threads are created, acting the main thread as muxer (multiplexer) thread. A "packet courier" takes care of data transfers among threads and limits the maximum number of data blocks (packets) being processed simultaneously. @@ -141,10 +172,11 @@ writes them to the output file. When decompressing from a regular file, the splitter is removed and the workers read directly from the input file. If the output file is -also a regular file, the muxer is also removed, and the workers write -directly to the output file. With these optimizations, decompression -speed of large files with many members is only limited by the number of -processors available and by I/O speed. +also a regular file, the muxer is also removed and the workers write +directly to the output file. With these optimizations, the use of RAM +is greatly reduced and the decompression speed of large files with many +members is only limited by the number of processors available and by +I/O speed. File: plzip.info, Node: Invoking plzip, Next: File format, Prev: Program design, Up: Top @@ -168,11 +200,11 @@ The format for running plzip is: '-B BYTES' '--data-size=BYTES' - Set the input data block size in bytes. The input file will be - divided in chunks of this size before compression is performed. - Valid values range from 8 KiB to 1 GiB. Default value is two times - the dictionary size. Plzip will reduce the dictionary size if it - is larger than the chosen data size. + Set the size of the input data blocks, in bytes. The input file + will be divided in chunks of this size before compression is + performed. Valid values range from 8 KiB to 1 GiB. Default value + is two times the dictionary size. Plzip will reduce the dictionary + size if it is larger than the chosen data size. '-c' '--stdout' @@ -418,13 +450,13 @@ Concept index Tag Table: Node: Top221 -Node: Introduction873 -Node: Program design5442 -Node: Invoking plzip6496 -Ref: --data-size6941 -Node: File format12090 -Node: Problems14595 -Node: Concept index15124 +Node: Introduction847 +Node: Program design6279 +Node: Invoking plzip7868 +Ref: --data-size8313 +Node: File format13471 +Node: Problems15976 +Node: Concept index16505 End Tag Table diff --git a/doc/plzip.texi b/doc/plzip.texi index 7678977..bc7af6d 100644 --- a/doc/plzip.texi +++ b/doc/plzip.texi @@ -6,8 +6,8 @@ @finalout @c %**end of header -@set UPDATED 1 July 2014 -@set VERSION 1.2-rc2 +@set UPDATED 29 August 2014 +@set VERSION 1.2 @dircategory Data Compression @direntry @@ -44,8 +44,7 @@ This manual is for Plzip (version @value{VERSION}, @value{UPDATED}). @end menu @sp 1 -Copyright @copyright{} 2009, 2010, 2011, 2012, 2013, 2014 -Antonio Diaz Diaz. +Copyright @copyright{} 2009-2014 Antonio Diaz Diaz. This manual is free documentation: you have unlimited permission to copy, distribute and modify it. @@ -55,15 +54,15 @@ to copy, distribute and modify it. @chapter Introduction @cindex introduction -Plzip is a massively parallel (multi-threaded), lossless data compressor +Plzip is a massively parallel (multi-threaded) lossless data compressor based on the lzlib compression library, with a user interface similar to the one of lzip, bzip2 or gzip. Plzip can compress/decompress large files on multiprocessor machines much faster than lzip, at the cost of a slightly reduced compression -ratio. Note that the number of usable threads is limited by file size, -so on files larger than a few GB plzip can use hundreds of processors, -but on files of only a few MB plzip is no faster than lzip. +ratio. Note that the number of usable threads is limited by file size; +on files larger than a few GB plzip can use hundreds of processors, but +on files of only a few MB plzip is no faster than lzip. Plzip uses the lzip file format; the files produced by plzip are fully compatible with lzip-1.4 or newer, and can be rescued with lziprecover. @@ -92,6 +91,11 @@ Additionally lzip is copylefted, which guarantees that it will remain free forever. @end itemize +A nice feature of the lzip format is that a corrupt byte is easier to +repair the nearer it is from the beginning of the file. Therefore, with +the help of lziprecover, losing an entire archive just because of a +corrupt byte near the beginning is a thing of the past. + The member trailer stores the 32-bit CRC of the original data, the size of the original data and the size of the member. These values, together with the value remaining in the range decoder and the end-of-stream @@ -105,7 +109,29 @@ wrong. It can't help you recover the original uncompressed data. Plzip uses the same well-defined exit status values used by lzip and bzip2, which makes it safer than compressors returning ambiguous warning -values (like gzip) when it is used as a back end for tar or zutils. +values (like gzip) when it is used as a back end for other programs like +tar or zutils. + +The amount of memory required @strong{per thread} is approximately the +following: + +@itemize @bullet +@item +For compression; 3 times the data size (@pxref{--data-size}) plus 11 +times the dictionary size. + +@item +For decompression or testing of a non-seekable file or of standard +input; 2 times the dictionary size plus up to 32 MiB. + +@item +For decompression of a regular file to a non-seekable file or to +standard output; the dictionary size plus up to 32 MiB. + +@item +For decompression of a regular file to another regular file, or for +testing of a regular file; the dictionary size. +@end itemize Plzip will automatically use the smallest possible dictionary size for each file without exceeding the given limit. Keep in mind that the @@ -154,6 +180,16 @@ you verify the compressed file with a command like @chapter Program design @cindex program design +When compressing, plzip divides the input file into chunks and +compresses as many chunks simultaneously as worker threads are chosen, +creating a multi-member compressed file. + +When decompressing, plzip decompresses as many members simultaneously as +worker threads are chosen. Files that were compressed with lzip will not +be decompressed faster than using lzip (unless the @samp{-b} option was +used) because lzip usually produces single-member files, which can't be +decompressed in parallel. + For each input file, a splitter thread and several worker threads are created, acting the main thread as muxer (multiplexer) thread. A "packet courier" takes care of data transfers among threads and limits the @@ -166,10 +202,10 @@ writes them to the output file. When decompressing from a regular file, the splitter is removed and the workers read directly from the input file. If the output file is also a -regular file, the muxer is also removed, and the workers write directly -to the output file. With these optimizations, decompression speed of -large files with many members is only limited by the number of -processors available and by I/O speed. +regular file, the muxer is also removed and the workers write directly +to the output file. With these optimizations, the use of RAM is greatly +reduced and the decompression speed of large files with many members is +only limited by the number of processors available and by I/O speed. @node Invoking plzip @@ -199,11 +235,11 @@ Print the version number of plzip on the standard output and exit. @item -B @var{bytes} @itemx --data-size=@var{bytes} @anchor{--data-size} -Set the input data block size in bytes. The input file will be divided -in chunks of this size before compression is performed. Valid values -range from 8 KiB to 1 GiB. Default value is two times the dictionary -size. Plzip will reduce the dictionary size if it is larger than the -chosen data size. +Set the size of the input data blocks, in bytes. The input file will be +divided in chunks of this size before compression is performed. Valid +values range from 8 KiB to 1 GiB. Default value is two times the +dictionary size. Plzip will reduce the dictionary size if it is larger +than the chosen data size. @item -c @itemx --stdout |