Merging upstream version 1.25~rc1.

Signed-off-by: Daniel Baumann <daniel.baumann@progress-linux.org>

1 files changed, 390 insertions, 368 deletions

diff --git a/doc/lzip.info b/doc/lzip.info
index 04f81c4..95587d8 100644
--- a/doc/lzip.info
+++ b/doc/lzip.info
@@ -11,17 +11,18 @@ File: lzip.info,  Node: Top,  Next: Introduction,  Up: (dir)
 Lzip Manual
 ***********
 
-This manual is for Lzip (version 1.24.1, 1 March 2024).
+This manual is for Lzip (version 1.25-rc1, 26 November 2024).
 
 * Menu:
 
 * Introduction::           Purpose and features of lzip
 * Output::                 Meaning of lzip's output
 * Invoking lzip::          Command-line interface
-* Quality assurance::      Design, development, and testing of lzip
-* Algorithm::              How lzip compresses the data
+* Argument syntax::        By convention, options start with a hyphen
 * File format::            Detailed format of the compressed file
 * Stream format::          Format of the LZMA stream in lzip files
+* Quality assurance::      Design, development, and testing of lzip
+* Algorithm::              How lzip compresses the data
 * Trailing data::          Extra data appended to the file
 * Examples::               A small tutorial with examples
 * Problems::               Reporting bugs
@@ -41,16 +42,16 @@ File: lzip.info,  Node: Introduction,  Next: Output,  Prev: Top,  Up: Top
 **************
 
 Lzip is a lossless data compressor with a user interface similar to the one
-of gzip or bzip2. Lzip uses a simplified form of the 'Lempel-Ziv-Markov
-chain-Algorithm' (LZMA) stream format to maximize interoperability. The
-maximum dictionary size is 512 MiB so that any lzip file can be decompressed
-on 32-bit machines. Lzip provides accurate and robust 3-factor integrity
-checking. Lzip can compress about as fast as gzip (lzip -0) or compress most
-files more than bzip2 (lzip -9). Decompression speed is intermediate between
-gzip and bzip2. Lzip is better than gzip and bzip2 from a data recovery
-perspective. Lzip has been designed, written, and tested with great care to
-replace gzip and bzip2 as the standard general-purpose compressed format for
-Unix-like systems.
+of gzip or bzip2. Lzip uses a simplified form of LZMA (Lempel-Ziv-Markov
+chain-Algorithm) designed to achieve complete interoperability between
+implementations. The maximum dictionary size is 512 MiB so that any lzip
+file can be decompressed on 32-bit machines. Lzip provides accurate and
+robust 3-factor integrity checking. 'lzip -0' compresses about as fast as
+gzip, while 'lzip -9' compresses most files more than bzip2. Decompression
+speed is intermediate between gzip and bzip2. Lzip provides better data
+recovery capabilities than gzip and bzip2. Lzip has been designed, written,
+and tested with great care to replace gzip and bzip2 as general-purpose
+compressed format for Unix-like systems.
 
    For compressing/decompressing large files on multiprocessor machines
 plzip can be much faster than lzip at the cost of a slightly reduced
@@ -64,12 +65,10 @@ alignment between tar members and lzip members. *Note tarlz manual:
    The lzip file format is designed for data sharing and long-term
 archiving, taking into account both data integrity and decoder availability:
 
-   * The lzip format provides very safe integrity checking and some data
-     recovery means. The program lziprecover can repair bit flip errors
-     (one of the most common forms of data corruption) in lzip files, and
-     provides data recovery capabilities, including error-checked merging
-     of damaged copies of a file. *Note Data safety: (lziprecover)Data
-     safety.
+   * The program lziprecover can repair bit flip errors (one of the most
+     common forms of data corruption) in lzip files, and provides data
+     recovery capabilities, including error-checked merging of damaged
+     copies of a file. *Note Data safety: (lziprecover)Data safety.
 
    * The lzip format is as simple as possible (but not simpler). The lzip
      manual provides the source code of a simple decompressor along with a
@@ -88,7 +87,7 @@ byte near the beginning is a thing of the past.
 
    The member trailer stores the 32-bit CRC of the original data, the size
 of the original data, and the size of the member. These values, together
-with the "End Of Stream" marker, provide a 3-factor integrity checking which
+with the 'End Of Stream' marker, provide a 3-factor integrity checking which
 guarantees that the decompressed version of the data is identical to the
 original. This guards against corruption of the compressed data, and against
 undetected bugs in lzip (hopefully very unlikely). The chances of data
@@ -101,9 +100,9 @@ makes it safer than compressors returning ambiguous warning values (like
 gzip) when it is used as a back end for other programs like tar or zutils.
 
    Lzip automatically uses for each file the largest dictionary size that
-does not exceed neither the file size nor the limit given. Keep in mind
-that the decompression memory requirement is affected at compression time
-by the choice of dictionary size limit.
+does not exceed neither the file size nor the limit given. The dictionary
+size used for decompression is the same dictionary size used for
+compression.
 
    The amount of memory required for compression is about 1 or 2 times the
 dictionary size limit (1 if input file size is less than dictionary size
@@ -142,8 +141,7 @@ also supported.
    Lzip can produce multimember files, and lziprecover can safely recover
 the undamaged members in case of file damage. Lzip can also split the
 compressed output in volumes of a given size, even when reading from
-standard input. This allows the direct creation of multivolume compressed
-tar archives.
+standard input.
 
    Lzip is able to compress and decompress streams of unlimited size by
 automatically creating multimember output. The members so created are large,
@@ -197,7 +195,7 @@ have been compressed. Decompressed is used to refer to data which have
 undergone the process of decompression.
 
 
-File: lzip.info,  Node: Invoking lzip,  Next: Quality assurance,  Prev: Output,  Up: Top
+File: lzip.info,  Node: Invoking lzip,  Next: Argument syntax,  Prev: Output,  Up: Top
 
 3 Invoking lzip
 ***************
@@ -212,8 +210,7 @@ means standard input. It can be mixed with other FILES and is read just
 once, the first time it appears in the command line. Remember to prepend
 './' to any file name beginning with a hyphen, or use '--'.
 
-   lzip supports the following options: *Note Argument syntax:
-(arg_parser)Argument syntax.
+lzip supports the following options: *Note Argument syntax::.
 
 '-h'
 '--help'
@@ -232,11 +229,12 @@ once, the first time it appears in the command line. Remember to prepend
 
 '-b BYTES'
 '--member-size=BYTES'
-     When compressing, set the member size limit to BYTES. It is advisable
-     to keep members smaller than RAM size so that they can be repaired with
-     lziprecover in case of corruption. A small member size may degrade
-     compression ratio, so use it only when needed. Valid values range from
-     100 kB to 2 PiB. Defaults to 2 PiB.
+     When compressing, set the member size limit to BYTES. If BYTES is
+     smaller than the compressed size, a multimember file is produced. It is
+     advisable to keep members smaller than RAM size so that they can be
+     repaired with lziprecover in case of corruption. A small member size
+     may degrade compression ratio, so use it only when needed. Valid
+     values range from 100 kB to 2 PiB. Defaults to 2 PiB.
 
 '-c'
 '--stdout'
@@ -257,7 +255,8 @@ once, the first time it appears in the command line. Remember to prepend
      status 1. If a file fails to decompress, or is a terminal, lzip exits
      immediately with error status 2 without decompressing the rest of the
      files. A terminal is considered an uncompressed file, and therefore
-     invalid.
+     invalid. A multimember file with one or more empty members is accepted
+     if redirected to standard input.
 
 '-f'
 '--force'
@@ -281,7 +280,8 @@ once, the first time it appears in the command line. Remember to prepend
      '-v', the dictionary size, the number of members in the file, and the
      amount of trailing data (if any) are also printed. With '-vv', the
      positions and sizes of each member in multimember files are also
-     printed.
+     printed. A multimember file with one or more empty members is accepted
+     if redirected to standard input.
 
      If any file is damaged, does not exist, can't be opened, or is not
      regular, the final exit status is > 0. '-lq' can be used to check
@@ -356,7 +356,8 @@ once, the first time it appears in the command line. Remember to prepend
      fails the test, does not exist, can't be opened, or is a terminal, lzip
      continues testing the rest of the files. A final diagnostic is shown at
      verbosity level 1 or higher if any file fails the test when testing
-     multiple files.
+     multiple files. A multimember file with one or more empty members is
+     accepted if redirected to standard input.
 
 '-v'
 '--verbose'
@@ -366,8 +367,8 @@ once, the first time it appears in the command line. Remember to prepend
      When decompressing or testing, further -v's (up to 4) increase the
      verbosity level, showing status, compression ratio, dictionary size,
      trailer contents (CRC, data size, member size), and up to 6 bytes of
-     trailing data (if any) both in hexadecimal and as a string of printable
-     ASCII characters.
+     trailing data (if any) both in hexadecimal and as a string of
+     printable ASCII characters.
      Two or more '-v' options show the progress of (de)compression.
 
 '-0 .. -9'
@@ -387,6 +388,7 @@ once, the first time it appears in the command line. Remember to prepend
      '-s64MiB -m273'
 
      Level   Dictionary size (-s)   Match length limit (-m)
+     ------------------------------------------------------
      -0      64 KiB                 16 bytes
      -1      1 MiB                  5 bytes
      -2      1.5 MiB                6 bytes
@@ -402,21 +404,11 @@ once, the first time it appears in the command line. Remember to prepend
 '--best'
      Aliases for GNU gzip compatibility.
 
-'--empty-error'
-     Exit with error status 2 if any empty member is found in the input
-     files.
-
-'--marking-error'
-     Exit with error status 2 if the first LZMA byte is non-zero in any
-     member of the input files. This may be caused by data corruption or by
-     deliberate insertion of tracking information in the file. Use
-     'lziprecover --clear-marking' to clear any such non-zero bytes.
-
 '--loose-trailing'
      When decompressing, testing, or listing, allow trailing data whose
      first bytes are so similar to the magic bytes of a lzip header that
      they can be confused with a corrupt header. Use this option if a file
-     triggers a "corrupt header" error and the cause is not indeed a
+     triggers a 'corrupt header' error and the cause is not indeed a
      corrupt header.
 
 
@@ -427,6 +419,7 @@ and may be followed by a multiplier and an optional 'B' for "byte".
    Table of SI and binary prefixes (unit multipliers):
 
 Prefix   Value                      |   Prefix   Value
+----------------------------------------------------------------------
 k        kilobyte   (10^3 = 1000)   |   Ki       kibibyte  (2^10 = 1024)
 M        megabyte   (10^6)          |   Mi       mebibyte  (2^20)
 G        gigabyte   (10^9)          |   Gi       gibibyte  (2^30)
@@ -445,278 +438,58 @@ corrupt or invalid input file, 3 for an internal consistency error (e.g.,
 bug) which caused lzip to panic.
 
 
-File: lzip.info,  Node: Quality assurance,  Next: Algorithm,  Prev: Invoking lzip,  Up: Top
-
-4 Design, development, and testing of lzip
-******************************************
-
-There are two ways of constructing a software design: One way is to make it
-so simple that there are obviously no deficiencies and the other way is to
-make it so complicated that there are no obvious deficiencies. The first
-method is far more difficult.
--- C.A.R. Hoare
-
-   Lzip has been designed, written, and tested with great care to replace
-gzip and bzip2 as the standard general-purpose compressed format for
-Unix-like systems. This chapter describes the lessons learned from these
-previous formats, and their application to the design of lzip. The lzip
-format specification has been reviewed carefully and is believed to be free
-from design errors.
-
-
-4.1 Format design
-=================
-
-When gzip was designed in 1992, computers and operating systems were much
-less capable than they are today. The designers of gzip tried to work around
-some of those limitations, like 8.3 file names, with additional fields in
-the file format.
-
-   Today those limitations have mostly disappeared, and the format of gzip
-has proved to be unnecessarily complicated. It includes fields that were
-never used, others that have lost their usefulness, and finally others that
-have become too limited.
-
-   Bzip2 was designed 5 years later, and its format is simpler than the one
-of gzip.
-
-   Probably the worst defect of the gzip format from the point of view of
-data safety is the variable size of its header. If the byte at offset 3
-(flags) of a gzip member gets corrupted, it may become difficult to recover
-the data, even if the compressed blocks are intact, because it can't be
-known with certainty where the compressed blocks begin.
-
-   By contrast, the header of a lzip member has a fixed length of 6. The
-LZMA stream in a lzip member always starts at offset 6, making it trivial to
-recover the data even if the whole header becomes corrupt.
-
-   Bzip2 also provides a header of fixed length and marks the begin and end
-of each compressed block with six magic bytes, making it possible to find
-the compressed blocks even in case of file damage. But bzip2 does not store
-the size of each compressed block, as lzip does.
-
-   Lziprecover is able to provide unique data recovery capabilities because
-the lzip format is extraordinarily safe. The simple and safe design of the
-file format complements the embedded error detection provided by the LZMA
-data stream. Any distance larger than the dictionary size acts as a
-forbidden symbol, allowing the decompressor to detect the approximate
-position of errors, and leaving very little work for the check sequence
-(CRC and data sizes) in the detection of errors. Lzip is usually able to
-detect all possible bit flips in the compressed data without resorting to
-the check sequence. It would be difficult to write an automatic recovery
-tool like lziprecover for the gzip format. And, as far as I know, it has
-never been written.
-
-   Lzip, like gzip and bzip2, uses a CRC32 to check the integrity of the
-decompressed data because it provides optimal accuracy in the detection of
-errors up to a compressed size of about 16 GiB, a size larger than that of
-most files. In the case of lzip, the additional detection capability of the
-decompressor reduces the probability of undetected errors several million
-times more, resulting in a combined integrity checking optimally accurate
-for any member size produced by lzip. Preliminary results suggest that the
-lzip format is safe enough to be used in critical safety avionics systems.
-
-   The lzip format is designed for long-term archiving. Therefore it
-excludes any unneeded features that may interfere with the future
-extraction of the decompressed data.
-
-
-4.1.1 Gzip format (mis)features not present in lzip
----------------------------------------------------
-
-'Multiple algorithms'
-     Gzip provides a CM (Compression Method) field that has never been used
-     because it is a bad idea to begin with. New compression methods may
-     require additional fields, making it impossible to implement new
-     methods and, at the same time, keep the same format. This field does
-     not solve the problem of format proliferation; it just makes the
-     problem less obvious.
-
-'Optional fields in header'
-     Unless special precautions are taken, optional fields are generally a
-     bad idea because they produce a header of variable size. The gzip
-     header has 2 fields that, in addition to being optional, are
-     zero-terminated. This means that if any byte inside the field gets
-     zeroed, or if the terminating zero gets altered, gzip won't be able to
-     find neither the header CRC nor the compressed blocks.
-
-'Optional CRC for the header'
-     Using an optional CRC for the header is not only a bad idea, it is an
-     error; it circumvents the Hamming distance (HD) of the CRC and may
-     prevent the extraction of perfectly good data. For example, if the CRC
-     is used and the bit enabling it is reset by a bit flip, then the
-     header seems to be intact (in spite of being corrupt) while the
-     compressed blocks seem to be totally unrecoverable (in spite of being
-     intact). Very misleading indeed.
-
-'Metadata'
-     The gzip format stores some metadata, like the modification time of the
-     original file or the operating system on which compression took place.
-     This complicates reproducible compression (obtaining identical
-     compressed output from identical input).
-
-
-4.1.2 Lzip format improvements over gzip and bzip2
---------------------------------------------------
-
-'64-bit size field'
-     Probably the most frequently reported shortcoming of the gzip format
-     is that it only stores the least significant 32 bits of the
-     uncompressed size. The size of any file larger or equal than 4 GiB
-     gets truncated.
-
-     Bzip2 does not store the uncompressed size of the file.
-
-     The lzip format provides a 64-bit field for the uncompressed size.
-     Additionally, lzip produces multimember output automatically when the
-     size is too large for a single member, allowing for an unlimited
-     uncompressed size.
-
-'Distributed index'
-     The lzip format provides a distributed index that, among other things,
-     helps plzip to decompress several times faster than pigz and helps
-     lziprecover do its job. Neither the gzip format nor the bzip2 format
-     do provide an index.
-
-     A distributed index is safer and more scalable than a monolithic
-     index. The monolithic index introduces a single point of failure in
-     the compressed file and may limit the number of members or the total
-     uncompressed size.
-
-
-4.2 Quality of implementation
-=============================
-
-Our civilization depends critically on software; it had better be quality
-software.
--- Bjarne Stroustrup
-
-'Accurate and robust error detection'
-     The lzip format provides 3-factor integrity checking, and the
-     decompressors report mismatches in each factor separately. This method
-     detects most false positives for corruption. If just one byte in one
-     factor fails but the other two factors match the data, it probably
-     means that the data are intact and the corruption just affects the
-     mismatching factor (CRC, data size, or member size) in the member
-     trailer.
-
-'Multiple implementations'
-     Just like the lzip format provides 3-factor protection against
-     undetected data corruption, the development methodology of the lzip
-     family of compressors provides 3-factor protection against undetected
-     programming errors.
-
-     Three related but independent compressor implementations, lzip, clzip,
-     and minilzip/lzlib, are developed concurrently. Every stable release
-     of any of them is tested to check that it produces identical output to
-     the other two. This guarantees that all three implement the same
-     algorithm, and makes it unlikely that any of them may contain serious
-     undiscovered errors. In fact, no errors have been discovered in lzip
-     since 2009.
-
-     Additionally, the three implementations have been extensively tested
-     with unzcrash, valgrind, and 'american fuzzy lop' without finding a
-     single vulnerability or false negative. *Note Unzcrash:
-     (lziprecover)Unzcrash.
-
-'Dictionary size'
-     Lzip automatically adapts the dictionary size to the size of each file.
-     In addition to reducing the amount of memory required for
-     decompression, this feature also minimizes the probability of being
-     affected by RAM errors during compression.
-
-'Exit status'
-     Returning a warning status of 2 is a design flaw of compress that
-     leaked into the design of gzip. Both bzip2 and lzip are free from this
-     flaw.
-
-
-
-File: lzip.info,  Node: Algorithm,  Next: File format,  Prev: Quality assurance,  Up: Top
+File: lzip.info,  Node: Argument syntax,  Next: File format,  Prev: Invoking lzip,  Up: Top
 
-5 Algorithm
-***********
+4 Syntax of command-line arguments
+**********************************
 
-In spite of its name (Lempel-Ziv-Markov chain-Algorithm), LZMA is not a
-concrete algorithm; it is more like "any algorithm using the LZMA coding
-scheme". LZMA compression consists in describing the uncompressed data as a
-succession of coding sequences from the set shown in Section 'What is
-coded' (*note what-is-coded::), and then encoding them using a range
-encoder. For example, the option '-0' of lzip uses the scheme in almost the
-simplest way possible; issuing the longest match it can find, or a literal
-byte if it can't find a match. Inversely, a much more elaborated way of
-finding coding sequences of minimum size than the one currently used by
-lzip could be developed, and the resulting sequence could also be coded
-using the LZMA coding scheme.
+POSIX recommends these conventions for command-line arguments.
 
-   Lzip currently implements two variants of the LZMA algorithm: fast (used
-by option '-0') and normal (used by all other compression levels).
-
-   The high compression of LZMA comes from combining two basic, well-proven
-compression ideas: sliding dictionaries (LZ77) and Markov models (the thing
-used by every compression algorithm that uses a range encoder or similar
-order-0 entropy coder as its last stage) with segregation of contexts
-according to what the bits are used for.
-
-   Lzip is a two stage compressor. The first stage is a Lempel-Ziv coder,
-which reduces redundancy by translating chunks of data to their
-corresponding distance-length pairs. The second stage is a range encoder
-that uses a different probability model for each type of data: distances,
-lengths, literal bytes, etc.
-
-   Here is how it works, step by step:
+   * A command-line argument is an option if it begins with a hyphen ('-').
 
-   1) The member header is written to the output stream.
-
-   2) The first byte is coded literally, because there are no previous
-bytes to which the match finder can refer to.
+   * Option names are single alphanumeric characters.
 
-   3) The main encoder advances to the next byte in the input data and
-calls the match finder.
+   * Certain options require an argument.
 
-   4) The match finder fills an array with the minimum distances before the
-current byte where a match of a given length can be found.
-
-   5) Go back to step 3 until a sequence (formed of pairs, repeated
-distances, and literal bytes) of minimum price has been formed. Where the
-price represents the number of output bits produced.
-
-   6) The range encoder encodes the sequence produced by the main encoder
-and sends the bytes produced to the output stream.
+   * An option and its argument may or may not appear as separate tokens.
+     (In other words, the whitespace separating them is optional). Thus,
+     '-o foo' and '-ofoo' are equivalent.
 
-   7) Go back to step 3 until the input data are finished or until the
-member or volume size limits are reached.
+   * One or more options without arguments, followed by at most one option
+     that takes an argument, may follow a hyphen in a single token. Thus,
+     '-abc' is equivalent to '-a -b -c'.
 
-   8) The range encoder is flushed.
+   * Options typically precede other non-option arguments.
 
-   9) The member trailer is written to the output stream.
+   * The argument '--' terminates all options; any following arguments are
+     treated as non-option arguments, even if they begin with a hyphen.
 
-   10) If there are more data to compress, go back to step 1.
+   * A token consisting of a single hyphen character is interpreted as an
+     ordinary non-option argument. By convention, it is used to specify
+     standard input, standard output, or a file named '-'.
 
+GNU adds "long options" to these conventions:
 
-   During compression, lzip reads data in large blocks (one dictionary size
-at a time). Therefore it may block for up to tens of seconds any process
-feeding data to it through a pipe. This is normal. The blocking intervals
-get longer with higher compression levels because dictionary size increases
-(and compression speed decreases) with compression level.
+   * A long option consists of two hyphens ('--') followed by a name made
+     of alphanumeric characters and hyphens. Option names are typically one
+     to three words long, with hyphens to separate words. Abbreviations can
+     be used for the long option names as long as the abbreviations are
+     unique.
 
-The ideas embodied in lzip are due to (at least) the following people:
-Abraham Lempel and Jacob Ziv (for the LZ algorithm), Andrei Markov (for the
-definition of Markov chains), G.N.N. Martin (for the definition of range
-encoding), Igor Pavlov (for putting all the above together in LZMA), and
-Julian Seward (for bzip2's CLI).
+   * A long option and its argument may or may not appear as separate
+     tokens. In the latter case they must be separated by an equal sign '='.
+     Thus, '--foo bar' and '--foo=bar' are equivalent.
 
 
-File: lzip.info,  Node: File format,  Next: Stream format,  Prev: Algorithm,  Up: Top
+File: lzip.info,  Node: File format,  Next: Stream format,  Prev: Argument syntax,  Up: Top
 
-6 File format
+5 File format
 *************
 
 Perfection is reached, not when there is no longer anything to add, but
 when there is no longer anything to take away.
 -- Antoine de Saint-Exupery
 
-
    In the diagram below, a box like this:
 
 +---+
@@ -731,12 +504,12 @@ when there is no longer anything to take away.
 
    represents a variable number of bytes.
 
-
-   A lzip file consists of one or more independent "members" (compressed
-data sets). The members simply appear one after another in the file, with no
+A lzip file consists of one or more independent "members" (compressed data
+sets). The members simply appear one after another in the file, with no
 additional information before, between, or after them. Each member can
 encode in compressed form up to 16 EiB - 1 byte of uncompressed data. The
-size of a multimember file is unlimited.
+size of a multimember file is unlimited. Empty members (data size = 0) are
+not allowed in multimember files.
 
    Each member has the following structure:
 
@@ -764,7 +537,7 @@ size of a multimember file is unlimited.
      Valid values for dictionary size range from 4 KiB to 512 MiB.
 
 'LZMA stream'
-     The LZMA stream, finished by an "End Of Stream" marker. Uses default
+     The LZMA stream, terminated by an 'End Of Stream' marker. Uses default
      values for encoder properties. *Note Stream format::, for a complete
      description.
 
@@ -781,14 +554,13 @@ size of a multimember file is unlimited.
      files. Lzip limits the member size to 2 PiB to prevent the data size
      field from overflowing.
 
-
 
-File: lzip.info,  Node: Stream format,  Next: Trailing data,  Prev: File format,  Up: Top
+File: lzip.info,  Node: Stream format,  Next: Quality assurance,  Prev: File format,  Up: Top
 
-7 Format of the LZMA stream in lzip files
+6 Format of the LZMA stream in lzip files
 *****************************************
 
-The LZMA algorithm has three parameters, called "special LZMA properties",
+The LZMA algorithm has three parameters, called 'special LZMA properties',
 to adjust it for some kinds of binary data. These parameters are:
 'literal_context_bits' (with a default value of 3),
 'literal_pos_state_bits' (with a default value of 0), and 'pos_state_bits'
@@ -797,14 +569,17 @@ uses the default values for these parameters. In particular
 'literal_pos_state_bits' has been optimized away and does not even appear
 in the code.
 
-   Lzip finishes the LZMA stream with an "End Of Stream" (EOS) marker (the
+   The first byte of the LZMA stream is set to zero to help tools like grep
+recognize lzip files as binary files.
+
+   The LZMA stream is terminated by an 'End Of Stream' (EOS) marker (the
 distance-length pair 0xFFFFFFFFU, 2), which in conjunction with the 'member
 size' field in the member trailer allows the checking of stream integrity.
 The EOS marker is the only LZMA marker allowed in lzip files. The LZMA
 stream in lzip files always has these two features (default properties and
 EOS marker) and is referred to in this document as LZMA-302eos. This
 simplified and marker-terminated form of the LZMA stream format has been
-chosen to maximize interoperability and safety.
+chosen to achieve complete interoperability and robust safety.
 
    The second stage of LZMA is a range encoder that uses a different
 probability model for each type of symbol: distances, lengths, literal
@@ -821,13 +596,12 @@ the range decoder that need to be described accurately, the source code of
 a real decompressor seems the only appropriate reference to use.
 
    What follows is a description of the decoding algorithm for LZMA-302eos
-streams using as reference the source code of "lzd", an educational
+streams using as reference the source code of lzd, an educational
 decompressor for lzip files, included in appendix A. *Note Reference source
 code::. Lzd is written in C++11 and can be downloaded from the lzip download
 directory.
 
-
-7.1 What is coded
+6.1 What is coded
 =================
 
 The LZMA stream includes literals, matches, and repeated matches (matches
@@ -901,7 +675,7 @@ slot + direct_bits                     distances from 4 to 127
 slot + (direct_bits - 4) + 4 bits      distances from 128 to 2^32 - 1
 
 
-7.2 The coding contexts
+6.2 The coding contexts
 =======================
 
 These contexts ('Bit_model' in the source), are integers or arrays of
@@ -991,8 +765,7 @@ corresponding bit in 'match_byte'. After the first difference is found, the
 rest of the byte is decoded using the normal bit tree context. (See
 'decode_matched' in the source).
 
-
-7.3 The range decoder
+6.3 The range decoder
 =====================
 
 The LZMA stream is consumed one byte at a time by the range decoder. (See
@@ -1005,29 +778,284 @@ of decoded bits, depending on how well these bits agree with their context.
 decoded) and 'code' (representing the current point within 'range').
 'range' is initialized to 2^32 - 1, and 'code' is initialized to 0.
 
-   The range encoder produces a first 0 byte that must be ignored by the
-range decoder. (See the 'Range_decoder' constructor in the source).
-
-
-7.4 Decoding and checking the LZMA stream
+6.4 Decoding and checking the LZMA stream
 =========================================
 
 After decoding the member header and obtaining the dictionary size, the
 range decoder is initialized and then the LZMA decoder enters a loop (see
 'decode_member' in the source) where it invokes the range decoder with the
 appropriate contexts to decode the different coding sequences (matches,
-repeated matches, and literal bytes), until the "End Of Stream" marker is
+repeated matches, and literal bytes), until the 'End Of Stream' marker is
 decoded.
 
-   Once the "End Of Stream" marker has been decoded, the decompressor reads
+   Once the 'End Of Stream' marker has been decoded, the decompressor reads
 and decodes the member trailer, and checks that the three integrity factors
 stored there (CRC, data size, and member size) match those computed from the
 data.
 
 
-File: lzip.info,  Node: Trailing data,  Next: Examples,  Prev: Stream format,  Up: Top
+File: lzip.info,  Node: Quality assurance,  Next: Algorithm,  Prev: Stream format,  Up: Top
 
-8 Extra data appended to the file
+7 Design, development, and testing of lzip
+******************************************
+
+There are two ways of constructing a software design: One way is to make it
+so simple that there are obviously no deficiencies and the other way is to
+make it so complicated that there are no obvious deficiencies. The first
+method is far more difficult.
+-- C.A.R. Hoare
+
+   Lzip has been designed, written, and tested with great care to replace
+gzip and bzip2 as general-purpose compressed format for Unix-like systems.
+This chapter describes the lessons learned from these previous formats, and
+their application to the design of lzip. The lzip format specification has
+been reviewed carefully and is believed to be free from design errors.
+
+7.1 Format design
+=================
+
+When gzip was designed in 1992, computers and operating systems were less
+capable than they are today. The designers of gzip tried to work around some
+of those limitations, like 8.3 file names, with additional fields in the
+file format.
+
+   Today those limitations have mostly disappeared, and the format of gzip
+has proved to be unnecessarily complicated. It includes fields that were
+never used, others that have lost their usefulness, and finally others that
+have become too limited.
+
+   Bzip2 was designed 5 years later, and its format is simpler than the one
+of gzip.
+
+   Probably the worst defect of the gzip format from the point of view of
+data safety is the variable size of its header. If the byte at offset 3
+(flags) of a gzip member gets corrupted, it may become difficult to recover
+the data, even if the compressed blocks are intact, because it can't be
+known with certainty where the compressed blocks begin.
+
+   By contrast, the header of a lzip member has a fixed length of 6. The
+LZMA stream in a lzip member always starts at offset 6, making it trivial to
+recover the data even if the whole header becomes corrupt.
+
+   Bzip2 also provides a header of fixed length and marks the begin and end
+of each compressed block with six magic bytes, making it possible to find
+the compressed blocks even in case of file damage. But bzip2 does not store
+the size of each compressed block, as lzip does.
+
+   Lziprecover is able to provide unique data recovery capabilities because
+the lzip format is extraordinarily safe. The simple and safe design of the
+file format complements the embedded error detection provided by the LZMA
+data stream. Any distance larger than the dictionary size acts as a
+forbidden symbol, allowing the decompressor to detect the approximate
+position of errors, and leaving very little work for the check sequence
+(CRC and data sizes) in the detection of errors. Lzip is usually able to
+detect all possible bit flips in the compressed data without resorting to
+the check sequence. It would be difficult to write an automatic recovery
+tool like lziprecover for the gzip format. And, as far as I know, it has
+never been written.
+
+   Lzip, like gzip and bzip2, uses a CRC32 to check the integrity of the
+decompressed data because it provides optimal accuracy in the detection of
+errors up to a compressed size of about 16 GiB, a size larger than that of
+most files. In the case of lzip, the additional detection capability of the
+decompressor reduces the probability of undetected errors several million
+times more, resulting in a combined integrity checking optimally accurate
+for any member size produced by lzip. Preliminary results suggest that the
+lzip format is safe enough to be used in critical safety avionics systems.
+
+   The lzip format is designed for long-term archiving. Therefore it
+excludes any unneeded features that may interfere with the future
+extraction of the decompressed data.
+
+7.1.1 Gzip format (mis)features not present in lzip
+---------------------------------------------------
+
+'Multiple algorithms'
+     Gzip provides a CM (Compression Method) field that has never been used
+     because it is a bad idea to begin with. New compression methods may
+     require additional fields, making it impossible to implement new
+     methods and, at the same time, keep the same format. This field does
+     not solve the problem of format proliferation; it just makes the
+     problem less obvious.
+
+'Optional fields in header'
+     Unless special precautions are taken, optional fields are generally a
+     bad idea because they produce a header of variable size. The gzip
+     header has 2 fields that, in addition to being optional, are
+     zero-terminated. This means that if any byte inside the field gets
+     zeroed, or if the terminating zero gets altered, gzip won't be able to
+     find neither the header CRC nor the compressed blocks.
+
+'Optional CRC for the header'
+     Using an optional CRC for the header is not only a bad idea, it is an
+     error; it circumvents the Hamming distance (HD) of the CRC and may
+     prevent the extraction of perfectly good data. For example, if the CRC
+     is used and the bit enabling it is reset by a bit flip, then the
+     header seems to be intact (in spite of being corrupt) while the
+     compressed blocks seem to be unrecoverable (in spite of being intact).
+     Very misleading indeed.
+
+'Metadata'
+     The gzip format stores some metadata, like the modification time of the
+     original file or the operating system on which compression took place.
+     This complicates reproducible compression (obtaining identical
+     compressed output from identical input).
+
+
+7.1.2 Lzip format improvements over gzip and bzip2
+--------------------------------------------------
+
+'64-bit size field'
+     Probably the most frequently reported shortcoming of the gzip format
+     is that it only stores the least significant 32 bits of the
+     uncompressed size. The size of any file larger or equal than 4 GiB
+     gets truncated.
+
+     Bzip2 does not store the uncompressed size of the file.
+
+     The lzip format provides a 64-bit field for the uncompressed size.
+     Additionally, lzip produces multimember output automatically when the
+     size is too large for a single member, allowing for an unlimited
+     uncompressed size.
+
+'Distributed index'
+     The lzip format provides a distributed index that, among other things,
+     helps plzip to decompress faster than pigz and helps lziprecover do
+     its job. Neither the gzip format nor the bzip2 format do provide an
+     index.
+
+     A distributed index is safer and more scalable than a monolithic
+     index. The monolithic index introduces a single point of failure in
+     the compressed file and may limit the number of members or the total
+     uncompressed size.
+
+
+7.2 Quality of implementation
+=============================
+
+Our civilization depends critically on software; it had better be quality
+software.
+-- Bjarne Stroustrup
+
+'Accurate and robust error detection'
+     The lzip format provides 3-factor integrity checking, and the
+     decompressors report mismatches in each factor separately. This method
+     detects most false positives for corruption. If just one byte in one
+     factor fails but the other two factors match the data, it probably
+     means that the data are intact and the corruption just affects the
+     mismatching factor (CRC, data size, or member size) in the member
+     trailer.
+
+'Multiple implementations'
+     Just like the lzip format provides 3-factor protection against
+     undetected data corruption, the development methodology of the lzip
+     family of compressors provides 3-factor protection against undetected
+     programming errors.
+
+     Three related but independent compressor implementations, lzip, clzip,
+     and minilzip/lzlib, are developed concurrently. Every stable release
+     of any of them is tested to check that it produces identical output to
+     the other two. This guarantees that all three implement the same
+     algorithm, and makes it unlikely that any of them may contain serious
+     undiscovered errors. In fact, no errors have been discovered in lzip
+     since 2009.
+
+     Additionally, the three implementations have been extensively tested
+     with unzcrash, valgrind, and 'american fuzzy lop' without finding a
+     single vulnerability or false negative. *Note Unzcrash:
+     (lziprecover)Unzcrash.
+
+'Dictionary size'
+     Lzip automatically adapts the dictionary size to the size of each file.
+     In addition to reducing the amount of memory required for
+     decompression, this feature also minimizes the probability of being
+     affected by RAM errors during compression.
+
+'Exit status'
+     Returning a warning status of 2 is a design flaw of compress that
+     leaked into the design of gzip. Both bzip2 and lzip are free from this
+     flaw.
+
+
+
+File: lzip.info,  Node: Algorithm,  Next: Trailing data,  Prev: Quality assurance,  Up: Top
+
+8 Algorithm
+***********
+
+In spite of its name (Lempel-Ziv-Markov chain-Algorithm), LZMA is not a
+concrete algorithm; it is more like "any algorithm using the LZMA coding
+scheme". LZMA compression consists in describing the uncompressed data as a
+succession of coding sequences from the set shown in Section 'What is
+coded' (*note what-is-coded::), and then encoding them using a range
+encoder. For example, the option '-0' of lzip uses the scheme in almost the
+simplest way possible; issuing the longest match it can find, or a literal
+byte if it can't find a match. Inversely, a more elaborate way of finding
+coding sequences of minimum size than the one currently used by lzip could
+be developed, and the resulting sequence could also be coded using the LZMA
+coding scheme.
+
+   Lzip currently implements two variants of the LZMA algorithm: fast (used
+by option '-0') and normal (used by all other compression levels).
+
+   The high compression of LZMA comes from combining two basic, well-proven
+compression ideas: sliding dictionaries (LZ77) and Markov models (the thing
+used by every compression algorithm that uses a range encoder or similar
+order-0 entropy coder as its last stage) with segregation of contexts
+according to what the bits are used for.
+
+   Lzip is a two stage compressor. The first stage is a Lempel-Ziv coder,
+which reduces redundancy by translating chunks of data to their
+corresponding distance-length pairs. The second stage is a range encoder
+that uses a different probability model for each type of data: distances,
+lengths, literal bytes, etc.
+
+   Here is how it works, step by step:
+
+   1) The member header is written to the output stream.
+
+   2) The first byte is coded literally, because there are no previous
+bytes to which the match finder can refer to.
+
+   3) The main encoder advances to the next byte in the input data and
+calls the match finder.
+
+   4) The match finder fills an array with the minimum distances before the
+current byte where a match of a given length can be found.
+
+   5) Go back to step 3 until a sequence (formed of pairs, repeated
+distances, and literal bytes) of minimum price has been formed. Where the
+price represents the number of output bits produced.
+
+   6) The range encoder encodes the sequence produced by the main encoder
+and sends the bytes produced to the output stream.
+
+   7) Go back to step 3 until the input data are finished or until the
+member or volume size limits are reached.
+
+   8) The range encoder is flushed.
+
+   9) The member trailer is written to the output stream.
+
+   10) If there are more data to compress, go back to step 1.
+
+
+   During compression, lzip reads data in large blocks (one dictionary size
+at a time). Therefore it may block for up to tens of seconds any process
+feeding data to it through a pipe. This is normal. The blocking intervals
+get longer with higher compression levels because dictionary size increases
+(and compression speed decreases) with compression level.
+
+The ideas embodied in lzip are due to (at least) the following people:
+Abraham Lempel and Jacob Ziv (for the LZ algorithm), Andrei Markov (for the
+definition of Markov chains), G.N.N. Martin (for the definition of range
+encoding), Igor Pavlov (for putting all the above together in LZMA), and
+Julian Seward (for bzip2's CLI).
+
+
+File: lzip.info,  Node: Trailing data,  Next: Examples,  Prev: Algorithm,  Up: Top
+
+9 Extra data appended to the file
 *********************************
 
 Sometimes extra data are found appended to a lzip file after the last
@@ -1037,7 +1065,7 @@ member. Such trailing data may be:
      example when writing to a tape. It is safe to append any amount of
      padding zero bytes to a lzip file.
 
-   * Useful data added by the user; an "End Of File" string (to check that
+   * Useful data added by the user; an 'End Of File' string (to check that
      the file has not been truncated), a cryptographically secure hash, a
      description of file contents, etc. It is safe to append any amount of
      text to a lzip file as long as none of the first four bytes of the
@@ -1073,8 +1101,8 @@ where a file containing trailing data must be rejected, the option
 
 File: lzip.info,  Node: Examples,  Next: Problems,  Prev: Trailing data,  Up: Top
 
-9 A small tutorial with examples
-********************************
+10 A small tutorial with examples
+*********************************
 
 WARNING! Even if lzip is bug-free, other causes may result in a corrupt
 compressed file (bugs in the system libraries, memory errors, etc).
@@ -1086,38 +1114,32 @@ comparing the compressed file with the original because the corruption
 happens before lzip compresses the RAM contents, resulting in a valid
 compressed file containing wrong data.
 
-
 Example 1: Extract all the files from archive 'foo.tar.lz'.
 
        tar -xf foo.tar.lz
      or
        lzip -cd foo.tar.lz | tar -xf -
 
-
 Example 2: Replace a regular file with its compressed version 'file.lz' and
 show the compression ratio.
 
      lzip -v file
 
-
 Example 3: Like example 2 but the created 'file.lz' is multimember with a
 member size of 1 MiB. The compression ratio is not shown.
 
      lzip -b 1MiB file
 
-
 Example 4: Restore a regular file from its compressed version 'file.lz'. If
 the operation is successful, 'file.lz' is removed.
 
      lzip -d file.lz
 
-
 Example 5: Check the integrity of the compressed file 'file.lz' and show
 status.
 
      lzip -tv file.lz
 
-
 Example 6: The right way of concatenating the decompressed output of two or
 more compressed files. *Note Trailing data::.
 
@@ -1126,19 +1148,16 @@ more compressed files. *Note Trailing data::.
      Do this instead
        lzip -cd file1.lz file2.lz file3.lz
 
-
 Example 7: Decompress 'file.lz' partially until 10 KiB of decompressed data
 are produced.
 
      lzip -cd file.lz | dd bs=1024 count=10
 
-
 Example 8: Decompress 'file.lz' partially from decompressed byte at offset
 10000 to decompressed byte at offset 14999 (5000 bytes are produced).
 
      lzip -cd file.lz | dd bs=1000 skip=10 count=5
 
-
 Example 9: Compress a whole device in /dev/sdc and send the output to
 'file.lz'.
 
@@ -1146,18 +1165,15 @@ Example 9: Compress a whole device in /dev/sdc and send the output to
      or
        lzip /dev/sdc -o file.lz
 
-
 Example 10: Create a multivolume compressed tar archive with a volume size
 of 1440 KiB.
 
      tar -c some_directory | lzip -S 1440KiB -o volume_name -
 
-
 Example 11: Extract a multivolume compressed tar archive.
 
      lzip -cd volume_name*.lz | tar -xf -
 
-
 Example 12: Create a multivolume compressed backup of a large database file
 with a volume size of 650 MB, where each volume is a multimember file with
 a member size of 32 MiB.
@@ -1167,7 +1183,7 @@ a member size of 32 MiB.
 
 File: lzip.info,  Node: Problems,  Next: Reference source code,  Prev: Examples,  Up: Top
 
-10 Reporting bugs
+11 Reporting bugs
 *****************
 
 There are probably bugs in lzip. There are certainly errors and omissions
@@ -1336,7 +1352,8 @@ public:
   Range_decoder()
     : member_pos( header_size ), code( 0 ), range( 0xFFFFFFFFU )
     {
-    get_byte();			// discard first byte of the LZMA stream
+    if( get_byte() != 0 )		// check first LZMA byte
+      { std::fputs( "Nonzero first LZMA byte.\n", stderr ); std::exit( 2 ); }
     for( int i = 0; i < 4; ++i ) code = ( code << 8 ) | get_byte();
     }
 
@@ -1579,8 +1596,7 @@ bool LZ_decoder::decode_member()	// Return false if error
                                              direct_bits );
         else
           {
-          rep0 +=
-            rdec.decode( direct_bits - dis_align_bits ) << dis_align_bits;
+          rep0 += rdec.decode( direct_bits-dis_align_bits ) << dis_align_bits;
           rep0 += rdec.decode_tree_reversed( bm_align, dis_align_bits );
           if( rep0 == 0xFFFFFFFFU )		// marker found
             {
@@ -1612,8 +1628,8 @@ int main( const int argc, const char * const argv[] )
       "Lzd decompresses from standard input to standard output.\n"
       "\nCopyright (C) 2024 Antonio Diaz Diaz.\n"
       "License 2-clause BSD.\n"
-      "This is free software: you are free to change and redistribute it.\n"
-      "There is NO WARRANTY, to the extent permitted by law.\n"
+      "This is free software: you are free to change and redistribute "
+      "it.\nThere is NO WARRANTY, to the extent permitted by law.\n"
       "Report bugs to lzip-bug@nongnu.org\n"
       "Lzd home page: http://www.nongnu.org/lzip/lzd.html\n",
       PROGVERSION, argv[0] );
@@ -1625,6 +1641,7 @@ int main( const int argc, const char * const argv[] )
   setmode( STDOUT_FILENO, O_BINARY );
 #endif
 
+  bool empty = false, multi = false;
   for( bool first_member = true; ; first_member = false )
     {
     Lzip_header header;				// check header
@@ -1639,12 +1656,12 @@ int main( const int argc, const char * const argv[] )
     unsigned dict_size = 1 << ( header[5] & 0x1F );
     dict_size -= ( dict_size / 16 ) * ( ( header[5] >> 5 ) & 7 );
     if( dict_size < min_dictionary_size || dict_size > max_dictionary_size )
-      { std::fputs( "Invalid dictionary size in member header.\n", stderr );
-        return 2; }
+      { std::fputs( "Invalid dictionary size in member header.\n",
+                    stderr ); return 2; }
 
     LZ_decoder decoder( dict_size );		// decode LZMA stream
     if( !decoder.decode_member() )
-      { std::fputs( "Data error\n", stderr ); return 2; }
+      { std::fputs( "Data error.\n", stderr ); return 2; }
 
     Lzip_trailer trailer;			// check trailer
     for( int i = 0; i < trailer_size; ++i ) trailer[i] = decoder.get_byte();
@@ -1652,25 +1669,28 @@ int main( const int argc, const char * const argv[] )
     unsigned crc = 0;
     for( int i = 3; i >= 0; --i ) crc = ( crc << 8 ) + trailer[i];
     if( crc != decoder.crc() )
-      { std::fputs( "CRC mismatch\n", stderr ); retval = 2; }
+      { std::fputs( "CRC mismatch.\n", stderr ); retval = 2; }
 
     unsigned long long data_size = 0;
     for( int i = 11; i >= 4; --i )
       data_size = ( data_size << 8 ) + trailer[i];
     if( data_size != decoder.data_position() )
-      { std::fputs( "Data size mismatch\n", stderr ); retval = 2; }
+      { std::fputs( "Data size mismatch.\n", stderr ); retval = 2; }
+    multi = !first_member; if( data_size == 0 ) empty = true;
 
     unsigned long long member_size = 0;
     for( int i = 19; i >= 12; --i )
       member_size = ( member_size << 8 ) + trailer[i];
     if( member_size != decoder.member_position() )
-      { std::fputs( "Member size mismatch\n", stderr ); retval = 2; }
+      { std::fputs( "Member size mismatch.\n", stderr ); retval = 2; }
     if( retval ) return retval;
     }
 
   if( std::fclose( stdout ) != 0 )
     { std::fprintf( stderr, "Error closing stdout: %s\n",
                     std::strerror( errno ) ); return 1; }
+  if( empty && multi )
+    { std::fputs( "Empty member not allowed.\n", stderr ); return 2; }
   return 0;
   }
 
@@ -1684,6 +1704,7 @@ Concept index
 * Menu:
 
 * algorithm:                             Algorithm.                 (line 6)
+* argument syntax:                       Argument syntax.           (line 6)
 * bugs:                                  Problems.                  (line 6)
 * examples:                              Examples.                  (line 6)
 * file format:                           File format.               (line 6)
@@ -1703,22 +1724,23 @@ Concept index
 
 Tag Table:
 Node: Top203
-Node: Introduction1197
-Node: Output7082
-Node: Invoking lzip8677
-Ref: --trailing-error9546
-Node: Quality assurance19644
-Node: Algorithm28457
-Node: File format31858
-Ref: coded-dict-size33287
-Node: Stream format34518
-Ref: what-is-coded36913
-Node: Trailing data45787
-Node: Examples48123
-Ref: concat-example49563
-Node: Problems50784
-Node: Reference source code51316
-Node: Concept index66361
+Node: Introduction1273
+Node: Output6965
+Node: Invoking lzip8560
+Ref: --trailing-error9397
+Node: Argument syntax19605
+Node: File format21367
+Ref: coded-dict-size22865
+Node: Stream format24097
+Ref: what-is-coded26621
+Node: Quality assurance35351
+Node: Algorithm44122
+Node: Trailing data47519
+Node: Examples49851
+Ref: concat-example51287
+Node: Problems52502
+Node: Reference source code53034
+Node: Concept index68345
 
 End Tag Table