summaryrefslogtreecommitdiffstats
path: root/tests/files/README
diff options
context:
space:
mode:
Diffstat (limited to '')
-rw-r--r--tests/files/README409
1 files changed, 409 insertions, 0 deletions
diff --git a/tests/files/README b/tests/files/README
new file mode 100644
index 0000000..119cac4
--- /dev/null
+++ b/tests/files/README
@@ -0,0 +1,409 @@
+
+.xz and .lzma Test Files
+------------------------
+
+0. Introduction
+
+ This directory contains bunch of files to test handling of .xz,
+ .lzma (LZMA_Alone), and .lz (lzip) files in decoder implementations.
+ Many of the files have been created by hand with a hex editor, thus
+ there is no better "source code" than the files themselves. All the
+ test files and this README have been put into the public domain.
+
+
+1. File Types
+
+ Good files (good-*) must decode successfully without requiring
+ a lot of CPU time or RAM.
+
+ Unsupported files (unsupported-*) are good files, but headers
+ indicate features not supported by the current file format
+ specification.
+
+ Bad files (bad-*) must cause the decoder to give an error. Like
+ with the good files, these files must not require a lot of CPU
+ time or RAM before they get detected to be broken.
+
+
+2. Descriptions of Individual .xz Files
+
+2.1. Good Files
+
+ good-0-empty.xz has one Stream with no Blocks.
+
+ good-0pad-empty.xz has one Stream with no Blocks followed by
+ four-byte Stream Padding.
+
+ good-0cat-empty.xz has two zero-Block Streams concatenated without
+ Stream Padding.
+
+ good-0catpad-empty.xz has two zero-Block Streams concatenated with
+ four-byte Stream Padding between the Streams.
+
+ good-1-check-none.xz has one Stream with one Block with two
+ uncompressed LZMA2 chunks and no integrity check.
+
+ good-1-check-crc32.xz has one Stream with one Block with two
+ uncompressed LZMA2 chunks and CRC32 check.
+
+ good-1-check-crc64.xz is like good-1-check-crc32.xz but with CRC64.
+
+ good-1-check-sha256.xz is like good-1-check-crc32.xz but with
+ SHA256.
+
+ good-2-lzma2.xz has one Stream with two Blocks with one uncompressed
+ LZMA2 chunk in each Block.
+
+ good-1-block_header-1.xz has both Compressed Size and Uncompressed
+ Size in the Block Header. This has also four extra bytes of Header
+ Padding.
+
+ good-1-block_header-2.xz has known Compressed Size.
+
+ good-1-block_header-3.xz has known Uncompressed Size.
+
+ good-1-delta-lzma2.tiff.xz is an image file that compresses
+ better with Delta+LZMA2 than with plain LZMA2.
+
+ good-1-x86-lzma2.xz uses the x86 filter (BCJ) and LZMA2. The
+ uncompressed file is compress_prepared_bcj_x86 found from the tests
+ directory.
+
+ good-1-sparc-lzma2.xz uses the SPARC filter and LZMA2. The
+ uncompressed file is compress_prepared_bcj_sparc found from the tests
+ directory.
+
+ good-1-arm64-lzma2-1.xz uses the ARM64 filter and LZMA2. The
+ uncompressed data is constructed so that it tests integer
+ wrap around and sign extension.
+
+ good-1-arm64-lzma2-2.xz is like good-1-arm64-lzma2-1.xz but with
+ non-zero start offset. XZ Embedded doesn't support this file.
+
+ good-1-lzma2-1.xz has two LZMA2 chunks, of which the second sets
+ new properties.
+
+ good-1-lzma2-2.xz has two LZMA2 chunks, of which the second resets
+ the state without specifying new properties.
+
+ good-1-lzma2-3.xz has two LZMA2 chunks, of which the first is
+ uncompressed and the second is LZMA. The first chunk resets dictionary
+ and the second sets new properties.
+
+ good-1-lzma2-4.xz has three LZMA2 chunks: First is LZMA, second is
+ uncompressed with dictionary reset, and third is LZMA with new
+ properties but without dictionary reset.
+
+ good-1-lzma2-5.xz has an empty LZMA2 stream with only the end of
+ payload marker. XZ Utils 5.0.1 and older incorrectly see this file
+ as corrupt.
+
+ good-1-3delta-lzma2.xz has three Delta filters and LZMA2.
+
+ good-1-empty-bcj-lzma2.xz has an empty Block that uses PowerPC BCJ
+ and LZMA2. liblzma from XZ Utils 5.0.1 and older may incorrectly
+ return LZMA_BUF_ERROR in some cases. See commit message
+ d8db706acb8316f9861abd432cfbe001dd6d0c5c for the details.
+
+
+2.2. Unsupported Files
+
+ unsupported-check.xz uses Check ID 0x02 which isn't supported by
+ the current version of the file format. It is implementation-defined
+ how this file handled (it may reject it, or decode it possibly with
+ a warning).
+
+ unsupported-block_header.xz has a non-null byte in Header Padding,
+ which may indicate presence of a new unsupported field.
+
+ unsupported-filter_flags-1.xz has unsupported Filter ID 0x7F.
+
+ unsupported-filter_flags-2.xz specifies only Delta filter in the
+ List of Filter Flags, but Delta isn't allowed as the last filter in
+ the chain. It could be a little more correct to detect this file as
+ corrupt instead of unsupported, but saying it is unsupported is
+ simpler in case of liblzma.
+
+ unsupported-filter_flags-3.xz specifies two LZMA2 filters in the
+ List of Filter Flags. LZMA2 is allowed only as the last filter in the
+ chain. It could be a little more correct to detect this file as
+ corrupt instead of unsupported, but saying it is unsupported is
+ simpler in case of liblzma.
+
+
+2.3. Bad Files
+
+ bad-0pad-empty.xz has one Stream with no Blocks followed by
+ five-byte Stream Padding. Stream Padding must be a multiple of four
+ bytes, thus this file is corrupt.
+
+ bad-0catpad-empty.xz has two zero-Block Streams concatenated with
+ five-byte Stream Padding between the Streams.
+
+ bad-0cat-alone.xz is good-0-empty.xz concatenated with an empty
+ LZMA_Alone file.
+
+ bad-0cat-header_magic.xz is good-0cat-empty.xz but with one byte
+ wrong in the Header Magic Bytes field of the second Stream. liblzma
+ gives LZMA_DATA_ERROR for this. (LZMA_FORMAT_ERROR is used only if
+ the first Stream of a file has invalid Header Magic Bytes.)
+
+ bad-0-header_magic.xz is good-0-empty.xz but with one byte wrong
+ in the Header Magic Bytes field. liblzma gives LZMA_FORMAT_ERROR for
+ this.
+
+ bad-0-footer_magic.xz is good-0-empty.xz but with one byte wrong
+ in the Footer Magic Bytes field. liblzma gives LZMA_DATA_ERROR for
+ this.
+
+ bad-0-empty-truncated.xz is good-0-empty.xz without the last byte
+ of the file.
+
+ bad-0-nonempty_index.xz has no Blocks but Index claims that there is
+ one Block.
+
+ bad-0-backward_size.xz has wrong Backward Size in Stream Footer.
+
+ bad-1-stream_flags-1.xz has different Stream Flags in Stream Header
+ and Stream Footer.
+
+ bad-1-stream_flags-2.xz has wrong CRC32 in Stream Header.
+
+ bad-1-stream_flags-3.xz has wrong CRC32 in Stream Footer.
+
+ bad-1-vli-1.xz has two-byte variable-length integer in the
+ Uncompressed Size field in Block Header while one-byte would be enough
+ for that value. It's important that the file gets rejected due to too
+ big integer encoding instead of due to Uncompressed Size not matching
+ the value stored in the Block Header. That is, the decoder must not
+ try to decode the Compressed Data field.
+
+ bad-1-vli-2.xz has ten-byte variable-length integer as Uncompressed
+ Size in Block Header. It's important that the file gets rejected due
+ to too big integer encoding instead of due to Uncompressed Size not
+ matching the value stored in the Block Header. That is, the decoder
+ must not try to decode the Compressed Data field.
+
+ bad-1-block_header-1.xz has Block Header that ends in the middle of
+ the Filter Flags field.
+
+ bad-1-block_header-2.xz has Block Header that has Compressed Size and
+ Uncompressed Size but no List of Filter Flags field.
+
+ bad-1-block_header-3.xz has wrong CRC32 in Block Header.
+
+ bad-1-block_header-4.xz has too big Compressed Size in Block Header
+ (2^63 - 1 bytes while maximum is a little less, because the whole
+ Block must stay smaller than 2^63). It's important that the file
+ gets rejected due to invalid Compressed Size value; the decoder
+ must not try decoding the Compressed Data field.
+
+ bad-1-block_header-5.xz has zero as Compressed Size in Block Header.
+
+ bad-1-block_header-6.xz has corrupt Block Header which may crash
+ xz -lvv in XZ Utils 5.0.3 and earlier. It was fixed in the commit
+ c0297445064951807803457dca1611b3c47e7f0f.
+
+ bad-2-index-1.xz has wrong Unpadded Sizes in Index.
+
+ bad-2-index-2.xz has wrong Uncompressed Sizes in Index.
+
+ bad-2-index-3.xz has non-null byte in Index Padding.
+
+ bad-2-index-4.xz wrong CRC32 in Index.
+
+ bad-2-index-5.xz has zero as Unpadded Size. It is important that the
+ file gets rejected specifically due to Unpadded Size having an invalid
+ value.
+
+ bad-3-index-uncomp-overflow.xz has Index whose Uncompressed Size
+ fields have huge values whose sum exceeds the maximum allowed size
+ of 2^63 - 1 bytes. In this file the sum is exactly 2^64.
+ lzma_index_append() in liblzma <= 5.2.6 lacks the integer overflow
+ check for the uncompressed size and thus doesn't catch the error
+ when decoding the Index field in this file. This makes "xz -l"
+ not detect the error and will display 0 as the uncompressed size.
+ Note that regular decompression isn't affected by this bug because
+ it uses lzma_index_hash_append() instead.
+
+ bad-2-compressed_data_padding.xz has non-null byte in the padding of
+ the Compressed Data field of the first Block.
+
+ bad-1-check-crc32.xz has wrong Check (CRC32).
+
+ bad-1-check-crc32-2.xz has Compressed Size and Uncompressed Size in
+ Block Header but wrong Check (CRC32) in the actual data. This file
+ differs by one byte from good-1-block_header-1.xz: the last byte of
+ the Check field is wrong. This file is useful for testing error
+ detection in the threaded decoder when a worker thread is configured
+ to pass input one byte at a time to the Block decoder.
+
+ bad-1-check-crc64.xz has wrong Check (CRC64).
+
+ bad-1-check-sha256.xz has wrong Check (SHA-256).
+
+ bad-1-lzma2-1.xz has LZMA2 stream whose first chunk (uncompressed)
+ doesn't reset the dictionary.
+
+ bad-1-lzma2-2.xz has two LZMA2 chunks, of which the second chunk
+ indicates dictionary reset, but the LZMA compressed data tries to
+ repeat data from the previous chunk.
+
+ bad-1-lzma2-3.xz sets new invalid properties (lc=8, lp=0, pb=0) in
+ the middle of Block.
+
+ bad-1-lzma2-4.xz has two LZMA2 chunks, of which the first is
+ uncompressed and the second is LZMA. The first chunk resets dictionary
+ as it should, but the second chunk tries to reset state without
+ specifying properties for LZMA.
+
+ bad-1-lzma2-5.xz is like bad-1-lzma2-4.xz but doesn't try to reset
+ anything in the header of the second chunk.
+
+ bad-1-lzma2-6.xz has reserved LZMA2 control byte value (0x03).
+
+ bad-1-lzma2-7.xz has EOPM at LZMA level.
+
+ bad-1-lzma2-8.xz is like good-1-lzma2-4.xz but doesn't set new
+ properties in the third LZMA2 chunk.
+
+ bad-1-lzma2-9.xz has LZMA2 stream that is truncated at the end of
+ a LZMA2 chunk (no end marker). The uncompressed size of the partial
+ LZMA2 stream exceeds the value stored in the Block Header.
+
+ bad-1-lzma2-10.xz has LZMA2 stream that, from point of view of a
+ LZMA2 decoder, extends past the end of Block (and even the end of
+ the file). Uncompressed Size in Block Header is bigger than the
+ invalid LZMA2 stream may produce (even if a decoder reads until
+ the end of the file). The Check type is None to nullify certain
+ simple size-based sanity checks in a Block decoder.
+
+ bad-1-lzma2-11.xz has LZMA2 stream that lacks the end of
+ payload marker. When Compressed Size bytes have been decoded,
+ Uncompressed Size bytes of output will have been produced but
+ the LZMA2 decoder doesn't indicate end of stream.
+
+
+3. Descriptions of Individual .lzma Files
+
+3.1. Good Files
+
+ good-unknown_size-with_eopm.lzma has unknown size in the header
+ and end of payload marker at the end.
+
+ good-known_size-without_eopm.lzma has a known size in the header
+ and no end of payload marker at the end.
+
+ good-known_size-with_eopm.lzma has a known size in the header
+ and end of payload marker at the end. XZ Utils 5.2.5 and older
+ will give an error at the end of the file after producing the
+ correct uncompressed output.
+
+
+3.2. Bad Files
+
+ bad-unknown_size-without_eopm.lzma has unknown size in the header
+ but no end of payload marker at the end. This file might be seen
+ by a decoder as if it were truncated.
+
+ bad-too_big_size-with_eopm.lzma has too big uncompressed size in
+ the header and the end of payload marker will be detected before
+ the specified number of bytes have been decoded.
+
+ bad-too_small_size-without_eopm-1.lzma has too small uncompressed
+ size in the header. The decoder will look for end of payload marker
+ but instead find a literal that would produce more output.
+
+ bad-too_small_size-without_eopm-2.lzma is like -1 above but instead
+ of a literal the problem occurs with a short repeated match.
+
+ bad-too_small_size-without_eopm-3.lzma is like -1 above but instead
+ of a literal the problem occurs in the middle of a match.
+
+
+4. Descriptions of Individual .lz (lzip) Files
+
+4.1. Good Files
+
+ good-1-v0.lz contains a single version 0 member. lzip 1.17 and
+ *older* can decompress this; support for version 0 was removed
+ in lzip 1.18.
+
+ good-1-v0-trailing-1.lz is like good-1-v0.lz but contains
+ trailing data that the decompressor must ignore.
+
+ good-1-v1.lz contains a single version 1 member. lzip 1.3 and
+ newer can decompress this.
+
+ good-1-v1-trailing-1.lz is like good-1-v1.lz but contains
+ trailing data that the decompressor must ignore.
+
+ good-1-v1-trailing-2.lz is like good-1-v1.lz but contains
+ trailing data whose first three bytes match the .lz magic bytes.
+ With lzip >= 1.20 this file results in an error unless one uses
+ the command line option --loose-trailing. lzip 1.3 to 1.19 decode
+ this file successfully by default. XZ Utils uses the old behavior
+ because it allows lzma_code() to stop at the first byte of the
+ trailing data as long as the first byte isn't 0x4C (L in US-ASCII);
+ otherwise the first 1-3 bytes that equal to the magic bytes are
+ consumed and lost in lzma_code(), and this is visible in xz too:
+
+ $ ( xz -dc ; cat ) < good-1-v1-trailing-2.lz
+ Hello
+ World!
+ Trailing garbage
+
+ $ ( xz -dc --single-stream ; cat ) < good-1-v1-trailing-2.lz
+ Hello
+ World!
+ LZITrailing garbage
+
+ good-2-v0-v1.lz contains two members of which the first is
+ version 0 and the second version 1. lzip versions 1.3 to 1.17
+ (inclusive) can decompress this.
+
+ good-2-v1-v0.lz contains two members of which the first is
+ version 1 and the second version 0. lzip versions 1.3 to 1.17
+ (inclusive) can decompress this.
+
+ good-2-v1-v1.lz contains two version 1 members. lzip versions 1.3
+ and newer can decompress this.
+
+
+4.2. Unsupported Files
+
+ unsupported-1-v234.lz is like good-1-v1.lz except the version
+ field has been set to 234 (0xEA) which, as of writing, isn't
+ defined or supported by any .lz implementation.
+
+
+4.3. Bad Files
+
+ bad-1-v1-magic-1.lz is like good-1-v1.lz but the first magic byte
+ is wrong.
+
+ bad-1-v1-magic-2.lz is like good-1-v1.lz but the last (fourth)
+ magic byte is wrong.
+
+ bad-1-v1-dict-1.lz has too low value in the dictionary size field.
+
+ bad-1-v1-dict-2.lz has too high value in the dictionary size field.
+
+ bad-1-v1-crc32.lz has wrong CRC32 value.
+
+ bad-1-v0-uncomp-size.lz is version 0 format with incorrect value
+ in the uncompressed size field.
+
+ bad-1-v1-uncomp-size.lz is version 1 format with incorrect value
+ in the uncompressed size field.
+
+ bad-1-v1-member-size.lz has incorrect value in the member size
+ field.
+
+ bad-1-v1-trailing-magic.lz has the four .lz magic bytes as trailing
+ data. This should be detected as a truncated file and thus result
+ in an error. That is, the last four bytes of the file should not be
+ ignored as trailing garbage. lzip >= 1.18 matches this behavior
+ while older versions ignore the last four bytes and don't indicate
+ an error.
+