This page contains a bunch of objects that implement various parts of compression algorithms.
They can be put together in different ways to construct many different algorithms.
Note that the compress_stream object contains complete compression algorithms. So if you
just want to compress some data then you can easily use that object and not bother with the others.
In the column to the right you can see benchmark data for each of the compress_stream
typedefs. The times measured are the time it takes to compress and then
decompress each file. It was run on a 3.0ghz P4. For reference see the Canterbury corpus
web site.
compress_streamdlib/compress_stream.hdlib/compress_stream/compress_stream_kernel_abstract.h
This object is pretty straight forward. It has no state and just
contains the functions compress and decompress.
They do just what their names imply to iostream objects.
compress_stream_ex.cpp.htmlfile_to_code_ex.cpp.htmlcompress_stream_kernel_1dlib/compress_stream/compress_stream_kernel_1.h
This implementation is done using the entropy_encoder_model and
entropy_decoder_model objects.
kernel_1ais a typedef for compress_stream_kernel_1 which uses entropy_decoder_model_kernel_1b and entropy_decoder_model_kernel_1bkernel_1bis a typedef for compress_stream_kernel_1 which uses entropy_decoder_model_kernel_2b and entropy_decoder_model_kernel_2b kernel_1cis a typedef for compress_stream_kernel_1 which uses entropy_decoder_model_kernel_3b and entropy_decoder_model_kernel_3b kernel_1dais a typedef for compress_stream_kernel_1 which uses entropy_decoder_model_kernel_4a and entropy_decoder_model_kernel_4a kernel_1dbis a typedef for compress_stream_kernel_1 which uses entropy_decoder_model_kernel_4b and entropy_decoder_model_kernel_4b kernel_1eais a typedef for compress_stream_kernel_1 which uses entropy_decoder_model_kernel_5a and entropy_decoder_model_kernel_5a kernel_1ebis a typedef for compress_stream_kernel_1 which uses entropy_decoder_model_kernel_5b and entropy_decoder_model_kernel_5b kernel_1ecis a typedef for compress_stream_kernel_1 which uses entropy_decoder_model_kernel_5c and entropy_decoder_model_kernel_5c compress_stream_kernel_2dlib/compress_stream/compress_stream_kernel_2.h
This implementation is done using the entropy_encoder_model and
entropy_decoder_model objects. It also uses the
lz77_buffer object. It uses the entropy coder models to
encode symbols when there is no match found by the lz77_buffer.
kernel_2ais a typedef for compress_stream_kernel_2 which uses entropy_encoder_model_kernel_2b, entropy_decoder_model_kernel_2b, and lz77_buffer_kernel_2a.compress_stream_kernel_3dlib/compress_stream/compress_stream_kernel_3.h
This implementation is done using the lzp_buffer object and
crc32 object. It does not use any sort of entropy coding, instead
a byte aligned output method is used.
kernel_3ais a typedef for compress_stream_kernel_3 which uses lzp_buffer_kernel_1.kernel_3bis a typedef for compress_stream_kernel_3 which uses lzp_buffer_kernel_2.conditioning_classdlib/conditioning_class.hdlib/conditioning_class/conditioning_class_kernel_abstract.h
This object represents a conditioning class used for arithmetic style
compression. It maintains the cumulative counts which are needed
by the entropy_encoder and entropy_decoder objects below.
conditioning_class_kernel_1dlib/conditioning_class/conditioning_class_kernel_1.h
This implementation is done using an array to store all the counts and they are summed
whenever the cumulative counts are requested. It's pretty straight forward.
kernel_1ais a typedef for conditioning_class_kernel_1conditioning_class_kernel_2dlib/conditioning_class/conditioning_class_kernel_2.h
This implementation is done using a binary tree where each node in the tree represents one symbol and
contains that symbols count and the sum of all the counts for the nodes to the left. This way
when you request a cumulative count it can be computed by visiting log n nodes where n is the
size of the alphabet.
kernel_2ais a typedef for conditioning_class_kernel_2conditioning_class_kernel_3dlib/conditioning_class/conditioning_class_kernel_3.h
This implementation is done using an array to store all the counts and they are
summed whenever the cumulative counts are requested. The counts are also kept in
semi-sorted order to speed up the calculation of the cumulative count.
kernel_3ais a typedef for conditioning_class_kernel_3conditioning_class_kernel_4dlib/conditioning_class/conditioning_class_kernel_4.h
This implementation is done using a linked list to store all the counts and they are
summed whenever the cumulative counts are requested. The counts are also kept in
semi-sorted order to speed up the calculation of the cumulative count. This implementation
also uses the memory_manager component to create a
memory pool of linked list nodes. This implementation is especially useful for high order
contexts and/or very large and sparse alphabets.
kernel_4ais a typedef for conditioning_class_kernel_4 with a memory pool of 10,000 nodes.kernel_4bis a typedef for conditioning_class_kernel_4 with a memory pool of 100,000 nodes.kernel_4cis a typedef for conditioning_class_kernel_4 with a memory pool of 1,000,000 nodes.kernel_4dis a typedef for conditioning_class_kernel_4 with a memory pool of 10,000,000 nodes.entropy_decoderdlib/entropy_decoder.hdlib/entropy_decoder/entropy_decoder_kernel_abstract.h
This object represents an entropy decoder. E.g. the decoding part of
an arithmetic coder.
entropy_decoder_kernel_1dlib/entropy_decoder/entropy_decoder_kernel_1.h
This object is implemented using arithmetic coding and is done in the
straight forward way using integers and fixed precision math.
kernel_1ais a typedef for entropy_decoder_kernel_1entropy_decoder_kernel_2dlib/entropy_decoder/entropy_decoder_kernel_2.h
This object is implemented using "range" coding and is done
in the straight forward way using integers and fixed precision math.
kernel_2ais a typedef for entropy_decoder_kernel_2entropy_encoderdlib/entropy_encoder.hdlib/entropy_encoder/entropy_encoder_kernel_abstract.h
This object represents an entropy encoder. E.g. the encoding part of
an arithmetic coder.
entropy_encoder_kernel_1dlib/entropy_encoder/entropy_encoder_kernel_1.h
This object is implemented using arithmetic coding and is done in the
straight forward way using integers and fixed precision math.
kernel_1ais a typedef for entropy_encoder_kernel_1entropy_encoder_kernel_2dlib/entropy_encoder/entropy_encoder_kernel_2.h
This object is implemented using "range" coding and is done
in the straight forward way using integers and fixed precision math.
kernel_2ais a typedef for entropy_encoder_kernel_2entropy_decoder_modeldlib/entropy_decoder_model.hdlib/entropy_decoder_model/entropy_decoder_model_kernel_abstract.h
This object represents some kind of statistical model. You
can use it to read symbols from an entropy_decoder and it will calculate
the cumulative counts/probabilities and manage contexts for you.
entropy_decoder_model_kernel_1dlib/entropy_decoder_model/entropy_decoder_model_kernel_1.h
This object is implemented using the conditioning_class component.
It implements an order-0 finite context model and uses lazy exclusions and update exclusions.
The escape method used is method D.
kernel_1ais a typedef for entropy_decoder_model_kernel_1 that uses conditioning_class_kernel_1akernel_1bis a typedef for entropy_decoder_model_kernel_1 that uses conditioning_class_kernel_2akernel_1cis a typedef for entropy_decoder_model_kernel_1 that uses conditioning_class_kernel_3aentropy_decoder_model_kernel_2dlib/entropy_decoder_model/entropy_decoder_model_kernel_2.h
This object is implemented using the conditioning_class component.
It implements an order-1-0 finite context model and uses lazy exclusions and update exclusions.
The escape method used is method D.
kernel_2ais a typedef for entropy_decoder_model_kernel_2 that uses conditioning_class_kernel_1akernel_2bis a typedef for entropy_decoder_model_kernel_2 that uses conditioning_class_kernel_2akernel_2cis a typedef for entropy_decoder_model_kernel_2 that uses conditioning_class_kernel_3akernel_2dis a typedef for entropy_decoder_model_kernel_2 that uses conditioning_class_kernel_2a for its order-0
context and conditioning_class_kernel_4b for its order-1 context.entropy_decoder_model_kernel_3dlib/entropy_decoder_model/entropy_decoder_model_kernel_3.h
This object is implemented using the conditioning_class component.
It implements an order-2-1-0 finite context model and uses lazy exclusions and update exclusions.
The escape method used is method D.
kernel_3a is a typedef for entropy_decoder_model_kernel_3 that uses conditioning_class_kernel_1a for orders 0 and 1
and conditioning_class_kernel_4b for order-2.kernel_3b is a typedef for entropy_decoder_model_kernel_3 that uses conditioning_class_kernel_2a for orders 0 and 1
and conditioning_class_kernel_4b for order-2.kernel_3c is a typedef for entropy_decoder_model_kernel_3 that uses conditioning_class_kernel_3a for orders 0 and 1
and conditioning_class_kernel_4b for order-2.entropy_decoder_model_kernel_4dlib/entropy_decoder_model/entropy_decoder_model_kernel_4.h
This object is implemented using a variation of the PPM algorithm described by Alistair Moffat in his paper "Implementing
the PPM data compression scheme."
It provides template arguments to select the maximum order and maximum memory to use. For speed,
exclusions are not used. The escape method used is method D.
kernel_4a is a typedef for entropy_decoder_model_kernel_4 with the max order set to 4 and the max number
of nodes set to 200,000kernel_4b is a typedef for entropy_decoder_model_kernel_4 with the max order set to 5 and the max number
of nodes set to 1,000,000entropy_decoder_model_kernel_5dlib/entropy_decoder_model/entropy_decoder_model_kernel_5.h
This object is implemented using a variation of the PPM algorithm described by Alistair Moffat in his paper "Implementing
the PPM data compression scheme."
It provides template arguments to select the maximum order and maximum memory to use. Exclusions are used. The escape method used is method D.
This implementation is very much like kernel_4 except it is tuned for higher compression rather than speed.
This also uses Dmitry Shkarin's Information Inheritance scheme.
kernel_5a is a typedef for entropy_decoder_model_kernel_5 with the max order set to 4 and the max number
of nodes set to 200,000kernel_5b is a typedef for entropy_decoder_model_kernel_5 with the max order set to 5 and the max number
of nodes set to 1,000,000kernel_5c is a typedef for entropy_decoder_model_kernel_5 with the max order set to 7 and the max number
of nodes set to 2,500,000entropy_decoder_model_kernel_6dlib/entropy_decoder_model/entropy_decoder_model_kernel_6.h
This object just assigns every symbol the same probability. I.e. it uses an order-(-1) model.
kernel_6ais a typedef for entropy_decoder_model_kernel_6entropy_encoder_modeldlib/entropy_encoder_model.hdlib/entropy_encoder_model/entropy_encoder_model_kernel_abstract.h
This object represents some kind of statistical model. You
can use it to write symbols to an entropy_encoder and it will calculate
the cumulative counts/probabilities and manage contexts for you.
entropy_encoder_model_kernel_1dlib/entropy_encoder_model/entropy_encoder_model_kernel_1.h
This object is implemented using the conditioning_class component.
It implements an order-0 finite context model and uses lazy exclusions and update exclusions.
The escape method used is method D.
kernel_1ais a typedef for entropy_encoder_model_kernel_1 that uses conditioning_class_kernel_1akernel_1bis a typedef for entropy_encoder_model_kernel_1 that uses conditioning_class_kernel_2akernel_1cis a typedef for entropy_encoder_model_kernel_1 that uses conditioning_class_kernel_3aentropy_encoder_model_kernel_2dlib/entropy_encoder_model/entropy_encoder_model_kernel_2.h
This object is implemented using the conditioning_class component.
It implements an order-1-0 finite context model and uses lazy exclusions and update exclusions.
The escape method used is method D.
kernel_2ais a typedef for entropy_encoder_model_kernel_2 that uses conditioning_class_kernel_1akernel_2bis a typedef for entropy_encoder_model_kernel_2 that uses conditioning_class_kernel_2akernel_2cis a typedef for entropy_encoder_model_kernel_2 that uses conditioning_class_kernel_3akernel_2dis a typedef for entropy_encoder_model_kernel_2 that uses conditioning_class_kernel_2a for its order-0
context and conditioning_class_kernel_4b for its order-1 context.entropy_encoder_model_kernel_3dlib/entropy_encoder_model/entropy_encoder_model_kernel_3.h
This object is implemented using the conditioning_class component.
It implements an order-2-1-0 finite context model and uses lazy exclusions and update exclusions.
The escape method used is method D.
kernel_3a is a typedef for entropy_encoder_model_kernel_3 that uses conditioning_class_kernel_1a for orders 0 and 1
and conditioning_class_kernel_4b for order-2.kernel_3b is a typedef for entropy_encoder_model_kernel_3 that uses conditioning_class_kernel_2a for orders 0 and 1
and conditioning_class_kernel_4b for order-2.kernel_3c is a typedef for entropy_encoder_model_kernel_3 that uses conditioning_class_kernel_3a for orders 0 and 1
and conditioning_class_kernel_4b for order-2.entropy_encoder_model_kernel_4dlib/entropy_encoder_model/entropy_encoder_model_kernel_4.h
This object is implemented using a variation of the PPM algorithm described by Alistair Moffat in his paper "Implementing
the PPM data compression scheme."
It provides template arguments to select the maximum order and maximum memory to use. For speed,
exclusions are not used. The escape method used is method D.
kernel_4a is a typedef for entropy_encoder_model_kernel_4 with the max order set to 4 and the max number
of nodes set to 200,000kernel_4b is a typedef for entropy_encoder_model_kernel_4 with the max order set to 5 and the max number
of nodes set to 1,000,000entropy_encoder_model_kernel_5dlib/entropy_encoder_model/entropy_encoder_model_kernel_5.h
This object is implemented using a variation of the PPM algorithm described by Alistair Moffat in his paper "Implementing
the PPM data compression scheme."
It provides template arguments to select the maximum order and maximum memory to use. Exclusions are used. The escape method used is method D.
This implementation is very much like kernel_4 except it is tuned for higher compression rather than speed.
This also uses Dmitry Shkarin's Information Inheritance scheme.
kernel_5a is a typedef for entropy_encoder_model_kernel_5 with the max order set to 4 and the max number
of nodes set to 200,000kernel_5b is a typedef for entropy_encoder_model_kernel_5 with the max order set to 5 and the max number
of nodes set to 1,000,000kernel_5c is a typedef for entropy_encoder_model_kernel_5 with the max order set to 7 and the max number
of nodes set to 2,500,000entropy_encoder_model_kernel_6dlib/entropy_encoder_model/entropy_encoder_model_kernel_6.h
This object just assigns every symbol the same probability. I.e. it uses an order-(-1) model.
kernel_6ais a typedef for entropy_encoder_model_kernel_6lz77_bufferdlib/lz77_buffer.hdlib/lz77_buffer/lz77_buffer_kernel_abstract.h
This object represents a pair of buffers (history and lookahead buffers)
used during lz77 style compression.
lz77_buffer_kernel_1dlib/lz77_buffer/lz77_buffer_kernel_1.h
This object is implemented using the sliding_buffer and it
just does simple linear searches of the history buffer to find matches.
kernel_1ais a typedef for lz77_buffer_kernel_1 that uses sliding_buffer_kernel_1lz77_buffer_kernel_2dlib/lz77_buffer/lz77_buffer_kernel_2.h
This object is implemented using the sliding_buffer. It
finds matches by using a hash table.
kernel_2ais a typedef for lz77_buffer_kernel_2 that uses sliding_buffer_kernel_1lzp_bufferdlib/lzp_buffer.hdlib/lzp_buffer/lzp_buffer_kernel_abstract.h
This object represents some variation on the LZP algorithm
described by Charles Bloom in his paper "LZP: a new data
compression algorithm"
lzp_buffer_kernel_1dlib/lzp_buffer/lzp_buffer_kernel_1.h
This object is implemented using the sliding_buffer and uses
an order-3 model to predict matches.
kernel_1ais a typedef for lzp_buffer_kernel_1 that uses sliding_buffer_kernel_1lzp_buffer_kernel_2dlib/lzp_buffer/lzp_buffer_kernel_2.h
This object is implemented using the sliding_buffer and uses
an order-5-4-3 model to predict matches.
kernel_2ais a typedef for lzp_buffer_kernel_2 that uses sliding_buffer_kernel_1