diff options
Diffstat (limited to 'NEWS')
| -rw-r--r-- | NEWS | 716 |
1 files changed, 716 insertions, 0 deletions
@@ -2,6 +2,566 @@ XZ Utils Release Notes ====================== +5.6.2 (2024-05-29) + + * Remove the backdoor (CVE-2024-3094). + + * Not changed: Memory sanitizer (MSAN) has a false positive + in the CRC CLMUL code which also makes OSS Fuzz unhappy. + Valgrind is smarter and doesn't complain. + + A revision to the CLMUL code is coming anyway and this issue + will be cleaned up as part of it. It won't be backported to + 5.6.x or 5.4.x because the old code isn't wrong. There is + no reason to risk introducing regressions in old branches + just to silence a false positive. + + * liblzma: + + - lzma_index_decoder() and lzma_index_buffer_decode(): Fix + a missing output pointer initialization (*i = NULL) if the + functions are called with invalid arguments. The API docs + say that such an initialization is always done. In practice + this matters very little because the problem can only occur + if the calling application has a bug and these functions + return LZMA_PROG_ERROR. + + - lzma_str_to_filters(): Fix a missing output pointer + initialization (*error_pos = 0). This is very similar + to the fix above. + + - Fix C standard conformance with function pointer types. + + - Remove GNU indirect function (IFUNC) support. This is *NOT* + done for security reasons even though the backdoor relied on + this code. The performance benefits of IFUNC are too tiny in + this project to make the extra complexity worth it. + + - FreeBSD on ARM64: Add error checking to CRC32 instruction + support detection. + + - Fix building with NVIDIA HPC SDK. + + * xz: + + - Fix a C standard conformance issue in --block-list parsing + (arithmetic on a null pointer). + + - Fix a warning from GNU groff when processing the man page: + "warning: cannot select font 'CW'" + + * xzdec: Add support for Linux Landlock ABI version 4. xz already + had the v3-to-v4 change but it had been forgotten from xzdec. + + * Autotools-based build system (configure): + + - Symbol versioning variant can now be overridden with + --enable-symbol-versions. Documentation in INSTALL was + updated to match. + + - Add new configure option --enable-doxygen to enable + generation and installation of the liblzma API documentation + using Doxygen. Documentation in INSTALL and PACKAGERS was + updated to match. + + CMake: + + - Fix detection of Linux Landlock support. The detection code + in CMakeLists.txt had been sabotaged. + + - Disable symbol versioning on non-glibc Linux to match what + the Autotools build does. For example, symbol versioning + isn't enabled with musl. + + - Symbol versioning variant can now be overridden by setting + SYMBOL_VERSIONING to "OFF", "generic", or "linux". + + - Add support for all tests in typical build configurations. + Now the only difference to the tests coverage to Autotools + is that CMake-based build will skip more tests if features + are disabled. Such builds are only for special cases like + embedded systems. + + - Separate the CMake code for the tests into tests/tests.cmake. + It is used conditionally, thus it is possible to + + rm -rf tests + + and the CMake-based build will still work normally except + that no tests are then available. + + - Add a option ENABLE_DOXYGEN to enable generation and + installation of the liblzma API documentation using Doxygen. + + * Documentation: + + - Omit the Doxygen-generated liblzma API documentation from the + package. Instead, the generation and installation of the API + docs can be enabled with a configure or CMake option if + Doxygen is available. + + - Remove the XZ logo which was used in the API documentation. + The logo has been retired and isn't used by the project + anymore. However, it's OK to use it in contexts that refer + to the backdoor incident. + + - Remove the PDF versions of the man pages from the source + package. These existed primarily for users of operating + systems which don't come with tools to render man page + source files. The plain text versions are still included + in doc/man/txt. PDF files can still be generated to doc/man, + if the required tools are available, using "make pdf" after + running "configure". + + - Update home page URLs back to their old locations on + tukaani.org. + + - Update maintainer info. + + * Tests: + + - In tests/files/README, explain how to recreate the ARM64 + test files. + + - Remove two tests that used tiny x86 and SPARC object files + as the input files. The matching .c file was included but + the object files aren't easy to reproduce. The test cases + weren't great anyway; they were from the early days (2009) + of the project when the test suite had very few tests. + + - Improve a few tests. + + +5.6.1 (2024-03-09) + + IMPORTANT: This fixed bugs in the backdoor (CVE-2024-3094) (someone + had forgot to run Valgrind). + + * liblzma: Fixed two bugs relating to GNU indirect function (IFUNC) + with GCC. The more serious bug caused a program linked with + liblzma to crash on start up if the flag -fprofile-generate was + used to build liblzma. The second bug caused liblzma to falsely + report an invalid write to Valgrind when loading liblzma. + + * xz: Changed the messages for thread reduction due to memory + constraints to only appear under the highest verbosity level. + + * Build: + + - Fixed a build issue when the header file <linux/landlock.h> + was present on the system but the Landlock system calls were + not defined in <sys/syscall.h>. + + - The CMake build now warns and disables NLS if both gettext + tools and pre-created .gmo files are missing. Previously, + this caused the CMake build to fail. + + * Minor improvements to man pages. + + * Minor improvements to tests. + + +5.6.0 (2024-02-24) + + IMPORTANT: This added a backdoor (CVE-2024-3094). It's enabled only + in the release tarballs. + + This bumps the minor version of liblzma because new features were + added. The API and ABI are still backward compatible with liblzma + 5.4.x and 5.2.x and 5.0.x. + + NOTE: As described in the NEWS for 5.5.2beta, the core components + are now under the BSD Zero Clause License (0BSD). + + Since 5.5.2beta: + + * liblzma: + + - Disabled the branchless C variant in the LZMA decoder based + on the benchmark results from the community. + + - Disabled x86-64 inline assembly on x32 to fix the build. + + * Sandboxing support in xz: + + - Landlock is now used even when xz needs to create files. + In this case the sandbox has to be more permissive than + when no files need to be created. A similar thing was + already in use with pledge(2) since 5.3.4alpha. + + - Landlock and pledge(2) are now stricter when reading from + more than one input file and only writing to standard output. + + - Added support for Landlock ABI version 4. + + * CMake: + + - Default to -O2 instead of -O3 with CMAKE_BUILD_TYPE=Release. + -O3 is not useful for speed and makes the code larger. + + - Now builds lzmainfo and lzmadec. + + - xzdiff, xzgrep, xzless, xzmore, and their symlinks are now + installed. The scripts are also tested during "make test". + + - Added translation support for xz, lzmainfo, and the + man pages. + + - Applied the symbol versioning workaround for MicroBlaze that + is used in the Autotools build. + + - The general XZ Utils and liblzma API documentation is now + installed. + + - The CMake component names were changed a little and several + were added. liblzma_Runtime and liblzma_Development are + unchanged. + + - Minimum required CMake version is now 3.14. However, + translation support is disabled with CMake versions + older than 3.20. + + - The CMake-based build is now close to feature parity with the + Autotools-based build. Most importantly a few tests aren't + run yet. Testing the CMake-based build on different operating + systems would be welcome now. See the comment at the top of + CMakeLists.txt. + + * Fixed a bug in the Autotools feature test for ARM64 CRC32 + instruction support for old versions of Clang. This did not + affect the CMake build. + + * Windows: + + - The build instructions in INSTALL and windows/INSTALL*.txt + were revised completely. + + - windows/build-with-cmake.bat along with the instructions + in windows/INSTALL-MinGW-w64_with_CMake.txt should make + it very easy to build liblzma.dll and xz.exe on Windows + using CMake and MinGW-w64 with either GCC or Clang/LLVM. + + - windows/build.bash was updated. It now works on MSYS2 and + on GNU/Linux (cross-compiling) to create a .zip and .7z + package for 32-bit and 64-bit x86 using GCC + MinGW-w64. + + * The TODO file is no longer installed as part of the + documentation. The file is out of date and does not reflect + the actual tasks that will be completed in the future. + + * Translations: + + - Translated lzmainfo man pages are now installed. These + had been forgotten in earlier versions. + + - Updated Croatian, Esperanto, German, Hungarian, Korean, + Polish, Romanian, Spanish, Swedish, Vietnamese, and Ukrainian + translations. + + - Updated German, Korean, Romanian, and Ukrainian man page + translations. + + * Added a few tests. + + Summary of new features added in the 5.5.x development releases: + + * liblzma: + + - LZMA decoder: Speed optimizations to the C code and + added GCC & Clang compatible inline assembly for x86-64. + + - Added lzma_mt_block_size() to recommend a Block size for + multithreaded encoding. + + - Added CLMUL-based CRC32 on x86-64 and E2K with runtime + processor detection. Similar to CRC64, on 32-bit x86 it + isn't available unless --disable-assembler is used. + + - Optimized the CRC32 calculation on ARM64 platforms using the + CRC32 instructions. Runtime detection for the instruction is + used on GNU/Linux, FreeBSD, Windows, and macOS. If the + compiler flags indicate unconditional CRC32 instruction + support (+crc) then the generic version is not built. + + - Added definitions of mask values like + LZMA_INDEX_CHECK_MASK_CRC32 to <lzma/index.h>. + + * xz: + + - Multithreaded mode is now the default. This improves + compression speed and creates .xz files that can be + decompressed in multithreaded mode. The downsides are + increased memory usage and slightly worse compression ratio. + + - Added a new command line option --filters to set the filter + chain using the liblzma filter string syntax. + + - Added new command line options --filters1 ... --filters9 to + set additional filter chains using the liblzma filter string + syntax. The --block-list option now allows specifying filter + chains that were set using these new options. + + - Ported the command line tools to Windows MSVC. + Visual Studio 2015 or later is required. + + * Added lz4 support to xzdiff/xzcmp and xzgrep. + + +5.5.2beta (2024-02-14) + + * Licensing change: The core components are now under the + BSD Zero Clause License (0BSD). In XZ Utils 5.4.6 and older + and 5.5.1alpha these components are in the public domain and + obviously remain so; the change affects the new releases only. + + 0BSD is an extremely permissive license which doesn't require + retaining or reproducing copyright or license notices when + distributing the code, thus in practice there is extremely + little difference to public domain. + + * liblzma + + - Significant speed optimizations to the LZMA decoder were + made. There are now three variants that can be chosen at + build time: + + * Basic C version: This is a few percent faster than + 5.4.x due to some new optimizations. + + * Branchless C: This is currently the default on platforms + for which there is no assembly code. This should be a few + percent faster than the basic C version. + + * x86-64 inline assembly. This works with GCC and Clang. + + The default choice can currently be overridden by setting + LZMA_RANGE_DECODER_CONFIG in CPPFLAGS: 0 means the basic + version and 3 means that branchless C version. + + - Optimized the CRC32 calculation on ARM64 platforms using the + CRC32 instructions. The instructions are optional in ARMv8.0 + and are required in ARMv8.1 and later. Runtime detection for + the instruction is used on GNU/Linux, FreeBSD, Windows, and + macOS. If the compiler flags indicate unconditional CRC32 + instruction support (+crc) then the generic version is not + built. + + * Added lz4 support to xzdiff/xzcmp and xzgrep. + + * Man pages of xzdiff/xzcmp, xzgrep, and xzmore were rewritten + to simplify licensing of the man page translations. + + * Translations: + + - Updated Chinese (simplified), German, Korean, Polish, + Romanian, Spanish, Swedish, and Ukrainian translations. + + - Updated German, Korean, Romanian, and Ukrainian man page + translations. + + * Small improvements to the tests. + + * Added doc/examples/11_file_info.c. It was added to the Git + repository in 2017 but forgotten to be added into distribution + tarballs. + + * Removed doc/examples_old. These were from 2012. + + * Removed the macos/build.sh script. It had not been updated + since 2013. + + +5.5.1alpha (2024-01-26) + + * Added a new filter for RISC-V binaries. The filter can be used + for 32-bit and 64-bit binaries with either little or big + endianness. In liblzma, the Filter ID is LZMA_FILTER_RISCV (0x0B) + and the xz option is --riscv. liblzma filter string syntax + recognizes this filter as "riscv". + + * liblzma: + + - Added lzma_mt_block_size() to recommend a Block size for + multithreaded encoding + + - Added CLMUL-based CRC32 on x86-64 and E2K with runtime + processor detection. Similar to CRC64, on 32-bit x86 it + isn't available unless --disable-assembler is used. + + - Implemented GNU indirect function (IFUNC) as a runtime + function dispatching method for CRC32 and CRC64 fast + implementations on x86. Only GNU/Linux (glibc) and FreeBSD + builds will use IFUNC, unless --enable-ifunc is specified to + configure. + + - Added definitions of mask values like + LZMA_INDEX_CHECK_MASK_CRC32 to <lzma/index.h>. + + - The XZ logo is now included in the Doxygen generated + documentation. It is licensed under Creative Commons + Attribution-ShareAlike 4.0. + + * xz: + + - Multithreaded mode is now the default. This improves + compression speed and creates .xz files that can be + decompressed multithreaded at the cost of increased memory + usage and slightly worse compression ratio. + + - Added new command line option --filters to set the filter + chain using liblzma filter string syntax. + + - Added new command line options --filters1 ... --filters9 to + set additional filter chains using liblzma filter string + syntax. The --block-list option now allows specifying filter + chains that were set using these new options. + + - Added support for Linux Landlock as a sandboxing method. + + - xzdec now supports pledge(2), Capsicum, and Linux Landlock as + sandboxing methods. + + - Progress indicator time stats remain accurate after pausing + xz with SIGTSTP. + + - Ported xz and xzdec to Windows MSVC. Visual Studio 2015 or + later is required. + + * CMake Build: + + - Supports pledge(2), Capsicum, and Linux Landlock sandboxing + methods. + + - Replacement functions for getopt_long() are used on platforms + that do not have it. + + * Enabled unaligned access by default on PowerPC64LE and on RISC-V + targets that define __riscv_misaligned_fast. + + * Tests: + + - Added two new fuzz targets to OSS-Fuzz. + + - Implemented Continuous Integration (CI) testing using + GitHub Actions. + + * Changed quoting style from `...' to '...' in all messages, + scripts, and documentation. + + * Added basic Codespell support to help catch typo errors. + + +5.4.7 (2024-05-29) + + * Not changed: Memory sanitizer (MSAN) has a false positive + in the CRC CLMUL code which also makes OSS Fuzz unhappy. + Valgrind is smarter and doesn't complain. + + A revision to the CLMUL code is coming anyway and this issue + will be cleaned up as part of it. It won't be backported to + 5.6.x or 5.4.x because the old code isn't wrong. There is + no reason to risk introducing regressions in old branches + just to silence a false positive. + + * liblzma: + + - lzma_index_decoder() and lzma_index_buffer_decode(): Fix + a missing output pointer initialization (*i = NULL) if the + functions are called with invalid arguments. The API docs + say that such an initialization is always done. In practice + this matters very little because the problem can only occur + if the calling application has a bug and these functions + return LZMA_PROG_ERROR. + + - lzma_str_to_filters(): Fix a missing output pointer + initialization (*error_pos = 0). This is very similar + to the fix above. + + - Fix C standard conformance with function pointer types. + This newly showed up with Clang 17 with -fsanitize=undefined. + There are no bug reports about this. + + - Fix building with NVIDIA HPC SDK. + + * xz: + + - Fix a C standard conformance issue in --block-list parsing + (arithmetic on a null pointer). + + - Fix a warning from GNU groff when processing the man page: + "warning: cannot select font 'CW'" + + - Fix outdated threading related information on the man page. + + * xzless: + + - With "less" version 451 and later, use "||-" instead of "|-" + in the environment variable LESSOPEN. This way compressed + files that contain no uncompressed data are shown correctly + as empty. + + - With "less" version 632 and later, use --show-preproc-errors + to make "less" show a warning on decompression errors. + + * Autotools-based build system (configure): + + - Symbol versioning variant can now be overridden with + --enable-symbol-versions. Documentation in INSTALL was + updated to match. + + CMake: + + - Linux on MicroBlaze is handled specially now. This matches + the changes made to the Autotools-based build in XZ Utils + 5.4.2 and 5.2.11. + + - Disable symbol versioning on non-glibc Linux to match what + the Autotools build does. For example, symbol versioning + isn't enabled with musl. + + - Symbol versioning variant can now be overridden by setting + SYMBOL_VERSIONING to "OFF", "generic", or "linux". + + * Documentation: + + - Clarify the description of --disable-assembler in INSTALL. + The option only affects 32-bit x86 assembly usage. + + - Add doc/examples/11_file_info.c. It was added to the + Git repository in 2017 but forgotten to be added into + distribution tarballs. + + - Don't install the TODO file as part of the documentation. + The file is out of date. + + - Update home page URLs back to their old locations on + tukaani.org. + + - Update maintainer info. + + +5.4.6 (2024-01-26) + + * Fixed a bug involving internal function pointers in liblzma not + being initialized to NULL. The bug can only be triggered if + lzma_filters_update() is called on a LZMA1 encoder, so it does + not affect xz or any application known to us that uses liblzma. + + * xz: + + - Fixed a regression introduced in 5.4.2 that caused encoding + in the raw format to unnecessarily fail if --suffix was not + used. For instance, the following command no longer reports + that --suffix must be used: + + echo foo | xz --format=raw --lzma2 | wc -c + + - Fixed an issue on MinGW-w64 builds that prevented reading + from or writing to non-terminal character devices like NUL. + + * Added a new test. + + 5.4.5 (2023-11-01) * liblzma: @@ -804,6 +1364,162 @@ XZ Utils Release Notes (FreeBSD >= 10). +5.2.13 (2024-05-29) + + * liblzma: + + - lzma_index_append(): Fix an assertion failure that could be + triggered by a large unpadded_size argument. It was verified + that there was no other bug than the assertion failure. + + - lzma_index_decoder() and lzma_index_buffer_decode(): Fix + a missing output pointer initialization (*i = NULL) if the + functions are called with invalid arguments. The API docs + say that such an initialization is always done. In practice + this matters very little because the problem can only occur + if the calling application has a bug and these functions + return LZMA_PROG_ERROR. + + - Fix C standard conformance with function pointer types. + This newly showed up with Clang 17 with -fsanitize=undefined. + There are no bug reports about this. + + - Fix building with NVIDIA HPC SDK. + + - Fix building with Windows Vista threads and --enable-small. + (CMake build doesn't support ENABLE_SMALL in XZ Utils 5.2.x.) + + * xz: + + - Fix a C standard conformance issue in --block-list parsing + (arithmetic on a null pointer). + + - Fix a warning from GNU groff when processing the man page: + "warning: cannot select font 'CW'" + + - Windows: Handle special files such as "con" or "nul". Earlier + the following wrote "foo" to the console and deleted the input + file "con_xz": + + echo foo | xz > con_xz + xz --suffix=_xz --decompress con_xz + + - Windows: Fix an issue that prevented reading from or writing + to non-terminal character devices like NUL. + + * xzless: + + - With "less" version 451 and later, use "||-" instead of "|-" + in the environment variable LESSOPEN. This way compressed + files that contain no uncompressed data are shown correctly + as empty. + + - With "less" version 632 and later, use --show-preproc-errors + to make "less" show a warning on decompression errors. + + * Build systems: + + - Add a new line to liblzma.pc for MSYS2 (Windows): + + Cflags.private: -DLZMA_API_STATIC + + When compiling code that will link against static liblzma, + the LZMA_API_STATIC macro needs to be defined on Windows. + + - Autotools (configure): + + * Symbol versioning variant can now be overridden with + --enable-symbol-versions. Documentation in INSTALL was + updated to match. + + - CMake: + + * Fix a bug that prevented other projects from including + liblzma multiple times using find_package(). + + * Fix a bug where configuring CMake multiple times resulted + in HAVE_CLOCK_GETTIME and HAVE_CLOCK_MONOTONIC not being + defined. + + * Fix the build with MinGW-w64-based Clang/LLVM 17. + llvm-windres now has more accurate GNU windres emulation + so the GNU windres workaround from 5.4.1 is needed with + llvm-windres version 17 too. + + * The import library on Windows is now properly named + "liblzma.dll.a" instead of "libliblzma.dll.a" + + * Add large file support by default for platforms that + need it to handle files larger than 2 GiB. This includes + MinGW-w64, even 64-bit builds. + + * Linux on MicroBlaze is handled specially now. This + matches the changes made to the Autotools-based build + in XZ Utils 5.4.2 and 5.2.11. + + * Disable symbol versioning on non-glibc Linux to match + what the Autotools build does. For example, symbol + versioning isn't enabled with musl. + + * Symbol versioning variant can now be overridden by + setting SYMBOL_VERSIONING to "OFF", "generic", or + "linux". + + * Documentation: + + - Clarify the description of --disable-assembler in INSTALL. + The option only affects 32-bit x86 assembly usage. + + - Don't install the TODO file as part of the documentation. + The file is out of date. + + - Update home page URLs back to their old locations on + tukaani.org. + + - Update maintainer info. + + +5.2.12 (2023-05-04) + + * Fixed a build system bug that prevented building liblzma as a + shared library when configured with --disable-threads. This bug + affected releases 5.2.6 to 5.2.11 and 5.4.0 to 5.4.2. + + * Include <intrin.h> for Windows intrinsic functions where they are + needed. This fixed a bug that prevented building liblzma using + clang-cl on Windows. + + * Minor update to the Croatian translation. The small change + applies to a string in both 5.2 and 5.4 branches. + + +5.2.11 (2023-03-18) + + * Removed all possible cases of null pointer + 0. It is undefined + behavior in C99 and C17. This was detected by a sanitizer and had + not caused any known issues. + + * Build systems: + + - Added a workaround for building with GCC on MicroBlaze Linux. + GCC 12 on MicroBlaze doesn't support the __symver__ attribute + even though __has_attribute(__symver__) returns true. The + build is now done without the extra RHEL/CentOS 7 symbols + that were added in XZ Utils 5.2.7. The workaround only + applies to the Autotools build (not CMake). + + - CMake: Ensure that the C compiler language is set to C99 or + a newer standard. + + - CMake changes from XZ Utils 5.4.1: + + * Added a workaround for a build failure with + windres from GNU binutils. + + * Included the Windows resource files in the xz + and xzdec build rules. + + 5.2.10 (2022-12-13) * xz: Don't modify argv[] when parsing the --memlimit* and |