# Changelog All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ## [Unreleased] ### Added - libsquashfs: Add a helper function to initialize objects - libsquashfs: Add `sqfs_xattr_t` and helpers, for handling combined, decoded key-value pairs - libsquashfs: `sqfs_xattr_reader_t` and `sqfs_xattr_writer_t` functions for handling `sqfs_xattr_t` directly - libsquashfs: `sqfs_istream_t` and `sqfs_ostream_t` stream I/O interfaces, cleaned up and moved from internal helper libraries into libsquashfs, along with native implementations used by tools. - libsquashfs: Add a `get_filename` method to `sqfs_file_t` - libsquashfs: Add native file handle type and I/O wrappers, used internally by native file implementations. - libsquashfs: Add `sqfs_dir_iterator_t` interface, Windows & Unix native implementations, stacked recursive implementation for tools - libsquashfs: Add a data reader based `sqfs_istream_t` implementation - Tools: collect and print statistics about the kind of files we are packing - tar2sqfs: Add option to exclude files ### Fixed - Fix broken C++ guard in rbtree.h - documentation: Compressor ID enumerator - rdsquashfs: improve unpacking error message on Windows - gensquashfs: sort by file breaking up the directory list - Win32: Fix fstree defaults in Windows version of `fstree_from_dir` - Win32: Fix fstree CLI mtime range check ### Changed - libsquashfs: add a threshold for extended directory inodes with index - libsquashfs: Make `sqfs_object_t` to reference counted - Internal cleanups and restructuring ### Removed - Build system: Remove without-tools feature switch - libsquashfs: remove the default block writer alignment feature ## [1.2.0] - 2022-12-03 ### Added - Make it possible for `gensquashfs` to add xattrs from a description file - Make it possible for `gensquashfs` to specify file sorting order - Support `.` and `..` directory browsing support in `sqfs_dir_reader_t` - Add `sqfs_tree_node_get_path` to `libsquashfs` ### Fixed - Only use actually available CPUs in `gensquashfs` and `tar2sqfs` - Overzealous bounds check in the `libsquashfs `block processor (#110) - In `sqfs_compressor_create`, clear the output pointer on error (#110) - Typo in `gensquashfs` and `tar2sqfs` block count statistics (#108) - Hard link resolution: report error if path cannonicalization fails - Documentation: signedness of the directory header inode number - Seek position in `libsquashfs` meta data reader erroneously out-of-bounds - Memory leak in `libfstree` error path - Double-free in `libfstree` error path ### Changed - Allow NULL pointer for `sqfs_destroy`, `sqfs_dir_tree_destroy` - Use 32 bit hard link counter in `libfstree` instead of just 16 bit - Internal cleanup and restructuring - Upgrade autotools version - Check in `autogen.sh` if the required tools are available - Trigger a build failure if no compressor library is available ## [1.1.4] - 2022-03-30 ### Added - libsquashfs: A flag for `sqfs_open_file` to *not* perform any local charset based transformations, e.g. UTF-8 to UTF-16 on Windows (#96). ### Fixed - libsquashfs: uninitializeed state in the directory reader caused by rewind call after reading an emtpy directory. - libsquashfs: stricter bounds checking in directory reader. - libsquashfs: Guard against multiplication overflow in the data reader on 32 bit systems. - structure alignment test on 32 bit x86 systems (#93) - Windows: error handling on FlushFileBuffers (#97) - Windows: loop exit condition in mkdir_p (#97) - Windows: error handling when reading from a pipe (#102) - Windows: heap corruption in stdout stream wrapper (#100) - Use the correct printf specifiers for 64 bit types on 32 bit systems. - Miscellaneous static analysis and compiler warnings. ### Changed - Windows: add a wrapper for main() that reads the UTF-16 command line and converts it to UTF-8 for propper Unicode support (#96). - Windows: when opening a file, treat the path as UTF-8, convert it to UTF-16 and use the WCHAR API for propper Unicode support (#96). - Windows: redirect stdio print functions, convert from UTF-8 to UTF-16 if the target is the console for propper Unicode support (#96). - libsquashfs: Windows: when opening files, interpret the API argument path as UTF-8, convert it to UTF-16 and use the WCHAR API, unless explictly told not to (#96). - Upgraded version of the builtin copy of zlib. ## [1.1.3] - 2021-08-15 ### Added - Overhaul format documentation, convert to ASCIIdoc (#88, #90) - Add an explicit license boiler plate to all the example programs (#89) - Additional example code & documentation ### Fixed - Typos and formating issues in the documentation (#86) - Symlink based path traversal in rdsqaushfs when unpacking to a directory - libsquashfs directory writer size accounting (#85) - tar2sqfs, sqfs2tar: 4 GiB limitation for files in tarballs (#87) ## [1.1.2] - 2021-06-25 ### Added - Test cases for concatenated stream decompression - A more "real-world" test suite for `tar2sqfs` pre-release testing ### Fixed - Replace tabs with spaces in format.txt - Some documentation clarifications and typo fixes - libsquashfs: preserve alignment flag in block processor - libsquashfs: broken block alignment in block write - allow concatenated Bzip2 streams - Use Automake conditional for zstd stream compression support - Use *_MAX from limits.h instead of configure-time type size checks - Additional compiler warnings were turned on and addressed - libfstream: Add printf format specifier attribute - libfstream: guard against potential integer overflows - libfstree: guard against link count and inode number overflow - libfstree: guard against possible overflow in readlink() - libcommon: potentially un-aligned data access in LZO compressor - libsquashfs: potentially unaligned data access in meta data handling - Some format string type/signedness mismatch issues ## [1.1.1] - 2021-05-07 ### Fixed - tar2sqfs: currectly process concatenated xz streams from parallel compression - sqfs2tar: don't report an error if fsync() returns EINVAL - libsquashfs: static linking on Windows - libsquashfs: visibillity of internal mempool functions - libsquashfs: add sqfs_free() function, mainly for Windows portabillity - libsquashfs: block processor: Fix account for manually submitted blocks - Fix build failure if configured with --without-tools ## [1.1.0] - 2021-03-28 ### Added - tar2sqfs: support transparently reading stream compressed archives - sqfs2tar: support creating stream compressed archives - gensquashfs: support file globbing/filtering in the description file - Support bzip2 compression for tar - Implement directory scanning for Windows - a `glob` keyword to `gensquashfs` for `find` like globbing from a directory. ### Changed - Rewrite file I/O in the tools around an I/O stream wrapper which is used to implement the transparent compression. - Internal cleanups & restructuring, trying to improve maintainabillity and testabillity of the code. - Updated benchmarks, including benchmarks for decompression. - Drastic performance improvements of the xattr writer (#68). - Performance improvements in the thread pool block processor. - Bump zstd & xz versions for Windows binary releases. ### Fixed - libsquashfs: Allow shared read access to generated images on Windows (#79) ## [1.0.6] - 2021-08-15 ### Added - Overhaul format documentation, convert to ASCIIdoc (#88, #90) - Add an explicit license boiler plate to all the example programs (#89) - Additional example code & documentation ### Fixed - Typos and formating issues in the documentation (#86) - Fix symlink path traversal in rdsqaushfs - Add a test case for the path traversal bug - Fix libsquashfs directory writer size accounting - tar2sqfs, sqfs2tar: 4 GiB limitation for files in tarballs (#87) ## [1.0.5] - 2021-06-25 ### Fixed - libsquashfs: Allow shared read access to generated images on Windows (#79) - libsquashfs: preserve alignment flag in block processor - libsquashfs: broken block alignment in block write - libsquashfs: add sqfs_free() function, mainly for Windows portabillity - libsquashfs: static linking on Windows - Some documentation clarifications and typo fixes - libfstree: guard against link count and inode number overflow - libcommon: potentially un-aligned data access in LZO compressor - libsquashfs: potentially unaligned data access in meta data handling - More compiler warning flags turned on and partially fixed ## [1.0.4] - 2021-01-23 ### Fixed - typos in documentation - libsquashfs: xattr_writer: fail if allocation fails - remove broken normalization of backslashes (#77, #78) - congestion in block processor when processing lots of tiny fragments - exact byte-for-byte matching during block deduplication - exact byte-for-byte matching during fragment deduplication ### Added - new, extensible block processor create function required for deduplication fixes ## [1.0.3] - 2020-11-05 ### Fixed - tar2sqfs: if --root-becomes is used, also retarget links (#70) - tar2sqfs: when skipping non-prefixed files, also skip contents (#70) - tar2sqfs: null-pointer dereference if a file with a broken path is skipped - rdsquashfs: correctly escape of the input file paths if prefixed ### Added - Packing scripts for a various GNU/Linux distributions (#65, #67) ## [1.0.2] - 2020-09-03 ### Fixed - Fix block processor single block with don't fragment flag bug (#63) - Fix libtar treatment of link targets that fill the header field (#64) - Fix tree node path generation for detached sub trees (#66) - Fix rdsquashfs unpack under Windows if a directory exists - Clarify copyright status of documentation & build system ### Added - Support for GNU tar sparse format 1.0. - A spec file to build RPM packages (#65). - A `--stat` option to rdsquashfs ## [1.0.1] - 2020-08-03 ### Fixed - Xattr reader: re-read the header after seaking to an OOL value (#60) - Documentation: mention the file name limit imposed by the kernel - Documentation: fix wrong magic value and stray tabs in format.txt (#61) - Fix block bounds checking in libsquashfs data reader (#58) - Fix build issue caused by demo code that didn't have (#57, #59) ## [1.0.0] - 2020-06-13 ### Added - Expose more fine grained control values & flags on the XZ and LZMA compressors - A flag for the `libsquashfs` block processor to micro manage block hashing and sparse block detection. - A raw block submission function. - A user pointer that can be forwarded to the block writer. ### Changed - `sqfsdiff` doesn't abort if it fails to read the compressor options - in `libsquashfs`, turn the block writer into an interface with a default implementation and remove the statistics and hooks. - Block processor can function without fragment table and without inode pointer - Strictly enfore min/max dictionary size in XZ & LZMA compressors - Make compression level a generic compression option in `libsquashfs` - More `libsquashfs` API and internal cleanups - Bump the major version number of libsquashfs ### Fixed - Propperly set the last block flag if fragments are disabled - Compilation on GCC4 and below - libtar: size computation of PAX line length (#50) - Semantics of the super block deduplication - Actually set the ZSTD compression level to something greater than 0 - Only add Selinux compile flags if WITH_SELINUX is set. Fixes Mingw cross build on Fedora. - Make `rdsquashfs` describe mode terminate with an error message if an illegal filename is encountered (#52). - Don't include alloca.h on systems that don't provide that header. ## [0.9.1] - 2020-05-03 ### Added - Options to `gensquashfs` for overriding the ownership if packing a directory. ### Changed - Various internal code and build system related cleanups. - Various performance improvements for `gensquashfs` and `tar2sqfs`. - Make `tar2sqfs` compeltely ignore PAX global headers, even if `--no-skip` is set (#45). - Make `gensquashfs` fail if extra arguments are passed, similar to `tar2sqfs`. ### Fixed - Missing assert header inclusion if built with `--without-lzo` (#41) - sqfs2tar: also emit trailing slashes for empty directories (#42) - Illegal implicit cast in public `libsquashfs` headers if used from C++. - If tar2sqfs or gensquashfs fails, delete the output file. - Make sure test cases also work if built with NDEBUG defined. - Add missing `--with-gzip` configure option. - In `tar2sqfs`, actually apply the `--no-tail-packing` if set. - Potential nondeterministic order of extended attributes in `gensquashfs` if packing a directory. - Manpages lagging behind. ## [0.9.0] - 2020-03-30 ### Added - Support parsing [device] block size argument with SI suffix. - Add a write-up on the on-disk format. - A couple demo programs that make use of `libsquashfs`. - Add statistics counters to the block writer and processor. - A compressor interface function to retrieve its configuration. ### Changed - For better compatibility, sqfs2tar appends `/` to directory names. (#37) - Extra data in sqfs_inode_generic_t no longer handled via payload pointers. - Add a currently unsued flag field to the id table create function. - Split off fragment table handling from data reader and writer into dedicated fragment table data type. - Split data writer up into a block writer and a block processor. - All abstract data types inhert from an sqfs_object_t with common functionality. - Lots of performance improvements. - Make block processor sync and flush seperate operations. - Combined the various payload size values in generic inode structure. - Change block processor API to manage creation/resizing of file inodes. - A "do not deduplicate flag" for the block processing pipeline. - Lots of API cleanups. ### Fixed - Include sys/sysmacros.h on any GNU libc platform. - Directory index accounting. - Memory leak in hard link detection code. - Broken iteration over directory children in sqfsdiff. - Data reader returning -1 instead of an error code. - Size accounting for sparse files in tar parsing code. - Stricter verification of the compressor configuration. - Broken builds with older liblz4 and zstd versions (e.g. on Ubuntu Xenial). - Various build issues on MacOS. ### Removed - A number of hook callbacks from the block writer. - sqfs_block_t from the public API. ## [0.8.0] - 2019-12-30 ### Added - Experimental Windows port using MinGW cross compilation toolchain. - Port to BSD systems. - Explicit argument invalid error code in `libsquashfs`. - A `--root-becomes` option to `tar2sqfs` and `sqfs2tar`. - A `--no-tail-packing` option to `tar2sqfs` and `sqfs2tar`. - CHANGELOG.md now references GitHub issue numbers. - Simple integration and regression test suit. - Simple creation of an NFS export table throught `libsquashfs`. - Support for non-directory hard links in `gensquashfs`, `tar2sqfs` and `sqfs2tar`. ### Changed - Return propper error code from `sqfs_get_xattr_prefix_id`. - Return propper error code from `sqfs_compressor_id_from_name`. - Combined error and value return from `sqfs_compressor_id_from_name`. - Return error code from compressor functions if input is larger than 2G. - Lots of cleanups, including the build system. ### Removed - Entirely redundant `sqfs_has_xattr` function from `libsquashfs`. ### Fixed - LZO compressor moved out of libsquashfs to avoid licensing problems. - Make overriding configure variables for LZO library actually work. - Do not follow symlinks when reading xattrs from input files. - Ignore directory entry named `./` in tar2sqfs. (#22, #31) - Reject empty string as directory name in libsquashfs. (#22, #31) - Fix memory leak in tar2sqfs if entries are skipped. - Fix tar_fuzz error check after seek. - Fix the `fstree_init` test to account for defaults from `SOURCE_DATE_EPOCH`. - Honor the no_xattr flag when generating SquashFS images. - Block size check in `sqfs_super_init`. (#29) - Fix pthread block processor interfering with application signal handling. - Added pthread flags to the programs using libsquashfs. - Fix "buffer too small" being treated as fatal error by the zstd compressor. - Fix out of bounds write in the LZO compressor. - Fix queue accounting in the compressor thread pool. (#29) - Fix name of `libsquashfs` pkg-config file. - Reading of binary (i.e. non-textual) xattr values from tar files. (#32) - A bug in parsing the GNU.sparse.name PAX attribute from tar files. - sqfsdiff: recurse into directories that are only in one image. ## [0.7.0] - 2019-10-08 ### Added - LGPLv3 licensed, shared library `libsquashfs.so` containing all the SquashFS related logic. - Sanitized, public headers and pkg-config file for libsquashfs. - Doxygen reference manual for libsquashfs. - Legacy LZMA compression support. (#17) - User configurable queue backlog for tar2sqfs and gensquashfs. ### Changed - Make sqfsdiff continue comparing even if the types are different, but compatible (e.g. extended directory vs basic directory). - Try to determine the number of available CPU cores and use the maximum by default. - Start numbering inodes at 1, instead of 2. - Only store permission bits in inodes, the reader reconstructs them from the inode type. - Make "--keep-time" the default for tar2sqfs and use flag to disable it. ### Fixed - An off-by-one error in the directory packing code. (#18) - Typo in configure fallback path searching for LZO library. - Typo that caused LZMA2 VLI filters to not be used at all. - Possible out-of-bounds access in LZO compressor constructor. - Inverted logic in sqfs2tar extended attributes processing. ### Removed - Comparisong with directory from sqfsdiff. ## [0.6.1] - 2019-08-27 ### Added - Add a change log - Add test programs for fuzzing tar and gensquashfs file format parsers ### Fixed - Harden against integer overflows when parsing squashfs images (#13, #14) - Test against format limits when parsing directory entries (#12) - More thorough bounds checking when reading metadata blocks (#13, #14, #15) ## [0.6.0] - 2019-08-22 ### Added - New utility `sqfsdiff` that can compare squashfs images - rdsquashfs can now dump extended attributes for an inode (#2) - rdsquashfs can now optionally set xattrs on unpacked files (#2) - rdsquashfs can now optionally restore timestamps on unpacked files (#2) - sqfs2tar can now optionally copy xattrs over to the resulting tarball (#2) - gensquashfs can now optionally read xattrs from input files (#2) - gensquashfs now has a --one-file-system option - tar2sqfs and gensquashfs now output some simple statistics - Full fragment and data block deduplication - Support for SOURCE_DATE_EPOCH environment variable - Optimized, faster file unpacking order - Faster, pthread based, parallel block compressor ### Fixed - Return the correct value from data_reader_create - Fix free() of stack pointer in id_table_read error path - Fix missing initialization of file fragment fields - Fix xattr OOL position - Fix super block flags: clear "no xattr" flag when writing xattrs - Fix xattr writer size accounting - Fix explicit NULL dereference in deserialize_fstree failure path - Fix tar header error reporting on 32 bit systems - Make sure file listing generated by rdsquashfs -d is properly escaped - Fix functions with side effect being used inside asserts - Fix zero padding of extracted data blocks - Fix forward seek when unpacking sparse files - Fix wrong argument type for gensquashfs --keep-time - Fix memory leak in dir-scan error code path - Fix chmod of symlinks in restore_fstree - Add proper copyright headers to all source files ### Changed - Various internal data structure and code cleanups - Overhaul README and convert it to markdown ## [0.5.0] - 2019-07-28 ### Added - Support for NFS export - Support for xattr value deduplication - Flag in packers to optionally keep the original time stamps - Largefile support - Implement simple, fork() based parallel unpacking in rdsquashfs ### Fixed - Remove unfriendly words about squashfs-tools from README (#10) - Propper error message for ZSTD compressor - Correct copy-and-paste mistake in the build system - Make sure xattr string table is propperly initialized - More lenient tar checksum verification - Fix xattr unit test - Fix possible leak in tar2sqfs if writing xattrs fails - Fix corner cases in directory list parsing - Fix processing of tar mtime on 32 bit systems (#8) - libfstree: fix signed/unsigned comparisons - Fix fragment reader out of bounds read when loading table - Fix checks of super block block size - Fix potential resource leak in deserialize_tree - Enforce reasonable upper and low bounds on the size of tar headers - Make sure target in fstree_mknode is always set when creating a symlink - Use safer string copy function to fill tar header ## [0.4.2] - 2019-07-17 ### Fixed - Sanity check id table size read from super block - Various bug fixes through Coverity scan - Fix dirindex writing for ext dir inode - fstree: mknode: initialize fragment data, add extra blocksize slot - Fix directory index creation - Support for reading files without fragments - Support for spaces in filenames - Eleminate use of temporary file ## [0.4.1] - 2019-07-07 ### Fixed - read_inode: determine mode bits from inode type - Actually encode/decode directory inode difference as signed - Fix regression in fstree_from_file device node format - Always initialize gensquashfs defaults option ## [0.4.0] - 2019-07-04 ### Added - no-skip option in tar2sqfs and sqfs2tar - Option for sqfs2tar to extract only some subdirectories - Support for xattr extensions in tar parser - Support repacking xattrs from tarball into squashfs filesystem ### Fixed - Null-pointer dereference in restore_unpack - Memory leak in gzip compressor - Stack pointer free() in fstree_from_dir - Use of uninitialized xattr structure - Initialize return status in fstree_relabel_selinux - Make pax header parser bail if parsing a number fails - Double free in GNU tar sparse file parser - Never used overflow error message in fstree_from_file - Unused variable assignment in tar header writer - Make sure fragment and id tables are initialized - Directory index offset calculation - Missing htole32 transformations - Don't blindly strcpy() the tar header name - Typos in README - Composition order of prefix + name for ustar - Actually check return value when writing xattrs - Possible out of bounds read in libcompress.a - Check block_log range before deriving block size from it - tar2sqfs: check for invalid file names first ### Changed - Tar writer: Use more widely supported GNU tar extensions instead of PAX - Simplify deduction logic for squashfs inode type ## [0.3.0] - 2019-06-30 ### Added - Add utility to turn a squashfs image into a POSIX tar archvie - Add utility to turn a POSIX/PAX tar archive into a squashfs image - Add unit tests - Add support for packing sparse files - Add support for unpacking sparse files as sparse files ### Fixed - Actually update permissions in fstree add by path - Always set permissions on symlinks to 0777 - gensquashfs: Fix typo in help text - Fix inode fragment & sparse counter initialization - Ommit fragment table if there really are no fragments ### Changed - Lots of internal cleanups and restructuring ## [0.2.0] - 2019-06-13 ### Fixed - Make empty directories with xattrs work - Flush the last, unfinished fragment block ### Changed - Add pushd/popd utility functions and replace directory traversal code - Lots of internal cleanups - Use abstractions for many operations and move them to support libraries ## [0.1.0] - 2019-06-08 ### Added - Salvage protoype from Pygos project and turn it into generic squashfs packer - Add unpacker ### Changed - Insert abstraction layers and split generic code off into support libraries [unreleased]: https://github.com/AgentD/squashfs-tools-ng/compare/v1.2.0...HEAD [1.2.0]: https://github.com/AgentD/squashfs-tools-ng/compare/v1.1.4...v1.2.0 [1.1.4]: https://github.com/AgentD/squashfs-tools-ng/compare/v1.1.3...v1.1.4 [1.1.3]: https://github.com/AgentD/squashfs-tools-ng/compare/v1.1.2...v1.1.3 [1.1.2]: https://github.com/AgentD/squashfs-tools-ng/compare/v1.1.1...v1.1.2 [1.1.1]: https://github.com/AgentD/squashfs-tools-ng/compare/v1.1.0...v1.1.1 [1.1.0]: https://github.com/AgentD/squashfs-tools-ng/compare/v1.0.2...v1.1.0 [1.0.6]: https://github.com/AgentD/squashfs-tools-ng/compare/v1.0.5...v1.0.6 [1.0.5]: https://github.com/AgentD/squashfs-tools-ng/compare/v1.0.4...v1.0.5 [1.0.4]: https://github.com/AgentD/squashfs-tools-ng/compare/v1.0.3...v1.0.4 [1.0.3]: https://github.com/AgentD/squashfs-tools-ng/compare/v1.0.2...v1.0.3 [1.0.2]: https://github.com/AgentD/squashfs-tools-ng/compare/v1.0.1...v1.0.2 [1.0.1]: https://github.com/AgentD/squashfs-tools-ng/compare/v1.0.0...v1.0.1 [1.0.0]: https://github.com/AgentD/squashfs-tools-ng/compare/v0.9.1...v1.0.0 [0.9.1]: https://github.com/AgentD/squashfs-tools-ng/compare/v0.9...v0.9.1 [0.9.0]: https://github.com/AgentD/squashfs-tools-ng/compare/v0.8...v0.9 [0.8.0]: https://github.com/AgentD/squashfs-tools-ng/compare/v0.7...v0.8 [0.7.0]: https://github.com/AgentD/squashfs-tools-ng/compare/v0.6.1...v0.7 [0.6.1]: https://github.com/AgentD/squashfs-tools-ng/compare/v0.6...v0.6.1 [0.6.0]: https://github.com/AgentD/squashfs-tools-ng/compare/v0.5...v0.6 [0.5.0]: https://github.com/AgentD/squashfs-tools-ng/compare/v0.4...v0.5 [0.4.2]: https://github.com/AgentD/squashfs-tools-ng/compare/v0.4.1...v0.4.2 [0.4.1]: https://github.com/AgentD/squashfs-tools-ng/compare/v0.4...v0.4.1 [0.4.0]: https://github.com/AgentD/squashfs-tools-ng/compare/v0.3...v0.4 [0.3.0]: https://github.com/AgentD/squashfs-tools-ng/compare/v0.2...v0.3 [0.2.0]: https://github.com/AgentD/squashfs-tools-ng/compare/v0.1...v0.2 [0.1.0]: https://github.com/AgentD/squashfs-tools-ng/releases/tag/v0.1