aboutsummaryrefslogtreecommitdiff
path: root/lib
AgeCommit message (Collapse)Author
2020-12-06libsqfs: implement exact matching in the default block writer.David Oberhollenzer
Instead of comparing (compresed, disk-size, checksum) tuples to find block matches, do an exact, byte-for-byte comparison of the data stored on disk to avoid the possibility of a spurious colision. Since this is the desired behaviour, make it the default, optionally overrideable through a flag. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-11-13Fix: include MacOS compat byteswab definitions in LZO compressorDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-11-07Fix: libsquashfs: xattr_writer: return NULL if calloc failsDavid Oberhollenzer
Instead of dereferencing the NULL pointer and crashing. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-11-07Cleanup: libcommon: try to split up some of the larger filesDavid Oberhollenzer
The sqsf_writer is moved to a separate header, as well as the compressor related stuff. The statistics function is moved into the writer code, as this is the only place that actually uses it. The writer code itself is split up into a hand full of file in their own subdirectory. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-09-29Update tar2sqfs/sqfs2tar documentationDavid Oberhollenzer
Modify the man pages and help texts to mention compression support, make the help option display a list of supported compressors. Also clarify the squashfs compressor list in the help text to state that the list shows block compressors supported for SquashFS compression. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-09-29Replace file/getline usage with istreamDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-09-29Add a more usefull getline-like function to libfstreamDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-09-29libfstream: Rework unix sparse file writingDavid Oberhollenzer
Instead of immediately seeking forward, record the sparse bytes "written" and then do the seek the next time we write data, or do an ftruncate if we do a sync. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-09-17Add a check whether libzstd supports stream compressionDavid Oberhollenzer
Streaming compression was added fairly recently to zstd (and then the API was changed a few times). Rather than rely on libzstd versioning macros, this commit adds an m4 script to test at configure time if the enums/functions we need are available by trying to compile a small sample. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-09-17Fix: add missing compiler/linker flagsDavid Oberhollenzer
If tar2sqfs and sqfs2tar want to use comperssion via libfstream, they need to link against the compressor libraries. Also, libfstream may need the apropriate cflags to access the headers on some platforms. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-09-16Add bzip2 stream compression supportDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-09-16Add zstd stream compressor implementation to libfstreamDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-09-16Re-import crc32 code to builtin zlib copyDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-09-16Implement istream decompression supportDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-09-16Remodel libtar/tar2sqfs to read data from an istream_tDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-09-16Implement ostream compressor wrapperDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-09-16Remodel file extraction tools to use libfstreamDavid Oberhollenzer
This commit rewrites the libtar write paths to use libfstream insead of a FILE pointer. Also, the libcommon file extraction function is remodeled to use libfstream. In accordance, rdsquashfs, sqfs2tar and sqfsdiff have some minor adjustments made to work with the ported libtar and libcommon. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-09-16Implement a data writer based output streamDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-09-16Add stream I/O abstraction libraryDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-09-03Fix integer bounds checking in GNU tar sparse format 1.0 parserDavid Oberhollenzer
- Make sure the file actually has that many records before trying to read one and fail if not. - Use the helper macros for size_t overflow checking instead of assuming size_t == uint64_t. - Impose a "reasonable" upper bound on the number of data segments and insist that there is at least one entry. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-09-02Fix nonexistant gnu tar sparse format 1.0 supportDavid Oberhollenzer
Contrary to previous claims, support for the GNU tar sparse format 1.0 was missing entirely (the newest of their 3 different sparse mapping formats). This oversight wasn't caught, because the unit test was compiling the wrong source file and tar2sqfs had no problem processing the test file because it is still a valid POSIX-ish tar archive (but the sparse part was missing and the mapping embedded in the file). Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-08-26Fix tree node path generation for detached sub treesDavid Oberhollenzer
The function sqfs_tree_node_get_path is used in several places within rdsquashfs to produce a path for a tree node, either when describing the file system, or when unpacking it. Unpacking can be done on sub-trees as well as the entire tree, in which case the root of the sub-tree has its parent pointer removed, so the full path terminates at the new root. This works with directories, since they receive special case handling anyway, but fails if the sub-tree to unpack is only a single file because the sqfs_tree_node_get_path function assumes that we are at the tree root and returns "/" as a path, which gets normalized to "". This commit adds a workaround to the function to simply use the nodes name (if available) in that case instead. The describe case in rdsquashfs is unaffacted, since it always starts at the root. Likewise, the sqfs2tar case should also be unaffacted, since it already employs special case handling for the [sub] tree root node. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-08-16Fix libtar treatment of link targets that fill the header fieldDavid Oberhollenzer
The tar header has a 100 byte field for symlink and hard link targets. If the target is longer than 100 bytes, an extension header has to be used. However, it is perfectly valid to fill all 100 bytes to the brim without adding a null terminator. In case of a symlink, this can result in garbage link targets, while for hard links it results in an immediate error since the target cannot be resolved later on. This commit attempts to fix the problem by replacing the strdup of the link target with an strndup that copies at most the size of the target header field. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-08-12Fix block processor single block with don't fragment flag bugDavid Oberhollenzer
This commit fixes a bug where the block processor state machine would not add the "last block" flag if there is only one not entirely filled block and the "don't fragment" flag is set. If the flag isn't set, the inode start block position is not updated and points to the beginning of the image instead. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-08-04Cleanup: move zlib/lz4 code from lib/sqfs/comp/ to lib/David Oberhollenzer
The source code of a modified liblz4 and zlib are included with the option to compile them into libsquashfs if they are not available on the system. So far, the source code was included directly in the compressor sub directory within libsqsuashfs. This commit moves the libraries out into the lib directory. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-07-29Fix: xattr reader: read the header after seaking to an OOL valueDavid Oberhollenzer
If an xattr value is stored OOL, the value actually holds an 8 byte reference to another, previously stored value. This reference points to the header that we need to read to know the actual size of the value before reading it, not the value itself, so after reading the reference and seeking to it, the xattr reader needs to read the actual header. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-06-20Fix block bounds checking in libsquashfs data readerDavid Oberhollenzer
Instead of doing the fragile size comparison in both loops, simply bail from the function if offset is out of bounds, clamp the size to the available range of the file and abail if it is zero. As a result, a lot of checks can be removed and the function will not return data beyond EOF. This problem occoured with files that have a short last block instead of a fragment. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-06-13Fix: don't include alloca.h on systems that don't provide this headerv1.0.0David Oberhollenzer
This commit fixes a build issue on BSD based systems, where alloca is defined in stdlib.h and there is no such thing as "alloca.h". Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-06-13Bump the so version number for libsquashfsDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-06-12Add an explicit defition for the libsquashfs so versionDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-06-11Add flags to functions that might logically be expanded in the futureDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-06-09Cleanup: mark sqfs_xattr_writer_flush writer argument as constDavid Oberhollenzer
It does not make any changes to the writer itself, so mark it as const. This also requires some similar changes to the string table. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-06-09Cleanup: remove refcount adjusting in sqfs_xattr_writer_endDavid Oberhollenzer
After finding a match, reducing the reference count of the matched elements and increasing them afterwards leaves the reference count identical, because they refere to the same entries. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-06-09Cleanup: split libsquashfs xattr writer codeDavid Oberhollenzer
This commit moves the libsquashfs xattr related code into a sub directory and splits the xattr writer code up into several files. No actual code is changed. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-06-07Fix uninitialized error code in block processor error pathDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-06-07Move the fragment deduplication hash table back into the block processorDavid Oberhollenzer
Fragment deduplication really doesn't belong into the public API of the fragment table. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-06-07block processor: add an internal common cleanup functionDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-06-04Cleanup: libcommon: use global LUTs for compressor optionsDavid Oberhollenzer
Instead of the convoluted logic, simply use a small number of LUTs that point to the available compressor flags for each compressor, the avaialble options and their ranges. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-06-04Cleanup: Pull compression level parameter out into compressor configDavid Oberhollenzer
Every compressor (except LC4) has a compression level parameter. This commit pulls the compression level field out into the generic configuration structure and applies some code clean ups as a result from this. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-06-04Strictly enfore min/max dictionary size in XZ & LZMA compressorsDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-06-04lzma compressor: support micro management optionsDavid Oberhollenzer
The LZMA compressor (through the xz-utils library) supports basically the same options for micro management as the XZ compressor. This commit enables support for those options in the compressor, the option parser and adds an option field to the configuration structure. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-06-04lzma compressor: add support for the "extreme" flagDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-06-03Cleanup: Add defines for minimum and maximum block sizeDavid Oberhollenzer
This commit adds propper defines in the super block header and removes some of the hard coded constants. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-05-30Cleanup: sqfs2tar: break up and simplify the repacking codeDavid Oberhollenzer
- Move the xattr extraction and repacking to xattr.c - Don't on-the-fly delete the tar xattr list, use the function from libtar.a - Split minor tasks into static helper functions - creating a libtar xattr struct from libsqfs xattr data - finding a hard link entry from current path and inode number Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-05-30Block processor: cleanup macros, merge windows & pthread initializationDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-05-29Block processor: merge finish & sync functionsDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-05-29Block processor: allow operation without a fragment tableDavid Oberhollenzer
This commit modifies the block processor to support operating without a fragment table. If that is the case, fragment deduplication is essentially disabled and fragment blocks aren't indexed anymore. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-05-29Block processor: Add a raw block submission functionDavid Oberhollenzer
This function allows submission of raw blocks to the block processor, completely bypassing the file API. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-05-29Block processor: add flags to manage hashing & sparse block detectionDavid Oberhollenzer
This commit adds 2 new user settable flags to the block processor: - A flag to ignore sparse blocks and treat them like normal data blocks. - A flag to disable checksum computation altogether. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-05-29Support associating a user pointer with data blocksDavid Oberhollenzer
This commit modifies the block processor to support associating a user data pointer with data blocks that it forwards to the block writer, which is modified to accept an optional user data pointer. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>