aboutsummaryrefslogtreecommitdiff
path: root/include
AgeCommit message (Collapse)Author
2022-08-19Fix: libfstree: actually use a full 32 bit hard link counterDavid Oberhollenzer
The squashfs on-disk format uses 32 bit link counters, but the fstree used 16 bit ones. Because the link count also includes child nodes, this artificially limited the number of entries in a directory to ~64k files. This patch removes the limit by switching libfstree to 32 bit counters. Reported-by: Marvin Renich <mrvn@renich.org> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-03-10Windows: redirect standard I/O and convert text to UTF-16David Oberhollenzer
Preprocessor magic is used to redirect putc/fputc/fputs/printf/fprintf to custom implementations. The custom implementations try to figure out if we are printing to the console and, if so, convert the resulting strings to UTF-16 and print them through ConsoleWriteW. If the output is redirected to a file or a pipe, the original (presummed) UTF-8 is kept. Simply setting the console output codepage to UTF-8 does not work, because the standard I/O facilities of MSVCRT either does not support unicode (in non-wchar mode), or has half-broken support through fputs, which can still break up multi-byte sequences through its internal buffering. Likewise, changing the codepage and using ConsoleWriteA, or trying to use fputws did not work in a test VM either. This approach is the one that worked most consistently among the ones tried, but also has problems. E.g. it breaks when setting the codepage to UTF-8 manually (using `chcp 65001`). Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-12-05Fix: consistently use the widechar file API on WindowsDavid Oberhollenzer
When opening files on windows, use the widechar versions and convert from (assumed) UTF-8 to UTF-16 as needed. Since the broken, code-page-random API may acutall be intended in some use cases, leave that option in through an additional flag. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-12-05Add a wrapper for the main function on WindowsDavid Oberhollenzer
A macro and forward declaration are added to compat.h that rename the main() function programs using compat.h into sqfs_tools_main. An actual main() function is added to libcompat.a, that uses the shell API to get the UTF-16 command line arguments, convert them to UTF-8 and call sqfs_tools_main. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-07-21Fix libsquashfs directory writer size accountingDavid Oberhollenzer
The squashfs readdir() implementation in the Linux kernel returns non-existing "." and ".." entries for offsets 0 and 1, and after that reads from disk. For convenience, it was decided to store an off-by-3 value on disk instead of doing complex primary school math to adjust for this. This didn't show up until now, because the kernel implementation trusts the value from the directory header more than the actual size in the inode and happily reads 3 more than the inode would allow it to. This only showed up with 7-zip which subtracts 3 from the size and expects the result to be exact and bails if the directory headers suggest otherwise. And yes, I did consider making a "Holy Hand Granade of Antioch" reference, but consciously decided not to. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-06-25Remove casual un-const casting in various placesDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-06-25libfstream: Add printf format specifier attributeDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-05-16Use *_MAX to remove need for configure-time type size checksMichael Forney
2021-04-09Fix: libsquashfs: allow static linking on WindowsDavid Oberhollenzer
If SQFS_STATIC is defined, dummy out the SQFS_API definition, so we don't try to pull stuff from a (in this case) non-existant DLL or try to export functions. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-04-08Fix: libsquashfs: visibillity of mempool functionsDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-04-08Fix: libsquashfs: add sqfs_free() functionDavid Oberhollenzer
On systems like Windows, the dynamic library and applications can easily end up being linked against different runtime libraries, so applications cannot be expected to be able to free() any malloc'd pointer that the library returns. This commit adds an sqfs_free function so the application can pass pointers back to the library to call the correct free() implementation. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-03-24Provide Musl derived fallbacks for getopt/getopt_long/getsuboptDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-03-22Cleanup the block processor file structureDavid Oberhollenzer
A cleaner separation between common code, frontend code and backend code is made. The "is this byte blob zero" function is moved out to libutil (with test case and everything) with a more optimized implementation. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-03-21Add a thread pool implementation to libutilDavid Oberhollenzer
The thread pool enforces ordering of items during dequeue similar to the already existing implementation in libsqfs. The idea is to eventually pull this functionality out of the block processor and turn it into a cleaner, separately tested module. The thread pool is implemented as an abstract interface, so we can have multiple implementations around, including the serial fallback implementation which we can then *always* test, irregardless of the compile config and run through static analysis as well. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-03-20Add libcompat fallback implementation for fnmatchDavid Oberhollenzer
This has basically been copied over from Musl and slightly modifed. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-03-07Optionally use a pool allocator for rb-tree nodesDavid Oberhollenzer
This commit restructures the rbtree code to optionally use a pool allocator for the nodes. The option is made depenend on the presence of a pre-processor flag. To the configure script is added an option to enable/disable the use of custom allocators. It makes sense to still allow the malloc/free based routes for better ASAN based instrumentation. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-03-07Implement a custom memory pool allocatorDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-03-07Rewrite the str_table to internally use the more opimized hash_tableDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-03-06Add a generic implementation of a dynamic array to libutilDavid Oberhollenzer
The intention is to get rid of all the ad-hoc array implementations in the other components and cut down code size. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-03-06Add a context pointer to the rbtree key comparisonDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-03-06Add a copy function to the rb-tree implementationDavid Oberhollenzer
If we use the rb-tree in libsquashfs objects, we need to be able top copy an entire tree as part of the object. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-03-06Cleanup: replace the void-ptr with an inode-ptr in the file tree nodeDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-02-19fstree_from_file: Implement basic file globbingDavid Oberhollenzer
The base path is passed to the fstree_from_file function and in turn to the individual callbacks. The line parsing function is modified to allow '*' as mode, uid and gid for specifically marked callbacks. A glob callback is added that internally uses the fstree_from_dir scanning functions in combination with a filter callback. Directory scanning flags are parsed from the extra arguments before interpreting it as a path fragment. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-02-19libfstree: Add a filter callback to the directory scanning functionDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-02-18libfstree: add a subdirectory scanning functionDavid Oberhollenzer
So we can scan a sub-directory within a the base directory without having to do string operations first. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-02-18fstree_from_dir: add filtering flags to skip certain inode typesDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-01-19Add a user pointer to the hash table implementationDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-01-19libcommon: Use the newer API for the block processorDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-01-19libsqfs: Add a sqfs_block_processor_create_ex functionDavid Oberhollenzer
This function creates a block processor from a structure describing it. A stub implementation for the old sqfs_block_processor_create is added that simply sets up such a struct and forwards the call. The current version of the description struct only contains the exact same parameters and a size field at the beginning. This approach is supposed to make extending the range of parameters easier without breaking ABI compatibillity. Currently already planned are: - Adding a sqfs_file_t pointer to double-check when deduplicating fragments. - When the scanning code reaches a usable state, add the abillity to pass scanned fragment data, so the block processor can be used for appending to an existing image. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-12-15libfstree: make the directory scanning code a little more genericDavid Oberhollenzer
- Instead of using the fstree root, let the caller specify it. - Add a flag to prevent recursion into sub directories. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-12-10Move fstree dirscan code back to libfstreeDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-12-06libsqfs: implement exact matching in the default block writer.David Oberhollenzer
Instead of comparing (compresed, disk-size, checksum) tuples to find block matches, do an exact, byte-for-byte comparison of the data stored on disk to avoid the possibility of a spurious colision. Since this is the desired behaviour, make it the default, optionally overrideable through a flag. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-11-07Cleanup: libcommon: try to split up some of the larger filesDavid Oberhollenzer
The sqsf_writer is moved to a separate header, as well as the compressor related stuff. The statistics function is moved into the writer code, as this is the only place that actually uses it. The writer code itself is split up into a hand full of file in their own subdirectory. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-09-29Replace file/getline usage with istreamDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-09-29Add a more usefull getline-like function to libfstreamDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-09-16Add bzip2 stream compression supportDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-09-16Add zstd stream compressor implementation to libfstreamDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-09-16Implement istream decompression supportDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-09-16Remodel libtar/tar2sqfs to read data from an istream_tDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-09-16Implement ostream compressor wrapperDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-09-16Remodel file extraction tools to use libfstreamDavid Oberhollenzer
This commit rewrites the libtar write paths to use libfstream insead of a FILE pointer. Also, the libcommon file extraction function is remodeled to use libfstream. In accordance, rdsquashfs, sqfs2tar and sqfsdiff have some minor adjustments made to work with the ported libtar and libcommon. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-09-16Implement a data writer based output streamDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-09-16Add stream I/O abstraction libraryDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-06-11Add flags to functions that might logically be expanded in the futureDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-06-09Cleanup: mark sqfs_xattr_writer_flush writer argument as constDavid Oberhollenzer
It does not make any changes to the writer itself, so mark it as const. This also requires some similar changes to the string table. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-06-07Move the fragment deduplication hash table back into the block processorDavid Oberhollenzer
Fragment deduplication really doesn't belong into the public API of the fragment table. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-06-04Cleanup: Pull compression level parameter out into compressor configDavid Oberhollenzer
Every compressor (except LC4) has a compression level parameter. This commit pulls the compression level field out into the generic configuration structure and applies some code clean ups as a result from this. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-06-04Strictly enfore min/max dictionary size in XZ & LZMA compressorsDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-06-04lzma compressor: support micro management optionsDavid Oberhollenzer
The LZMA compressor (through the xz-utils library) supports basically the same options for micro management as the XZ compressor. This commit enables support for those options in the compressor, the option parser and adds an option field to the configuration structure. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-06-04lzma compressor: add support for the "extreme" flagDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
generated by cgit v1.2.3 (git 2.39.1) at 2024-12-26 14:56:07 +0000