aboutsummaryrefslogtreecommitdiff
path: root/include
AgeCommit message (Collapse)Author
2022-07-08Cleanup: split libtar header, move to sub directoryDavid Oberhollenzer
Some of the on-disk format internals are moved to a separate header and some of the stuff from internal.h is moved to that format header. C++ guards are added in addtion. Everything PAX related is moved to pax_header.c, some internal functions are marked as static. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-07-08Cleanup: rename libfstream to libio, split headersDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-07-08Cleanup: move libutil headers to sub directoryDavid Oberhollenzer
Move all the libutil stuff from the toplevel include/ to a util/ sub directory and fix up the includes that make use of them. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-07-08Fix: libfstree: actually use a full 32 bit hard link counterDavid Oberhollenzer
The squashfs on-disk format uses 32 bit link counters, but the fstree used 16 bit ones. Because the link count also includes child nodes, this artificially limited the number of entries in a directory to ~64k files. This patch removes the limit by switching libfstree to 32 bit counters. Reported-by: Marvin Renich <mrvn@renich.org> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-06-03Typo fixDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-06-02Cleanup: libsqfs: sqfs_dir_reader_find_by_pathDavid Oberhollenzer
Split out several repated patterns into helper functions and move the rest of the code back into dir_reader.c Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-06-02Cleanup: libsqfs: move directory iteration out of the directory readerDavid Oberhollenzer
Add a simple directory state object to the meta data reader and use that to iterate directory entries. The code for reading the directory listing is movde to readdir.c Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-04-09Add support for '.' and '..' entries in sqfs_dir_reader_tDavid Oberhollenzer
Two flags are added to the dir reader API, one for the create function that the dir reader should report those entries and one to the open function to suppress that if it was enabled. To implement the feature, a mapping of visited directory inodes is maintained internally, that mapps inode numbers to inode references. When opening a directory, state is maintained to generate the fake entries for '.' and '..'. Since all the other functions are based on the open/read/rewind API, no alterations need to be made. The tree scan function is modified, to use the suppress flag, so it does not accidentally catch those entries. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-03-30Cleanup: remove struct stat from libtarDavid Oberhollenzer
The idea was originally to use struct stat in the libfstree code, so we can simply hose data read from a directory into the fstree_t. The struct was then also used with libtar, for simpler interoperation, but it turned out to introduce a lot of platform quirks and causes more trouble than it's worth. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-03-30Add a rudimentary unit test for sort filesDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-03-30Add sort-file implementationDavid Oberhollenzer
A `flags` field and `priority` are added to all file information structs. A news fstree function is introduced for parsing a "sort-file". Each line in the file is space separated, and has the following format: priority [flags] filename Priority is a 64 bit number, flags are optional and filename can be put in quotes if it is supposed to start or end with spaces. Single line comments can be used. The flags can be used to set block-processor flags (e.g. don't fragment, or don't compress), as well as instructing the parser to use file globbing to match the filename. After parsing the file, the list of file info structure is sorted according to the priority (default is 0) using a stable sort algorithm. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-03-30sqfs_dir_tree_destroy/sqfs_destroy: allow NULL inputLuca Boccassi
Many library destructor functions (like free()) allow a NULL pointer as input, and do nothing in that case. This allows easier cleanup patterns: initialize pointers to NULL and then always pass them to the destroyer functions, no need for verbose goto/if-else patterns. Signed-off-by: Luca Boccassi <luca.boccassi@microsoft.com>
2022-03-10Windows: redirect standard I/O and convert text to UTF-16David Oberhollenzer
Preprocessor magic is used to redirect putc/fputc/fputs/printf/fprintf to custom implementations. The custom implementations try to figure out if we are printing to the console and, if so, convert the resulting strings to UTF-16 and print them through ConsoleWriteW. If the output is redirected to a file or a pipe, the original (presummed) UTF-8 is kept. Simply setting the console output codepage to UTF-8 does not work, because the standard I/O facilities of MSVCRT either does not support unicode (in non-wchar mode), or has half-broken support through fputs, which can still break up multi-byte sequences through its internal buffering. Likewise, changing the codepage and using ConsoleWriteA, or trying to use fputws did not work in a test VM either. This approach is the one that worked most consistently among the ones tried, but also has problems. E.g. it breaks when setting the codepage to UTF-8 manually (using `chcp 65001`). Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-12-05Fix: consistently use the widechar file API on WindowsDavid Oberhollenzer
When opening files on windows, use the widechar versions and convert from (assumed) UTF-8 to UTF-16 as needed. Since the broken, code-page-random API may acutall be intended in some use cases, leave that option in through an additional flag. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-12-05Add a wrapper for the main function on WindowsDavid Oberhollenzer
A macro and forward declaration are added to compat.h that rename the main() function programs using compat.h into sqfs_tools_main. An actual main() function is added to libcompat.a, that uses the shell API to get the UTF-16 command line arguments, convert them to UTF-8 and call sqfs_tools_main. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-07-21Fix libsquashfs directory writer size accountingDavid Oberhollenzer
The squashfs readdir() implementation in the Linux kernel returns non-existing "." and ".." entries for offsets 0 and 1, and after that reads from disk. For convenience, it was decided to store an off-by-3 value on disk instead of doing complex primary school math to adjust for this. This didn't show up until now, because the kernel implementation trusts the value from the directory header more than the actual size in the inode and happily reads 3 more than the inode would allow it to. This only showed up with 7-zip which subtracts 3 from the size and expects the result to be exact and bails if the directory headers suggest otherwise. And yes, I did consider making a "Holy Hand Granade of Antioch" reference, but consciously decided not to. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-06-25Remove casual un-const casting in various placesDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-06-25libfstream: Add printf format specifier attributeDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-05-16Use *_MAX to remove need for configure-time type size checksMichael Forney
2021-04-09Fix: libsquashfs: allow static linking on WindowsDavid Oberhollenzer
If SQFS_STATIC is defined, dummy out the SQFS_API definition, so we don't try to pull stuff from a (in this case) non-existant DLL or try to export functions. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-04-08Fix: libsquashfs: visibillity of mempool functionsDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-04-08Fix: libsquashfs: add sqfs_free() functionDavid Oberhollenzer
On systems like Windows, the dynamic library and applications can easily end up being linked against different runtime libraries, so applications cannot be expected to be able to free() any malloc'd pointer that the library returns. This commit adds an sqfs_free function so the application can pass pointers back to the library to call the correct free() implementation. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-03-24Provide Musl derived fallbacks for getopt/getopt_long/getsuboptDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-03-22Cleanup the block processor file structureDavid Oberhollenzer
A cleaner separation between common code, frontend code and backend code is made. The "is this byte blob zero" function is moved out to libutil (with test case and everything) with a more optimized implementation. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-03-21Add a thread pool implementation to libutilDavid Oberhollenzer
The thread pool enforces ordering of items during dequeue similar to the already existing implementation in libsqfs. The idea is to eventually pull this functionality out of the block processor and turn it into a cleaner, separately tested module. The thread pool is implemented as an abstract interface, so we can have multiple implementations around, including the serial fallback implementation which we can then *always* test, irregardless of the compile config and run through static analysis as well. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-03-20Add libcompat fallback implementation for fnmatchDavid Oberhollenzer
This has basically been copied over from Musl and slightly modifed. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-03-07Optionally use a pool allocator for rb-tree nodesDavid Oberhollenzer
This commit restructures the rbtree code to optionally use a pool allocator for the nodes. The option is made depenend on the presence of a pre-processor flag. To the configure script is added an option to enable/disable the use of custom allocators. It makes sense to still allow the malloc/free based routes for better ASAN based instrumentation. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-03-07Implement a custom memory pool allocatorDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-03-07Rewrite the str_table to internally use the more opimized hash_tableDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-03-06Add a generic implementation of a dynamic array to libutilDavid Oberhollenzer
The intention is to get rid of all the ad-hoc array implementations in the other components and cut down code size. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-03-06Add a context pointer to the rbtree key comparisonDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-03-06Add a copy function to the rb-tree implementationDavid Oberhollenzer
If we use the rb-tree in libsquashfs objects, we need to be able top copy an entire tree as part of the object. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-03-06Cleanup: replace the void-ptr with an inode-ptr in the file tree nodeDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-02-19fstree_from_file: Implement basic file globbingDavid Oberhollenzer
The base path is passed to the fstree_from_file function and in turn to the individual callbacks. The line parsing function is modified to allow '*' as mode, uid and gid for specifically marked callbacks. A glob callback is added that internally uses the fstree_from_dir scanning functions in combination with a filter callback. Directory scanning flags are parsed from the extra arguments before interpreting it as a path fragment. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-02-19libfstree: Add a filter callback to the directory scanning functionDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-02-18libfstree: add a subdirectory scanning functionDavid Oberhollenzer
So we can scan a sub-directory within a the base directory without having to do string operations first. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-02-18fstree_from_dir: add filtering flags to skip certain inode typesDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-01-19Add a user pointer to the hash table implementationDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-01-19libcommon: Use the newer API for the block processorDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-01-19libsqfs: Add a sqfs_block_processor_create_ex functionDavid Oberhollenzer
This function creates a block processor from a structure describing it. A stub implementation for the old sqfs_block_processor_create is added that simply sets up such a struct and forwards the call. The current version of the description struct only contains the exact same parameters and a size field at the beginning. This approach is supposed to make extending the range of parameters easier without breaking ABI compatibillity. Currently already planned are: - Adding a sqfs_file_t pointer to double-check when deduplicating fragments. - When the scanning code reaches a usable state, add the abillity to pass scanned fragment data, so the block processor can be used for appending to an existing image. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-12-15libfstree: make the directory scanning code a little more genericDavid Oberhollenzer
- Instead of using the fstree root, let the caller specify it. - Add a flag to prevent recursion into sub directories. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-12-10Move fstree dirscan code back to libfstreeDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-12-06libsqfs: implement exact matching in the default block writer.David Oberhollenzer
Instead of comparing (compresed, disk-size, checksum) tuples to find block matches, do an exact, byte-for-byte comparison of the data stored on disk to avoid the possibility of a spurious colision. Since this is the desired behaviour, make it the default, optionally overrideable through a flag. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-11-07Cleanup: libcommon: try to split up some of the larger filesDavid Oberhollenzer
The sqsf_writer is moved to a separate header, as well as the compressor related stuff. The statistics function is moved into the writer code, as this is the only place that actually uses it. The writer code itself is split up into a hand full of file in their own subdirectory. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-09-29Replace file/getline usage with istreamDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-09-29Add a more usefull getline-like function to libfstreamDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-09-16Add bzip2 stream compression supportDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-09-16Add zstd stream compressor implementation to libfstreamDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-09-16Implement istream decompression supportDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-09-16Remodel libtar/tar2sqfs to read data from an istream_tDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>