aboutsummaryrefslogtreecommitdiff
path: root/include
AgeCommit message (Collapse)Author
2023-04-17Internalize fstree_insert_sortedDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2023-04-17Implement a version of the directory iterator for UnixDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2023-04-12Split out Windows directory iteration code to a dir_iterator_t typeDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2023-04-06libfstree: simplify hard link resolution codeDavid Oberhollenzer
We do not allow hard links to directories, so we can toss the special case handling code for that. The visited mechanism was pointless anyway, because we don't even descend down hard links in the recursive tree handling functions. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2023-04-06Cleanup: libfstree: move all the hardlink related code to hardlink.cDavid Oberhollenzer
For some reason, the recursive hardlink resolution ended up in post process, calling into the non recrusive one in hardlink.c that wasn't used elsewhere. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2023-04-06libfstree: Move left over dirscan flags to gensquashfsDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2023-04-04Move scan callback from libfstree to gensquashfsDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2023-04-02Move fstree CLI code to libcommonDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2023-04-01libsquashfs: remove the default block writer alignment featureDavid Oberhollenzer
The idea of the block align feature was to allow micro-managing that some files are forcefully aligned to 1k/4k ("device block") boundaries, hoping to improve access time at the cost of data density. The feature was not exposed in the tools for a long time and eventuall added to the sort file. Measurement and experimentation showed, that it in fact worsened the read performance on a test system with an old micro SD card as the bottle neck. The feature is removed, and if needed, can be brought back simply by wrapping/sub-classing the default block writer, if need be.. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2023-02-20Remove ostream_printfDavid Oberhollenzer
By cobbling together the xattr lines manually in libtar, the need (and thus the function itself) are removed. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2023-02-15Fix broken C++ guard in rbtree.hDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2023-02-08libtar: remove need for skip_padding functionDavid Oberhollenzer
In the istream implementation, automatically skip the padding when we reach end-of-file. Also skip file AND padding when we destroy the object. Replace the remaining instances with a simple istream_skip instead and remove the wrapper from libtar. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2023-02-08libtar: Add an istream_t implementationDavid Oberhollenzer
The tar_istream_t reads the data from a tar file, having been given the header, and synthesizes zero bytes for sparse regions. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2023-02-04libtar: internalize the declaration of read_octalDavid Oberhollenzer
Use read_number in the places that remain. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2023-02-04libtar: simplfy parsing of old GNU sparse formatDavid Oberhollenzer
There was some code duplication for extracting the sparse entry fields from the start record and the subsequent extended record. This commit introduces a data structure for both and unifies the parsing code paths. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2023-02-04libtar: some minor cleanupsDavid Oberhollenzer
- Use is_memory_zero from libutil - Move checksum update function to tar writer code - Move checksum verify function to tar reader code - Only export the function to compute the checksum Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2023-01-26libio: move simple wrappers back into header as inline functionsDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2023-01-19Split stream compression out of libioDavid Oberhollenzer
Move it to a separate libxfrm library, where it can be independently tested as well. The bulk of the new code is also mainly test cases for the compressors. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2023-01-19Implement rudimentary reference counting for sqfs_object_tDavid Oberhollenzer
Implement grab/drop functions to increase/decrease reference count and destroy the object if the count drops to 0. Make sure that all objects that maintain internal references actually grab that reference, duplicate it in the copy function, drop it in the destroy handler. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2023-01-19Add a helper function to initialize libsquashfs objectsDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2023-01-19Drop backwards compatibillity note for sqfs_freeDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2023-01-19libsqfs: add a threshold for extended directory inodes with indexDavid Oberhollenzer
mksquashfs generates extended inodes if a directory contains 256 entries. libsquashfs so far only generated extended inodes if there is no other way to encode it. Mimic the behaviour of mksquashfs by adding a threshold. For this to work, the "sqfs_inode_set_xattr_index" function has to be changed to not immediately try to demote inodes to basic types. The fstree serialization is modified to do that itself if the index is 0xFFFFFFFF and the target is not a directory inode. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-11-22Move gensquashfs specific code from libfstree to gensquashfsDavid Oberhollenzer
The "from dir" and from "from file" code, as well as the "sort file" code is specific to gensquashfs, so move them there and the test cases as well. The medium term idea is to reduce libfstree to a stub, merge it into the generic writer and ultimately hoist that into libsquashfs. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-11-18Add a single, central base64 decoderDavid Oberhollenzer
Similar to the hex blob decoder, we need this once for tar and once for the filemap xattr parser. Simply add a single, central implementation to libutil, with a simple unit test, and then use it in both libtar and gensquashfs. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-11-18Add a single, central hex blob decoderDavid Oberhollenzer
Since we need it twice (once for tar, once for the filemap xattr parser), add a single, central implementation to libutil, add a unit test for that implementation and then use it in both libtar and gensquashfs. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-09-20block writer: move block comaprison to utility functionDavid Oberhollenzer
Slightly modify the byte-for-byte comparison function to compare an arbitrary range in a file and move it to libutil. Instead of calling it for each block in the block writer, simply let it check an entire range in the block writer and compute the range position/size of the reference ahead, before looking for potential matches. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-07-08Cleanup: Get rid of libfstree "internal.h" headerDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-07-08Cleanup: move source date epoch code back to libutilDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-07-08Cleanup: move filename_sane & canonicalize_path functions to libutilDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-07-08Make sqfs_tree_node_get_path more robustDavid Oberhollenzer
Test against various invariants: - Every non-root node must have a name - The root node muts not have a name - The name must not be ".." or "." - The name must not contain '/' - The loop that chases parent pointers must terminate, i.e. we must never reach the starting state again (link loop). Furthermore, make sure the sum of all path components plus separators does not overflow. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-07-08Move sqfs_tree_node_get_path to libsquashfsDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-07-08Cleanup: move inode_stat to sqfs2tarDavid Oberhollenzer
After trying removing many instances of `struct stat`, there is only one user of inode_stat left, so move the function there. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-07-08Cleanup: move mkdir_p from libcommon to libutilDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-07-08Cleanup: move test.h to libutilDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-07-08Cleanup: split libtar header, move to sub directoryDavid Oberhollenzer
Some of the on-disk format internals are moved to a separate header and some of the stuff from internal.h is moved to that format header. C++ guards are added in addtion. Everything PAX related is moved to pax_header.c, some internal functions are marked as static. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-07-08Cleanup: rename libfstream to libio, split headersDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-07-08Cleanup: move libutil headers to sub directoryDavid Oberhollenzer
Move all the libutil stuff from the toplevel include/ to a util/ sub directory and fix up the includes that make use of them. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-07-08Fix: libfstree: actually use a full 32 bit hard link counterDavid Oberhollenzer
The squashfs on-disk format uses 32 bit link counters, but the fstree used 16 bit ones. Because the link count also includes child nodes, this artificially limited the number of entries in a directory to ~64k files. This patch removes the limit by switching libfstree to 32 bit counters. Reported-by: Marvin Renich <mrvn@renich.org> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-06-03Typo fixDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-06-02Cleanup: libsqfs: sqfs_dir_reader_find_by_pathDavid Oberhollenzer
Split out several repated patterns into helper functions and move the rest of the code back into dir_reader.c Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-06-02Cleanup: libsqfs: move directory iteration out of the directory readerDavid Oberhollenzer
Add a simple directory state object to the meta data reader and use that to iterate directory entries. The code for reading the directory listing is movde to readdir.c Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-04-09Add support for '.' and '..' entries in sqfs_dir_reader_tDavid Oberhollenzer
Two flags are added to the dir reader API, one for the create function that the dir reader should report those entries and one to the open function to suppress that if it was enabled. To implement the feature, a mapping of visited directory inodes is maintained internally, that mapps inode numbers to inode references. When opening a directory, state is maintained to generate the fake entries for '.' and '..'. Since all the other functions are based on the open/read/rewind API, no alterations need to be made. The tree scan function is modified, to use the suppress flag, so it does not accidentally catch those entries. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-03-30Cleanup: remove struct stat from libtarDavid Oberhollenzer
The idea was originally to use struct stat in the libfstree code, so we can simply hose data read from a directory into the fstree_t. The struct was then also used with libtar, for simpler interoperation, but it turned out to introduce a lot of platform quirks and causes more trouble than it's worth. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-03-30Add a rudimentary unit test for sort filesDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-03-30Add sort-file implementationDavid Oberhollenzer
A `flags` field and `priority` are added to all file information structs. A news fstree function is introduced for parsing a "sort-file". Each line in the file is space separated, and has the following format: priority [flags] filename Priority is a 64 bit number, flags are optional and filename can be put in quotes if it is supposed to start or end with spaces. Single line comments can be used. The flags can be used to set block-processor flags (e.g. don't fragment, or don't compress), as well as instructing the parser to use file globbing to match the filename. After parsing the file, the list of file info structure is sorted according to the priority (default is 0) using a stable sort algorithm. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-03-30sqfs_dir_tree_destroy/sqfs_destroy: allow NULL inputLuca Boccassi
Many library destructor functions (like free()) allow a NULL pointer as input, and do nothing in that case. This allows easier cleanup patterns: initialize pointers to NULL and then always pass them to the destroyer functions, no need for verbose goto/if-else patterns. Signed-off-by: Luca Boccassi <luca.boccassi@microsoft.com>
2022-03-10Windows: redirect standard I/O and convert text to UTF-16David Oberhollenzer
Preprocessor magic is used to redirect putc/fputc/fputs/printf/fprintf to custom implementations. The custom implementations try to figure out if we are printing to the console and, if so, convert the resulting strings to UTF-16 and print them through ConsoleWriteW. If the output is redirected to a file or a pipe, the original (presummed) UTF-8 is kept. Simply setting the console output codepage to UTF-8 does not work, because the standard I/O facilities of MSVCRT either does not support unicode (in non-wchar mode), or has half-broken support through fputs, which can still break up multi-byte sequences through its internal buffering. Likewise, changing the codepage and using ConsoleWriteA, or trying to use fputws did not work in a test VM either. This approach is the one that worked most consistently among the ones tried, but also has problems. E.g. it breaks when setting the codepage to UTF-8 manually (using `chcp 65001`). Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-12-05Fix: consistently use the widechar file API on WindowsDavid Oberhollenzer
When opening files on windows, use the widechar versions and convert from (assumed) UTF-8 to UTF-16 as needed. Since the broken, code-page-random API may acutall be intended in some use cases, leave that option in through an additional flag. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-12-05Add a wrapper for the main function on WindowsDavid Oberhollenzer
A macro and forward declaration are added to compat.h that rename the main() function programs using compat.h into sqfs_tools_main. An actual main() function is added to libcompat.a, that uses the shell API to get the UTF-16 command line arguments, convert them to UTF-8 and call sqfs_tools_main. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-07-21Fix libsquashfs directory writer size accountingDavid Oberhollenzer
The squashfs readdir() implementation in the Linux kernel returns non-existing "." and ".." entries for offsets 0 and 1, and after that reads from disk. For convenience, it was decided to store an off-by-3 value on disk instead of doing complex primary school math to adjust for this. This didn't show up until now, because the kernel implementation trusts the value from the directory header more than the actual size in the inode and happily reads 3 more than the inode would allow it to. This only showed up with 7-zip which subtracts 3 from the size and expects the result to be exact and bails if the directory headers suggest otherwise. And yes, I did consider making a "Holy Hand Granade of Antioch" reference, but consciously decided not to. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>