summaryrefslogtreecommitdiff
path: root/lib
AgeCommit message (Collapse)Author
2022-04-09Add support for '.' and '..' entries in sqfs_dir_reader_tDavid Oberhollenzer
Two flags are added to the dir reader API, one for the create function that the dir reader should report those entries and one to the open function to suppress that if it was enabled. To implement the feature, a mapping of visited directory inodes is maintained internally, that mapps inode numbers to inode references. When opening a directory, state is maintained to generate the fake entries for '.' and '..'. Since all the other functions are based on the open/read/rewind API, no alterations need to be made. The tree scan function is modified, to use the suppress flag, so it does not accidentally catch those entries. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-04-05libsqfs: move dir reader code to sub directory, add internal headerDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-03-30Cleanup: remove struct stat from libtarDavid Oberhollenzer
The idea was originally to use struct stat in the libfstree code, so we can simply hose data read from a directory into the fstree_t. The struct was then also used with libtar, for simpler interoperation, but it turned out to introduce a lot of platform quirks and causes more trouble than it's worth. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-03-30Fix: Directory insertion order on WindowsDavid Oberhollenzer
The fstree sorting code got removed recently, in favour of inserting at the correct position. The Windows directory scanning code still used it's own list insertion code instead of mknode (so it could allocate and translate the directory entry name in-place), which broke the sorting order. This issue is fixed in this commit. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-03-30Add a rudimentary unit test for sort filesDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-03-30Add sort-file implementationDavid Oberhollenzer
A `flags` field and `priority` are added to all file information structs. A news fstree function is introduced for parsing a "sort-file". Each line in the file is space separated, and has the following format: priority [flags] filename Priority is a 64 bit number, flags are optional and filename can be put in quotes if it is supposed to start or end with spaces. Single line comments can be used. The flags can be used to set block-processor flags (e.g. don't fragment, or don't compress), as well as instructing the parser to use file globbing to match the filename. After parsing the file, the list of file info structure is sorted according to the priority (default is 0) using a stable sort algorithm. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-03-30Cleanup: remove node sorting from libfstreeDavid Oberhollenzer
Always insert the tree nodes in the correct oder and remove the post-process sorting step. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-03-30Cleanup: table driven pax header parsingDavid Oberhollenzer
Instead of having a long if-else-if chain, replace the PAX header field parsing with a table driven approach. Altough it is more code, it is hopefully more readable, maintainable, extensible and it dedupliates some of the value parsing code. The GNU.sparse parsing is left as is, because it requires maintaining state. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-03-30Cleanup: pax header parsingDavid Oberhollenzer
Split the key/value pairs right in the header and terminate the key name. This way, some of the magic numbers can be eliminated. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-03-30sqfs_dir_tree_destroy/sqfs_destroy: allow NULL inputLuca Boccassi
Many library destructor functions (like free()) allow a NULL pointer as input, and do nothing in that case. This allows easier cleanup patterns: initialize pointers to NULL and then always pass them to the destroyer functions, no need for verbose goto/if-else patterns. Signed-off-by: Luca Boccassi <luca.boccassi@microsoft.com>
2022-03-30Update built-in zlib versionDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-03-11More defensive programming in mem_pool_allocateDavid Oberhollenzer
Abort and retry in situations that should logically _never_ _ever_ happen. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-03-10Fix: guard against potential overflow in file size calculationDavid Oberhollenzer
The block_count is a size_t, so on 32 bit platforms the multiplication might be truncated before the comparison with filesz. On 64 bit platforms, it could potentially also overflow the 64 bit bounds of the data type. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-03-10Fix warning if __SIZEOF_INT128__ is not definedDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-03-10Cleanup libtar mkxattr, explicitly null-terminate stringsDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-03-10Windows: redirect standard I/O and convert text to UTF-16David Oberhollenzer
Preprocessor magic is used to redirect putc/fputc/fputs/printf/fprintf to custom implementations. The custom implementations try to figure out if we are printing to the console and, if so, convert the resulting strings to UTF-16 and print them through ConsoleWriteW. If the output is redirected to a file or a pipe, the original (presummed) UTF-8 is kept. Simply setting the console output codepage to UTF-8 does not work, because the standard I/O facilities of MSVCRT either does not support unicode (in non-wchar mode), or has half-broken support through fputs, which can still break up multi-byte sequences through its internal buffering. Likewise, changing the codepage and using ConsoleWriteA, or trying to use fputws did not work in a test VM either. This approach is the one that worked most consistently among the ones tried, but also has problems. E.g. it breaks when setting the codepage to UTF-8 manually (using `chcp 65001`). Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-03-09Fix: Windows: libfstream: allocation size of stdout stream structDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-01-29Fix: libfstream: don't fail on Windows when reading from a pipeDavid Oberhollenzer
When piping the output of another program into tar2sqfs.exe, and the source program terminates, tar2sqfs.exe gets an ERROR_BROKEN_PIPE when the end is reached and it trys to pre-cache more data. This commit adds a work around, to propperly handle this as and end-of-file condition. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-12-14Fix Windows main wrapper after mingw upgradeDavid Oberhollenzer
Apparently, mingw implicitly included stdlib.h indirectly from either windows.h or shellapi.h. After an upgrade, the windows build now fails with EXIT_FAILURE being undefined. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-12-05Fix: consistently use the widechar file API on WindowsDavid Oberhollenzer
When opening files on windows, use the widechar versions and convert from (assumed) UTF-8 to UTF-16 as needed. Since the broken, code-page-random API may acutall be intended in some use cases, leave that option in through an additional flag. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-12-05Add a wrapper for the main function on WindowsDavid Oberhollenzer
A macro and forward declaration are added to compat.h that rename the main() function programs using compat.h into sqfs_tools_main. An actual main() function is added to libcompat.a, that uses the shell API to get the UTF-16 command line arguments, convert them to UTF-8 and call sqfs_tools_main. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-11-24Fix: libcommon: Correctly restore prefix path in mkdir_p on WindowsDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-11-24Fix: libfstream: Correctly handle FlushFileBuffers resturn statusDavid Oberhollenzer
The Windows port uses FlushFileBuffers in libfstream for the implmentation of the file flush method. Unlike other winapi functions, this function returns a boolean and not an error code. Previously, the error code path was executed on success, printing a rather confusing error message, that this file already exists. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-08-22Tighten bounds checks in sqfs_dir_reader_readerDavid Oberhollenzer
Use the same size check as sqfs_dir_reader_open_dir and report EOF, even if it is possible to read the header itself, but nothing beyond that. Also check if it should be possible to read an entry header before attempting and report EOF if not. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-08-22Fix half done initialization of sqfs_dir_reader_open_dirDavid Oberhollenzer
The sqfs_dir_reader_open_dir function tried to take a short-cut by returning early if the target directory is empty. However, this left some field unchanged from the previous directory. If iterating over a directory and then deciding to enter a sub-directory that happens to be empty, the directory reader will keep the settings for the current directory. After calling sqfs_dir_reader_rewind, the sub-directory will suddenly report the contents of the parent. A similar check is added to the rewind function to not track back on the meta data reader in that case. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-07-21Fix libsquashfs directory writer size accountingDavid Oberhollenzer
The squashfs readdir() implementation in the Linux kernel returns non-existing "." and ".." entries for offsets 0 and 1, and after that reads from disk. For convenience, it was decided to store an off-by-3 value on disk instead of doing complex primary school math to adjust for this. This didn't show up until now, because the kernel implementation trusts the value from the directory header more than the actual size in the inode and happily reads 3 more than the inode would allow it to. This only showed up with 7-zip which subtracts 3 from the size and expects the result to be exact and bails if the directory headers suggest otherwise. And yes, I did consider making a "Holy Hand Granade of Antioch" reference, but consciously decided not to. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-07-09Fix printf format specifiers used for generating tarballsDavid Oberhollenzer
When processing files > 4G, using "%o" truncates the result and the tarball is not readable. This should have been discovered when auto-patching the printf format specifiers, but a cast was added instead and the issue was overlooked. This commit replaces the down-cast and printf format specifiers. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-06-25libfstream: sanity check the buffer size in the gzip stream compressorDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-06-25Add default cases for every switch blockDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-06-25Remove casual un-const casting in various placesDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-06-25libutil: cleanup alignment trickery in mempoolDavid Oberhollenzer
- Store the return value of the page allocation directly into the pool variable instead of an intermediate unsigned char pointer. - Make the blob[] array the same type as the bitmap, this saves us manual alignment trickery. - Cleanup the pointer arithmetic, let the compiler do the sizeof() multiplication. - Use uintptr_t for the manual alignment of the data pointer, so we don't run into signdness problems there. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-06-25libsquashfs: get rid of potentially unaligned access and VLAsDavid Oberhollenzer
The same problem with the meta data header again, 16 bit read from a buffer: copy the buffer data into a 16 bit variable instead of casting to something potentially unaligned. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-06-25libcommon: remove potentially un-aligned access in LZO compressorDavid Oberhollenzer
When accessing the 16 bit header, don't cast the buffer pointer to an uint16_t pointer, the result might not be aligned propperly. Instead memcpy to and from an uint16_t. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-06-25libfstree: guard against possible overflow in readlink()David Oberhollenzer
*in theory*, say on a 32 bit system, we could have a 32 bit size_t and a 64 bit off_t. If the filesystem permitted this, we *could* then have a symlink with a target > 4G. Or the target is exacetely 4G, but adding a null-terminator could exceed addressable memory. This commit adds a check to guard against such an overflow and throw an error, instead of silently wrapping around. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-06-25libfstree: guard against link count and inode number overflowDavid Oberhollenzer
If the hard link counter or the inode number counter overflow the maximum representable value (for SquashFS 16 bit and 32 bit respecitively), abort with an error message. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-06-25libfstream: guard against potential integer overflowsDavid Oberhollenzer
The differen compressor libraries use differnt integer types to tally the buffer sizes. The libfstream library uses size_t, which may be bigger than the actualy types, potentially causing an overflow if trying to compress to much at once. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-06-07libsquashfs: fix: also preserve alignment flag in block processorDavid Oberhollenzer
Currently, when the block processor aggreagtes fragments into a fragment block, it applies the "don't compress" flag if any of the original framgnets has it set, but the "align to device block" flag is lost. This commit ensures that both flags get applied to the fragment block if set. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-06-07libsquashfs: fix block alignment if requestedDavid Oberhollenzer
1) If the block alignment flag is set, the padding bytes must be inserted _before_ recording the start position, otherwise the resulting image is not readable. 2) Also perform alignment if the flag is set on a fragment block. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-06-04Fix: allow concatenated Bzip2 streamsDavid Oberhollenzer
This is a followup to dd4e6ead142e58568aec89d76b0b2e867ee983f2. Basically the same problem occours with Bzip2, but it so far it wasn't possible to find a sampel that reproduces it. Unlike libxz, the libbz2 API does not support concatenated streams by itself and will choke when trying to decompress after the stream end, so this commit adds a workaround to simply initialize the decompressor on-the-fly and tear it down again when and end-of-stream is returned. The end-of-file condition is only set when there actually is no more data to read. Otherwise, the decompressor will be re-initialized in the next round. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-05-06Fix: allow concatenated xz streamsDavid Oberhollenzer
Some xz compressed tarballs (e.g. from kernel.org) are not made up of a single xz stream, but rather contain several, independendly compressed streams. In that case, the xz decompressor hits an LZMA_STREAM_END early on and reports EOF. If you are lucky, the tar reader bails (premature end-of-file). If you are unlucky, it happens exactely between two records and is interpeted as regular end-of-file. As this seems to be a normal use case for xz, it has a flag to just read across the seams and only report end-of-stream if the action is set to finish. This commit adds the flag to the initialization propperly sets the lzma_action depending on whether the underlying stream hit EOF or not. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-04-08Fix: libsquashfs: add sqfs_free() functionDavid Oberhollenzer
On systems like Windows, the dynamic library and applications can easily end up being linked against different runtime libraries, so applications cannot be expected to be able to free() any malloc'd pointer that the library returns. This commit adds an sqfs_free function so the application can pass pointers back to the library to call the correct free() implementation. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-03-30Fix: don't throw an error if fsync() returns EINVALDavid Oberhollenzer
This indicates that sync isn't possible on the underlying file descriptor (e.g. a pipe), which currently causes sqfs2tar to err if the output isn't written directly to a file. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-03-30libsqfs: block processor: Fix account for manually submitted blocksDavid Oberhollenzer
This was already in the original block processor but got dropped by accident when restructuring it. The problem manifests itself when manually submitting fragment blocks. They no longer get correct I/O queue tickets, clog up the queue and the processor eventually throws an internal error. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-03-25libfstree: allow the glob path to be emptyDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-03-25libfstree: add an assertion that root is not NULLDavid Oberhollenzer
If the path argument is "", we assume that referes to root and set the *existing* target node to the root node and skip ahead across the tree search. This leaves "name" uninitialized, which makes coverity panic, because fs->root could be NULL, going down the wrong path. Obviously, this should never, *ever* happen and there is no reasonable recovery strategy if it suddenly does, so simply add an assertion. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-03-25Fix fail branch in block processor fragment backendDavid Oberhollenzer
Only clean up the fragment if it hasn't been re-assigned to the fragment block. The NULL check is definitely wrong, because we no longer re-assign it as NULL. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-03-25libfstree: Allow / as argument for "glob" and "dir" commandsDavid Oberhollenzer
This allows putting globbed files & directories into the filesystem root, as well as explicitly setting attributes of the root directory from the file lisiting. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-03-24Provide Musl derived fallbacks for getopt/getopt_long/getsuboptDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-03-24Port the pool allocator to WindowsDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-03-24Fix block processor queue accountingDavid Oberhollenzer
Dequeuing won't work if we have a backlog of 1 or 2 and the blocks are used for internal buffering. Take that into account, similar to the sync code. Also bump the minimum backlog to 3, just to make absolutely sure we cannot run into a dequeue loop trying to allocate a block. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>