aboutsummaryrefslogtreecommitdiff
path: root/include/sqfs
AgeCommit message (Collapse)Author
2023-06-18libsqfs: consistently rename the native file handle functionsDavid Oberhollenzer
Start with a common prefix "sqfs_native_file_*" Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2023-06-17Move data writer ostream into libsquashfsDavid Oberhollenzer
It is mainly a very thin wrapper on top of the block processor so far. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2023-06-16libsquashfs: merge windows & unix file I/O primitivesDavid Oberhollenzer
By adding additional close/duplicate primitives, the remaining, mostly identical istream/ostream/file code is mostly identical between Windows and Unix and be merged, mostly without stitches. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2023-06-15libsquashfs: cleanup the flag situation on istream/ostream functionsDavid Oberhollenzer
- The ostream creation functions already have flag arguments, but make them an sqfs_u32 instead of int. - Add flag arguments to the istream functions, sanitzie and forward them when opening the handle. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2023-06-15Migrate file istream/ostream from libio to libsquashfsDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2023-06-15libsquashfs: Add sqfs_open_native_file functionDavid Oberhollenzer
Having a function to interpret the flags and open a native file handle simplifies the istream/ostream/file code which shares that decoding part, particularly on windows where the character set needs to be transformed. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2023-06-15libsquashfs: Add a get_filename function to sqfs_file_tDavid Oberhollenzer
Similar to the sqfs_istream_t & sqfs_ostream_t interfaces. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2023-06-15Move sqfs_istream_t & sqfs_ostream_t into libsquashfsDavid Oberhollenzer
For now, only the interfaces and helper functions are moved, the concrete implementations remain in libio. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2023-06-15libio: replace OSTREAM_OPEN_* flags with SQFS_FILE_OPEN_* flagsDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2023-06-07libsquashfs: Add utility functions to read xattrs into listDavid Oberhollenzer
The common pattern is used in rdsquashfs and sqfs2tar, move the code to libsquashfs. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2023-06-05libsqfs: Add an xattr writer function to add a combined key-value structDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2023-06-05Move dir_entry_xattr_t from libio to libsquashfsDavid Oberhollenzer
The structure and functions are renamed to sqfs_xattr_* instead, an additional helper is added to accept an encoded xattr. Documentation and unit test are added as well. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2023-04-01libsquashfs: remove the default block writer alignment featureDavid Oberhollenzer
The idea of the block align feature was to allow micro-managing that some files are forcefully aligned to 1k/4k ("device block") boundaries, hoping to improve access time at the cost of data density. The feature was not exposed in the tools for a long time and eventuall added to the sort file. Measurement and experimentation showed, that it in fact worsened the read performance on a test system with an old micro SD card as the bottle neck. The feature is removed, and if needed, can be brought back simply by wrapping/sub-classing the default block writer, if need be.. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2023-01-19Implement rudimentary reference counting for sqfs_object_tDavid Oberhollenzer
Implement grab/drop functions to increase/decrease reference count and destroy the object if the count drops to 0. Make sure that all objects that maintain internal references actually grab that reference, duplicate it in the copy function, drop it in the destroy handler. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2023-01-19Add a helper function to initialize libsquashfs objectsDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2023-01-19Drop backwards compatibillity note for sqfs_freeDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2023-01-19libsqfs: add a threshold for extended directory inodes with indexDavid Oberhollenzer
mksquashfs generates extended inodes if a directory contains 256 entries. libsquashfs so far only generated extended inodes if there is no other way to encode it. Mimic the behaviour of mksquashfs by adding a threshold. For this to work, the "sqfs_inode_set_xattr_index" function has to be changed to not immediately try to demote inodes to basic types. The fstree serialization is modified to do that itself if the index is 0xFFFFFFFF and the target is not a directory inode. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-07-08Make sqfs_tree_node_get_path more robustDavid Oberhollenzer
Test against various invariants: - Every non-root node must have a name - The root node muts not have a name - The name must not be ".." or "." - The name must not contain '/' - The loop that chases parent pointers must terminate, i.e. we must never reach the starting state again (link loop). Furthermore, make sure the sum of all path components plus separators does not overflow. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-07-08Move sqfs_tree_node_get_path to libsquashfsDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-06-02Cleanup: libsqfs: move directory iteration out of the directory readerDavid Oberhollenzer
Add a simple directory state object to the meta data reader and use that to iterate directory entries. The code for reading the directory listing is movde to readdir.c Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-04-09Add support for '.' and '..' entries in sqfs_dir_reader_tDavid Oberhollenzer
Two flags are added to the dir reader API, one for the create function that the dir reader should report those entries and one to the open function to suppress that if it was enabled. To implement the feature, a mapping of visited directory inodes is maintained internally, that mapps inode numbers to inode references. When opening a directory, state is maintained to generate the fake entries for '.' and '..'. Since all the other functions are based on the open/read/rewind API, no alterations need to be made. The tree scan function is modified, to use the suppress flag, so it does not accidentally catch those entries. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-03-30sqfs_dir_tree_destroy/sqfs_destroy: allow NULL inputLuca Boccassi
Many library destructor functions (like free()) allow a NULL pointer as input, and do nothing in that case. This allows easier cleanup patterns: initialize pointers to NULL and then always pass them to the destroyer functions, no need for verbose goto/if-else patterns. Signed-off-by: Luca Boccassi <luca.boccassi@microsoft.com>
2021-12-05Fix: consistently use the widechar file API on WindowsDavid Oberhollenzer
When opening files on windows, use the widechar versions and convert from (assumed) UTF-8 to UTF-16 as needed. Since the broken, code-page-random API may acutall be intended in some use cases, leave that option in through an additional flag. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-07-21Fix libsquashfs directory writer size accountingDavid Oberhollenzer
The squashfs readdir() implementation in the Linux kernel returns non-existing "." and ".." entries for offsets 0 and 1, and after that reads from disk. For convenience, it was decided to store an off-by-3 value on disk instead of doing complex primary school math to adjust for this. This didn't show up until now, because the kernel implementation trusts the value from the directory header more than the actual size in the inode and happily reads 3 more than the inode would allow it to. This only showed up with 7-zip which subtracts 3 from the size and expects the result to be exact and bails if the directory headers suggest otherwise. And yes, I did consider making a "Holy Hand Granade of Antioch" reference, but consciously decided not to. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-06-25Remove casual un-const casting in various placesDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-04-09Fix: libsquashfs: allow static linking on WindowsDavid Oberhollenzer
If SQFS_STATIC is defined, dummy out the SQFS_API definition, so we don't try to pull stuff from a (in this case) non-existant DLL or try to export functions. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-04-08Fix: libsquashfs: add sqfs_free() functionDavid Oberhollenzer
On systems like Windows, the dynamic library and applications can easily end up being linked against different runtime libraries, so applications cannot be expected to be able to free() any malloc'd pointer that the library returns. This commit adds an sqfs_free function so the application can pass pointers back to the library to call the correct free() implementation. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-01-19libsqfs: Add a sqfs_block_processor_create_ex functionDavid Oberhollenzer
This function creates a block processor from a structure describing it. A stub implementation for the old sqfs_block_processor_create is added that simply sets up such a struct and forwards the call. The current version of the description struct only contains the exact same parameters and a size field at the beginning. This approach is supposed to make extending the range of parameters easier without breaking ABI compatibillity. Currently already planned are: - Adding a sqfs_file_t pointer to double-check when deduplicating fragments. - When the scanning code reaches a usable state, add the abillity to pass scanned fragment data, so the block processor can be used for appending to an existing image. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-12-06libsqfs: implement exact matching in the default block writer.David Oberhollenzer
Instead of comparing (compresed, disk-size, checksum) tuples to find block matches, do an exact, byte-for-byte comparison of the data stored on disk to avoid the possibility of a spurious colision. Since this is the desired behaviour, make it the default, optionally overrideable through a flag. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-06-11Add flags to functions that might logically be expanded in the futureDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-06-09Cleanup: mark sqfs_xattr_writer_flush writer argument as constDavid Oberhollenzer
It does not make any changes to the writer itself, so mark it as const. This also requires some similar changes to the string table. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-06-07Move the fragment deduplication hash table back into the block processorDavid Oberhollenzer
Fragment deduplication really doesn't belong into the public API of the fragment table. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-06-04Cleanup: Pull compression level parameter out into compressor configDavid Oberhollenzer
Every compressor (except LC4) has a compression level parameter. This commit pulls the compression level field out into the generic configuration structure and applies some code clean ups as a result from this. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-06-04Strictly enfore min/max dictionary size in XZ & LZMA compressorsDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-06-04lzma compressor: support micro management optionsDavid Oberhollenzer
The LZMA compressor (through the xz-utils library) supports basically the same options for micro management as the XZ compressor. This commit enables support for those options in the compressor, the option parser and adds an option field to the configuration structure. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-06-04lzma compressor: add support for the "extreme" flagDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-06-03Cleanup: Add defines for minimum and maximum block sizeDavid Oberhollenzer
This commit adds propper defines in the super block header and removes some of the hard coded constants. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-06-02Cleanup: clear naming confusion lzma vs xz in some commentsDavid Oberhollenzer
Previously, the XZ compressor was called LZMA at some points in the code, because it actually referres to LZMAv2 and the older LZMAv1 wasn't implemented. Since LZMAv1 is supported for some time now, simply calling it LZMA is confusing. This commit tries to replace the last bits where XZ is referred to as LZMA, which turns out to be only a hand full of comments. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-05-29Block processor: Add a raw block submission functionDavid Oberhollenzer
This function allows submission of raw blocks to the block processor, completely bypassing the file API. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-05-29Block processor: add flags to manage hashing & sparse block detectionDavid Oberhollenzer
This commit adds 2 new user settable flags to the block processor: - A flag to ignore sparse blocks and treat them like normal data blocks. - A flag to disable checksum computation altogether. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-05-29Support associating a user pointer with data blocksDavid Oberhollenzer
This commit modifies the block processor to support associating a user data pointer with data blocks that it forwards to the block writer, which is modified to accept an optional user data pointer. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-05-29Make the block processor inode management optionalDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-05-29Turn the sqfs_block_writer_t into an interfaceDavid Oberhollenzer
This way, everything that could be done through the hooks (and more) can be done by simply providign a custom implementation. The result is a lot clener that the previous hook based version. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-05-29cleanup: libsqfs: eliminate block writer statisticsDavid Oberhollenzer
- the "bytes submitted" can be moved over to the block processor - the number of blocks submitted are already there (implcitily, by adding the data block count to the fragment block count) - actual data bytes written can be computed from the super block - the remaining block count can be changed to simple counter that can be obtained through a function. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-05-29cleanup: libsqfs: remove hooks from sqfs_block_writer_tDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-05-21Fix the semantics of the super block deduplicationDavid Oberhollenzer
Its purely informational, but make sure other programs don't print out scary messages that imply the data has been ineficiently. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-05-04Expose more fine grained control values & flags on the XZ compressorDavid Oberhollenzer
This patch allows external users to fiddle with the XZ compressors compression strength, alignment and other values. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-04-16Propperly cast void pointer in sqfs_object_t inline functionDavid Oberhollenzer
Otherwise, C++ compilers will scream. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-03-22doxygen: fix some minor typosDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-03-22doxygen: propperly label generic inode helper functions as membersDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>