aboutsummaryrefslogtreecommitdiff
path: root/lib
AgeCommit message (Collapse)Author
2020-03-19Fix pthread_join check for valid thread handlesDavid Oberhollenzer
On Linux, checking for > 0 worked because pthread_t is internally an integer type. On other platforms (*caugh* Mac OS X *caugh*), it is typedefed to an opaque pointer, causing a warning if used in an integer relational comparison. The intended use is to allow the generic cleanup function to be used in the error path of the block processor creation function, while preventing pthread_join being called on threads that haven't been created at all. Since they are calloc'ed to 0, testing for non-zero values should suffice in both cases. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-03-18Fix build of lz4 compressor with older versions of liblz4David Oberhollenzer
Older versions of liblz4 don't define LZ4HC_CLEVEL_MAX. This commit adds a definition if liblz4 doesn't provide one. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-03-18Restore workaround for unaligned reads in xxhashDavid Oberhollenzer
The code was originally used inside the block processor, where 32 bit aligned data could be guaranteed. If it is available in libutil, I cannot possibly guarantee for alignment in future use elsewhere. Even for the block processor it was rather risky "remember this detail very well" buisness. This commit restores the unaligned read treatment of the original. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-03-18Cleanup: Move xxhash32 code to libutilDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-03-05Get rid of sqfs_compressor_existsDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-03-05Change the signature of sqfs_compressor_create to return an error codeDavid Oberhollenzer
Make sure the function has a way of telling the caller *why* it failed. This way, the function can convey whether it had an internal error, an allocation failure, whether the arguments are totaly nonsensical, or simply that the compressor *or specific configuration* is not supported. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-03-05Cleanup: Remove the E_ prefix from all libsquashfs enumeratorsDavid Oberhollenzer
Avoid namespace polution. Make sure all exportet symbols are prefixed with either sqfs_ or SQFS_. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-03-04Fix block writer inheritance of sqfs_object_tDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-03-04Cleanup: match xattr reader API closer to id table APIDavid Oberhollenzer
Instead of creating everything in the "create" function, cleanup and create/initialize stuff in a "load" function. This allows the xattr reader to be reset/re-used and adds the benefit of not having to lug around references to the super block, compressor and file (altough the later two are hidden inside the meta reader). Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-03-04Add a generic copying mechanism to sqfs_object_tDavid Oberhollenzer
This patch adds a deep-copy callback to sqfs_object_t and removes the copying mechanism from sqfs_compressor_t. This is also interesting for other types. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-03-04Add a deep copy function for the str_table_t helperDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-03-01Add a "do not deduplicate" block flagDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-03-01Fix printf format specifies for sqfs_u64David Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-03-01Fix: Replace bit shifts in parse_size with SZ_MUL_OVDavid Oberhollenzer
On 32 bit systems, size_t is a 32 bit integer and doing 64 bit shifts won't do us any good. So instead, use the existing size_t multiply and catch overflow macros. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-03-01Fix alloca in write_inode.c for windows buildDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-28Cleanup pax header parser a littleDavid Oberhollenzer
This commit tries to untangle the logic of parsing and sanitizing the pax header length field and the associated bounds checks. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-27Fix: strictly verify compressor settings in config initializationDavid Oberhollenzer
Make sure the function throws an error if a given compressor ID or flag is not known. This way, libsquahfs supports *exactly* and *only* what the on-disk format specifies. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-27Add a function to the compressor interface to get the configurationDavid Oberhollenzer
This allows getting the compressor configuration back after creating it, for various purposes. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-23Remove the sqfs_inode_copy functionDavid Oberhollenzer
With unified payload size counters, copying an inode is now trivial. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-23Turn file inode management completely over to the block processorDavid Oberhollenzer
If the block processor allocates and dynamically resizes inodes on the fly, we can add data indefinitely without knowing the size of the file ahead of time. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-23Unify the payload counters in the sqfs_inode_generic_tDavid Oberhollenzer
Instead of having seperate counters for blocks, dir index bytes and having to fiddle out the link target size, simply use a single value that stores the number of payload bytes used. A seperate "payload bytes available" is used for dynamically growing inodes during processing. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-22libcommon: stdin file: Fix size accounting for sparse filesDavid Oberhollenzer
The file has to report the "apparent size" for sparse files, but internally work with the actual size in the tar ball. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-22Move inode size accounting completely to the block processorDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-22Cleanup block processor: merge common initialization codeDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-22Cleanup block processor: Merge destructors for Windows & pthreadsDavid Oberhollenzer
Since the merged destructor checks if the objects it destroys were actually initialized, the pthread implementation can also replace its error path cleanup with simply calling the destructor. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-22Add a seperate sqfs_block_processor_sync functionDavid Oberhollenzer
This function waits for all pending blocks to be written to disk, but doesn't flush the fragment block, so processing can continue afterwards as if nothing happened. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-22Make hard link detection scale betterDavid Oberhollenzer
Instead of doing a linear search, which scales quadratically, use a red-black tree with inode numbers as key for finding hard links. To reiterate: we can't just use a flat table, because the SquashFS file is potentially untrusted and the inode numbers can be anything, no matter what the super block says. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-22Add a very simple red-black tree implementationDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-21Cleanup: move utilities back out of libsquashfsDavid Oberhollenzer
This commit removes the allocation helpers and string table functions out of libsquashfs back into a "libutil.a". The problem of libsquashfs exporting stuff that it shouldn't is resolved by retaining the internal attributes and directly adding the source to libsquashfs instead of trying to somehow link against libutil.la. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-20Thread pool block processor: Cleanup after restructuringDavid Oberhollenzer
- Merge duplicated code from append_to_work_queue and sqfs_block_processor_finish Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-20Restructure thread pool block processorDavid Oberhollenzer
Implement the io-queue based design as outline in doc/parallelism.txt Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-18Simplify the thread pool block processor somewhatDavid Oberhollenzer
- Split the worker function up into smaller functions that are a little more readable. - Only dequeue one block at a time. Makes the dequeueing a lot more readable and understandable. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-16block processor: move the internals to the respective implementationsDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-16block processor: merge rest of fileapi.c into common.cDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-16Replace crc32 with xxhash32David Oberhollenzer
On the one hand, benchmarking and profiling determined xxhash32 to be faster than the zlib implementation of crc32, on the other hand profiling determined that crc32 computation contributed signifficantly to the overall runtime. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-16Move all the queue-waiting logic to the thread pool implemenationDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-16Minor cleanupDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-16block processor: move sparse block detection into worker threadDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-15Move block block accounting to the other end of the block pipelineDavid Oberhollenzer
This commit moves all of the fragment/block accounting in the block processor over to the writing end of the pipeline. In order to do this, the sparse blocks are allowed to bubble through the pipeline. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-15Cleanup: block processor: move init/cleanup functions into implemenationsDavid Oberhollenzer
Again, the generic init/cleanup functions do way too many things that are specific to the thread pool implementation. Thanks to the splitting up of the block processor, they also have become quite trivial. This commit moves those functions into their respective implementations, allowing even further simplificiation. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-15Cleanup: block processor: move finish function back into implementationsDavid Oberhollenzer
The "generic" version actually uses specific internals of the thread pool implementation. Move it back into the thread pool based implementation and simplify the serial processor. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-15Cleanup: block processor: remove test_and_set_statusDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-15Don't set error status on block processor for non-fatal errorsDavid Oberhollenzer
If the API is mis-used or allocation fails, return an appropriate error code, but don't permanently break the block processor. It's easy to recover from that. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-15Add a "sequence error" to libsquashfs error codesDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-15Cleanup: block processor: remove delayed thread notificationDavid Oberhollenzer
Under the assumption that block processing is CPU bound and not I/O bound, this is entirely useless. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-12Use a hash table for fragment lookup instead of linear searchDavid Oberhollenzer
Profiling on a sample filesystem determined that fragment deduplication lookups rank third place directly after crc32 and the actual compression. By using a hash table instead of linear search, this time can be reduced drastically. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-12Implement a more explicit object systemDavid Oberhollenzer
Make every dynamically allocated, opaque data structure inherit from a common sqfs_object_t structure with common entry points (e.g. destroy). This removes tons of public API functions and replaces them with a simple sqfs_destroy instead. If semantics of the (until now implicit) object system need to be extended, it can be much more conveniantely done this way. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-12Cleanup: Move sqfs_block_t to block processor internalsDavid Oberhollenzer
It was basically built around the block processor and exposed way too many internals. Removing it from other places was mostly trivial. This commit completely removes it from the public API and even most of the libsquashfs internals. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-12Remove usage of sqfs_block_t from block readerDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-12Clenaup: remove useage of sqfs_block_t from block writerDavid Oberhollenzer
The sqfs_block_t structure has been written for the block processor and exposes way too many internals. This commit removes its usage from the block writer, cutting it down to the bare essentials, so the structure can be removed from the public API later on. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
generated by cgit v1.2.3 (git 2.39.1) at 2024-11-05 06:52:47 +0000