aboutsummaryrefslogtreecommitdiff
path: root/lib
AgeCommit message (Collapse)Author
2020-02-23Turn file inode management completely over to the block processorDavid Oberhollenzer
If the block processor allocates and dynamically resizes inodes on the fly, we can add data indefinitely without knowing the size of the file ahead of time. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-23Unify the payload counters in the sqfs_inode_generic_tDavid Oberhollenzer
Instead of having seperate counters for blocks, dir index bytes and having to fiddle out the link target size, simply use a single value that stores the number of payload bytes used. A seperate "payload bytes available" is used for dynamically growing inodes during processing. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-22libcommon: stdin file: Fix size accounting for sparse filesDavid Oberhollenzer
The file has to report the "apparent size" for sparse files, but internally work with the actual size in the tar ball. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-22Move inode size accounting completely to the block processorDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-22Cleanup block processor: merge common initialization codeDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-22Cleanup block processor: Merge destructors for Windows & pthreadsDavid Oberhollenzer
Since the merged destructor checks if the objects it destroys were actually initialized, the pthread implementation can also replace its error path cleanup with simply calling the destructor. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-22Add a seperate sqfs_block_processor_sync functionDavid Oberhollenzer
This function waits for all pending blocks to be written to disk, but doesn't flush the fragment block, so processing can continue afterwards as if nothing happened. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-22Make hard link detection scale betterDavid Oberhollenzer
Instead of doing a linear search, which scales quadratically, use a red-black tree with inode numbers as key for finding hard links. To reiterate: we can't just use a flat table, because the SquashFS file is potentially untrusted and the inode numbers can be anything, no matter what the super block says. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-22Add a very simple red-black tree implementationDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-21Cleanup: move utilities back out of libsquashfsDavid Oberhollenzer
This commit removes the allocation helpers and string table functions out of libsquashfs back into a "libutil.a". The problem of libsquashfs exporting stuff that it shouldn't is resolved by retaining the internal attributes and directly adding the source to libsquashfs instead of trying to somehow link against libutil.la. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-20Thread pool block processor: Cleanup after restructuringDavid Oberhollenzer
- Merge duplicated code from append_to_work_queue and sqfs_block_processor_finish Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-20Restructure thread pool block processorDavid Oberhollenzer
Implement the io-queue based design as outline in doc/parallelism.txt Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-18Simplify the thread pool block processor somewhatDavid Oberhollenzer
- Split the worker function up into smaller functions that are a little more readable. - Only dequeue one block at a time. Makes the dequeueing a lot more readable and understandable. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-16block processor: move the internals to the respective implementationsDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-16block processor: merge rest of fileapi.c into common.cDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-16Replace crc32 with xxhash32David Oberhollenzer
On the one hand, benchmarking and profiling determined xxhash32 to be faster than the zlib implementation of crc32, on the other hand profiling determined that crc32 computation contributed signifficantly to the overall runtime. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-16Move all the queue-waiting logic to the thread pool implemenationDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-16Minor cleanupDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-16block processor: move sparse block detection into worker threadDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-15Move block block accounting to the other end of the block pipelineDavid Oberhollenzer
This commit moves all of the fragment/block accounting in the block processor over to the writing end of the pipeline. In order to do this, the sparse blocks are allowed to bubble through the pipeline. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-15Cleanup: block processor: move init/cleanup functions into implemenationsDavid Oberhollenzer
Again, the generic init/cleanup functions do way too many things that are specific to the thread pool implementation. Thanks to the splitting up of the block processor, they also have become quite trivial. This commit moves those functions into their respective implementations, allowing even further simplificiation. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-15Cleanup: block processor: move finish function back into implementationsDavid Oberhollenzer
The "generic" version actually uses specific internals of the thread pool implementation. Move it back into the thread pool based implementation and simplify the serial processor. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-15Cleanup: block processor: remove test_and_set_statusDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-15Don't set error status on block processor for non-fatal errorsDavid Oberhollenzer
If the API is mis-used or allocation fails, return an appropriate error code, but don't permanently break the block processor. It's easy to recover from that. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-15Add a "sequence error" to libsquashfs error codesDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-15Cleanup: block processor: remove delayed thread notificationDavid Oberhollenzer
Under the assumption that block processing is CPU bound and not I/O bound, this is entirely useless. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-12Use a hash table for fragment lookup instead of linear searchDavid Oberhollenzer
Profiling on a sample filesystem determined that fragment deduplication lookups rank third place directly after crc32 and the actual compression. By using a hash table instead of linear search, this time can be reduced drastically. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-12Implement a more explicit object systemDavid Oberhollenzer
Make every dynamically allocated, opaque data structure inherit from a common sqfs_object_t structure with common entry points (e.g. destroy). This removes tons of public API functions and replaces them with a simple sqfs_destroy instead. If semantics of the (until now implicit) object system need to be extended, it can be much more conveniantely done this way. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-12Cleanup: Move sqfs_block_t to block processor internalsDavid Oberhollenzer
It was basically built around the block processor and exposed way too many internals. Removing it from other places was mostly trivial. This commit completely removes it from the public API and even most of the libsquashfs internals. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-12Remove usage of sqfs_block_t from block readerDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-12Clenaup: remove useage of sqfs_block_t from block writerDavid Oberhollenzer
The sqfs_block_t structure has been written for the block processor and exposes way too many internals. This commit removes its usage from the block writer, cutting it down to the bare essentials, so the structure can be removed from the public API later on. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-12Fix data reader return codesDavid Oberhollenzer
Return an error number as document instead of throwing -1. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-10Cleanup: remove block hooks entirely from block processorDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-10Cleanup: remove the fragment store/discard and block discard hooksDavid Oberhollenzer
There is no obvious non-footgun use for those other than tallying statistics, which is now done by the data structures themselves. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-10Cleanup statistics print outDavid Oberhollenzer
- Give a rounded input/output byte count. - Seperate groups by new lines. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-10Cleanup: Use stat structs instead of hooks in tar2sqfs/gensquashfsDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-10Add run time statistics to the block writer and processorDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-09block processor: merge left overs of block.c/fragment.c into common.cDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-09Move block writer and fragment table management out of block processorDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-01-31Split the block writing/deduplication away from the block processorDavid Oberhollenzer
This commit moves the entire block writing and deduplication of data blocks over to a different data type named "block writer". For simplicity, the interfaces of the block processor are left as is and are turned into warppers. Likewise, most of the code in the block writer is just verbatim from the block processor, to be cleaned up in subsequent commits. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-01-29Rename sqfs_data_writer_t back to sqfs_block_processor_tDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-01-26Fix memory leak in hard link detection codeDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-01-26Cleanup: Move fragment deduplication code over to fragment tableDavid Oberhollenzer
This removes further clutter from the data writer. Any future efforts on making fragment by hash lookup faster can focus on that area only and don't clutter the block processor. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-01-25Add missing headers to installed header listDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-01-24Cleanup: use fragment table primitive in data writerDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-01-24Cleanup: remove single use helper functions from data writerDavid Oberhollenzer
This commit moves the single use helper functions that are called from worker thread context into the worker thread function. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-01-24Cleanup: use the frag table data type in the data readerDavid Oberhollenzer
Instead of doing everything by manually. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-01-24Add a fragment table primitive to libsquashfsDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-01-20Add a flag field to the id table create functionDavid Oberhollenzer
Just to be safe in case there needs to be an extension in the future. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-01-19Cleanup: remove the payload pointers from sqfs_inode_generic_tDavid Oberhollenzer
There are 3 types of extra payload: - Directory index - File block sizes - Symlink target This commit removes the type specific pointers and modifies the code to use the payload area directly. To simplify the file block case and mitigate alignment issues, the type of the extra field is changed to sqfs_u32. For symlink target, the extra field can simply be cast to a character pointer (it had to be cast anyway for most uses). For block sizes, probably the most common usecase, it can be used as is. For directory indices, there is a helper function anyway. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
generated by cgit v1.2.3 (git 2.39.1) at 2024-12-26 05:22:58 +0000