squashfs-tools-ng.git - A new set of tools and libraries for working with SquashFS images

Age	Commit message (Collapse)	Author
2020-03-04	Add a generic copying mechanism to sqfs_object_t	David Oberhollenzer
	This patch adds a deep-copy callback to sqfs_object_t and removes the copying mechanism from sqfs_compressor_t. This is also interesting for other types. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-03-01	Add a "do not deduplicate" block flag	David Oberhollenzer
	Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-23	Turn file inode management completely over to the block processor	David Oberhollenzer
	If the block processor allocates and dynamically resizes inodes on the fly, we can add data indefinitely without knowing the size of the file ahead of time. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-23	Unify the payload counters in the sqfs_inode_generic_t	David Oberhollenzer
	Instead of having seperate counters for blocks, dir index bytes and having to fiddle out the link target size, simply use a single value that stores the number of payload bytes used. A seperate "payload bytes available" is used for dynamically growing inodes during processing. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-22	Move inode size accounting completely to the block processor	David Oberhollenzer
	Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-22	Cleanup block processor: merge common initialization code	David Oberhollenzer
	Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-22	Cleanup block processor: Merge destructors for Windows & pthreads	David Oberhollenzer
	Since the merged destructor checks if the objects it destroys were actually initialized, the pthread implementation can also replace its error path cleanup with simply calling the destructor. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-22	Add a seperate sqfs_block_processor_sync function	David Oberhollenzer
	This function waits for all pending blocks to be written to disk, but doesn't flush the fragment block, so processing can continue afterwards as if nothing happened. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-21	Cleanup: move utilities back out of libsquashfs	David Oberhollenzer
	This commit removes the allocation helpers and string table functions out of libsquashfs back into a "libutil.a". The problem of libsquashfs exporting stuff that it shouldn't is resolved by retaining the internal attributes and directly adding the source to libsquashfs instead of trying to somehow link against libutil.la. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-20	Thread pool block processor: Cleanup after restructuring	David Oberhollenzer
	- Merge duplicated code from append_to_work_queue and sqfs_block_processor_finish Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-20	Restructure thread pool block processor	David Oberhollenzer
	Implement the io-queue based design as outline in doc/parallelism.txt Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-18	Simplify the thread pool block processor somewhat	David Oberhollenzer
	- Split the worker function up into smaller functions that are a little more readable. - Only dequeue one block at a time. Makes the dequeueing a lot more readable and understandable. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-16	block processor: move the internals to the respective implementations	David Oberhollenzer
	Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-16	block processor: merge rest of fileapi.c into common.c	David Oberhollenzer
	Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-16	Replace crc32 with xxhash32	David Oberhollenzer
	On the one hand, benchmarking and profiling determined xxhash32 to be faster than the zlib implementation of crc32, on the other hand profiling determined that crc32 computation contributed signifficantly to the overall runtime. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-16	Move all the queue-waiting logic to the thread pool implemenation	David Oberhollenzer
	Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-16	Minor cleanup	David Oberhollenzer
	Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-16	block processor: move sparse block detection into worker thread	David Oberhollenzer
	Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-15	Move block block accounting to the other end of the block pipeline	David Oberhollenzer
	This commit moves all of the fragment/block accounting in the block processor over to the writing end of the pipeline. In order to do this, the sparse blocks are allowed to bubble through the pipeline. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-15	Cleanup: block processor: move init/cleanup functions into implemenations	David Oberhollenzer
	Again, the generic init/cleanup functions do way too many things that are specific to the thread pool implementation. Thanks to the splitting up of the block processor, they also have become quite trivial. This commit moves those functions into their respective implementations, allowing even further simplificiation. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-15	Cleanup: block processor: move finish function back into implementations	David Oberhollenzer
	The "generic" version actually uses specific internals of the thread pool implementation. Move it back into the thread pool based implementation and simplify the serial processor. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-15	Cleanup: block processor: remove test_and_set_status	David Oberhollenzer
	Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-15	Don't set error status on block processor for non-fatal errors	David Oberhollenzer
	If the API is mis-used or allocation fails, return an appropriate error code, but don't permanently break the block processor. It's easy to recover from that. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-15	Cleanup: block processor: remove delayed thread notification	David Oberhollenzer
	Under the assumption that block processing is CPU bound and not I/O bound, this is entirely useless. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-12	Implement a more explicit object system	David Oberhollenzer
	Make every dynamically allocated, opaque data structure inherit from a common sqfs_object_t structure with common entry points (e.g. destroy). This removes tons of public API functions and replaces them with a simple sqfs_destroy instead. If semantics of the (until now implicit) object system need to be extended, it can be much more conveniantely done this way. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-12	Cleanup: Move sqfs_block_t to block processor internals	David Oberhollenzer
	It was basically built around the block processor and exposed way too many internals. Removing it from other places was mostly trivial. This commit completely removes it from the public API and even most of the libsquashfs internals. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-12	Clenaup: remove useage of sqfs_block_t from block writer	David Oberhollenzer
	The sqfs_block_t structure has been written for the block processor and exposes way too many internals. This commit removes its usage from the block writer, cutting it down to the bare essentials, so the structure can be removed from the public API later on. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-10	Cleanup: remove block hooks entirely from block processor	David Oberhollenzer
	Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-10	Cleanup: remove the fragment store/discard and block discard hooks	David Oberhollenzer
	There is no obvious non-footgun use for those other than tallying statistics, which is now done by the data structures themselves. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-10	Add run time statistics to the block writer and processor	David Oberhollenzer
	Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-09	block processor: merge left overs of block.c/fragment.c into common.c	David Oberhollenzer
	Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-09	Move block writer and fragment table management out of block processor	David Oberhollenzer
	Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-01-31	Split the block writing/deduplication away from the block processor	David Oberhollenzer
	This commit moves the entire block writing and deduplication of data blocks over to a different data type named "block writer". For simplicity, the interfaces of the block processor are left as is and are turned into warppers. Likewise, most of the code in the block writer is just verbatim from the block processor, to be cleaned up in subsequent commits. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-01-29	Rename sqfs_data_writer_t back to sqfs_block_processor_t	David Oberhollenzer
	Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>