squashfs-tools-ng.git - A new set of tools and libraries for working with SquashFS images

Age	Commit message (Collapse)	Author
2020-05-23	block processor: recycle blocks to reduce allocation pressure	David Oberhollenzer
	Instead of freeing/allocating blocks all the time in the locked, serial path, use a free list to "recycle" blocks. Once a block is no longer used, throw it onto the free list. If a new block is, needed try to get one from the free list before calling malloc. After a few iterations, the block processor should stop allocating new blocks and only re-use the ones it already has. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-05-23	block processor: don't zero initialize the block payload area	David Oberhollenzer
	In the block processor, the payload area is only accessed up to the indicated size. Even the part that is accessed is initialized by copying data into the block before increasing the size, so there is no real point in zero-initializing hundres of kilobytes if not megabytes of payload area, especially since this is done in the locked, serial path of the block processor. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-05-04	Fix: propperly set the last block flag if fragments are disabled	David Oberhollenzer
	If a file consisting of multiple blocks is produced, the last block is short and the don't fragment flag is set, the last block flag has to be set on the block when we flush it, so the processing pipeline does it's job correctly. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-03-01	Add a "do not deduplicate" block flag	David Oberhollenzer
	Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-23	Turn file inode management completely over to the block processor	David Oberhollenzer
	If the block processor allocates and dynamically resizes inodes on the fly, we can add data indefinitely without knowing the size of the file ahead of time. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-23	Unify the payload counters in the sqfs_inode_generic_t	David Oberhollenzer
	Instead of having seperate counters for blocks, dir index bytes and having to fiddle out the link target size, simply use a single value that stores the number of payload bytes used. A seperate "payload bytes available" is used for dynamically growing inodes during processing. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-22	Move inode size accounting completely to the block processor	David Oberhollenzer
	Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-20	Restructure thread pool block processor	David Oberhollenzer
	Implement the io-queue based design as outline in doc/parallelism.txt Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-16	block processor: merge rest of fileapi.c into common.c	David Oberhollenzer
	Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-16	Replace crc32 with xxhash32	David Oberhollenzer
	On the one hand, benchmarking and profiling determined xxhash32 to be faster than the zlib implementation of crc32, on the other hand profiling determined that crc32 computation contributed signifficantly to the overall runtime. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-16	Minor cleanup	David Oberhollenzer
	Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-16	block processor: move sparse block detection into worker thread	David Oberhollenzer
	Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-15	Move block block accounting to the other end of the block pipeline	David Oberhollenzer
	This commit moves all of the fragment/block accounting in the block processor over to the writing end of the pipeline. In order to do this, the sparse blocks are allowed to bubble through the pipeline. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-15	Cleanup: block processor: move init/cleanup functions into implemenations	David Oberhollenzer
	Again, the generic init/cleanup functions do way too many things that are specific to the thread pool implementation. Thanks to the splitting up of the block processor, they also have become quite trivial. This commit moves those functions into their respective implementations, allowing even further simplificiation. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-12	Clenaup: remove useage of sqfs_block_t from block writer	David Oberhollenzer
	The sqfs_block_t structure has been written for the block processor and exposes way too many internals. This commit removes its usage from the block writer, cutting it down to the bare essentials, so the structure can be removed from the public API later on. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-10	Cleanup: remove block hooks entirely from block processor	David Oberhollenzer
	Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-10	Cleanup: remove the fragment store/discard and block discard hooks	David Oberhollenzer
	There is no obvious non-footgun use for those other than tallying statistics, which is now done by the data structures themselves. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-10	Add run time statistics to the block writer and processor	David Oberhollenzer
	Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-09	block processor: merge left overs of block.c/fragment.c into common.c	David Oberhollenzer
	Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-02-09	Move block writer and fragment table management out of block processor	David Oberhollenzer
	Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-01-31	Split the block writing/deduplication away from the block processor	David Oberhollenzer
	This commit moves the entire block writing and deduplication of data blocks over to a different data type named "block writer". For simplicity, the interfaces of the block processor are left as is and are turned into warppers. Likewise, most of the code in the block writer is just verbatim from the block processor, to be cleaned up in subsequent commits. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-01-29	Rename sqfs_data_writer_t back to sqfs_block_processor_t	David Oberhollenzer
	Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>