squashfs-tools-ng.git/lib/sqfs/block_processor, branch v1.1.0

Fix fail branch in block processor fragment backend

2021-03-25T13:47:06+00:00

Only clean up the fragment if it hasn't been re-assigned to the fragment block. The NULL check is definitely wrong, because we no longer re-assign it as NULL. Signed-off-by: David Oberhollenzer

Fix block processor queue accounting

2021-03-24T13:05:18+00:00

Dequeuing won't work if we have a backlog of 1 or 2 and the blocks are used for internal buffering. Take that into account, similar to the sync code. Also bump the minimum backlog to 3, just to make absolutely sure we cannot run into a dequeue loop trying to allocate a block. Signed-off-by: David Oberhollenzer

block processor: Re-implement exact fragment matching

2021-03-23T10:39:52+00:00

In the hash-table equals callback, if the hash and size match, do an exact, byte-for-byte comparison of the fragment in question. The fragment can either be in a fragment block that is in-flight (for which we have the in-flight list), in the current, unfinished fragment block, or it can be on disk. In the later case, the fragment block is resolved through the fragment table and read back from disk into a scratch buffer and decompressed. After that, the fragment is checked for byte-for-byte equality with the one we resolved through the hash table. Signed-off-by: David Oberhollenzer

block processor: keep duplicate copies of in-flight fragment blocks

2021-03-23T10:00:49+00:00

If we want full, byte-for byte, verification of fragments during de-duplication we need to check back with the blocks already written to disk, or with the ones that are in flight. The previous, extremely hacky approach simply locked up the thread pool and investigated the queues. For the new approach, we treat the thread pool as completely opaque and don't try to touch it. This commit modifies the block processor to keep duplicate copies of each submitted fragment block around, that are cleaned up once the block is dequeued and written to disk. So instead of touching the thread pool, we can simply investigate the in-fligth-block list and the current block, before resorting to reading back fragment blocks from the file. Signed-off-by: David Oberhollenzer

block processor: simplify backlog accounting

2021-03-22T20:09:54+00:00

Simply count the number of blocks we hand out (malloc'ed or recycled) and decrease the counter when we put blocks back for recycling. The sync() part becomes a little more complicated, because we can get stuck with a backlog of 1 or 2 because we have a fragment or current block buffer in use. We also need to accout for this when creating the processor, because we need to be able to request at least 2 blocks without stalling. Signed-off-by: David Oberhollenzer

Cleanup the block processor file structure

2021-03-22T14:26:47+00:00

A cleaner separation between common code, frontend code and backend code is made. The "is this byte blob zero" function is moved out to libutil (with test case and everything) with a more optimized implementation. Signed-off-by: David Oberhollenzer

Fix missing error code initialization

2021-03-21T16:41:47+00:00

Signed-off-by: David Oberhollenzer

Cleanup: Rewrite block processor to use the libutil thread_pool_t

2021-03-21T16:29:18+00:00

Throw out the messy thread pool implementation and temporarily also remove the exact fragment matching for simplicity. Signed-off-by: David Oberhollenzer

Always use the correct data type for realloc return value

2021-02-10T11:19:36+00:00

This commit mainly serves the static analysis tooling. Signed-off-by: David Oberhollenzer

libsqfs: Implement exact matching of fragments

2021-01-19T05:01:51+00:00

Signed-off-by: David Oberhollenzer