| Age | Commit message (Collapse) | Author |
|---|
|
If we want full, byte-for byte, verification of fragments during
de-duplication we need to check back with the blocks already written
to disk, or with the ones that are in flight.
The previous, extremely hacky approach simply locked up the thread
pool and investigated the queues. For the new approach, we treat the
thread pool as completely opaque and don't try to touch it.
This commit modifies the block processor to keep duplicate copies of
each submitted fragment block around, that are cleaned up once the
block is dequeued and written to disk. So instead of touching the
thread pool, we can simply investigate the in-fligth-block list and
the current block, before resorting to reading back fragment blocks
from the file.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Simply count the number of blocks we hand out (malloc'ed or recycled)
and decrease the counter when we put blocks back for recycling.
The sync() part becomes a little more complicated, because we can get
stuck with a backlog of 1 or 2 because we have a fragment or current
block buffer in use. We also need to accout for this when creating the
processor, because we need to be able to request at least 2 blocks
without stalling.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
A cleaner separation between common code, frontend code and backend
code is made.
The "is this byte blob zero" function is moved out to libutil (with
test case and everything) with a more optimized implementation.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Throw out the messy thread pool implementation and temporarily also
remove the exact fragment matching for simplicity.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This commit restructures the rbtree code to optionally use a pool
allocator for the nodes. The option is made depenend on the presence
of a pre-processor flag.
To the configure script is added an option to enable/disable the use
of custom allocators. It makes sense to still allow the malloc/free
based routes for better ASAN based instrumentation.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Technically, this should *never* **ever** happen, because a SquashFS
file always starts with a super block, which isn't wrapped in a meta
data block, so a valid SquashFS file will never have a reason to read
from offset 0.
However, this does bite us when doing unit tests where the meta reader
and writer are used on an otherwise empty file. When trying to read
from offset 0, the caching code assumes that we already have that
block, since tha block_offset got initialized to 0.
This commit changes the initialization to set the current block
location to the maximum 64 bit integer, a location we are never
going to read from, since it will always be after the limit.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
By storing the blocks in a tree, the de-duplication can lookup
existing blocks in logartihmic instead of linear time.
The linked list is still maintained, because we need to iterate
over the blocks in creation order during serialization.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
|
|
This commit mainly serves the static analysis tooling.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This function creates a block processor from a structure describing it.
A stub implementation for the old sqfs_block_processor_create is added
that simply sets up such a struct and forwards the call.
The current version of the description struct only contains the exact
same parameters and a size field at the beginning.
This approach is supposed to make extending the range of parameters
easier without breaking ABI compatibillity.
Currently already planned are:
- Adding a sqfs_file_t pointer to double-check when deduplicating
fragments.
- When the scanning code reaches a usable state, add the abillity
to pass scanned fragment data, so the block processor can be used
for appending to an existing image.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Keeping a list of fragments stored away in the current fragment block
and consolidating them in the thread pool takes them out of circulation.
If we have a lot of tiny fragments, this can lead to a situation where
all the limit is reached, but we cannot do anything, because we are
waiting for a block to complete, but they are all attached to the
current fragment block and the queue is empty.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
It looks like the last commit missed a couple more occurences
where '\' was treated incorrectly.
Fixes were still needed in sqfs_dir_reader_find_by_path and
sqfs_dir_reader_get_full_hierarchy.
This path is used in extras/browse.c.
|
|
Instead of comparing (compresed, disk-size, checksum) tuples to find
block matches, do an exact, byte-for-byte comparison of the data
stored on disk to avoid the possibility of a spurious colision.
Since this is the desired behaviour, make it the default, optionally
overrideable through a flag.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Instead of dereferencing the NULL pointer and crashing.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This commit fixes a bug where the block processor state machine would
not add the "last block" flag if there is only one not entirely filled
block and the "don't fragment" flag is set. If the flag isn't set, the
inode start block position is not updated and points to the beginning
of the image instead.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
The source code of a modified liblz4 and zlib are included with the
option to compile them into libsquashfs if they are not available on
the system.
So far, the source code was included directly in the compressor sub
directory within libsqsuashfs. This commit moves the libraries out
into the lib directory.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
If an xattr value is stored OOL, the value actually holds an 8 byte
reference to another, previously stored value. This reference points
to the header that we need to read to know the actual size of the
value before reading it, not the value itself, so after reading the
reference and seeking to it, the xattr reader needs to read the actual
header.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Instead of doing the fragile size comparison in both loops, simply
bail from the function if offset is out of bounds, clamp the size
to the available range of the file and abail if it is zero.
As a result, a lot of checks can be removed and the function will not
return data beyond EOF.
This problem occoured with files that have a short last block instead
of a fragment.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This commit fixes a build issue on BSD based systems, where alloca
is defined in stdlib.h and there is no such thing as "alloca.h".
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
It does not make any changes to the writer itself, so mark it as
const. This also requires some similar changes to the string table.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
After finding a match, reducing the reference count of the matched
elements and increasing them afterwards leaves the reference count
identical, because they refere to the same entries.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This commit moves the libsquashfs xattr related code into a sub
directory and splits the xattr writer code up into several files.
No actual code is changed.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Fragment deduplication really doesn't belong into the public API of
the fragment table.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Every compressor (except LC4) has a compression level parameter. This
commit pulls the compression level field out into the generic
configuration structure and applies some code clean ups as a result
from this.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
The LZMA compressor (through the xz-utils library) supports basically
the same options for micro management as the XZ compressor.
This commit enables support for those options in the compressor, the
option parser and adds an option field to the configuration structure.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This commit adds propper defines in the super block header and removes
some of the hard coded constants.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This commit modifies the block processor to support operating without
a fragment table. If that is the case, fragment deduplication is
essentially disabled and fragment blocks aren't indexed anymore.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This function allows submission of raw blocks to the block processor,
completely bypassing the file API.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This commit adds 2 new user settable flags to the block processor:
- A flag to ignore sparse blocks and treat them like normal
data blocks.
- A flag to disable checksum computation altogether.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This commit modifies the block processor to support associating a user
data pointer with data blocks that it forwards to the block writer,
which is modified to accept an optional user data pointer.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
