| Age | Commit message (Collapse) | Author |
|---|
|
The squashfs readdir() implementation in the Linux kernel returns
non-existing "." and ".." entries for offsets 0 and 1, and after
that reads from disk. For convenience, it was decided to store an
off-by-3 value on disk instead of doing complex primary school math
to adjust for this. This didn't show up until now, because the kernel
implementation trusts the value from the directory header more than
the actual size in the inode and happily reads 3 more than the inode
would allow it to. This only showed up with 7-zip which subtracts 3
from the size and expects the result to be exact and bails if the
directory headers suggest otherwise.
And yes, I did consider making a "Holy Hand Granade of Antioch"
reference, but consciously decided not to.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
The same problem with the meta data header again, 16 bit read from
a buffer: copy the buffer data into a 16 bit variable instead of
casting to something potentially unaligned.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
On systems like Windows, the dynamic library and applications can
easily end up being linked against different runtime libraries, so
applications cannot be expected to be able to free() any malloc'd
pointer that the library returns.
This commit adds an sqfs_free function so the application can pass
pointers back to the library to call the correct free() implementation.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Currently, when the block processor aggreagtes fragments into a
fragment block, it applies the "don't compress" flag if any of the
original framgnets has it set, but the "align to device block" flag
is lost.
This commit ensures that both flags get applied to the fragment block
if set.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
1) If the block alignment flag is set, the padding bytes must be
inserted _before_ recording the start position, otherwise the
resulting image is not readable.
2) Also perform alignment if the flag is set on a fragment block.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
|
|
This commit is an amalgamation of the commits on master that
implement exact matching of fragment blocks during deduplication.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Instead of comparing (compresed, disk-size, checksum) tuples to find
block matches, do an exact, byte-for-byte comparison of the data
stored on disk to avoid the possibility of a spurious colision.
Since this is the desired behaviour, make it the default, optionally
overrideable through a flag.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Keeping a list of fragments stored away in the current fragment block
and consolidating them in the thread pool takes them out of circulation.
If we have a lot of tiny fragments, this can lead to a situation where
all the limit is reached, but we cannot do anything, because we are
waiting for a block to complete, but they are all attached to the
current fragment block and the queue is empty.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
It looks like the last commit missed a couple more occurences
where '\' was treated incorrectly.
Fixes were still needed in sqfs_dir_reader_find_by_path and
sqfs_dir_reader_get_full_hierarchy.
This path is used in extras/browse.c.
|
|
Instead of dereferencing the NULL pointer and crashing.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This commit fixes a bug where the block processor state machine would
not add the "last block" flag if there is only one not entirely filled
block and the "don't fragment" flag is set. If the flag isn't set, the
inode start block position is not updated and points to the beginning
of the image instead.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
The source code of a modified liblz4 and zlib are included with the
option to compile them into libsquashfs if they are not available on
the system.
So far, the source code was included directly in the compressor sub
directory within libsqsuashfs. This commit moves the libraries out
into the lib directory.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
If an xattr value is stored OOL, the value actually holds an 8 byte
reference to another, previously stored value. This reference points
to the header that we need to read to know the actual size of the
value before reading it, not the value itself, so after reading the
reference and seeking to it, the xattr reader needs to read the actual
header.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Instead of doing the fragile size comparison in both loops, simply
bail from the function if offset is out of bounds, clamp the size
to the available range of the file and abail if it is zero.
As a result, a lot of checks can be removed and the function will not
return data beyond EOF.
This problem occoured with files that have a short last block instead
of a fragment.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This commit fixes a build issue on BSD based systems, where alloca
is defined in stdlib.h and there is no such thing as "alloca.h".
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
It does not make any changes to the writer itself, so mark it as
const. This also requires some similar changes to the string table.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
After finding a match, reducing the reference count of the matched
elements and increasing them afterwards leaves the reference count
identical, because they refere to the same entries.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This commit moves the libsquashfs xattr related code into a sub
directory and splits the xattr writer code up into several files.
No actual code is changed.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Fragment deduplication really doesn't belong into the public API of
the fragment table.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Every compressor (except LC4) has a compression level parameter. This
commit pulls the compression level field out into the generic
configuration structure and applies some code clean ups as a result
from this.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
The LZMA compressor (through the xz-utils library) supports basically
the same options for micro management as the XZ compressor.
This commit enables support for those options in the compressor, the
option parser and adds an option field to the configuration structure.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This commit adds propper defines in the super block header and removes
some of the hard coded constants.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This commit modifies the block processor to support operating without
a fragment table. If that is the case, fragment deduplication is
essentially disabled and fragment blocks aren't indexed anymore.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This function allows submission of raw blocks to the block processor,
completely bypassing the file API.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This commit adds 2 new user settable flags to the block processor:
- A flag to ignore sparse blocks and treat them like normal
data blocks.
- A flag to disable checksum computation altogether.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This commit modifies the block processor to support associating a user
data pointer with data blocks that it forwards to the block writer,
which is modified to accept an optional user data pointer.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This way, everything that could be done through the hooks (and more)
can be done by simply providign a custom implementation. The result is
a lot clener that the previous hook based version.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
- the "bytes submitted" can be moved over to the block processor
- the number of blocks submitted are already there (implcitily, by
adding the data block count to the fragment block count)
- actual data bytes written can be computed from the super block
- the remaining block count can be changed to simple counter that
can be obtained through a function.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
- Move the inode modifications out of do_block. The inode may be
reallocated in parallel by the process_completed_block function, so
it is not safe to store the fragment location in the do_block
function which is used from the worker threads.
- Move the accounting of fragment blocks to the
process_completed_block function.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This commit breaks the common code up again by moving the data submission
code to a separate file, making both a little bit more readable.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Instead of [potentially] allocating a new fragment block, take an
existing fragment and promote it to the fragmenet block. This saves
as a potential block allocation and a memcpy of the initial data.
Also it *definitely* removes block allocation from the backend path
of the block processor.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Instead of merging fragments into the fragment block inside the
process_completed_fragment function, store a linked list of fragments
in the fragment block and do the actual merging (several memcpy calls
totaling of up to 1M of data in worst case) in the worker thread
instead of the locked, serial path.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Instead of freeing/allocating blocks all the time in the locked,
serial path, use a free list to "recycle" blocks. Once a block is
no longer used, throw it onto the free list. If a new block is,
needed try to get one from the free list before calling malloc.
After a few iterations, the block processor should stop allocating
new blocks and only re-use the ones it already has.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
In the block processor, the payload area is only accessed up to
the indicated size. Even the part that is accessed is initialized
by copying data into the block before increasing the size, so there
is no real point in zero-initializing hundres of kilobytes if not
megabytes of payload area, especially since this is done in the
locked, serial path of the block processor.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
In the zstd compressor, the compression level from the configuration
structure wasn't used at all. Instead, the zstd compressor was told
to use level 0 and compressor options with that parameter were written
to disk.
This commit makes sure the level parameter is propperly initialized.
Reported-by: Sébastien Gross
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
