| Age | Commit message (Collapse) | Author |
|---|
|
Fragment deduplication really doesn't belong into the public API of
the fragment table.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Instead of the convoluted logic, simply use a small number of LUTs
that point to the available compressor flags for each compressor,
the avaialble options and their ranges.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Every compressor (except LC4) has a compression level parameter. This
commit pulls the compression level field out into the generic
configuration structure and applies some code clean ups as a result
from this.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
The LZMA compressor (through the xz-utils library) supports basically
the same options for micro management as the XZ compressor.
This commit enables support for those options in the compressor, the
option parser and adds an option field to the configuration structure.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This commit adds propper defines in the super block header and removes
some of the hard coded constants.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Previously, the XZ compressor was called LZMA at some points in the
code, because it actually referres to LZMAv2 and the older LZMAv1
wasn't implemented. Since LZMAv1 is supported for some time now,
simply calling it LZMA is confusing.
This commit tries to replace the last bits where XZ is referred to
as LZMA, which turns out to be only a hand full of comments.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
On a current Fedora 31, when cross compiling with mingw, pkg config
manages to find the Selinux library, but not the header. The configure
script disables WITH_SELINUX, but the library variable is still set,
causing a linker error further down the line.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
- Move the xattr extraction and repacking to xattr.c
- Don't on-the-fly delete the tar xattr list, use the function
from libtar.a
- Split minor tasks into static helper functions
- creating a libtar xattr struct from libsqfs xattr data
- finding a hard link entry from current path and inode number
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
- Split up the huge file in bin/ into one small file per sub directory
- Remove the stub in doc/
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This commit breaks tar2sqfs.c into multiple files:
- options.c contains the command line argument processing
- process_tarball.c contains the main tar repacking code
- tar2sqfs.c contains what is left (the main function)
- A header is added for gluing it all together.
No actual code is changed. The tar repacking code is slightly modified
to pass the sqfs writer and input file pointer around as argument rather
than using a global variable.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This comit breaks sqfs2tar into multiple files:
- options.c contains the command line argument processing
- write_tree.c contains the code for generating the actual tar archive
- sqfs2tar.c contains what is left
- A header is added for glueing it all together.
No actual code is changed.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This commit modifies the block processor to support operating without
a fragment table. If that is the case, fragment deduplication is
essentially disabled and fragment blocks aren't indexed anymore.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This function allows submission of raw blocks to the block processor,
completely bypassing the file API.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This commit adds 2 new user settable flags to the block processor:
- A flag to ignore sparse blocks and treat them like normal
data blocks.
- A flag to disable checksum computation altogether.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This commit modifies the block processor to support associating a user
data pointer with data blocks that it forwards to the block writer,
which is modified to accept an optional user data pointer.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This way, everything that could be done through the hooks (and more)
can be done by simply providign a custom implementation. The result is
a lot clener that the previous hook based version.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
- the "bytes submitted" can be moved over to the block processor
- the number of blocks submitted are already there (implcitily, by
adding the data block count to the fragment block count)
- actual data bytes written can be computed from the super block
- the remaining block count can be changed to simple counter that
can be obtained through a function.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This commit modifies sqfsdiff to extract the compressor options from
the squashfs image and store them in a compressor configuration if
possible. The failure path is modified to *not* burst into flames on
error, because those options are not required by any compressor to
read data from the disk and pretty much every vendor modifed SquashFS
has messed with those to the point that they cannot be propperly
decoded (or the flag is set and there are no options).
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
- Move the inode modifications out of do_block. The inode may be
reallocated in parallel by the process_completed_block function, so
it is not safe to store the fragment location in the do_block
function which is used from the worker threads.
- Move the accounting of fragment blocks to the
process_completed_block function.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This commit breaks the common code up again by moving the data submission
code to a separate file, making both a little bit more readable.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Instead of [potentially] allocating a new fragment block, take an
existing fragment and promote it to the fragmenet block. This saves
as a potential block allocation and a memcpy of the initial data.
Also it *definitely* removes block allocation from the backend path
of the block processor.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Instead of merging fragments into the fragment block inside the
process_completed_fragment function, store a linked list of fragments
in the fragment block and do the actual merging (several memcpy calls
totaling of up to 1M of data in worst case) in the worker thread
instead of the locked, serial path.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Instead of freeing/allocating blocks all the time in the locked,
serial path, use a free list to "recycle" blocks. Once a block is
no longer used, throw it onto the free list. If a new block is,
needed try to get one from the free list before calling malloc.
After a few iterations, the block processor should stop allocating
new blocks and only re-use the ones it already has.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
In the block processor, the payload area is only accessed up to
the indicated size. Even the part that is accessed is initialized
by copying data into the block before increasing the size, so there
is no real point in zero-initializing hundres of kilobytes if not
megabytes of payload area, especially since this is done in the
locked, serial path of the block processor.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
In the zstd compressor, the compression level from the configuration
structure wasn't used at all. Instead, the zstd compressor was told
to use level 0 and compressor options with that parameter were written
to disk.
This commit makes sure the level parameter is propperly initialized.
Reported-by: Sébastien Gross
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Its purely informational, but make sure other programs don't print
out scary messages that imply the data has been ineficiently.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This commit attempts to fix the following two problems:
- The number of digits computation returning an off-by-one result
if the number is 10, or the resulting digit string starts
with "10". This results in one-too-many padding bytes, corrupting
the rest of the archive since the headers now don't start at
multiples of 512 anymore.
- Adding the line length prefix affects the line length (duh). If it
grows far enough to require more digits, the result is a similar
problem. This is a converging series that we need to compute the
limit of.
Unit tests for this still need to be added. Or maybe I can convince a
bored undergrad student to provide an induction proof.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
- Some clarifications
- Some typo fixes
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
When compiling with GCC4 the following error occurs.
> lib/util/rbtree.c:140: undefined reference to `__builtin_uaddl_overflow'
This is because __builtin_uaddl_overflow() and the other
__builtin_u{add,mul}{,l,ll}_overflow() functions are only defined in
GNUC < 5 for Clang. When using GCC4 and below they are not defined.
Since the SZ_ADD_OV and SZ_MUL_OV are only used to check 'size_t' type
values. And overflow on add and multiply of unsigned types is defined
behaviour (C Standard 6.2.5 paragraph 9). It's simple to write overflow
functions for this specific case. These are based on the overflow
wrappers from the SEI CERT C Standard INT30-C.
[1] https://gcc.gnu.org/gcc-5/changes.html
Signed-off-by: Brandon Maier <brandon.maier@rockwellcollins.com>
|
|
This patch allows external users to fiddle with the XZ compressors
compression strength, alignment and other values.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
If a file consisting of multiple blocks is produced, the last block is
short and the don't fragment flag is set, the last block flag has to
be set on the block when we flush it, so the processing pipeline does
it's job correctly.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
