| Age | Commit message (Collapse) | Author |
|---|
|
This commit is an amalgamation of the commits on master that
implement exact matching of fragment blocks during deduplication.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Instead of comparing (compresed, disk-size, checksum) tuples to find
block matches, do an exact, byte-for-byte comparison of the data
stored on disk to avoid the possibility of a spurious colision.
Since this is the desired behaviour, make it the default, optionally
overrideable through a flag.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
It does not make any changes to the writer itself, so mark it as
const. This also requires some similar changes to the string table.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Fragment deduplication really doesn't belong into the public API of
the fragment table.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Every compressor (except LC4) has a compression level parameter. This
commit pulls the compression level field out into the generic
configuration structure and applies some code clean ups as a result
from this.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
The LZMA compressor (through the xz-utils library) supports basically
the same options for micro management as the XZ compressor.
This commit enables support for those options in the compressor, the
option parser and adds an option field to the configuration structure.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This commit adds propper defines in the super block header and removes
some of the hard coded constants.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Previously, the XZ compressor was called LZMA at some points in the
code, because it actually referres to LZMAv2 and the older LZMAv1
wasn't implemented. Since LZMAv1 is supported for some time now,
simply calling it LZMA is confusing.
This commit tries to replace the last bits where XZ is referred to
as LZMA, which turns out to be only a hand full of comments.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
- Move the xattr extraction and repacking to xattr.c
- Don't on-the-fly delete the tar xattr list, use the function
from libtar.a
- Split minor tasks into static helper functions
- creating a libtar xattr struct from libsqfs xattr data
- finding a hard link entry from current path and inode number
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This function allows submission of raw blocks to the block processor,
completely bypassing the file API.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This commit adds 2 new user settable flags to the block processor:
- A flag to ignore sparse blocks and treat them like normal
data blocks.
- A flag to disable checksum computation altogether.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This commit modifies the block processor to support associating a user
data pointer with data blocks that it forwards to the block writer,
which is modified to accept an optional user data pointer.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This way, everything that could be done through the hooks (and more)
can be done by simply providign a custom implementation. The result is
a lot clener that the previous hook based version.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
- the "bytes submitted" can be moved over to the block processor
- the number of blocks submitted are already there (implcitily, by
adding the data block count to the fragment block count)
- actual data bytes written can be computed from the super block
- the remaining block count can be changed to simple counter that
can be obtained through a function.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Its purely informational, but make sure other programs don't print
out scary messages that imply the data has been ineficiently.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
When compiling with GCC4 the following error occurs.
> lib/util/rbtree.c:140: undefined reference to `__builtin_uaddl_overflow'
This is because __builtin_uaddl_overflow() and the other
__builtin_u{add,mul}{,l,ll}_overflow() functions are only defined in
GNUC < 5 for Clang. When using GCC4 and below they are not defined.
Since the SZ_ADD_OV and SZ_MUL_OV are only used to check 'size_t' type
values. And overflow on add and multiply of unsigned types is defined
behaviour (C Standard 6.2.5 paragraph 9). It's simple to write overflow
functions for this specific case. These are based on the overflow
wrappers from the SEI CERT C Standard INT30-C.
[1] https://gcc.gnu.org/gcc-5/changes.html
Signed-off-by: Brandon Maier <brandon.maier@rockwellcollins.com>
|
|
This patch allows external users to fiddle with the XZ compressors
compression strength, alignment and other values.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Tar archives can contain set two kinds of PAX headers:
- local headers that modify the attributes of the next file
- global headers that set defaults for all files
The later is used "... not widely used", according to tar(5)
and has been deliberately not implemented.
Some programs (e.g. git-archive) *do* generate them (in the case
of git, it stores the commit hash).
This commit adds a code path that skips a PAX global header entirely
and resumes tar parsing, instead of erroneusly reporting it as an
entry.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This commit changes the tar2sqfs & gensquashfs code to pass the exit
status on to sqfs_writer_cleanup in libcommon.
The function sqfs writer code in libcommon is changed to retain the
output file name and delete it if the status passed to the cleanup
function is anything other than EXIT_SUCCESS.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Otherwise, C++ compilers will scream.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
The PRIu64 et al are missing a "%" sign in front.
Fixes: aaf7e68c75a907c3c08e83dfd2972665a0f1c1a3
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Make sure the function has a way of telling the caller *why* it failed.
This way, the function can convey whether it had an internal error, an
allocation failure, whether the arguments are totaly nonsensical, or
simply that the compressor *or specific configuration* is not supported.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Avoid namespace polution. Make sure all exportet symbols are prefixed
with either sqfs_ or SQFS_.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Instead of creating everything in the "create" function, cleanup and
create/initialize stuff in a "load" function. This allows the xattr
reader to be reset/re-used and adds the benefit of not having to
lug around references to the super block, compressor and file (altough
the later two are hidden inside the meta reader).
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This patch adds a deep-copy callback to sqfs_object_t and removes the
copying mechanism from sqfs_compressor_t. This is also interesting for
other types.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This allows getting the compressor configuration back after
creating it, for various purposes.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
With unified payload size counters, copying an inode is now trivial.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
If the block processor allocates and dynamically resizes inodes on
the fly, we can add data indefinitely without knowing the size of
the file ahead of time.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Instead of having seperate counters for blocks, dir index bytes
and having to fiddle out the link target size, simply use a single
value that stores the number of payload bytes used.
A seperate "payload bytes available" is used for dynamically
growing inodes during processing.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This function waits for all pending blocks to be written to disk, but
doesn't flush the fragment block, so processing can continue afterwards
as if nothing happened.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This commit removes the allocation helpers and string table functions
out of libsquashfs back into a "libutil.a". The problem of libsquashfs
exporting stuff that it shouldn't is resolved by retaining the internal
attributes and directly adding the source to libsquashfs instead of
trying to somehow link against libutil.la.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This commit moves all of the fragment/block accounting in the block
processor over to the writing end of the pipeline. In order to do
this, the sparse blocks are allowed to bubble through the pipeline.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Make every dynamically allocated, opaque data structure inherit from
a common sqfs_object_t structure with common entry points (e.g. destroy).
This removes tons of public API functions and replaces them with a
simple sqfs_destroy instead. If semantics of the (until now implicit)
object system need to be extended, it can be much more conveniantely
done this way.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
It was basically built around the block processor and exposed way too
many internals. Removing it from other places was mostly trivial. This
commit completely removes it from the public API and even most of the
libsquashfs internals.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
