| Age | Commit message (Collapse) | Author |
|---|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This change removes the need for passing a list of files around for
deduplication. Also the deduplication code no longer needs to worry
about order, since the file being deduplicated is only added after
deduplication is done.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This commit moves the file unpacking order & job scheduling to a libfstree
function. The ordering is improved by making sure fragment blocks are not
extracted more than once and files with data blocks are extracted in order.
This way, serial unpacking of a 2GiB Debian live image could be reduced
from ~5' on my test machine to ~3.5', whereas parallel unpacking stays
roughly the same (~3' for -j 4).
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
If -DNDEBUG is set, the entire thing is omitted from the output.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
reproducible-builds.org suggests the use of an environment variable
as a source for time stamps:
https://reproducible-builds.org/specs/source-date-epoch/
This commit adds support for setting the default mtime from the variable,
if it is set and only defaulting to 0 if not. The timestamp given by the
command line switch takes precedence.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Since it is actually completely independend of libsqfs and only works
on file_info_t lists, it can be safely moved over to libfstree and
the data writer becomes less cluttered as a result.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
For merging, the use of a pointer to a pointer can simplify linked list
operations
For sorting, find the half-way point of the list in a single iteration
over the list
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Despite having a flag for that now, they still need to be initialized
because they are written straight to disk.
Fixes: d4d1854aaed867d28ebfc97afb3518254ab6fd4b
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Simplifies some task if we can just add a flag that a file has a framgent
or that it has already been detected as a duplicate.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Instead of doing DFS on the fly in gensquashfs, churn out a linked list
of all files in an archive.
Future improvements in packing strategies can go into this file.
This can also be usefull for other purposes in the future, such as file
deduplication or as a work queue for the unpacker.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This commit removes all the code for parsing and processing atime/ctime
and values and related test code.
Caring about those is kind of pointless because squashfs can only store
mtime in inodes. The only relevant place is when generating a struct
stat from a squashfs inode or an fstree node.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Requires that config.h be included before other headers, since the macro
_FILE_OFFSET_BITS changes the definitions of things like 'struct stat'.
I chose to simply include it at the top of every C file and at
immediately after the double-inclusion guards of every header.
Signed-off-by: Matt Turner <mattst88@gmail.com>
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
First of all, this commit adds a mod_time field to a tree node. When
creating the tree node, the field is set from the struct stat. When
scanning a directory, the time stamps from the input are used if set.
Second, the libsqfs code that reads inodes is modified to store the
mod_time from the inode in the fstree node and to write the tree node
into a generated inode.
Finally, tar2sqfs is modified to optionally keep the timestamps from
the tar archive instead of setting defaults. gensquashfs is similarly
modified to keep the input timestamps if specified.
The result is as follows:
- sqfs2tar will always carry the timestamps from the squashfs over
to the tar ball.
- tar2sqfs will set defaults, unless explicitly asked to preserve
the mtime from the tar ball.
- gensquashfs can optionally preserve the mtime from the input
hierarchy it processes if only --pack-dir is specified.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
- Store them in a struct instead of a hacky uint64_t with magic shifts
- Split up key/value pair write function to write_key and write_value
- Move the size accounting into those functions respectively
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This commit adds a reference count functionality to the string table
implementation and uses this functionality in the fstree code to
count how often each key and value is referenced by the deduplicated
Xattr blocks. This is needed to support deduplication through
out-of-band storage of xattrs.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This commit adds a mechanism to fstree_from_file to support filenames
with spaces in them by quoting the entire string. Quote marks can still
be used inside file names by escaping them with a backslash. Back slashes
(if that is your thing) can also be escaped.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This commit makes sure that regular file tree nodes have one more slot
than necessary to support files that don't have fragments.
Also, it initializes the fragment data, which should help to deduplicate
some code ahead.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
The string is actually always null-terminated, since the calloc above
adds one more byte than what we tell readlink about.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
The format is documented as "<c|b> major minor" but the parser was
accidentally changed to require a colon in between.
Fixes: 9864ea5b2045f4bd72633152d71dd1c7f8b0b7f9
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Bug found using scan-build.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Should the inode table theoretically be empty, the function returns an
undefined value. This commit fixes that case by initializing the return
status to 0, so it returns success status in that case.
Bug found using scan-build.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
The refactor of the xattr table grow code merged all allocation code
paths into realloc(), including the initial allocation. This means that
the xattr structure is used uninitialized. This commit makes sure the
reallocated structure is alwayes cleared.
Bug found using scan-build.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Most code path that use 'extra' allocate it using calloc/malloc/strdup
and then converge into a common path that uses free() on extra.
This commit removes a stray one that uses alloca for symlink targets,
so we don't free() a stack buffer further down in the common path.
Bug found using scan-build.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This IMO makes it somewhat easier to read and understand what's going on.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Instead of decomposing a default string in gensquashfs option processing,
move that to fstree_init instead and pass the option string directly to
fstree_init.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This commit removes the packdir/packfile based directory setup magic
from fstree_from_file and moves it to gensquashfs. Over there, the
common parts are deduplicated.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Make tree node list sort and recursive variant available and independend
of the fstree_t.
This is considered cleaner, since the fstree_t actually isn't needed for
any of this and we can just call the recusvie sort on the root instead,
and we can use the sort implementation directly for things like the
upcoming unit test.
Also this commit splits up the merge/sort implementation into a seperate
split and merge functions to make the code somewhat more readable.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This commit moves the SELinux label code after the tree is sorted and
the inode table is generated. Sorting helps to make sure that the tree
will always be traversed in a defined, deterministic order and likewise
the creation of xattrs happens in a defined, deterministic order.
Second, we can now use the inode table instead of having to implement a
recursive tree traversal yet again.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
When creating a directory that has previously been created implicitly,
actually update the permissions as the documentation says.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Replace default_$FIELD with a struct stat. Merge the rest of
add_node/add_file into mknode and add by path+stat.
Expose the mknode function and replace all the duplicated node pointer
arithmetic magic with a call to mknode.
Plus a generic cleanup by utlizing new util functions.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
- use generic struct stat based tree node add function
- simplify some of the parsing code
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Instead of allocating inode numbers as we go, generate and populat an
inode table from the fstree ahead of time. This makes processing nodes
a little bit simpler and we will need that table anyway for NFS export
support later on.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This commit replaces the chdir to the input directory with pushd/popd
when building the fstree and again when packing files.
This simplifies handling of other file paths given on the command line
that have to be accessed and are relative to the original working
directories.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
If packdir and packfile are both specified, use packdir as
alternate root.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This makes path handling a lot easier. We no longer need to paste paths
together to get input file relative paths, since we are in the same
directory.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
