| Age | Commit message (Collapse) | Author |
|---|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Move all the libutil stuff from the toplevel include/ to a util/
sub directory and fix up the includes that make use of them.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
The squashfs on-disk format uses 32 bit link counters, but the fstree
used 16 bit ones. Because the link count also includes child nodes,
this artificially limited the number of entries in a directory to ~64k
files.
This patch removes the limit by switching libfstree to 32 bit counters.
Reported-by: Marvin Renich <mrvn@renich.org>
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Split out several repated patterns into helper functions and move the
rest of the code back into dir_reader.c
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Add a simple directory state object to the meta data reader and use
that to iterate directory entries. The code for reading the directory
listing is movde to readdir.c
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Two flags are added to the dir reader API, one for the create function
that the dir reader should report those entries and one to the open
function to suppress that if it was enabled.
To implement the feature, a mapping of visited directory inodes is
maintained internally, that mapps inode numbers to inode references.
When opening a directory, state is maintained to generate the fake
entries for '.' and '..'. Since all the other functions are based on
the open/read/rewind API, no alterations need to be made. The tree
scan function is modified, to use the suppress flag, so it does not
accidentally catch those entries.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
The idea was originally to use struct stat in the libfstree code, so
we can simply hose data read from a directory into the fstree_t. The
struct was then also used with libtar, for simpler interoperation,
but it turned out to introduce a lot of platform quirks and causes
more trouble than it's worth.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
A `flags` field and `priority` are added to all file information
structs. A news fstree function is introduced for parsing a "sort-file".
Each line in the file is space separated, and has the following format:
priority [flags] filename
Priority is a 64 bit number, flags are optional and filename can be put
in quotes if it is supposed to start or end with spaces. Single line
comments can be used.
The flags can be used to set block-processor flags (e.g. don't fragment,
or don't compress), as well as instructing the parser to use file
globbing to match the filename.
After parsing the file, the list of file info structure is sorted
according to the priority (default is 0) using a stable sort algorithm.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Many library destructor functions (like free()) allow a NULL
pointer as input, and do nothing in that case.
This allows easier cleanup patterns: initialize pointers to NULL
and then always pass them to the destroyer functions, no need for
verbose goto/if-else patterns.
Signed-off-by: Luca Boccassi <luca.boccassi@microsoft.com>
|
|
Preprocessor magic is used to redirect putc/fputc/fputs/printf/fprintf
to custom implementations.
The custom implementations try to figure out if we are printing to the
console and, if so, convert the resulting strings to UTF-16 and print
them through ConsoleWriteW. If the output is redirected to a file or
a pipe, the original (presummed) UTF-8 is kept.
Simply setting the console output codepage to UTF-8 does not work,
because the standard I/O facilities of MSVCRT either does not support
unicode (in non-wchar mode), or has half-broken support through fputs,
which can still break up multi-byte sequences through its internal
buffering.
Likewise, changing the codepage and using ConsoleWriteA, or trying to
use fputws did not work in a test VM either.
This approach is the one that worked most consistently among the
ones tried, but also has problems. E.g. it breaks when setting the
codepage to UTF-8 manually (using `chcp 65001`).
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
When opening files on windows, use the widechar versions and convert
from (assumed) UTF-8 to UTF-16 as needed.
Since the broken, code-page-random API may acutall be intended in some
use cases, leave that option in through an additional flag.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
A macro and forward declaration are added to compat.h that rename
the main() function programs using compat.h into sqfs_tools_main.
An actual main() function is added to libcompat.a, that uses the
shell API to get the UTF-16 command line arguments, convert them
to UTF-8 and call sqfs_tools_main.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
The squashfs readdir() implementation in the Linux kernel returns
non-existing "." and ".." entries for offsets 0 and 1, and after
that reads from disk. For convenience, it was decided to store an
off-by-3 value on disk instead of doing complex primary school math
to adjust for this. This didn't show up until now, because the kernel
implementation trusts the value from the directory header more than
the actual size in the inode and happily reads 3 more than the inode
would allow it to. This only showed up with 7-zip which subtracts 3
from the size and expects the result to be exact and bails if the
directory headers suggest otherwise.
And yes, I did consider making a "Holy Hand Granade of Antioch"
reference, but consciously decided not to.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
|
|
If SQFS_STATIC is defined, dummy out the SQFS_API definition, so we
don't try to pull stuff from a (in this case) non-existant DLL or
try to export functions.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
On systems like Windows, the dynamic library and applications can
easily end up being linked against different runtime libraries, so
applications cannot be expected to be able to free() any malloc'd
pointer that the library returns.
This commit adds an sqfs_free function so the application can pass
pointers back to the library to call the correct free() implementation.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
A cleaner separation between common code, frontend code and backend
code is made.
The "is this byte blob zero" function is moved out to libutil (with
test case and everything) with a more optimized implementation.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
The thread pool enforces ordering of items during dequeue similar
to the already existing implementation in libsqfs. The idea is to
eventually pull this functionality out of the block processor and
turn it into a cleaner, separately tested module.
The thread pool is implemented as an abstract interface, so we can
have multiple implementations around, including the serial fallback
implementation which we can then *always* test, irregardless of the
compile config and run through static analysis as well.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This has basically been copied over from Musl and slightly modifed.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This commit restructures the rbtree code to optionally use a pool
allocator for the nodes. The option is made depenend on the presence
of a pre-processor flag.
To the configure script is added an option to enable/disable the use
of custom allocators. It makes sense to still allow the malloc/free
based routes for better ASAN based instrumentation.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
The intention is to get rid of all the ad-hoc array implementations
in the other components and cut down code size.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
If we use the rb-tree in libsquashfs objects, we need to be able
top copy an entire tree as part of the object.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
The base path is passed to the fstree_from_file function and in turn
to the individual callbacks.
The line parsing function is modified to allow '*' as mode, uid and gid
for specifically marked callbacks.
A glob callback is added that internally uses the fstree_from_dir scanning
functions in combination with a filter callback.
Directory scanning flags are parsed from the extra arguments before
interpreting it as a path fragment.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
So we can scan a sub-directory within a the base directory without
having to do string operations first.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This function creates a block processor from a structure describing it.
A stub implementation for the old sqfs_block_processor_create is added
that simply sets up such a struct and forwards the call.
The current version of the description struct only contains the exact
same parameters and a size field at the beginning.
This approach is supposed to make extending the range of parameters
easier without breaking ABI compatibillity.
Currently already planned are:
- Adding a sqfs_file_t pointer to double-check when deduplicating
fragments.
- When the scanning code reaches a usable state, add the abillity
to pass scanned fragment data, so the block processor can be used
for appending to an existing image.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
- Instead of using the fstree root, let the caller specify it.
- Add a flag to prevent recursion into sub directories.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Instead of comparing (compresed, disk-size, checksum) tuples to find
block matches, do an exact, byte-for-byte comparison of the data
stored on disk to avoid the possibility of a spurious colision.
Since this is the desired behaviour, make it the default, optionally
overrideable through a flag.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
The sqsf_writer is moved to a separate header, as well as the compressor
related stuff. The statistics function is moved into the writer code, as
this is the only place that actually uses it. The writer code itself is
split up into a hand full of file in their own subdirectory.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
