| Age | Commit message (Collapse) | Author |
|---|
|
Move all the libutil stuff from the toplevel include/ to a util/
sub directory and fix up the includes that make use of them.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Abort and retry in situations that should logically _never_
_ever_ happen.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
- Store the return value of the page allocation directly into the
pool variable instead of an intermediate unsigned char pointer.
- Make the blob[] array the same type as the bitmap, this saves us
manual alignment trickery.
- Cleanup the pointer arithmetic, let the compiler do the
sizeof() multiplication.
- Use uintptr_t for the manual alignment of the data pointer, so we
don't run into signdness problems there.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
When we already hold the mutex, try to pre-emtively dequeue items into
a "safe queue". When actually asked to dequeue, take blocks from there
first and avoid having to enter the critical section if possible.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
A cleaner separation between common code, frontend code and backend
code is made.
The "is this byte blob zero" function is moved out to libutil (with
test case and everything) with a more optimized implementation.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Hopeing that coverity can now tell the two appart.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
The thread pool enforces ordering of items during dequeue similar
to the already existing implementation in libsqfs. The idea is to
eventually pull this functionality out of the block processor and
turn it into a cleaner, separately tested module.
The thread pool is implemented as an abstract interface, so we can
have multiple implementations around, including the serial fallback
implementation which we can then *always* test, irregardless of the
compile config and run through static analysis as well.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This commit restructures the rbtree code to optionally use a pool
allocator for the nodes. The option is made depenend on the presence
of a pre-processor flag.
To the configure script is added an option to enable/disable the use
of custom allocators. It makes sense to still allow the malloc/free
based routes for better ASAN based instrumentation.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
The intention is to get rid of all the ad-hoc array implementations
in the other components and cut down code size.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
If we use the rb-tree in libsquashfs objects, we need to be able
top copy an entire tree as part of the object.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
It does not make any changes to the writer itself, so mark it as
const. This also requires some similar changes to the string table.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
I forgot to enable this when I copied it over from Mesa. Mesa's
meson configuration system checks that a C program using the uint128_t
type compiles, but I think this is likely unnecessary. Simply check the
macro that clang and gcc define.
This cuts the .text size of hash_table.o by 160 bytes or about 4% on my
system.
Signed-off-by: Matt Turner <mattst88@gmail.com>
|
|
Not only does this build the hashtable into libutil.a, it also makes
sure the headers end up in the distribution tarball.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
With `perf record`/`perf report` I saw that 30% of the time was spent in
`sqfs_frag_table_find_tail_end` with tar2sqfs for a tarball containing
the Gentoo ebuild repository (many thousands of small files).
The reason was the bucketing hash table in frag_table.c: too many
elements in too few buckets meant lots of walking over the linked lists.
This patch replaces that hash table with the hash table implementation
from Mesa. Its implementation is more complex (is is an open-addressing,
linear-reprobing) hash table, but it is much better suited for the task.
On my 4c/8t Skylake, the time to run tar2sqfs drops from 7.5s to less
than 3s. CPU usage increases from ~207% to ~356%, presumably indicating
an increase in available parallelism due to the removal of the hash
table as a bottleneck. The `perf report` profile with this patch shows
that the time spent in `sqfs_frag_table_find_tail_end` has dropped from
~30% to 0.01%.
Output from ministat:
x before
+ after
N Min Max Median Avg Stddev
x 20 7.476 7.685 7.5725 7.5615 0.051254268
+ 20 2.79 2.901 2.846 2.84475 0.03543842
Difference at 95.0% confidence
-4.71675 +/- 0.0282015
-62.3785% +/- 0.241477%
(Student's t, pooled s = 0.0440618)
I imported only the bits of the hash table implementation that were
needed for frag_table.c. Among the changes I made after importing are
- removed usage of ralloc, Mesa's recursive memory allocator
- Replaced ralloc -> malloc
ralloc_free -> free
rzalloc_array -> calloc
- Removed mem_ctx parameters
- Added free()s to the appropriate places (valgrind confirms there
are no leaks)
- removed _mesa_-prefix from function names
Fixes: #40
Signed-off-by: Matt Turner <mattst88@gmail.com>
|
|
The code was originally used inside the block processor, where 32 bit
aligned data could be guaranteed. If it is available in libutil, I
cannot possibly guarantee for alignment in future use elsewhere. Even
for the block processor it was rather risky "remember this detail very
well" buisness.
This commit restores the unaligned read treatment of the original.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This commit removes the allocation helpers and string table functions
out of libsquashfs back into a "libutil.a". The problem of libsquashfs
exporting stuff that it shouldn't is resolved by retaining the internal
attributes and directly adding the source to libsquashfs instead of
trying to somehow link against libutil.la.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
There were only a hand full of instances outside libsquashfs that used
the alloc code. In most cases, the thing allocated hat its size derived
from something already in memory anyway, so it is safe to assume its
size fits into a size_t.
At the same time, the opencoded Windows path conversion functions are
all unified into a single helper function.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
"Some" "non-POSIX systems" don't have that.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
The -static-libgcc flag has to be passed through the compiler with
a "-Wc," prefix, because libtool tries to be clever about linker
flags. If added directly to LDFLAGS, libtool removes it.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Its the only user. The other code doesn't touch raw file
descriptors anymore.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
It's only ever used for padding tarballs.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
- str_table_t is used by libsquashfs; Don't write to stderr, report
an error code instead.
- Fix the comments about that and fix the SPDX license identifier
while we're at it.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This commit implements the part of the API responsible for recoding
and deduplicating xattr key-value blocks.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
- We don't have "endian.h" everywhere. On some BSDs its in sys and
on some BSDs the macros have different names.
- We definitely don't have sysmacros.h on non-Unix-like systems.
- Likewise for sys/types.h, sys/stat.h and their contents.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This is a fully automated search and replace, i.e. I ran this:
git grep -l uint8_t | xargs sed -i 's/uint8_t/sqfs_u8/g'
git grep -l uint16_t | xargs sed -i 's/uint16_t/sqfs_u16/g'
git grep -l uint32_t | xargs sed -i 's/uint32_t/sqfs_u32/g'
git grep -l uint64_t | xargs sed -i 's/uint64_t/sqfs_u64/g'
git grep -l int8_t | xargs sed -i 's/int8_t/sqfs_s8/g'
git grep -l int16_t | xargs sed -i 's/int16_t/sqfs_s16/g'
git grep -l int32_t | xargs sed -i 's/int32_t/sqfs_s32/g'
git grep -l int64_t | xargs sed -i 's/int64_t/sqfs_s64/g'
and than added the appropriate definitions to sqfs/predef.h
The whole point being better compatibillity with platforms that may
not have an stdint.h with the propper definitions.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
