aboutsummaryrefslogtreecommitdiff
AgeCommit message (Collapse)Author
2020-04-27Cleanup/fix: gensquashfs: split directory scanning from xattr scanningDavid Oberhollenzer
On the one hand, this commit cleanes the code a bit by splitting the "scan directory contents" code from the "scan xattrs from directory contents" and moving the later in a seperate file. On the other hand, the xattr scanning is now done *after* the fstree is post processed, which includes sorting it. This way, the xattrs are always added in a deterministic, reproducible order. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-04-27Enable uint128_t pathMatt Turner
I forgot to enable this when I copied it over from Mesa. Mesa's meson configuration system checks that a C program using the uint128_t type compiles, but I think this is likely unnecessary. Simply check the macro that clang and gcc define. This cuts the .text size of hash_table.o by 160 bytes or about 4% on my system. Signed-off-by: Matt Turner <mattst88@gmail.com>
2020-04-27gensquashfs: Add options to globally override UID/GID valuesDavid Oberhollenzer
A common use case for mksquashfs is to simply pack a directory and set a magic option to force all user/group IDs to root. This commit adds similar options to gensquashfs to maek it better suited as a direct replacement for packing an input directory. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-04-27Add hash table code to libutil.aDavid Oberhollenzer
Not only does this build the hashtable into libutil.a, it also makes sure the headers end up in the distribution tarball. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-04-27Add propper license text for Mesa hash table implementationDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-04-27Cleanup directory structure of the binary programsDavid Oberhollenzer
Instead of having the binary programs in randomly named subdirectories, move all of them to a "bin" subdirectory, similar to the utility libraries that have subdirectories within "lib" and give the subdirectories the propper names (e.g. have gensquashfs source in a directory *actually* named "gensquashfs"). Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-04-27Fix tar2sqfs: Actually apply the block processor flagsDavid Oberhollenzer
This fixes a bug in tar2sqfs where the -T option has no effect, because the block processor flags were propperly generated, but not passed on. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-04-23Update benchmark dataDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-04-22Import and use Mesa's hash tableMatt Turner
With `perf record`/`perf report` I saw that 30% of the time was spent in `sqfs_frag_table_find_tail_end` with tar2sqfs for a tarball containing the Gentoo ebuild repository (many thousands of small files). The reason was the bucketing hash table in frag_table.c: too many elements in too few buckets meant lots of walking over the linked lists. This patch replaces that hash table with the hash table implementation from Mesa. Its implementation is more complex (is is an open-addressing, linear-reprobing) hash table, but it is much better suited for the task. On my 4c/8t Skylake, the time to run tar2sqfs drops from 7.5s to less than 3s. CPU usage increases from ~207% to ~356%, presumably indicating an increase in available parallelism due to the removal of the hash table as a bottleneck. The `perf report` profile with this patch shows that the time spent in `sqfs_frag_table_find_tail_end` has dropped from ~30% to 0.01%. Output from ministat: x before + after N Min Max Median Avg Stddev x 20 7.476 7.685 7.5725 7.5615 0.051254268 + 20 2.79 2.901 2.846 2.84475 0.03543842 Difference at 95.0% confidence -4.71675 +/- 0.0282015 -62.3785% +/- 0.241477% (Student's t, pooled s = 0.0440618) I imported only the bits of the hash table implementation that were needed for frag_table.c. Among the changes I made after importing are - removed usage of ralloc, Mesa's recursive memory allocator - Replaced ralloc -> malloc ralloc_free -> free rzalloc_array -> calloc - Removed mem_ctx parameters - Added free()s to the appropriate places (valgrind confirms there are no leaks) - removed _mesa_-prefix from function names Fixes: #40 Signed-off-by: Matt Turner <mattst88@gmail.com>
2020-04-22COPYING: Fix a couple of typosMatt Turner
Signed-off-by: Matt Turner <mattst88@gmail.com>
2020-04-22Skip PAX global headersDavid Oberhollenzer
Tar archives can contain set two kinds of PAX headers: - local headers that modify the attributes of the next file - global headers that set defaults for all files The later is used "... not widely used", according to tar(5) and has been deliberately not implemented. Some programs (e.g. git-archive) *do* generate them (in the case of git, it stores the commit hash). This commit adds a code path that skips a PAX global header entirely and resumes tar parsing, instead of erroneusly reporting it as an entry. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-04-21Fix: add missing --with-gzip/--without-gzip configure handleDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-04-17Remove some configure time sizeof checksDavid Oberhollenzer
In libtar, the sizeof time_t checked when trying to store a time value. It is pointless using the preprocessor here, as we can simply do an if (sizeof(time_t) < ...) check and the compiler will take care optimizing away one or the other branch. After changing the libtar check and the corresponding unit tests, the sizeof check can be removed from configure.ac, along with other unused sizeof checks. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-04-17Cleanup: split read_header.c in libtar.aDavid Oberhollenzer
Simply moving the pax header decoding to a separate file and splitting out the common helper functions should be a good start. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-04-17tests: improve diagnostics output & make "release builds" workDavid Oberhollenzer
This commit adds a few macros and helper functions for the unit test programs. Those are used instead of asserts to provide more fine grained diagnostics on the one hand and on the other hand because they also work if NDEBUG is defined, unlike asserts that get eliminated in that case. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-04-16tar2sqfs & gensquashfs: Delete the output file on failureDavid Oberhollenzer
This commit changes the tar2sqfs & gensquashfs code to pass the exit status on to sqfs_writer_cleanup in libcommon. The function sqfs writer code in libcommon is changed to retain the output file name and delete it if the status passed to the cleanup function is anything other than EXIT_SUCCESS. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-04-16Propperly cast void pointer in sqfs_object_t inline functionDavid Oberhollenzer
Otherwise, C++ compilers will scream. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-04-16sqfs2tar: Fix trailing slashes for directory namesDavid Oberhollenzer
sqfs2tar is supposed to append slashes to directory names. Until now, it assumed a tree node to be a directory if it has children. This simple check obviously fails for empty directories. This commit fixes the check by actually testing the inode mode. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-04-01Fix missing header without LZOAlyssa Ross
lib/common/compress.c: In function 'compressor_get_default': lib/common/compress.c:39:2: warning: implicit declaration of function 'assert' [-Wimplicit-function-declaration] 39 | assert(0); | ^~~~~~ lib/common/compress.c:8:1: note: 'assert' is defined in header '<assert.h>'; did you forget to '#include <assert.h>'? 7 | #include "common.h" +++ |+#include <assert.h> 8 |
2020-03-30Bump release date for 0.9, fix version number formatingv0.9David Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-03-25Remove mingw based cross build from travis configDavid Oberhollenzer
- It can't find the headers on the travis CI machines, but works perfectly on the Ubuntu Bionic VM I set up localy. Don't know why yet. - The mkwinbins.sh script runs the unit tests through wine. I don't want to remove that from the script but I also don't want to install all of wine on the CI machines for every build. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-03-23Fix dependencies for mingw CI buildDavid Oberhollenzer
Now who would have thought we're gonna need some headers and libraries if we actually want to build software. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-03-22Add Windows cross build and S390 targets to the travis CI fileDavid Oberhollenzer
Why the windows cross build should be tested should be fairly self explanatory. The SYSTEM/390 build offers the possibillity to test on a big-endian target (besides being a rather uncommon target machine). Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-03-22Do not try to read back the compressor optionsDavid Oberhollenzer
None of the currently implemented compressors do anything with that data. They are all at the mercy of the data actually in the image. This commit removes the code from sqfs2tar and rdsquashfs that decodes the options, which also has the side effect of increasing compatibillity with some non-confirming images. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-03-22doxygen: fix some minor typosDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-03-22doxygen: add rudmentary main page to the API reference manualDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-03-22doxygen: propperly label generic inode helper functions as membersDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-03-22Update Doxygen fileDavid Oberhollenzer
Add description and example directory. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-03-22Bump version numberDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-03-22mkwinbins.sh: don't hard code version numberDavid Oberhollenzer
Use the same trick as the coverity script. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-03-22Update CHANGELOG.mdDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-03-20Overhaul README.mdDavid Oberhollenzer
Rewrite introduction. Move copyright section further up. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-03-19Fix: properly terminate the getopt_long arraysDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-03-19Fix compressor availability check in libcommonDavid Oberhollenzer
Initialize have_compressor to false before testing, to make the check work. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-03-19Fix destruction of NULL pointer in xattr reader cleanupDavid Oberhollenzer
This fixes a copy and paste error in the cleanup path, destroying a previously destroyed object again instead of the one being tested for. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-03-19Add references to the external testing & CI servicesDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-03-19Add a script to semi-automate coverity scan submissionsDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-03-19Add macOS workaround for lsetxattrDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-03-19Fix pthread_join check for valid thread handlesDavid Oberhollenzer
On Linux, checking for > 0 worked because pthread_t is internally an integer type. On other platforms (*caugh* Mac OS X *caugh*), it is typedefed to an opaque pointer, causing a warning if used in an integer relational comparison. The intended use is to allow the generic cleanup function to be used in the error path of the block processor creation function, while preventing pthread_join being called on threads that haven't been created at all. Since they are calloc'ed to 0, testing for non-zero values should suffice in both cases. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-03-19Fix xattr scanning on Mac OS XDavid Oberhollenzer
Mac OS X does not have llistxattr or lgetxattr. Instead, the listxattr and getxattr functions have additional an flag parameter that can be set to not follow symlinks. This commit adds a pre-processor define on OS X as a work around. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-03-19Improve travis CI build configurationDavid Oberhollenzer
- Build for arm64 and ppc64el as well as amd64. - Use a newer Ubuntu version. - Add a separate build for the serial (non-threaded) block processor. - Add a Mac OS X target. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-03-19Fix: don't use Linux specific st_mtim stat memberDavid Oberhollenzer
Always access the st_mtime field, which on Linux is #defined to a member of st_mtim. This fixes a travis CI build failure on Mac OS X. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-03-18Set a minimal libzstd version in autoconf scriptDavid Oberhollenzer
Version 1.3.1 is the first one which has fixed error codes exposed through the public API, which are used in the zstd compressor to determine whether compression *actually* failed or if the destination buffer was too small. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-03-18Fix build of lz4 compressor with older versions of liblz4David Oberhollenzer
Older versions of liblz4 don't define LZ4HC_CLEVEL_MAX. This commit adds a definition if liblz4 doesn't provide one. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-03-18Add basic travis CI configurationDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-03-18Add a test case for xxhash32David Oberhollenzer
Technically the code was imported by a third party library, but some modifications have been made. This commit adds a simple test case with some test vectors and expected results that have to match. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-03-18Add a test for the minimal rb-tree implementationDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-03-18Restore workaround for unaligned reads in xxhashDavid Oberhollenzer
The code was originally used inside the block processor, where 32 bit aligned data could be guaranteed. If it is available in libutil, I cannot possibly guarantee for alignment in future use elsewhere. Even for the block processor it was rather risky "remember this detail very well" buisness. This commit restores the unaligned read treatment of the original. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-03-18Cleanup: Move xxhash32 code to libutilDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-03-10Fix the printf format specifies (again)David Oberhollenzer
The PRIu64 et al are missing a "%" sign in front. Fixes: aaf7e68c75a907c3c08e83dfd2972665a0f1c1a3 Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
generated by cgit v1.2.3 (git 2.39.1) at 2024-12-29 18:43:52 +0000