summaryrefslogtreecommitdiff
AgeCommit message (Collapse)Author
2019-12-22Add basic support for handling and serializing hard linksDavid Oberhollenzer
In libfstree, add a function to add a hard link to the fstree. The hard links stores the target in the data.target field, canonicalizes the target and sets a sentinel mode. A second function is used to resolve link, i.e. replacing it with a direct pointer, setting another sentinel mode and increasing the targets link count. The post process function tries to resolve unresolved hard links and only allocates inode numbers for nodes that aren't hard links. If the target node of a hard link does not have an inode number yet, the two need to be swapped, since this is also the order in which they are serialized. The serialization function in libcommon simply has to skip hard link nodes and when writing directory entries, use the inode num/ref of the target node. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-19Split the libfstree add_by_path tree traversal function outDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-19Fix link count for file inodes in tree serialization.David Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-19Fix spelling mistakes found by lintianBenjamin Drung
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
2019-12-18Add an explicit link count to the fstree nodesDavid Oberhollenzer
Gets initialized to 2 for directories, 1 for all other types. The count of the parent node is automatically incremented. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-18Rename fstree "slink_target" to "target"David Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-18Move is_filename_sane to libfstree, add test casesDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-18Cleanup: internalize some fstree functionsDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-18Cleanup: remove the unused "block_size" from the fstreeDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-18Cleanup: merge the fstree post processing functionsDavid Oberhollenzer
Instead of having 3 different functions for sorting the tree, numbering the nodes and generating a file list, that all have to be used in the right order, this commit merges them into a single "fstree_post_process" function. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-16Remove fstree inode tableDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-16Do the fstree serialization in a recursive tree walkDavid Oberhollenzer
Remove usage of the "inode table". Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-16Use the sqfs_dir_writer_t to create the NFS export tableDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-16Add ability to sqfs_dir_writer to create an export tableDavid Oberhollenzer
If the dir writer is used to create the directory table, it neccessarily sees every single inode number and coresponding location for all inodes that are referenced by the filesystem tree. This means it can easily collect that information internally to create an export table later on. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-16Cleanup: split sqfs_serialize_fstree into smaller functionsDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-16Corpora tests: fix behaviour if SOURCE_DATE_EPOCH is setDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-15Add a test script that turns all test tarballs into SquashFS imagesDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-15Clamp mtime in tar2sqfs to the representable 32 bit range.David Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-15Fix tar GNU sparse header parsingDavid Oberhollenzer
Extract the filename correctly. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-14Add a test case for reading binary xattrs from pax/schily headersDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-13rdsquashfs: print hex dump of xattrs if it isn't an ASCII/UTF-8 stringDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-13Better support for reading/writing non-ASCII xattr values from/to tarDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-13Make the PAX header parser more strictDavid Oberhollenzer
Actually parse the length field and insist it matches the line length, then use the length field to advance to the next line. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-13Update CHANGELOG.mdDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-13Fix name of libsquashfs pkg-config fileDavid Oberhollenzer
This fixes both the name inside the file, as well as the file name by adding the major version suffix. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-13Fix help text for --with-lz4David Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-13Add a simplistic integration and regression testDavid Oberhollenzer
This test basically consists of the Canterbury Corpus which is turned into a SquashFS image using every supported compressor, with every supported block size, with and without tail end packing. The results MUST have an exact, given sha512sum. If that changes, it means either a regression in the output format, something broke or non-deterministic packing behaviour. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-13Merge windows and pthread thread pool implementationsDavid Oberhollenzer
Since they are both structured the same way using condition variables, they are only a few defines away from removing code duplication. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-13Cleanup data writerDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-13Add --no-tail-packing option to gensquashfs, tar2sqfsDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-12Don't kick off the threads until the queue is fullDavid Oberhollenzer
The idea is as follows: Initially let the user submit blocks until the queue is filled, then kick of the threads. Every thread will end up getting a block without any waits until they completely deplete the queue. Assuming the threads take longer to process the data than it takes the main thread to do I/O and submit new blocks, the queue should stay mostly filled with minimal wait times. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-12Fix thread pool queue accountingDavid Oberhollenzer
- ONLY manipulate the back log counter in the main thread. - Fix the order of operations when submitting blocks. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-12Fix out of bounds writes in lzo compressorDavid Oberhollenzer
The liblzo compression functions don't do any bounds checking, instead they expect the destination buffer to be large enough to hold the worst case encoding. This commit modifies the lzo compressor to use a scratch buffer for compression (with worst case size) and only copy to the actual destination if the result fits. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-12Fix "buffer to small" being treated as error in zstd compressorDavid Oberhollenzer
The zstd compress function returns an error code if it cannot fit the compressed data into the given destination buffer. This commit adds a check for this error and reports that the libsquashfs compressor implementation was unable to shrink the input instead of claiming a fatal error happened. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-12Fix: programs linking against libsquashfs also need pthreadDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-09Add github issue numbers to CHANGELOG.mdDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-09Update CHANGELOG.mdDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-09Only check for OS specific bad filenames when unpackingDavid Oberhollenzer
When converting a SquashFS image to a tarball, it makes no sense to refuse conversion if the filename is considered evil by the OS. This patch adds an option to is_filename_sane to check if the OS has a problem with the given file name. sqfs2tar sets it to false and converts everything while rdsquashfs sets it to true when unpacking. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-08Replace SetEvent synchronization with condition variablesDavid Oberhollenzer
It's cleaner, more stable and works pretty much the same way as the pthread version. The downside is that the minimum target for the library is now Windows Vista, or Server 2008. But both are over a decade old anyway, so this shouldn't be an issue. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-08Fix Windows file creation modeDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-08Add native Windows port of the multi-threaded data writerDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-05Minor data writer cleanupDavid Oberhollenzer
Move "do block" function over to the rest of the block related code and internalizie the pthread worker structure. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-05Fix pthread data writer interfering with signal handlingDavid Oberhollenzer
If a signal is sent to a process, the POSIX spec says that ANY thread could be picked ARBITRARILY to handle the signal. In our case this could be one of the internal worker threads. The problem here is that an unsuspecting user of the library might suddenly have their signal handler run in parallel to their main thread and run into weird concurrency issues, because they didn't expect that to happen. In fact, the libsquashfs API tries to be transparent about whether or not it uses a thread pool internally and does everything other than number crunching (e.g. I/O that may happen through user supplied callbacks) in the same thread as the one that submitts the blocks. Unfortunately, pthread doesn't have a way to set a signal mask for the new thread and setting it inside the thread is racy (i.e. a signal might be delivered before the worker thread sets the mask). The only portable and non-racy way to do this, is to disable all signals in the calling thread that sets up the data writer, create the threads (which will inherit the mask) and then resetore the previous signal mask, hoping for the best. The downside to this is that signals may be lost in that short time. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-02Update man pagesDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-02Add a --root-becomes option to tar2sqfs and sqfs2tarDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-01Update CHANGELOG.mdDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-11-28Cleanup: Return combined return value from compressor id by nameDavid Oberhollenzer
Instead of returning the ID through a pointer and an error code as return status, return a single int that could be a compressor ID (positive values) or an error code (negative values). Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-11-25Polish up that configure scriptDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-11-25Cleanup: remove what is left of libutilDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-11-25Cleanup: move overflow safe alloc code into libsquashfsDavid Oberhollenzer
There were only a hand full of instances outside libsquashfs that used the alloc code. In most cases, the thing allocated hat its size derived from something already in memory anyway, so it is safe to assume its size fits into a size_t. At the same time, the opencoded Windows path conversion functions are all unified into a single helper function. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
generated by cgit v1.2.3 (git 2.39.1) at 2024-11-01 21:37:50 +0000