aboutsummaryrefslogtreecommitdiff
path: root/lib/sqfs
AgeCommit message (Collapse)Author
2019-12-12Don't kick off the threads until the queue is fullDavid Oberhollenzer
The idea is as follows: Initially let the user submit blocks until the queue is filled, then kick of the threads. Every thread will end up getting a block without any waits until they completely deplete the queue. Assuming the threads take longer to process the data than it takes the main thread to do I/O and submit new blocks, the queue should stay mostly filled with minimal wait times. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-12Fix thread pool queue accountingDavid Oberhollenzer
- ONLY manipulate the back log counter in the main thread. - Fix the order of operations when submitting blocks. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-12Fix "buffer to small" being treated as error in zstd compressorDavid Oberhollenzer
The zstd compress function returns an error code if it cannot fit the compressed data into the given destination buffer. This commit adds a check for this error and reports that the libsquashfs compressor implementation was unable to shrink the input instead of claiming a fatal error happened. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-12Fix: programs linking against libsquashfs also need pthreadDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-08Replace SetEvent synchronization with condition variablesDavid Oberhollenzer
It's cleaner, more stable and works pretty much the same way as the pthread version. The downside is that the minimum target for the library is now Windows Vista, or Server 2008. But both are over a decade old anyway, so this shouldn't be an issue. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-08Fix Windows file creation modeDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-08Add native Windows port of the multi-threaded data writerDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-05Minor data writer cleanupDavid Oberhollenzer
Move "do block" function over to the rest of the block related code and internalizie the pthread worker structure. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-05Fix pthread data writer interfering with signal handlingDavid Oberhollenzer
If a signal is sent to a process, the POSIX spec says that ANY thread could be picked ARBITRARILY to handle the signal. In our case this could be one of the internal worker threads. The problem here is that an unsuspecting user of the library might suddenly have their signal handler run in parallel to their main thread and run into weird concurrency issues, because they didn't expect that to happen. In fact, the libsquashfs API tries to be transparent about whether or not it uses a thread pool internally and does everything other than number crunching (e.g. I/O that may happen through user supplied callbacks) in the same thread as the one that submitts the blocks. Unfortunately, pthread doesn't have a way to set a signal mask for the new thread and setting it inside the thread is racy (i.e. a signal might be delivered before the worker thread sets the mask). The only portable and non-racy way to do this, is to disable all signals in the calling thread that sets up the data writer, create the threads (which will inherit the mask) and then resetore the previous signal mask, hoping for the best. The downside to this is that signals may be lost in that short time. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-11-28Cleanup: Return combined return value from compressor id by nameDavid Oberhollenzer
Instead of returning the ID through a pointer and an error code as return status, return a single int that could be a compressor ID (positive values) or an error code (negative values). Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-11-25Cleanup: remove what is left of libutilDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-11-25Cleanup: move overflow safe alloc code into libsquashfsDavid Oberhollenzer
There were only a hand full of instances outside libsquashfs that used the alloc code. In most cases, the thing allocated hat its size derived from something already in memory anyway, so it is safe to assume its size fits into a size_t. At the same time, the opencoded Windows path conversion functions are all unified into a single helper function. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-11-24Cleanup: completely move str_table into libsquashfs internalsDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-11-24Fix check of block size in super blockDavid Oberhollenzer
The check needs to include 1M, which is still a valid block size. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-11-24Cleanup libsquashfs invalid argument error codesDavid Oberhollenzer
- Add an explicit "you're holding it wrong" error code. - Consistently return error codes and not have some special places where -1 is returned. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-11-24Cleanup: split out sqfs_write_super similar to sqfs_read_superDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-11-24Cleanup: remove the entirely redundant sqfs_has_xattr functionDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-11-24Fix: Move LZO compressor from libsquashfs to libcommonDavid Oberhollenzer
The liblzo2 library is licensed under GPLv2, so it is not possible to distribute binaries of libsquashfs that link against liblzo2 under LGPL. This commit moves the LZO compressor implementation to libcommon, where this isn't a problem, since the tools themselves are licensed under GPLv3. It removes the ability of libsquashfs to read or generate LZO compressed SquashFS images, but the tools still can. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-11-23Move some unix header inclusions to compat.hDavid Oberhollenzer
In most cases, including unistd.h and fcntl.h was a left over anyway. In the cases where it was not, move it to compat.h. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-11-22Cleanup: move all the compatibillity fluff to a dedicated "libcompat"David Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-10-23For Windows builds, remove the so-version, link libgcc staticallyDavid Oberhollenzer
The -static-libgcc flag has to be passed through the compiler with a "-Wc," prefix, because libtool tries to be clever about linker flags. If added directly to LDFLAGS, libtool removes it. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-10-23Fix sqfs_dir_writer_add_entry behaviour on empty stringsDavid Oberhollenzer
SquashFS cannot represent empty string file names in the directory entry structure, so reject them and report an error. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-10-11Add the required libtool magic to build a Windows DLLDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-10-11Remove usage of strchrnul from libsquashfsDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-10-11Add Windows implementation for sqfs_file_tDavid Oberhollenzer
This commit moves the generic unix implementation into a "unix" subdirectory and adds a "win32" subdirectory with a winapi based implementation. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-10-08Fix: propperly clear index entryDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-10-08Fix storing of the directory index in the inode, size accountingDavid Oberhollenzer
After the last modification, the directory index size counter no longer counts the size, but the number of index entries. To avoid future confusion, remove the thing entirely and compute the size on the fly. Furthermore, eliminate unaligned memory writes and fix the darn counter once again. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-10-08Fix recovery of directory index in inode readerDavid Oberhollenzer
Apparently mksquashfs writes an actual usage count (1 = 1 entry, 2 = 2 entries; i.e. NOT off by one). Also, if it does happen to be garbage, guard against an overflow. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-10-08Fix unaligned reads in write_dir_indexDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-10-07Cleanup: consistent, fixed size data types for generic flag fieldsDavid Oberhollenzer
Instead of using int or unsigned int for generic function flag arguments, consistently use an unsigned, fixed size type. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-10-07Cleanup: move libutil related headers to "util" sub directoryDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-10-07Fix spelling of "align"David Oberhollenzer
Before the misspelled version has a chance to become stable API. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-10-07Improve ABI backward & forward compatibillityDavid Oberhollenzer
- Padd the compressor config union - 128 bytes aught to be enough for everyone, i.e. future compressors. - Insist that the padding space is initialized to 0. If a field gets added to an existing compressor, it can test for 0 as a sentinel value. - Add a size field to the hook structure, aka "the Microsoft way". - The explanation is in the comment. - Don't make the Microsoft mistake of checking for >=, insist on *exact* size match. Future users will need a fallback if their hooks are rejected. But at least they will be rejected instead of silently not being used. - Add an unsupported flag check to the dir tree reader. - Add a basic abi unit test that, for now, checks the size of the compressor config struct fields. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-10-06Fix resource leaks reported by coverityDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-10-03Remove accidental internal use of stdint typesDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-10-03Add a counter-part to sqfs_inode_get_xattr_indexDavid Oberhollenzer
Combines all the type depenend attribute setting and inode type promotion into a single function. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-10-03Store directory index in generic inode structureDavid Oberhollenzer
This commit modifies the dir writer to store the directory index directly in the payload area of the inode it creates. The size of the index in bytes is stored in a seperate field. Storing the index is moved to the write inode function. The read inode function is in turn also modified to read the directory index from disk and actually store it in the inode structure in memory for later use. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-09-30Add flag field to directory writer begin for future expansionDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-09-29Cleanup: rename "compress.h" to "compressor.h"David Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-09-29Fix inode writer: do not store the file mode, only permissionsDavid Oberhollenzer
SquashFS restores the file mode from the type, so those bits are completely redundant. On a side note: this also reduces entropy, allowing the compressor to *maybe* squeeze a few more bytes out. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-09-29Bring back the don't fragment flagDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-09-29Make the data writer padding hookableDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-09-29Fix data writer: remember where we stored the paddingDavid Oberhollenzer
If we don't store the padding location in the deduplication list, the block deduplication might try to deduplicate across two files with padding in between, which can't be done. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-09-28Fix xattr writer: don't generate an xattr table if there is noneDavid Oberhollenzer
If there were no key-value pairs recoreded during the lifetime of the xattr writer, take a short cut out of the flush function and set the super block values to indicate that. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-09-28Fix xattr writer: handle the case that no attributes where addedDavid Oberhollenzer
If begin/end was called without recording *any* attributes, simply set the resulting index to the 0xFFFFFFFF sentinel. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-09-28Fix dir writer size accountingDavid Oberhollenzer
Count the *corrent* entry structure size, i.e. the on-disk one, not the in-memory one. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-09-28Fix dir writer: the index size is stored off-by-oneDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-09-28Implement flush function that generates all the tablesDavid Oberhollenzer
Mostly straight copy-pasta from the sqfshelper code, except it doesn't use the fstree functions. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-09-28Add recoding implementation for the xattr writerDavid Oberhollenzer
This commit implements the part of the API responsible for recoding and deduplicating xattr key-value blocks. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-09-28Add header with declarations for the sqfs_xattr_writer_tDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
generated by cgit v1.2.3 (git 2.39.1) at 2024-12-27 12:35:21 +0000