summaryrefslogtreecommitdiff
path: root/lib/sqfs
AgeCommit message (Collapse)Author
2020-01-31Split the block writing/deduplication away from the block processorDavid Oberhollenzer
This commit moves the entire block writing and deduplication of data blocks over to a different data type named "block writer". For simplicity, the interfaces of the block processor are left as is and are turned into warppers. Likewise, most of the code in the block writer is just verbatim from the block processor, to be cleaned up in subsequent commits. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-01-29Rename sqfs_data_writer_t back to sqfs_block_processor_tDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-01-26Cleanup: Move fragment deduplication code over to fragment tableDavid Oberhollenzer
This removes further clutter from the data writer. Any future efforts on making fragment by hash lookup faster can focus on that area only and don't clutter the block processor. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-01-25Add missing headers to installed header listDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-01-24Cleanup: use fragment table primitive in data writerDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-01-24Cleanup: remove single use helper functions from data writerDavid Oberhollenzer
This commit moves the single use helper functions that are called from worker thread context into the worker thread function. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-01-24Cleanup: use the frag table data type in the data readerDavid Oberhollenzer
Instead of doing everything by manually. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-01-24Add a fragment table primitive to libsquashfsDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-01-20Add a flag field to the id table create functionDavid Oberhollenzer
Just to be safe in case there needs to be an extension in the future. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-01-19Cleanup: remove the payload pointers from sqfs_inode_generic_tDavid Oberhollenzer
There are 3 types of extra payload: - Directory index - File block sizes - Symlink target This commit removes the type specific pointers and modifies the code to use the payload area directly. To simplify the file block case and mitigate alignment issues, the type of the extra field is changed to sqfs_u32. For symlink target, the extra field can simply be cast to a character pointer (it had to be cast anyway for most uses). For block sizes, probably the most common usecase, it can be used as is. For directory indices, there is a helper function anyway. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-01-19Add a helper function to unpack directory index entriesDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-01-19Fix directory index accounting when reading inodesDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-01-12Add a start anchor parameter to sqfs_dir_reader_find_by_pathDavid Oberhollenzer
It optionally allows code that does tree traversal to start at an inode that it obtained previously and makes it easier to keep state externally. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-01-12Add an inode deep copy helper function to libsquashfsDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-30Fix zlib paths in automake fileDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-29Add a small version of zlib that can be built in staticallyDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-28Add a small version of liblz4 that can be built in staticallyDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-16Add ability to sqfs_dir_writer to create an export tableDavid Oberhollenzer
If the dir writer is used to create the directory table, it neccessarily sees every single inode number and coresponding location for all inodes that are referenced by the filesystem tree. This means it can easily collect that information internally to create an export table later on. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-13Fix name of libsquashfs pkg-config fileDavid Oberhollenzer
This fixes both the name inside the file, as well as the file name by adding the major version suffix. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-13Merge windows and pthread thread pool implementationsDavid Oberhollenzer
Since they are both structured the same way using condition variables, they are only a few defines away from removing code duplication. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-13Cleanup data writerDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-12Don't kick off the threads until the queue is fullDavid Oberhollenzer
The idea is as follows: Initially let the user submit blocks until the queue is filled, then kick of the threads. Every thread will end up getting a block without any waits until they completely deplete the queue. Assuming the threads take longer to process the data than it takes the main thread to do I/O and submit new blocks, the queue should stay mostly filled with minimal wait times. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-12Fix thread pool queue accountingDavid Oberhollenzer
- ONLY manipulate the back log counter in the main thread. - Fix the order of operations when submitting blocks. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-12Fix "buffer to small" being treated as error in zstd compressorDavid Oberhollenzer
The zstd compress function returns an error code if it cannot fit the compressed data into the given destination buffer. This commit adds a check for this error and reports that the libsquashfs compressor implementation was unable to shrink the input instead of claiming a fatal error happened. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-12Fix: programs linking against libsquashfs also need pthreadDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-08Replace SetEvent synchronization with condition variablesDavid Oberhollenzer
It's cleaner, more stable and works pretty much the same way as the pthread version. The downside is that the minimum target for the library is now Windows Vista, or Server 2008. But both are over a decade old anyway, so this shouldn't be an issue. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-08Fix Windows file creation modeDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-08Add native Windows port of the multi-threaded data writerDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-05Minor data writer cleanupDavid Oberhollenzer
Move "do block" function over to the rest of the block related code and internalizie the pthread worker structure. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-05Fix pthread data writer interfering with signal handlingDavid Oberhollenzer
If a signal is sent to a process, the POSIX spec says that ANY thread could be picked ARBITRARILY to handle the signal. In our case this could be one of the internal worker threads. The problem here is that an unsuspecting user of the library might suddenly have their signal handler run in parallel to their main thread and run into weird concurrency issues, because they didn't expect that to happen. In fact, the libsquashfs API tries to be transparent about whether or not it uses a thread pool internally and does everything other than number crunching (e.g. I/O that may happen through user supplied callbacks) in the same thread as the one that submitts the blocks. Unfortunately, pthread doesn't have a way to set a signal mask for the new thread and setting it inside the thread is racy (i.e. a signal might be delivered before the worker thread sets the mask). The only portable and non-racy way to do this, is to disable all signals in the calling thread that sets up the data writer, create the threads (which will inherit the mask) and then resetore the previous signal mask, hoping for the best. The downside to this is that signals may be lost in that short time. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-11-28Cleanup: Return combined return value from compressor id by nameDavid Oberhollenzer
Instead of returning the ID through a pointer and an error code as return status, return a single int that could be a compressor ID (positive values) or an error code (negative values). Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-11-25Cleanup: remove what is left of libutilDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-11-25Cleanup: move overflow safe alloc code into libsquashfsDavid Oberhollenzer
There were only a hand full of instances outside libsquashfs that used the alloc code. In most cases, the thing allocated hat its size derived from something already in memory anyway, so it is safe to assume its size fits into a size_t. At the same time, the opencoded Windows path conversion functions are all unified into a single helper function. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-11-24Cleanup: completely move str_table into libsquashfs internalsDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-11-24Fix check of block size in super blockDavid Oberhollenzer
The check needs to include 1M, which is still a valid block size. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-11-24Cleanup libsquashfs invalid argument error codesDavid Oberhollenzer
- Add an explicit "you're holding it wrong" error code. - Consistently return error codes and not have some special places where -1 is returned. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-11-24Cleanup: split out sqfs_write_super similar to sqfs_read_superDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-11-24Cleanup: remove the entirely redundant sqfs_has_xattr functionDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-11-24Fix: Move LZO compressor from libsquashfs to libcommonDavid Oberhollenzer
The liblzo2 library is licensed under GPLv2, so it is not possible to distribute binaries of libsquashfs that link against liblzo2 under LGPL. This commit moves the LZO compressor implementation to libcommon, where this isn't a problem, since the tools themselves are licensed under GPLv3. It removes the ability of libsquashfs to read or generate LZO compressed SquashFS images, but the tools still can. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-11-23Move some unix header inclusions to compat.hDavid Oberhollenzer
In most cases, including unistd.h and fcntl.h was a left over anyway. In the cases where it was not, move it to compat.h. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-11-22Cleanup: move all the compatibillity fluff to a dedicated "libcompat"David Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-10-23For Windows builds, remove the so-version, link libgcc staticallyDavid Oberhollenzer
The -static-libgcc flag has to be passed through the compiler with a "-Wc," prefix, because libtool tries to be clever about linker flags. If added directly to LDFLAGS, libtool removes it. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-10-23Fix sqfs_dir_writer_add_entry behaviour on empty stringsDavid Oberhollenzer
SquashFS cannot represent empty string file names in the directory entry structure, so reject them and report an error. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-10-11Add the required libtool magic to build a Windows DLLDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-10-11Remove usage of strchrnul from libsquashfsDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-10-11Add Windows implementation for sqfs_file_tDavid Oberhollenzer
This commit moves the generic unix implementation into a "unix" subdirectory and adds a "win32" subdirectory with a winapi based implementation. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-10-08Fix: propperly clear index entryDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-10-08Fix storing of the directory index in the inode, size accountingDavid Oberhollenzer
After the last modification, the directory index size counter no longer counts the size, but the number of index entries. To avoid future confusion, remove the thing entirely and compute the size on the fly. Furthermore, eliminate unaligned memory writes and fix the darn counter once again. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-10-08Fix recovery of directory index in inode readerDavid Oberhollenzer
Apparently mksquashfs writes an actual usage count (1 = 1 entry, 2 = 2 entries; i.e. NOT off by one). Also, if it does happen to be garbage, guard against an overflow. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-10-08Fix unaligned reads in write_dir_indexDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
generated by cgit v1.2.3 (git 2.39.1) at 2024-11-09 06:02:41 +0000