summaryrefslogtreecommitdiff
path: root/lib
AgeCommit message (Collapse)Author
2019-07-28Implement fragment deduplication in data writerDavid Oberhollenzer
The strategy is simple: - The data writer function that write data/fragment blocks get access to the list files. - When writing a fragment, we look for an already written file that has a fragment with the same size and checksum. - If we find one, we throw away the fragment and reuse the existing one. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-28Unify common file start/end code from data writer in helper functionsDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-28Compute per-block and per-fragment checksums in data wrtierDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-28Add fragment and block checksum fields to file_info_tDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-28Add utility function to compute crc32 check sumsDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-28Merge remaining code of fragment reader into data readerDavid Oberhollenzer
After the table read unification, there wasn't much left of the fragment reader and the remains could easily be moved over to the data reader. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-28Split data_reader_dump_file into smaller functionsDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-25Use safer string copy function to fill tar headerDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-25Make sure symlink in fstree_mknode is always set when creating a symlinkDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-25Enforce reasonable upper and low bounds on the size of tar headersDavid Oberhollenzer
Since they are read directly into memory, blindly allocating the size from the tar ball is probably a bad idea. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-25Fix potential resource leak in deserialize_treeDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-25Fix checks of super block block sizeDavid Oberhollenzer
Make sure range is checked when reading a block and that the check is made correctly. Also make the block log check a little more strict. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-25Fix acciedental usage of left over local variable instead struct memberDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-25Generate linear file list in fstreeDavid Oberhollenzer
Instead of doing DFS on the fly in gensquashfs, churn out a linked list of all files in an archive. Future improvements in packing strategies can go into this file. This can also be usefull for other purposes in the future, such as file deduplication or as a work queue for the unpacker. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-25Add generic read_table function similar to write_tableDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-25Cleanup sqfs_write_tableDavid Oberhollenzer
This commit attempts to make the generic table writer more readable. A few changes are made, including heap allocation of the block list. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-25Rename table.c to write_table.c in accordance to function it containsDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-25Fix fragment reader out of bounds read when loading tableDavid Oberhollenzer
This commit fixes a bug in the fragment table reader where the reader tries to read data into an out of bounds location due to an oversight in size calculation. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-25Replace reads in squashfs with positional readsDavid Oberhollenzer
In most cases, we know exactely where the data that we want to read is on disk, so instead of using read() on the squashfs (or lseek + read), the code can in many places be cleaned up to use the pread wrapper read_data_at instead. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-25libutil: add read_data style wrapper around pread()David Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-24libfstree: fix signed/unsigned comparisonsDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-24Fix processing of tar mtime on 32 bit systemsDavid Oberhollenzer
struct stat uses time_t to store time values. On some 32 bit systems, this may be a 32 bit integer. This patch adds a broken-out 64 bit time value to tar_header_decoded_t and makes sure to clamp the value to +/- (2^32 - 1) if required when writing it back to a struct stat. Reported-by: Matt Turner <mattst88@gmail.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-24cleanup: remove atime/ctime processing codeDavid Oberhollenzer
This commit removes all the code for parsing and processing atime/ctime and values and related test code. Caring about those is kind of pointless because squashfs can only store mtime in inodes. The only relevant place is when generating a struct stat from a squashfs inode or an fstree node. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-24Include the reason ZSTD gave us for the errorMatt Turner
Without it you're left guessing or using a debugger to figure out what's wrong. Signed-off-by: Matt Turner <mattst88@gmail.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-24Enable largefile supportMatt Turner
Requires that config.h be included before other headers, since the macro _FILE_OFFSET_BITS changes the definitions of things like 'struct stat'. I chose to simply include it at the top of every C file and at immediately after the double-inclusion guards of every header. Signed-off-by: Matt Turner <mattst88@gmail.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-24libtar: more lenient tar checksum verificationDavid Oberhollenzer
Until now, the tar checksum verification simply copied the header, recomputed the checksum and compared it byte-for-byte with the original. However, not all implementations store the checksum the same way. For instance, git-archive generates tar balls that use the same format as for other octal numbers. This patch makes the checksum verification more lenient by parsing the checksum from the header and comparing it with the computed value instead of copying the entire block and insisting on byte-for-byte equivalence. The result is better interoperabillity with existing tools and perhaps slightly faster processing since the block doesn't have to be copied. Reported-by: Matt Turner <mattst88@gmail.com> Suggested-by: René Scharfe <l.s.r@web.de> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-23Fix tree node scanningDavid Oberhollenzer
- Bail early on empty directories without touching the meta readers. - Aport the directory read loop if we can't even read a header anymore, no matter if there are bytes remaining. - Also add that same condition to the inner loop. The later two actually caused a numeric overflow on some particularly malformed squashfs images, going into a RAM filling infinite loop. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-22Add a way to optionally keep the original time stampsDavid Oberhollenzer
First of all, this commit adds a mod_time field to a tree node. When creating the tree node, the field is set from the struct stat. When scanning a directory, the time stamps from the input are used if set. Second, the libsqfs code that reads inodes is modified to store the mod_time from the inode in the fstree node and to write the tree node into a generated inode. Finally, tar2sqfs is modified to optionally keep the timestamps from the tar archive instead of setting defaults. gensquashfs is similarly modified to keep the input timestamps if specified. The result is as follows: - sqfs2tar will always carry the timestamps from the squashfs over to the tar ball. - tar2sqfs will set defaults, unless explicitly asked to preserve the mtime from the tar ball. - gensquashfs can optionally preserve the mtime from the input hierarchy it processes if only --pack-dir is specified. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-21Fix indexing into export tableDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-21Implement generating an inode table for NFS exportDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-21Add support for storing xattr values out-of-lineDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-21Cleanup xattr handlingDavid Oberhollenzer
- Store them in a struct instead of a hacky uint64_t with magic shifts - Split up key/value pair write function to write_key and write_value - Move the size accounting into those functions respectively Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-21cleanup: remove left over, unused assignmentDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-21Keep track of xattr key & value references AFTER deduplicationDavid Oberhollenzer
This commit adds a reference count functionality to the string table implementation and uses this functionality in the fstree code to count how often each key and value is referenced by the deduplicated Xattr blocks. This is needed to support deduplication through out-of-band storage of xattrs. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-20Make raw fragment table accessible through fragment/data readersDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-19libutil.a: make sure str_table_init initializes all fieldsDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-17fstree: add support for spaces in filenamesDavid Oberhollenzer
This commit adds a mechanism to fstree_from_file to support filenames with spaces in them by quoting the entire string. Quote marks can still be used inside file names by escaping them with a backslash. Back slashes (if that is your thing) can also be escaped. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-16cleanup: move error handling into read_retryDavid Oberhollenzer
If read_retry fails to read the expected amount of data (EOF or otherwise), it is almost always an error. This commit renames read_retry to read_data and moves error handling into the function, making a lot of error handling code redundant. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-16cleanup: move error handling into write_retryDavid Oberhollenzer
If write_retry fails to write everything, it is *always* an error. This commit renames write_retry to write_data and moves error handling into the function, making a lot of error handling code redundant. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-16Fix directory index creationDavid Oberhollenzer
Digging around in kernel internals and mksquashfs reveals that it is actually a buffer offset into the raw directory buffer. The error hasn't been noted until now because of the bug fixed by a5428e0. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-15Add flags to data writer to micro manage behaviourDavid Oberhollenzer
The added flags allow controlling the following on a per file level: - forcing a file to be written uncompressed - forcing a file to not have a fragment, i.e. the last truncated block actually being written as a block - padding a file to be alligned to device block size The flags are not yet exposed to anything user controllable (such as command line flags). Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-12Add generic support for reading files without fragmentsDavid Oberhollenzer
This commit extends the special case handling for sparse files to generically support reading files that don't have a fragment but instead have a trunkated final block. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-12Simplify writer code for files without a fragmentDavid Oberhollenzer
The data writer sparse block code can take advantage that it can add a block size instead of a fragment and doesn't have to initialize the framgent location. In return, the tree node to inode serialization code doesn't need a special case for sparse file anymore and can now also handle files that are forced to not have a fragment. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-12fstree: mknode: initialize fragment data, add extra blocksize slotDavid Oberhollenzer
This commit makes sure that regular file tree nodes have one more slot than necessary to support files that don't have fragments. Also, it initializes the fragment data, which should help to deduplicate some code ahead. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-10Fix dirindex writing for ext dir inodeDavid Oberhollenzer
This commit makes sure we write the index count to the inode before writing out the inode itself. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-10libsqfs: eleminate use of temporary fileDavid Oberhollenzer
Tell the directory writer to keep the directory blocks in memory, then write them out once everything is done and the inodes are all written out. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-10Add a way to keep meta data blocks in memoryDavid Oberhollenzer
Instead of writing meta data blocks directly to disk, the writer can now alternatively keep the blocks in memory until explicitly told to write to disk. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-10Make coverity happyDavid Oberhollenzer
The string is actually always null-terminated, since the calloc above adds one more byte than what we tell readlink about. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-09Sanity check id table size read from super blockDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-07Fix regression in fstree_from_file device node formatDavid Oberhollenzer
The format is documented as "<c|b> major minor" but the parser was accidentally changed to require a colon in between. Fixes: 9864ea5b2045f4bd72633152d71dd1c7f8b0b7f9 Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
generated by cgit v1.2.3 (git 2.39.1) at 2024-11-01 21:30:10 +0000