summaryrefslogtreecommitdiff
path: root/include
AgeCommit message (Collapse)Author
2019-08-20Add explicit padding to tree_node_tDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-08-18Replace update_crc32 helper function with crc32 from zlibDavid Oberhollenzer
It is optimized to the maximum and if we already use zlib anyway, why not use zlib crc32? This also makes zlib a hard dependency which also means the whole "do we have a compressor" sanity check in the build system can be removed. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-08-18Make data writer use block processorDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-08-18Restructure data writer around passing block_t structuresDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-08-18Minor interface change to data writerDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-08-18cleanup: internalize deduplication list in data_writerDavid Oberhollenzer
This change removes the need for passing a list of files around for deduplication. Also the deduplication code no longer needs to worry about order, since the file being deduplicated is only added after deduplication is done. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-08-18Add block processor data structureDavid Oberhollenzer
The interface is designed for parallel, asynchronuous processing of data blocks with an I/O callback that handles the serialized result. The underlying implementation is currently still synchronuous. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-08-16Add deep-copy function to compressor interfaceDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-08-11Add gensquashfs option to read xattrs from input filesDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-08-11Add --one-file-system option to gensquashfsDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-08-11Replace fstree_from_dir boolean with flag fieldDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-08-07Add pread(2) like function to data_readerDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-08-05cleanup data readerDavid Oberhollenzer
- Split block reading code out from "dump_blocks" into precache_data_block, similar to precache_fragment_block - Merge the code paths for fragment/data block reading and uncompression Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-08-05cleanup: unify all the code that reads squashfs imagesDavid Oberhollenzer
This commit creates a new data structure called 'sqfs_reader_t' that takes care of all the repetetive tasks like opening the file, reading the super block, creating the compressor, deserializing an fstree and creating a data reader. This in turn makes it possible to remove all the duplicate code from rdsquashfs and sqfs2tar. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-08-04Improve file unpacking orderDavid Oberhollenzer
This commit moves the file unpacking order & job scheduling to a libfstree function. The ordering is improved by making sure fragment blocks are not extracted more than once and files with data blocks are extracted in order. This way, serial unpacking of a 2GiB Debian live image could be reduced from ~5' on my test machine to ~3.5', whereas parallel unpacking stays roughly the same (~3' for -j 4). Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-08-02Implement support for SOURCE_DATE_EPOCH environment variableDavid Oberhollenzer
reproducible-builds.org suggests the use of an environment variable as a source for time stamps: https://reproducible-builds.org/specs/source-date-epoch/ This commit adds support for setting the default mtime from the variable, if it is set and only defaulting to 0 if not. The timestamp given by the command line switch takes precedence. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-08-01Add ability to write_tar_header to embedd extended attributesDavid Oberhollenzer
This commit patches the tar writer to generate a PAX header with SCHILY xattr key/value pairs if requested. The Schily format is used for two reasons: - It is simple - It is apparently more widely supported than the libarchive format Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-08-01Add option to restore xattrs to deserialize_fstreeDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-08-01Add xattr reader implementation to recover xattrs from squashfsDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-30Add propper copyright headers to all source filesDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-29Cleanup: move deduplication code from data writer to fstreeDavid Oberhollenzer
Since it is actually completely independend of libsqfs and only works on file_info_t lists, it can be safely moved over to libfstree and the data writer becomes less cluttered as a result. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-28Add some nice statistics output to tar2sqfs and gensquashfsDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-28Add general purpose flags field to file_info_tDavid Oberhollenzer
Simplifies some task if we can just add a flag that a file has a framgent or that it has already been detected as a duplicate. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-28Implement fragment deduplication in data writerDavid Oberhollenzer
The strategy is simple: - The data writer function that write data/fragment blocks get access to the list files. - When writing a fragment, we look for an already written file that has a fragment with the same size and checksum. - If we find one, we throw away the fragment and reuse the existing one. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-28Add fragment and block checksum fields to file_info_tDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-28Add utility function to compute crc32 check sumsDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-28Merge remaining code of fragment reader into data readerDavid Oberhollenzer
After the table read unification, there wasn't much left of the fragment reader and the remains could easily be moved over to the data reader. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-25Generate linear file list in fstreeDavid Oberhollenzer
Instead of doing DFS on the fly in gensquashfs, churn out a linked list of all files in an archive. Future improvements in packing strategies can go into this file. This can also be usefull for other purposes in the future, such as file deduplication or as a work queue for the unpacker. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-25Add generic read_table function similar to write_tableDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-25Cleanup sqfs_write_tableDavid Oberhollenzer
This commit attempts to make the generic table writer more readable. A few changes are made, including heap allocation of the block list. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-25libutil: add read_data style wrapper around pread()David Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-24Fix processing of tar mtime on 32 bit systemsDavid Oberhollenzer
struct stat uses time_t to store time values. On some 32 bit systems, this may be a 32 bit integer. This patch adds a broken-out 64 bit time value to tar_header_decoded_t and makes sure to clamp the value to +/- (2^32 - 1) if required when writing it back to a struct stat. Reported-by: Matt Turner <mattst88@gmail.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-24Enable largefile supportMatt Turner
Requires that config.h be included before other headers, since the macro _FILE_OFFSET_BITS changes the definitions of things like 'struct stat'. I chose to simply include it at the top of every C file and at immediately after the double-inclusion guards of every header. Signed-off-by: Matt Turner <mattst88@gmail.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-22Add a way to optionally keep the original time stampsDavid Oberhollenzer
First of all, this commit adds a mod_time field to a tree node. When creating the tree node, the field is set from the struct stat. When scanning a directory, the time stamps from the input are used if set. Second, the libsqfs code that reads inodes is modified to store the mod_time from the inode in the fstree node and to write the tree node into a generated inode. Finally, tar2sqfs is modified to optionally keep the timestamps from the tar archive instead of setting defaults. gensquashfs is similarly modified to keep the input timestamps if specified. The result is as follows: - sqfs2tar will always carry the timestamps from the squashfs over to the tar ball. - tar2sqfs will set defaults, unless explicitly asked to preserve the mtime from the tar ball. - gensquashfs can optionally preserve the mtime from the input hierarchy it processes if only --pack-dir is specified. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-21Implement generating an inode table for NFS exportDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-21Cleanup xattr handlingDavid Oberhollenzer
- Store them in a struct instead of a hacky uint64_t with magic shifts - Split up key/value pair write function to write_key and write_value - Move the size accounting into those functions respectively Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-21Keep track of xattr key & value references AFTER deduplicationDavid Oberhollenzer
This commit adds a reference count functionality to the string table implementation and uses this functionality in the fstree code to count how often each key and value is referenced by the deduplicated Xattr blocks. This is needed to support deduplication through out-of-band storage of xattrs. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-20Make raw fragment table accessible through fragment/data readersDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-16cleanup: move error handling into read_retryDavid Oberhollenzer
If read_retry fails to read the expected amount of data (EOF or otherwise), it is almost always an error. This commit renames read_retry to read_data and moves error handling into the function, making a lot of error handling code redundant. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-16cleanup: move error handling into write_retryDavid Oberhollenzer
If write_retry fails to write everything, it is *always* an error. This commit renames write_retry to write_data and moves error handling into the function, making a lot of error handling code redundant. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-16Fix directory index creationDavid Oberhollenzer
Digging around in kernel internals and mksquashfs reveals that it is actually a buffer offset into the raw directory buffer. The error hasn't been noted until now because of the bug fixed by a5428e0. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-15Add flags to data writer to micro manage behaviourDavid Oberhollenzer
The added flags allow controlling the following on a per file level: - forcing a file to be written uncompressed - forcing a file to not have a fragment, i.e. the last truncated block actually being written as a block - padding a file to be alligned to device block size The flags are not yet exposed to anything user controllable (such as command line flags). Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-10Add a way to keep meta data blocks in memoryDavid Oberhollenzer
Instead of writing meta data blocks directly to disk, the writer can now alternatively keep the blocks in memory until explicitly told to write to disk. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-07Actually encode/decode directory inode difference as signedDavid Oberhollenzer
The directory listing stores a signed difference of the inode number. Actually treating it as signed saves emitting extra headers if hard links or file deduplication are finally implemented. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-04Fix: simplify deduction logic for squashfs inode typeDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-04tar2sqfs: repack extended attributes into squashfs filesystemDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-04libtar: add support for xattr extensionsDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-03cleanup: move tree node from path function to libfstree.aDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-03tar writer: replace PAX headers with GNU extensionsDavid Oberhollenzer
Some experiments seem to indicate that the various GNU extensions are more widely supported than their POSIX equivalents[1]. Possibly because they are easier to implement and possibly because of the wide spread use of GNU tar. This commit replaces the PAX writer in the write_tar_header implementation with a GNU extension based writer. The writer is also cleaned up by removing all global state. The record counter is moved outside into the tar2sqfs program and passed in as function argument. [1] https://dev.gentoo.org/~mgorny/articles/portability-of-tar-features.html Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-01cleanup: split tar code up, remove some duplicationsDavid Oberhollenzer
This commit attempts to split some of the monolitic tar parsing code up into multiple functions in seperate files. Also, some code duplication (like reading a record into memory which was implemented twice) is removed. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
generated by cgit v1.2.3 (git 2.39.1) at 2024-06-02 16:41:46 +0000