aboutsummaryrefslogtreecommitdiff
path: root/include
AgeCommit message (Collapse)Author
2019-07-28Implement fragment deduplication in data writerDavid Oberhollenzer
The strategy is simple: - The data writer function that write data/fragment blocks get access to the list files. - When writing a fragment, we look for an already written file that has a fragment with the same size and checksum. - If we find one, we throw away the fragment and reuse the existing one. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-28Add fragment and block checksum fields to file_info_tDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-28Add utility function to compute crc32 check sumsDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-28Merge remaining code of fragment reader into data readerDavid Oberhollenzer
After the table read unification, there wasn't much left of the fragment reader and the remains could easily be moved over to the data reader. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-25Generate linear file list in fstreeDavid Oberhollenzer
Instead of doing DFS on the fly in gensquashfs, churn out a linked list of all files in an archive. Future improvements in packing strategies can go into this file. This can also be usefull for other purposes in the future, such as file deduplication or as a work queue for the unpacker. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-25Add generic read_table function similar to write_tableDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-25Cleanup sqfs_write_tableDavid Oberhollenzer
This commit attempts to make the generic table writer more readable. A few changes are made, including heap allocation of the block list. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-25libutil: add read_data style wrapper around pread()David Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-24Fix processing of tar mtime on 32 bit systemsDavid Oberhollenzer
struct stat uses time_t to store time values. On some 32 bit systems, this may be a 32 bit integer. This patch adds a broken-out 64 bit time value to tar_header_decoded_t and makes sure to clamp the value to +/- (2^32 - 1) if required when writing it back to a struct stat. Reported-by: Matt Turner <mattst88@gmail.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-24Enable largefile supportMatt Turner
Requires that config.h be included before other headers, since the macro _FILE_OFFSET_BITS changes the definitions of things like 'struct stat'. I chose to simply include it at the top of every C file and at immediately after the double-inclusion guards of every header. Signed-off-by: Matt Turner <mattst88@gmail.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-22Add a way to optionally keep the original time stampsDavid Oberhollenzer
First of all, this commit adds a mod_time field to a tree node. When creating the tree node, the field is set from the struct stat. When scanning a directory, the time stamps from the input are used if set. Second, the libsqfs code that reads inodes is modified to store the mod_time from the inode in the fstree node and to write the tree node into a generated inode. Finally, tar2sqfs is modified to optionally keep the timestamps from the tar archive instead of setting defaults. gensquashfs is similarly modified to keep the input timestamps if specified. The result is as follows: - sqfs2tar will always carry the timestamps from the squashfs over to the tar ball. - tar2sqfs will set defaults, unless explicitly asked to preserve the mtime from the tar ball. - gensquashfs can optionally preserve the mtime from the input hierarchy it processes if only --pack-dir is specified. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-21Implement generating an inode table for NFS exportDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-21Cleanup xattr handlingDavid Oberhollenzer
- Store them in a struct instead of a hacky uint64_t with magic shifts - Split up key/value pair write function to write_key and write_value - Move the size accounting into those functions respectively Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-21Keep track of xattr key & value references AFTER deduplicationDavid Oberhollenzer
This commit adds a reference count functionality to the string table implementation and uses this functionality in the fstree code to count how often each key and value is referenced by the deduplicated Xattr blocks. This is needed to support deduplication through out-of-band storage of xattrs. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-20Make raw fragment table accessible through fragment/data readersDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-16cleanup: move error handling into read_retryDavid Oberhollenzer
If read_retry fails to read the expected amount of data (EOF or otherwise), it is almost always an error. This commit renames read_retry to read_data and moves error handling into the function, making a lot of error handling code redundant. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-16cleanup: move error handling into write_retryDavid Oberhollenzer
If write_retry fails to write everything, it is *always* an error. This commit renames write_retry to write_data and moves error handling into the function, making a lot of error handling code redundant. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-16Fix directory index creationDavid Oberhollenzer
Digging around in kernel internals and mksquashfs reveals that it is actually a buffer offset into the raw directory buffer. The error hasn't been noted until now because of the bug fixed by a5428e0. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-15Add flags to data writer to micro manage behaviourDavid Oberhollenzer
The added flags allow controlling the following on a per file level: - forcing a file to be written uncompressed - forcing a file to not have a fragment, i.e. the last truncated block actually being written as a block - padding a file to be alligned to device block size The flags are not yet exposed to anything user controllable (such as command line flags). Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-10Add a way to keep meta data blocks in memoryDavid Oberhollenzer
Instead of writing meta data blocks directly to disk, the writer can now alternatively keep the blocks in memory until explicitly told to write to disk. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-07Actually encode/decode directory inode difference as signedDavid Oberhollenzer
The directory listing stores a signed difference of the inode number. Actually treating it as signed saves emitting extra headers if hard links or file deduplication are finally implemented. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-04Fix: simplify deduction logic for squashfs inode typeDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-04tar2sqfs: repack extended attributes into squashfs filesystemDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-04libtar: add support for xattr extensionsDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-03cleanup: move tree node from path function to libfstree.aDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-03tar writer: replace PAX headers with GNU extensionsDavid Oberhollenzer
Some experiments seem to indicate that the various GNU extensions are more widely supported than their POSIX equivalents[1]. Possibly because they are easier to implement and possibly because of the wide spread use of GNU tar. This commit replaces the PAX writer in the write_tar_header implementation with a GNU extension based writer. The writer is also cleaned up by removing all global state. The record counter is moved outside into the tar2sqfs program and passed in as function argument. [1] https://dev.gentoo.org/~mgorny/articles/portability-of-tar-features.html Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-01cleanup: split tar code up, remove some duplicationsDavid Oberhollenzer
This commit attempts to split some of the monolitic tar parsing code up into multiple functions in seperate files. Also, some code duplication (like reading a record into memory which was implemented twice) is removed. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-06-30libtar: clarify actual size vs on-disk record sizeDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-06-30Add support for repacking condensed sparse filesDavid Oberhollenzer
This commit broadly does the following things: - Rename and move the sparse mapping structure to libutil - Add a function to the data writer for writing condensed versions of sparse files, given the mapping. - This shares code with the already existing function for regular files. The shared code is moved to a common helper function. - Add support to tar2sqfs for repacking sparse files. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-06-30tar reader: also store condensed size of sparse filesDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-06-29Add support for reading old style GNU sparse tar file formatDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-06-28Add support for unpacking sparse files as sparse filesDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-06-28Add support for packing sparse filesDavid Oberhollenzer
This commit adds support for packing sparse files into squashfs images as follows: - In the data writer: simply detect zero blocks and write a zero to the block size field and don't emit any data. Record the number of bytes saved this way. For fragments, set the fragment offset to invalid. - In the inode writer: write out the number of bytes saved for sparse files. If there should be a fragment but there is none, append a block count of 0. - In the data reader: if the block size is 0, read nothing from disk and emit an empty block. Do the same if the fragment is missing. - In the inode reader: restore the number of bytes saved for sparse files. The sparse files can be packed and unpacked, but the unpacking will not create sparse files for now. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-06-28Add basic support for the GNU tar formatDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-06-27Relax tar header parser to accept pre-posix formatsDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-06-24Add missing stdint.h header to tar.hDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-06-23Move fstree default option processing to fstree codeDavid Oberhollenzer
Instead of decomposing a default string in gensquashfs option processing, move that to fstree_init instead and pass the option string directly to fstree_init. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-06-23Move all handling of compressor names to libcompress.aDavid Oberhollenzer
This commit removes handling of compressor names from gensquashfs. Instead, functions are added to libcompress to obtain name from ID, ID from name and to print out defaults. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-06-22Cleanup: unify packdir/packfile based directory changes in gensquashfsDavid Oberhollenzer
This commit removes the packdir/packfile based directory setup magic from fstree_from_file and moves it to gensquashfs. Over there, the common parts are deduplicated. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-06-22Cleanup: split fstree sort into 2 fstree independend functionsDavid Oberhollenzer
Make tree node list sort and recursive variant available and independend of the fstree_t. This is considered cleaner, since the fstree_t actually isn't needed for any of this and we can just call the recusvie sort on the root instead, and we can use the sort implementation directly for things like the upcoming unit test. Also this commit splits up the merge/sort implementation into a seperate split and merge functions to make the code somewhat more readable. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-06-20simplify and improve canonicalize_nameDavid Oberhollenzer
- Replace scan and memmove with simple automaton - Convert forward to back slashes, lots of file systems treat them equivalenty, this saves us a few more checks down the road - Remove './' path components. We can saveley remove them in a string processing step instead of throwing an error. Also they actually appear often in tar balls, possibly not under user control. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-06-19Split generic tar code off to static libraryDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-06-19Cleanup fstree codeDavid Oberhollenzer
Replace default_$FIELD with a struct stat. Merge the rest of add_node/add_file into mknode and add by path+stat. Expose the mknode function and replace all the duplicated node pointer arithmetic magic with a call to mknode. Plus a generic cleanup by utlizing new util functions. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-06-17libfstree: add generic function to add node from struct statDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-06-15libfstree.a: add function to get struct stat from tree nodeDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-06-15Move function to padd a file with 0 bytes to libutilDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-06-11Move file extraction code to generic data_reader_t in libsquashfs.aDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-06-11Cleanup dependency handlingDavid Oberhollenzer
Always try to gather all compressor libraries available, but only complain about missing a one if it has been *explicitly* selected. If a compressor has been explicityl disabled, we can still turn it off after checking. Also, rework gensquashfs to set the default compressor based on what's available. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-06-11Move write_xattr to libsqfs.aDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-06-11Move data writer to libsqfs.aDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>