aboutsummaryrefslogtreecommitdiff
path: root/include/fstree.h
AgeCommit message (Collapse)Author
2022-08-19Fix: libfstree: actually use a full 32 bit hard link counterDavid Oberhollenzer
The squashfs on-disk format uses 32 bit link counters, but the fstree used 16 bit ones. Because the link count also includes child nodes, this artificially limited the number of entries in a directory to ~64k files. This patch removes the limit by switching libfstree to 32 bit counters. Reported-by: Marvin Renich <mrvn@renich.org> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-23Bring back the flat list of inodes in libfstreeDavid Oberhollenzer
It makes further processing simpler and doesn't leak the abstraction into upper layers. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-22Add basic support for handling and serializing hard linksDavid Oberhollenzer
In libfstree, add a function to add a hard link to the fstree. The hard links stores the target in the data.target field, canonicalizes the target and sets a sentinel mode. A second function is used to resolve link, i.e. replacing it with a direct pointer, setting another sentinel mode and increasing the targets link count. The post process function tries to resolve unresolved hard links and only allocates inode numbers for nodes that aren't hard links. If the target node of a hard link does not have an inode number yet, the two need to be swapped, since this is also the order in which they are serialized. The serialization function in libcommon simply has to skip hard link nodes and when writing directory entries, use the inode num/ref of the target node. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-19Split the libfstree add_by_path tree traversal function outDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-19Fix spelling mistakes found by lintianBenjamin Drung
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
2019-12-18Add an explicit link count to the fstree nodesDavid Oberhollenzer
Gets initialized to 2 for directories, 1 for all other types. The count of the parent node is automatically incremented. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-18Rename fstree "slink_target" to "target"David Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-18Move is_filename_sane to libfstree, add test casesDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-18Cleanup: internalize some fstree functionsDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-18Cleanup: remove the unused "block_size" from the fstreeDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-18Cleanup: merge the fstree post processing functionsDavid Oberhollenzer
Instead of having 3 different functions for sorting the tree, numbering the nodes and generating a file list, that all have to be used in the right order, this commit merges them into a single "fstree_post_process" function. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-16Remove fstree inode tableDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-11-24Cleanup: move canonicalize_name back to libfstree.aDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-11-22Cleanup: move all the compatibillity fluff to a dedicated "libcompat"David Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-10-07Cleanup: move libutil related headers to "util" sub directoryDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-09-29Cleanup: fstree no longer has any use for the block sizeDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-09-28Replace fstree/sqfshelper xattr code with sqfs_xattr_writer_tDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-09-28Move fstree selinux code to gensquashfsDavid Oberhollenzer
Same rational as for the dir-scanner code: It's actually the only user and it is going to get a lot closer integerated with libsquashfs. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-09-28Move fstree_from_dir to gensquashfs codeDavid Oberhollenzer
It's actually the only user and the dir-scanner xattr code is going to get a lot closer integerated with libsquashfs. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-09-27Add a header for platform compatibillity fluffDavid Oberhollenzer
- We don't have "endian.h" everywhere. On some BSDs its in sys and on some BSDs the macros have different names. - We definitely don't have sysmacros.h on non-Unix-like systems. - Likewise for sys/types.h, sys/stat.h and their contents. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-09-27Cleanup: replace fixed with data types with typedefsDavid Oberhollenzer
This is a fully automated search and replace, i.e. I ran this: git grep -l uint8_t | xargs sed -i 's/uint8_t/sqfs_u8/g' git grep -l uint16_t | xargs sed -i 's/uint16_t/sqfs_u16/g' git grep -l uint32_t | xargs sed -i 's/uint32_t/sqfs_u32/g' git grep -l uint64_t | xargs sed -i 's/uint64_t/sqfs_u64/g' git grep -l int8_t | xargs sed -i 's/int8_t/sqfs_s8/g' git grep -l int16_t | xargs sed -i 's/int16_t/sqfs_s16/g' git grep -l int32_t | xargs sed -i 's/int32_t/sqfs_s32/g' git grep -l int64_t | xargs sed -i 's/int64_t/sqfs_s64/g' and than added the appropriate definitions to sqfs/predef.h The whole point being better compatibillity with platforms that may not have an stdint.h with the propper definitions. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-09-27Cleanup: remove most of the payload pointer magic from libfstreeDavid Oberhollenzer
Now that dir_info_t and file_info_t have reasonably small, use them in tree_node_t directly instead of doing pointer arithmetic magic on the payload area. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-09-25Remove no-longer-used cruft from libfstreeDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-09-22Use inode structures in the data writer instead of fstree file infoDavid Oberhollenzer
Changes are minor, but needs some temporary hacks in fstree again. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-09-21Minor fstree related cleanupDavid Oberhollenzer
- file list is no longer needed for statistics - the size field in the directory info structure is no longer in use - sqfs2tar, rdsquashfs and sqfsdiff no longer depend on libfstree.a Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-09-20Dead code removal, round #2David Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-09-20Large round of dead code removalDavid Oberhollenzer
Remove all the library functions that no longer have any users. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-09-20Move canonicalize_name back to libutilDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-09-20Move "optimize unpack order" to from fstree to rdsquashfsDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-09-20Remove parallel unpackingDavid Oberhollenzer
Parallel unpacking didn't really improve the speed that much. Actually sorting the files for optimized unpack order improved speed much more than the parallel unpacker. Furthermore, the fork based parallel unpacker was actually pretty messy to begin with. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-09-19Add directory reader data structureDavid Oberhollenzer
This moves a lot of the stuff that is done manually in the tree deserializer to a generic helper in libsquashfs. Due to how the fstree is implemented, as a work around, the inode needs to be temporarily stored in the tree node, but some of the directory details could be removed. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-09-14Remove fstree file flagsDavid Oberhollenzer
As a side effect, this requires the data writer to keep track of statistics. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-09-14Move data deduplication from fstree code to data writerDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-09-01Move some application specific stuff out of libutilDavid Oberhollenzer
This commit does the following: - canonicalize_name is moved to libfstree - source_date_epoch is only used inside libfstree, so it's also moved over and can later be completely internalized - print_version is moved over to sqfshelper. Mainly so it doesn't end up in libsquashfs.so for no sane reason. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-08-20Add explicit padding to tree_node_tDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-08-18cleanup: internalize deduplication list in data_writerDavid Oberhollenzer
This change removes the need for passing a list of files around for deduplication. Also the deduplication code no longer needs to worry about order, since the file being deduplicated is only added after deduplication is done. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-08-11Add gensquashfs option to read xattrs from input filesDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-08-11Add --one-file-system option to gensquashfsDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-08-11Replace fstree_from_dir boolean with flag fieldDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-08-04Improve file unpacking orderDavid Oberhollenzer
This commit moves the file unpacking order & job scheduling to a libfstree function. The ordering is improved by making sure fragment blocks are not extracted more than once and files with data blocks are extracted in order. This way, serial unpacking of a 2GiB Debian live image could be reduced from ~5' on my test machine to ~3.5', whereas parallel unpacking stays roughly the same (~3' for -j 4). Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-30Add propper copyright headers to all source filesDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-29Cleanup: move deduplication code from data writer to fstreeDavid Oberhollenzer
Since it is actually completely independend of libsqfs and only works on file_info_t lists, it can be safely moved over to libfstree and the data writer becomes less cluttered as a result. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-28Add general purpose flags field to file_info_tDavid Oberhollenzer
Simplifies some task if we can just add a flag that a file has a framgent or that it has already been detected as a duplicate. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-28Add fragment and block checksum fields to file_info_tDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-25Generate linear file list in fstreeDavid Oberhollenzer
Instead of doing DFS on the fly in gensquashfs, churn out a linked list of all files in an archive. Future improvements in packing strategies can go into this file. This can also be usefull for other purposes in the future, such as file deduplication or as a work queue for the unpacker. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-24Enable largefile supportMatt Turner
Requires that config.h be included before other headers, since the macro _FILE_OFFSET_BITS changes the definitions of things like 'struct stat'. I chose to simply include it at the top of every C file and at immediately after the double-inclusion guards of every header. Signed-off-by: Matt Turner <mattst88@gmail.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-22Add a way to optionally keep the original time stampsDavid Oberhollenzer
First of all, this commit adds a mod_time field to a tree node. When creating the tree node, the field is set from the struct stat. When scanning a directory, the time stamps from the input are used if set. Second, the libsqfs code that reads inodes is modified to store the mod_time from the inode in the fstree node and to write the tree node into a generated inode. Finally, tar2sqfs is modified to optionally keep the timestamps from the tar archive instead of setting defaults. gensquashfs is similarly modified to keep the input timestamps if specified. The result is as follows: - sqfs2tar will always carry the timestamps from the squashfs over to the tar ball. - tar2sqfs will set defaults, unless explicitly asked to preserve the mtime from the tar ball. - gensquashfs can optionally preserve the mtime from the input hierarchy it processes if only --pack-dir is specified. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-21Cleanup xattr handlingDavid Oberhollenzer
- Store them in a struct instead of a hacky uint64_t with magic shifts - Split up key/value pair write function to write_key and write_value - Move the size accounting into those functions respectively Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-03cleanup: move tree node from path function to libfstree.aDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-06-28Add support for packing sparse filesDavid Oberhollenzer
This commit adds support for packing sparse files into squashfs images as follows: - In the data writer: simply detect zero blocks and write a zero to the block size field and don't emit any data. Record the number of bytes saved this way. For fragments, set the fragment offset to invalid. - In the inode writer: write out the number of bytes saved for sparse files. If there should be a fragment but there is none, append a block count of 0. - In the data reader: if the block size is 0, read nothing from disk and emit an empty block. Do the same if the fragment is missing. - In the inode reader: restore the number of bytes saved for sparse files. The sparse files can be packed and unpacked, but the unpacking will not create sparse files for now. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>