aboutsummaryrefslogtreecommitdiff
path: root/include/fstree.h
AgeCommit message (Collapse)Author
2019-11-24Cleanup: move canonicalize_name back to libfstree.aDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-11-22Cleanup: move all the compatibillity fluff to a dedicated "libcompat"David Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-10-07Cleanup: move libutil related headers to "util" sub directoryDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-09-29Cleanup: fstree no longer has any use for the block sizeDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-09-28Replace fstree/sqfshelper xattr code with sqfs_xattr_writer_tDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-09-28Move fstree selinux code to gensquashfsDavid Oberhollenzer
Same rational as for the dir-scanner code: It's actually the only user and it is going to get a lot closer integerated with libsquashfs. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-09-28Move fstree_from_dir to gensquashfs codeDavid Oberhollenzer
It's actually the only user and the dir-scanner xattr code is going to get a lot closer integerated with libsquashfs. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-09-27Add a header for platform compatibillity fluffDavid Oberhollenzer
- We don't have "endian.h" everywhere. On some BSDs its in sys and on some BSDs the macros have different names. - We definitely don't have sysmacros.h on non-Unix-like systems. - Likewise for sys/types.h, sys/stat.h and their contents. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-09-27Cleanup: replace fixed with data types with typedefsDavid Oberhollenzer
This is a fully automated search and replace, i.e. I ran this: git grep -l uint8_t | xargs sed -i 's/uint8_t/sqfs_u8/g' git grep -l uint16_t | xargs sed -i 's/uint16_t/sqfs_u16/g' git grep -l uint32_t | xargs sed -i 's/uint32_t/sqfs_u32/g' git grep -l uint64_t | xargs sed -i 's/uint64_t/sqfs_u64/g' git grep -l int8_t | xargs sed -i 's/int8_t/sqfs_s8/g' git grep -l int16_t | xargs sed -i 's/int16_t/sqfs_s16/g' git grep -l int32_t | xargs sed -i 's/int32_t/sqfs_s32/g' git grep -l int64_t | xargs sed -i 's/int64_t/sqfs_s64/g' and than added the appropriate definitions to sqfs/predef.h The whole point being better compatibillity with platforms that may not have an stdint.h with the propper definitions. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-09-27Cleanup: remove most of the payload pointer magic from libfstreeDavid Oberhollenzer
Now that dir_info_t and file_info_t have reasonably small, use them in tree_node_t directly instead of doing pointer arithmetic magic on the payload area. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-09-25Remove no-longer-used cruft from libfstreeDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-09-22Use inode structures in the data writer instead of fstree file infoDavid Oberhollenzer
Changes are minor, but needs some temporary hacks in fstree again. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-09-21Minor fstree related cleanupDavid Oberhollenzer
- file list is no longer needed for statistics - the size field in the directory info structure is no longer in use - sqfs2tar, rdsquashfs and sqfsdiff no longer depend on libfstree.a Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-09-20Dead code removal, round #2David Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-09-20Large round of dead code removalDavid Oberhollenzer
Remove all the library functions that no longer have any users. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-09-20Move canonicalize_name back to libutilDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-09-20Move "optimize unpack order" to from fstree to rdsquashfsDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-09-20Remove parallel unpackingDavid Oberhollenzer
Parallel unpacking didn't really improve the speed that much. Actually sorting the files for optimized unpack order improved speed much more than the parallel unpacker. Furthermore, the fork based parallel unpacker was actually pretty messy to begin with. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-09-19Add directory reader data structureDavid Oberhollenzer
This moves a lot of the stuff that is done manually in the tree deserializer to a generic helper in libsquashfs. Due to how the fstree is implemented, as a work around, the inode needs to be temporarily stored in the tree node, but some of the directory details could be removed. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-09-14Remove fstree file flagsDavid Oberhollenzer
As a side effect, this requires the data writer to keep track of statistics. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-09-14Move data deduplication from fstree code to data writerDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-09-01Move some application specific stuff out of libutilDavid Oberhollenzer
This commit does the following: - canonicalize_name is moved to libfstree - source_date_epoch is only used inside libfstree, so it's also moved over and can later be completely internalized - print_version is moved over to sqfshelper. Mainly so it doesn't end up in libsquashfs.so for no sane reason. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-08-20Add explicit padding to tree_node_tDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-08-18cleanup: internalize deduplication list in data_writerDavid Oberhollenzer
This change removes the need for passing a list of files around for deduplication. Also the deduplication code no longer needs to worry about order, since the file being deduplicated is only added after deduplication is done. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-08-11Add gensquashfs option to read xattrs from input filesDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-08-11Add --one-file-system option to gensquashfsDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-08-11Replace fstree_from_dir boolean with flag fieldDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-08-04Improve file unpacking orderDavid Oberhollenzer
This commit moves the file unpacking order & job scheduling to a libfstree function. The ordering is improved by making sure fragment blocks are not extracted more than once and files with data blocks are extracted in order. This way, serial unpacking of a 2GiB Debian live image could be reduced from ~5' on my test machine to ~3.5', whereas parallel unpacking stays roughly the same (~3' for -j 4). Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-30Add propper copyright headers to all source filesDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-29Cleanup: move deduplication code from data writer to fstreeDavid Oberhollenzer
Since it is actually completely independend of libsqfs and only works on file_info_t lists, it can be safely moved over to libfstree and the data writer becomes less cluttered as a result. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-28Add general purpose flags field to file_info_tDavid Oberhollenzer
Simplifies some task if we can just add a flag that a file has a framgent or that it has already been detected as a duplicate. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-28Add fragment and block checksum fields to file_info_tDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-25Generate linear file list in fstreeDavid Oberhollenzer
Instead of doing DFS on the fly in gensquashfs, churn out a linked list of all files in an archive. Future improvements in packing strategies can go into this file. This can also be usefull for other purposes in the future, such as file deduplication or as a work queue for the unpacker. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-24Enable largefile supportMatt Turner
Requires that config.h be included before other headers, since the macro _FILE_OFFSET_BITS changes the definitions of things like 'struct stat'. I chose to simply include it at the top of every C file and at immediately after the double-inclusion guards of every header. Signed-off-by: Matt Turner <mattst88@gmail.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-22Add a way to optionally keep the original time stampsDavid Oberhollenzer
First of all, this commit adds a mod_time field to a tree node. When creating the tree node, the field is set from the struct stat. When scanning a directory, the time stamps from the input are used if set. Second, the libsqfs code that reads inodes is modified to store the mod_time from the inode in the fstree node and to write the tree node into a generated inode. Finally, tar2sqfs is modified to optionally keep the timestamps from the tar archive instead of setting defaults. gensquashfs is similarly modified to keep the input timestamps if specified. The result is as follows: - sqfs2tar will always carry the timestamps from the squashfs over to the tar ball. - tar2sqfs will set defaults, unless explicitly asked to preserve the mtime from the tar ball. - gensquashfs can optionally preserve the mtime from the input hierarchy it processes if only --pack-dir is specified. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-21Cleanup xattr handlingDavid Oberhollenzer
- Store them in a struct instead of a hacky uint64_t with magic shifts - Split up key/value pair write function to write_key and write_value - Move the size accounting into those functions respectively Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-03cleanup: move tree node from path function to libfstree.aDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-06-28Add support for packing sparse filesDavid Oberhollenzer
This commit adds support for packing sparse files into squashfs images as follows: - In the data writer: simply detect zero blocks and write a zero to the block size field and don't emit any data. Record the number of bytes saved this way. For fragments, set the fragment offset to invalid. - In the inode writer: write out the number of bytes saved for sparse files. If there should be a fragment but there is none, append a block count of 0. - In the data reader: if the block size is 0, read nothing from disk and emit an empty block. Do the same if the fragment is missing. - In the inode reader: restore the number of bytes saved for sparse files. The sparse files can be packed and unpacked, but the unpacking will not create sparse files for now. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-06-23Move fstree default option processing to fstree codeDavid Oberhollenzer
Instead of decomposing a default string in gensquashfs option processing, move that to fstree_init instead and pass the option string directly to fstree_init. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-06-22Cleanup: unify packdir/packfile based directory changes in gensquashfsDavid Oberhollenzer
This commit removes the packdir/packfile based directory setup magic from fstree_from_file and moves it to gensquashfs. Over there, the common parts are deduplicated. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-06-22Cleanup: split fstree sort into 2 fstree independend functionsDavid Oberhollenzer
Make tree node list sort and recursive variant available and independend of the fstree_t. This is considered cleaner, since the fstree_t actually isn't needed for any of this and we can just call the recusvie sort on the root instead, and we can use the sort implementation directly for things like the upcoming unit test. Also this commit splits up the merge/sort implementation into a seperate split and merge functions to make the code somewhat more readable. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-06-19Cleanup fstree codeDavid Oberhollenzer
Replace default_$FIELD with a struct stat. Merge the rest of add_node/add_file into mknode and add by path+stat. Expose the mknode function and replace all the duplicated node pointer arithmetic magic with a call to mknode. Plus a generic cleanup by utlizing new util functions. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-06-17libfstree: add generic function to add node from struct statDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-06-15libfstree.a: add function to get struct stat from tree nodeDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-06-11cleanup: remove the linked list of fragment relationsDavid Oberhollenzer
Don't need that, we already know the index when we write add the fragment. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-06-10Add generic function to produce a full path from an fstree nodeDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-06-10Generate a flat inode table from the fstree ahead of timeDavid Oberhollenzer
Instead of allocating inode numbers as we go, generate and populat an inode table from the fstree ahead of time. This makes processing nodes a little bit simpler and we will need that table anyway for NFS export support later on. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-06-10gensquashfs: do pushd/popd when needed instead of chdirDavid Oberhollenzer
This commit replaces the chdir to the input directory with pushd/popd when building the fstree and again when packing files. This simplifies handling of other file paths given on the command line that have to be accessed and are relative to the original working directories. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-05-31gensquashfs: allow combining packdir and packfileDavid Oberhollenzer
If packdir and packfile are both specified, use packdir as alternate root. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-05-26gensquashfs: add option to simply pack an input directoryDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>