aboutsummaryrefslogtreecommitdiff
path: root/lib/fstree
AgeCommit message (Collapse)Author
2023-04-20Collect and print statistics about the kind of files we are packingDavid Oberhollenzer
Using depth-first search, we collect some crude statistics about directory tree types (e.g. regular files, directories, device special files and so on) and print them out after serializing the tree. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2023-04-20libfstree: Remove special modes for hard linksDavid Oberhollenzer
Instead of adding special sentinel modes, simply treat hard links as special case of symlinks, setting a flag to indicate that it is a hard link and another flag to indicate that it has been resolved. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2023-04-19libfstree: Make hard link resolution non-recursiveDavid Oberhollenzer
Use the next_by_type pointer to create a list of all unresolved hard links and iterate over that list for link resolution. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2023-04-19libfstree: hoist file link pointer into parent structureDavid Oberhollenzer
Instead of having a file_info_t next pointer, requiring an up-cast to tree_node_t all the time, simply add a "next_by_type" pointer to the tree node itself, which can also be used for other purposes by other node types and removes the need for up-casting. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2023-04-19libfstree: get rid of dir_info_tDavid Oberhollenzer
The single boolean created_implicitly can be replaced with a general purpose flag field. The "children" pointer can then be hoisted directly into the data union of tree_node_t. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2023-04-17Internalize fstree_insert_sortedDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2023-04-06libfstree: simplify hard link resolution codeDavid Oberhollenzer
We do not allow hard links to directories, so we can toss the special case handling code for that. The visited mechanism was pointless anyway, because we don't even descend down hard links in the recursive tree handling functions. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2023-04-06Cleanup: libfstree: move all the hardlink related code to hardlink.cDavid Oberhollenzer
For some reason, the recursive hardlink resolution ended up in post process, calling into the non recrusive one in hardlink.c that wasn't used elsewhere. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2023-04-02Move fstree CLI code to libcommonDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2023-01-31Reintegrate test code with library codeDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2023-01-31Move library source into src sub-directoryDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-11-22Move gensquashfs specific code from libfstree to gensquashfsDavid Oberhollenzer
The "from dir" and from "from file" code, as well as the "sort file" code is specific to gensquashfs, so move them there and the test cases as well. The medium term idea is to reduce libfstree to a stub, merge it into the generic writer and ultimately hoist that into libsquashfs. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-07-08Cleanup: Get rid of libfstree "internal.h" headerDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-07-08Cleanup: move source date epoch code back to libutilDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-07-08Cleanup: move filename_sane & canonicalize_path functions to libutilDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-07-08Cleanup: rename libfstream to libio, split headersDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-07-08Fix: libfstree: actually use a full 32 bit hard link counterDavid Oberhollenzer
The squashfs on-disk format uses 32 bit link counters, but the fstree used 16 bit ones. Because the link count also includes child nodes, this artificially limited the number of entries in a directory to ~64k files. This patch removes the limit by switching libfstree to 32 bit counters. Reported-by: Marvin Renich <mrvn@renich.org> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-07-08Fix: libfstree: double free in error pathDavid Oberhollenzer
If fstree_mknode fails, because the parent link count would overflow, the function fails and cleans up behind it. The problem arises because the function does this check *after* inserting the node in the parent node, so it is later free'd again, when destroying the rest of the tree. This patch moves the insertion after the check to mitigate the problem. Reported-by: Marvin Renich <mrvn@renich.org> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-06-03Fix: libfstree: free() path string in error code pathDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-03-30Fix: Directory insertion order on WindowsDavid Oberhollenzer
The fstree sorting code got removed recently, in favour of inserting at the correct position. The Windows directory scanning code still used it's own list insertion code instead of mknode (so it could allocate and translate the directory entry name in-place), which broke the sorting order. This issue is fixed in this commit. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-03-30Add a rudimentary unit test for sort filesDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-03-30Add sort-file implementationDavid Oberhollenzer
A `flags` field and `priority` are added to all file information structs. A news fstree function is introduced for parsing a "sort-file". Each line in the file is space separated, and has the following format: priority [flags] filename Priority is a 64 bit number, flags are optional and filename can be put in quotes if it is supposed to start or end with spaces. Single line comments can be used. The flags can be used to set block-processor flags (e.g. don't fragment, or don't compress), as well as instructing the parser to use file globbing to match the filename. After parsing the file, the list of file info structure is sorted according to the priority (default is 0) using a stable sort algorithm. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2022-03-30Cleanup: remove node sorting from libfstreeDavid Oberhollenzer
Always insert the tree nodes in the correct oder and remove the post-process sorting step. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-06-25libfstree: guard against possible overflow in readlink()David Oberhollenzer
*in theory*, say on a 32 bit system, we could have a 32 bit size_t and a 64 bit off_t. If the filesystem permitted this, we *could* then have a symlink with a target > 4G. Or the target is exacetely 4G, but adding a null-terminator could exceed addressable memory. This commit adds a check to guard against such an overflow and throw an error, instead of silently wrapping around. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-06-25libfstree: guard against link count and inode number overflowDavid Oberhollenzer
If the hard link counter or the inode number counter overflow the maximum representable value (for SquashFS 16 bit and 32 bit respecitively), abort with an error message. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-03-25libfstree: allow the glob path to be emptyDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-03-25libfstree: add an assertion that root is not NULLDavid Oberhollenzer
If the path argument is "", we assume that referes to root and set the *existing* target node to the root node and skip ahead across the tree search. This leaves "name" uninitialized, which makes coverity panic, because fs->root could be NULL, going down the wrong path. Obviously, this should never, *ever* happen and there is no reasonable recovery strategy if it suddenly does, so simply add an assertion. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-03-25libfstree: Allow / as argument for "glob" and "dir" commandsDavid Oberhollenzer
This allows putting globbed files & directories into the filesystem root, as well as explicitly setting attributes of the root directory from the file lisiting. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-03-24libfstree: implement directory scanning code for WindowsDavid Oberhollenzer
It's rather simplistic and doesn't account for junction/reparse points, which is the closest thing Windows has to symlinks, hard links and mount points, but it's consistent with the unpacking code that assumes Windows only has files and directories. Using the 32 bit mingw toolchain, this seems to satisfy the unit tests on wine. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-03-20Add libcompat fallback implementation for fnmatchDavid Oberhollenzer
This has basically been copied over from Musl and slightly modifed. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-02-19Fix: libfstree: add an assert the canonicalize_name return valueDavid Oberhollenzer
Since the canonicalize_name function only fails if the path contains ".." and the one we are constructing from the scanned fstree (built using canonicalized names), it should NEVER fail. However, coverity does get concerned, because we are checking the return value elesewhere. So do what we do at other, similar locations and add an assert(). Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-02-19Fix: canonicalize path names in glob pattern matchingDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-02-19libfstree: reject unknown glob options to allow future expansionsDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-02-19fstree_from_file: Add fnmatch() pattern matching to file globbingDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-02-19fstree_from_file: Implement basic file globbingDavid Oberhollenzer
The base path is passed to the fstree_from_file function and in turn to the individual callbacks. The line parsing function is modified to allow '*' as mode, uid and gid for specifically marked callbacks. A glob callback is added that internally uses the fstree_from_dir scanning functions in combination with a filter callback. Directory scanning flags are parsed from the extra arguments before interpreting it as a path fragment. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-02-19libfstree: Add a filter callback to the directory scanning functionDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-02-18libfstree: add a subdirectory scanning functionDavid Oberhollenzer
So we can scan a sub-directory within a the base directory without having to do string operations first. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-02-18fstree_from_dir: add filtering flags to skip certain inode typesDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2021-02-10cleanup: fstree_from_file: split & simplify line parsing functionDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-12-29Fix normalization of slashes in filenamesDavid Oberhollenzer
All paths were canonicalized internally, which includes filtering sequences of slashes and converting backslashes to slashes. Furthermore, when unpacking files, filenames are sanity checked and rejected if they contain forward OR backward slashes. This is a problem on Unix-like systems, where files containing backslashes are a legitimate use case (*cough* SystemD *cough*). This patch removes the backslash conversion from the canonicalization and modifies the sanity check to reject backslashes only on Windows. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-12-15libfstree: make the directory scanning code a little more genericDavid Oberhollenzer
- Instead of using the fstree root, let the caller specify it. - Add a flag to prevent recursion into sub directories. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-12-10Move fstree dirscan code back to libfstreeDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2020-09-29Replace file/getline usage with istreamDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-23Simplify hard link handlingDavid Oberhollenzer
- For now, enforce that hard links don't point to a directories. - Instead of doing the swaping trickery, just reorder the flat list and hand out new inode numbers. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-23Minor cleanup in inode allocationDavid Oberhollenzer
- Remove unnecessary counter argument, we already have the total count. - Remove the return status, there is no failure branch. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-23Bring back the flat list of inodes in libfstreeDavid Oberhollenzer
It makes further processing simpler and doesn't leak the abstraction into upper layers. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-22Add hard link support to gensquashfs and tar2sqfsDavid Oberhollenzer
In libtar, set a special flag if the header is actually a hard link. In tar2sqfs, create a hard link node and skip the rest for hard links. Also refues to set the root attributes from a hard link, it may refere to a node that we have missed earlier, there is nothing else that we can do here. In fstree_from_file, add a "link" command for adding hard links. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-22Add basic support for handling and serializing hard linksDavid Oberhollenzer
In libfstree, add a function to add a hard link to the fstree. The hard links stores the target in the data.target field, canonicalizes the target and sets a sentinel mode. A second function is used to resolve link, i.e. replacing it with a direct pointer, setting another sentinel mode and increasing the targets link count. The post process function tries to resolve unresolved hard links and only allocates inode numbers for nodes that aren't hard links. If the target node of a hard link does not have an inode number yet, the two need to be swapped, since this is also the order in which they are serialized. The serialization function in libcommon simply has to skip hard link nodes and when writing directory entries, use the inode num/ref of the target node. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-19Split the libfstree add_by_path tree traversal function outDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-12-18Add an explicit link count to the fstree nodesDavid Oberhollenzer
Gets initialized to 2 for directories, 1 for all other types. The count of the parent node is automatically incremented. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>