aboutsummaryrefslogtreecommitdiff
path: root/mkfs
AgeCommit message (Collapse)Author
2019-08-18cleanup: internalize deduplication list in data_writerDavid Oberhollenzer
This change removes the need for passing a list of files around for deduplication. Also the deduplication code no longer needs to worry about order, since the file being deduplicated is only added after deduplication is done. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-08-18Add pthread based, parallel block processor implementationDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-08-11cleanup: zero-initialize gensquashfs options with memsetDavid Oberhollenzer
Manual initialization is not only verbose, it actually caused a bug in a past version. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-08-11cleanup: replace gensquashfs bool options for dir scan with flagsDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-08-11Add gensquashfs option to read xattrs from input filesDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-08-11Fix wrong argument type for gensquashfs --keep-timeDavid Oberhollenzer
Copy paste mistake: the propper type should be no_argument as it is only a simple flag and doesn't take extra arguments. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-08-11Split gensquashfs help string because it got too longDavid Oberhollenzer
There is an upper limit of characters in a string literal that it managed to cross. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-08-11Add --one-file-system option to gensquashfsDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-08-11Replace fstree_from_dir boolean with flag fieldDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-30Add propper copyright headers to all source filesDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-28Add some nice statistics output to tar2sqfs and gensquashfsDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-28Implement fragment deduplication in data writerDavid Oberhollenzer
The strategy is simple: - The data writer function that write data/fragment blocks get access to the list files. - When writing a fragment, we look for an already written file that has a fragment with the same size and checksum. - If we find one, we throw away the fragment and reuse the existing one. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-25Generate linear file list in fstreeDavid Oberhollenzer
Instead of doing DFS on the fly in gensquashfs, churn out a linked list of all files in an archive. Future improvements in packing strategies can go into this file. This can also be usefull for other purposes in the future, such as file deduplication or as a work queue for the unpacker. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-25cleanup: remove unneccessary conditionals from automake filesDavid Oberhollenzer
If some library isn't present, the variable $(FOO_LIBS) simply evaluates to empty string, so it can be simply be added to LDADD without any checks. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-25cleanup: remove some redundant header inclusionsDavid Oberhollenzer
The utility programs pretty much all have the same structure of including one central header per C file that includes all required library headers, so we can simply include config.h which we need for large file support from there and remove it from the C files. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-24Enable largefile supportMatt Turner
Requires that config.h be included before other headers, since the macro _FILE_OFFSET_BITS changes the definitions of things like 'struct stat'. I chose to simply include it at the top of every C file and at immediately after the double-inclusion guards of every header. Signed-off-by: Matt Turner <mattst88@gmail.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-22Add a way to optionally keep the original time stampsDavid Oberhollenzer
First of all, this commit adds a mod_time field to a tree node. When creating the tree node, the field is set from the struct stat. When scanning a directory, the time stamps from the input are used if set. Second, the libsqfs code that reads inodes is modified to store the mod_time from the inode in the fstree node and to write the tree node into a generated inode. Finally, tar2sqfs is modified to optionally keep the timestamps from the tar archive instead of setting defaults. gensquashfs is similarly modified to keep the input timestamps if specified. The result is as follows: - sqfs2tar will always carry the timestamps from the squashfs over to the tar ball. - tar2sqfs will set defaults, unless explicitly asked to preserve the mtime from the tar ball. - gensquashfs can optionally preserve the mtime from the input hierarchy it processes if only --pack-dir is specified. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-21Implement generating an inode table for NFS exportDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-17fstree: add support for spaces in filenamesDavid Oberhollenzer
This commit adds a mechanism to fstree_from_file to support filenames with spaces in them by quoting the entire string. Quote marks can still be used inside file names by escaping them with a backslash. Back slashes (if that is your thing) can also be escaped. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-15Add flags to data writer to micro manage behaviourDavid Oberhollenzer
The added flags allow controlling the following on a per file level: - forcing a file to be written uncompressed - forcing a file to not have a fragment, i.e. the last truncated block actually being written as a block - padding a file to be alligned to device block size The flags are not yet exposed to anything user controllable (such as command line flags). Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-07-07Fix: always initialize gensquashfs defaults optionDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-06-23gensquashfs: Remove rest of read_number from options parserDavid Oberhollenzer
Use a simple strtol for block size and device block size: - We don't need to check the block size, sqfs_super_init does that. - We only need to make sure the device block size is at least 1k. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-06-23Move fstree default option processing to fstree codeDavid Oberhollenzer
Instead of decomposing a default string in gensquashfs option processing, move that to fstree_init instead and pass the option string directly to fstree_init. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-06-23Move all handling of compressor names to libcompress.aDavid Oberhollenzer
This commit removes handling of compressor names from gensquashfs. Instead, functions are added to libcompress to obtain name from ID, ID from name and to print out defaults. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-06-23gensquashfs: remove command line number parsing codeDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-06-23gensquashfs: Fix typo in help textDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-06-22Cleanup: unify packdir/packfile based directory changes in gensquashfsDavid Oberhollenzer
This commit removes the packdir/packfile based directory setup magic from fstree_from_file and moves it to gensquashfs. Over there, the common parts are deduplicated. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-06-22Cleanup: split fstree sort into 2 fstree independend functionsDavid Oberhollenzer
Make tree node list sort and recursive variant available and independend of the fstree_t. This is considered cleaner, since the fstree_t actually isn't needed for any of this and we can just call the recusvie sort on the root instead, and we can use the sort implementation directly for things like the upcoming unit test. Also this commit splits up the merge/sort implementation into a seperate split and merge functions to make the code somewhat more readable. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-06-22simplify SELinux labelingDavid Oberhollenzer
This commit moves the SELinux label code after the tree is sorted and the inode table is generated. Sorting helps to make sure that the tree will always be traversed in a defined, deterministic order and likewise the creation of xattrs happens in a defined, deterministic order. Second, we can now use the inode table instead of having to implement a recursive tree traversal yet again. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-06-15Move function to padd a file with 0 bytes to libutilDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-06-11Cleanup dependency handlingDavid Oberhollenzer
Always try to gather all compressor libraries available, but only complain about missing a one if it has been *explicitly* selected. If a compressor has been explicityl disabled, we can still turn it off after checking. Also, rework gensquashfs to set the default compressor based on what's available. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-06-11cleanup: mkfs: remove redundant header inclusionsDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-06-11cleanup: merge rest of block.c into mkfs.cDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-06-11cleanup: remove rest of sqfs_info_tDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-06-11Move write_xattr to libsqfs.aDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-06-11Remove sqfs_info_t from xattr writerDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-06-11Move data writer to libsqfs.aDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-06-11Encapsulate data/fragment processing in data_writer_t data structureDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-06-11Split process_file functionDavid Oberhollenzer
Add a generic one that reads and processes N bytes from an input file descriptor. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-06-11Major cleanup of mkfs block processorDavid Oberhollenzer
This commit tries to make the block and fragment processing code a lot more readable by simplifying it as follows. The following pattern is used repeatedly: - Try to compress a block - Write the compressed if smaller (the original otherwise) to a file - Print errors on the way - Set the magic (1 << 24) flag if we wrote the original - Increment the super block byte counter => Move that to a seperate function called "write_compressed" => Move the code to grow the fragment table is moved to a seperate function. => Clearing the fragment buffer is pointless, so remove that The add_fragment/add_block functions are now trivial => Merge them into process_file => Global block counter no longer needed, so remove that too Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-06-11cleanup: remove the linked list of fragment relationsDavid Oberhollenzer
Don't need that, we already know the index when we write add the fragment. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-06-11fix: flush the last, unfinished fragment block as wellDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-06-11cleanup: remove unused prototypesDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-06-10Rename table.h to highlevel.hDavid Oberhollenzer
The idea is to move various higher level helper functions there. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-06-10cleanup: move entire fstree serialization to libsqfs.aDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-06-10Make sqfs_write_inodes independend of sqfs_info_tDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-06-10cleanup: move write_inode to separate fileDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-06-10Generate a flat inode table from the fstree ahead of timeDavid Oberhollenzer
Instead of allocating inode numbers as we go, generate and populat an inode table from the fstree ahead of time. This makes processing nodes a little bit simpler and we will need that table anyway for NFS export support later on. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-06-10cleanup: move directory serialization to separate fileDavid Oberhollenzer
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
2019-06-10gensquashfs: do pushd/popd when needed instead of chdirDavid Oberhollenzer
This commit replaces the chdir to the input directory with pushd/popd when building the fstree and again when packing files. This simplifies handling of other file paths given on the command line that have to be accessed and are relative to the original working directories. Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>