| Age | Commit message (Collapse) | Author |
|---|
|
Currently, when the block processor aggreagtes fragments into a
fragment block, it applies the "don't compress" flag if any of the
original framgnets has it set, but the "align to device block" flag
is lost.
This commit ensures that both flags get applied to the fragment block
if set.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This commit is an amalgamation of the commits on master that
implement exact matching of fragment blocks during deduplication.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Keeping a list of fragments stored away in the current fragment block
and consolidating them in the thread pool takes them out of circulation.
If we have a lot of tiny fragments, this can lead to a situation where
all the limit is reached, but we cannot do anything, because we are
waiting for a block to complete, but they are all attached to the
current fragment block and the queue is empty.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Fragment deduplication really doesn't belong into the public API of
the fragment table.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This commit modifies the block processor to support operating without
a fragment table. If that is the case, fragment deduplication is
essentially disabled and fragment blocks aren't indexed anymore.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This function allows submission of raw blocks to the block processor,
completely bypassing the file API.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This commit adds 2 new user settable flags to the block processor:
- A flag to ignore sparse blocks and treat them like normal
data blocks.
- A flag to disable checksum computation altogether.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This commit modifies the block processor to support associating a user
data pointer with data blocks that it forwards to the block writer,
which is modified to accept an optional user data pointer.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This way, everything that could be done through the hooks (and more)
can be done by simply providign a custom implementation. The result is
a lot clener that the previous hook based version.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
- the "bytes submitted" can be moved over to the block processor
- the number of blocks submitted are already there (implcitily, by
adding the data block count to the fragment block count)
- actual data bytes written can be computed from the super block
- the remaining block count can be changed to simple counter that
can be obtained through a function.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
- Move the inode modifications out of do_block. The inode may be
reallocated in parallel by the process_completed_block function, so
it is not safe to store the fragment location in the do_block
function which is used from the worker threads.
- Move the accounting of fragment blocks to the
process_completed_block function.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This commit breaks the common code up again by moving the data submission
code to a separate file, making both a little bit more readable.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Instead of [potentially] allocating a new fragment block, take an
existing fragment and promote it to the fragmenet block. This saves
as a potential block allocation and a memcpy of the initial data.
Also it *definitely* removes block allocation from the backend path
of the block processor.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Instead of merging fragments into the fragment block inside the
process_completed_fragment function, store a linked list of fragments
in the fragment block and do the actual merging (several memcpy calls
totaling of up to 1M of data in worst case) in the worker thread
instead of the locked, serial path.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Instead of freeing/allocating blocks all the time in the locked,
serial path, use a free list to "recycle" blocks. Once a block is
no longer used, throw it onto the free list. If a new block is,
needed try to get one from the free list before calling malloc.
After a few iterations, the block processor should stop allocating
new blocks and only re-use the ones it already has.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
In the block processor, the payload area is only accessed up to
the indicated size. Even the part that is accessed is initialized
by copying data into the block before increasing the size, so there
is no real point in zero-initializing hundres of kilobytes if not
megabytes of payload area, especially since this is done in the
locked, serial path of the block processor.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
If a file consisting of multiple blocks is produced, the last block is
short and the don't fragment flag is set, the last block flag has to
be set on the block when we flush it, so the processing pipeline does
it's job correctly.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
If the block processor allocates and dynamically resizes inodes on
the fly, we can add data indefinitely without knowing the size of
the file ahead of time.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Instead of having seperate counters for blocks, dir index bytes
and having to fiddle out the link target size, simply use a single
value that stores the number of payload bytes used.
A seperate "payload bytes available" is used for dynamically
growing inodes during processing.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Implement the io-queue based design as outline in doc/parallelism.txt
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
On the one hand, benchmarking and profiling determined xxhash32 to be
faster than the zlib implementation of crc32, on the other hand
profiling determined that crc32 computation contributed signifficantly
to the overall runtime.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This commit moves all of the fragment/block accounting in the block
processor over to the writing end of the pipeline. In order to do
this, the sparse blocks are allowed to bubble through the pipeline.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Again, the generic init/cleanup functions do way too many things that
are specific to the thread pool implementation. Thanks to the splitting
up of the block processor, they also have become quite trivial. This
commit moves those functions into their respective implementations,
allowing even further simplificiation.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
The sqfs_block_t structure has been written for the block processor
and exposes way too many internals. This commit removes its usage
from the block writer, cutting it down to the bare essentials, so
the structure can be removed from the public API later on.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
There is no obvious non-footgun use for those other than tallying
statistics, which is now done by the data structures themselves.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
This commit moves the entire block writing and deduplication of data
blocks over to a different data type named "block writer".
For simplicity, the interfaces of the block processor are left as is
and are turned into warppers. Likewise, most of the code in the block
writer is just verbatim from the block processor, to be cleaned up in
subsequent commits.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
|
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
|
