New I/O Layer, Sparse Files and Sub-Second Timestamps

New I/O Layer

A completely new I/O layer abstraction replaces the ubiquitous use of memory mapped files across the DwarFS code base. Memory-mapping is still the default, but processing is done in "segments" rather than in whole files. This required a significant amount of changes (this release adds/touches more than 5,000 lines of code and almost 10,000 lines of new tests) in almost every part of the code that was processing file data, code that previously assumed any file could simply be accessed as a contiguous piece of memory.

In the new abstraction layer, backends are pluggable and configurable through the DWARFS_IOLAYER_OPTS environment variable, letting you:

• Configure the size up to which files are mapped "eagerly", i.e. as a whole and not in segments. This is mostly relevant for 32-bit systems, on which this is set to a reasonable default (32 MiB).
• Switch from mmap() to classic read() for maximum robustness on unreliable storage or faulty hardware.

The latter is relevant if you're seeing "bus errors" (SIGBUS), as many have done in the past (#45, #50, #108, #163, #213). You can switch to the read-based backend using:

$ export DWARFS_IOLAYER_OPTS=open_mode=read

Sparse File Support

This release includes end-to-end sparse file support:

• mkdwarfs detects holes in files and preserves sparseness in the image.
• The FUSE driver exposes sparsity via lseek() where supported (Linux, FreeBSD).
• dwarfsextract writes sparse files and preserves them when targeting archive formats that support holes (e.g., tar).

Compatibility: images containing sparse files require DwarFS ≥ 0.14.0. You can use --no-sparse-files to explicitly treat sparse files as non-sparse and keep compatibility with older versions. If your input does not contain sparse files, the images remain backwards-compatible even without that flag.

Sparse file support matrix

FUSE-level sparseness requires lseek() support in the FUSE implementation (currently Linux and FreeBSD). On macOS and Windows, files are exposed via FUSE as non-sparse even though dwarfsextract can still write sparse files when extracting. Missing lseek() support is tracked here for Windows and here for macOS.

Sub-second timestamps

Configurable sub-second timestamp resolution down to nanoseconds (using --time-resolution). The default remains one second. This is fully backwards-compatible: older DwarFS versions can read images with sub-second resolution, but will ignore the sub-second part.

Bug fixes

• Leading dots in --input-list file paths were incorrectly treated as literal directory names instead of being expanded. This has been fixed. Fixes #292.

• The SPDX license identifier in GPL-licensed source files was incorrectly specified as GPL-3.0-only instead of GPL-3.0-or-later. This has been corrected. Fixes #275.

• Fixed an off-by-one error when recovering self_index fields in metadata, which could cause the sentinel directory to have a non-zero self_entry. While harmless by itself (since that entry is never actually used), this would cause the metadata consistency check to fail. The fix covers three aspects: correcting the off-by-one error; ensuring the self_entry recovery code does not run for the sentinel directory; and changing the metadata consistency check to only warn about a non-zero self_entry rather than fail. Running mkdwarfs with --rebuild-metadata will also reset a non-zero sentinel self_entry to zero.

• Fixed the implementation of the read operation in the FUSE driver to send positive error code values to libfuse. This was likely never triggered in practice, but in cases where parts of the filesystem image vanish while being accessed (which previously caused SIGBUS crashes), libfuse would not understand the negative error codes.

• Moved the FUSE driver binaries from sbin to bin and kept only the mount.dwarfs/mount.dwarfs2 symlinks in sbin. This better aligns with user expectations, other FUSE drivers, and the fact that the man pages are installed in section 1. (Thanks to Ahmad Khalifa for the fix.)

• The dwarfs2 binary was broken in builds using shared libraries. (Thanks to Ahmad Khalifa for the fix.)

• When setting CPU thread affinity for worker group threads via DWARFS_WORKER_GROUP_AFFINITY, the code did not CPU_ZERO the cpu_set_t structure before setting individual CPUs. This could pin threads to random CPUs in addition to the requested ones.

• The FITS categorizer would scan entire files for the end-of-header marker if their size was a multiple of 2880 bytes, causing significant slowdowns on large non-FITS files. Additional checks now ensure scanning only continues if the data truly looks like a standards-compliant FITS header.

• GCC caught a potential null-pointer dereference on error when opening a file in mkdwarfs. This has been fixed.

• Numerous fixes for 32-bit architectures, mostly related to integer overflows with file sizes larger than 4 GiB.

• Another off-by-one error caused the first regular file inode to be excluded from the file-size cache. This would be hard to notice unless that file was highly fragmented. The cache will be fixed when rebuilding the metadata.

• The FUSE driver’s enable_nlink option is now the default behavior and cannot be disabled. The previous optimization skipped building a table of hardlink counts, which produced inherently incorrect file status information (hardlinked files share an inode, so reporting a link count of 1 is wrong). The hardlink table is now stored in the metadata by default; if there are no hardlinks, it consumes no space. You can still omit the hardlink table with --no-hardlink-table, at the cost of building it on-the-fly when the filesystem image is loaded (typically fast — e.g., ~300 ms for 14 million files).

• Fixed a typo in dwarfs-format.md. (Thanks to Dennis Brakhane for spotting this and sending a PR.)

Features

• New I/O layer abstraction that supports “classic” mmap-based file access, granular mmap-based access on 32-bit systems, and fully mmap-less access if desired. This applies to all DwarFS tools. By default, tools use the most efficient method—memory-mapping whole files on 64-bit systems and mapping file segments on 32-bit systems (to conserve address space). This can be controlled via the new DWARFS_IOLAYER_OPTS environment variable described in dwarfs-env(7).

• Full support for sparse files. mkdwarfs now detects and efficiently processes sparse files, skipping holes where possible and preserving them in the filesystem image. This is supported on all platforms. The FUSE driver implements lseek() where supported by the FUSE library (currently Linux and FreeBSD); Windows and macOS fall back to showing files as non-sparse. dwarfsextract extracts sparse files as such and preserves sparse representations when extracting to archive formats that support them (e.g., tar). Note: Sparse file support is not backwards compatible; images containing sparse files cannot be processed by DwarFS versions prior to 0.14.0. By default, mkdwarfs enables sparse file support if it detects sparse input. Use --no-sparse-files to disable it and ensure compatibility with older versions.

• Support for subsecond timestamp resolution. The default remains one second, but finer resolutions (down to nanoseconds) can be specified with --time-resolution. mkdwarfs will warn if the requested resolution is finer than the native filesystem resolution. This is fully backwards compatible: older DwarFS versions will handle such images but ignore the subsecond parts. Fixes #294.

• Desktop integration for Linux. A new --auto-mountpoint option automatically creates or selects a mount-point directory, making it easier to mount DwarFS images from file managers. Desktop files and MIME type definitions are now installed to enable double-click mounting of .dwarfs files. (Thanks to Ahmad Khalifa for the implementation.)

• Shell completion for mkdwarfs (bash and zsh). (Thanks to Ahmad Khalifa for the contribution.)

• Improved error handling when DwarFS tools encounter SIGBUS (usually caused by accessing memory-mapped files on unreliable or faulty storage like network shares or flaky USB drives). When SIGBUS is caught, tools now print an error suggesting switching from mmap- to read-based I/O via DWARFS_IOLAYER_OPTS.

• dwarfsck now checks metadata consistency by default (unless --no-check is given), improving detection of filesystem image corruption.

• If sparse files are supported by the FUSE library, the FUSE driver exposes new options cache_sparse and no_cache_sparse to control whether sparse files should be cached in the kernel page cache. See dwarfs(1) for details.

• The JSON output from dwarfsck now contains a complete raw metadata dump when the detail level includes metadata_full_dump.

• dwarfsck no longer artificially limits string sizes when dumping metadata. (Thanks to Dennis Brakhane for the contribution.)

• Accelerated search for the start of a DwarFS image in files with custom headers; the new code is about four times faster, scanning at more than 6 GiB/s on a modern CPU.

• The cache size can now be configured for dwarfsck, useful with the --checksum option.

• Both dwarfsck and dwarfsextract now limit the amount of ...

FreeBSD, big-endian, and many new architectures

This release finally includes FreeBSD in the list of supported operating systems. That should make it much easier to port DwarFS to other *BSDs as well. Big-endian platforms are also supported, and the file system images (which use little-endian) are fully portable between architectures. Binary releases are now available for a wide range of architectures: aarch64, arm, i386, loongarch64, ppc64, ppc64le, riscv64, s390x, and x86_64.

Metadata manipulation

Before this release, file system metadata was immutable once a DwarFS image had been created, and the only way to manipulate metadata was to build a new image from scratch. This release adds two options to mkdwarfs:

• --rebuild-metadata allows changes/upgrades to the metadata block. This makes it easy to change how the metadata is packed, or even perform manipulations such as --chmod after the fact.
• --change-block-size allows you to change the physical block size of the file system image.

Bug fixes

• The linker configuration for the release binaries was broken. The symptom was that, very occasionally, tests would fail in CI with std::terminate being called after the exception handling code failed to unwind the stack. The root cause was that, in clang builds, code from libunwind and libstdc++ was arbitrarily mixed, which—depending on the order in which individual threads were scheduled in the unit tests—could lead to stack unwinding working flawlessly or not at all. This was one of the hardest bugs to track down this year; fortunately, the fix was quite simple. It’s possible this issue is present in previously released binaries, although there have been no reports.

• Made section index discovery more robust. Fixes #264.

• A recent kernel change (LKML thread, 2025-05-05) caused tools_test to fail on Linux 6.14 and later. This has been fixed by accepting both EPERM and ENOSYS as valid error codes for link() calls.

Features

• FreeBSD support. Everything that works on Linux should also work on FreeBSD. No static binaries are provided for FreeBSD, but the build should work out of the box once all dependencies are installed.

• Big-endian architectures. This is still experimental, even though all unit tests pass under QEMU and the benchmark suite runs on real hardware. This currently requires forked versions of folly and fsst. The changes are small, and the pull requests will hopefully be merged upstream soon. (facebook/folly#2484, cwida/fsst#36)

• Experimental 32-bit support. While DwarFS should largely “just work” on 32-bit with small images (a few hundred megabytes), limited address space is a problem due to DwarFS’s extensive use of memory-mapped files. There will be changes to limit the use of mmap in the future (primarily due to other issues), which should improve 32-bit compatibility as a side effect. Fixes #268.

• Wider CPU architecture coverage. Static binary releases (including universal binaries) are now available for x86_64, aarch64, i386, arm, ppc64, ppc64le, riscv64, s390x, and loongarch64. Building the new release binaries uncovered a few bugs in clang (llvm/llvm-project#150913), binutils (https://sourceware.org/bugzilla/show_bug.cgi?id=33223), mold (rui314/mold#1490, rui314/mold#1496, rui314/mold#1497, rui314/mold#1498), and UPX (upx/upx#925), not all of which have been fixed. As a result, the binaries use slightly different toolchains and configurations depending on the architecture. Fixes #266, #268.

• Custom self-extracting stub for universal binaries. It aims for simplicity and portability and should work on most Linux systems. It is used if UPX support for an architecture is unavailable, or if the binaries extract much faster than the UPX-compressed version. The stub also supports --extract-wrapped-binary <file> to extract the embedded binary.

• Category metadata stored by default. The category metadata for categorized blocks is now stored in the metadata block by default. This allows recompressing blocks with a metadata-dependent algorithm (e.g., FLAC) even if they were previously compressed with a metadata-independent algorithm. You can disable this with --no-category-metadata. See the mkdwarfs man page for details.

• Options for smaller metadata. The --no-category-names and --no-category-metadata options can be used to reduce metadata size. However, this makes it impossible to use metadata-dependent compression algorithms (e.g., FLAC) or to select category-specific compression when recompressing the image.

• Metadata rebuilding in mkdwarfs. In addition to recompressing, it is now possible to change metadata packing and apply operations such as --set-owner, --set-group, --set-time, --time-resolution, --chmod, or --no-create-timestamp. Note that these operations are potentially lossy and may be irreversible. By default, the history of metadata rebuilds is tracked in the metadata itself; you can disable this with --no-metadata-version-history.

• Change block size on existing images. You can now change the block size of an existing image using --change-block-size. This implies --rebuild-metadata and --recompress=all and can be useful for tuning performance without recreating the image from scratch.

• Runtime memory display in mkdwarfs. mkdwarfs now shows its current memory usage while running. Note that -L/--memory-limit still only limits the memory used for the block queue, not overall memory usage. Fixing this is on the roadmap; there’s no need to file an issue.

• dwarfsextract format controls. New options --format-options and --format-filters control the output format. There is also --format=auto to automatically “guess” the format and filters based on the output file name. (Thanks to @oxalica for the pull request.)

• dwarfsck detail level. New frozen_details detail level shows frozen_analysis content ordered by memory location instead of memory usage, and also shows the address range of each section.

• Lean LRU cache. Replaced folly’s EvictingCacheMap with a simple in-repo LRU cache implementation. This reduces external dependencies and binary size without sacrificing performance.

• Windows extended attributes. The pxattr utility now supports all extended attribute operations on Windows, including setxattr() and removexattr(). Error handling and reporting for extended attributes on Windows has also been improved.

Docs

• Updated dwarfs-format.md with more information on section types, compression metadata, and the Frozen2 binary metadata layout. (Thanks to @oxalica for asking the right questions, reporting bugs, and ultimately releasing a Rust library to read/write DwarFS images.)

• Added a notable users section to the README. (Thanks to Vitaly Zdanevich for the PRs.)

• Updated mkdwarfs docs with more information on worker threads and the requirements for bit-identical images.

• Major README overhaul: added a Quick Start section, added more links, and fixed typos and wording.

Full Changelog: v0.12.4...v0.13.0

eb4a21fe560721a17059eb26b14abb894420008cc0dc990b829bdfda08e97af8  dwarfs-0.13.0-Linux-aarch64.tar.xz
3781cf4e5dde77f4e7da9900dca4250d300b57bbf5ba6640ba7f61e2efb5782f  dwarfs-0.13.0-Linux-arm.tar.xz
d7157d7a2faedea61829835060edfb1614d75b8559252f237aae38a97a684e9c  dwarfs-0.13.0-Linux-i386.tar.xz
786351112f1659d041e40ef70a4376c55f68bb783fc710ff71a1cb612b6786f5  dwarfs-0.13.0-Linux-loongarch64.tar.xz
465c0a2c14f13612a0413ba4e79f853cefdcc25a82d8ea7e74e3a349ac887983  dwarfs-0.13.0-Linux-ppc64le.tar.xz
cb44187074793b4aca9e252fb0bee269725180448e7a47abbf17e38b0b32e6ac  dwarfs-0.13.0-Linux-ppc64.tar.xz
0d8eac724b5c00f22592c68c2f61a1ae7be92aef0b236ace208e73e759e6965f  dwarfs-0.13.0-Linux-riscv64.tar.xz
52dbd8e44deede8e237d4b8c2fe1388b1478c14b1f76b63b4045a7103110acfb  dwarfs-0.13.0-Linux-s390x.tar.xz
81d2c5178b51367207df819a3a754fc3141d0f74d4ac80dbf20b1f0bd9d1be44  dwarfs-0.13.0-Linux-x86_64.tar.xz
d0654fcc1219bfd11c96f737011d141c3ae5929620cd22928e49f25c37a15dc9  dwarfs-0.13.0.tar.xz
fbbdf50657caf6be3c6864768bd2f0c2f6ea955b66e07875408e0a78bed2f9b9  dwarfs-0.13.0-Windows-AMD64.7z
78c52f9ca120e11d4a4620c614d532425ab694c0f8b6c25f04b475f10b3e0b2e  dwarfs-fuse-extract-0.13.0-Linux-aarch64
451744a2be3312fcd2968aa7821d61d97ece24177cebdb4cb8cf463409f9de7d  dwarfs-fuse-extract-0.13.0-Linux-aarch64.upx
0079697de87e14ea5bbf3a0dd0e95488d8343bd423f2ca3643462520a6c95b89  dwarfs-fuse-extract-0.13.0-Linux-arm.upx
7d3fc8474c1372a92b4301c28124fac16f858d81686ee4cafe21e8b6a1cebc14  dwarfs-fuse-extract-0.13.0-Linux-i386.upx
da9558153264fd14ba49f02d9fb26a8ef5c090a3e99a643fbeb9e88b68c3fd93  dwarfs-fuse-extract-0.13.0-Linux-loongarch64
b42ad7d4a229d22ecf219fd1759f1f1ebb0ccd28b4be16e49cf307f05b154f63  dwarfs-fuse-extract-0.13.0-Linux-ppc64
a81001d3a195eab8cdfd2b29768718ff5eae64a69174d33c5114aa04b87bdb6a  dwarfs-fuse-extract-0.13.0-Linux-ppc64le
9c9f4dcee5c78466cda8d3f3b5dccaa57be0270774e373198ab5cdc6f46a8c5d  dwarfs-fuse-extract-0.13.0-Linux-riscv64
aeae39035e52632a013f318909b38e0525dc04ba4e52179ab678807b6a3755ea  dwarfs-fuse-extract-0.13.0-Linux-s390x
04c9587395dbbdfdb6562f945daf2aa83e5ff80691e1dfeb7854dfc2e0e6993e  dwarfs-fuse-extract-0.13.0-Linux-x86_64
4a31bcbf9145fa422d3cfaf3f1f58b40cd095ba9a4bee508123f97e208684f52  dwarfs-fuse-extract-0.13.0-Linux-x86_64.upx
74b730031dfe4d4aa0caaa411ea806d02ba44f32dc2618e52739ce7...

This release mainly fixes a bug that was introduced with v0.12.0 (06f8728 to be precise). When re-compressing a file system image where some blocks cannot be compressed using the selected algorithm because of a bad_compression_ratio_error (i.e. when the compressed size is actually larger than the uncompressed size), the resulting block object was left empty, which subsequently led to a segfault.

A few small bugs have also been fixed and couple of features were added to the dwarfsck tool, mostly triggered by the discussion in #263.

Most other changes that made it into the release were related to how the static binaries are built. All the dependencies have been updated, in particular also for the Windows build. The Windows build was also switched from openssl to libressl. Overall, the size of the Windows universal binary was reduced by 30% and is now comparable in size to the Linux binaries.

Bugfixes

• Segfault on bad_compression_ratio_error. When recompressing a filesystem where some blocks cannot be compressed using the selected algorithm because of a bad_compression_ratio_error, the resulting block was left empty after the refactoring done in 06f8728.
• Add history unless --no-history is given when rewriting a file system image.
• Allow dumping frozen_layout w/o frozen_analysis in dwarfsck.
• Logging timestamps should show local time.
• Workaround a weird MSVC bug.
• Remove useless cast causing compiler warning.

Features

• More complete breakdown of metadata in dwarfsck.
• Add schema_raw_dump flag to dwarfsck --detail.

Build

• Switch static build to libressl on Windows.
• Update static build libraries.
• Update folly/fbthrift/fsst.

Other

• Introduce and use safe_localtime() to prevent issues with fmt deprecating fmt::localtime.
• Speed up a few slow tests on Windows.

Full Changelog: v0.12.3...v0.12.4

1492c0796ab3479a80e4e191e651f20c005e634b73ec684edd48465830f0aab3  dwarfs-0.12.4-Linux-aarch64.tar.xz
bff9da50918cf9272dbb9322733a629ccd84efc723e355a86eafcd075081f968  dwarfs-0.12.4-Linux-x86_64.tar.xz
352d13a3c7d9416e0a7d0d959306a25908b58d1ff47fb97e30a7c8490fcff259  dwarfs-0.12.4.tar.xz
5574d6aeb970c4cbabd50526994159c0551b2dc3e940edc774ac0952e1528c93  dwarfs-0.12.4-Windows-AMD64.7z
72f688800faf74acfdc75b6d77a456930db7088ee7c7e115c4b95414ab751c93  dwarfs-fuse-extract-0.12.4-Linux-aarch64
ea182efeb3ac55f8f79ee80009854c2ac8410b37d427da90e42b96eec73c470f  dwarfs-fuse-extract-0.12.4-Linux-x86_64
06c35c1dc99bc9c19f73ceda238866f9b4a631c08ab4ad19bbf8ea5c0a3ff9f0  dwarfs-fuse-extract-mimalloc-0.12.4-Linux-aarch64
afa48dfda2692e5b3cba6c523ab66696140d48dc8c4dfd57b114c73632e45326  dwarfs-fuse-extract-mimalloc-0.12.4-Linux-x86_64
7efdbd93954f1f88898b690466dca4b2ae3d20e799247c4ee4459acb35824389  dwarfs-universal-0.12.4-Linux-aarch64
48fc469bca251b932904b33d6161fc9a964cdefdb9d4d128828b4766b9c63eaa  dwarfs-universal-0.12.4-Linux-x86_64
094eaaf821df47daaa11899927f269f57f71843b39ecd53bae70692447cb3e65  dwarfs-universal-0.12.4-Windows-AMD64.exe

This release provides a fix for cases where automatic image offset detection could fail to work correctly as well as further size optimizations of the release binaries. The dwarfs-universal binary now uses LibreSSL's libcrypto, whereas the binaries from the release tarball use OpenSSL's libcrypto. This is a trade-off favoring size for the universal binary and speed for the "regular" binaries. Note, however, that this will be imperceptible unless you use dwarfsck with either --check-integrity or --checksum.

Bugfixes

• Automatic image offset detection (for images using a custom header) did not work correctly if the header contained a string that would be identified as the start of a v1 section header (these were only used before dwarfs-0.3.0). If there was either "DWARFS\x02\x00" or "DWARFS\x02\x01" in the header, offset detection would fail. The check has been modified to peek further into the data and ensure this really is a v1 section header, and also checking if the next section header position can be derived from the length field. It is still possible to construct a file system image where offset detection will ultimately fail, but it is much less likely with the change.

Build

• The build process for the release binaries has been further tweaked to reduce binary size. The dwarfs-fuse-extract binary now again supports extracting files by pattern; I didn't realize that this was actually a widely used feature before dropping it in the last release. dwarfs-universal is now linked against LibreSSL's libcrypto instead of OpenSSL's. This significantly reduces the size at the expense slightly slower cryptographic hash functions. However, this will likely only be perceivable when using --tool=dwarfsck with either --check-integrity or --checksum. The binaries from the release tarballs are still linked against libcrypto from OpenSSL.

Full Changelog: v0.12.2...v0.12.3

7e2c1d6f4bf8f19cedc8f050f405906e269d96197e238226f334c7ae2fb1f489  dwarfs-0.12.3-Linux-aarch64.tar.xz
9a2590deb3069d7e677604304d94226fa10b7385ccee46ba8c66f4e6c902168c  dwarfs-0.12.3-Linux-x86_64.tar.xz
bd2d54178c59e229f2280eea747479a569e6f6d38340e90360220d00988f5589  dwarfs-0.12.3.tar.xz
acbfbf5a48a8fa53dd6a39d7450c8d6849ef99d305563461596ac5c52767387a  dwarfs-0.12.3-Windows-AMD64.7z
16a450e6996ab59b89f69b0e30dab508d1e88c526586df5db5b7a43ede252398  dwarfs-fuse-extract-0.12.3-Linux-aarch64
e658a0513bc9168ff6f366b2dc19f6360408663fc2f0015653912075526c05a2  dwarfs-fuse-extract-0.12.3-Linux-x86_64
755599e8b52e36a87d6b90e860f38c7cf3c71bcebd8f63e11834cca0fbed9708  dwarfs-fuse-extract-mimalloc-0.12.3-Linux-aarch64
372c3090fd966e881978aaa7ccfcbb1476cf16a3cc3ddbaaddeac14cf141cb5d  dwarfs-fuse-extract-mimalloc-0.12.3-Linux-x86_64
9c66639a66a122d964ee297b93a638c61f25c917e9aee49339fde17320757f2a  dwarfs-universal-0.12.3-Linux-aarch64
c5e83388f41cddb59b3c490fc9fad8fbd73b134f8da5b8bba77297ab593d0efb  dwarfs-universal-0.12.3-Linux-x86_64
9dbc03ce6dfc9df25e8c9e77e5ee15f624ef244aaa9b4e235fb121761c698789  dwarfs-universal-0.12.3-Windows-AMD64.exe

This release provides a fix for a performance regression, switches the default memory allocator back to jemalloc, and further reduces the size of the dwarfs-fuse-extract binary. The latter is available as both a jemalloc and a mimalloc version. jemalloc offers a lot more configuration options that can be crucial in optimizing the memory profile of e.g. the FUSE driver. If you don't need that flexibility, you can save a few bits by using the -mimalloc version.

Bugfixes

• The dwarfs-0.12.0 release introduced a performance regression where FLAC compression took more than twice as long as in the previous releases. This has been fixed. FLAC decompression was unaffected.

Build

• A few small refactoring changes to further reduce the size of the fuse-extract binary. In particular, the performance monitor and the history feature are now fully removed. Also, the functionality to extract in different archive formats as well as to extract only files matching a pattern have been removed, so the image can only be fully extracted to disk.

Full Changelog: v0.12.1...v0.12.2

7d58b4125171befb5457a6318cda99607e32c2226db74de5f7449dee0e10764f  dwarfs-0.12.2-Linux-aarch64.tar.xz
61d239c0583d88443ca3e0080f1fe8bc97979a3ad67ed15ca3516e27ea7e7f53  dwarfs-0.12.2-Linux-x86_64.tar.xz
9b256d1f2bc17917cd63a1bee3bd5f505076b4d880fcf9daa18a6ca5bca35aeb  dwarfs-0.12.2.tar.xz
1ffbe8bbf44c5168aba5d0132705ad46837e4925ff57798333efcc6312bd7441  dwarfs-0.12.2-Windows-AMD64.7z
2aea873299cecc68dc0d8028d55cb00dc5c3289d12896e70e2038ea09d780c4e  dwarfs-fuse-extract-0.12.2-Linux-aarch64
88559806c8f2a98108e9ecf24926a317ea0afe655ef45f76c476307c4e71d971  dwarfs-fuse-extract-0.12.2-Linux-x86_64
99202109637d4d49d3b3945b35487deada758358b4f185869975c7c4be9870fa  dwarfs-fuse-extract-mimalloc-0.12.2-Linux-aarch64
8af6f46b5c39fa7fa9294b652bd023302599f2723f33571b3e2bf2376f420770  dwarfs-fuse-extract-mimalloc-0.12.2-Linux-x86_64
79a8e5d729650d8f26e1759228a10c2ea49ae88c1a491741f9196ce2937b4e2e  dwarfs-universal-0.12.2-Linux-aarch64
29d3195831c8ff3aca46b2a731eee7899d3735a71b870e9510adeaeb34dd135c  dwarfs-universal-0.12.2-Linux-x86_64
1232104b7c44dda3da46fdc6f7667e542e2e916ac926e2395965bcc035ae8046  dwarfs-universal-0.12.2-Windows-AMD64.exe

A quick update to v0.12.0 that addresses a few issues and improves the performance of the release binaries while mostly making them even smaller. The universal x86_64 binary is slightly bigger, but that's a different story [1].

Bugfixes

• Attempt to fix a linking issue in the Homebrew build.

Features

• Added --memory-limit=auto to mkdwarfs to use a more reasonably (hopefully) default for the block queue. The old default of 1 GiB was quite arbitrary and definitely not suitable for low-end systems. The new auto default will determine the limit based on the number of workers (which in turn is based on the number of CPUs), the block size, and the amount of physical memory of the system.

• Replaced vector_byte_buffer with malloc_byte_buffer, which is internally based around a simple buffer that doesn't incur the cost of initializing each element like std::vector. Especially for large blocks which are known to be overwritten immediately, this can save a few CPU cycles.

• The x86_64 release binaries now use an optimized memcpy implementation (if supported by the CPU) instead of the rather slow musl memcpy implementation. This makes mkdwarfs a few percent faster and dwarfsextract up to 20% faster.

Build

• Switched the release binaries to use mimalloc instead of jemalloc. The primary reason for this change is a reduction in binary size.

• Updated the xz library to the latest 5.8.1 release.

Full Changelog: v0.12.0...v0.12.1

f61d49436ad6b02e7e496f746ce09d69a3f050592286b5ff3d6e3edba765b82a  dwarfs-0.12.1-Linux-aarch64.tar.xz
6685622e6bf1edea138023dfb5a84ec241c6a57619ec5d7ee86344057d89296b  dwarfs-0.12.1-Linux-x86_64.tar.xz
5523a5c3aea244cbfbccfe64f1df6053b3901e6af8916fac1530faf0f7a5f07f  dwarfs-0.12.1.tar.xz
d51c2e5ed021a7322928aeb8f09cc3c392362c8a1ea6217e2ba177f241f8a809  dwarfs-0.12.1-Windows-AMD64.7z
8850a0002d7008791c2629fd3a5bc718c50606dfc391bf5b1259d9f6d79c8401  dwarfs-fuse-extract-0.12.1-Linux-aarch64
d01dd82068e0d2020fd35ba5a2ddf416aeba32a9f6f7ac4544a173eefc6c743f  dwarfs-fuse-extract-0.12.1-Linux-x86_64
3dfc3d8d2152f4d9d29c543d0a30b2ba1bc0bba470742e08924ad75dc8c23967  dwarfs-universal-0.12.1-Linux-aarch64
518faa4f5a476dcc4ec75d8ed4ac31d076990bb515cd32a752a8165b3ad04885  dwarfs-universal-0.12.1-Linux-x86_64
1ad50c0b6127e56dc026de7d2f7a3df28fa3983bdd628584a652d05f97e28b88  dwarfs-universal-0.12.1-Windows-AMD64.exe

[1] I've noticed that the universal binary, and sometimes the other binaries as well, fluctuate dramatically in size after being UPX compressed. In the last CI build before tagging the new DwarFS release, the UPX-compressed universal binary had a size of 2,713,752 bytes. The only change after tagging the release was the version info (although, with LTO, you never know what the compiler makes of that). Interestingly, when unpacked, both binaries have exactly the same size. But the packed size differs by ~190 KiB.

The main features of this release are new licensing conditions and significantly smaller binaries.

New Licensing Conditions

Instead of being all GPL-3.0 like all the previous releases, this release changes the license of a large fraction of the DwarFS code to MIT. All tools and libraries that only read DwarFS images are now MIT-licensed. Everything else (e.g. mkdwarfs) is still GPL-3.0 for the time being.

Significantly reduced binary size

By moving the build pipeline to Alpine Linux and through some major refactoring, the size of the release binaries has been reduced significantly.

There's now also an additional binary called dwarfs-fuse-extract that combines the functionality of the FUSE driver dwarfs and dwarfsextract in a single, extremely small binary:

The main use case for this binary is single-file application image runtimes (e.g. uruntime). This binary doesn't have built-in manual pages or support for the performance monitor. It also only supports zstd and lzma compression.

Bugfixes

• Build release binaries against an up-to-date libfuse. Fixes github #252.

• Changes for compatibility with Boost.Process v2.

Features

• Re-licensed all libraries required for reading DwarFS images under the MIT license. The source of all tools that just read DwarFS images (i.e. everything except for mkdwarfs) are also under the MIT license now. Everything else is still GPL-3.0. Addresses github #255.

• Significantly reduced binary size in the static release builds. This is the result of refactoring code that unconditionally pulled in code-heavy dependencies such as libcrypto, as well as optimizing the build pipeline (e.g. building dependencies with only the necessary set of features) and turning on link time optimization.

• A new kind of "universal" binary dwarfs-fuse-extract is part of the release now. This combines the FUSE driver (dwarfs) and dwarfsextract into a single binary, but does not include the mkdwarfs and dwarfsck tools that are also part of the regular universal binary. dwarfs-fuse-extract is much smaller than the regular universal binary and especially suitable to AppImage-like applications.

• New hotness categorizer in mkdwarfs that allows a list of "hot" files to be stored in distinct file system blocks.

• New explicit ordering mode in mkdwarfs that allows files to be ordered accoring to the order in a given list file.

• dwarfs now shows the version of the FUSE library used.

• New dwarfs options preload_all and preload_category to populate the block cache immediately after mounting.

• New dwarfs option analysis_file that can be used for profiling and as input to mkdwarfs new hotness categorizer and explicit ordering mode.

• New dwarfs option block_allocator that allows the user to switch from a malloc-based block allocator to an mmap-based one. This can help with returning memory back to the system if the blocks are evicted from the cache.

Full Changelog: v0.11.3...v0.12.0

15867a3e1b5a310ea27700806ce7d504c912de8741dddbb40430eee419c4532f  dwarfs-0.12.0-Linux-aarch64.tar.xz
4fd1e23a97d871d0536b818c11a58c58859627e347d10c1e211522c8dd56b328  dwarfs-0.12.0-Linux-x86_64.tar.xz
91d5a22e5cf125a9871bcbdb4875bdd661557757b9f50e88553da4b47f8351d2  dwarfs-0.12.0.tar.xz
e9a4d513085bb91ab26f86277cfdfbf7930adf4bcb3c44be9f67f205d1db29fb  dwarfs-0.12.0-Windows-AMD64.7z
f7e7909b779a96dccced8e6749729ccf73ac5782ed5721cc27199498b84a27c1  dwarfs-fuse-extract-0.12.0-Linux-aarch64
264e6cbac5ade98275241082770e7aefb6e931612c543deb87322be03fc65bb8  dwarfs-fuse-extract-0.12.0-Linux-x86_64
41981a4f7a068f6c3bc32dad5307aa4466d0d74bf13083a495151d0ece34b17c  dwarfs-universal-0.12.0-Linux-aarch64
1425e102fd3b8251629025fc3b8aab0f74b7079466261f6fcef6c0e0531249ac  dwarfs-universal-0.12.0-Linux-x86_64
fe5904d64f1bd08f8f2f7dd271587b415bef1237562da4a77ee30982b48e24c4  dwarfs-universal-0.12.0-Windows-AMD64.exe

Bugfixes

• Handle absolute paths in --input-list. Fixes github #259.
• Don't prefetch blocks that are already in the active list within the block cache.
• Ensure that statistics for block tidying are correctly updated in the block cache.
• A few build fixes, mainly to simplify building on Alpine.

New Contributors

• @hexahigh made their first contribution in #257

Full Changelog: v0.11.2...v0.11.3

de7f6609a4ddd6f2feff4cb4e43c4481515c5da178bbe12db24de9e7fec48bac  dwarfs-0.11.3-Linux-aarch64-clang-reldbg-stacktrace.tar.xz
7c0835b89871e48025b2e30577fb1b3c39927f9b86940dd3c5d7c41871e12533  dwarfs-0.11.3-Linux-aarch64-clang.tar.xz
2e771b53ebee66278b3a2e6e18fd04e20abc0f6defccb5a347dbfc2b7436b729  dwarfs-0.11.3-Linux-x86_64-clang-reldbg-stacktrace.tar.xz
adc3fc58d36848a312e846f0e737056b7e406894e24fa20d80fcc476ca7f401f  dwarfs-0.11.3-Linux-x86_64-clang.tar.xz
5ccfc293d74e0509a848d10416b9682cf7318c8fa9291ba9e92e967b9a6bb994  dwarfs-0.11.3.tar.xz
33b3488bc1097b1b2b54194eaa5fb169dfded9a6046de6c4fee693d9a97ece32  dwarfs-0.11.3-Windows-AMD64.7z
e14c0caa38a8d10273a84e57f532e513b2cbc50bb8df707b57c01d575f040a43  dwarfs-universal-0.11.3-Linux-aarch64-clang
7d4857ee18ffae705a41f164a0a810f173bf8d69bc8bef8dcbd1018fa8287f6e  dwarfs-universal-0.11.3-Linux-aarch64-clang-reldbg-stacktrace
64b349aec059b9d460211af6c517f6edd89e79c5e9581381229af745ebf3cc87  dwarfs-universal-0.11.3-Linux-x86_64-clang
07a9ef68256e76e7bda552b24955a3db12c3b34312d7d664e47639995ccdabf1  dwarfs-universal-0.11.3-Linux-x86_64-clang-reldbg-stacktrace
772f00d5d02fdebca4cbd74f4b1b37ee7b57fb3004a078c053b4f58e97a794ed  dwarfs-universal-0.11.3-Windows-AMD64.exe

Bugfixes

• macOS Ventura's version of clang appears to be missing an implementation of std::hash<std::filesystem::path, making it hard to define an unordered_map<filesystem::path>. Work around by simply using an unordered_map<string> instead.
• Installing the binaries using cmake did not honor the CMAKE_INSTALL_BINDIR or CMAKE_INSTALL_SBINDIR variables. Fixes github #253.

Full Changelog: v0.11.1...v0.11.2

61fce8eaa6bbdf10917a5a12331e192748a54ab1aa175ed6f55cb26825ab3177  dwarfs-0.11.2-Linux-aarch64-clang-reldbg-stacktrace.tar.xz
06fc4ed91ee5c348dbfc70771fe3e3ea6834277e4a58f1f99e0bc98cb16ed3d4  dwarfs-0.11.2-Linux-aarch64-clang.tar.xz
15905007cff432bb9be0bdabed93473764c1706796e0da6f3af083f0a142db6d  dwarfs-0.11.2-Linux-x86_64-clang-reldbg-stacktrace.tar.xz
3c82708e00af9d1622e78047efd216e4e29213a60aff3afa8326bade8353ea38  dwarfs-0.11.2-Linux-x86_64-clang.tar.xz
1b38faf399a6d01cd0e5f919b176e1cab76e4a8507088d060a91b92c174d912b  dwarfs-0.11.2.tar.xz
8a028693ce0a7ab083b25dc491b100f41fbf98f28413a38f6773fe1cf27574fb  dwarfs-0.11.2-Windows-AMD64.7z
600134267dd0ad51dd9d8bd1b58fa614b0a0da9a7a3d57f5fce4dbda9bb80460  dwarfs-universal-0.11.2-Linux-aarch64-clang
a9f5f79afeff4eba5cc23893de46e4c8eaa3b51b8f5938ed7f9e6cb92560fa4f  dwarfs-universal-0.11.2-Linux-aarch64-clang-reldbg-stacktrace
1bee828de84c1a3a1c2134bc866f28bdf93a62927cb7e8c416813f389f7745ad  dwarfs-universal-0.11.2-Linux-x86_64-clang
ddbd62d3bf0bf420a1720af6c03bea21ce6a77a73cedc46f28c8d79e4ac26827  dwarfs-universal-0.11.2-Linux-x86_64-clang-reldbg-stacktrace
d95dab93a7e9d8349d4a4393213a401d9ded79040b1c48e661df7dfe118b72a7  dwarfs-universal-0.11.2-Windows-AMD64.exe

Bugfixes

• macOS Ventura's version of clang appears to be missing the <source_location> header, despite Apple claiming otherwise. Fix this by shipping a wrapper and providing a fallback implementation.

Full Changelog: v0.11.0...v0.11.1

3e1b6331cf2f589d7058700aa2c5dc41f1825f3954f3828eb709034ba57a7c97  dwarfs-0.11.1-Linux-aarch64-clang-reldbg-stacktrace.tar.xz
23b1e0b18a7c3ffeb6c5fcc97ab032a7c1c651454d0fa5cb9741918d97a14ab3  dwarfs-0.11.1-Linux-aarch64-clang.tar.xz
4ec6614e87064ac96dfdb9b7957620bd889f5f1e95416409369d00606cfff0a1  dwarfs-0.11.1-Linux-x86_64-clang-reldbg-stacktrace.tar.xz
1eebf6e66eb5d6dc7cfb9c9b3c7c6e67084acc5ced7a018c15a511e929598f99  dwarfs-0.11.1-Linux-x86_64-clang.tar.xz
7a0cccb1ec3c2a18e9a014893c1d3e1f8f2c44ade6936c9f6d3bab5ec14b2052  dwarfs-0.11.1.tar.xz
c43fd9f2089b94ddd2819c7853dd6d8c34951e2d42cbfdc2e4470cde9c3e18fb  dwarfs-0.11.1-Windows-AMD64.7z
e9bf1f8bcccf363be25396d2f60d9f4e7765eba5bd647f071aa4d0ba5cb3785b  dwarfs-universal-0.11.1-Linux-aarch64-clang
22966f1dba98697db0cad127d1d8c50ef5952b5c9816cc2564b9410a37cdbaa3  dwarfs-universal-0.11.1-Linux-aarch64-clang-reldbg-stacktrace
0a025dc0f854ad9f3a5f9ca89ac43ada5305de14fe4b2e03088ce9ee5a23dbf4  dwarfs-universal-0.11.1-Linux-x86_64-clang
a38de846f48c7979c204699af6a1fda0d8d634caac68223b2b46fcb1c52c0e56  dwarfs-universal-0.11.1-Linux-x86_64-clang-reldbg-stacktrace
b8299e5f2102283c52c1cbd8ad14a0c3b71244937327e895e0aee809d92e4474  dwarfs-universal-0.11.1-Windows-AMD64.exe