Bug fixes

• The new Windows code from the v0.14.0 release that implemented lstat()-like functionality to get file status information did not work correctly for reparse points that were not symbolic links (i.e. "real" symlinks or junctions). When dealing with such reparse points, the code would incorrectly determine the allocated size of the file to be zero, since it tried to determine that size from the reparse point and not from the target. This caused the progress percentage to be calculated incorrectly, since much more data was processed than was expected, leading to progress percentages well over 100%. Thankfully, @lunighty not only reported this issue, but also provided significant context by mentioning that this was on a Windows Server NTFS volume with deduplication enabled. Fortunately, this was also reproducible on "regular" Windows using a mounted WIM image. The fix will now correctly handle reparse points by re-opening the handle without FILE_FLAG_OPEN_REPARSE_POINT in order to open the target. This bug only affected Windows builds of mkdwarfs, and only when actually used on filesystems with reparse points that were not symlinks. Even then, the resulting DwarFS image would still be correct, except for a way too small total_allocated_fs_size metadata entry. However, this entry is purely informational. Huge thanks to @lunighty for both reporting and helping to debug this issue! Fixes github #307.

• Related to the above, the metadata_builder now recomputes all total sizes (total_fs_size, total_allocated_fs_size, and total_hardlink_size) as part of the build() function. This not only ensures that the totals are correct even if the allocated size changes between scanning and segmenting (which has been happening at least on ZFS volumes), but it also allows images affected by the above bug to be fixed by rebuilding the metadata.

• Instead of making the FUSE drivers fail hard when seeing the options that were removed in v0.14.0, they now just log a warning and ignore them. The options may still be fully removed in a future release. Fixes github #303.

• The pcmaudio categorizer had two minor issues discovered by @lunighty when compressing a large number of WAV files. One was reporting an unsupported format: 3/0 or unsupported format: 65,534/3 warning, which isn't very useful for the end user. These format codes correspond to IEEE floating point formats, which are indeed unsupported. However, the format appears to be quite common, so the warning has been downgraded to an info message that explicitly mentions the floating point format. The second issue was an unexpected fmt chunk size of 20 bytes, which caused the file to be rejected as a PCM audio file (meaning it was added using a generic compressor instead of FLAC). It turns out that these non-conforming fmt chunks are also quite common in practice, so the code has been changed to accept the non-conforming file, but also logging an info message mentioning the non-conformance. Fixes github #309.

• A bunch of changes were made to make the static (Linux) release binaries work on FreeBSD with the Linux emulation layer. While you can easily build DwarFS on FreeBSD natively, the static binaries might be bundled in Linux application images that you may want to run on FreeBSD as well. This required patching musl to fall back to FUTEX_WAKE if FUTEX_REQUEUE is not supported (which is the case on FreeBSD's Linux emulation); otherwise, the binaries would regularly hang when trying to shut down thread pools. Furthermore, the SFX stub did not work with the emulation layer due to lack of support for fexecve; so it now uses a fallback of temporary file + execve in case create_memfd and/or fexecve are not supported. This fallback is very similar to the one that was already in use for old Linux kernels, only that cleanup of the temporary file is now done using a child "janitor" process as soon as the execve call succeeds. Lastly, all static binaries are now branded as using the Linux ABI rather than the generic "System V" in the ELF header, which makes it unnecessary to explicitly brandelf the binaries on FreeBSD. See VHSgunzo/uruntime#21 for more details.

• The test_helpers library used for unit tests was not explicitly listing xxhash as a dependency, which caused linker errors on macOS. These unfortunately only showed up in the Homebrew CI. The dependency has been added.

• The help text for the mkdwarfs compress level option (-l) was misleading in combination with the manual page as neither mentioned that the table with details was shown only by -H / --long-help. Fixes github #312.

Features

• Added shell completion for dwarfsck and dwarfsextract. (Thanks to Ahmad Khalifa for the contribution.)

• Added sample desktop unmount handlers. (Thanks to Ahmad Khalifa for the contribution.)

Documentation

• Updated diagram in dwarfs-format.md to show new inode metadata fields.

Build

• Bumped libarchive, libressl, and libdwarf versions in the static build pipeline.

Test

• Added test for file_stat on symbolic links containing multibyte characters on Windows.

• Added test for obsolete FUSE driver options being ignored and producing a warning.

• Added test for the FUSE driver's imagesize option.

• Added more tests for the --auto-mountpoint option in the FUSE driver.

Full Changelog: v0.14.0...v0.14.1

8c17434c4fc8c33ba303e06eb62904047ee1aab3ea7bd98d2797b04d86b426eb  dwarfs-0.14.1-Linux-aarch64.tar.xz
98cdca2e9406cbb88093bd2ea03cbd8a62b7a3aa5ba0271d22934fe4e81128ac  dwarfs-0.14.1-Linux-arm.tar.xz
531b2c318a810920d4f11f0efac12cec06627068d91291232000a883108952ab  dwarfs-0.14.1-Linux-i386.tar.xz
93ec3d9d649cdb77eed3d1b5c6533ee05e24589be63cca3b0a9c32c18c55ef1a  dwarfs-0.14.1-Linux-loongarch64.tar.xz
813d021f3025968dc7fa0797553b55bd7a5ed1c85af1f6b2c36b2ad5897294c8  dwarfs-0.14.1-Linux-ppc64le.tar.xz
1437bd811bfcf0f9d7db080075906086625ab1465ed9a068bb881da1c8f68681  dwarfs-0.14.1-Linux-ppc64.tar.xz
0a6e1e66b0b0ce63acdc37a8a68ce173d37b135cfad9ca652ca2827e4789d3ec  dwarfs-0.14.1-Linux-riscv64.tar.xz
9f20a528d9757ae7825341438c7d38eb3c91953c34c8d6bb6ace5cc4fe3d2f52  dwarfs-0.14.1-Linux-s390x.tar.xz
7dba2bc4482defc5987652643ab0d58330b74e73be0f4d2d184879e7815c2476  dwarfs-0.14.1-Linux-x86_64.tar.xz
620cf27f2e142a5f8fc05552a70704c3bf4df23c3279c6026b3f37954d0529c5  dwarfs-0.14.1.tar.xz
66f4d28514eb9cd8a28f634d76f274eb899c461cd54775f74efb6a97239c04ec  dwarfs-0.14.1-Windows-AMD64.7z
24bdd5c6adea556ca3a28f26a3774684da7efe4a5e70a13ed7149edd12b3f738  dwarfs-fuse-extract-0.14.1-Linux-aarch64
9f04547e224aa97c3187474de4d49ee12cfc1a414b9a3864f4338272201278fd  dwarfs-fuse-extract-0.14.1-Linux-aarch64.upx
fa31490ac078cd90be86cb43cf12836ad142f115ff5bf4a01771b6e07b73acab  dwarfs-fuse-extract-0.14.1-Linux-arm
6e39a297c75ae2fbb69b2d6075b32e34b3c4a386d6961e4a85b76ff944ccf30b  dwarfs-fuse-extract-0.14.1-Linux-arm.upx
cd637922614ba20b4614c37e61d0bb5ee703ccfce13ba328f9c07ed2848d9424  dwarfs-fuse-extract-0.14.1-Linux-i386
b6274b71bd695d62de21c9631e6d9f62be8a55512e072f5c6124eaa047e1fdf3  dwarfs-fuse-extract-0.14.1-Linux-i386.upx
6b1b699aba544ef407ebc33003f900fa691685c2be4195ca9690da7d7da4bcef  dwarfs-fuse-extract-0.14.1-Linux-loongarch64
948227dc2f87c37a3c584af61b189cbc04557088530335794f54a8a9a17e113f  dwarfs-fuse-extract-0.14.1-Linux-ppc64
8a60b79fc535afeb3352c24252a69db8b0dae0e176a46cc321c0e4e644eb0e08  dwarfs-fuse-extract-0.14.1-Linux-ppc64le
4cb8a50135ad0d3f93f3750e738fcd0f92751445493665d7316e2384a20ba107  dwarfs-fuse-extract-0.14.1-Linux-riscv64
aebd9c079fdafc11e5669f1a2e49192f19e5c3f78d62f8c138eb28208ed1eb8d  dwarfs-fuse-extract-0.14.1-Linux-s390x
b121f472429bfba72b79f1b5de126e8a70872fedefd4d42aba98fc5fc402ae20  dwarfs-fuse-extract-0.14.1-Linux-x86_64
0ded4f3b5c324cd662babece00bf0434fd98b9c98e6268ea745e160bc96a500b  dwarfs-fuse-extract-0.14.1-Linux-x86_64.upx
3a47eda6d496ba12fb6d9af1b1914d4ba7ab73df8df3172ee3cad66c29822cc1  dwarfs-universal-0.14.1-Linux-aarch64
ad9d6e86c120c2a4e635e4843234d9e046ac186772ad719ba508c6f1051a5f57  dwarfs-universal-0.14.1-Linux-aarch64.upx
958ab2e0ae8b1c65b14090b1f56cf062becc955c8287d31f1f7026adad01ac7a  dwarfs-universal-0.14.1-Linux-arm
33c06c3320e9c9907f0803fba3913e59f39c2b07c2a54722a0c8fbd2ea20f542  dwarfs-universal-0.14.1-Linux-arm.upx
dae5171425af53ed1357d6f63c5fc45d6a61b0c70588c7cd8b5c8eb432080673  dwarfs-universal-0.14.1-Linux-i386
7a073913b6c682816e7dbcddde1bcb8ddca9d9c32f679369df21bb7d54f0291c  dwarfs-universal-0.14.1-Linux-i386.upx
17630a7f41d5974a0747f28052bf0339140b22ce67a66fce443ebcc1bc8594bd  dwarfs-universal-0.14.1-Linux-loongarch64
ca378379612b66093e677d45616f242bd08f5e5e808283f4055530c51048b149  dwarfs-universal-0.14.1-Linux-ppc64
06472149ea534d94eb5799f6a2037edc5ff5f3146546af15b3a9d5b427be9db6  dwarfs-universal-0.14.1-Linux-ppc64le
bf827447162016098ef193c6104264a52aaef9089bab434872631a8bf6ed2ffe  dwarfs-universal-0.14.1-Linux-riscv64
a89e4152096b422bda48cb9ec1a957dd9a4745ec9cd83f2d036d00e5e2dbb2ba  dwarfs-universal-0.14.1-Linux-s390x
f3a117fd6d5b7304944b199af7fdb8086a48c509ea2e9832255d8f9a54c98587  dwarfs-universal-0.14.1-Linux-x86_64
fb4ea4a267acf19a5bf9bf0eb13f3d7d88b5f7687fca2aae7136e21810fde0cd  dwarfs-universal-0.14.1-Linux-x86_64.upx
c59967d05cb97937c642964a258f29e437f68e2f156ff71b7205dd489b0f5c00  dwarfs-universal-0.14.1-Windows-AMD64.exe

New I/O Layer, Sparse Files and Sub-Second Timestamps

New I/O Layer

A completely new I/O layer abstraction replaces the ubiquitous use of memory mapped files across the DwarFS code base. Memory-mapping is still the default, but processing is done in "segments" rather than in whole files. This required a significant amount of changes (this release adds/touches more than 5,000 lines of code and almost 10,000 lines of new tests) in almost every part of the code that was processing file data, code that previously assumed any file could simply be accessed as a contiguous piece of memory.

In the new abstraction layer, backends are pluggable and configurable through the DWARFS_IOLAYER_OPTS environment variable, letting you:

• Configure the size up to which files are mapped "eagerly", i.e. as a whole and not in segments. This is mostly relevant for 32-bit systems, on which this is set to a reasonable default (32 MiB).
• Switch from mmap() to classic read() for maximum robustness on unreliable storage or faulty hardware.

The latter is relevant if you're seeing "bus errors" (SIGBUS), as many have done in the past (#45, #50, #108, #163, #213). You can switch to the read-based backend using:

$ export DWARFS_IOLAYER_OPTS=open_mode=read

Sparse File Support

This release includes end-to-end sparse file support:

• mkdwarfs detects holes in files and preserves sparseness in the image.
• The FUSE driver exposes sparsity via lseek() where supported (Linux, FreeBSD).
• dwarfsextract writes sparse files and preserves them when targeting archive formats that support holes (e.g., tar).

Compatibility: images containing sparse files require DwarFS ≥ 0.14.0. You can use --no-sparse-files to explicitly treat sparse files as non-sparse and keep compatibility with older versions. If your input does not contain sparse files, the images remain backwards-compatible even without that flag.

Sparse file support matrix

FUSE-level sparseness requires lseek() support in the FUSE implementation (currently Linux and FreeBSD). On macOS and Windows, files are exposed via FUSE as non-sparse even though dwarfsextract can still write sparse files when extracting. Missing lseek() support is tracked here for Windows and here for macOS.

Sub-second timestamps

Configurable sub-second timestamp resolution down to nanoseconds (using --time-resolution). The default remains one second. This is fully backwards-compatible: older DwarFS versions can read images with sub-second resolution, but will ignore the sub-second part.

Bug fixes

• Leading dots in --input-list file paths were incorrectly treated as literal directory names instead of being expanded. This has been fixed. Fixes #292.

• The SPDX license identifier in GPL-licensed source files was incorrectly specified as GPL-3.0-only instead of GPL-3.0-or-later. This has been corrected. Fixes #275.

• Fixed an off-by-one error when recovering self_index fields in metadata, which could cause the sentinel directory to have a non-zero self_entry. While harmless by itself (since that entry is never actually used), this would cause the metadata consistency check to fail. The fix covers three aspects: correcting the off-by-one error; ensuring the self_entry recovery code does not run for the sentinel directory; and changing the metadata consistency check to only warn about a non-zero self_entry rather than fail. Running mkdwarfs with --rebuild-metadata will also reset a non-zero sentinel self_entry to zero.

• Fixed the implementation of the read operation in the FUSE driver to send positive error code values to libfuse. This was likely never triggered in practice, but in cases where parts of the filesystem image vanish while being accessed (which previously caused SIGBUS crashes), libfuse would not understand the negative error codes.

• Moved the FUSE driver binaries from sbin to bin and kept only the mount.dwarfs/mount.dwarfs2 symlinks in sbin. This better aligns with user expectations, other FUSE drivers, and the fact that the man pages are installed in section 1. (Thanks to Ahmad Khalifa for the fix.)

• The dwarfs2 binary was broken in builds using shared libraries. (Thanks to Ahmad Khalifa for the fix.)

• When setting CPU thread affinity for worker group threads via DWARFS_WORKER_GROUP_AFFINITY, the code did not CPU_ZERO the cpu_set_t structure before setting individual CPUs. This could pin threads to random CPUs in addition to the requested ones.

• The FITS categorizer would scan entire files for the end-of-header marker if their size was a multiple of 2880 bytes, causing significant slowdowns on large non-FITS files. Additional checks now ensure scanning only continues if the data truly looks like a standards-compliant FITS header.

• GCC caught a potential null-pointer dereference on error when opening a file in mkdwarfs. This has been fixed.

• Numerous fixes for 32-bit architectures, mostly related to integer overflows with file sizes larger than 4 GiB.

• Another off-by-one error caused the first regular file inode to be excluded from the file-size cache. This would be hard to notice unless that file was highly fragmented. The cache will be fixed when rebuilding the metadata.

• The FUSE driver’s enable_nlink option is now the default behavior and cannot be disabled. The previous optimization skipped building a table of hardlink counts, which produced inherently incorrect file status information (hardlinked files share an inode, so reporting a link count of 1 is wrong). The hardlink table is now stored in the metadata by default; if there are no hardlinks, it consumes no space. You can still omit the hardlink table with --no-hardlink-table, at the cost of building it on-the-fly when the filesystem image is loaded (typically fast — e.g., ~300 ms for 14 million files).

• Fixed a typo in dwarfs-format.md. (Thanks to Dennis Brakhane for spotting this and sending a PR.)

Features

• New I/O layer abstraction that supports “classic” mmap-based file access, granular mmap-based access on 32-bit systems, and fully mmap-less access if desired. This applies to all DwarFS tools. By default, tools use the most efficient method—memory-mapping whole files on 64-bit systems and mapping file segments on 32-bit systems (to conserve address space). This can be controlled via the new DWARFS_IOLAYER_OPTS environment variable described in dwarfs-env(7).

• Full support for sparse files. mkdwarfs now detects and efficiently processes sparse files, skipping holes where possible and preserving them in the filesystem image. This is supported on all platforms. The FUSE driver implements lseek() where supported by the FUSE library (currently Linux and FreeBSD); Windows and macOS fall back to showing files as non-sparse. dwarfsextract extracts sparse files as such and preserves sparse representations when extracting to archive formats that support them (e.g., tar). Note: Sparse file support is not backwards compatible; images containing sparse files cannot be processed by DwarFS versions prior to 0.14.0. By default, mkdwarfs enables sparse file support if it detects sparse input. Use --no-sparse-files to disable it and ensure compatibility with older versions.

• Support for subsecond timestamp resolution. The default remains one second, but finer resolutions (down to nanoseconds) can be specified with --time-resolution. mkdwarfs will warn if the requested resolution is finer than the native filesystem resolution. This is fully backwards compatible: older DwarFS versions will handle such images but ignore the subsecond parts. Fixes #294.

• Desktop integration for Linux. A new --auto-mountpoint option automatically creates or selects a mount-point directory, making it easier to mount DwarFS images from file managers. Desktop files and MIME type definitions are now installed to enable double-click mounting of .dwarfs files. (Thanks to Ahmad Khalifa for the implementation.)

• Shell completion for mkdwarfs (bash and zsh). (Thanks to Ahmad Khalifa for the contribution.)

• Improved error handling when DwarFS tools encounter SIGBUS (usually caused by accessing memory-mapped files on unreliable or faulty storage like network shares or flaky USB drives). When SIGBUS is caught, tools now print an error suggesting switching from mmap- to read-based I/O via DWARFS_IOLAYER_OPTS.

• dwarfsck now checks metadata consistency by default (unless --no-check is given), improving detection of filesystem image corruption.

• If sparse files are supported by the FUSE library, the FUSE driver exposes new options cache_sparse and no_cache_sparse to control whether sparse files should be cached in the kernel page cache. See dwarfs(1) for details.

• The JSON output from dwarfsck now contains a complete raw metadata dump when the detail level includes metadata_full_dump.

• dwarfsck no longer artificially limits string sizes when dumping metadata. (Thanks to Dennis Brakhane for the contribution.)

• Accelerated search for the start of a DwarFS image in files with custom headers; the new code is about four times faster, scanning at more than 6 GiB/s on a modern CPU.

• The cache size can now be configured for dwarfsck, useful with the --checksum option.

• Both dwarfsck and dwarfsextract now limit the amount of ...

FreeBSD, big-endian, and many new architectures

This release finally includes FreeBSD in the list of supported operating systems. That should make it much easier to port DwarFS to other *BSDs as well. Big-endian platforms are also supported, and the file system images (which use little-endian) are fully portable between architectures. Binary releases are now available for a wide range of architectures: aarch64, arm, i386, loongarch64, ppc64, ppc64le, riscv64, s390x, and x86_64.

Metadata manipulation

Before this release, file system metadata was immutable once a DwarFS image had been created, and the only way to manipulate metadata was to build a new image from scratch. This release adds two options to mkdwarfs:

• --rebuild-metadata allows changes/upgrades to the metadata block. This makes it easy to change how the metadata is packed, or even perform manipulations such as --chmod after the fact.
• --change-block-size allows you to change the physical block size of the file system image.

Bug fixes

• The linker configuration for the release binaries was broken. The symptom was that, very occasionally, tests would fail in CI with std::terminate being called after the exception handling code failed to unwind the stack. The root cause was that, in clang builds, code from libunwind and libstdc++ was arbitrarily mixed, which—depending on the order in which individual threads were scheduled in the unit tests—could lead to stack unwinding working flawlessly or not at all. This was one of the hardest bugs to track down this year; fortunately, the fix was quite simple. It’s possible this issue is present in previously released binaries, although there have been no reports.

• Made section index discovery more robust. Fixes #264.

• A recent kernel change (LKML thread, 2025-05-05) caused tools_test to fail on Linux 6.14 and later. This has been fixed by accepting both EPERM and ENOSYS as valid error codes for link() calls.

Features

• FreeBSD support. Everything that works on Linux should also work on FreeBSD. No static binaries are provided for FreeBSD, but the build should work out of the box once all dependencies are installed.

• Big-endian architectures. This is still experimental, even though all unit tests pass under QEMU and the benchmark suite runs on real hardware. This currently requires forked versions of folly and fsst. The changes are small, and the pull requests will hopefully be merged upstream soon. (facebook/folly#2484, cwida/fsst#36)

• Experimental 32-bit support. While DwarFS should largely “just work” on 32-bit with small images (a few hundred megabytes), limited address space is a problem due to DwarFS’s extensive use of memory-mapped files. There will be changes to limit the use of mmap in the future (primarily due to other issues), which should improve 32-bit compatibility as a side effect. Fixes #268.

• Wider CPU architecture coverage. Static binary releases (including universal binaries) are now available for x86_64, aarch64, i386, arm, ppc64, ppc64le, riscv64, s390x, and loongarch64. Building the new release binaries uncovered a few bugs in clang (llvm/llvm-project#150913), binutils (https://sourceware.org/bugzilla/show_bug.cgi?id=33223), mold (rui314/mold#1490, rui314/mold#1496, rui314/mold#1497, rui314/mold#1498), and UPX (upx/upx#925), not all of which have been fixed. As a result, the binaries use slightly different toolchains and configurations depending on the architecture. Fixes #266, #268.

• Custom self-extracting stub for universal binaries. It aims for simplicity and portability and should work on most Linux systems. It is used if UPX support for an architecture is unavailable, or if the binaries extract much faster than the UPX-compressed version. The stub also supports --extract-wrapped-binary <file> to extract the embedded binary.

• Category metadata stored by default. The category metadata for categorized blocks is now stored in the metadata block by default. This allows recompressing blocks with a metadata-dependent algorithm (e.g., FLAC) even if they were previously compressed with a metadata-independent algorithm. You can disable this with --no-category-metadata. See the mkdwarfs man page for details.

• Options for smaller metadata. The --no-category-names and --no-category-metadata options can be used to reduce metadata size. However, this makes it impossible to use metadata-dependent compression algorithms (e.g., FLAC) or to select category-specific compression when recompressing the image.

• Metadata rebuilding in mkdwarfs. In addition to recompressing, it is now possible to change metadata packing and apply operations such as --set-owner, --set-group, --set-time, --time-resolution, --chmod, or --no-create-timestamp. Note that these operations are potentially lossy and may be irreversible. By default, the history of metadata rebuilds is tracked in the metadata itself; you can disable this with --no-metadata-version-history.

• Change block size on existing images. You can now change the block size of an existing image using --change-block-size. This implies --rebuild-metadata and --recompress=all and can be useful for tuning performance without recreating the image from scratch.

• Runtime memory display in mkdwarfs. mkdwarfs now shows its current memory usage while running. Note that -L/--memory-limit still only limits the memory used for the block queue, not overall memory usage. Fixing this is on the roadmap; there’s no need to file an issue.

• dwarfsextract format controls. New options --format-options and --format-filters control the output format. There is also --format=auto to automatically “guess” the format and filters based on the output file name. (Thanks to @oxalica for the pull request.)

• dwarfsck detail level. New frozen_details detail level shows frozen_analysis content ordered by memory location instead of memory usage, and also shows the address range of each section.

• Lean LRU cache. Replaced folly’s EvictingCacheMap with a simple in-repo LRU cache implementation. This reduces external dependencies and binary size without sacrificing performance.

• Windows extended attributes. The pxattr utility now supports all extended attribute operations on Windows, including setxattr() and removexattr(). Error handling and reporting for extended attributes on Windows has also been improved.

Docs

• Updated dwarfs-format.md with more information on section types, compression metadata, and the Frozen2 binary metadata layout. (Thanks to @oxalica for asking the right questions, reporting bugs, and ultimately releasing a Rust library to read/write DwarFS images.)

• Added a notable users section to the README. (Thanks to Vitaly Zdanevich for the PRs.)

• Updated mkdwarfs docs with more information on worker threads and the requirements for bit-identical images.

• Major README overhaul: added a Quick Start section, added more links, and fixed typos and wording.

Full Changelog: v0.12.4...v0.13.0

eb4a21fe560721a17059eb26b14abb894420008cc0dc990b829bdfda08e97af8  dwarfs-0.13.0-Linux-aarch64.tar.xz
3781cf4e5dde77f4e7da9900dca4250d300b57bbf5ba6640ba7f61e2efb5782f  dwarfs-0.13.0-Linux-arm.tar.xz
d7157d7a2faedea61829835060edfb1614d75b8559252f237aae38a97a684e9c  dwarfs-0.13.0-Linux-i386.tar.xz
786351112f1659d041e40ef70a4376c55f68bb783fc710ff71a1cb612b6786f5  dwarfs-0.13.0-Linux-loongarch64.tar.xz
465c0a2c14f13612a0413ba4e79f853cefdcc25a82d8ea7e74e3a349ac887983  dwarfs-0.13.0-Linux-ppc64le.tar.xz
cb44187074793b4aca9e252fb0bee269725180448e7a47abbf17e38b0b32e6ac  dwarfs-0.13.0-Linux-ppc64.tar.xz
0d8eac724b5c00f22592c68c2f61a1ae7be92aef0b236ace208e73e759e6965f  dwarfs-0.13.0-Linux-riscv64.tar.xz
52dbd8e44deede8e237d4b8c2fe1388b1478c14b1f76b63b4045a7103110acfb  dwarfs-0.13.0-Linux-s390x.tar.xz
81d2c5178b51367207df819a3a754fc3141d0f74d4ac80dbf20b1f0bd9d1be44  dwarfs-0.13.0-Linux-x86_64.tar.xz
d0654fcc1219bfd11c96f737011d141c3ae5929620cd22928e49f25c37a15dc9  dwarfs-0.13.0.tar.xz
fbbdf50657caf6be3c6864768bd2f0c2f6ea955b66e07875408e0a78bed2f9b9  dwarfs-0.13.0-Windows-AMD64.7z
78c52f9ca120e11d4a4620c614d532425ab694c0f8b6c25f04b475f10b3e0b2e  dwarfs-fuse-extract-0.13.0-Linux-aarch64
451744a2be3312fcd2968aa7821d61d97ece24177cebdb4cb8cf463409f9de7d  dwarfs-fuse-extract-0.13.0-Linux-aarch64.upx
0079697de87e14ea5bbf3a0dd0e95488d8343bd423f2ca3643462520a6c95b89  dwarfs-fuse-extract-0.13.0-Linux-arm.upx
7d3fc8474c1372a92b4301c28124fac16f858d81686ee4cafe21e8b6a1cebc14  dwarfs-fuse-extract-0.13.0-Linux-i386.upx
da9558153264fd14ba49f02d9fb26a8ef5c090a3e99a643fbeb9e88b68c3fd93  dwarfs-fuse-extract-0.13.0-Linux-loongarch64
b42ad7d4a229d22ecf219fd1759f1f1ebb0ccd28b4be16e49cf307f05b154f63  dwarfs-fuse-extract-0.13.0-Linux-ppc64
a81001d3a195eab8cdfd2b29768718ff5eae64a69174d33c5114aa04b87bdb6a  dwarfs-fuse-extract-0.13.0-Linux-ppc64le
9c9f4dcee5c78466cda8d3f3b5dccaa57be0270774e373198ab5cdc6f46a8c5d  dwarfs-fuse-extract-0.13.0-Linux-riscv64
aeae39035e52632a013f318909b38e0525dc04ba4e52179ab678807b6a3755ea  dwarfs-fuse-extract-0.13.0-Linux-s390x
04c9587395dbbdfdb6562f945daf2aa83e5ff80691e1dfeb7854dfc2e0e6993e  dwarfs-fuse-extract-0.13.0-Linux-x86_64
4a31bcbf9145fa422d3cfaf3f1f58b40cd095ba9a4bee508123f97e208684f52  dwarfs-fuse-extract-0.13.0-Linux-x86_64.upx
74b730031dfe4d4aa0caaa411ea806d02ba44f32dc2618e52739ce7...

This release mainly fixes a bug that was introduced with v0.12.0 (06f8728 to be precise). When re-compressing a file system image where some blocks cannot be compressed using the selected algorithm because of a bad_compression_ratio_error (i.e. when the compressed size is actually larger than the uncompressed size), the resulting block object was left empty, which subsequently led to a segfault.

A few small bugs have also been fixed and couple of features were added to the dwarfsck tool, mostly triggered by the discussion in #263.

Most other changes that made it into the release were related to how the static binaries are built. All the dependencies have been updated, in particular also for the Windows build. The Windows build was also switched from openssl to libressl. Overall, the size of the Windows universal binary was reduced by 30% and is now comparable in size to the Linux binaries.

Bugfixes

• Segfault on bad_compression_ratio_error. When recompressing a filesystem where some blocks cannot be compressed using the selected algorithm because of a bad_compression_ratio_error, the resulting block was left empty after the refactoring done in 06f8728.
• Add history unless --no-history is given when rewriting a file system image.
• Allow dumping frozen_layout w/o frozen_analysis in dwarfsck.
• Logging timestamps should show local time.
• Workaround a weird MSVC bug.
• Remove useless cast causing compiler warning.

Features

• More complete breakdown of metadata in dwarfsck.
• Add schema_raw_dump flag to dwarfsck --detail.

Build

• Switch static build to libressl on Windows.
• Update static build libraries.
• Update folly/fbthrift/fsst.

Other

• Introduce and use safe_localtime() to prevent issues with fmt deprecating fmt::localtime.
• Speed up a few slow tests on Windows.

Full Changelog: v0.12.3...v0.12.4

1492c0796ab3479a80e4e191e651f20c005e634b73ec684edd48465830f0aab3  dwarfs-0.12.4-Linux-aarch64.tar.xz
bff9da50918cf9272dbb9322733a629ccd84efc723e355a86eafcd075081f968  dwarfs-0.12.4-Linux-x86_64.tar.xz
352d13a3c7d9416e0a7d0d959306a25908b58d1ff47fb97e30a7c8490fcff259  dwarfs-0.12.4.tar.xz
5574d6aeb970c4cbabd50526994159c0551b2dc3e940edc774ac0952e1528c93  dwarfs-0.12.4-Windows-AMD64.7z
72f688800faf74acfdc75b6d77a456930db7088ee7c7e115c4b95414ab751c93  dwarfs-fuse-extract-0.12.4-Linux-aarch64
ea182efeb3ac55f8f79ee80009854c2ac8410b37d427da90e42b96eec73c470f  dwarfs-fuse-extract-0.12.4-Linux-x86_64
06c35c1dc99bc9c19f73ceda238866f9b4a631c08ab4ad19bbf8ea5c0a3ff9f0  dwarfs-fuse-extract-mimalloc-0.12.4-Linux-aarch64
afa48dfda2692e5b3cba6c523ab66696140d48dc8c4dfd57b114c73632e45326  dwarfs-fuse-extract-mimalloc-0.12.4-Linux-x86_64
7efdbd93954f1f88898b690466dca4b2ae3d20e799247c4ee4459acb35824389  dwarfs-universal-0.12.4-Linux-aarch64
48fc469bca251b932904b33d6161fc9a964cdefdb9d4d128828b4766b9c63eaa  dwarfs-universal-0.12.4-Linux-x86_64
094eaaf821df47daaa11899927f269f57f71843b39ecd53bae70692447cb3e65  dwarfs-universal-0.12.4-Windows-AMD64.exe

This release provides a fix for cases where automatic image offset detection could fail to work correctly as well as further size optimizations of the release binaries. The dwarfs-universal binary now uses LibreSSL's libcrypto, whereas the binaries from the release tarball use OpenSSL's libcrypto. This is a trade-off favoring size for the universal binary and speed for the "regular" binaries. Note, however, that this will be imperceptible unless you use dwarfsck with either --check-integrity or --checksum.

Bugfixes

• Automatic image offset detection (for images using a custom header) did not work correctly if the header contained a string that would be identified as the start of a v1 section header (these were only used before dwarfs-0.3.0). If there was either "DWARFS\x02\x00" or "DWARFS\x02\x01" in the header, offset detection would fail. The check has been modified to peek further into the data and ensure this really is a v1 section header, and also checking if the next section header position can be derived from the length field. It is still possible to construct a file system image where offset detection will ultimately fail, but it is much less likely with the change.

Build

• The build process for the release binaries has been further tweaked to reduce binary size. The dwarfs-fuse-extract binary now again supports extracting files by pattern; I didn't realize that this was actually a widely used feature before dropping it in the last release. dwarfs-universal is now linked against LibreSSL's libcrypto instead of OpenSSL's. This significantly reduces the size at the expense slightly slower cryptographic hash functions. However, this will likely only be perceivable when using --tool=dwarfsck with either --check-integrity or --checksum. The binaries from the release tarballs are still linked against libcrypto from OpenSSL.

Full Changelog: v0.12.2...v0.12.3

7e2c1d6f4bf8f19cedc8f050f405906e269d96197e238226f334c7ae2fb1f489  dwarfs-0.12.3-Linux-aarch64.tar.xz
9a2590deb3069d7e677604304d94226fa10b7385ccee46ba8c66f4e6c902168c  dwarfs-0.12.3-Linux-x86_64.tar.xz
bd2d54178c59e229f2280eea747479a569e6f6d38340e90360220d00988f5589  dwarfs-0.12.3.tar.xz
acbfbf5a48a8fa53dd6a39d7450c8d6849ef99d305563461596ac5c52767387a  dwarfs-0.12.3-Windows-AMD64.7z
16a450e6996ab59b89f69b0e30dab508d1e88c526586df5db5b7a43ede252398  dwarfs-fuse-extract-0.12.3-Linux-aarch64
e658a0513bc9168ff6f366b2dc19f6360408663fc2f0015653912075526c05a2  dwarfs-fuse-extract-0.12.3-Linux-x86_64
755599e8b52e36a87d6b90e860f38c7cf3c71bcebd8f63e11834cca0fbed9708  dwarfs-fuse-extract-mimalloc-0.12.3-Linux-aarch64
372c3090fd966e881978aaa7ccfcbb1476cf16a3cc3ddbaaddeac14cf141cb5d  dwarfs-fuse-extract-mimalloc-0.12.3-Linux-x86_64
9c66639a66a122d964ee297b93a638c61f25c917e9aee49339fde17320757f2a  dwarfs-universal-0.12.3-Linux-aarch64
c5e83388f41cddb59b3c490fc9fad8fbd73b134f8da5b8bba77297ab593d0efb  dwarfs-universal-0.12.3-Linux-x86_64
9dbc03ce6dfc9df25e8c9e77e5ee15f624ef244aaa9b4e235fb121761c698789  dwarfs-universal-0.12.3-Windows-AMD64.exe

This release provides a fix for a performance regression, switches the default memory allocator back to jemalloc, and further reduces the size of the dwarfs-fuse-extract binary. The latter is available as both a jemalloc and a mimalloc version. jemalloc offers a lot more configuration options that can be crucial in optimizing the memory profile of e.g. the FUSE driver. If you don't need that flexibility, you can save a few bits by using the -mimalloc version.

Bugfixes

• The dwarfs-0.12.0 release introduced a performance regression where FLAC compression took more than twice as long as in the previous releases. This has been fixed. FLAC decompression was unaffected.

Build

• A few small refactoring changes to further reduce the size of the fuse-extract binary. In particular, the performance monitor and the history feature are now fully removed. Also, the functionality to extract in different archive formats as well as to extract only files matching a pattern have been removed, so the image can only be fully extracted to disk.

Full Changelog: v0.12.1...v0.12.2

7d58b4125171befb5457a6318cda99607e32c2226db74de5f7449dee0e10764f  dwarfs-0.12.2-Linux-aarch64.tar.xz
61d239c0583d88443ca3e0080f1fe8bc97979a3ad67ed15ca3516e27ea7e7f53  dwarfs-0.12.2-Linux-x86_64.tar.xz
9b256d1f2bc17917cd63a1bee3bd5f505076b4d880fcf9daa18a6ca5bca35aeb  dwarfs-0.12.2.tar.xz
1ffbe8bbf44c5168aba5d0132705ad46837e4925ff57798333efcc6312bd7441  dwarfs-0.12.2-Windows-AMD64.7z
2aea873299cecc68dc0d8028d55cb00dc5c3289d12896e70e2038ea09d780c4e  dwarfs-fuse-extract-0.12.2-Linux-aarch64
88559806c8f2a98108e9ecf24926a317ea0afe655ef45f76c476307c4e71d971  dwarfs-fuse-extract-0.12.2-Linux-x86_64
99202109637d4d49d3b3945b35487deada758358b4f185869975c7c4be9870fa  dwarfs-fuse-extract-mimalloc-0.12.2-Linux-aarch64
8af6f46b5c39fa7fa9294b652bd023302599f2723f33571b3e2bf2376f420770  dwarfs-fuse-extract-mimalloc-0.12.2-Linux-x86_64
79a8e5d729650d8f26e1759228a10c2ea49ae88c1a491741f9196ce2937b4e2e  dwarfs-universal-0.12.2-Linux-aarch64
29d3195831c8ff3aca46b2a731eee7899d3735a71b870e9510adeaeb34dd135c  dwarfs-universal-0.12.2-Linux-x86_64
1232104b7c44dda3da46fdc6f7667e542e2e916ac926e2395965bcc035ae8046  dwarfs-universal-0.12.2-Windows-AMD64.exe

A quick update to v0.12.0 that addresses a few issues and improves the performance of the release binaries while mostly making them even smaller. The universal x86_64 binary is slightly bigger, but that's a different story [1].

Bugfixes

• Attempt to fix a linking issue in the Homebrew build.

Features

• Added --memory-limit=auto to mkdwarfs to use a more reasonably (hopefully) default for the block queue. The old default of 1 GiB was quite arbitrary and definitely not suitable for low-end systems. The new auto default will determine the limit based on the number of workers (which in turn is based on the number of CPUs), the block size, and the amount of physical memory of the system.

• Replaced vector_byte_buffer with malloc_byte_buffer, which is internally based around a simple buffer that doesn't incur the cost of initializing each element like std::vector. Especially for large blocks which are known to be overwritten immediately, this can save a few CPU cycles.

• The x86_64 release binaries now use an optimized memcpy implementation (if supported by the CPU) instead of the rather slow musl memcpy implementation. This makes mkdwarfs a few percent faster and dwarfsextract up to 20% faster.

Build

• Switched the release binaries to use mimalloc instead of jemalloc. The primary reason for this change is a reduction in binary size.

• Updated the xz library to the latest 5.8.1 release.

Full Changelog: v0.12.0...v0.12.1

f61d49436ad6b02e7e496f746ce09d69a3f050592286b5ff3d6e3edba765b82a  dwarfs-0.12.1-Linux-aarch64.tar.xz
6685622e6bf1edea138023dfb5a84ec241c6a57619ec5d7ee86344057d89296b  dwarfs-0.12.1-Linux-x86_64.tar.xz
5523a5c3aea244cbfbccfe64f1df6053b3901e6af8916fac1530faf0f7a5f07f  dwarfs-0.12.1.tar.xz
d51c2e5ed021a7322928aeb8f09cc3c392362c8a1ea6217e2ba177f241f8a809  dwarfs-0.12.1-Windows-AMD64.7z
8850a0002d7008791c2629fd3a5bc718c50606dfc391bf5b1259d9f6d79c8401  dwarfs-fuse-extract-0.12.1-Linux-aarch64
d01dd82068e0d2020fd35ba5a2ddf416aeba32a9f6f7ac4544a173eefc6c743f  dwarfs-fuse-extract-0.12.1-Linux-x86_64
3dfc3d8d2152f4d9d29c543d0a30b2ba1bc0bba470742e08924ad75dc8c23967  dwarfs-universal-0.12.1-Linux-aarch64
518faa4f5a476dcc4ec75d8ed4ac31d076990bb515cd32a752a8165b3ad04885  dwarfs-universal-0.12.1-Linux-x86_64
1ad50c0b6127e56dc026de7d2f7a3df28fa3983bdd628584a652d05f97e28b88  dwarfs-universal-0.12.1-Windows-AMD64.exe

[1] I've noticed that the universal binary, and sometimes the other binaries as well, fluctuate dramatically in size after being UPX compressed. In the last CI build before tagging the new DwarFS release, the UPX-compressed universal binary had a size of 2,713,752 bytes. The only change after tagging the release was the version info (although, with LTO, you never know what the compiler makes of that). Interestingly, when unpacked, both binaries have exactly the same size. But the packed size differs by ~190 KiB.

The main features of this release are new licensing conditions and significantly smaller binaries.

New Licensing Conditions

Instead of being all GPL-3.0 like all the previous releases, this release changes the license of a large fraction of the DwarFS code to MIT. All tools and libraries that only read DwarFS images are now MIT-licensed. Everything else (e.g. mkdwarfs) is still GPL-3.0 for the time being.

Significantly reduced binary size

By moving the build pipeline to Alpine Linux and through some major refactoring, the size of the release binaries has been reduced significantly.

There's now also an additional binary called dwarfs-fuse-extract that combines the functionality of the FUSE driver dwarfs and dwarfsextract in a single, extremely small binary:

The main use case for this binary is single-file application image runtimes (e.g. uruntime). This binary doesn't have built-in manual pages or support for the performance monitor. It also only supports zstd and lzma compression.

Bugfixes

• Build release binaries against an up-to-date libfuse. Fixes github #252.

• Changes for compatibility with Boost.Process v2.

Features

• Re-licensed all libraries required for reading DwarFS images under the MIT license. The source of all tools that just read DwarFS images (i.e. everything except for mkdwarfs) are also under the MIT license now. Everything else is still GPL-3.0. Addresses github #255.

• Significantly reduced binary size in the static release builds. This is the result of refactoring code that unconditionally pulled in code-heavy dependencies such as libcrypto, as well as optimizing the build pipeline (e.g. building dependencies with only the necessary set of features) and turning on link time optimization.

• A new kind of "universal" binary dwarfs-fuse-extract is part of the release now. This combines the FUSE driver (dwarfs) and dwarfsextract into a single binary, but does not include the mkdwarfs and dwarfsck tools that are also part of the regular universal binary. dwarfs-fuse-extract is much smaller than the regular universal binary and especially suitable to AppImage-like applications.

• New hotness categorizer in mkdwarfs that allows a list of "hot" files to be stored in distinct file system blocks.

• New explicit ordering mode in mkdwarfs that allows files to be ordered accoring to the order in a given list file.

• dwarfs now shows the version of the FUSE library used.

• New dwarfs options preload_all and preload_category to populate the block cache immediately after mounting.

• New dwarfs option analysis_file that can be used for profiling and as input to mkdwarfs new hotness categorizer and explicit ordering mode.

• New dwarfs option block_allocator that allows the user to switch from a malloc-based block allocator to an mmap-based one. This can help with returning memory back to the system if the blocks are evicted from the cache.

Full Changelog: v0.11.3...v0.12.0

15867a3e1b5a310ea27700806ce7d504c912de8741dddbb40430eee419c4532f  dwarfs-0.12.0-Linux-aarch64.tar.xz
4fd1e23a97d871d0536b818c11a58c58859627e347d10c1e211522c8dd56b328  dwarfs-0.12.0-Linux-x86_64.tar.xz
91d5a22e5cf125a9871bcbdb4875bdd661557757b9f50e88553da4b47f8351d2  dwarfs-0.12.0.tar.xz
e9a4d513085bb91ab26f86277cfdfbf7930adf4bcb3c44be9f67f205d1db29fb  dwarfs-0.12.0-Windows-AMD64.7z
f7e7909b779a96dccced8e6749729ccf73ac5782ed5721cc27199498b84a27c1  dwarfs-fuse-extract-0.12.0-Linux-aarch64
264e6cbac5ade98275241082770e7aefb6e931612c543deb87322be03fc65bb8  dwarfs-fuse-extract-0.12.0-Linux-x86_64
41981a4f7a068f6c3bc32dad5307aa4466d0d74bf13083a495151d0ece34b17c  dwarfs-universal-0.12.0-Linux-aarch64
1425e102fd3b8251629025fc3b8aab0f74b7079466261f6fcef6c0e0531249ac  dwarfs-universal-0.12.0-Linux-x86_64
fe5904d64f1bd08f8f2f7dd271587b415bef1237562da4a77ee30982b48e24c4  dwarfs-universal-0.12.0-Windows-AMD64.exe

Bugfixes

• Handle absolute paths in --input-list. Fixes github #259.
• Don't prefetch blocks that are already in the active list within the block cache.
• Ensure that statistics for block tidying are correctly updated in the block cache.
• A few build fixes, mainly to simplify building on Alpine.

New Contributors

• @hexahigh made their first contribution in #257

Full Changelog: v0.11.2...v0.11.3

de7f6609a4ddd6f2feff4cb4e43c4481515c5da178bbe12db24de9e7fec48bac  dwarfs-0.11.3-Linux-aarch64-clang-reldbg-stacktrace.tar.xz
7c0835b89871e48025b2e30577fb1b3c39927f9b86940dd3c5d7c41871e12533  dwarfs-0.11.3-Linux-aarch64-clang.tar.xz
2e771b53ebee66278b3a2e6e18fd04e20abc0f6defccb5a347dbfc2b7436b729  dwarfs-0.11.3-Linux-x86_64-clang-reldbg-stacktrace.tar.xz
adc3fc58d36848a312e846f0e737056b7e406894e24fa20d80fcc476ca7f401f  dwarfs-0.11.3-Linux-x86_64-clang.tar.xz
5ccfc293d74e0509a848d10416b9682cf7318c8fa9291ba9e92e967b9a6bb994  dwarfs-0.11.3.tar.xz
33b3488bc1097b1b2b54194eaa5fb169dfded9a6046de6c4fee693d9a97ece32  dwarfs-0.11.3-Windows-AMD64.7z
e14c0caa38a8d10273a84e57f532e513b2cbc50bb8df707b57c01d575f040a43  dwarfs-universal-0.11.3-Linux-aarch64-clang
7d4857ee18ffae705a41f164a0a810f173bf8d69bc8bef8dcbd1018fa8287f6e  dwarfs-universal-0.11.3-Linux-aarch64-clang-reldbg-stacktrace
64b349aec059b9d460211af6c517f6edd89e79c5e9581381229af745ebf3cc87  dwarfs-universal-0.11.3-Linux-x86_64-clang
07a9ef68256e76e7bda552b24955a3db12c3b34312d7d664e47639995ccdabf1  dwarfs-universal-0.11.3-Linux-x86_64-clang-reldbg-stacktrace
772f00d5d02fdebca4cbd74f4b1b37ee7b57fb3004a078c053b4f58e97a794ed  dwarfs-universal-0.11.3-Windows-AMD64.exe

Bugfixes

• macOS Ventura's version of clang appears to be missing an implementation of std::hash<std::filesystem::path, making it hard to define an unordered_map<filesystem::path>. Work around by simply using an unordered_map<string> instead.
• Installing the binaries using cmake did not honor the CMAKE_INSTALL_BINDIR or CMAKE_INSTALL_SBINDIR variables. Fixes github #253.

Full Changelog: v0.11.1...v0.11.2

61fce8eaa6bbdf10917a5a12331e192748a54ab1aa175ed6f55cb26825ab3177  dwarfs-0.11.2-Linux-aarch64-clang-reldbg-stacktrace.tar.xz
06fc4ed91ee5c348dbfc70771fe3e3ea6834277e4a58f1f99e0bc98cb16ed3d4  dwarfs-0.11.2-Linux-aarch64-clang.tar.xz
15905007cff432bb9be0bdabed93473764c1706796e0da6f3af083f0a142db6d  dwarfs-0.11.2-Linux-x86_64-clang-reldbg-stacktrace.tar.xz
3c82708e00af9d1622e78047efd216e4e29213a60aff3afa8326bade8353ea38  dwarfs-0.11.2-Linux-x86_64-clang.tar.xz
1b38faf399a6d01cd0e5f919b176e1cab76e4a8507088d060a91b92c174d912b  dwarfs-0.11.2.tar.xz
8a028693ce0a7ab083b25dc491b100f41fbf98f28413a38f6773fe1cf27574fb  dwarfs-0.11.2-Windows-AMD64.7z
600134267dd0ad51dd9d8bd1b58fa614b0a0da9a7a3d57f5fce4dbda9bb80460  dwarfs-universal-0.11.2-Linux-aarch64-clang
a9f5f79afeff4eba5cc23893de46e4c8eaa3b51b8f5938ed7f9e6cb92560fa4f  dwarfs-universal-0.11.2-Linux-aarch64-clang-reldbg-stacktrace
1bee828de84c1a3a1c2134bc866f28bdf93a62927cb7e8c416813f389f7745ad  dwarfs-universal-0.11.2-Linux-x86_64-clang
ddbd62d3bf0bf420a1720af6c03bea21ce6a77a73cedc46f28c8d79e4ac26827  dwarfs-universal-0.11.2-Linux-x86_64-clang-reldbg-stacktrace
d95dab93a7e9d8349d4a4393213a401d9ded79040b1c48e661df7dfe118b72a7  dwarfs-universal-0.11.2-Windows-AMD64.exe