Bugfixes

• Remove the access implementation from the FUSE driver. There's no point here trying to be more clever than FUSE's default. This makes sure DwarFS will behave more like other FUSE file systems. See github discussion #244 for details.
• Limit the number of chunks returned in inodeinfo xattr. Highly fragmented files would have megabytes in inodeinfo, which not only breaks the xattr interface, but can also dramatically slow down tools like eza who like to read xattrs for no apparent reason.
• Avoid nested indentation in manpages to work around ronn-ng bug. Fixes github #249.
• Don't link library against jemalloc. This fixes both issues with pydwarfs and issues building with jemalloc support on macOS. Only the binaries are now linked against jemalloc, which should be sufficient.

Features

• Support case-insensitive lookups. Fixes github #232.
• Allow setting image size in FUSE driver. Fixes github #239.
• Support extracting a subset of files with dwarfsextract using the new --pattern option. The same glob patterns can be used as for the filter rules in mkdwarfs. Fixes github #243.
• Allow overriding UID / GID for the whole file system when using the FUSE driver on non-Windows platforms. See github discussion #244.
• Expose more LZMA options (mode, mf, nice, depth).
• Improve filter patterns, which now support ranges and complementation.
• Improve speed of filesystem walk / walk_data_order calls by 80% / 40%. The impact of this will largely depend on what the code is being run for each inode, but, for example, the speed of listing more than 14 million files with dwarfsck will take about 16 seconds compared to 17 seconds with the previous release.
• Added an inode size cache to the metadata to speed up file size computation for large, highly fragmented files. The configuration is currently fixed using a conservative default. Only files with at least 128 chunks will be added to the cache, so in a lot of cases this cache may be completely empty and not contribute to the size of the file system image at all.
• Use bit-packing for hardlink, shared files, and chunk tables. This will consume less memory when loading a DwarFS image.
• Show total hardlink size in dwarfsck output.
• Library: return a dir_entry_view from readdir and find. This is more consistent, but was previously not easily possible due to the lack of a "self" dir entry in the metadata. The "self" entry has been added and will only impact the size of the metadata if directories metadata is not packed.
• Library: prefer std::string_view over char const*.
• Library: add directory iterator to directory_view.
• Library: support for maxiov parameter in readv call.

Other

• Lots of internal refactoring to improve overall code quality.

Full Changelog: v0.10.2...v0.11.0

2040a951697ddb78a4b6ee887e06be4295f9d2e3708a311ac72ad9ad7bd28aa3  dwarfs-0.11.0-Linux-aarch64-clang-reldbg-stacktrace.tar.xz
0db0d6bc823d26f86d47f1edf8b4ddbcf85fab24e118be7de9ee091234b5623e  dwarfs-0.11.0-Linux-aarch64-clang.tar.xz
a7214b10902653c582aa4c21e05e2476518ed1d15e4640cc3eb2bbe53a297120  dwarfs-0.11.0-Linux-x86_64-clang-reldbg-stacktrace.tar.xz
35e851bce5ba6a17b6b53081d1305ebcee5698d8bc770b8b1a875d2986fd6d7c  dwarfs-0.11.0-Linux-x86_64-clang.tar.xz
852c96133444493eff6f03324bc2700e31859d75410a937f0714eae9f75d2dd4  dwarfs-0.11.0.tar.xz
15591223010400488c5066a864bcee3ad71c045e2aa4bf60b7c05e9d45909b9f  dwarfs-0.11.0-Windows-AMD64.7z
da197d19b3eadfea5180034765d70c050ae9b85ade58dd0aa91b65283a079236  dwarfs-universal-0.11.0-Linux-aarch64-clang
d58ad14583345d4e7efb4ddb0278ec39c836646a39868422ca1358fa22a990b7  dwarfs-universal-0.11.0-Linux-aarch64-clang-reldbg-stacktrace
72fe171dd9d9abd0bba46e52a983934affbcc9a7349d07854eda91d788ea686b  dwarfs-universal-0.11.0-Linux-x86_64-clang
1c5b19c21aca4dc6df8cff3e06358c96fb4e3bb1e969ed3ceef0eb381d84f98b  dwarfs-universal-0.11.0-Linux-x86_64-clang-reldbg-stacktrace
f2451ed0832c13157f869a3d7ba3596fcb4bb0c5c55741fc054ce6b1bdc977c8  dwarfs-universal-0.11.0-Windows-AMD64.exe

Bugfixes

• Gracefully handle localized error message on Windows. These error messages can contain characters from a Windows (non-UTF-8) code page, which could cause a fatal error in fmt::print in the logging code. Call sites that log such error messages now try to convert these from the code page to UTF-8 or, if that fails, simply replace all characters that are invalid from a UTF-8 point-of-view. Partial fix for #241.
• Handle invalid wide chars in file names on Windows. For some reason, Windows allows invalid UTF-16 characters in file names. Try to handle these gracefully when converting to UTF-8. Partial fix for #241.
• Workaround for new boost versions which have a process component.
• Workaround for a deprecated boost header.
• Support for upcoming Boost 1.87.0. io_service was deprecated and replaced by io_context in 1.66.0. The upcoming Boost 1.87.0 will remove the deprecated API. (Thanks to Michael Cho for the fix.)
• Disable extended output algorithms (shake(128|256)).
• Install libraries to CMAKE_INSTALL_LIBDIR. Fixes #240.
• mode/uid/gid checks were expecting 16-bit types.
• stricter metadata checks and improved error messages.
• Various fixes for filesystem_extractor to prevent memory leaks, correctly handle errors during extraction, and prevent creation of invalid archive outputs due to padding.
• Various minor fixes: non-virtual dtors, missing includes, std::move vs. std::forward, unused code removal.

Other

• More test cases for stricter metadata checks. Also enable the strict checks in in unit tests by default.
• Fix typos in README.md. (Thanks to Christian Clauss for the fix.)
• Fix typos in man pages.

New Contributors

• @cclauss made their first contribution in #235
• @cho-m made their first contribution in #245

Full Changelog: v0.10.1...v0.10.2

2f4d275d006228acb2280c3bf5327f02098c2ba110d81fe3854a80f5fd848058  dwarfs-0.10.2-Linux-aarch64-clang-reldbg-stacktrace.tar.xz
75878252ef0bfc490e5bd6ad5870bc5a02531650ceacf1258807e09606069561  dwarfs-0.10.2-Linux-aarch64-clang.tar.xz
74b52460ebd2d8e752ad7fbe976c683be542a8a581fdf25ac59ba1dea5bc5d0c  dwarfs-0.10.2-Linux-x86_64-clang-reldbg-stacktrace.tar.xz
a018bfe2531763a273a2d78bc507b1c89fe58a44f7955c980c854a55f9adbaea  dwarfs-0.10.2-Linux-x86_64-clang.tar.xz
36767290a39f92782e41daaa3eb45e39550ad1a4294a6d8365bc0f456f75f00c  dwarfs-0.10.2.tar.xz
c15280d920b67b51b42117612bd8a959eb5ca9ed0202fd765e19743aad61a728  dwarfs-0.10.2-Windows-AMD64.7z
36f72f1ff049a1d955e68547540b932539beab44b0cba23efbdb7a1b0bfd32d4  dwarfs-universal-0.10.2-Linux-aarch64-clang
4d55e783e352a5aafc321f7ac36964b0493601320d3d93d021634e78e743505d  dwarfs-universal-0.10.2-Linux-aarch64-clang-reldbg-stacktrace
b565399a0a671d06be3e078376e02b388ee14133680b8d19483fc93c294b12d2  dwarfs-universal-0.10.2-Linux-x86_64-clang
cb374fc2d64bbf3bd4dd4714f1be37e3d6fc6ecffc7afd93714b6897e9d3751a  dwarfs-universal-0.10.2-Linux-x86_64-clang-reldbg-stacktrace
eb69b1bf4703d28bd3d5f477dca1ab3460dda4250c7ce1899eb4192c2c1bef69  dwarfs-universal-0.10.2-Windows-AMD64.exe

Bugfixes

• Allow building utils_test against a non-compatible, system-installed version of gtest. This is a common issue when trying to integrate dwarfs into a package manager, as these generally disallow fetching external dependencies at build time.
• dwarfsck was always reporting a block size of 1 byte rather than the actual block size of the image.
• DWARFS_HAVE_LIBBROTLI was not set correctly in the config file, causing build errors if the library was built without brotli.
• Several small fixes for building with Homebrew.

Full Changelog: v0.10.0...v0.10.1

53bb766f3a22f019c4bac7cbf995376cb4f3f0ad69e4793471af11c954185227  dwarfs-0.10.1-Linux-aarch64-clang-stacktrace.tar.xz
f272f667649d71ec7d29d6822ad4198e13a33e997d722f74f2bca23b239de72f  dwarfs-0.10.1-Linux-aarch64-clang.tar.xz
671ce264938ab4cacc8af0aabcacb1ecfffa01284b4959441e921264ae19b47e  dwarfs-0.10.1-Linux-x86_64-clang-stacktrace.tar.xz
84894bf6a26cac2eb2c8d43d6fccf1ece7665c4c15050cec494d09199bd8310e  dwarfs-0.10.1-Linux-x86_64-clang.tar.xz
4041ed9aa19e03f44dbe69b470f31423a3c358bcd07e78230311b859629785b6  dwarfs-0.10.1-Windows-AMD64.7z
db785e0e0f257fa4363d90153db34127add4552791a72998b30ded787840d039  dwarfs-0.10.1.tar.xz
3e003c9a5fbf31b75548c11a2c2c1958f606ce2c2022db4baa6d62b80201c76d  dwarfs-universal-0.10.1-Linux-aarch64-clang
44ad0a3f2d89e373b0279d1db7c19aeca46879972a2db226e31ec7ebe8ff103e  dwarfs-universal-0.10.1-Linux-aarch64-clang-stacktrace
18f99297c7425bd1bea87d47a2046bfc7e00fe9cc566f024e631ed39a6bb1913  dwarfs-universal-0.10.1-Linux-x86_64-clang
c60821be4a248be2feb54905b5bb6c5cd323014bcb7107f0d586ba7f70deb488  dwarfs-universal-0.10.1-Linux-x86_64-clang-stacktrace
768d013d55cd030c1fbabd35ad522648581c79435da4994cc39de75b3a7eda30  dwarfs-universal-0.10.1-Windows-AMD64.exe

This release doesn't bring a lot of new features or bugfixes to the command-line tools. However, large parts of the code base have been refactored (in more than 300 commits since the last release) and a couple of long-standing issues have been resolved:

• It is finally possible to properly install the libraries that implement most of DwarFS. These libraries come with all necessary headers and a CMake config script so you can start building your own tools.

• Shared library builds are now explicitly supported. These have caused some trouble in the past, but the problematic code has been refactored to avoid issue like missing compression / categorizer code.

• The code now builds about twice as fast. This is due to shipping generated code in the source tarball and removing a lot of dependencies into the folly library.

• It is now possible to do modular builds, i.e. the library, tools and FUSE driver can be built independently.

Bugfixes

• Fixed a race condition identified by ThreadSanitizer in the root node name processing.

• The terminal abstraction code did not check any errors when trying to determine the terminal width, leading to a random terminal width value. This caused the manual page tests to occasionally crash.

• Fixed some flaky tests, e.g. unmounting the FUSE driver on macOS.

Features

• Two sets of universal binaries and binary tarballs are provided for Linux platforms: one without any debug symbols, the other with minimal debug symbols and support for stack traces. For the universal binary, only the version without debug symbols will be UPX-compressed, as the stack trace functionality doesn't work with a compressed binary.

• Symbolic links to the universal binary may now be suffixed with a version (i.e. any part of the name starting with - and followed by a digit will be ignored, e.g. the symlink could be mkdwarfs-0.10 and it would be treated as mkdwarfs).

• Introduced support for extended attributes on Windows, including a new utility for cross-platform xattr manipulation (pxattr, for portable xattr).

• Enhanced file system API, adding error-code based and exception-safe versions for getattr, access, and similar functions.

• Filter rules now consistently use Unix path separators, even for the root path component. Addresses a comment in github discussion #228.

Improvements

• Extensive refactoring to improve code modularity, maintainability and to provide proper libraries. The library code has been moved to different namespaces to make it easier to understand the role of different components (e.g. reader, writer, extractor).

• Replaced all folly library dependencies in the public DwarFS library interface with alternatives from libraries like e.g. boost or nlohmann::json which are more broadly available. folly and fbthrift are still used as implementation details, but no longer leak into the public library interfaces.

• A much smaller subset of folly is now used in DwarFS and only the necessary components are built, significantly reducing the number of compilation units when building DwarFS.

• It is now possible to do modular builds in addition to the default monolithic build, i.e. you can build and install just the DwarFS libraries and later build/install the tools (mkdwarfs, ...) and/or the FUSE driver against these libraries. This is particularly useful for packaging (e.g. in Homebrew, which has removed all FUSE support from the core formulae).

• Shared library builds are now explicitly supported. This fixes issues such as github #184.

• The source tarball now contains all auto-generated code, e.g. manual pages or generated thrift code. This reduces the number of build-time dependencies (e.g. ronn or mistletoe are no longer required) and significantly reduces the build steps (it is no longer necessary to build the thrift compiler). The build is now roughly twice as fast as in the 0.9.x releases.

• The parallel-hashmap, xxHash and zstd submodules have been removed from the git repo and are no longer added to the source tarball. Both xxHash and zstd are now widely available. If a suitable version of parallel-hashmap is found on the system, this will be used, otherwise it will be fetched during the build. Being a header-only library and only used internally, there's no need for it to be installed.

• A lot of GCC warnings have been fixed and upstreamed to folly / fbthrift.

New Contributors

• @mindfocus made their first contribution in #222

Full Changelog: v0.9.10...v0.10.0

SHA-256 Checksums

52045506839249823a9b05b711f89e3fd18a729b264149fbe74f557bd8f16b81  dwarfs-0.10.0-Linux-aarch64-clang-stacktrace.tar.xz
b12a33650694baa6f546ac18768b8a7b2e1dd90c1c674d26bec7f5149a285537  dwarfs-0.10.0-Linux-aarch64-clang.tar.xz
8593eaecc1bed4f570626ff07b0e56a7f6a025c3a5849f4b0056006ab9801393  dwarfs-0.10.0-Linux-x86_64-clang-stacktrace.tar.xz
53f02da7ff7fe484b3bb91e6532987c3fb59db28131309ccbb46d2c2c9cec9ce  dwarfs-0.10.0-Linux-x86_64-clang.tar.xz
f24ee132cf0c77b0c94e49b3d8108fdf7e0ff965031bbc34aa51e3c2e066fa43  dwarfs-0.10.0-Windows-AMD64.7z
c01ae59d4662e4f027a7c8a5934b7aebe6edc4f7affd836fa73e6d861d18bf35  dwarfs-0.10.0.tar.xz
573656e33a171017046a32500b9ebe50ebd117df16212e6ef8f8d4b034618210  dwarfs-system-gtest.patch
149c9518d5229bf6dd07b5c39df02ed5cbc4bddbf9bf54f6fa5617d76a594b1c  dwarfs-universal-0.10.0-Linux-aarch64-clang
bfaad42a15c687fdaf502283792322692e479c563c56e8734efd9a298956d716  dwarfs-universal-0.10.0-Linux-aarch64-clang-stacktrace
5f3401cdd7267b6eec6f758f902d7d874534cc77b5be7233944832992e2c0081  dwarfs-universal-0.10.0-Linux-x86_64-clang
307f973fd7da525c5f06f6090b4a4f7a0bea041d785dcb718fb226f6b15f942f  dwarfs-universal-0.10.0-Linux-x86_64-clang-stacktrace
99cf71972156ec6f5d88ab0331709ecdc1e80f571582fce95c89b694d3be258b  dwarfs-universal-0.10.0-Windows-AMD64.exe

Bugfixes

• When cloning LZMA compressor objects, the LZMA filter options of the cloned instance would still point to an options object in the original instance. This could lead to LZMA errors when initializing a new compressor. Fixes github #224.
• Fetch range-v3 if no suitable version is found. Fixes github #221.
• Filter rules did not work correctly when input is root dir
• duf reports odd sizes due to using bsize instead of frsize

Bugfixes

• A bug introduced by an optimization to skip hashing of large files if they already differ in the first 4 KiB could, under rare circumstances, lead to an unexpected "inode has no file" exception after the scanning phase. This bug did not cause any file system inconsistency issues; mkdwarfs either crashes with the exception, or its output will be correct. Fixes github #217 (see also for more details).

Features

• A sequential access detector was added to the block cache, which can trigger a prefetch of blocks assumed to be read in the future. This improves sequential read throughput roughly by a factor of two. Random access should typically be unaffected. Can be configured / disabled using -o seq_detector.

• Added tracing support in FUSE driver and dwarfsextract, which allows simple performance analysis using chrome://tracing. Traces can be enabled using -o perfmon_trace and --perfmon-trace.

• Added performance monitoring and tracing support for the block cache.

Performance

• Significantly improved the speed of dwarfsck --checksum.

Bugfixes

• Build custom version of libcrypto to link with the release binaries in order for them to run properly on FIPS-enabled setups. Fixes github #210.

• When mounting a DwarFS image on macOS and viewing the volume in Finder, only the directories were shown, but no files. The root cause was that a non-existent extended attribute is reported via a different error code in macOS (ENOATTR) compared to Linux (ENODATA) and the wrong error code was returned for certain Finder-related attributes. Fixes github #211.

• macOS builds using jemalloc were crashing when calling mallctl("version", ...). The root cause of the crash is still unclear, but as a workaround, the jemalloc version is compiled in from a preprocessor constant rather than using mallctl.

Bugfixes

• Handle root uid correctly in access() implementation. Fixes github #204.

Features

• Show and track library dependencies. Dependencies will be displayed in the command line help; they will also be tracked in the history metadata of a DwarFS image. See also github #207.

Documentation

• Describe nilsimsa ordering algorithm more accurately.

Performance

• Reorder branches to improve ricepp speed with real world data.
• Some tweaks to improve segmenter speed.

UPDATE: All prebuilt binaries available for download here are unaffected by the xz/liblzma backdoor issue. The binaries are statically linked against liblzma-5.4.1, which does not include the backdoor, so even on systems with an affected version of liblzma, the statically linked binaries are safe. #207 has more details. Thanks to @bionade24 for bringing this to my attention.

Bugfixes

• Add workaround for new glog release breaking the folly build. Fixes github #201.

Performance

• Improve ricepp decoding speed by about 25% on x86 and arm, and up to 100% on Windows. Also improve encoding speed on Windows by 25%.

Bugfixes

• Windows path handling was wrong and didn't work properly for e.g. network shares. This is hopefully fixed for all tools now.