Crate structured_zstd

Expand description

Pure-Rust Zstandard codec with a production-grade decoder, dictionary handle reuse, and an actively-improved encoder.

The crate ships:

decoding — RFC 8878 decoder (decoding::StreamingDecoder, decoding::FrameDecoder, dictionary-backed paths via decoding::DictionaryHandle).
encoding — frame compressor, streaming encoder, named and numeric compression levels (encoding::CompressionLevel).
dictionary (feature dict-builder) — COVER / FastCOVER training plus raw-to-finalized dictionary helpers.

No FFI, no cmake, no system zstd. no_std builds are supported by disabling the default std feature.

§CPU kernel features

Both the decode and the encode hot paths ship per-CPU-tier SIMD kernels. On x86 and aarch64 with std the tier is chosen at runtime (CPU-feature detection, cached on first use); on no_std it is chosen at compile time from cfg(target_feature). WebAssembly is compile-time only either way: its kernels additionally require target_feature = "simd128", so a wasm build without -C target-feature=+simd128 stays scalar.

Each tier is gated by a cargo feature: kernel-scalar, kernel-sse, kernel-bmi2, kernel-avx2, kernel-vbmi2 (x86) and kernel-neon, kernel-sve (aarch64). All are on by default except kernel-vbmi2, which is opt-in because the AVX-512 decode tier measures slower than AVX2 on the bursty decode path, so the default build is a universal binary that picks the best available tier per the above. kernel-scalar gates no code: the scalar path is the mandatory fallback and is always compiled, so the flag exists only to name that tier explicitly in a feature set. kernel-vbmi2 and kernel-sve are decoder-only; the encoder has no AVX-512 or SVE tier. The chain mirrors the ISA dependency (kernel-avx2 implies kernel-bmi2 implies kernel-sse; kernel-sve implies kernel-neon). kernel-sse covers two x86 tiers: SSE4.2 where the CPU has it, and a plain-SSE2 tier otherwise, so a pre-SSE4.2 CPU still gets vector match compares. Any subset is valid, and a flag is inert on architectures it doesn’t apply to. Constrained targets can shrink the binary by trimming tiers: --no-default-features --features kernel-scalar compiles out every per-tier dispatch, the BMI2/AVX2/VBMI2/NEON trampolines, and the explicit SSE2/NEON intrinsics in both the copy primitives and the encoder match-finder. The kernel_* features control the crate’s own explicit SIMD; they do not constrain the compiler’s autovectorizer, which may still emit vector instructions from ordinary scalar code regardless of the enabled tiers.

The packaged README is included below for the docs.rs landing page; the API anchors above link straight into the per-module documentation.

§structured-zstd

Pure-Rust Zstandard (zstd) compression and decompression. All 22 standard levels plus the negative ultra-fast range, streaming, dictionaries, no_std support and a WebAssembly build — with plain cargo: no cmake, no system zstd, no FFI.

§Highlights

Production-grade decoder — complete RFC 8878 implementation: dictionary-backed streams, raw / RLE / compressed blocks, the full frame format, optional content checksums, runtime-dispatched SIMD kernels (SSE2 / BMI2 / AVX2 / NEON, opt-in AVX-512).
Full-range encoder — every C-zstd level (-131072..=22) produces valid frames decodable by this crate and by upstream C zstd; named presets, per-knob parameter overrides, long-distance matching, streaming via std::io::Write.
Dictionaries end to end — compress and decompress with the same dictionary format C zstd consumes; reusable parsed handles; pure-Rust COVER / FastCOVER training behind the dict-builder feature.
Wire-compatible both ways — frames interoperate with C zstd in either direction; interop is enforced in CI against the reference implementation.
no_std ready — the decoder builds with --no-default-features for embedded and sandboxed targets.
WebAssembly / npm — the same codec as an npm package with automatic SIMD selection; no native addons, no postinstall scripts.
Continuously benchmarked — a public dashboard tracks speed and ratio against C zstd on every merge.

§Quick start

cargo add structured-zstd

use structured_zstd::encoding::{compress_to_vec, CompressionLevel};

let compressed = compress_to_vec(&b"hello world"[..], CompressionLevel::from_level(7));

For no_std builds disable the default features:

cargo add structured-zstd --no-default-features

Release notes for every version live in zstd/CHANGELOG.md (maintained by release-plz).

§Command-line tool

The tool ships from this same crate, so one name covers both uses:

cargo add structured-zstd        # the library
cargo install structured-zstd    # the `structured-zstd` binary

It carries no dependencies of its own — argument parsing, progress display and error reporting are written against std — so depending on the library pulls in nothing extra.

The binary speaks the upstream zstd command line: levels (-1..-19, --ultra for -20..-22, --fast[=N]), -d, -c, -o, -t, -l, -D, --train, -b, and the usual -f/-k/--rm file handling.

Flags that only steer how the work is done (-T, -B, --adapt, --[no-]progress, …) are accepted and ignored — their values are still validated, so a typo is an error rather than silence. --target-compressed-block-size does take effect: it bounds what goes into a block, so blocks flush sooner. --long means --long=27, as upstream documents, and is capped there: a larger window would produce frames this build’s decoder refuses. It needs level 16 or above, where long-distance matching actually runs — below that it is refused rather than accepted as a wider window and nothing else. A window is never declared larger than the source can fill, so a small file compressed with --long does not ask its decoders to reserve 128 MiB.

Flags that would change the result are refused instead: --format= for anything but zstd, --patch-from, --rsyncable, --no-check, --[no-]compress-literals, and the not-yet-implemented --pass-through / --exclude-compressed. -M is treated as the safety promise it is: on the runs that decode, a limit covering the 128 MiB window, the decoder’s buffers and the -D dictionary is kept and a tighter one is refused rather than ignored. Compressing, listing and training allocate no decoder, so the flag is accepted there and describes nothing, as upstream has it.

--train and --train-fastcover both train with FastCOVER, the algorithm upstream also defaults to. --train-cover and --train-legacy name algorithms this build does not have, so they are refused rather than quietly served by FastCOVER. -D takes either a dictionary produced by --train or any file at all, which is then used as raw content the way upstream does — such a dictionary has no ID, so the same bytes must be supplied when decoding.

It is deliberately not installed as zstd, so it never shadows the system tool. It does dispatch on the name it is invoked under, so linking it as the familiar names works:

ln -s "$(command -v structured-zstd)" ~/.local/bin/unzstd    # defaults to -d
ln -s "$(command -v structured-zstd)" ~/.local/bin/zstdcat   # defaults to -d -c

Distributions should register it with their alternatives mechanism rather than overwriting /usr/bin/zstd, e.g.

update-alternatives --install /usr/bin/zstd zstd /usr/bin/structured-zstd 100

§Usage

§Compression

use structured_zstd::encoding::{compress, compress_to_vec, CompressionLevel};

let data: &[u8] = b"hello world";
// Named level
let compressed = compress_to_vec(data, CompressionLevel::Fastest);
// Numeric level (C zstd compatible: 0 = default, 1-22, negative for ultra-fast)
let compressed = compress_to_vec(data, CompressionLevel::from_level(7));

use structured_zstd::encoding::{CompressionLevel, StreamingEncoder};
use std::io::Write;

let mut out = Vec::new();
let mut encoder = StreamingEncoder::new(&mut out, CompressionLevel::Fastest);
encoder.write_all(b"hello ")?;
encoder.write_all(b"world")?;
encoder.finish()?;

Named presets: Fastest (≈1), Default (≈3), Better (≈7), Best (≈13)
Frame Content Size: FrameCompressor writes FCS automatically; StreamingEncoder requires set_pledged_content_size() before the first write
Content checksums: opt-in via set_content_checksum(true)

§Fine-grained parameters

Override individual compression knobs (the drop-in equivalent of C zstd’s ZSTD_CCtx_setParameter). Every knob left unset inherits the base level’s default, so a parameter set that overrides nothing reproduces plain level-based compression. Long-distance matching is off at every level preset and is activated only here; it also needs the (default-on) ldm feature. Without it the builder still accepts enable_long_distance_matching(true) and the frame is still valid, but no long-distance matches are produced:

use structured_zstd::encoding::{
    compress_with_parameters, CompressionLevel, CompressionParameters, Strategy,
};

let data: &[u8] = b"hello world";
let params = CompressionParameters::builder(CompressionLevel::Level(19))
    .window_log(22)
    .strategy(Strategy::Btultra2)
    .enable_long_distance_matching(true)
    .build()
    .expect("parameters within bounds");

let compressed = compress_with_parameters(data, &params);

Each parameter’s valid range is queryable via CParameter::bounds() (the analogue of ZSTD_cParam_getBounds); the builder validates every set knob.

§Decompression

use structured_zstd::decoding::StreamingDecoder;
use structured_zstd::io::Read;

let compressed_data: Vec<u8> = vec![];
let mut source: &[u8] = &compressed_data;
let mut decoder = StreamingDecoder::new(&mut source).unwrap();

let mut result = Vec::new();
decoder.read_to_end(&mut result).unwrap();

§Dictionaries

use structured_zstd::decoding::{DictionaryHandle, FrameDecoder, StreamingDecoder};
use structured_zstd::io::Read;

let compressed: Vec<u8> = vec![];
let dict_bytes: Vec<u8> = vec![];
let mut output = vec![0u8; 1024];

// Parse dictionary once, then reuse handle.
let handle = DictionaryHandle::decode_dict(&dict_bytes).unwrap();
let mut decoder = FrameDecoder::new();
let _written = decoder
    .decode_all_with_dict_handle(compressed.as_slice(), &mut output, &handle)
    .unwrap();

// Compatibility path: pass raw dictionary bytes directly.
let mut decoder = FrameDecoder::new();
let _written = decoder
    .decode_all_with_dict_bytes(compressed.as_slice(), &mut output, &dict_bytes)
    .unwrap();

// Streaming helpers exist for both handle- and bytes-based paths.
let mut source: &[u8] = &compressed;
let mut stream = StreamingDecoder::new_with_dictionary_handle(&mut source, &handle).unwrap();
let mut sink = Vec::new();
stream.read_to_end(&mut sink).unwrap();

Compression takes the same dictionary format through FrameCompressor::set_dictionary_from_bytes / EncoderDictionary::from_bytes (one parse, reusable across frames).

Behind the dict-builder feature, the dictionary module trains dictionaries in pure Rust:

COVER (create_raw_dict_from_source) and FastCOVER (create_fastcover_raw_dict_from_source) raw dictionaries
finalize_raw_dict to produce the full zstd dictionary format
create_fastcover_dict_from_source for train + finalize in one call

§Feature flags

Feature	Default	What it enables
`std`	✅	Runtime CPU detection, `std::io` adapters
`hash`	✅	XXH64 content checksums
`ldm`	✅	Long-distance matching (implies `hash`: LDM hashes each window with XXH64)
`kernel-sse`, `kernel-bmi2`, `kernel-avx2`	✅	x86 SIMD kernels (`kernel-sse` covers both the SSE2 and SSE4.2 tiers)
`kernel-neon`, `kernel-sve`	✅	aarch64 SIMD kernels
`kernel-simd128`	✅	WebAssembly SIMD kernel (needs `-C target-feature=+simd128`)
`kernel-vbmi2`	❌	AVX-512 decode kernel (see note below)
`kernel-scalar`	✅	Marker for the always-compiled scalar fallback
`dict-builder`	❌	Pure-Rust COVER / FastCOVER dictionary training
`lsm`	❌	Storage-format extensions

Each flag gates its tier wherever that tier exists. kernel-sse, kernel-bmi2, kernel-avx2, kernel-neon and kernel-simd128 cover both the decoder and the encoder; kernel-vbmi2 and kernel-sve are decoder-only (the encoder has no AVX-512 or SVE tier), and kernel-scalar gates nothing, since the scalar path is the mandatory fallback and always compiled. So --no-default-features (optionally with --features kernel-scalar) compiles every per-tier dispatch and all explicit SIMD intrinsics out of the crate.

On x86 and aarch64 with std, the tier is picked at runtime from CPU detection; on no_std it comes from the target’s target_feature set at compile time. x86 has two 128-bit tiers under kernel-sse: SSE4.2 when available, otherwise a plain-SSE2 tier, so pre-SSE4.2 CPUs still get vector match compares instead of dropping to scalar.

WebAssembly is compile-time only, with or without std: wasm has no runtime feature detection, so both the decoder kernels and the encoder fastpath additionally require target_feature = "simd128". Building for wasm32 with default features and no extra flags therefore stays scalar — pass -C target-feature=+simd128 to get the SIMD tier. (The npm package sidesteps this by shipping separately compiled scalar and +simd128 payloads and picking one at load time.)

In every case these features control only the crate’s own explicit SIMD; the compiler’s autovectorizer is unaffected.

Why AVX-512 is off by default

On AVX-512 hosts the kernel-vbmi2 tier measures slower than kernel-avx2 for this decode workload: AVX-512’s license-based frequency downclocking stalls the surrounding bursty, memory-bound code and the heavier kernel never amortizes. By default runtime dispatch is therefore capped at AVX2, and AVX-512 hosts use the (faster) AVX2 tier. Opt in with --features kernel-vbmi2 for a sustained AVX-512 workload that genuinely benefits.

§Performance

Per-merge benchmarks publish to a public dashboard: structured-world.github.io/structured-zstd/dev/bench — speed and ratio against upstream C zstd over time.
The CI matrix covers x86_64-linux-gnu, i686-linux-gnu and x86_64-musl, with per-target / stage / scenario / level filtering on the dashboard.
A dedicated section tracks the WebAssembly build (simd128 + scalar) against the most popular npm wasm zstd, @bokuweb/zstd-wasm.
Methodology in BENCHMARKS.md: small payloads, entropy extremes, a 100 MiB large-stream scenario, repository corpus fixtures, optional local Silesia corpora.

Internal: compression strategy backends

Level range	Strategy	Backend
1-2	`Fast`	`Simple` matcher
3-4	`Dfast`	`Dfast` two-tier hash
5-12	`Greedy` / `Lazy` / `Lazy2`	`Row` lazy parse (`lazy_depth=0/1/2`): row match-finder above a 2^14 window, hash chain at or below it
13-15	`Btlazy2`	`Row` lazy parse over the lazily-sorted binary tree
16-17	`BtOpt`	`HashChain` candidates + `btopt` price parser
18	`BtUltra`	`HashChain` candidates + `btultra` price parser
19-22	`BtUltra2`	`HashChain` candidates + `btultra2` dual-profile parse

The level → strategy column matches upstream zstd ZSTD_defaultCParameters[0] at zstd/lib/compress/clevels.h:25-50 (srcSize > 256 KiB tier); smaller sources shift the row per upstream’s size tiers. The whole greedy..btlazy2 band runs upstream’s ZSTD_compressBlock_lazy_generic parse on the Row backend over the three upstream match finders (rows / hash chain per ZSTD_resolveRowMatchFinderMode, lazily-sorted binary tree for btlazy2).

§WebAssembly / npm

JavaScript / TypeScript consumers can use the codec from npm — no native addons, no build step:

npm install @structured-world/structured-zstd

import { compress, decompress } from "@structured-world/structured-zstd";
const framed = await compress(new TextEncoder().encode("hello"), 19);
const plain = await decompress(framed);

The package ships two WebAssembly payloads — one built with the simd128 SIMD tier, one scalar — and selects the fast one at runtime from the host engine’s capabilities. Pure ESM, strict TypeScript types. Frames interoperate with native zstd. Source lives in zstd-wasm/; see the package README.

§Storage-format extensions

Behind the lsm feature (default off), the crate adds building blocks for storage-format authors:

Skippable frames — a typed SkippableFrame API (structured_zstd::skippable) for interleaving application metadata with zstd data.
Block-subset partial decode — FrameDecoder::decode_blocks_partial decodes only the inner blocks covering a requested range (skipping the trailing ones) and preserves the clean prefix on a corrupt block.
Block-to-byte-range lookup — FrameEmitInfo::decompressed_byte_range(block_index) maps a block to its decompressed byte range, so a range query can locate which blocks cover a target byte window.
Resumable decoding — request a ResumeState (cross-block entropy tables + repcode history + next-block coordinates) from a partial decode, then feed it back to continue from a later block, even across a dropped decoder. The state does not carry the match window: the resuming call also supplies the tail of the already-decompressed output (the last min(window_size, resume_offset) bytes) via ResumeInput::window_prime.

[dependencies]
structured-zstd = { version = "0", features = ["lsm"] }

The ecosystem registry of allocated skippable-frame magic variants and the allocation policy live in docs/SKIPPABLE_MAGIC_ALLOCATIONS.md.

§Project relationship

Maintained fork of KillingSpark/zstd-rs (ruzstd) by Dmitry Prudnikov. We sync periodically with upstream but maintain an independent development trajectory focused on the CoordiNode database engine’s per-label dictionary needs.

§Support the project

USDT TRC-20 Donation QR Code

USDT (TRC-20): TFDsezHa1cBkoeZT5q2T49Wp66K8t2DmdA

§License

Apache License 2.0. Contributions will be published under the same Apache 2.0 license.

Re-exports§

pub use io_std as io;std

Modules§

decoding: RFC 8878 Zstandard decoder.
dictionarydict-builder: Code for creating a separate content dictionary.
encoding: Zstandard encoder — frame compression, streaming, dictionary support.
io_stdstd: Re-exports of std traits or local reimplementations if std is not available
skippablelsm: Typed Rust API for zstd skippable frames (RFC 8878 §3.1).

Constants§

MIN_TARGET_BLOCK_SIZE: Smallest accepted block-size target (the ZSTD_TARGETCBLOCKSIZE_MIN bound): the single source of truth shared by the Rust setters (set_target_block_size) and the C ABI parameter surface. Smallest accepted block-size target (upstream ZSTD_TARGETCBLOCKSIZE_MIN): below this the per-block header overhead dominates any latency benefit. Re-exported at the crate root as the single source of truth; the C ABI parameter bounds import it from there.
WILDCOPY_OVERLENGTH: SIMD wildcopy overshoot slack carried by every decoder backend (currently 32 bytes). Sized so the AVX2 chunked kernel in simd_copy::copy_bytes_overshooting (32-byte stride on x86-64) can fire on tail copies near the end of a fixed-capacity output buffer. Upstream zstd’s WILDCOPY_OVERLENGTH is also 32 bytes today; this matches that contract.

Functions§

active_cpu_kernel_name: Name of the active CPU kernel tier (entropy / sequence hot paths) for this process — for diagnostics and benchmark/dashboard reporting. See cpu_kernel::active_cpu_kernel_name. Name of the CPU kernel tier this process selected for the entropy / sequence hot paths: decode (literals + FSE sequence decode) and encode (entropy) share this dispatch (see #247). Returned as a stable lowercase string for diagnostics and benchmark/dashboard reporting; the value is what the runtime CPU-feature detection (or compile-time target_feature on no_std) actually resolves to on this machine, so a dashboard can attribute a measurement to the kernel that produced it.