codec-corpus

Runtime access to the imazen/codec-corpus test image collection. No data ships with the crate — datasets download on first use and cache locally.

let corpus = codec_corpus::Corpus::new()?;
let valid = corpus.get("webp-conformance/valid")?;
for entry in std::fs::read_dir(valid)? {
    let path = entry?.path();
    // decode, validate, benchmark...
}

What it does

1. You call corpus.get("some-folder/optional-subpath")
2. If the folder isn't cached (or the cache is stale), it downloads via git sparse checkout or HTTP tarball
3. Returns the local path. Done.

Downloads use shell git (preferred) with fallback to curl/wget/powershell. No heavy HTTP crate dependencies.

Install

[dev-dependencies]
codec-corpus = "1"

Usage

use codec_corpus::Corpus;

#[test]
#[ignore] // network access required
fn jpeg_conformance() {
    let corpus = Corpus::new().unwrap();
    let valid = corpus.get("jpeg-conformance/valid").unwrap();

    for entry in std::fs::read_dir(valid).unwrap() {
        let path = entry.unwrap().path();
        let data = std::fs::read(&path).unwrap();
        // test your decoder...
    }
}

Custom cache location

let corpus = Corpus::with_cache_root("/mnt/fast-storage")?;

Or via environment variable:

CODEC_CORPUS_CACHE=/mnt/fast-storage cargo test -- --ignored

Check what's cached

if corpus.is_cached("pngsuite") {
    println!("already downloaded");
}

for name in corpus.list_cached() {
    println!("cached: {name}");
}

Datasets

Any top-level folder in the codec-corpus repo is a valid path. Pass any path into get() — the first component determines the download unit.

Quality calibration

Format conformance

Decoder robustness

Full dataset descriptions and per-file attribution: codec-corpus README.

Cache layout

~/.cache/codec-corpus/v1/          # Linux
~/Library/Caches/codec-corpus/v1/  # macOS
%LOCALAPPDATA%\codec-corpus\v1\    # Windows

  .version          # "1.0.0" — triggers re-download on version change
  .lock             # fd-lock for concurrent access
  pngsuite/
  jpeg-conformance/
  ...

Different major versions coexist (v1/, v2/). Any crate version change within a major version triggers a re-download to ensure correctness.

CI integration

- uses: actions/cache@v4
  with:
    path: ~/.cache/codec-corpus
    key: corpus-v1
- run: cargo test --release -- --ignored

No special setup. The crate handles downloading; the CI cache avoids re-downloading across runs.

Dependencies

Two Rust crates, both small:

• dirs — cross-platform cache directory
• fd-lock — file locking for concurrent safety

Archive extraction uses the system tar command. No reqwest, ureq, gix, serde, or toml.

License

The crate itself is Apache-2.0. Each dataset in the corpus has its own license — see the table above and the full license summary.