Safe, ergonomic Rust bindings and high-level APIs for OpenZL — a graph-based typed compression library optimized for structured data.

• openzl — Safe, high-level Rust API over OpenZL with typed compression primitives (graphs, TypedRef, TypedBuffer) and ergonomic helpers like compress_numeric, decompress_numeric, and compress_with_graph.
• openzl-sys — Low-level bindgen-generated FFI bindings to the OpenZL C library (openzl). Includes a custom build.rs to generate bindings and link against a local OpenZL installation.
• Root crate (openzl-workspace) — minimal binary (src/main.rs) used for workspace scaffolding.

OpenZL is a graph-based typed compression system. Instead of treating all input as an opaque byte stream, OpenZL uses compression graphs that describe how specific data structures should be compressed. This enables type-aware strategies such as delta encoding, bitpacking, field-level LZ, and entropy coding for superior compression of structured data.

Key ideas:

• Compression Graphs define the strategy (e.g., ZSTD, NUMERIC, FIELD_LZ, STORE).
• TypedRef provides typed input (serial bytes, numeric arrays, strings, structs).
• TypedBuffer owns decompression output with type metadata and typed accessors.

Implemented in openzl:

• Safe wrappers for core contexts: CCtx, DCtx, Compressor.
• Graph selection helpers and built-in graphs: ZstdGraph, NumericGraph, StoreGraph, FieldLzGraph.
• High-level APIs:
  • compress_serial / decompress_serial
  • compress_with_graph<G: GraphFn>
  • compress_typed_ref, compress_multi_typed_ref
  • compress_numeric<T> / decompress_numeric<T>
• Typed interfaces: TypedRef constructors for serial, numeric, strings, and structs; TypedBuffer typed accessors.

Implemented in openzl-sys:

• bindgen-generated bindings to OpenZL headers.
• Linker configuration to libopenzl via build.rs.

Examples:

• openzl/examples/numeric.rs demonstrates compress_numeric/decompress_numeric over several numeric datasets and prints sizes and ratios.

• Rust toolchain (edition 2024).
• OpenZL C library installed locally and accessible to the build script.
  • The build script in openzl-sys/build.rs expects OpenZL at: /home/mahmudsudo/openzl with:
    • Headers in /home/mahmudsudo/openzl/include
    • Library in /home/mahmudsudo/openzl/lib (containing libopenzl.*)

If your OpenZL install is elsewhere, update openzl-sys/build.rs accordingly.

cargo build

If linking fails, ensure libopenzl is discoverable at the path configured in openzl-sys/build.rs and that the headers are available for bindgen.

cargo run --package openzl --example numeric

This will print compression stats and verify round-trip decompression for several numeric datasets.

• Serial data:

use openzl::{compress_serial, decompress_serial};

let data = b"Hello, OpenZL!";
let compressed = compress_serial(data)?;
let decompressed = decompress_serial(&compressed)?;
assert_eq!(data.as_slice(), decompressed.as_slice());
# Ok::<(), openzl::Error>(())

• Numeric data (type-optimized):

use openzl::{compress_numeric, decompress_numeric};

let data: Vec<u32> = (0..10000).collect();
let compressed = compress_numeric(&data)?;
let decompressed: Vec<u32> = decompress_numeric(&compressed)?;
assert_eq!(data, decompressed);
# Ok::<(), openzl::Error>(())

• Graph-based compression:

use openzl::{compress_with_graph, decompress_serial, ZstdGraph, NumericGraph};

let data = b"Repeated data...".repeat(100);
let compressed = compress_with_graph(&data, &ZstdGraph)?;
let decompressed = decompress_serial(&compressed)?;
assert_eq!(data.as_slice(), decompressed.as_slice());
# Ok::<(), openzl::Error>(())

The openzl crate provides safe abstractions over unsafe FFI calls:

• RAII and Drop for resource management.
• Lifetime-checked TypedRef to avoid use-after-free.
• Type and width validation for numeric paths.
• Errors surfaced as Result<T, openzl::Error> with contextual messages.

• Build errors about missing headers: verify the include path in openzl-sys/build.rs points to your OpenZL headers.
• Linker errors: ensure libopenzl is in the configured library directory and is compiled for your target.
• Runtime mismatches (numeric width/type): confirm the type parameter T matches the compressed data element width.

This project is licensed under the MIT License. See LICENSE for details.