The High-Performance, Embeddable Storage Engine for Zig & SQLite.
Axion is a next-generation LSM-tree storage engine written in Zig. It is designed to exploit the full potential of modern NVMe SSDs and multi-core CPUs, offering distinct advantages: blazing fast writes, ACID compliance, MVCC, and seamless integration as a SQLite Virtual Table.
Whether you need a raw key-value store for your Zig application or a turbo-charged backend for your SQLite queries, Axion delivers.
Warning: Axion is currently in heavy development and relies on features anticipated in the upcoming Zig 0.16 release. It is not yet suitable for production environments.
APIs, file formats, and internal structures may change without notice. We welcome intrepid contributors and feedback, but please use with caution!
- LSM-Tree Architecture: Optimized for heavy write workloads with Leveled Compaction.
- ACID Transactions: Strict serializability and snapshot isolation using MVCC (Multi-Version Concurrency Control).
- Modern I/O: Utilizes
io_uringon Linux for asynchronous, high-throughput WAL writes. - SQLite Integration: Drop-in replacement for SQLite tables via the Virtual Table mechanism. Run SQL on top of an LSM tree!
- Efficiency:
- Prefix Compression: Reduces storage footprint for keys with common prefixes.
- Block Cache: Sharded, concurrent block cache to minimize disk reads.
- Bloom Filters: fast negative lookups to avoid unnecessary disk seeks.
- Zero-Copy: Optimized read paths to reduce memory churn.
- Observability: Built-in structured logging and atomic metrics for deep runtime insights.
Add Axion to your build.zig.zon and import it:
const std = @import("std");
const Axion = @import("axion").DB;
pub fn main() !void {
var gpa = std.heap.GeneralPurposeAllocator(.{}){};
const allocator = gpa.allocator();
// Open the database
var db = try Axion.open(allocator, "./my_data", .{
.memtable_size_bytes = 64 * 1024 * 1024,
.block_cache_size_bytes = 1 * 1024 * 1024 * 1024, // 1GB Cache
});
defer db.close();
// Write
try db.put("user:101", "Alice");
// Read
if (try db.get("user:101")) |value| {
defer value.deinit();
std.debug.print("Found: {s}\n", .{value.data});
}
// Atomic Transaction
var txn = try db.beginTransaction();
defer txn.deinit();
try txn.put("balance:A", "500");
try txn.put("balance:B", "300");
try txn.commit();
}Axion exposes itself as a SQLite Virtual Table module. This allows you to keep the SQL language you love but swap the B-Tree engine for Axion's LSM tree.
# Start the Axion Shell (bundled with SQLite)
./zig-out/bin/axion_shell-- Mount an Axion database as a virtual table
-- Syntax: CREATE VIRTUAL TABLE <name> USING axion('<path>', '<wal_mode>', '', '<options>');
CREATE VIRTUAL TABLE users USING axion('./users_db', 'NORMAL', '', 'memtable_mb=64;cache_mb=256');
-- Use standard SQL
INSERT INTO users (key, value) VALUES ('u1', '{"name": "John", "age": 30}');
INSERT INTO users (key, value) VALUES ('u2', '{"name": "Jane", "age": 25}');
SELECT * FROM users WHERE key >= 'u1';Axion provides a native C ABI (libaxion), making it easy to integrate with any language that supports FFI (Foreign Function Interface) or CGO.
1. Build the Shared Library:
zig build
# Output: zig-out/lib/libaxion.so (or .dll/.dylib)
# Header: include/axion.h2. Usage in C:
#include "axion.h"
#include <stdio.h>
int main() {
axion_db_t* db;
if (axion_db_open("./c_data", &db) != 0) return 1;
axion_db_put(db, "key", 3, "val", 3);
char* val;
size_t len;
if (axion_db_get(db, "key", 3, &val, &len) == 0 && val) {
printf("Got: %.*s\n", (int)len, val);
axion_db_free_val(val, len);
}
axion_db_close(db);
return 0;
}3. Usage in Python, Rust, Go:
- Python: Use
ctypesorcffito loadlibaxion.so. - Rust: Use
bindgento generate bindings fromaxion.h. - Go: Use
cgoto link againstlibaxion.
Axion requires Zig 0.16 (master).
# Build release optimized binaries
zig build -Doptimize=ReleaseFast
# Run the full test suite (Unit + Integration)
zig build test
# Run the benchmark suite
zig build -Doptimize=ReleaseFast bench_native_axion bench_native_sqlite bench_vtab_axion Axion is built on first principles, avoiding over-engineering while ensuring robustness.
+-----------------------------------------------------------------------+
| Axion - Architecture Overview |
+-----------------------------------------------------------------------+
User Application (Zig / C / Go / Python) OR SQLite Query Engine
| |
| (Native API) | (Virtual Table)
v v
+-----------------------------------------------------------------------+
| INTERFACE LAYER: c_api.zig / sqlite/vtab.zig |
+-----------------------------------------------------------------------+
|
v
+-----------------------------------------------------------------------+
| TRANSACTION LAYER (MVCC) |
| |
| [ CommitBatcher ] <---(Group Commit)---> [ Transaction Manager ] |
| | | |
| +--- Serializes WAL Entries +-- Global Version |
| +--- Detects Conflicts +-- Snapshots |
+-----------------------------------------------------------------------+
| (Writes) | (Reads)
v v
+--------------------------+ +--------------------------------+
| STORAGE LAYER (LSM) | | READ PATH |
| | | |
| +--------------------+ | | 1. Check MemTable (Skiplist) |
| | MemTable (Memory) | | | (Latest updates) |
| | [Sharded Skiplist] | | | |
| +--------------------+ | | v |
| | (Flush) | | |
| v | | 2. Check Block Cache (LRU) |
| +--------------------+ | | (Hot data in memory) |
| | WAL (Disk) | | | |
| | [Append-Only Log] | | | v |
| +--------------------+ | | |
| | (Compact) | | 3. Check Bloom Filters |
| v | | (Skip missing keys) |
| +--------------------+ | | |
| | SSTables (Disk) | | | v |
| | Level 0 (Unsorted) | | | |
| | Level 1 (Sorted) | <------------+ 4. Scan SSTables (Disk) |
| | Level 2 (Sorted) | | (Binary search blocks) |
| +--------------------+ | |
+--------------------------+ +--------------------------------+
**Key pieces**
- `WAL`: append-only log with CRC; replay trims torn tails, resets offsets, and replays idempotently.
- `MemTable`: 16-shard skiplist for writes; rotates into immutables when the configured byte budget is hit.
- `VersionSet`: snapshot container (active + immutable memtables, SSTable metadata, block cache handle) with ref-counting so iterators stay stable.
- `Compaction`: background threads pick work via level/size scores, rewrite SSTables, and install edits atomically with manifest updates.
- `BlockCache`: sharded LRU keyed by file id + block offset; ref-counted handles avoid copying block data.
- `TransactionManager`: MVCC with monotonic versions, group commit batching for WAL/memtable application, and conflict checks against recent history plus latest-visible versions.
+-----------------------------------------------------------------------+
| WORKFLOW: WRITE (PUT) |
+-----------------------------------------------------------------------+
1. User calls db.put(key, val)
2. Transaction created, assigned Sequence Number.
3. Written to WAL (Write-Ahead Log) for durability.
(Optimized: Pre-serialized, patched with checksum later)
4. Written to MemTable (In-Memory SkipList).
5. Commit confirmed to user.
... LATER (Background) ...
6. MemTable Full? -> FLUSH to Disk (Level 0 SSTable).
7. Level 0 Full? -> COMPACTION (Merge L0 -> L1).
+-----------------------------------------------------------------------+
| WORKFLOW: READ (GET) |
+-----------------------------------------------------------------------+
1. User calls db.get(key)
2. Acquire Snapshot (Current Max Sequence Number).
3. Search MemTable. Found? -> Return (if visible to snapshot).
4. Search Block Cache. Found? -> Return.
5. For each Level (L0 -> L1...):
a. Check Bloom Filter. (Negative? -> Skip File).
b. Read Index Block.
c. Read Data Block from Disk.
d. Check CRC32 Checksum.
e. Search Key in Block. Found? -> Return.
Axion can be configured via the Options string in CREATE VIRTUAL TABLE or via the C API.
| Option | Default | Description |
|---|---|---|
memtable_mb |
64 | Size of in-memory buffer (MB) before flushing. |
cache_mb |
10 | Size of the in-memory LRU block cache (MB). |
l0_limit |
4 | Number of SSTables in Level 0 before compaction triggers. |
l1_mb |
256 | Target size of Level 1 (MB). Subsequent levels are 10x larger. |
compaction_threads |
1 | Number of background threads for merging tables. |
wal_sync_mode |
Full |
Full (fsync), Normal (write), or Off (unsafe). |
config_file |
- | Path to an external configuration file (k=v format). |
Example with multiple options:
CREATE VIRTUAL TABLE logs USING axion('./logs_db', 'NORMAL', '', 'memtable_mb=128;l0_limit=8;compaction_threads=2');Axion demonstrates significant performance advantages over standard SQLite (B-Tree) for write-heavy workloads, while maintaining competitive read speeds.
Environment: Linux, Zig 0.16 (master) (ReleaseFast), 100 Threads, 50k Keys.
Safety first: Every write is physically flushed to disk.
| Metric | SQLite (Native) | Axion (Native) | Axion (VTab) |
|---|---|---|---|
| Write (Random) | 170 | 7,286 | 2,984 |
| Write (Sequential) | 145 | 7,501 | 3,131 |
| Read (Point) | 1,272,793 | 3,247,513 | 2,512,192 |
| Range Scan | 608,382 | 241,331 | 147,694 |
| Mix (Read/Write) | 2,783,051 | 13,743 | 6,286 |
Performance balanced: Writes are buffered to OS cache; io_uring used where applicable.
| Metric | SQLite (Native) | Axion (Native) | Axion (VTab) |
|---|---|---|---|
| Write (Random) | 147 | 414,050 | 311,247 |
| Write (Sequential) | 172 | 443,205 | 323,952 |
| Read (Point) | 1,277,213 | 2,631,517 | 2,326,464 |
| Range Scan | 611,262 | 244,644 | 154,379 |
| Mix (Read/Write) | 2,667,385 | 607,981 | 536,348 |
Raw in-memory speed: No disk sync guarantees.
| Metric | SQLite (Native) | Axion (Native) | Axion (VTab) |
|---|---|---|---|
| Write (Random) | 147 | 356,085 | 317,070 |
| Write (Sequential) | 172 | 409,023 | 329,262 |
| Read (Point) | 1,204,112 | 2,507,509 | 2,325,622 |
| Range Scan | 562,927 | 247,068 | 153,207 |
| Mix (Read/Write) | 2,587,663 | 727,515 | 546,838 |
Axion comes with a handy shell for management and inspection.
# Open a DB and check internal stats
./zig-out/bin/axion_shell ./my_data --sql ".stats"Output:
--- Axion Metrics ---
Puts: 1250430
Gets: 4021
Iterators: 50
Compactions: 12
Flushes: 8
BlockCache Hits: 8540
BlockCache Misses: 120
Txn Success: 1250000
Txn Conflict: 0
---------------------
This project is open-source software.
This project's codebase was initially written by human developers and has since evolved through AI-assisted audits and contributions.
- Code Origin: The core logic and initial implementation are human-authored.
- AI Role: AI tools are used for code auditing, documentation generation, website building, and intelligent auto-completion.
- Code Verification: AI does not write code directly without human oversight. There is no "vibe coding" involved; all AI-suggested changes are reviewed.
- Documentation: All documentation and website content are primarily generated and maintained by AI to ensure clarity and consistency.