Octopii: A distributed systems kernel for building replicated, fault-tolerant services

Octopii is a distributed systems kernel that provides the fundamental primitives for building replicated, crash-resistant services. Unlike typical frameworks that glue libraries together, Octopii abstracts over time, storage, network, and randomness to enable both production deployment and rigorous deterministic testing.

• Raft Consensus - Two implementations: OpenRaft (async, production) and raft-rs (simulation mode)
• Walrus WAL - Custom Write-Ahead Log with two-phase commit, dual-topic durability, and crash recovery
• QUIC Transport - Encrypted, multiplexed networking with connection pooling via Quinn
• Deterministic Simulation - Controlled time, I/O fault injection, and reproducible randomness for crash testing
• Durability Verification - Multi-cycle crash recovery validation with invariant checking

[dependencies]
octopii = { git = "https://github.com/octopii-rs/octopii", features = ["openraft"] }

use octopii::{Config, OctopiiNode, OctopiiRuntime};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let runtime = OctopiiRuntime::new(4);

    let config = Config {
        node_id: 1,
        bind_addr: "127.0.0.1:5001".parse()?,
        peers: vec!["127.0.0.1:5002".parse()?],
        wal_dir: "./data/node1".into(),
        is_initial_leader: true,
        ..Default::default()
    };

    let node = OctopiiNode::new(config, runtime).await?;
    node.start().await?;

    // Write with consensus
    node.propose(b"SET key value".to_vec()).await?;

    // Read from state machine
    let value = node.query(b"GET key").await?;
    println!("Value: {:?}", String::from_utf8(value.to_vec()));

    Ok(())
}

Implement StateMachineTrait to replicate your own logic:

impl StateMachineTrait for MyStateMachine {
    fn apply(&self, command: &[u8]) -> Result<Bytes, String> { /* deterministic execution */ }
    fn snapshot(&self) -> Vec<u8> { /* serialize state */ }
    fn restore(&self, data: &[u8]) -> Result<(), String> { /* restore from snapshot */ }
    fn compact(&self) -> Result<(), String> { Ok(()) } // optional
}

Two-phase commit with dual topics for crash resistance:

1. Write to main topic → 2. Write to recovery topic → 3. Write commit marker

On recovery, only entries with a commit marker are visible. Prevents partial write corruption even with mid-operation crashes.

• API.md - API reference
• ARCHITECTURE.md - System internals
• CUSTOM_STATE_MACHINES.md - State machine guide
• SHIPPING_LANE.md - P2P file transfers

node/, custom_state_machine/, shipping_lane/, rpc/, rpc_shipping_lane/, wal/, runtime/

cargo run --example node --features openraft

Octopii implements DST, the methodology pioneered by FoundationDB and adopted by TigerBeetle. Find bugs that would take years to surface in production by running millions of fault-injected operations in minutes.

Given a seed, tests are fully reproducible.

I/O errors, partial writes (torn pages), fsync failures, artificial latency.

Consistency, durability (DurabilityOracle), Raft invariants, recovery idempotence.

11,000+ lines of tests across 19 test files. The cluster simulation suite covers:

Each category runs across 50 seeds by default, 1000 operations per run (5000 for stress).

cargo test --features openraft                           # Standard tests
cargo test --features simulation                         # DST with fault injection
CLUSTER_SEED_COUNT=200 cargo test --features simulation  # Extended coverage

Under active development. The DST infrastructure has validated crash recovery across hundreds of thousands of fault-injected operations.

Licensed under the Apache License, Version 2.0. See LICENSE for details.