A standalone, high-performance Kafka-compatible protocol server written in Rust.

• Full Kafka wire protocol compatibility (v0-v3 for most APIs)
• Zero-copy message handling for optimal performance
• Multiple compression algorithms (Gzip, Snappy, LZ4, Zstandard)
• Pluggable storage backends (S3/MinIO, local filesystem, in-memory)
• Built-in metrics and monitoring via Prometheus
• Consumer group coordination
• SASL/PLAIN authentication support
• Configurable performance tuning

# Run with default settings (local storage)
cargo run --bin heraclitus

# Run with in-memory storage
cargo run --bin heraclitus -- --storage-path memory://

# Run with S3/MinIO storage
export AWS_ACCESS_KEY_ID=minioadmin
export AWS_SECRET_ACCESS_KEY=minioadmin
export AWS_ENDPOINT_URL=http://localhost:9000
cargo run --bin heraclitus -- --storage-path s3://my-bucket/heraclitus

# Run with custom ports
cargo run --bin heraclitus -- --kafka-port 9092 --http-port 9093

# Run with debug logging
cargo run --bin heraclitus -- --debug

Create a heraclitus.toml file:

kafka_port = 9092
http_port = 9093

[storage]
# Local filesystem path
path = "/var/lib/heraclitus"
# Or use in-memory storage (data lost on restart)
# path = "memory://"
# Or use S3/MinIO (configure via environment variables)
# path = "s3://my-bucket/heraclitus"
# dsn = "s3://my-bucket/heraclitus"  # 'dsn' is an alias for 'path'

[auth]
enabled = false
mechanism = "PLAIN"
# plain_username = "admin"
# plain_password = "secret"

[batching]
max_batch_size = 1048576      # 1MB
max_batch_messages = 1000
flush_interval_ms = 100

[metrics]
enabled = true
prefix = "heraclitus"

[performance]
tcp_nodelay = true
socket_send_buffer_bytes = 131072  # 128KB
socket_recv_buffer_bytes = 131072  # 128KB
buffer_pool_size = 1000
compression_level = 6

Run with configuration file:

cargo run --bin heraclitus -- --config heraclitus.toml

When using S3 storage (s3://bucket/path), configure via environment variables:

export AWS_ACCESS_KEY_ID=your-access-key
export AWS_SECRET_ACCESS_KEY=your-secret-key
export AWS_DEFAULT_REGION=us-east-1
export AWS_ENDPOINT_URL=http://localhost:9000  # For MinIO or custom S3 endpoint

The S3 URL format is: s3://bucket-name/optional/path/prefix

Heraclitus is designed as a standalone server with clean separation of concerns:

• Protocol Handler: Implements the Kafka wire protocol with per-API handlers
• State Manager: Manages topics, partitions, offsets, and consumer groups
• Storage Layer: Efficient storage using Arrow/Parquet format
• Batch Writer: Optimizes writes with configurable batching
• Metrics: Prometheus-compatible metrics exposed via HTTP

Each Kafka API is implemented as a separate handler:

• Auth: ApiVersions, SaslHandshake, SaslAuthenticate
• Metadata: Metadata, ListOffsets
• Produce: Produce, InitProducerId
• Fetch: Fetch
• Consumer Groups: JoinGroup, Heartbeat, LeaveGroup, SyncGroup, etc.
• Admin: CreateTopics, DeleteTopics

• Zero-copy message handling using bytes::Bytes
• Async compression to avoid blocking the event loop
• Buffer pooling to reduce allocations
• Sticky partitioning for better batching efficiency
• Configurable socket options for network tuning

# Debug build
cargo build

# Release build with optimizations
cargo build --release

# Run tests
cargo test

# Run benchmarks
cargo bench

Metrics are exposed at http://localhost:9093/metrics (by default) in Prometheus format.

Key metrics include:

• Request rates and latencies per API
• Message throughput and sizes
• Consumer group states
• Storage statistics
• Connection counts

Apache License 2.0