Note

The KV-store described in this post was also touched upon in a talk at HYTRADBOI 2025. Check out the video!

Overview

S2 is fundamentally a building block. We believe it is an excellent fit for a variety of use cases that require fast, scalable, and cost-effective access to sequenced data: change data capture, feature transformation, observability events, and click streams, to name just a few. A lot of the time, you really just want an object-storage-like interface for durable streams of records.

We are continuing to iterate on the core capabilities of S2, such as supporting record timestamps in addition to sequence numbers (on its way), and key-based compaction (roadmap). We also have our sights on offering higher-level layers for compatibility with protocols like Kafka that offer features like consumer groups and queue semantics, via software that runs on top of S2.

As we embark on building these additional layers, we are making the core S2 service available for anyone to use as a foundation for their own data-intensive systems. This post explores one concrete example of what that can look like, using a beloved "hello world" distributed data system: a replicated key-value store.

The S2 API is deliberately simple; however, it is by no means limited to basic use cases. S2 is designed to provide a first-class implementation of the shared log abstraction, a powerful tool for distributed systems engineers that has gained traction in recent years.

This post is structured in three parts:

An intro to the concept of shared logs.
Exploring how S2's stream API can be used to design a multi-primary replicated KV store with strong consistency properties. (This part won't be language specific at all.)
Digging into a sample implementation of the system which uses S2's Rust SDK.

Code example

The code example we'll be walking through is on GitHub. If you want to jump right into that, head over to the README where you can find some quickstart materials for getting it up and running locally, as well as sample curl invocations to exercise it.

Part 1: The shared log abstraction

Write-ahead logs (or "WAL"s) have been a cornerstone of database durability for decades. Traditionally, WALs are stored on non-volatile storage that is co-located with compute, and are used to provide crash recovery for database nodes. As soon as a mutation has been persisted in the WAL, it can be considered durable — its effects can be deterministically recovered after a crash — and the database is free to asynchronously complete more expensive actions, like rewriting a page in a B-tree.

In the distributed context, maintaining copies of a database for scalability, availability, and performance is notoriously tricky but well-researched. There are many types of consistency guarantees that a system may target, but an underlying challenge in a replicated setup is making nodes agree about the content of the database at a given moment. Strong consistency models promise that even a highly distributed database appears as if we are communicating with one that were only running on a single machine.

This is where consensus protocols like Paxos or Raft enter the picture to power state machine replication (or "SMR"). Generally, the database implementor has to take on a lot of the heavy-lifting involved in SMR, which has a whole spectrum of design possibilities.

Suppose that instead of residing on a block storage device attached to a single machine, the WAL were distributed — able to be written to and read from multiple nodes on different hosts concurrently, and independently durable across node failures.

If we had something like this, we could use the WAL itself, "disaggregated" from local compute, as the mechanism for distributing state across our replicas. Our core database code could then be much more focused on indexing and querying semantics.

This is the concept of the shared log, which Mahesh Balakrishnan explores in depth in his 2024 paper, Taming Consensus in the Wild:

As an analogy, think of the SMR platform as a filesystem and the shared log as a block device. In much the same way that a block device makes it easier to build a filesystem without worrying about hardware internals (e.g., HDD vs. SSD), a shared log helps us write an SMR layer without reasoning about the internals of the consensus protocol. The SMR layer is then free to focus on the complexity of materialization, snapshot management, query scalability, single-node failure atomicity, etc., in much the same way that a filesystem can focus on file-grain multiplexing, directories, crash consistency, etc.

This idea is at the heart of the Delos system at Meta, used for powering control plane services.

More recently, Amazon has shared how they built MemoryDB, a strongly-consistent replicated Redis, using this same technique:

MemoryDB offloads durability concerns to a separate low-latency, durable transaction log service, allowing us to scale performance, availability, and durability independently from the in-memory execution engine.

Access to a system that provides sufficient durability, performance, and a suitable API for using as a shared log service is, alas, pretty hard to come by unless you happen to find yourself at Meta or Amazon.

We want S2 to be the public implementation of exactly this type of service, which has to date been a super-power mostly just for super-scalers.

Part 2: Designing a replicated KV store

Let's see how a key-value database can be implemented using S2 as a shared log. Our example will not have nearly as sophisticated of an API as Redis — to start with, we'll just support get/put/delete.

Goals

Some things we would like to see in our KV store:

HTTP interface over a get/put/delete REST API: it should be easy to interact with the system using curl.
Durability: specifically, in multiple availability zones of a cloud region. Writes modifying the state of the store cannot be lost once acknowledged.
Horizontal scalability: we aim to distribute our requests across an arbitrary number of replicas, for improved performance and availability.
Multi-primary: each of our nodes should be capable of servicing both reads and writes. S2 does support concurrency control mechanisms¹ which simplify the robust implementation of leader/follower schemes, but we will dive into that in a future post.
Strongly consistent, linearizable reads and writes: we want every read or write to reflect all prior writes that have completed before it (and also in the order in which those writes occurred). For clients that don't require strong consistency, we can provide better performance with an opt-in "eventual consistency" mode.

Non-goals

Some aspects we will consider acceptable limitations, but all of which make for good "exercises left for the reader" 😄:

Returning prior values of a key from put and delete.
Supporting values larger than 1 MiB, which is the limit on an S2 record's size.
Snapshotting to allow for restoring state via a combination of a snapshot and, only for recent writes, the log.
Sharding or any sort of data partitioning.

Connecting to S2 concepts

To build this system, we need a shared log, stored durably, where each entry in that log provides some payload representing a change to our KV store. These log entries will need to be totally ordered, so that there is no ambiguity about the sequence of events, and we can deterministically reconstruct state from the log.

Maybe you can already see where this is going... this is exactly what we get with an S2 stream! We are going to use "log entry" and "record" interchangeably for the rest of this post.

`AppendSession` for log writes

S2 streams can be written to via unary or streaming RPCs. Since our KV store is always going to need access to its log, each node can maintain a streaming AppendSession for its lifetime.

Only two of the routes in our KV store's API actually "write": put which supplies a new value, and delete which removes the existing value. Any time a node receives a request for one of these, it will also need to prepare a log entry corresponding to those actions, and send that as an append over the session.

The actual format of this entry can be virtually any lossless representation of the key it applies to and the value being supplied (in the case of put).

Once we have encoded our entry and sent it to S2, we have to wait to receive a corresponding acknowledgement that it has become durably sequenced within the log before we can in turn send an acknowledgment to the KV store requestor.

The AppendSession RPC is full-duplex. We can send records to it and concurrently also receive acknowledgements (or an error) from it. The acknowledgements will always arrive in the same order as the inputs.


Sequence diagram for a `put` to the KV store

S2 append latency is in the critical path of put and delete calls on our store, so we can never return faster than the "round-trip time" to append a record and receive its acknowledgment. Accordingly, these write-triggering calls on our KV store will typically be in the low tens of milliseconds (50 ms at p99) — assuming our KV store is also running in the same cloud region as S2,² and that we elect to use S2's Express storage class. (We do plan to offer a faster storage class in the future for single-digit millisecond latencies.)

`ReadSession` and `CheckTail` for log reads

We have not covered the actual "materialized state" of our KV store yet.

If we were not going for horizontal scalability, and could guarantee that there would only ever be one node running, then serving reads would simply be a matter of consulting with the node's internal state stored in an in-memory dictionary of some sort. Writes could update that state directly, after receiving an acknowledgment from S2. Get requests would also be incredibly fast, as there would be no external service to communicate with.³

In a multi-primary setup, however, any of the KV store replicas might be concurrently writing to the shared log, so updating the state directly in response to an append wouldn't work. We could be missing any writes that occurred on other nodes, and replicas would quickly get out of sync.

Instead, we can have each node keep a tailing ReadSession open for its lifetime. This is a half-duplex RPC that will let us read all entries on the shared log, in the same order in which they were appended, with minimal latency overhead. If all nodes then apply those entries to their local copies of the materialized state, they will end up as identical replicas of our KV store, and any of them can service get requests by consulting their local copy.

Almost, anyway. ReadSessions do tail updates to the log in real time, but to provide strong consistency we also have to ensure that the node which is servicing the get is fully caught up with the log.

Suppose a get request comes in to one of our nodes just after a put completed (either on the same node or a different one), but our read session hasn't quite caught up with it yet. If we didn't do any extra coordination, and simply served the get using the current materialized state, we could return a stale value, and our system wouldn't actually be linearizable.

This is where CheckTail comes in. This operation gives us the current sequence number representing the tail of the stream (i.e., the sequence number that would be assigned to the next record appended). When a get arrives, our KV store just needs to check the current tail of its log, and compare it to the last sequence number which has been reflected in the local state. If the applied local state lags from the tail, we simply need to wait until we've caught up to it over the ReadSession.


Sequence diagram for a strongly consistent `get` to the KV store

Similar to put and delete, which are bounded below by the time it takes to append and receive an S2 acknowlegement, any get on our KV store is bounded by the time it takes for check_tail to return. Unlike appends, this latency is not a function of the stream's storage class, and should be quite fast. Nevertheless, clients that can tolerate eventual consistency reads (potentially stale values) may elect to forego the check_tail operation for better latency.

Wrapping it up

If we do all of this, we will end up with a KV store that can satisfy our lofty goals. We only have to deal with three fundamental operations on a stream: append, read, and check_tail. This is the power of decoupled storage and compute!

Part 3: Implementing the KV store using the Rust SDK

Now we'll walk through an actual implementation of this system that is built around the S2 Rust SDK.

If Rust is not your jam, SDKs for other languages are on their way — starting with Go, Python, and Java — as well as a REST API.

Setup

Data formats

Keys in our system can simply be Strings. For values, we'd like to support several common datatypes, for convenience.

A reasonable approach to this is to create a new Value enum which wraps other types, e.g.:

enum Value {
    Str(String),
    UInt(u64),
    List(Vec<Value>)
    // ...
}

Blog