delimitedbuffer

A Go library for reading and writing streams of delimited blobs. Each data record is preceded by a 4-byte length prefix (little-endian, max uint32), enabling easy parsing of variable-length streams.

Features

• Streaming interface: Implements io.Writer and io.Reader (via composition)
• Length-prefixed records: Each blob is preceded by a 4-byte little-endian length
• Full read guarantee: Uses io.ReadFull to ensure complete data retrieval
• Proper EOF handling: Returns io.EOF when no more records are available
• Partial write detection: Returns io.ErrShortWrite on incomplete writes

Installation

go get github.com/sdotz/delimitedbuffer

Basic Usage

Writer and Reader

package main

import (
    "bytes"
    "fmt"

    "github.com/sdotz/delimitedbuffer"
)

func main() {
    buf := bytes.NewBuffer(nil)
    writer := delimitedbuffer.NewProtoBufWriter(buf)

    // Write some data
    msg1 := []byte("Hello, World!")
    n, err := writer.Write(msg1)
    if err != nil {
        panic(err)
    }
    fmt.Printf("Wrote %d bytes\n", n)

    msg2 := []byte("Second message")
    n, err = writer.Write(msg2)
    if err != nil {
        panic(err)
    }
    fmt.Printf("Wrote %d bytes\n", n)

    // Read back
    reader := delimitedbuffer.NewProtoBufReader(buf)
    for {
        data, err := reader.Read()
        if err != nil {
            if err == io.EOF {
                break
            }
            panic(err)
        }
        fmt.Printf("Read: %s\n", string(data))
    }
}

Using with Files

func main() {
    file, err := os.Create("output.bin")
    if err != nil {
        panic(err)
    }
    defer file.Close()

    writer := delimitedbuffer.NewProtoBufWriter(file)

    msg := []byte("Hello from file!")
    n, err := writer.Write(msg)
    if err != nil {
        panic(err)
    }
}

Chained Readers/Writers

// Writer chain: original -> delimitedbuffer -> gzip writer
origWriter := gzip.NewWriter(ioWriter)
delimitedWriter := delimitedbuffer.NewProtoBufWriter(origWriter)

// Ensure all buffers are flushed and the chain is closed properly
// See the full example above for proper cleanup

Use Cases

1. Protocol Buffer Streaming

// Stream multiple protobuf messages without serialization overhead
pbFiles := []string{
    "users.pb",
    "products.pb",
    "orders.pb",
}

for _, fileName := range pbFiles {
    data, _ := os.ReadFile(fileName)
    _, err := writer.Write(data)
    if err != nil {
        log.Fatal(err)
    }
}

2. Message Bus

// Broadcast messages to multiple subscribers
for _, subscriber := range subscribers {
    subWriter := delimitedbuffer.NewProtoBufWriter(subscriberConn)
    message, _ := encoder.Encode(event)
    if _, err := subWriter.Write(message); err != nil {
        log.Printf("Failed to send to subscriber: %v", err)
    }
}

3. Batch Log Aggregation

func processLogs() error {
    batch := make([][]byte, 0, batchSize)
    for logChunk := range logStream {
        batch = append(batch, compress(logChunk))
        
        // Write when batch is full
        if len(batch) >= batchSize {
            if err := writeBatch(toDelimitedWriter(batch)); err != nil {
                return err
            }
            batch = nil
        }
    }
    return nil
}

API Reference

NewProtoBufWriter(w io.Writer) *ProtoBufWriter

Creates a new writer that prepends a 4-byte length prefix to each write.

NewProtoBufReader(r io.Reader) *ProtoBufReader

Creates a new reader that reads length-prefixed records.

Methods

Method	Description
Write(data []byte) (int, error)	Writes length-prefixed data
Read() ([]byte, error)	Reads next length-prefixed record

Error Conventions:

• io.EOF: No more records available
• io.ErrShortWrite: Underlying writer couldn't write all bytes
• Other error: Underlying IO error

Error Handling

writer := delimitedbuffer.NewProtoBufWriter(buf)

// Write
n, err := writer.Write(data)
if err == io.ErrShortWrite {
    log.Printf("Partial write: wrote %d of %d bytes", n, len(data))
}

// Read
data, err := reader.Read()
if err == io.EOF {
    // End of stream reached
    break
}
if err != nil {
    panic(err)
}

License

MIT License