Crate aol

Append Only Log

Generic purpose append only log implementation.

English | 简体中文

Introducation

When developing infrastructure softwares, write-ahead log or append-only log plays an important role, and people re-implement same funcationalities multiple times, but actually, the core for append-only log is just atomic append, append_batch, replay, and rewrite.

This crate provides generic purpose append-only log implementation based on std::fs::File.

• aol::fs::AppendLog:

  Generic purpose append-only log implementation based on std::fs::File.

  • It is good for:

    • The encoded entry size is smaller than 64 bytes.
    • Manifest file.
    • Write is not too frequently.

  • Pros:

    • It is growable, do not require pre-allocated.
    • Support automatically rewrite.
    • No holes in the file.

File Structure

+----------------------+--------------------------+-----------------------+
| magic text (4 bytes) | external magic (2 bytes) | magic (2 bytes)       |
+----------------------+--------------------------+-----------------------+-----------------------+-----------------------+
| op (1 bit)           | custom flag (7 bits)     | len (4 bytes)         | data (N bytes)        | checksum (8 bytes)    |
+----------------------+--------------------------+-----------------------+-----------------------+-----------------------+
| op (1 bit)           | custom flag (7 bits)     | len (4 bytes)         | data (N bytes)        | checksum (8 bytes)    |
+----------------------+--------------------------+-----------------------+-----------------------+-----------------------+
| ...                  | ...                      | ...                   | ...                   | ...                   |
+----------------------+--------------------------+-----------------------+-----------------------+-----------------------+

Installation

[dependencies]
aol = "0.3"

Example

• Manifest file for bitcask database

License

aol is under the terms of both the MIT license and the Apache License (Version 2.0).

See LICENSE-APACHE, LICENSE-MIT for details.

Copyright (c) 2024 Al Liu.

Re-exports

pub use either;

Modules

buffer

A vacant buffer that can be filled with bytes.

checksum

Traits and structs for checksuming.

leb128

LEB128 encoding and decoding

Structs

AppendLogstd

Append-only log implementation based on std::fs::File.

Builderstd

A builder used to create a AppendLog.

CustomFlags

A 7-bit custom flag.

Entry

The entry in the append-only file.

EntryFlags

Flags for the snapshot entry.

MaybeEntryRef

Maybe a reference entry type or an owned entry.

Optionsstd

Options for the append only log.

Enums

Errorstd

Errors for append-only file.

RewritePolicystd

Rewrite policy.

Traits

Batch

A batch of entries.

Record

Record for the Entry.

RecordRef

The reference type for the record.

Snapshot

The snapshot trait, snapshot may contain some in-memory information about the append-only log.