afero

Afero: The Universal Filesystem Abstraction for Go

Afero is a powerful and extensible filesystem abstraction system for Go. It provides a single, unified API for interacting with diverse filesystems—including the local disk, memory, archives, and network storage.

Afero acts as a drop-in replacement for the standard os package, enabling you to write modular code that is agnostic to the underlying storage, dramatically simplifies testing, and allows for sophisticated architectural patterns through filesystem composition.

Why Afero?

Afero elevates filesystem interaction beyond simple file reading and writing, offering solutions for testability, flexibility, and advanced architecture.

🔑 Key Features:

• Universal API: Write your code once. Run it against the local OS, in-memory storage, ZIP/TAR archives, or remote systems (SFTP, GCS).
• Ultimate Testability: Utilize MemMapFs, a fully concurrent-safe, read/write in-memory filesystem. Write fast, isolated, and reliable unit tests without touching the physical disk or worrying about cleanup.
• Powerful Composition: Afero's hidden superpower. Layer filesystems on top of each other to create sophisticated behaviors:
  • Sandboxing: Use CopyOnWriteFs to create temporary scratch spaces that isolate changes from the base filesystem.
  • Caching: Use CacheOnReadFs to automatically layer a fast cache (like memory) over a slow backend (like a network drive).
  • Security Jails: Use BasePathFs to restrict application access to a specific subdirectory (chroot).
• os Package Compatibility: Afero mirrors the functions in the standard os package, making adoption and refactoring seamless.
• io/fs Compatibility: Fully compatible with the Go standard library's io/fs interfaces.

Installation

go get github.com/spf13/afero

import "github.com/spf13/afero"

Quick Start: The Power of Abstraction

The core of Afero is the afero.Fs interface. By designing your functions to accept this interface rather than calling os.* functions directly, your code instantly becomes more flexible and testable.

1. Refactor Your Code

Change functions that rely on the os package to accept afero.Fs.

// Before: Coupled to the OS and difficult to test
// func ProcessConfiguration(path string) error {
//     data, err := os.ReadFile(path)
//     ...
// }

import "github.com/spf13/afero"

// After: Decoupled, flexible, and testable
func ProcessConfiguration(fs afero.Fs, path string) error {
    // Use Afero utility functions which mirror os/ioutil
    data, err := afero.ReadFile(fs, path)
    // ... process the data
    return err
}

2. Usage in Production

In your production environment, inject the OsFs backend, which wraps the standard operating system calls.

func main() {
    // Use the real OS filesystem
    AppFs := afero.NewOsFs()
    ProcessConfiguration(AppFs, "/etc/myapp.conf")
}

3. Usage in Testing

In your tests, inject MemMapFs. This provides a blazing-fast, isolated, in-memory filesystem that requires no disk I/O and no cleanup.

func TestProcessConfiguration(t *testing.T) {
    // Use the in-memory filesystem
    AppFs := afero.NewMemMapFs()
    
    // Pre-populate the memory filesystem for the test
    configPath := "/test/config.json"
    afero.WriteFile(AppFs, configPath, []byte(`{"feature": true}`), 0644)

    // Run the test entirely in memory
    err := ProcessConfiguration(AppFs, configPath)
    if err != nil {
        t.Fatal(err)
    }
}

Afero's Superpower: Composition

Afero's most unique feature is its ability to combine filesystems. This allows you to build complex behaviors out of simple components, keeping your application logic clean.

Example 1: Sandboxing with Copy-on-Write

Create a temporary environment where an application can "modify" system files without affecting the actual disk.

// 1. The base layer is the real OS, made read-only for safety.
baseFs := afero.NewReadOnlyFs(afero.NewOsFs())

// 2. The overlay layer is a temporary in-memory filesystem for changes.
overlayFs := afero.NewMemMapFs()

// 3. Combine them. Reads fall through to the base; writes only hit the overlay.
sandboxFs := afero.NewCopyOnWriteFs(baseFs, overlayFs)

// The application can now "modify" /etc/hosts, but the changes are isolated in memory.
afero.WriteFile(sandboxFs, "/etc/hosts", []byte("127.0.0.1 sandboxed-app"), 0644)

// The real /etc/hosts on disk is untouched.

Example 2: Caching a Slow Filesystem

Improve performance by layering a fast cache (like memory) over a slow backend (like a network drive or cloud storage).

import "time"

// Assume 'remoteFs' is a slow backend (e.g., SFTP or GCS)
var remoteFs afero.Fs 

// 'cacheFs' is a fast in-memory backend
cacheFs := afero.NewMemMapFs()

// Create the caching layer. Cache items for 5 minutes upon first read.
cachedFs := afero.NewCacheOnReadFs(remoteFs, cacheFs, 5*time.Minute)

// The first read is slow (fetches from remote, then caches)
data1, _ := afero.ReadFile(cachedFs, "data.json")

// The second read is instant (serves from memory cache)
data2, _ := afero.ReadFile(cachedFs, "data.json")

Example 3: Security Jails (chroot)

Restrict an application component's access to a specific subdirectory.

osFs := afero.NewOsFs()

// Create a filesystem rooted at /home/user/public
// The application cannot access anything above this directory.
jailedFs := afero.NewBasePathFs(osFs, "/home/user/public")

// To the application, this is reading "/"
// In reality, it's reading "/home/user/public/"
dirInfo, err := afero.ReadDir(jailedFs, "/")

// Attempts to access parent directories fail
_, err = jailedFs.Open("../secrets.txt") // Returns an error

Real-World Use Cases

Build Cloud-Agnostic Applications

Write applications that seamlessly work with different storage backends:

type DocumentProcessor struct {
    fs afero.Fs
}

func NewDocumentProcessor(fs afero.Fs) *DocumentProcessor {
    return &DocumentProcessor{fs: fs}
}

func (p *DocumentProcessor) Process(inputPath, outputPath string) error {
    // This code works whether fs is local disk, cloud storage, or memory
    content, err := afero.ReadFile(p.fs, inputPath)
    if err != nil {
        return err
    }
    
    processed := processContent(content)
    return afero.WriteFile(p.fs, outputPath, processed, 0644)
}

// Use with local filesystem
processor := NewDocumentProcessor(afero.NewOsFs())

// Use with Google Cloud Storage
processor := NewDocumentProcessor(gcsFS)

// Use with in-memory filesystem for testing
processor := NewDocumentProcessor(afero.NewMemMapFs())

Treating Archives as Filesystems

Read files directly from .zip or .tar archives without unpacking them to disk first.

import (
    "archive/zip"
    "github.com/spf13/afero/zipfs"
)

// Assume 'zipReader' is a *zip.Reader initialized from a file or memory
var zipReader *zip.Reader 

// Create a read-only ZipFs
archiveFS := zipfs.New(zipReader)

// Read a file from within the archive using the standard Afero API
content, err := afero.ReadFile(archiveFS, "/docs/readme.md")

Serving Any Filesystem over HTTP

Use HttpFs to expose any Afero filesystem—even one created dynamically in memory—through a standard Go web server.

import (
    "net/http"
    "github.com/spf13/afero"
)

func main() {
    memFS := afero.NewMemMapFs()
    afero.WriteFile(memFS, "index.html", []byte("<h1>Hello from Memory!</h1>"), 0644)

    // Wrap the memory filesystem to make it compatible with http.FileServer.
    httpFS := afero.NewHttpFs(memFS)

    http.Handle("/", http.FileServer(httpFS.Dir("/")))
    http.ListenAndServe(":8080", nil)
}

Testing Made Simple

One of Afero's greatest strengths is making filesystem-dependent code easily testable:

func SaveUserData(fs afero.Fs, userID string, data []byte) error {
    filename := fmt.Sprintf("users/%s.json", userID)
    return afero.WriteFile(fs, filename, data, 0644)
}

func TestSaveUserData(t *testing.T) {
    // Create a clean, fast, in-memory filesystem for testing
    testFS := afero.NewMemMapFs()
    
    userData := []byte(`{"name": "John", "email": "[email protected]"}`)
    err := SaveUserData(testFS, "123", userData)
    
    if err != nil {
        t.Fatalf("SaveUserData failed: %v", err)
    }
    
    // Verify the file was saved correctly
    saved, err := afero.ReadFile(testFS, "users/123.json")
    if err != nil {
        t.Fatalf("Failed to read saved file: %v", err)
    }
    
    if string(saved) != string(userData) {
        t.Errorf("Data mismatch: got %s, want %s", saved, userData)
    }
}

Benefits of testing with Afero:

• ⚡ Fast - No disk I/O, tests run in memory
• 🔄 Reliable - Each test starts with a clean slate
• 🧹 No cleanup - Memory is automatically freed
• 🔒 Safe - Can't accidentally modify real files
• 🏃 Parallel - Tests can run concurrently without conflicts

Backend Reference

Afero vs. io/fs (Go 1.16+)

Go 1.16 introduced the io/fs package, which provides a standard abstraction for read-only filesystems.

Afero complements io/fs by focusing on different needs:

• Use io/fs when: You only need to read files and want to conform strictly to the standard library interfaces.
• Use Afero when:
  • Your application needs to create, write, modify, or delete files.
  • You need to test complex read/write interactions (e.g., renaming, concurrent writes).
  • You need advanced compositional features (Copy-on-Write, Caching, etc.).

Afero is fully compatible with io/fs. You can wrap any Afero filesystem to satisfy the fs.FS interface using afero.NewIOFS:

import "io/fs"

// Create an Afero filesystem (writable)
var myAferoFs afero.Fs = afero.NewMemMapFs()

// Convert it to a standard library fs.FS (read-only view)
var myIoFs fs.FS = afero.NewIOFS(myAferoFs)

Third-Party Backends & Ecosystem

The Afero community has developed numerous backends and tools that extend the library's capabilities. Below are curated, well-maintained options organized by maturity and reliability.

Featured Community Backends

These are mature, reliable backends that we can confidently recommend for production use:

Amazon S3 - fclairamb/afero-s3

Production-ready S3 backend built on the official AWS SDK for Go.

import "github.com/fclairamb/afero-s3"

s3fs := s3.NewFs(bucket, session)

MinIO - cpyun/afero-minio

MinIO object storage backend providing S3-compatible object storage with deduplication and optimization features.

import "github.com/cpyun/afero-minio"

minioFs := miniofs.NewMinioFs(ctx, "minio://endpoint/bucket")

Community & Specialized Backends

Cloud Storage

• Google Drive - fclairamb/afero-gdrive
  Streaming support; no write-seeking or POSIX permissions; no files listing cache

• Dropbox - fclairamb/afero-dropbox
  Streaming support; no write-seeking or POSIX permissions

Version Control Systems

• Git Repositories - tobiash/go-gitfs
  Read-only filesystem abstraction for Git repositories. Works with bare repositories and provides filesystem view of any git reference. Uses go-git for repository access.

Container and Remote Systems

• Docker Containers - unmango/aferox
  Access Docker container filesystems as if they were local filesystems

• GitHub API - unmango/aferox
  Turn GitHub repositories, releases, and assets into browsable filesystems

FUSE Integration

• Generic FUSE - JakWai01/sile-fystem
  Mount any Afero filesystem as a FUSE filesystem, allowing any Afero backend to be used as a real mounted filesystem

Specialized Filesystems

• FAT32 Support - aligator/GoFAT
  Pure Go FAT filesystem implementation (currently read-only)

Interface Adapters & Utilities

Cross-Interface Compatibility:

• jfontan/go-billy-desfacer - Adapter between Afero and go-billy interfaces (for go-git compatibility)
• Maldris/go-billy-afero - Alternative wrapper for using Afero with go-billy
• c4milo/afero2billy - Another Afero to billy filesystem adapter

Working Directory Management:

• carolynvs/aferox - Working directory-aware filesystem wrapper

Advanced Filtering:

• unmango/aferox includes multiple specialized filesystems:
  • FilterFs - Predicate-based file filtering
  • IgnoreFs - .gitignore-aware filtering
  • WriterFs - Dump writes to io.Writer for debugging

Developer Tools & Utilities

nhatthm Utility Suite - Essential tools for Afero development:

• nhatthm/aferocopy - Copy files between any Afero filesystems
• nhatthm/aferomock - Mocking toolkit for testing
• nhatthm/aferoassert - Assertion helpers for filesystem testing

Ecosystem Showcase

Windows Virtual Drives - balazsgrill/potatodrive
Mount any Afero filesystem as a Windows drive letter. Brilliant demonstration of Afero's power!

Modern Asset Embedding (Go 1.16+)

Instead of third-party tools, use Go's native //go:embed with Afero:

import (
    "embed"
    "github.com/spf13/afero"
)

//go:embed assets/*
var assetsFS embed.FS

func main() {
    // Convert embedded files to Afero filesystem
    fs := afero.FromIOFS(assetsFS)
    
    // Use like any other Afero filesystem
    content, _ := afero.ReadFile(fs, "assets/config.json")
}

Contributing

We welcome contributions! The project is mature, but we are actively looking for contributors to help implement and stabilize network/cloud backends.

• 🔥 Microsoft Azure Blob Storage
• 🔒 Modern Encryption Backend - Built on secure, contemporary crypto (not legacy EncFS)
• 🐙 Canonical go-git Adapter - Unified solution for Git integration
• 📡 SSH/SCP Backend - Secure remote file operations
• Stabilization of existing experimental backends (GCS, SFTP)

To contribute:

1. Fork the repository
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Add some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create a new Pull Request

📄 License

Afero is released under the Apache 2.0 license. See LICENSE.txt for details.

🔗 Additional Resources

• 📖 Full API Documentation
• 🎯 Examples Repository
• 📋 Release Notes
• ❓ GitHub Discussions

Afero comes from the Latin roots Ad-Facere, meaning "to make" or "to do" - fitting for a library that empowers you to make and do amazing things with filesystems.