Redd-Archiver

⭐ If you find this project useful, please star the repo! It helps others discover the tool and motivates continued development.

Transform compressed data dumps into browsable HTML archives with flexible deployment options. Redd-Archiver supports offline browsing via sorted index pages, full-text search with Docker deployment, or fully dynamic serving straight from PostgreSQL. Archives stay current with monthly incremental updates from Arctic Shift dumps. Features mobile-first design, multi-platform support, operator-selectable themes, and PostgreSQL full-text indexing.

Supported Platforms:

Platform	Format	Status	Available Posts	Data
Reddit	.zst JSON Lines (Pushshift/Arctic Shift)	✅ Full support	2.38B+ posts (40,029 subreddits; rolling — monthly dumps keep archives current)	Download
Voat	SQL dumps	✅ Full support	3.81M posts, 24.1M comments (22,637 subverses, complete archive)	Download
Ruqqus	.7z JSON Lines	✅ Full support	500K posts (6,217 guilds, complete archive)	Download

Tracked content: 2.38B+ posts across 68,883 communities (full Reddit dataset plus monthly Arctic Shift dumps via incremental updates; Voat/Ruqqus complete archives)

Version 1.1 “Living Archive” adds three serving modes, monthly incremental updates, 11 theme palettes, community metadata/wiki enrichment, and a major performance pass on top of 1.0's multi-platform archiving, REST API, and MCP server. See CHANGELOG.md.

🧭 Serving Modes

One archive, three ways to serve it — the database is the canonical store, switch modes anytime:

	Static	Hybrid (default)	Dynamic
Runtime requirements	Any web host	nginx + Flask + PostgreSQL	Flask + PostgreSQL
Full-text search / REST API	—	✅	✅
Dynamic filtering (`?flair=&min_score=&from=`) & `/all/` view	—	—	✅
Content live immediately after import	export step	export step	✅ instantly
GitHub Pages / USB-stick / offline	✅	pages only	—

# Static: export once, host anywhere
reddarc.py /data --subreddit privacy ... --output /var/www/html/

# Hybrid (current default): static pages + search server
docker compose up -d

# Dynamic: no export step, Flask renders pages from PostgreSQL
reddarc.py --import-only /data --subreddit privacy ...
REDDARCHIVER_SERVE_MODE=dynamic python search_server.py

🔄 Keep Archives Current (Incremental Updates)

Apply monthly Arctic Shift dumps to an existing archive — only tracked subreddits are imported, re-runs are skipped by checksum, and scores refresh without altering preserved content:

# One month
reddarc.py --update RS_2026-01.zst --comments-file RC_2026-01.zst

# Or point at a downloaded monthly torrent (comments/ + submissions/ layout
# is auto-discovered) and apply every unprocessed month in order:
reddarc.py --update-all /data/monthly/reddit/
reddarc.py --update-status   # audit what has been applied

🚀 Quick Start

Archive internet history before it disappears - Deploy in 2 minutes, no domain required.

Try the live demo: Browse Example Archive →

→ QUICKSTART.md - Step-by-step deployment:

2 min: Tor hidden service (no domain, no port forwarding, works behind CGNAT)
5 min: Local testing (HTTP on localhost)
15 min: Production HTTPS (automated Let's Encrypt)

Why now? Communities get banned, platforms shut down, discussions vanish. Start preserving today.

Documentation

→ First time here? QUICKSTART.md - Deploy in 2-15 minutes

→ Quick answers? FAQ - Common questions answered in 30 seconds

→ Need help? Troubleshooting - Fix common issues

→ Using the API? API Reference - 30+ REST endpoints

→ How it works? Architecture - Technical deep-dive

→ Deployment guides:

Tor Hidden Service - .onion setup (2 min, no domain needed)
HTTPS Production - Let's Encrypt SSL (15 min)
Static Hosting - GitHub/Codeberg Pages (browse-only)
Docker Reference - Complete Docker guide

→ Operations:

Incremental Updates - Keep archives current with monthly dumps
Performance - Memory, storage, and tuning
Scaling - Multi-instance deployments
Search Setup - Full-text search configuration

→ Advanced:

MCP Server - AI integration (Claude Desktop/Code)
Scanner Tools - Data discovery utilities
Registry Setup - Instance leaderboard
Installation Guide - Platform-specific setup
Contributing · Security Policy · License

🎯 Key Features

🌐 Multi-Platform Support

Archive content from multiple link aggregator platforms in a single unified archive:

Platform	Format	CLI Flag	URL Prefix
Reddit	.zst JSON Lines	`--subreddit`	`/r/`
Voat	SQL dumps	`--subverse`	`/v/`
Ruqqus	.7z JSON Lines	`--guild`	`/g/`

Automatic Detection: Platform auto-detected from file extensions
Unified Search: PostgreSQL FTS searches across all platforms
Mixed Archives: Combine Reddit, Voat, and Ruqqus in single archive

🤖 MCP Server (AI Integration)

29 MCP tools auto-generated from OpenAPI for AI assistants:

Full Archive Access: Query posts, comments, users, search via Claude Desktop or Claude Code
Token Overflow Prevention: Built-in LLM guidance with field selection and truncation
5 MCP Resources: Instant access to stats, top posts, subreddits, search help
Claude Code Ready: Copy-paste configuration for immediate use

{
  "mcpServers": {
    "reddarchiver": {
      "command": "uv",
      "args": ["--directory", "/path/to/mcp_server", "run", "python", "server.py"],
      "env": { "REDDARCHIVER_API_URL": "http://localhost:5000" }
    }
  }
}

See MCP Server Documentation for complete setup guide.

📖 For Readers (offline, mobile, Tor)

📱 Mobile-First Design: Responsive layout optimized for all devices with touch-friendly navigation
⚡ JavaScript Free: Complete functionality without JS, pure CSS interactions — Tor-optimized, no external dependencies
📇 Offline Browsing Aids: Per-letter title indexes (Ctrl+F-friendly), flair indexes, and an archive map page — search-like navigation with zero server
🔍 Full-Text Search (server deployments): PostgreSQL FTS with Google-style operators — keywords, subreddit, author, date, score
📰 Community Context: Subreddit descriptions, rules, and wikis; Voat subverse metadata, user profiles, and flair
♿ Accessibility: WCAG compliant — Lighthouse 100 accessibility score across page types
🚄 Performance: ~13KB gzipped CSS, 3–32KB gzipped pages, Lighthouse 94–100, designed for low-bandwidth networks

🛠️ For Operators

🧭 Three Serving Modes: static (host anywhere), hybrid (static + search server), dynamic (everything served live from PostgreSQL) — switch anytime, same database
🔄 Incremental Updates: monthly dumps apply idempotently; archives stay current without rebuilds
🎨 Themes: 11 palettes (default, sepia, nord, solarized, dracula, gruvbox, cyberpunk, midnight OLED, old-reddit, phosphor, high-contrast) via --theme / REDDARCHIVER_THEME, plus --accent-color and --custom-css; CSS-only dark/light mode follows system preference with a manual toggle
🗄️ PostgreSQL Backend: streaming imports with constant memory; COPY protocol; resume from checkpoints
🚀 Deployment Options: localhost/LAN (2 commands), HTTPS with automated Let's Encrypt (15 min), Tor hidden service (2 min, works behind CGNAT), HTTPS+Tor dual-mode, or GitHub/Codeberg Pages (static)
🔗 SEO Ready: meta tags, XML sitemaps, structured data; --precompress + gzip_static for high-traffic static serving
🏆 Instance Registry: leaderboard with completeness-weighted scoring for distributed archiving

🔬 For Researchers & AI

🌐 REST API v1: 30+ endpoints — posts, comments, users, statistics, search, aggregations, batch and export (CSV/NDJSON) — with field selection and truncation controls
🤖 MCP Server: 29 tools for Claude Desktop/Claude Code (see above)
📊 Rich Statistics: analytics dashboard, per-community metrics, user activity summaries

📸 Screenshots

Dashboard

Main landing page showing archive overview with statistics for 9,592 posts across Reddit, Voat, and Ruqqus. Features customizable branding (site name, project URL), responsive cards, activity metrics, and content statistics. (Works offline)

Subreddit Index

Post listing with sorting options (score, comments, date), pagination, and badge coloring. Includes navigation and theme toggle. (Works offline - sorted by score/comments/date)

Post Page with Comments

Individual post displaying nested comment threads with collapsible UI, user flair, and timestamps. Comments include anchor links for direct navigation from user pages. (Works offline)

Mobile Responsive Design

Fully optimized for mobile devices with touch-friendly navigation and responsive layout.

Search Interface

PostgreSQL full-text search with Google-style operators. Supports filtering by subreddit, author, date range, and score. (Requires Docker deployment)

Search results with highlighted excerpts using PostgreSQL ts_headline(). Sub-second response times with GIN indexing. (Server-based, Tor-compatible)

🎨 Theme Gallery

All 11 palettes on the same post page (each ships dark and light variants; --accent-color re-tints any of them):


default	nord
solarized	dracula
gruvbox	sepia
cyberpunk	midnight (OLED)
old-reddit	phosphor
high-contrast

reddarc.py ... --theme nord                    # static export
REDDARCHIVER_THEME=nord search_server.py       # dynamic mode

Sample Archive: Multi-platform archive featuring programming and technology communities from Reddit, Voat, and Ruqqus · See all screenshots →

🛠️ Installation

Prerequisites: Python 3.10+, PostgreSQL 14+, 4GB+ RAM

Quick Install (Docker):

git clone https://github.com/19-84/redd-archiver.git
cd redd-archiver

# Create required directories
mkdir -p data output/.postgres-data logs tor-public

# Configure environment (IMPORTANT: change passwords!)
cp .env.example .env
nano .env  # Edit POSTGRES_PASSWORD and DATABASE_URL

# Start services
docker compose up -d

# Generate archive (after downloading .zst files to data/)
docker compose exec reddarchiver-builder python reddarc.py /data \
  --subreddit privacy \
  --comments-file /data/privacy_comments.zst \
  --submissions-file /data/privacy_submissions.zst \
  --output /output/

Detailed installation procedures (Docker, Ubuntu/Debian, macOS, Windows WSL2):

Installation Guide - Platform-specific setup and troubleshooting

📊 Usage

Quick workflow: Download data → Run archive generator → Deploy

Basic Example

# Docker (canonical): generate archive inside the builder container
docker compose up -d
docker compose exec reddarchiver-builder python reddarc.py /data \
  --subreddit privacy \
  --comments-file /data/privacy_comments.zst \
  --submissions-file /data/privacy_submissions.zst \
  --output /output/
# Access at http://localhost

# Local development (uv): same flags, host paths
export DATABASE_URL="postgresql://user:pass@localhost:5432/reddarchiver"
uv run python reddarc.py data/ --subreddit privacy \
  --comments-file data/privacy_comments.zst \
  --submissions-file data/privacy_submissions.zst --output output/

Multi-Platform Support

Reddit: .zst files from Pushshift (3.28TB, 2.38B posts)
Voat: SQL dumps from Archive.org (15GB, 3.8M posts) - Use pre-split files for 1000x speedup
Ruqqus: .7z files from Archive.org (752MB, 500K posts)

Detailed Guides

QUICKSTART.md - Step-by-step deployment (2-15 min)
Scanner Tools - Identify high-priority communities
Installation Guide - Detailed setup procedures
Docker Deployment - PostgreSQL, nginx, HTTPS, Tor in one compose file

CLI options and advanced workflows: See QUICKSTART.md for complete reference.

Environment Variables:

# Required
DATABASE_URL=postgresql://user:pass@host:5432/reddarchiver

# Serving (search server)
REDDARCHIVER_SERVE_MODE=dynamic         # hybrid (default) | dynamic
REDDARCHIVER_THEME=nord                 # 11 palettes: default|sepia|high-contrast|nord|solarized|dracula|gruvbox|cyberpunk|midnight|old-reddit|phosphor
REDDARCHIVER_ACCENT_COLOR="#8b6914"     # Accent override (hex), both modes
REDDARCHIVER_HTTP_CACHE_MAX_AGE=300     # Cache-Control/ETag for GET responses (0 disables)
REDDARCHIVER_LISTING_CACHE_TTL=300      # Listing count/stats cache (dynamic mode)
GUNICORN_WORKERS=8                      # Override CPU-scaled worker count (Docker)

# Optional Performance Tuning (auto-detected if not set)
REDDARCHIVER_MAX_DB_CONNECTIONS=8       # Connection pool size
REDDARCHIVER_MAX_PARALLEL_WORKERS=4     # Parallel processing workers
REDDARCHIVER_USER_BATCH_SIZE=2000       # User page batch size
REDDARCHIVER_QUEUE_MAX_BATCHES=10       # Queue backpressure control
REDDARCHIVER_CHECKPOINT_INTERVAL=10     # Progress save frequency
REDDARCHIVER_USER_PAGE_WORKERS=4        # User page generation workers

🏗️ Architecture

Modular PostgreSQL-backed design with specialized HTML modules and multi-platform import support:

Core Components:

reddarc.py - Main CLI entry point with platform auto-detection
core/ - PostgreSQL backend, streaming importers (Reddit/Voat/Ruqqus), incremental updates, HTML generation
api/ - REST API v1 with 30+ endpoints
mcp_server/ - MCP server for AI integration (29 tools)
dynamic_pages.py - Flask page routes for dynamic serving mode
html_modules/ - 24 specialized modules (Jinja2 rendering, SEO, themes, statistics, CSS minification)
templates_jinja2/ - 27 Jinja2 templates with inheritance system
processing/ - Parallel user processing, batch optimization, statistics
monitoring/ - Performance tracking, auto-tuning, system optimization

Key Features:

Streaming architecture with constant memory (4GB regardless of dataset size)
PostgreSQL COPY protocol for 15K+ inserts/sec
Keyset pagination for O(1) queries
Resume capability with database checkpoints
Multi-platform unified search with FTS GIN indexing

Learn more: ARCHITECTURE.md - Complete technical deep-dive

🌐 REST API & Search

30+ REST API Endpoints for programmatic access with MCP/AI optimization:

System (5): Health checks, stats, schema, OpenAPI spec
Posts (13): List, single, comments, context, tree, related, random, aggregate, batch
Comments (7): List, single, random, aggregate, batch
Users (8): Profiles, summary, activity, aggregate, batch
Subreddits (4): List, statistics, summary
Search (3): Full-text search with Google-style operators, query debugging

AI-Optimized Features: Field selection, truncation controls, export formats (CSV/NDJSON), batch endpoints, context endpoints. Rate limited to 100 req/min.

PostgreSQL Full-Text Search: Lightning-fast GIN-indexed search with relevance ranking, highlighted excerpts, and advanced filters (subreddit, author, date, score). Sub-second results for large datasets.

Instance Registry: Distributed leaderboard system for tracking archive instances. Configure metadata, automate scoring, group teams for coordinated archiving.

Learn more: API Documentation · Search Setup · Registry Guide

🎯 Use Cases

Research & Academia

Studying online discourse and community dynamics
Analyzing social movements and trends
Preserving internet culture

Community Archiving

Backing up subreddits before potential removal
Creating offline-accessible community resources
Distributing knowledge repositories

Investigation & Analysis

Pattern analysis in deleted/removed content
User behavior studies
Content moderation research

🚨 Get Involved: Help Preserve Internet History

Internet content disappears every day. Communities get banned, platforms shut down, and valuable discussions vanish. You can help prevent this.

📥 Download & Mirror Data Now

Don't wait for content to disappear. Download these datasets today:

Platform	Size	Posts	Download
Reddit	3.28TB	2.38B posts	Academic Torrents · Magnet Link
Voat	~15GB	3.8M posts	Archive.org †
Ruqqus	~752MB	500K posts	Archive.org ‡

† Voat Performance Tip: Use pre-split files for 1000x faster imports (2-5 min vs 30+ min per subverse)
‡ Ruqqus: Docker image includes p7zip for automatic .7z decompression

Every mirror matters. Store locally, seed torrents, share with researchers. Be part of the preservation network.

🌐 Join the Registry: Deploy Your Instance

Already running an archive? Register it on our public leaderboard:

Deploy your instance (Quick Start - 2-15 minutes)
Submit via Registry Template
Join coordinated preservation efforts with other teams

Benefits:

Public visibility and traffic
Coordinated archiving to avoid duplication
Team collaboration opportunities
Leaderboard recognition

👉 Register Your Instance Now →

🆕 Submit New Data Sources

Found a new platform dataset? Help expand the archive network:

Lemmy databases
Hacker News archives
Alternative Reddit archives
Other link aggregator platforms

👉 Submit Data Source →

Why submit?

Makes data discoverable for other archivists
Prevents duplicate preservation efforts
Builds comprehensive multi-platform archive ecosystem
Tracks data availability before platforms disappear

🤝 Contributing

We welcome contributions! Please see CONTRIBUTING.md for development guidelines, code structure, and testing procedures.

Key areas for contribution:

PostgreSQL query optimizations
Additional export formats
Enhanced search features
Documentation improvements

See our modular architecture (24 specialized HTML modules plus importer, API, and processing packages) for easy entry points to contribute.

📝 License

This is free and unencumbered software released into the public domain. See the LICENSE file (Unlicense) for details.

Anyone is free to copy, modify, publish, use, compile, sell, or distribute this software for any purpose, commercial or non-commercial, and by any means.

📦 Data Sources

This project leverages public datasets from the following sources:

Pushshift - Reddit data access and archival infrastructure
Watchful1's PushshiftDumps - Comprehensive data dump tools and torrent management
Arctic Shift - Making Reddit data accessible to researchers and the public
Ruqqus Public Dataset - 752 MB Ruqqus archive (comments and submissions)
SearchVoat Archive - 16.8 GB Voat.co complete backup

🙏 Acknowledgments

This project builds upon the work of several excellent archival projects:

reddit-html-archiver by libertysoft3 - Original inspiration and foundation for static HTML generation
redarc - Self-hosted Reddit archiving with PostgreSQL and full-text search
red-arch - Static website generator for Reddit subreddit archives
zst_blocks_format - Efficient block-based compression format for processing large datasets

📧 Contact

GitHub Issues: Report bugs or request features
GitHub Discussions: Ask questions or share ideas
Security Issues: Report via GitHub Security Advisories

💰 Support the Project

Redd-Archiver was built by one person over 6 months as a labor of love to preserve internet history before it disappears forever.

This isn't backed by a company or institution—just an individual committed to keeping valuable discussions accessible. Your support helps:

Continue development and bug fixes
Maintain documentation and support
Cover infrastructure costs (servers, storage, bandwidth)
Preserve more data sources and platforms

Every donation, no matter the size, helps keep this preservation effort alive.

Donation addresses (BTC / XMR)

Bitcoin (BTC)

bc1q8wpdldnfqt3n9jh2n9qqmhg9awx20hxtz6qdl7

Scan to donate Bitcoin

Monero (XMR)

42zJZJCqxyW8xhhWngXHjhYftaTXhPdXd9iJ2cMp9kiGGhKPmtHV746EknriN4TNqYR2e8hoaDwrMLfv7h1wXzizMzhkeQi

Scan to donate Monero

Thank you for supporting internet archival efforts! Every contribution helps maintain and improve this project.

Star History

This software is provided "as is" under the Unlicense. See LICENSE for details. Users are responsible for compliance with applicable laws and terms of service when processing data.

Name		Name	Last commit message	Last commit date
Latest commit History 198 Commits
.github		.github
api		api
core		core
docker		docker
docs		docs
html_modules		html_modules
mcp_server		mcp_server
monitoring		monitoring
processing		processing
qr-codes		qr-codes
roadmap		roadmap
screenshots		screenshots
scripts		scripts
seo-assets		seo-assets
sql		sql
static		static
templates_jinja2		templates_jinja2
tests		tests
tools		tools
utils		utils
.dockerignore		.dockerignore
.env.example		.env.example
.gitignore		.gitignore
.mcp.json.example		.mcp.json.example
.pre-commit-config.yaml		.pre-commit-config.yaml
ARCHITECTURE.md		ARCHITECTURE.md
CHANGELOG.md		CHANGELOG.md
CLAUDE.md		CLAUDE.md
CODE_OF_CONDUCT.md		CODE_OF_CONDUCT.md
CONTRIBUTING.md		CONTRIBUTING.md
Dockerfile		Dockerfile
LICENSE		LICENSE
Makefile		Makefile
QUICKSTART.md		QUICKSTART.md
README.md		README.md
SECURITY.md		SECURITY.md
docker-compose.prod.yml		docker-compose.prod.yml
docker-compose.tor-only.yml		docker-compose.tor-only.yml
docker-compose.yml		docker-compose.yml
dynamic_pages.py		dynamic_pages.py
postgres.conf		postgres.conf
pyproject.toml		pyproject.toml
reddarc.py		reddarc.py
requirements.txt		requirements.txt
search_server.py		search_server.py
spec.md		spec.md
uv.lock		uv.lock
version.py		version.py

Folders and files

Latest commit

History

Repository files navigation

Redd-Archiver

🧭 Serving Modes

🔄 Keep Archives Current (Incremental Updates)

🚀 Quick Start

Documentation

🎯 Key Features

🌐 Multi-Platform Support

🤖 MCP Server (AI Integration)

📖 For Readers (offline, mobile, Tor)

🛠️ For Operators

🔬 For Researchers & AI

📸 Screenshots

Dashboard

Subreddit Index

Post Page with Comments

Mobile Responsive Design

Search Interface

🎨 Theme Gallery

🛠️ Installation

📊 Usage

Basic Example

Multi-Platform Support

Detailed Guides

🏗️ Architecture

🌐 REST API & Search

🎯 Use Cases

Research & Academia

Community Archiving

Investigation & Analysis

🚨 Get Involved: Help Preserve Internet History

📥 Download & Mirror Data Now

🌐 Join the Registry: Deploy Your Instance

🆕 Submit New Data Sources

🤝 Contributing

📝 License

📦 Data Sources

🙏 Acknowledgments

📧 Contact

💰 Support the Project

Bitcoin (BTC)

Monero (XMR)

Star History

About

Topics

Resources

License

Code of conduct

Contributing

Security policy

Uh oh!

Stars

Watchers

Forks

Uh oh!

Contributors

Uh oh!

Languages