Thanks to visit codestin.com
Credit goes to github.com

Skip to content

A python migration tool from s3 to storacha

License

Mahd-Mehn/MM-python-s3-storacha

Folders and files

NameName
Last commit message
Last commit date

Latest commit

Β 

History

25 Commits
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation

py-s3-storacha

Python 3.10+ License: MIT Node.js 18+

A Python library for migrating files from AWS S3 to Storacha (IPFS). Upload your S3 data to the decentralized web with a simple Python API or CLI tool.

✨ What is This?

This library makes it easy to migrate your files from Amazon S3 to Storacha, a decentralized storage network built on IPFS. Your files become permanently accessible via IPFS gateways and content-addressed by their CID (Content Identifier).

πŸš€ Features

  • Simple Python API - Integrate S3 to Storacha migration in a few lines of code
  • Async Support - Built on asyncio for efficient concurrent operations
  • Progress Tracking - Real-time progress callbacks during migration
  • Error Handling - Comprehensive error handling with automatic retries
  • Flexible Configuration - Environment variables, config files, or direct parameters
  • Type Safe - Full type hints for better IDE support
  • IPFS Integration - Files become permanently accessible via IPFS gateways

πŸ“‹ Requirements

  • Python 3.10 or higher
  • Node.js 18 or higher (required by Storacha client)
  • AWS S3 credentials
  • Storacha account (sign up at storacha.network)

πŸ”§ Installation

Quick Install (Recommended)

# 1. Install the package
pip install git+https://github.com/Mahd-Mehn/MM-python-s3-storacha.git

# 2. Install JavaScript dependencies (automatic helper)
python -m py_s3_storacha.setup_helpers

# 3. Set up Storacha authentication (interactive)
python -m py_s3_storacha.auth_helper --setup

Manual Install

# 1. Clone and install
git clone https://github.com/Mahd-Mehn/MM-python-s3-storacha.git
cd MM-python-s3-storacha
pip install -e ".[dev]"

# 2. Install JavaScript dependencies
py-s3-storacha-setup

# 3. Set up authentication
py-s3-storacha-auth --setup

Verify Installation

# Check installation status
py-s3-storacha-setup --check

# Check authentication status
py-s3-storacha-auth --status

What Gets Installed

The installation process:

  1. βœ… Installs Python package
  2. βœ… Checks for Node.js 18+ (required)
  3. βœ… Installs @storacha/client and @aws-sdk/client-s3 via npm
  4. βœ… Sets up Storacha authentication (email-based)
  5. βœ… Creates a Storacha space for your uploads

πŸš€ Quick Start

Step 1: Authenticate with Storacha

First time only - authenticate with your email:

# Install Storacha CLI
npm install -g @storacha/cli

# Login (check your email for verification)
storacha login [email protected]

# Create a space
storacha space create my-migration-space

Step 2: Set Up Configuration

Create a .env file:

# S3 Configuration
S3_ACCESS_KEY_ID=your_aws_access_key
S3_SECRET_ACCESS_KEY=your_aws_secret_key
S3_REGION=us-east-1
S3_BUCKET_NAME=your-bucket-name

# Storacha Configuration
[email protected]
STORACHA_ENDPOINT_URL=https://api.storacha.network
STORACHA_SPACE_NAME=my-migration-space

# Migration Settings (optional)
MIGRATION_DRY_RUN=false
MIGRATION_VERBOSE=true

Step 3: Run Migration

import asyncio
from py_s3_storacha import (
    S3Config,
    StorachaConfig,
    MigrationRequest,
    S3ToStorachaMigrator,
)

async def migrate():
    # Load configuration from environment
    s3_config = S3Config.from_env()
    storacha_config = StorachaConfig.from_env()

    # Create migrator
    migrator = S3ToStorachaMigrator(s3_config, storacha_config)

    # Create migration request
    request = MigrationRequest(
        source_path="my-folder/",
        destination_path="backup/"
    )

    # Execute migration
    result = await migrator.migrate(request)

    # Access your files via IPFS
    print(f"βœ“ Migrated {result.objects_migrated} files")
    print(f"βœ“ Access via: https://{result.root_cid}.ipfs.storacha.link/")

    return result

# Run it
asyncio.run(migrate())

That's it! Your files are now on IPFS.

πŸ“– API Reference

Configuration Classes

S3Config

from py_s3_storacha import S3Config

# Create from parameters
config = S3Config(
    access_key_id="your_key",
    secret_access_key="your_secret",
    region="us-east-1",
    bucket_name="my-bucket",
    endpoint_url="https://s3.amazonaws.com"  # Optional
)

# Or load from environment variables
config = S3Config.from_env(prefix="S3_")

# Or from dictionary
config = S3Config.from_dict({
    "access_key_id": "your_key",
    "secret_access_key": "your_secret",
    "region": "us-east-1",
    "bucket_name": "my-bucket"
})

StorachaConfig

from py_s3_storacha import StorachaConfig

# Create from parameters
config = StorachaConfig(
    api_key="[email protected]",  # Your email for authentication
    endpoint_url="https://api.storacha.network",
    space_name="my-space"
)

# Or load from environment
config = StorachaConfig.from_env(prefix="STORACHA_")

MigrationConfig

from py_s3_storacha import MigrationConfig

config = MigrationConfig(
    batch_size=100,           # Objects per batch
    timeout_seconds=300,      # Operation timeout
    retry_attempts=3,         # Retry failed operations
    verbose=True,             # Detailed logging
    dry_run=False            # Set True to test without uploading
)

Migration Classes

S3ToStorachaMigrator

Main class for performing migrations:

from py_s3_storacha import S3ToStorachaMigrator, MigrationRequest

# Create migrator
migrator = S3ToStorachaMigrator(
    s3_config=s3_config,
    storacha_config=storacha_config,
    migration_config=migration_config  # Optional
)

# Create request
request = MigrationRequest(
    source_path="folder/",
    destination_path="backup/",
    include_pattern="*.jpg",      # Optional: only migrate matching files
    exclude_pattern="temp/*",     # Optional: skip matching files
    overwrite_existing=False,     # Optional: skip existing files
    verify_checksums=True         # Optional: verify file integrity
)

# Execute migration
result = await migrator.migrate(request)

MigrationResult

Returned after migration completes:

result.success              # bool: True if successful
result.objects_migrated     # int: Number of files migrated
result.total_size_bytes     # int: Total bytes transferred
result.duration_seconds     # float: Time taken
result.errors               # list: Any errors encountered
result.warnings             # list: Warnings (includes root CID)
result.skipped_objects      # list: Files skipped
result.failed_objects       # list: Files that failed

Progress Tracking

from py_s3_storacha import MigrationProgress

def on_progress(progress: MigrationProgress):
    print(f"Progress: {progress.progress_percentage:.1f}%")
    print(f"Files: {progress.objects_completed}/{progress.total_objects}")
    print(f"Bytes: {progress.bytes_transferred}/{progress.total_bytes}")

result = await migrator.migrate(request, progress_callback=on_progress)

οΏ½ Authentication

Storacha Authentication

Storacha uses email-based authentication with UCAN delegations. The first time you run a migration:

  1. The script sends a verification email to the address you provide
  2. Click the verification link in your email
  3. The script continues automatically once verified
  4. Credentials are stored for future use
# First run - requires email verification
storacha_config = StorachaConfig(
    api_key="[email protected]",  # Your email
    endpoint_url="https://api.storacha.network",
    space_name="my-space"
)

# Subsequent runs - uses stored credentials automatically

Alternative: Use Storacha CLI

If you prefer, authenticate once with the CLI:

npm install -g @storacha/cli
storacha login [email protected]
storacha space create my-space

Then the library will use those credentials automatically.

S3 Authentication

Standard AWS credentials:

# Option 1: Environment variables
export S3_ACCESS_KEY_ID=your_key
export S3_SECRET_ACCESS_KEY=your_secret

# Option 2: AWS credentials file (~/.aws/credentials)
# Option 3: IAM role (if running on EC2/ECS)

πŸ›‘οΈ Error Handling

Exception Types

from py_s3_storacha import (
    S3StorachaError,        # Base exception
    ConfigurationError,     # Invalid configuration
    MigrationError,         # Migration failures
    JSWrapperError         # JavaScript execution errors
)

try:
    result = await migrator.migrate(request)
except ConfigurationError as e:
    print(f"Config error: {e}")
    print(f"Context: {e.context}")
except MigrationError as e:
    print(f"Migration failed: {e}")
    print(f"Failed objects: {e.failed_objects}")
except S3StorachaError as e:
    print(f"Error: {e}")

Automatic Retries

The library automatically retries failed operations:

  • Network errors: 3 retries with exponential backoff
  • Transient failures: Automatic retry with backoff
  • Configuration errors: No retry (fail fast)

Accessing Error Details

try:
    result = await migrator.migrate(request)
except MigrationError as e:
    # Get error context
    print(f"Operation: {e.operation}")
    print(f"Source: {e.source_path}")
    print(f"Destination: {e.destination_path}")
    print(f"Objects processed: {e.objects_processed}")

    # Get original error
    if e.original_error:
        print(f"Caused by: {e.original_error}")

πŸ› οΈ Helper Commands

The library includes helper commands for setup and authentication:

Setup Helper

# Install JavaScript dependencies
py-s3-storacha-setup

# Force reinstall
py-s3-storacha-setup --force

# Check installation status
py-s3-storacha-setup --check

Authentication Helper

# Interactive setup
py-s3-storacha-auth --setup

# Setup with specific email
py-s3-storacha-auth --setup --email [email protected]

# Check authentication status
py-s3-storacha-auth --status

Programmatic Usage

from py_s3_storacha import (
    install_js_dependencies,
    verify_installation,
    StorachaAuthHelper
)

# Install JS dependencies
install_js_dependencies()

# Check installation
status = verify_installation()
print(f"Ready: {status['ready']}")

# Setup authentication
helper = StorachaAuthHelper()
helper.setup_authentication(email="[email protected]")

πŸ§ͺ Testing

Run the test suite:

# Install development dependencies
pip install -e ".[dev]"

# Run tests
pytest

# Run tests with coverage
pytest --cov=py_s3_storacha

# Run specific test categories
pytest tests/unit/          # Unit tests only
pytest tests/integration/   # Integration tests only

🀝 Contributing

We welcome contributions! Please see our Contributing Guide for details.

Development Setup

  1. Fork the repository
  2. Clone your fork: git clone https://github.com/yourusername/MM-python-s3-storacha.git
  3. Create a virtual environment: python -m venv venv
  4. Activate it: source venv/bin/activate (Linux/Mac) or venv\Scripts\activate (Windows)
  5. Install in development mode: pip install -e ".[dev]"
  6. Run tests: pytest

Code Quality

We use several tools to maintain code quality:

# Format code
ruff format .

# Lint code
ruff check .

# Type checking
pyright

πŸ“„ License

This project is licensed under the MIT License - see the LICENSE file for details.

πŸ™ Acknowledgments

  • Built on top of existing JavaScript S3 to Storacha implementation
  • Uses hatchling for modern Python packaging
  • Inspired by the need for seamless data migration between cloud storage providers

πŸ“ž Support

πŸ—ΊοΈ Roadmap

  • Support for additional cloud storage providers
  • GUI interface for non-technical users
  • Incremental sync capabilities
  • Advanced filtering and transformation options
  • Performance optimizations for large-scale migrations

Made with ❀️ by the py-s3-storacha team

About

A python migration tool from s3 to storacha

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Contributors 3

  •  
  •  
  •