A Python library for migrating files from AWS S3 to Storacha (IPFS). Upload your S3 data to the decentralized web with a simple Python API or CLI tool.
This library makes it easy to migrate your files from Amazon S3 to Storacha, a decentralized storage network built on IPFS. Your files become permanently accessible via IPFS gateways and content-addressed by their CID (Content Identifier).
- Simple Python API - Integrate S3 to Storacha migration in a few lines of code
- Async Support - Built on asyncio for efficient concurrent operations
- Progress Tracking - Real-time progress callbacks during migration
- Error Handling - Comprehensive error handling with automatic retries
- Flexible Configuration - Environment variables, config files, or direct parameters
- Type Safe - Full type hints for better IDE support
- IPFS Integration - Files become permanently accessible via IPFS gateways
- Python 3.10 or higher
- Node.js 18 or higher (required by Storacha client)
- AWS S3 credentials
- Storacha account (sign up at storacha.network)
# 1. Install the package
pip install git+https://github.com/Mahd-Mehn/MM-python-s3-storacha.git
# 2. Install JavaScript dependencies (automatic helper)
python -m py_s3_storacha.setup_helpers
# 3. Set up Storacha authentication (interactive)
python -m py_s3_storacha.auth_helper --setup# 1. Clone and install
git clone https://github.com/Mahd-Mehn/MM-python-s3-storacha.git
cd MM-python-s3-storacha
pip install -e ".[dev]"
# 2. Install JavaScript dependencies
py-s3-storacha-setup
# 3. Set up authentication
py-s3-storacha-auth --setup# Check installation status
py-s3-storacha-setup --check
# Check authentication status
py-s3-storacha-auth --statusThe installation process:
- β Installs Python package
- β Checks for Node.js 18+ (required)
- β
Installs
@storacha/clientand@aws-sdk/client-s3via npm - β Sets up Storacha authentication (email-based)
- β Creates a Storacha space for your uploads
First time only - authenticate with your email:
# Install Storacha CLI
npm install -g @storacha/cli
# Login (check your email for verification)
storacha login [email protected]
# Create a space
storacha space create my-migration-spaceCreate a .env file:
# S3 Configuration
S3_ACCESS_KEY_ID=your_aws_access_key
S3_SECRET_ACCESS_KEY=your_aws_secret_key
S3_REGION=us-east-1
S3_BUCKET_NAME=your-bucket-name
# Storacha Configuration
[email protected]
STORACHA_ENDPOINT_URL=https://api.storacha.network
STORACHA_SPACE_NAME=my-migration-space
# Migration Settings (optional)
MIGRATION_DRY_RUN=false
MIGRATION_VERBOSE=trueimport asyncio
from py_s3_storacha import (
S3Config,
StorachaConfig,
MigrationRequest,
S3ToStorachaMigrator,
)
async def migrate():
# Load configuration from environment
s3_config = S3Config.from_env()
storacha_config = StorachaConfig.from_env()
# Create migrator
migrator = S3ToStorachaMigrator(s3_config, storacha_config)
# Create migration request
request = MigrationRequest(
source_path="my-folder/",
destination_path="backup/"
)
# Execute migration
result = await migrator.migrate(request)
# Access your files via IPFS
print(f"β Migrated {result.objects_migrated} files")
print(f"β Access via: https://{result.root_cid}.ipfs.storacha.link/")
return result
# Run it
asyncio.run(migrate())That's it! Your files are now on IPFS.
from py_s3_storacha import S3Config
# Create from parameters
config = S3Config(
access_key_id="your_key",
secret_access_key="your_secret",
region="us-east-1",
bucket_name="my-bucket",
endpoint_url="https://s3.amazonaws.com" # Optional
)
# Or load from environment variables
config = S3Config.from_env(prefix="S3_")
# Or from dictionary
config = S3Config.from_dict({
"access_key_id": "your_key",
"secret_access_key": "your_secret",
"region": "us-east-1",
"bucket_name": "my-bucket"
})from py_s3_storacha import StorachaConfig
# Create from parameters
config = StorachaConfig(
api_key="[email protected]", # Your email for authentication
endpoint_url="https://api.storacha.network",
space_name="my-space"
)
# Or load from environment
config = StorachaConfig.from_env(prefix="STORACHA_")from py_s3_storacha import MigrationConfig
config = MigrationConfig(
batch_size=100, # Objects per batch
timeout_seconds=300, # Operation timeout
retry_attempts=3, # Retry failed operations
verbose=True, # Detailed logging
dry_run=False # Set True to test without uploading
)Main class for performing migrations:
from py_s3_storacha import S3ToStorachaMigrator, MigrationRequest
# Create migrator
migrator = S3ToStorachaMigrator(
s3_config=s3_config,
storacha_config=storacha_config,
migration_config=migration_config # Optional
)
# Create request
request = MigrationRequest(
source_path="folder/",
destination_path="backup/",
include_pattern="*.jpg", # Optional: only migrate matching files
exclude_pattern="temp/*", # Optional: skip matching files
overwrite_existing=False, # Optional: skip existing files
verify_checksums=True # Optional: verify file integrity
)
# Execute migration
result = await migrator.migrate(request)Returned after migration completes:
result.success # bool: True if successful
result.objects_migrated # int: Number of files migrated
result.total_size_bytes # int: Total bytes transferred
result.duration_seconds # float: Time taken
result.errors # list: Any errors encountered
result.warnings # list: Warnings (includes root CID)
result.skipped_objects # list: Files skipped
result.failed_objects # list: Files that failedfrom py_s3_storacha import MigrationProgress
def on_progress(progress: MigrationProgress):
print(f"Progress: {progress.progress_percentage:.1f}%")
print(f"Files: {progress.objects_completed}/{progress.total_objects}")
print(f"Bytes: {progress.bytes_transferred}/{progress.total_bytes}")
result = await migrator.migrate(request, progress_callback=on_progress)Storacha uses email-based authentication with UCAN delegations. The first time you run a migration:
- The script sends a verification email to the address you provide
- Click the verification link in your email
- The script continues automatically once verified
- Credentials are stored for future use
# First run - requires email verification
storacha_config = StorachaConfig(
api_key="[email protected]", # Your email
endpoint_url="https://api.storacha.network",
space_name="my-space"
)
# Subsequent runs - uses stored credentials automaticallyIf you prefer, authenticate once with the CLI:
npm install -g @storacha/cli
storacha login [email protected]
storacha space create my-spaceThen the library will use those credentials automatically.
Standard AWS credentials:
# Option 1: Environment variables
export S3_ACCESS_KEY_ID=your_key
export S3_SECRET_ACCESS_KEY=your_secret
# Option 2: AWS credentials file (~/.aws/credentials)
# Option 3: IAM role (if running on EC2/ECS)from py_s3_storacha import (
S3StorachaError, # Base exception
ConfigurationError, # Invalid configuration
MigrationError, # Migration failures
JSWrapperError # JavaScript execution errors
)
try:
result = await migrator.migrate(request)
except ConfigurationError as e:
print(f"Config error: {e}")
print(f"Context: {e.context}")
except MigrationError as e:
print(f"Migration failed: {e}")
print(f"Failed objects: {e.failed_objects}")
except S3StorachaError as e:
print(f"Error: {e}")The library automatically retries failed operations:
- Network errors: 3 retries with exponential backoff
- Transient failures: Automatic retry with backoff
- Configuration errors: No retry (fail fast)
try:
result = await migrator.migrate(request)
except MigrationError as e:
# Get error context
print(f"Operation: {e.operation}")
print(f"Source: {e.source_path}")
print(f"Destination: {e.destination_path}")
print(f"Objects processed: {e.objects_processed}")
# Get original error
if e.original_error:
print(f"Caused by: {e.original_error}")The library includes helper commands for setup and authentication:
# Install JavaScript dependencies
py-s3-storacha-setup
# Force reinstall
py-s3-storacha-setup --force
# Check installation status
py-s3-storacha-setup --check# Interactive setup
py-s3-storacha-auth --setup
# Setup with specific email
py-s3-storacha-auth --setup --email [email protected]
# Check authentication status
py-s3-storacha-auth --statusfrom py_s3_storacha import (
install_js_dependencies,
verify_installation,
StorachaAuthHelper
)
# Install JS dependencies
install_js_dependencies()
# Check installation
status = verify_installation()
print(f"Ready: {status['ready']}")
# Setup authentication
helper = StorachaAuthHelper()
helper.setup_authentication(email="[email protected]")Run the test suite:
# Install development dependencies
pip install -e ".[dev]"
# Run tests
pytest
# Run tests with coverage
pytest --cov=py_s3_storacha
# Run specific test categories
pytest tests/unit/ # Unit tests only
pytest tests/integration/ # Integration tests onlyWe welcome contributions! Please see our Contributing Guide for details.
- Fork the repository
- Clone your fork:
git clone https://github.com/yourusername/MM-python-s3-storacha.git - Create a virtual environment:
python -m venv venv - Activate it:
source venv/bin/activate(Linux/Mac) orvenv\Scripts\activate(Windows) - Install in development mode:
pip install -e ".[dev]" - Run tests:
pytest
We use several tools to maintain code quality:
# Format code
ruff format .
# Lint code
ruff check .
# Type checking
pyrightThis project is licensed under the MIT License - see the LICENSE file for details.
- Built on top of existing JavaScript S3 to Storacha implementation
- Uses hatchling for modern Python packaging
- Inspired by the need for seamless data migration between cloud storage providers
- Issues: GitHub Issues
- Discussions: GitHub Discussions
- Documentation: Full Documentation
- Support for additional cloud storage providers
- GUI interface for non-technical users
- Incremental sync capabilities
- Advanced filtering and transformation options
- Performance optimizations for large-scale migrations
Made with β€οΈ by the py-s3-storacha team