A high-performance Rust CLI for moving large files (>5GB) between Amazon S3 buckets. Built for speed, cost-efficiency, and reliability.
I wrote this application due to a lack of tools and simplicity for moving large files between S3 buckets. Currently, AWS's simple copy object does not support objects larger than 5 GB in a single operation (see official documentation). Another solution available from AWS Support is to use a Python Lambda with S3 batch operations. Unfortunately, it takes hours and often fails after the 15-minute timeout for large files. My goal was simple: I wanted a clean and simple command-line interface (CLI) that could do the job (VERY) fast, clean, and with a lot of tuning, options, and customization. It is satisfying to see that the transfer of a single 100 GB file takes less than 30 seconds with a simple command line, from an EC2 instance in my VPC.
- 🚀 Performance: High-concurrency multipart engine with adaptive tuning.
- 💰 Cost Aware: Real-time cost estimation and request optimization.
- 🛠️ Flexible: Support for all storage classes, KMS encryption, and property preservation.
- ✅ Reliable: Automatic cleanup on failure and checksum-based integrity verification.
- 🚄 Auto-Mode: Intelligent optimization of part sizes and thread counts.
# Basic copy
./s3_largecopy -s source-bucket -k data.iso -b dest-bucket -t data.isoDetailed documentation is available in the docs/ directory:
- Installation - Prerequisites and build instructions (incl. static binaries).
- Usage & Examples - Basic commands, advanced features, and CLI reference.
- Architecture - How the engine works, diagrams, and internal modules.
- Auto Mode - Deep dive into adaptive tuning and profiles.
- Cost Analysis - Comprehensive guide to S3 transfer costs and estimation logic.
- Permissions - IAM policy requirements and security configuration.
- Troubleshooting - Performance tips and common error resolutions.
- Changelog - Detailed history of changes and releases.
Created and maintained by Bart Leboeuf.
This project is licensed under the MIT License.
Contributions are welcome! See the Future Features for roadmap ideas.