get-all-crates

Download .crate files of all versions of all crates from crates.io.

Useful for things like noisy-clippy which need to analyze the source code of all crates.

I wrote this tool to saturate a 1000 Mbps connection. From scratch, it can finish downloading in under 20 minutes. It can also check the checksums of a directory of downloaded crates in 40 seconds on 64 cores. For a more fully featured tool, which some of this code is based on, but which is much slower in my experience, check out https://git.shipyard.rs/jstrong/registry-backup.

Usage

$ cargo install get-all-crates
$ git clone https://github.com/rust-lang/crates.io-index /path/to/index
$ get-all-crates --index /path/to/index --out /path/to/crates

Warning: as of 2022 this writes out more than 100 GB of crates.

The output directory structure is similar to how Cargo's registry index is structured:

/path/to/crates
 ├── 1
 │  └── m
 │     └── m-0.1.1.crate
 ├── 2
 │  └── nu
 │     └── nu-0.73.0.crate
 ├── 3
 │  └── s
 │     └── syn
 │        └── syn-1.0.107.crate
 └── se
    └── rd
       ├── serde
       │  └── serde-1.0.151.crate
       └── serde_json
          └── serde_json-1.0.91.crate

License

MIT license.