11# archive
22
3- A fast, friendly command line for the [ Internet Archive] ( https://archive.org ) .
4- One binary that searches millions of items, reads metadata, downloads and
5- verifies files, uploads to your own items, reads view counts, and travels
6- through the Wayback Machine.
3+ [ ![ CI] ( https://github.com/tamnd/archive-cli/actions/workflows/ci.yml/badge.svg )] ( https://github.com/tamnd/archive-cli/actions/workflows/ci.yml )
4+ [ ![ Release] ( https://img.shields.io/github/v/release/tamnd/archive-cli )] ( https://github.com/tamnd/archive-cli/releases/latest )
5+ [ ![ Go Reference] ( https://pkg.go.dev/badge/github.com/tamnd/archive-cli.svg )] ( https://pkg.go.dev/github.com/tamnd/archive-cli )
6+ [ ![ Go Report Card] ( https://goreportcard.com/badge/github.com/tamnd/archive-cli )] ( https://goreportcard.com/report/github.com/tamnd/archive-cli )
7+ [ ![ License] ( https://img.shields.io/github/license/tamnd/archive-cli )] ( ./LICENSE )
78
8- ```
9- archive item nasa
10- ```
9+ A command line for the [ Internet Archive] ( https://archive.org ) . ` archive `
10+ searches millions of items, reads metadata, downloads and verifies files,
11+ travels through the Wayback Machine, and uploads to your own items. One
12+ pure-Go binary, no credentials needed for public data.
1113
12- ```
13- field value
14- identifier nasa
15- title NASA
16- mediatype collection
17- files 9
18- size 131.1 KB
19- server ia801607.us.archive.org
20- details https://archive.org/details/nasa
21- ```
14+ [ Install] ( #install ) • [ Commands] ( #commands ) • [ Usage] ( #usage ) • [ Credentials] ( #credentials )
2215
23- Full documentation: [ archive-cli.tamnd.com ] ( https://archive-cli.tamnd.com ) .
16+ ! [ archive searching the Internet Archive and querying the Wayback Machine ] ( docs/static/demo.gif )
2417
25- ## Why
18+ It talks to the public Internet Archive APIs over HTTPS: the Metadata API, the
19+ Advanced Search (Solr) endpoint, the CDX Wayback server, and the S3-like IAS3
20+ upload interface. Every request is paced, retried on transient failures, and
21+ cached on disk. No login is needed for anything read-only.
2622
27- Working with the Internet Archive usually means juggling the Metadata API, the
28- Solr search endpoint, S3-style upload headers, and the Wayback CDX server by
29- hand. archive puts all of it behind one tool with sensible defaults, real output
30- formats, and pipelines that compose. It talks to the public data on
31- ` archive.org ` over HTTPS, so there is nothing to sign up for; credentials are
32- only needed to upload, delete, or read your task queue.
23+ ` archive ` is an independent tool. It is not affiliated with or endorsed by the
24+ Internet Archive.
3325
3426## Install
3527
36- ``` sh
28+ ``` bash
3729go install github.com/tamnd/archive-cli/cmd/archive@latest
3830```
3931
4032Or grab a prebuilt binary, a Linux package (` deb ` /` rpm ` /` apk ` ), or a container
41- image from the [ releases page] ( https://github.com/tamnd/archive-cli/releases ) .
42- The binary is pure Go with no runtime dependencies.
33+ image from the [ releases] ( https://github.com/tamnd/archive-cli/releases ) :
34+
35+ ``` bash
36+ brew install tamnd/tap/archive
37+ docker run --rm ghcr.io/tamnd/archive:latest search ' collection:nasa' -n 5
38+ ```
4339
44- ``` sh
45- brew install tamnd/tap/archive # macOS / Linux
46- docker run --rm ghcr.io/tamnd/archive search nasa -n 5
40+ Shell completion is built in: ` archive completion bash|zsh|fish|powershell ` .
41+
42+ ## Commands
43+
44+ | Command | Does |
45+ | --- | --- |
46+ | ` archive search <query> ` | search the Solr index; any Lucene query, ` --all ` for cursor-based export |
47+ | ` archive item <identifier> ` | a friendly summary of an item |
48+ | ` archive metadata <identifier> [subpath] ` | the raw Metadata API document, or one field |
49+ | ` archive files <identifier> ` | files in an item; ` --format ` , ` --glob ` to filter |
50+ | ` archive download <identifier> [files...] ` | download and md5-verify; ` --workers ` , ` -d ` dir |
51+ | ` archive upload <identifier> <file...> ` | upload into an item over IAS3; ` --meta ` |
52+ | ` archive delete <identifier> <file...> ` | delete files from an item over IAS3 |
53+ | ` archive views <identifier...> ` | view statistics for one or more items |
54+ | ` archive tasks <identifier> ` | catalog and derive task history of an item |
55+ | ` archive wayback available <url> ` | closest archived snapshot of a URL |
56+ | ` archive wayback list <url> ` | capture history from the CDX server |
57+ | ` archive wayback get <url> ` | fetch the content of a snapshot; ` --text ` , ` -t ` timestamp |
58+ | ` archive wayback save <url> ` | trigger a fresh Save Page Now capture |
59+ | ` archive open <identifier\|url> ` | open the details or Wayback URL in the browser |
60+ | ` archive configure ` | store IAS3 credentials |
61+ | ` archive whoami ` | show configured credentials |
62+ | ` archive config ` | show resolved configuration and data paths |
63+ | ` archive cache path\|info\|clear ` | inspect or clear the on-disk cache |
64+ | ` archive version ` | print version information |
65+
66+ Full reference and guides live at [ archive-cli.tamnd.com] ( https://archive-cli.tamnd.com ) .
67+
68+ ## Usage
69+
70+ ``` bash
71+ archive search ' collection:nasa' -n 10 # find items
72+ archive item nasa # item summary
73+ archive metadata nasa metadata/title # one metadata field
74+ archive files nasa --format JPEG -o url # file listing as URLs
75+ archive download nasa --format JPEG -d . # download and verify
76+ archive views nasa # view statistics
77+ archive wayback get https://example.com -t 2010 # a page as it was in 2010
4778```
4879
49- Build from source:
80+ Records come out as a table (the default on a terminal), JSON, JSONL, CSV, TSV,
81+ url, or raw:
5082
51- ``` sh
52- git clone https://github.com/tamnd/archive-cli
53- cd archive-cli
54- make build # produces ./bin/archive
83+ ``` bash
84+ archive search ' collection:nasa' --fields identifier,title,downloads -o table
85+ archive search ' collection:nasa' --fields identifier,downloads -o csv
86+ archive search ' collection:nasa' --fields identifier -o raw | xargs -n1 archive item
87+ archive files nasa --format JPEG -o url | head -20
88+ archive wayback list https://archive.org -n 50 -o jsonl | jq .timestamp
5589```
5690
57- ## Quick start
91+ Export a large result set with the cursor-based Scraping API:
5892
59- ``` sh
60- archive search ' collection:nasa' -n 5 # find items
61- archive item nasa # what an item is, at a glance
62- archive metadata nasa metadata/title # a single metadata field
63- archive files nasa --format JPEG -o url # a file listing, as URLs
64- archive download nasa --format JPEG -d . # download and verify by md5
65- archive views nasa # view counts
66- archive wayback get example.com -t 2010 --text # a page as it was in 2010
93+ ``` bash
94+ archive search ' subject:jazz mediatype:audio' --all --fields identifier,title -o jsonl > jazz.jsonl
6795```
6896
69- ## What you can do with it
70-
71- - ** Search.** Query the Advanced Search (Solr) index with any Lucene query,
72- sort and project fields, and render the result as a table, JSONL, CSV, or just
73- identifiers. Large sets export through the cursor-based Scraping API with
74- ` --all ` .
75- - ** Inspect items.** Read the raw Metadata API document, a friendly summary, or
76- a single field, and list files filtered by glob or format.
77- - ** Download and verify.** Pull whole items or selected files concurrently,
78- resume partial downloads with HTTP range requests, and verify each file
79- against its md5.
80- - ** Upload and manage.** Push files into your own items over the S3-like IAS3
81- interface with metadata headers, and delete files.
82- - ** Travel the Wayback Machine.** Find the closest snapshot of a URL, list its
83- capture history from the CDX server, fetch a snapshot as text, links, or raw
84- bytes, and trigger a fresh capture with Save Page Now.
85-
86- ## Output formats
87-
88- Every command renders through one output layer. Pick with ` -o ` : ` table ` , ` json ` ,
89- ` jsonl ` , ` csv ` , ` tsv ` , ` url ` , or ` raw ` . ` auto ` (the default) is a table on a
90- terminal and JSONL in a pipe. ` --fields ` projects columns; ` --template ` applies
91- a Go template per row.
92-
93- ``` sh
94- archive search ' collection:nasa' --fields identifier,downloads -o csv
95- archive search ' collection:nasa' --fields identifier -o raw | xargs -n1 archive item
97+ ### Global flags
98+
99+ ```
100+ -o, --output table|json|jsonl|csv|tsv|url|raw (auto: table on a TTY, jsonl when piped)
101+ --fields comma-separated columns to include
102+ --no-header omit the header row in table/csv/tsv
103+ --template Go text/template applied per record
104+ -n, --limit max records (0 = unlimited)
105+ -q, --quiet suppress progress output
106+ --color auto|always|never
107+ --rate min spacing between requests (default 250ms)
108+ --timeout per-request timeout (default 2m)
109+ --retries retry attempts on 429/5xx (default 5)
110+ -j, --workers concurrency for downloads (default 8)
111+ --no-cache bypass the on-disk cache
112+ --dry-run print actions without performing them
96113```
97114
98115## Credentials
@@ -101,34 +118,57 @@ Reading public data needs no account. To upload, delete, or read a private task
101118queue, get an IAS3 key pair from
102119[ archive.org/account/s3.php] ( https://archive.org/account/s3.php ) and store it:
103120
104- ``` sh
105- archive configure # prompts , writes ~/.config/archive/credentials (0600)
106- archive whoami # show what is configured
121+ ``` bash
122+ archive configure # prompts for access and secret keys , writes ~/.config/archive/credentials
123+ archive whoami # verify what is configured
107124```
108125
109- Credentials resolve from ` --access ` /` --secret ` , then ` ARCHIVE_ACCESS_KEY ` /
110- ` ARCHIVE_SECRET_KEY ` (or ` IA_* ` ), then the credentials file.
126+ Credentials resolve in order: ` --access ` /` --secret ` flags, then
127+ ` ARCHIVE_ACCESS_KEY ` /` ARCHIVE_SECRET_KEY ` environment variables (or ` IA_* `
128+ aliases), then the credentials file.
111129
112130## Exit codes
113131
114- ` 0 ` success, ` 1 ` generic error, ` 2 ` usage error, ` 3 ` no results, ` 4 `
115- authentication required/failed, ` 5 ` not found.
132+ ```
133+ 0 success
134+ 1 error
135+ 2 usage error
136+ 3 no results
137+ 4 authentication required or failed
138+ 5 not found
139+ ```
116140
117141## Development
118142
119- ``` sh
120- make build # build ./bin/archive
121- make test # go test ./...
122- make vet # go vet ./...
123- make fmt # gofmt -w -s .
143+ ```
144+ cmd/archive/ thin main entry point
145+ cli/ cobra commands and output rendering
146+ ia/ HTTP client, API calls, and models
147+ docs/ documentation site (Hugo, tago-doks theme)
148+ ```
149+
150+ ``` bash
151+ make build # ./bin/archive
152+ make test # go test ./...
153+ make vet # go vet ./...
154+ make fmt # gofmt -w -s .
155+ ```
156+
157+ Requires Go 1.23+.
158+
159+ ## Releasing
160+
161+ Push a version tag and GitHub Actions runs GoReleaser, which builds archives,
162+ Linux packages, a multi-arch GHCR image, checksums, an SBOM, a cosign
163+ signature, and Homebrew and Scoop entries:
164+
165+ ``` bash
166+ git tag -a v0.2.0 -m " v0.2.0"
167+ git push --tags
124168```
125169
126- CI runs build, test (with the race detector) on Linux and macOS, gofmt, vet,
127- golangci-lint, govulncheck, and a go.mod tidiness check. Releases are cut by
128- pushing a ` vX.Y.Z ` tag, which GoReleaser turns into archives, Linux packages, a
129- multi-arch GHCR image, checksums, an SBOM, a cosign signature, and Homebrew and
130- Scoop entries.
170+ The image tag carries no ` v ` prefix (` ghcr.io/tamnd/archive:0.2.0 ` ).
131171
132172## License
133173
134- [ Apache-2.0] ( LICENSE ) .
174+ Apache-2.0. See [ LICENSE ] ( LICENSE ) .
0 commit comments