Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Commit abdc1be

Browse files
authored
Add a demo GIF and rewrite README in the same style as the other CLI tools (#2)
Badge row, nav links, GIF early, a full commands table with all 19 commands, usage examples with the cursor-export recipe and jq pipeline, global flags block, credentials section, exit codes, and development / releasing sections. The GIF was rendered with ascii-gif from docs/demo/archive.tape and shows three interactions: a NASA collection search as a table, an item summary, and a Wayback Machine availability check.
1 parent a32bd9b commit abdc1be

3 files changed

Lines changed: 159 additions & 90 deletions

File tree

README.md

Lines changed: 130 additions & 90 deletions
Original file line numberDiff line numberDiff line change
@@ -1,98 +1,115 @@
11
# archive
22

3-
A fast, friendly command line for the [Internet Archive](https://archive.org).
4-
One binary that searches millions of items, reads metadata, downloads and
5-
verifies files, uploads to your own items, reads view counts, and travels
6-
through the Wayback Machine.
3+
[![CI](https://github.com/tamnd/archive-cli/actions/workflows/ci.yml/badge.svg)](https://github.com/tamnd/archive-cli/actions/workflows/ci.yml)
4+
[![Release](https://img.shields.io/github/v/release/tamnd/archive-cli)](https://github.com/tamnd/archive-cli/releases/latest)
5+
[![Go Reference](https://pkg.go.dev/badge/github.com/tamnd/archive-cli.svg)](https://pkg.go.dev/github.com/tamnd/archive-cli)
6+
[![Go Report Card](https://goreportcard.com/badge/github.com/tamnd/archive-cli)](https://goreportcard.com/report/github.com/tamnd/archive-cli)
7+
[![License](https://img.shields.io/github/license/tamnd/archive-cli)](./LICENSE)
78

8-
```
9-
archive item nasa
10-
```
9+
A command line for the [Internet Archive](https://archive.org). `archive`
10+
searches millions of items, reads metadata, downloads and verifies files,
11+
travels through the Wayback Machine, and uploads to your own items. One
12+
pure-Go binary, no credentials needed for public data.
1113

12-
```
13-
field value
14-
identifier nasa
15-
title NASA
16-
mediatype collection
17-
files 9
18-
size 131.1 KB
19-
server ia801607.us.archive.org
20-
details https://archive.org/details/nasa
21-
```
14+
[Install](#install)[Commands](#commands)[Usage](#usage)[Credentials](#credentials)
2215

23-
Full documentation: [archive-cli.tamnd.com](https://archive-cli.tamnd.com).
16+
![archive searching the Internet Archive and querying the Wayback Machine](docs/static/demo.gif)
2417

25-
## Why
18+
It talks to the public Internet Archive APIs over HTTPS: the Metadata API, the
19+
Advanced Search (Solr) endpoint, the CDX Wayback server, and the S3-like IAS3
20+
upload interface. Every request is paced, retried on transient failures, and
21+
cached on disk. No login is needed for anything read-only.
2622

27-
Working with the Internet Archive usually means juggling the Metadata API, the
28-
Solr search endpoint, S3-style upload headers, and the Wayback CDX server by
29-
hand. archive puts all of it behind one tool with sensible defaults, real output
30-
formats, and pipelines that compose. It talks to the public data on
31-
`archive.org` over HTTPS, so there is nothing to sign up for; credentials are
32-
only needed to upload, delete, or read your task queue.
23+
`archive` is an independent tool. It is not affiliated with or endorsed by the
24+
Internet Archive.
3325

3426
## Install
3527

36-
```sh
28+
```bash
3729
go install github.com/tamnd/archive-cli/cmd/archive@latest
3830
```
3931

4032
Or grab a prebuilt binary, a Linux package (`deb`/`rpm`/`apk`), or a container
41-
image from the [releases page](https://github.com/tamnd/archive-cli/releases).
42-
The binary is pure Go with no runtime dependencies.
33+
image from the [releases](https://github.com/tamnd/archive-cli/releases):
34+
35+
```bash
36+
brew install tamnd/tap/archive
37+
docker run --rm ghcr.io/tamnd/archive:latest search 'collection:nasa' -n 5
38+
```
4339

44-
```sh
45-
brew install tamnd/tap/archive # macOS / Linux
46-
docker run --rm ghcr.io/tamnd/archive search nasa -n 5
40+
Shell completion is built in: `archive completion bash|zsh|fish|powershell`.
41+
42+
## Commands
43+
44+
| Command | Does |
45+
| --- | --- |
46+
| `archive search <query>` | search the Solr index; any Lucene query, `--all` for cursor-based export |
47+
| `archive item <identifier>` | a friendly summary of an item |
48+
| `archive metadata <identifier> [subpath]` | the raw Metadata API document, or one field |
49+
| `archive files <identifier>` | files in an item; `--format`, `--glob` to filter |
50+
| `archive download <identifier> [files...]` | download and md5-verify; `--workers`, `-d` dir |
51+
| `archive upload <identifier> <file...>` | upload into an item over IAS3; `--meta` |
52+
| `archive delete <identifier> <file...>` | delete files from an item over IAS3 |
53+
| `archive views <identifier...>` | view statistics for one or more items |
54+
| `archive tasks <identifier>` | catalog and derive task history of an item |
55+
| `archive wayback available <url>` | closest archived snapshot of a URL |
56+
| `archive wayback list <url>` | capture history from the CDX server |
57+
| `archive wayback get <url>` | fetch the content of a snapshot; `--text`, `-t` timestamp |
58+
| `archive wayback save <url>` | trigger a fresh Save Page Now capture |
59+
| `archive open <identifier\|url>` | open the details or Wayback URL in the browser |
60+
| `archive configure` | store IAS3 credentials |
61+
| `archive whoami` | show configured credentials |
62+
| `archive config` | show resolved configuration and data paths |
63+
| `archive cache path\|info\|clear` | inspect or clear the on-disk cache |
64+
| `archive version` | print version information |
65+
66+
Full reference and guides live at [archive-cli.tamnd.com](https://archive-cli.tamnd.com).
67+
68+
## Usage
69+
70+
```bash
71+
archive search 'collection:nasa' -n 10 # find items
72+
archive item nasa # item summary
73+
archive metadata nasa metadata/title # one metadata field
74+
archive files nasa --format JPEG -o url # file listing as URLs
75+
archive download nasa --format JPEG -d . # download and verify
76+
archive views nasa # view statistics
77+
archive wayback get https://example.com -t 2010 # a page as it was in 2010
4778
```
4879

49-
Build from source:
80+
Records come out as a table (the default on a terminal), JSON, JSONL, CSV, TSV,
81+
url, or raw:
5082

51-
```sh
52-
git clone https://github.com/tamnd/archive-cli
53-
cd archive-cli
54-
make build # produces ./bin/archive
83+
```bash
84+
archive search 'collection:nasa' --fields identifier,title,downloads -o table
85+
archive search 'collection:nasa' --fields identifier,downloads -o csv
86+
archive search 'collection:nasa' --fields identifier -o raw | xargs -n1 archive item
87+
archive files nasa --format JPEG -o url | head -20
88+
archive wayback list https://archive.org -n 50 -o jsonl | jq .timestamp
5589
```
5690

57-
## Quick start
91+
Export a large result set with the cursor-based Scraping API:
5892

59-
```sh
60-
archive search 'collection:nasa' -n 5 # find items
61-
archive item nasa # what an item is, at a glance
62-
archive metadata nasa metadata/title # a single metadata field
63-
archive files nasa --format JPEG -o url # a file listing, as URLs
64-
archive download nasa --format JPEG -d . # download and verify by md5
65-
archive views nasa # view counts
66-
archive wayback get example.com -t 2010 --text # a page as it was in 2010
93+
```bash
94+
archive search 'subject:jazz mediatype:audio' --all --fields identifier,title -o jsonl > jazz.jsonl
6795
```
6896

69-
## What you can do with it
70-
71-
- **Search.** Query the Advanced Search (Solr) index with any Lucene query,
72-
sort and project fields, and render the result as a table, JSONL, CSV, or just
73-
identifiers. Large sets export through the cursor-based Scraping API with
74-
`--all`.
75-
- **Inspect items.** Read the raw Metadata API document, a friendly summary, or
76-
a single field, and list files filtered by glob or format.
77-
- **Download and verify.** Pull whole items or selected files concurrently,
78-
resume partial downloads with HTTP range requests, and verify each file
79-
against its md5.
80-
- **Upload and manage.** Push files into your own items over the S3-like IAS3
81-
interface with metadata headers, and delete files.
82-
- **Travel the Wayback Machine.** Find the closest snapshot of a URL, list its
83-
capture history from the CDX server, fetch a snapshot as text, links, or raw
84-
bytes, and trigger a fresh capture with Save Page Now.
85-
86-
## Output formats
87-
88-
Every command renders through one output layer. Pick with `-o`: `table`, `json`,
89-
`jsonl`, `csv`, `tsv`, `url`, or `raw`. `auto` (the default) is a table on a
90-
terminal and JSONL in a pipe. `--fields` projects columns; `--template` applies
91-
a Go template per row.
92-
93-
```sh
94-
archive search 'collection:nasa' --fields identifier,downloads -o csv
95-
archive search 'collection:nasa' --fields identifier -o raw | xargs -n1 archive item
97+
### Global flags
98+
99+
```
100+
-o, --output table|json|jsonl|csv|tsv|url|raw (auto: table on a TTY, jsonl when piped)
101+
--fields comma-separated columns to include
102+
--no-header omit the header row in table/csv/tsv
103+
--template Go text/template applied per record
104+
-n, --limit max records (0 = unlimited)
105+
-q, --quiet suppress progress output
106+
--color auto|always|never
107+
--rate min spacing between requests (default 250ms)
108+
--timeout per-request timeout (default 2m)
109+
--retries retry attempts on 429/5xx (default 5)
110+
-j, --workers concurrency for downloads (default 8)
111+
--no-cache bypass the on-disk cache
112+
--dry-run print actions without performing them
96113
```
97114

98115
## Credentials
@@ -101,34 +118,57 @@ Reading public data needs no account. To upload, delete, or read a private task
101118
queue, get an IAS3 key pair from
102119
[archive.org/account/s3.php](https://archive.org/account/s3.php) and store it:
103120

104-
```sh
105-
archive configure # prompts, writes ~/.config/archive/credentials (0600)
106-
archive whoami # show what is configured
121+
```bash
122+
archive configure # prompts for access and secret keys, writes ~/.config/archive/credentials
123+
archive whoami # verify what is configured
107124
```
108125

109-
Credentials resolve from `--access`/`--secret`, then `ARCHIVE_ACCESS_KEY` /
110-
`ARCHIVE_SECRET_KEY` (or `IA_*`), then the credentials file.
126+
Credentials resolve in order: `--access`/`--secret` flags, then
127+
`ARCHIVE_ACCESS_KEY`/`ARCHIVE_SECRET_KEY` environment variables (or `IA_*`
128+
aliases), then the credentials file.
111129

112130
## Exit codes
113131

114-
`0` success, `1` generic error, `2` usage error, `3` no results, `4`
115-
authentication required/failed, `5` not found.
132+
```
133+
0 success
134+
1 error
135+
2 usage error
136+
3 no results
137+
4 authentication required or failed
138+
5 not found
139+
```
116140

117141
## Development
118142

119-
```sh
120-
make build # build ./bin/archive
121-
make test # go test ./...
122-
make vet # go vet ./...
123-
make fmt # gofmt -w -s .
143+
```
144+
cmd/archive/ thin main entry point
145+
cli/ cobra commands and output rendering
146+
ia/ HTTP client, API calls, and models
147+
docs/ documentation site (Hugo, tago-doks theme)
148+
```
149+
150+
```bash
151+
make build # ./bin/archive
152+
make test # go test ./...
153+
make vet # go vet ./...
154+
make fmt # gofmt -w -s .
155+
```
156+
157+
Requires Go 1.23+.
158+
159+
## Releasing
160+
161+
Push a version tag and GitHub Actions runs GoReleaser, which builds archives,
162+
Linux packages, a multi-arch GHCR image, checksums, an SBOM, a cosign
163+
signature, and Homebrew and Scoop entries:
164+
165+
```bash
166+
git tag -a v0.2.0 -m "v0.2.0"
167+
git push --tags
124168
```
125169

126-
CI runs build, test (with the race detector) on Linux and macOS, gofmt, vet,
127-
golangci-lint, govulncheck, and a go.mod tidiness check. Releases are cut by
128-
pushing a `vX.Y.Z` tag, which GoReleaser turns into archives, Linux packages, a
129-
multi-arch GHCR image, checksums, an SBOM, a cosign signature, and Homebrew and
130-
Scoop entries.
170+
The image tag carries no `v` prefix (`ghcr.io/tamnd/archive:0.2.0`).
131171

132172
## License
133173

134-
[Apache-2.0](LICENSE).
174+
Apache-2.0. See [LICENSE](LICENSE).

docs/demo/archive.tape

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
# Demo tape for archive. Rendered with ascii-gif (github.com/tamnd/ascii-gif),
2+
# which supplies the window chrome and theme; this file is just the action.
3+
#
4+
# ascii-gif render docs/demo/archive.tape -o docs/static/demo.gif
5+
#
6+
# archive must be on PATH inside the recording shell.
7+
8+
Hide
9+
Type "export PS1='$ ' PATH=/Users/apple/bin:$PATH"
10+
Enter
11+
Type "clear"
12+
Enter
13+
Show
14+
15+
Sleep 500ms
16+
Type "archive search 'collection:nasa mediatype:movies' -n 5 --fields identifier,title,downloads"
17+
Sleep 800ms
18+
Enter
19+
Sleep 3s
20+
21+
Type "archive item nasa -o table"
22+
Sleep 800ms
23+
Enter
24+
Sleep 2.5s
25+
26+
Type "archive wayback available https://google.com -o table"
27+
Sleep 800ms
28+
Enter
29+
Sleep 2.5s

docs/static/demo.gif

524 KB
Loading

0 commit comments

Comments
 (0)