🔢 Tokenizer — Universal AI Token Counter

Fast, accurate, 100% client‑side token counting for GPT, Llama, Mistral, Qwen, DeepSeek, Phi, and more — all in your browser. No API keys. No backend. Just drop text or files and get counts, context usage, and optional cost estimates instantly.

🔗 Live Demo: https://nepomuceno.github.io/tokenizer/

_{Deploys automatically on push to main.}

1. User pastes text or drops files.
2. Parsers extract plain text (PDF via pdfjs-dist, DOCX via mammoth, Markdown via remark, raw readers for TXT/CSV).
3. Text is chunked (100–500KiB) and sent to a Web Worker.
4. The worker loads the appropriate tokenizer adapter:
   • @dqbd/tiktoken for OpenAI‑style models (WASM)
   • @huggingface/tokenizers JSON for Llama/Mistral/Qwen/Phi/DeepSeek
5. The adapter counts tokens and streams progress back.
6. UI displays per‑file stats, model context usage, optional price math, and token preview.

All processing stays in the browser. Nothing is uploaded.

Models are defined centrally in src/modelRegistry.ts with: key, adapter type, resource (internal tokenizer name or JSON path), context window, and optional pricing reference. This keeps logic declarative and avoids hard‑coding throughout the UI.

Prereqs: Bun (https://bun.sh) installed.

git clone https://github.com/Nepomuceno/tokenizer.git
cd tokenizer
bun install
bun run dev

Build:

bun run build

Preview production build:

bun run preview

Run tests:

bun run test

Lint:

bun run lint

Minimal, focused Vitest tests cover tokenizer adapters and integration boundaries. When adding a new model or parser:

1. Add adapter tests (src/test/*).
2. Add a small fixture (avoid large binaries in repo).
3. Ensure deterministic count vs. known sample.

React UI (components)  --->  Tokenizer Factory  --->  Adapter (tiktoken | hf)  --->  WASM / JSON tokenizer data
        |                        ^                        
        |                        |                        
File Parsers (pdf/markdown/docx/text) ----> Web Worker (tokenize.worker.ts) <---- Model Registry (ctx + meta)

Key principles:

• Keep UI pure & declarative.
• Lazy load heavyweight assets (WASM / tokenizer JSON).
• Use transferable data (ArrayBuffer) when scaling.
• Avoid blocking the main thread.

1. Drop tokenizer JSON in public/tokenizers/ (if HF style).
2. Add entry to modelRegistry.ts with key + ctx size.
3. (Optional) Add pricing metadata.
4. Write/extend adapter tests.
5. Build & verify counts.

The app is deployed under a subpath (https://<user>.github.io/tokenizer/). To keep static assets resolving correctly:

• vite.config.ts uses base: './' so built asset references are relative.
• All favicon / manifest links in index.html use ./ prefixes (no leading /).
• public/manifest.json sets icons[].src, start_url, and scope to ./ forms.
• The tiktoken adapter builds its local encodings path from import.meta.env.BASE_URL so it fetches ./encodings/*.tiktoken instead of hitting the domain root (/encodings), avoiding 404 + CORS when hosted in a subfolder.
• Avoid introducing absolute leading-slash asset paths unless you change deployment to the domain root.

If you fork and deploy at another subpath, no changes are needed; if you deploy at root, everything still works because relative paths resolve there as well.

Contributions welcome! See CONTRIBUTING.md for detailed setup, guidelines, and PR checklist.

Please follow conventional commits where practical (e.g., feat: add qwen tokenizer).

Report vulnerabilities privately via GitHub Security Advisories or the email listed in SECURITY.md. Avoid opening public issues for sensitive disclosures.

If this project saves you time, you can buy me a coffee:

https://buymeacoffee.com/gabrielbici

Sharing the repo also helps a ton! 💙

Licensed under the MIT License.

Inspired by community token counters and upstream tokenizer libraries. Thanks to:

• @dqbd/tiktoken
• HuggingFace tokenizers
• Open‑source parser ecosystems (pdfjs, remark, mammoth)

• Dark mode toggle
• PWA / offline caching
• Custom user‑uploaded tokenizer JSON
• Cost presets per vendor
• CSV Export of batch results

_{Have an idea? Open a discussion or PR!}