tokenicer
A (nicer) tokenizer you want to use for model `inference` and `training`: with all known peventable `gotchas` normalized or auto-fixed.
tokenicer has been downloaded 223,178 times in total on PyPI, including 25,365 in the last 30 days. The latest version is 0.0.5, released Sep 4, 2025.
Version0.0.5
Downloads
223.18k
License—
AuthorModelCloud
UpdatedSep 4, 2025
Downloads
Weekly, last 90d.
Includes CI traffic.
VersionsTotal0.*
Range
View
Granularity
Group by
CI traffic
Stack: OffCI: Included2 / 15 series
Selected total148.83k
223.2kAll-time
25.4kLast 30 days
1.1kLast 24 h
<0.01/sPer second
Sponsored
Sponsorships keep pepy free to read
Version distribution
Share of downloads by released version. Computed over the last quarter.
- 0146.5%
0.0.13
34.3k downloadsDownloads34.3k46.5% - 0231.0%
0.0.4
22.8k downloadsDownloads22.8k31.0% - 0313.2%
0.0.12
9.7k downloadsDownloads9.7k13.2% - 043.9%
0.0.11
2.9k downloadsDownloads2.9k3.9% - 052.0%
0.0.8
1.4k downloadsDownloads1.4k2.0% - 060.9%
0.0.5
637 downloadsDownloads6370.9% - 070.8%
0.0.6
557 downloadsDownloads5570.8% - 080.3%
0.0.9
242 downloadsDownloads2420.3% - 091.5%
Other
1.1k downloadsDownloads1.1k1.5%
Guess the next day
Thirteen recent days of tokenicer downloads. Drag the green handle on the right to guess where day fourteen lands.