New NLTK build in rust #3765
hariomlohardev
started this conversation in
Ideas
New NLTK build in rust
#3765
Replies: 2 comments
|
It would be interesting to know how much the rust language contributes to the speedup, vs. how much is due to a quicker tokenizer. |
0 replies
UPDATEThere are some errors in implemnetation but knwo fixed and verifiedThis are new benchmark things
|
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
hey i am building the nltk in rust soon be availabel for open-sourse
bellow are the speed details
🚀 Speed — every tokenizer, microsecond per call
Benchmark: median of 7 runs × warmup (LazyLock init excluded), same
FUNCTION_PAIRSas the correctness matrix — one source of truth for what is compared. Python 3.12 / manylinux x86_64,--release(opt-level=3,lto=fat,codegen-units=1,strip).nltk(µs)ported_lib(µs)sent_tokenize████████████████████████████word_tokenize██████████████████████regexp_tokenize███████████████casual_tokenize██████████████sexpr_tokenize█████████toktok_tokenize███████is_cjk██████mwe_tokenize█████string_span_tokenize████regexp_span_tokenize███xml_escape██align_tokens██spans_to_relative█xml_unescape█example.add·Read it this way: at 10k calls,
nltkspends ~11.9 s onword_tokenizewhereported_libspends ~0.27 s — you get your seconds back. The biggest wins are exactly where real pipelines hurt: the two functions most callers actually use (word_/sent_tokenize) and the regex-heavy social tokenizer (casual).All reactions