* Revert "use `int32_t` for call to tokenize (#4738)"

This reverts commit 763bb65dbb88004cd046c8acc0c8e889816e1828.

* Revert "vocab only"

This reverts commit bf54c845e9ea63ec58762a991dcea78d2c934b47.

* Revert "use ffi for tokenizing/detokenizing"

This reverts commit 26a00a04108f6cae625802e69faa4b48480bc208.