v0.3.1

• Fixed go.mod/go.sum.

v0.3.0

• Bumped the version of GoMLX in tests and documentation.
• Bumped version of dependencies: including github.com/daulet/tokenizers, which requires a fresh download of the
  corresponding c++ library libtokenizers.a.

v0.3.0

• Bumped the version of GoMLX in tests and documentation.
• Bumped version of dependencies: including github.com/daulet/tokenizers, which requires a fresh download of the
  corresponding c++ library libtokenizers.a.

• Fixed file truncation issues during download.

• Forcefully refresh (download) the revision's hash at least once before using.

• Add Windows support by moving to cross platform flock: see PR #6, thanks to @mrmichaeladavis

• If verbosity is 0, it won't print progress.
• Added support for custom end-points. Default being "https://huggingface.co" or the environment variable
  $HF_ENDPOINT if defined.

• Fixed URL resolution of non-model repos.
• Fixed sentencepiece Tokenizer and tokenizer API string methods (using enumer).
• Added dataset example.
• Added usage with Rust tokenizer.
• Improved README.md
• Added SentencePiece proto support -- to be used in future conversion of SentencePiece models.
• Improved documentation.

v0.1.0

• package hub: inspect and download files from arbitrary repos. Very functional.
• package tokenizers:
  • Interfaces, types and constants.
  • Gemma tokenizer implementation.
  • Not any other tokenizer implemented yet.
• Examples in README.md.