add Llama to mapping names in tokenization_auto.py#43911
Conversation
Without this like `AutoTokenizer.from_pretrained(...)` does not create LlamaTokenizer object.
|
cc @itazap you mentioned this mapping might not be necessary? I'm not sure why llama is missing. |
|
Fixes tokenizer from this repo. Currently instantiated as generic |
|
[For maintainers] Suggested jobs to run (before merge) run-slow: auto |
ArthurZucker
left a comment
There was a problem hiding this comment.
Hey! sorry this does not make sense: you need either update the tokenizer.json on the hub, or the tokenizer_config.json needs the tokenizer_class to be valid.
This IS a breaking change closer to be a fix. What might be wrong is the conversion as well. as the tokenizer return is just converted from the tokenizer.model.
What is a valid tokenizer class in v5 and why this class in |
|
Sorry my bad, in this case we should indeed follow the class. #44127 fixes it! |
What does this PR do?
Without this like
AutoTokenizer.from_pretrained(...)does not createLlamaTokenizerobject.Before submitting
Pull Request section?
to it if that's the case.
documentation guidelines, and
here are tips on formatting docstrings. - Not applicable
Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.