• Supports streaming and no streaming
• Tested on local project
• I used the same code that generated schema for tool calls in functionary chat handler, rewrote almost everything else as they completely changed the format.
• Built-in tokenizer had problems tokenizing special tokens like <|content|> and <|stop|> so I had to use transformers package to get an AutoTokenizer that works with this model and just plug it with llama.generate.
  • As a side effect of this, sentencepiece is now a requirement.
• Updated readme to reflect support for the newest model the I tested as of now.
• Chat format is set as functionary2 to retain compatibility with any code that uses the old formatter with older models.

Here are samples of the new chat format and old chat format.

Please note that current functionary handler didn't work with functionary model v1.4.