I was inspired by the way simpleaichat works and I wanted to try running it on my own hardware with my own API server. This code relies on the logit_bias parameter to work, so I needed to add it.

The way OpenAI's API works is that the logit_bias is a mapping from input_ids (as a string) to a bias (as a float). Because of the need to use IDs instead of just the token strings themselves, then it means this parameter and its functionality is tied to ChatGPT's tokenizer. This is a problem for other models, obviously.

So in addition to implementing the API as OpenAI intended, I also added another parameter that changes the meaning of logit_bias to be a mapping from token strings (as a string) to a bias (as a float).

I tested this with the following pair of requests:

$ curl --verbose --location -H 'Content-Type: application/json' http://localhost:7777/v1/completions --data @<(exec jq --null-input --arg prompt "USER: What is the result of 20+4? ASSISTANT:" 'reduce ( range(0;10) | { (.|tostring): 100.0 } ) as $x ({}; . + $x) | . as $logit_bias | { prompt: $prompt, logit_bias: $logit_bias, logit_bias_type: "tokens", max_tokens: 1 }')

$ curl --verbose --location -H 'Content-Type: application/json' http://localhost:7777/v1/chat/completions --data @<(exec jq --null-input --arg prompt "What is the result of 3+5?" 'reduce ( range(0;10) | { (.|tostring): 100.0 } ) as $x ({}; . + $x) | . as $logit_bias | { messages: [{ "role": "user", "content": $prompt }], logit_bias: $logit_bias, logit_bias_type: "tokens", max_tokens: 1 }')

The jq commands correspond to the following JSON objects.

{
  "prompt": "USER: What is the result of 20+4? ASSISTANT:",
  "logit_bias": {
    "0": 100,
    "1": 100,
    "2": 100,
    "3": 100,
    "4": 100,
    "5": 100,
    "6": 100,
    "7": 100,
    "8": 100,
    "9": 100
  },
  "logit_bias_type": "tokens",
  "max_tokens": 1
}

{
  "messages": [
    {
      "role": "user",
      "content": "What is the result of 3+5?"
    }
  ],
  "logit_bias": {
    "0": 100,
    "1": 100,
    "2": 100,
    "3": 100,
    "4": 100,
    "5": 100,
    "6": 100,
    "7": 100,
    "8": 100,
    "9": 100
  },
  "logit_bias_type": "tokens",
  "max_tokens": 1
}