@abetlen, could you review when you have a moment? Thank you!

Hey @Yard1 sorry to take so long on this reply. The issue I see with this at the moment is that the stop token is still being generated and appended to the eval_logits and eval_tokens internally. This will probably cause some kind of issue in generation, what you really want is a min_tokens inside of generate or something that essentially ignores / sets the eos token probability to 0 until a certain number of tokens are generated.

Also, it should probably be noted that fewer than min_tokens may be returned if e.g. another stop criteria like a stop sequence is encountered.

The proposed change would be:

• Add ignore_eos to sample
• Move min_tokens check to generate and where the returned tokens is less than min_tokens set ignore_eos=True

Got it, thanks! Let me take a look at this.

Hmm, now that I think about it, this can be easily implemented through a LogitsProcessor. Therefore, we don't need a separate argument (and by extension, this PR). Let me know if that makes sense - we can close this PR then.

any updates on this?

@Yard1 we've added the parameter to the server and it's available in the python API via a new MinTokensLogitProcessor