Use begin_of_sequence token in all sliding windows for correct model behaviour#40524
Use begin_of_sequence token in all sliding windows for correct model behaviour#40524TiAn5046 wants to merge 1 commit into
Conversation
…behaviour Testing with models employing bos tokens (eg Llama) resulted in much higher perplexity for all tokens but those in the first window. Reason was that if we tokenize only once, then only the first window will get the bos token. However for correct model behaviour, each forward pass/each sliding window needs to have a bos token as first token. Note: Since inserted bos token its the first token, we do not need to set its label to -100 due to causal model internal label shift. Note: if custom attention masks are used, always ensure bos token gets attention - otherwise unexpected behaviour (eg perplexity spikes for first three tokens experiencing lack of bos token)
|
cc @ArthurZucker for text models |
ArthurZucker
left a comment
There was a problem hiding this comment.
Hey!
Can you add some graph of the comparison / help update:
Running this with the stride length equal to the max input length is equivalent to the suboptimal, non-sliding-window
strategy we discussed above. The smaller the stride, the more context the model will have in making each prediction,
and the better the reported perplexity will typically be.
When we run the above withstride = 1024, i.e. no overlap, the resulting PPL is19.44, which is about the same
as the19.93reported in the GPT-2 paper. By usingstride = 512and thereby employing our striding window
strategy, this jumps down to16.44. This is not only a more favorable score, but is calculated in a way that is
closer to the true autoregressive decomposition of a sequence likelihood.
🤗
Testing with models employing bos tokens (eg Llama) resulted in much higher perplexity for all tokens but those in the first window. Reason was that if we tokenize only once, then only the first window will get the bos token. However for correct model behaviour, each forward pass/each sliding window needs to have a bos token as first token (otherwise perplexity values are significantly higher).
Note: Since inserted bos token is the first token, we do not need to set its label to -100 due to causal model internal label shift.
Note: if custom attention masks are used, always ensure bos token gets attention - otherwise unexpected behaviour (eg perplexity spikes for first three tokens experiencing lack of bos token)