What does this PR do?

In Whisper, model.generate, the function _retrieve_segment forgets to add back the time offset for the segment which was triggered from the seeking mechanism. Then this whole chunk (concatenated segments) with a false timestamp in its last segment is later processed in def _decode_asr in tokenization_whisper.py.

Inside _decode_asr depends on the correct last_timestamp of a chunk to confirm the start timestamp of the first chunk (Reference code), thus it relies on the robust concatenated segments from _retrieve_segment.

Final output is corrupted and misaligned with the original OpenAI's Whisper model output. The reproduction code is below.

This PR fixes the function _retrieve_segment in class WhisperGenerationMixin. In the case when a chunk only contains a sentence.

This bug is well hidden due to:

1. Whisper applies an initial timestamp constraint between 0.0 and 1.0 second as the default. Whisper Paper: “Finally, to avoid a failure mode where the model ignores the first few words in the input, we constrained the initial timestamp token to be between 0.0 and 1.0 second.”.
2. Despite of 1., Whisper is still prone to predict the first timestamp as <|0.00|> with the highest logit, even when the input audio is completely silent at the beginning. Especially in the less capable model (small and tiny). As long as the first timestamp is <|0.00|>, def _decode_asr can function properly by luck.
3. retrieve_segment , does add back the time offset set in the case when a chunk contains multiple sentences, but not when it only contains one sentence.

Reproduction code on Huggingface Whisper

Input audio is the 4th testing audio hf-internal-testing/librispeech_asr_dummy with 6s of silence added in the beginning. sample_4_6s_silence.flac - GoogleDrive

import os
import torch
import numpy as np
from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor, pipeline
from transformers import GenerationConfig


device = 'cuda' if torch.cuda.is_available() else 'cpu'
torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32

model_id = "openai/whisper-large-v3"
model = AutoModelForSpeechSeq2Seq.from_pretrained(model_id, dtype=torch_dtype, use_safetensors=True).to(device)
processor = AutoProcessor.from_pretrained(model_id)

audio_path = 'sample_4_6s_silence.flac'

# Enable initial timestamp constraint
# Default value is 1 second = 50*0.02, where 0.02 is the time precision
max_initial_timestamp_index = 50 

# Disable initial timestamp constraint
# max_initial_timestamp_index = None

generation_kwargs = {
        "task": "transcribe",
        "language": 'en',
        "max_initial_timestamp_index": max_initial_timestamp_index,
    }

pipe = pipeline(
        "automatic-speech-recognition",
        model=model,
        tokenizer=processor.tokenizer,
        feature_extractor=processor.feature_extractor,
        dtype=torch_dtype,
        device=device,
        batch_size=1,
    )

output = pipe(
            audio_path,
            chunk_length_s=30,
            generate_kwargs=generation_kwargs,
            return_timestamps=True,
        )

for seg in output['chunks']:
    print(f"[{seg['timestamp'][0]:.2f}---{seg['timestamp'][1]:.2f}] {seg['text']}\n")

Code for comparing to the original OpenAI Whisper

import whisper

model = whisper.load_model("large-v3")

audio_path = 'sample_4_6s_silence.flac'
audio = whisper.load_audio(audio_path)

# Enable initial timestamp constraint (Default)
# Default value is 1 second
max_initial_timestamp = 1.0

# # Disable initial timestamp constraint
# max_initial_timestamp = None

options = {
    "task": "transcribe",
    "language": 'en',
    "max_initial_timestamp": max_initial_timestamp,
    "without_timestamps": False
}
result = model.transcribe(audio, **options)

for seg in result["segments"]:
    print(f"[{seg['start']:.2f}---{seg['end']:.2f}] {seg['text']}\n")

Result - When enabling the initial timestamp constraint (Default setting)

Huggingface whisper-large-v3

[0.00---27.82] Linnell's pictures are a sort of Upguards and Adam paintings, and Mason's exquisite idylls are as national as a jingo poem. Mr. Burkett Foster's landscapes smile at one much in the same way that Mr. Carker used to flash his teeth. And Mr. John Collier gives his
[27.82---34.86] sitter a cheerful slap on the back before he says, like a shampooer in a Turkish bath, next man.

OpenAI whisper-medium whisper-large-v3

[0.00---12.76] Linnell's pictures are a sort of Upguards and Adam paintings, and Mason's exquisite
[12.76---17.60] idylls are as national as a jingo poem.
[17.60---23.32] Mr. Burkett Foster's landscapes smile at one much in the same way that Mr. Carker used
[23.32---25.88] to flash his teeth.
[25.88---32.56] And Mr. John Collier gives his sitter a cheerful slap on the back before he says, like a shampooer
[32.56---34.98] in a Turkish bath, Next man!

As we can see, when keeping the default initial timestamp constraint, the start times of the first sentence in both HuggingFace and OpenAI’s Whisper are inaccurate since the input audio has 6 seconds of leading silence.

Result - When disabling the initial timestamp constraint

Huggingface whisper-large-v3

[27.82---34.86] Linnell's pictures are a sort of Upguards and Adam paintings, and Mason's exquisite idylls are as national as a jingo poem. Mr. Burkett Foster's landscapes smile at one much in the same way that Mr. Carker used to flash his teeth. And Mr. John Collier gives his sitter a cheerful slap on the back before he says, like a shampooer in a Turkish bath, next man.

OpenAI whisper-medium whisper-large-v3

[6.56---12.76] Linnell's pictures are a sort of Upguards and Adam paintings, and Mason's exquisite
[12.76---17.60] idylls are as national as a jingo poem.
[17.60---23.32] Mr. Burkett Foster's landscapes smile at one much in the same way that Mr. Carker used
[23.32---25.90] to flash his teeth.
[25.90---32.54] And Mr. John Collier gives his sitter a cheerful slap on the back before he says, like a shampooer
[32.54---34.96] in a Turkish bath, Next man!

Here are the incorrect results from HuggingFace’s Whisper. When disabling the initial timestamp constraint, OpenAI’s whisper-large-v3 model can accurately identify that the actual human voice starts after 6 seconds of leading silence.

Correct timestamp at the start time of the first sentence after this PR fix.

Huggingface whisper-large-v3

[6.56---12.78] Linnell's pictures are a sort of Upguards and Adam paintings, and Mason's exquisite
[12.78---17.60] idylls are as national as a jingo poem.
[17.60---23.32] Mr. Burkett Foster's landscapes smile at one much in the same way that Mr. Carker used
[23.32---27.82] to flash his teeth. Mr. John Collier gives his sitter a cheerful slap in the same way that Mr. Carker used to flash his teeth. And Mr. John Collier gives his
[27.82---34.86] sitter a cheerful slap on the back before he says, like a shampooer in a Turkish bath, next man.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Just tag some account I have seen in some related issues.
@zucchini-nlp @ArthurZucker @eustlb