Why are lengths relative? #2971

It is documented, but not where you were looking. There is a section titled "Why relative lengths instead of absolute lengths?" in the speech recognition from scratch tutorial, and a one-line version in the data loading tutorial:

The use of relative lengths instead of absolute indexes guarantees that that these values do not change even after feature extraction: the relative true length remains the same even after STFT whatever is the window.

The API reference does not say it anywhere. speechbrain/dataio/batch.py contains the string "relative" zero times, and PaddedBatch is the first thing anyone touches. The only hint in the docstring is a doctest where the numbers happen to be floats (batch.py:81-82):

>>> batch.foo.lengths
tensor([0.5000, 1.0000])

So your debugging session was not your fault.

Where the float comes from

pad_right_to divides the true shape by the padded shape, speechbrain/utils/data_utils.py:450:

valid_vals.append(tensor.shape[j] / target_shape[j])

batch_pad_right collects one of those per item and stacks them, data_utils.py:511-519:

padded, valid_percent = pad_right_to(t, max_shape, mode=mode, value=value)
batched.append(padded)
valid.append(valid_percent[0])
...
return batched, torch.tensor(valid)

and PaddedBatch uses that as its default padding function, speechbrain/dataio/batch.py:148,178.

Where it is turned back into counts

Every consumer multiplies by the current time dimension. CTC loss, speechbrain/nnet/losses.py:254-255:

input_lens = (input_lens * log_probs.shape[1]).round().int()
target_lens = (target_lens * targets.shape[1]).round().int()

Attention decoder, speechbrain/nnet/RNN.py:989:

enc_len = torch.round(enc_states.shape[1] * wav_len).long()

Note both read shape[1] off the tensor in front of them, not off the waveform. length_to_mask (speechbrain/dataio/dataio.py:803) takes absolute counts, so the conversion happens at the point of use.

Why relative survives what absolute does not

The tensor is never rescaled when time resolution changes, and it does not need to be. Fbank.forward does not even accept a lengths argument (speechbrain/lobes/features.py:148, def forward(self, wav)), which is only sound because the caller's lengths stay valid across it. The ASR template says so in a comment, templates/speech_recognition/ASR/train.py:159:

fea_lens = wav_lens  # Relative lengths are preserved

Run against a current checkout:

speechbrain 1.1.0 | from .../speechbrain/speechbrain/__init__.py
padded waveform : (2, 16000)
wav_lens        : [1.0, 0.6000000238418579]
abs samples     : [16000, 9600]
fbank           : (2, 101, 40)
abs frames      : [101, 61]
resampled 8k    : (2, 8000)
abs samples 8k  : [8000, 4800]
mask frames kept: [101, 61]

One wav_lens tensor, three different time axes: 16000 samples, 101 frames, 8000 samples. round(0.6 * N) is right in all three. An absolute 9600 is right in exactly one of them and silently out of range in the other two.

That is the whole argument. Pooling, strided convolution, STFT hop, resampling and speed perturbation all change the number of time steps, and a pipeline that passed absolute counts would have to thread a rescale through every one of them, with each stage needing to know the previous stage's hop and padding. The relative form makes the lengths invariant to the transform, so a module only needs the tensor it was handed.

The cost is the one you paid: it is a float in (0, 1] where the type system suggests a count, the rounding is not exact, and nothing warns you.

Docstring convention statement, for reference, speechbrain/utils/callchains.py:51:

In SpeechBrain the convention is to use relative lengths.