Thanks!
What does it look like?
Could you paste an example?

It is like the tqdm progress bar, with percentage progress, current batch / total_batch, time_so_far / est_time_to_finish, time_per_batch.

0%|          | 1/1577 [00:20<8:49:13, 20.15s/it]

Maybe, you can add a bad example as well?

Sure.

@ftshijt, can you review it?

You may need to increase --num_threads to deal with large memory consumptions in scikit-learn?
(Ideally, I want you to solve it by avoiding using such less-refined implementation.

OK. Working on this item.

since CI does not support 1.5.0, we can remove these lines

CTC BPE token part looks complicated and tricky.
It requires some documentation (in the source code and asr2 or st2 documents).

The idea is simple. Maybe my way is a bit complicated. In st2, we use different the text targets for CTC and attention-decoder.

• CTC target: ASR transcriptions for ASR or ST, while <not_available> is used in MT.
• Att-Dec target: ASR transcriptions for ASR, while translation transcriptions for ST / MT.

For this purpose, we need different text input as data. In ESPnet preprocessor, the number of tokenizer should match the text input.

        assert (
            len(token_type) == len(token_list) == len(bpemodel) == len(text_name)
        ), "token_type, token_list, bpemodel, or processing text_name mismatched"

But in practice, the bpe model for CTC and Att-Dec is the same. We combine the vocabulary for ASR language and Translation language. However, I made CTC text tokenizer option as an explicit argument, which is easy to change.

Can you explain about it and embed the explanation in the source coed?

I'll put the following explanation down below.

        # not_available_symbol is a special symbol as placeholder in the text. e.g.
        #      "utt_id <na>" an item in the text input
        # Then such samples will not have the corresponding text signals.
        # The resulting tensor would be processed as torch.longtensor([-1])

space between or "/" between?

Since it was for multi-task, is it a good idea to put in st task or a more general one?

I'm asking cause the major reason we split st from asr in the previous context is:

• the different architectures between each other (for st only tasks, we also have a few unique architecture designs for each component (e.g., separate asr/mt decoder, two-pass framework with multi-decoder, etc.)
• data preparation (design for tgt_text, src_text, and src_speech)
• specific evaluation (bleu calculation and multi-reference support)

however, I feel many of the above parts are not shared (e.g., the architecture is still discrete asr; we skip the joint-task framework, instead, we do either asr or st in a single run; no support to multi-reference scoring)

Given the above reasons, I lean to call it s2t2 instead of st2. Please let me know you thoughts!

The script looks great!

One issue remains on my side is the support of multi-reference scenarios for ST evaluation (I double checked the support, but it seems not supported in preparation yet) Please refer https://github.com/espnet/espnet/blob/master/egs2/TEMPLATE/st1/st.sh#L547-L553 for some details on how we process those

For more information on the name:
https://github.com/espnet/espnet/blob/master/egs2/TEMPLATE/st1/st.sh#L1585C75-L1585C75

consider changing it?