Codestin Search App

juice500ml · 2023-09-21T02:04:00Z

What?

ASR2 recipe for the Multilingual Librispeech
Training from scratch, 10h split (10h per language, total 8 language)
ASR2, ASR1 Fbank baseline, ASR1 SSL baseline

Why?

MLS experiments for the ASR2 paper

Codecov Report

Merging #5441 (63940e1) into master (01037ca) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #5441   +/-   ##
=======================================
  Coverage   75.37%   75.37%           
=======================================
  Files         709      709           
  Lines       65291    65291           
=======================================
  Hits        49212    49212           
  Misses      16079    16079

Flag	Coverage Δ
test_configuration_espnet2	`∅ <ø> (∅)`
test_integration_espnet1	`65.67% <ø> (ø)`
test_integration_espnet2	`48.71% <ø> (ø)`
test_python_espnet1	`19.16% <ø> (ø)`
test_python_espnet2	`51.40% <ø> (ø)`
test_utils	`23.10% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

…to asr_mls_exp

for more information, see https://pre-commit.ci

juice500ml · 2023-10-19T19:52:48Z

Just found out that fr evaluation was killed during mid-eval... Need to run eval again and update the README.md. I'll come back to this after eval is done 🥲

…to asr_mls_exp

juice500ml · 2023-10-21T06:29:05Z

I tried to apply all the great reviews to the PR, and it seems that now it is close to merging! Evaluation + model upload to huggingface is also done.

sw005320 · 2023-10-21T11:34:50Z

egs2/mls/asr2/RESULTS.md

I observed degradations for some languages from fbank.
Can you summarize them here?
Will you plan to tune it more?

CER comparison

Model ASR1 FBANK ASR1 SSL ASR2

EN 22.1 15.3 12.4

ES 7.0 7.0 7.9

DE 10.3 10.4 11.9

FR 13.5 13.6 17.0

IT 7.1 6.9 7.9

NL 11.4 10.8 14.6

PL 8.2 7.2 11.3

PT 12.7 11.8 13.2

Currently, all the experiments aren't using the language model (asr1 ssl, asr1 fbank, asr2), so it can be somewhat worse compared to other reported numbers on using MLS 10h (10h per each language, total 8 languages).
Also, for asr1, I tried tuning the learning rate (1e-3, 1e-4, 1e-5) and chose the best among three. In comparison, for asr2, I tuned learning rate, batch size, kmeans k, source bpe, and target bpe. I think, to increase the asr2 performance, we need some modifications on the algorithm.
Interestingly, even though asr2 is generally a bit worse, it shows much better performance on English while showing limited performance on French. I'm suspecting that underlying ssl model is English-friendly so asr1 ssl and asr2 is somewhat better than asr1 fbank, but I'm not so sure.
For now, I wasn't thinking about tuning it more (because ES performance is similar to the original ES-only model's performance with CER=7.1), but if you think it's necessary, I'm open to more tuning.

sw005320 · 2023-10-21T11:36:38Z

I think this PR is almost there.

sw005320 · 2023-10-22T20:34:17Z

@simpleoier and/or @ftshijt, if it is OK for you, please merge this PR.

ftshijt · 2023-10-23T08:51:23Z

LGTM! Please also consider:

uploading the pre-trained models
adding additional results for the full set

Add baseline results

e2d65d6

mergify bot added ESPnet2 CI Travis, Circle CI, etc labels Sep 21, 2023

juice500ml and others added 11 commits September 20, 2023 22:04

add ssl baseline

bfa0682

[pre-commit.ci] auto fixes from pre-commit.com hooks

d1c9e7f

for more information, see https://pre-commit.ci

[pre-commit.ci] auto fixes from pre-commit.com hooks

653dc8e

for more information, see https://pre-commit.ci

Add asr2

49792fc

Merge branch 'asr_mls_exp' of https://github.com/juice500ml/espnet in…

a13b910

…to asr_mls_exp

[pre-commit.ci] auto fixes from pre-commit.com hooks

6f11c39

for more information, see https://pre-commit.ci

remove slack hook

1271802

Merge branch 'asr_mls_exp' of https://github.com/juice500ml/espnet in…

70e6141

…to asr_mls_exp

[pre-commit.ci] auto fixes from pre-commit.com hooks

af812c8

for more information, see https://pre-commit.ci

fix typo

a32df1d

Merge branch 'asr_mls_exp' of https://github.com/juice500ml/espnet in…

858347f

…to asr_mls_exp

sw005320 requested review from ftshijt and simpleoier September 27, 2023 10:29

sw005320 added Recipe ASR Automatic speech recogntion labels Sep 27, 2023

sw005320 added this to the v.202312 milestone Sep 27, 2023

ftshijt reviewed Sep 27, 2023

View reviewed changes

simpleoier reviewed Oct 2, 2023

View reviewed changes

Emrys365 marked this pull request as ready for review October 12, 2023 14:39

juice500ml mentioned this pull request Oct 12, 2023

Multilingual Librispeech (MLS) refactor ASR1 recipe #5323

Merged

17 tasks

juice500ml and others added 2 commits October 17, 2023 03:46

Update egs2/mls/asr2/cmd.sh

8b64368

Co-authored-by: Jiatong <[email protected]>

Merge branch 'master' into asr_mls_exp

98dcecd

juice500ml added 2 commits October 18, 2023 17:34

Update wavlm baseline results

d70bf9d

Merge branch 'asr_mls_exp' of https://github.com/juice500ml/espnet in…

5d00e77

…to asr_mls_exp

pre-commit-ci bot and others added 2 commits October 18, 2023 21:37

[pre-commit.ci] auto fixes from pre-commit.com hooks

2a1ae7f

for more information, see https://pre-commit.ci

apply reviews

91c2b06

juice500ml added 6 commits October 19, 2023 16:01

set default setting to SSL baseline

6d5c422

Update mls_fr_test results

115cfb5

move from utils/ to egs2/TEMPLATE/asr1/scripts/utils

29d7cb8

Add hf links

5584524

make pr happy

adbce2e

Merge branch 'master' into asr_mls_exp

0df81a0

juice500ml changed the title ~~[WIP] Multilingual Librispeech ASR2 + ASR1 baselines~~ Multilingual Librispeech ASR2 + ASR1 baselines Oct 21, 2023

juice500ml added 2 commits October 21, 2023 01:49

make pr happy

0e258fd

Merge branch 'asr_mls_exp' of https://github.com/juice500ml/espnet in…

24cba28

…to asr_mls_exp

juice500ml requested review from ftshijt and simpleoier October 21, 2023 06:28

Make pr happy

9774955

sw005320 reviewed Oct 21, 2023

View reviewed changes

simpleoier mentioned this pull request Oct 22, 2023

[SVS2] discrete svs development #5483

Closed

4 tasks

juice500ml added 2 commits October 22, 2023 15:36

add more comments

5672114

Merge branch 'master' into asr_mls_exp

63940e1

ftshijt merged commit d95b221 into espnet:master Oct 23, 2023

juice500ml deleted the asr_mls_exp branch October 23, 2023 16:42

+                  --tgt_nbpe $tgt_nbpe \
+                  --src_case ${src_case} \
+                  --tgt_case ${tgt_case} \
+                  --speed_perturb_factors "" \

               htmlcov
               coverage.xml*
-              bats-core/
+              test_utils/bats-core/

+                  # --src_bpe_train_text "data/${train_set}/text.${src_case}.${src_lang}" \
+                  # --tgt_bpe_train_text "data/${train_set}/text.${tgt_case}.${tgt_lang}" \
+                  # --lm_train_text "data/${train_set}/text.${tgt_case}.${tgt_lang} data/local/other_text/text" \

	# --src_bpe_train_text "data/${train_set}/text.${src_case}.${src_lang}" \
	# --tgt_bpe_train_text "data/${train_set}/text.${tgt_case}.${tgt_lang}" \
	# --lm_train_text "data/${train_set}/text.${tgt_case}.${tgt_lang} data/local/other_text/text" \

+              - Git hash: `a8bc43b1bfc9518da7dd8be4cad0ef346ef222fc`
+                - Commit date: `Sun Aug 20 16:17:23 2023 -0400`
+              ## exp/asr_smallerbatch_wamrup10k_lr0.0001_e200_raw_wavlm_large_21_full_km1000_bpe_rm3000_bpe_ts150

+              nclusters=1000
+              src_lang=$(echo "${kmeans_feature}_full_km${nclusters}" | tr "/" "_")
+              tgt_lang=en

+              ./asr2.sh \
+                  --local_data_opts "--lang ${lang} --data_split ${data_split}" \
+                  --portion 1.0 \

               if [ ${stage} -le 4 ] && [ ${stop_stage} -ge 4 ] && ! [[ " ${skip_stages} " =~ [[:space:]]4[[:space:]] ]]; then
                   log "Stage 4a: Perform Kmeans using ${kmeans_feature_type} features"
+                  if [ ${ngpu} -gt 0 ]; then

+                  export train_cmd="slurm.pl"
+                  export cuda_cmd="slurm.pl"
+                  export decode_cmd="slurm.pl --num_threads 4 --mem 2000M"

Model	ASR1 FBANK	ASR1 SSL	ASR2
EN	22.1	15.3	12.4
ES	7.0	7.0	7.9
DE	10.3	10.4	11.9
FR	13.5	13.6	17.0
IT	7.1	6.9	7.9
NL	11.4	10.8	14.6
PL	8.2	7.2	11.3
PT	12.7	11.8	13.2

		@@ -0,0 +1,18 @@
		# Default configuration

		src_nbpe=3000
		tgt_nbpe=150

Conversation

juice500ml commented Sep 21, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What?

Why?

See also

Uh oh!

sw005320 commented Sep 27, 2023

Uh oh!

juice500ml commented Sep 27, 2023

Uh oh!

ftshijt left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

simpleoier left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov bot commented Oct 16, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

juice500ml commented Oct 19, 2023

Uh oh!

juice500ml commented Oct 21, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sw005320 commented Oct 21, 2023

Uh oh!

sw005320 commented Oct 22, 2023

juice500ml commented Sep 21, 2023 •

edited

Loading

codecov bot commented Oct 16, 2023 •

edited

Loading

juice500ml commented Oct 21, 2023 •

edited

Loading