Codestin Search App

Emrys365 · 2023-05-31T03:45:06Z

This PR adds two main functions to ESPnet-SE (speech enhancement):

DNSMOS and PESQ evaluation
- We can now evaluate the DNSMOS scores with espnet2/enh/layers/dnsmos.py.
- We can also evaluate the PESQ score (be careful about the license).
utt2category support
- Now the enh.sh pipeline can automatically add the category information (utt2category) of each sample to the mini-batch by setting --use_utt2category true, which will be later processed by the preprocessor (espnet2/train/preprocessor.py) to be converted into unique integers.
- The chunk-based iterator also supports constructing mini-batches based on the category information, so that each mini-batch only contains samples of the same category.

In addition:

The bugs in the data preparation for recipes egs2/vctk_noisy/enh1 and egs2/vctk_noisyreverb/enh1 have been fixed.
The SE losses with a weight of 0 will now be calculated without gradient to reduce the memory footprint.
A new argument channel_reordering is added to EnhPreprocessor and TSEPreprocessor to support data augmentation for multi-channel signals.
A new argument --skip_stats_npz is added to espnet2/tasks/abs_task.py to allow skipping the preparation of *_stats.npz during the "Collecting Stats" stage . This is enabled by default for the SE task because these stats will not be used later and can save the computation.
A new learning rate scheduler has been added in espnet2/schedulers/warmup_reducelronplateau.py, which combines the WarmupLR and ReduceLROnPlateau schedulers.

…cement related tasks

for more information, see https://pre-commit.ci

simpleoier

Thanks for the efforts! I left some comments. I mainly concern about the utt2category and collect_stats parts.

egs2/TEMPLATE/enh1/enh.sh

simpleoier · 2023-05-31T12:32:45Z

egs2/TEMPLATE/enh1/enh.sh

    --use_noise_ref    # Whether or not to use noise signal as an additional reference
                         for training a denoising model (default="${use_noise_ref}")
    --extra_wav_list   # Extra list of scp files for wav formatting (default="${extra_wav_list}")
+    --use_utt2category # Whether or not to load the utt2category file for training (default="${use_utt2category}")


simpleoier · 2023-05-31T12:38:01Z

egs2/TEMPLATE/enh1/enh.sh

            done
        fi

+        # Add the category information at the end of the data path list


It is a bit redundant I think. Maybe we can do something in the sampler and preprocessor instead of here.

I think that will require more changes in more scripts. The current design only needs modifications in enh.sh, espnet2/tasks/enh.py, and espnet2/train/preprocessor.py.

simpleoier · 2023-05-31T12:39:49Z

espnet2/bin/enh_scoring.py

        format="%(asctime)s (%(module)s:%(lineno)d) %(levelname)s: %(message)s",
    )

+    if dnsmos:


This is a good feature. Can you add pesq as well?

Yes, I can add the code. Please review whether we will have the license issue.

The pesq python package seems to violate the PESQ license.

simpleoier · 2023-05-31T12:41:43Z

espnet2/bin/enh_scoring.py

+                    "You can download the model from https://github.com/microsoft/"
+                    "DNS-Challenge/tree/master/DNSMOS/DNSMOS"
+                )
+            if not Path(dnsmos_args["p808_model"]).exists():


A minor suggestion, this dnsmos is very specific to p808 model. It would be better if this can be more flexible for future.

I don't know if there are other options for the DNS model. I just followed the script provided by the official repository https://github.com/microsoft/DNS-Challenge/blob/master/DNSMOS/dnsmos_local.py#LL23C47-L23C47

espnet2/tasks/abs_task.py

for more information, see https://pre-commit.ci

codecov · 2023-06-02T22:58:37Z

Codecov Report

Merging #5193 (5bf2ed3) into master (d26489e) will decrease coverage by 0.57%.
The diff coverage is 54.64%.

@@            Coverage Diff             @@
##           master    #5193      +/-   ##
==========================================
- Coverage   75.00%   74.43%   -0.57%     
==========================================
  Files         630      642      +12     
  Lines       56821    57611     +790     
==========================================
+ Hits        42619    42885     +266     
- Misses      14202    14726     +524

Flag	Coverage Δ
test_integration_espnet1	`66.28% <ø> (ø)`
test_integration_espnet2	`47.52% <46.09%> (-0.10%)`	⬇️
test_python	`65.15% <39.34%> (-0.53%)`	⬇️
test_utils	`23.28% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
espnet2/enh/layers/dnsmos.py	`0.00% <0.00%> (ø)`
espnet2/bin/enh_scoring.py	`68.86% <35.29%> (-15.04%)`	⬇️
espnet2/train/preprocessor.py	`33.23% <65.43%> (+4.07%)`	⬆️
espnet2/schedulers/warmup_reducelronplateau.py	`80.18% <80.18%> (ø)`
espnet2/bin/enh_tse_inference.py	`91.02% <100.00%> (ø)`
espnet2/enh/espnet_model.py	`85.64% <100.00%> (-1.02%)`	⬇️
espnet2/enh/espnet_model_tse.py	`79.06% <100.00%> (-1.58%)`	⬇️
espnet2/iterators/chunk_iter_factory.py	`91.66% <100.00%> (+9.48%)`	⬆️
espnet2/tasks/abs_task.py	`76.67% <100.00%> (+0.14%)`	⬆️
espnet2/tasks/enh.py	`97.45% <100.00%> (+0.01%)`	⬆️
... and 1 more

... and 18 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

sw005320 · 2023-06-04T18:42:32Z

Can you add some tests for skipping the statistics collection and some changes in preprocessor?
It's OK if it is difficult.

Emrys365 · 2023-06-04T18:46:57Z

Can you add some tests for skipping the statistics collection and some changes in preprocessor? It's OK if it is difficult.

Sure. For the skipping the statistics collection part, I think it is automatically covered by the integration tests. I will try to add some tests for the preprocessor.

for more information, see https://pre-commit.ci

sw005320 · 2023-06-05T23:10:46Z

See https://github.com/espnet/espnet/actions/runs/5182160044/jobs/9338608132?pr=5193#step:18:2471

This PR is good for my side.
So, if @Emrys365, @simpleoier, and CI are OK, please merge this PR.

simpleoier

LGTM!
A minor suggestion, it would be better to add acknowledgement or reference to DNSMOS or PESQ.

Emrys365 added 5 commits May 17, 2023 12:59

Add DNSMOS evaluation in espnet2/bin/enh_scoring.py

6afebc1

Merge branch 'tse' of github.com:Emrys365/espnet into tse

6c7e783

Merge branch 'master' of github.com:espnet/espnet into tse

498dc3c

Fix bugs in VCTK-based recipes

0deb541

Add support for utt2category in chunk-based iterator and speech enhan…

bda95db

…cement related tasks

Emrys365 added ESPnet2 SE Speech enhancement labels May 31, 2023

[pre-commit.ci] auto fixes from pre-commit.com hooks

187bb69

for more information, see https://pre-commit.ci

sw005320 added this to the v.202307 milestone May 31, 2023

sw005320 added Enhancement Enhancement Bugfix New Features labels May 31, 2023

sw005320 requested a review from simpleoier May 31, 2023 12:03

simpleoier reviewed May 31, 2023

View reviewed changes

Emrys365 and others added 9 commits May 31, 2023 09:06

Update descriptions

66015ea

Merge branch 'tse' of github.com:Emrys365/espnet into tse

d733ccf

Redesign interfaces for controlling the behavior in collect_stats

30ad631

Reflect comments

e7684b3

Add PESQ evaluation

029441d

[pre-commit.ci] auto fixes from pre-commit.com hooks

9207310

for more information, see https://pre-commit.ci

Reformat espnet2/bin/enh_scoring.py

0aadb07

Fix minor issues

17035bf

Fix errors in test/espnet2/bin/test_enh_scoring.py

d7f4768

Emrys365 added 4 commits June 4, 2023 14:02

Fix a bug in espnet2/enh/layers/dnsmos.py

cbf8c1d

Fix a bug in EnhPreprocessor

48163e3

Revert changes in espnet2/main_funcs/collect_stats.py

1f12c93

Add unit tests for new LR schedulers

4dded89

Emrys365 and others added 6 commits June 4, 2023 15:08

Add unit tests for new functions in chunk_iter_factory

dbf5e98

[pre-commit.ci] auto fixes from pre-commit.com hooks

23cc5d7

for more information, see https://pre-commit.ci

Modify egs2/mini_an4/enh1 to test the new utt2category functions

fa6da16

Fix a bug in chunk_iter_factory

b004e30

Update show_enh_score.sh

fd84b43

Add utt2category support for DynamicMixingPreprocessor

1ee586b

mergify bot added the CI Travis, Circle CI, etc label Jun 5, 2023

Emrys365 and others added 6 commits June 5, 2023 01:51

Add utt2category support for DynamicMixingPreprocessor

873e776

Fix a bug when GPU is used in espnet2/bin/enh_tse_inference.py

b7041cb

Fix a bug in DynamicMixingPreprocessor when utt2category is used

683aab2

[pre-commit.ci] auto fixes from pre-commit.com hooks

025a527

for more information, see https://pre-commit.ci

Update egs2/TEMPLATE/enh1/enh.sh

9aa1d5f

Fix a bug in egs2/mini_an4/tse1 when utt2category is used

66f8d24

Emrys365 added 2 commits June 6, 2023 14:39

Remove categories from espnet2/enh/espnet_model.py

6666017

Fix a bug in egs2/mini_an4/enh1 when utt2category is used

5bf2ed3

simpleoier approved these changes Jun 7, 2023

View reviewed changes

simpleoier merged commit 096e2bb into espnet:master Jun 7, 2023

sw005320 mentioned this pull request Jun 12, 2023

RuntimeError: istft requires a complex-valued input tensor matching the output from stft with return_complex=True. #5227

Closed

Conversation

Emrys365 commented May 31, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

simpleoier left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

simpleoier May 31, 2023

Choose a reason for hiding this comment

Uh oh!

simpleoier May 31, 2023

Choose a reason for hiding this comment

Uh oh!

Emrys365 May 31, 2023

Choose a reason for hiding this comment

Uh oh!

simpleoier May 31, 2023

Choose a reason for hiding this comment

Uh oh!

Emrys365 May 31, 2023

Choose a reason for hiding this comment

Uh oh!

simpleoier May 31, 2023

Choose a reason for hiding this comment

Uh oh!

Emrys365 May 31, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

codecov bot commented Jun 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

sw005320 commented Jun 4, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Emrys365 commented Jun 4, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sw005320 commented Jun 5, 2023

Uh oh!

simpleoier left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Emrys365 commented May 31, 2023 •

edited

Loading

codecov bot commented Jun 2, 2023 •

edited

Loading

sw005320 commented Jun 4, 2023 •

edited

Loading

Emrys365 commented Jun 4, 2023 •

edited

Loading