Thanks to visit codestin.com
Credit goes to github.com

Skip to content

add tf-gridnet#4864

Merged
mergify[bot] merged 18 commits intoespnet:masterfrom
zqwang7:tfgridnet
Feb 14, 2023
Merged

add tf-gridnet#4864
mergify[bot] merged 18 commits intoespnet:masterfrom
zqwang7:tfgridnet

Conversation

@zqwang7
Copy link
Contributor

@zqwang7 zqwang7 commented Jan 11, 2023

Code of TF-GridNet proposed in

[1] Z.-Q. Wang, S. Cornell, S. Choi, Y. Lee, B.-Y. Kim, and S. Watanabe,
"TF-GridNet: Integrating Full- and Sub-Band Modeling for Speech Separation",
in arXiv preprint arXiv:2211.12433, 2022.
[2] Z.-Q. Wang, S. Cornell, S. Choi, Y. Lee, B.-Y. Kim, and S. Watanabe,
"TF-GridNet: Making Time-Frequency Domain Models Great Again for Monaural Speaker Separation",
in arXiv preprint arXiv:2209.03952, 2022.

The SI-SDR result of using D=48, I=4, J=1, H=192 (and 16 ms window and 8 ms hop sizes) obtains 22.8 dB SI-SNR on WSJ0-2mix. It is close to the 23.2 dB (obtained using 32 ms window and 8 ms hop sizes) reported in the jounral submission. We think the result is reasonably close, considering that, when using 4 s chunk_length and the default SI-SNR loss for training, the current chunking mechanism in ESPNet-SE discards ~15% training examples and some trailing segments (each of which can be up to 2-second long).

@sw005320 sw005320 requested a review from Emrys365 January 12, 2023 02:15
@sw005320 sw005320 added New Features SE Speech enhancement labels Jan 12, 2023
@sw005320 sw005320 added this to the v.202301 milestone Jan 12, 2023
@sw005320
Copy link
Contributor

@Emrys365
Copy link
Collaborator

Emrys365 commented Jan 14, 2023

Could you add some unit tests under test/espnet2/enh?

Generally, for a new model, we would add tests in test/espnet2/enh/test_espnet_model.py and test/espnet2/enh/separator/test_[new model name].py.

@Emrys365
Copy link
Collaborator

Emrys365 commented Jan 16, 2023

It would be great if you also share the trained model on HuggingFace.

You could check this link for more information on how to upload the model: https://github.com/espnet/espnet/blob/master/CONTRIBUTING.md#132-espnet2-recipes

@sw005320
Copy link
Contributor

Any update, @zqwang7?
We're very keen to use TF gridnet in the other applications.

@sw005320
Copy link
Contributor

@zqwang7
Copy link
Contributor Author

zqwang7 commented Jan 24, 2023

FYI, https://github.com/espnet/espnet/actions/runs/4000248137/jobs/6865197032#step:8:9966

In the most recent commits, "fix isort" changes

from espnet2.enh.loss.criterions.tf_domain import FrequencyDomainL1, FrequencyDomainMSE

to

from espnet2.enh.loss.criterions.tf_domain import (FrequencyDomainL1,
FrequencyDomainMSE)

But then, there is a code format error, and I need to run black (run black), and black changes

from espnet2.enh.loss.criterions.tf_domain import (FrequencyDomainL1,
FrequencyDomainMSE)

to

from espnet2.enh.loss.criterions.tf_domain import FrequencyDomainL1, FrequencyDomainMSE

I have no ideas how to deal with this....

@zqwang7
Copy link
Contributor Author

zqwang7 commented Jan 24, 2023

FYI, https://github.com/espnet/espnet/actions/runs/4000248137/jobs/6865197032#step:8:9966

In the most recent commits, "fix isort" changes

from espnet2.enh.loss.criterions.tf_domain import FrequencyDomainL1, FrequencyDomainMSE

to

from espnet2.enh.loss.criterions.tf_domain import (FrequencyDomainL1, FrequencyDomainMSE)

But then, there is a code format error, and I need to run black (run black), and black changes

from espnet2.enh.loss.criterions.tf_domain import (FrequencyDomainL1, FrequencyDomainMSE)

to

from espnet2.enh.loss.criterions.tf_domain import FrequencyDomainL1, FrequencyDomainMSE

I have no ideas how to deal with this....

Would changing

from espnet2.enh.loss.criterions.tf_domain import FrequencyDomainL1, FrequencyDomainMSE

to

from espnet2.enh.loss.criterions.tf_domain import FrequencyDomainL1
from espnet2.enh.loss.criterions.tf_domain import FrequencyDomainMSE

solve the problem?

@Emrys365
Copy link
Collaborator

It seems a version issue. Could you update both packages to the latest version and retry?

@sw005320
Copy link
Contributor

See https://github.com/espnet/espnet/actions/runs/4023093431/jobs/6913553984#step:8:9974
Is it the same issue that you mentioned before in #4864 (comment)?

@sw005320
Copy link
Contributor

sw005320 commented Feb 1, 2023

@kan-bayashi kan-bayashi modified the milestones: v.202301, v.202303 Feb 1, 2023
@codecov
Copy link

codecov bot commented Feb 14, 2023

Codecov Report

Merging #4864 (29573f1) into master (478ba00) will increase coverage by 0.07%.
The diff coverage is 98.87%.

@@            Coverage Diff             @@
##           master    #4864      +/-   ##
==========================================
+ Coverage   76.56%   76.63%   +0.07%     
==========================================
  Files         603      604       +1     
  Lines       53738    53934     +196     
==========================================
+ Hits        41142    41334     +192     
- Misses      12596    12600       +4     
Flag Coverage Δ
test_integration_espnet1 66.33% <ø> (+0.13%) ⬆️
test_integration_espnet2 47.42% <17.97%> (-0.18%) ⬇️
test_python 66.55% <98.87%> (+0.09%) ⬆️
test_utils 23.35% <ø> (+0.26%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
espnet2/enh/separator/tfgridnet_separator.py 98.87% <98.87%> (ø)
espnet2/tasks/enh.py 97.43% <100.00%> (+0.01%) ⬆️
espnet2/iterators/chunk_iter_factory.py 70.00% <0.00%> (-5.00%) ⬇️
espnet2/bin/enh_tse_inference.py 91.02% <0.00%> (-0.72%) ⬇️
espnet2/asr/espnet_model.py 76.83% <0.00%> (-0.51%) ⬇️
espnet2/train/preprocessor.py 27.17% <0.00%> (+0.02%) ⬆️
espnet2/layers/stft.py 88.00% <0.00%> (+0.12%) ⬆️
espnet/tts/pytorch_backend/tts.py 78.63% <0.00%> (+0.29%) ⬆️
...et/nets/pytorch_backend/e2e_asr_mix_transformer.py 84.97% <0.00%> (+0.46%) ⬆️
espnet/asr/asr_utils.py 76.53% <0.00%> (+0.87%) ⬆️
... and 3 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@sw005320 sw005320 added the auto-merge Enable auto-merge label Feb 14, 2023
@mergify mergify bot merged commit a24d72a into espnet:master Feb 14, 2023
@zqwang7 zqwang7 mentioned this pull request Aug 4, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

auto-merge Enable auto-merge ESPnet2 New Features README SE Speech enhancement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants