Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Add PyTorch & GPU support for DNSMOS calculation#5548

Merged
sw005320 merged 7 commits intoespnet:masterfrom
Emrys365:espnet2_enh
Jan 4, 2024
Merged

Add PyTorch & GPU support for DNSMOS calculation#5548
sw005320 merged 7 commits intoespnet:masterfrom
Emrys365:espnet2_enh

Conversation

@Emrys365
Copy link
Collaborator

@Emrys365 Emrys365 commented Nov 15, 2023 •
edited
Loading

What?

This PR adds support for PyTorch-based DNSMOS calculation in addition to the existing ONNX-based one. This is based on a third-party package onnx2torch which converts the ONNX-format model to a PyTorch module.

Why?

Current DNSMOS calculation basically relies on the ONNX wrapper, where the GPU-based processing is not well supported. By converting the ONNX-format model to a PyTorch module using onnx2torch, we can now easily use CUDA devices to greatly speed up the calculation.

In addition, since the model is converted to a PyTorch module, gradient backpropagation is also supported, which can be used to provide supervision during training later on.

@Emrys365 Emrys365 added ESPnet2 SE Speech enhancement labels Nov 15, 2023
@sw005320
Copy link
Contributor

sw005320 commented Nov 21, 2023

It requires additional packages for the test https://github.com/espnet/espnet/actions/runs/6872983326/job/18692312021?pr=5548#step:9:2429

@codecov
Copy link

codecov bot commented Nov 21, 2023 •
edited
Loading

Codecov Report

Attention: 40 lines in your changes are missing coverage. Please review.

Comparison is base (d325413) 76.54% compared to head (2453f85) 76.57%.
Report is 200 commits behind head on master.

Files Patch % Lines
espnet2/enh/layers/dnsmos.py 54.66% 34 Missing ⚠️
espnet2/bin/enh_scoring.py 25.00% 6 Missing ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##           master    #5548      +/-   ##
==========================================
+ Coverage   76.54%   76.57%   +0.03%     
==========================================
  Files         720      720              
  Lines       66599    66695      +96     
==========================================
+ Hits        50975    51072      +97     
+ Misses      15624    15623       -1
Flag Coverage Δ
test_configuration_espnet2 ∅ <ø> (∅)
test_integration_espnet1 62.92% <ø> (-0.01%) ⬇️
test_integration_espnet2 50.10% <25.00%> (+<0.01%) ⬆️
test_python_espnet1 19.06% <0.00%> (-0.03%) ⬇️
test_python_espnet2 52.45% <51.80%> (+0.06%) ⬆️
test_utils 22.15% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@Emrys365
Copy link
Collaborator Author

Emrys365 commented Nov 22, 2023

It requires additional packages for the test https://github.com/espnet/espnet/actions/runs/6872983326/job/18692312021?pr=5548#step:9:2429

I see. I intended not to make it a mandatory package in espnet. Just for unit test purposes, do you think I should add this dependency in setup.py? Should I add it in the "test" part of the requirements?

@sw005320
Copy link
Contributor

sw005320 commented Nov 27, 2023

How about only adding it in https://github.com/espnet/espnet/blob/master/ci/install.sh?

@Emrys365
Copy link
Collaborator Author

Emrys365 commented Nov 27, 2023

OK. I'll try it.

@mergify mergify bot added the CI Travis, Circle CI, etc label Nov 27, 2023
@sw005320 sw005320 added this to the v.202312 milestone Jan 4, 2024
@sw005320
Copy link
Contributor

sw005320 commented Jan 4, 2024

Thanks a lot!

@sw005320 sw005320 merged commit 6d53110 into espnet:master Jan 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

CI Travis, Circle CI, etc ESPnet2 SE Speech enhancement

Projects

None yet

Milestone

v.202312

Development

Successfully merging this pull request may close these issues.

2 participants

@Emrys365 @sw005320