Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Espnet Multi-convformer implementation#5832

Merged
sw005320 merged 16 commits intoespnet:masterfrom
Darshan7575:master
Jul 22, 2024
Merged

Espnet Multi-convformer implementation#5832
sw005320 merged 16 commits intoespnet:masterfrom
Darshan7575:master

Conversation

@Darshan7575
Copy link
Contributor

@Darshan7575 Darshan7575 commented Jul 8, 2024

What?

  • This PR adds a new encoder architecture coined Multi-convformer that extends conformer with gMLP style gating and a multi-convolution setup. More details about this architecture can be found here.
  • It also adds recipes and our results w/ links to models to Librispeech-100, Librispeech and SLURP datasets.

Known issues:

  • Turns out, grouped 2d convolutions are slower in PyTorch v1.x, and it has not yet been resolved to date (see here). Since some of our code relies on this, we recommend using PyTorch v2 when training on large datasets.

Citation:

If you use our work, please consider citing us:

@misc{prabhu2024multiconvformerextendingconformermultiple,
      title={Multi-Convformer: Extending Conformer with Multiple Convolution Kernels}, 
      author={Darshan Prabhu and Yifan Peng and Preethi Jyothi and Shinji Watanabe},
      year={2024},
      eprint={2407.03718},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2407.03718}, 
}

@sw005320 sw005320 requested a review from pyf98 July 8, 2024 11:36
@sw005320 sw005320 added Recipe New Features ASR Automatic speech recogntion labels Jul 8, 2024
@sw005320 sw005320 added this to the v.202405 milestone Jul 8, 2024
@sw005320
Copy link
Contributor

sw005320 commented Jul 8, 2024

@pyf98, can you review this PR?

@sw005320
Copy link
Contributor

sw005320 commented Jul 12, 2024

@pyf98, this is a reminder.

@sw005320
Copy link
Contributor

sw005320 commented Jul 12, 2024

Please check https://github.com/espnet/espnet/actions/runs/9908186288/job/27373422523#step:8:292

@Darshan7575 Darshan7575 changed the title Espnet Multi-convformer implementation WIP: Espnet Multi-convformer implementation Jul 12, 2024
@codecov
Copy link

codecov bot commented Jul 12, 2024 •
edited
Loading

Codecov Report

Attention: Patch coverage is 14.34783% with 197 lines in your changes missing coverage. Please review.

Project coverage is 47.40%. Comparing base (f2431f4) to head (ab1d0a6).

Files Patch % Lines
espnet2/asr/encoder/multiconvformer_encoder.py 15.82% 117 Missing ⚠️
espnet2/asr/layers/multiconv_cgmlp.py 11.11% 80 Missing ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##           master    #5832      +/-   ##
==========================================
- Coverage   48.35%   47.40%   -0.95%     
==========================================
  Files         497      781     +284     
  Lines       44007    71798   +27791     
==========================================
+ Hits        21279    34035   +12756     
- Misses      22728    37763   +15035
Flag Coverage Δ
test_integration_espnet2 48.17% <13.97%> (-0.19%) ⬇️
test_integration_espnetez 28.19% <14.34%> (?)
test_python_espnet1 17.72% <0.00%> (?)
test_utils 20.61% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

pyf98
pyf98 reviewed Jul 12, 2024
View reviewed changes
@pyf98
Copy link
Collaborator

pyf98 commented Jul 12, 2024

Looks good to me.

@sw005320
Copy link
Contributor

sw005320 commented Jul 18, 2024

@Darshan7575, I observed that the test coverage by codecov was dropped.
It might be an issue in codecov since you added the unit test, but can you double-check it?
Then, I'll merge this PR.

@Darshan7575
Copy link
Contributor Author

Darshan7575 commented Jul 22, 2024 •
edited
Loading

I couldn't find any apparent problem with the new unit test I added. Plus, the unit test I wrote should cover almost all aspects of the newly defined classes. Weirdly, the codecov/patch test coverage fails even before all the unit tests finish. It seems like a problem with codecov.

@sw005320 sw005320 merged commit b8b31c0 into espnet:master Jul 22, 2024
@sw005320
Copy link
Contributor

sw005320 commented Jul 22, 2024

Thanks!
Then, I'll merge it.

@Darshan7575 Darshan7575 changed the title WIP: Espnet Multi-convformer implementation Espnet Multi-convformer implementation Jul 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pyf98 pyf98 pyf98 left review comments

Assignees

No one assigned

Labels

ASR Automatic speech recogntion ESPnet2 New Features README Recipe

Projects

None yet

Milestone

v.202409

Development

Successfully merging this pull request may close these issues.

3 participants

@Darshan7575 @sw005320 @pyf98