Codestin Search App

jwrh · 2023-03-19T19:52:54Z

Hi, this is the first part of FindAdaptNet PR update -- the adapter module.

for more information, see https://pre-commit.ci

simpleoier

Thanks for the PR! I left some comments.

espnet2/asr/frontend/adapters_utils/README.md

espnet2/asr/frontend/adapters_utils/adapters/adapter.py

espnet2/asr/frontend/adapters_utils/adapters/adapter_transformer.py

espnet2/asr/frontend/adapters_utils/add_adapters.py

espnet2/asr/frontend/adapters_utils/example.py

…er.py Co-authored-by: xuankai@cmu <[email protected]>

Adapter fix

for more information, see https://pre-commit.ci

simpleoier · 2023-04-01T13:17:13Z

espnet2/asr/frontend/adapters_utils/README.md

@@ -0,0 +1,43 @@
+## Use adapters for ASR in ESPnet2


Can you also add something about adapter in the espnet top level readme and tutorial doc as well?

Sure thing, I would add a section in the main README

section added in top level README - ASR

Hi, where would be a good place to add in the tutorial docs? Right now I have the full tutorial for adapters in adapter_utils/README.md.

How about here? https://github.com/espnet/espnet/blob/master/doc/espnet2_tutorial.md#usage-of-self-supervised-learning-representations-as-feature

Hi just a quick reminder that the updated documentes have been updated to contain the documentation of adapter modules and how to use them. Could you give the PR in its current state a quick look?

sw005320 · 2023-06-12T02:48:57Z

@simpleoier, could I ask you to review this again?

simpleoier

Correct me if I'm wrong. I feel this PR is not complete. The Adapter function is not able to use after this PR. For example, when adding adapter-related configs, where are they used?

egs2/mini_an4/asr1/run_adapters.sh

simpleoier · 2023-06-15T04:26:27Z

espnet2/asr/frontend/adapters_utils/README.md

@@ -0,0 +1,43 @@
+## Use adapters for ASR in ESPnet2


How about here? https://github.com/espnet/espnet/blob/master/doc/espnet2_tutorial.md#usage-of-self-supervised-learning-representations-as-feature

simpleoier · 2023-06-15T04:28:15Z

espnet2/asr/frontend/adapters_utils/adapters/adapter.py

+        self,
+        orig_dim: int,
+        down_dim: int,
+        layer_norm: str = None,


Did you compare the performance of those? If not, I think we can do 1 as default.

espnet2/asr/frontend/adapters_utils/add_adapters.py

for more information, see https://pre-commit.ci

sw005320 · 2023-06-28T11:27:42Z

Can you add unit test and integration test?

for more information, see https://pre-commit.ci

jwrh · 2023-07-08T19:23:21Z

Hi, I've added some unit testing (tentative) for adapters modules. Integration testing, I think, can be added in the training scheme PR as discussed earlier here.

simpleoier

Thanks for the updates! Please address the CI test errors first.

simpleoier · 2023-07-10T15:43:12Z

egs2/mini_an4/asr1/conf/adapter_example.yaml

Can you clean this config a bit? For example, the number of white space for each indent, unnecessary empty lines in each block, etc.

simpleoier · 2023-07-10T15:44:16Z

espnet2/asr/frontend/adapters_utils/adapters/adapter_transformer.py

+    Modified fairseq's TransformerSentenceEncoderLayer for wav2vec2 with adapters.
+    Link:
+    https://github.com/
+    facebookresearch/fairseq/blob/


minor comment: why new line for this?

simpleoier · 2023-07-10T15:45:25Z

espnet2/asr/frontend/adapters_utils/adapters/adapter_transformer.py

+        activation_fn: str = "relu",
+        layer_norm_first: bool = False,
+        adapter_down_dim: int = 192,
+    ) -> None:


Please add some comments to explain the arguments here. So new users can easily understand what to tune.

simpleoier · 2023-07-10T15:46:26Z

espnet2/asr/frontend/adapters_utils/adapters/adapter_transformer.py

+        self_attn_padding_mask: torch.Tensor = None,
+        need_weights: bool = False,
+        att_args=None,
+    ) -> torch.Tensor:


Ditto. It's better to add some the shape information of each input tensor. It may help users to debug.

simpleoier · 2023-07-10T15:48:17Z

espnet2/asr/frontend/adapters_utils/add_adapters.py

+
+        return
+
+    # freeze all layers


A minor suggestion: is it cleaner / easier to understand if you merge this and the following parameter-freezing lines?

simpleoier · 2023-07-10T15:49:51Z

doc/espnet2_tutorial.md

+Note this is done for model training, so modifications would be at __stage 11__ of ESPnet2 recipes.
+### Prerequisites
+1. Install [S3PRL](https://github.com/s3prl/s3prl) by `tools/installers/install_s3prl.sh`.
+2. Wav2Vec is needed, [fairseq](https://github.com/pytorch/fairseq) should be installed by `tools/installers/install_fairseq.sh`.


Can you double check this? I think the s3prl now doesn't depend on the fairseq to use adapters. If necessary, you may need to update your adapter code a bit.

Thanks for the catch! Yes, indeed the fairseq part is unnecessary since s3prl has a prototype of TransformerSentenceEncoderLayer in their codebase which I can refer to when testing the whole module (Originally importing fairseq was for using the TransformerSentenceEncoderLayer at testing stage). Will remove this prereq in the next commit

simpleoier · 2023-07-10T15:54:54Z

test/espnet2/asr/frontend/test_adapter_utils.py

+from espnet2.asr.frontend.adapter_utils import *
+
+
+def test_add_adapters_wav2vec2():


Does this code support other ssl models, e.g. hubert / wavlm? Another concern is that we need to download the wav2vec2 checkpoint if we use this. It will slow down the CI test a lot and may cause some issues if downloading fails. Can you change some configs to avoid it?

Currently the code does not. I am certainly looking to support those! Do you think it it's better to do with currently wav2vec2 only or support those model within this PR?

The current code gets around downloading the wav2vec2 checkpoint by onlying simulating with 3 TransformerSentenceEncoderLayer s and adding adapters to 2 of them. Does this work?

Thanks for the information.

Can you please remind me if you support other models in your paper?

OK. If it skip downloading ckpt during, it is good. But another concern is how you ensure that it is compatible if fairseq updates the related implementation?

1.We did also support HuBERT in our paper, do you prefer integrating that part of code in this PR as well?
2.Yes, the problem about updated implementation is very real and it seems (from my view) that we could either do

pulling the wav2vec2.0 model and assume we don't know its internal implementation. This way I think we do not need to worry about fairseq updating implementation. But it's costly to pull the model and run them

or current way which gets around the need of pulling the whole model. But this way assumes knowledge of wav2vec's implementation and may fail if implementation are updated.
which one do we prefer? Any other way of testing is greatly appreciated!

github-actions · 2025-06-27T02:12:32Z

This PR is stale because it has been open for 90 days with no activity.
It will be closed if no further activity occurs.
Thank you for your contributions.

github-actions · 2025-07-12T02:13:20Z

This PR is closed. Please re-open if needed.

adapter modules update

9aa5ca2

mergify bot added ESPnet2 README labels Mar 19, 2023

pre-commit-ci bot and others added 4 commits March 19, 2023 19:53

[pre-commit.ci] auto fixes from pre-commit.com hooks

d3c1e32

for more information, see https://pre-commit.ci

training scheme update

f0fe178

Merge branch 'master' into master

314226d

[pre-commit.ci] auto fixes from pre-commit.com hooks

c0c0d69

for more information, see https://pre-commit.ci

sw005320 requested a review from simpleoier March 21, 2023 12:24

sw005320 added New Features ASR Automatic speech recogntion labels Mar 21, 2023

sw005320 added this to the v.202303 milestone Mar 21, 2023

simpleoier reviewed Mar 21, 2023

View reviewed changes

jwrh mentioned this pull request Mar 26, 2023

Training scheme update #5035

Closed

jwrh and others added 13 commits March 26, 2023 16:39

make espnet#5035 mutuually exclusive from espnet#5034

669dd21

adapter readme

29e8475

Update espnet2/asr/frontend/adapters_utils/adapters/adapter_transform…

113e81b

…er.py Co-authored-by: xuankai@cmu <[email protected]>

fixed according to comment

87668d7

update

11eb927

Merge pull request #2 from jwrh/adapter

b6bcea7

Adapter fix

[pre-commit.ci] auto fixes from pre-commit.com hooks

e1591d6

for more information, see https://pre-commit.ci

Merge branch 'espnet:master' into master

3cb3dc8

this is an attempt to create a clean master

f117cc4

update adapter.py

affa0be

[pre-commit.ci] auto fixes from pre-commit.com hooks

4b820a2

for more information, see https://pre-commit.ci

update add_adapters.py

94fab0f

[pre-commit.ci] auto fixes from pre-commit.com hooks

7ee475e

for more information, see https://pre-commit.ci

simpleoier reviewed Apr 1, 2023

View reviewed changes

jwrh and others added 3 commits April 8, 2023 15:33

Merge branch 'espnet:master' into master

c04b036

update README

89af3ed

correct README https pointer

5cfd732

simpleoier reviewed Jun 15, 2023

View reviewed changes

jwrh added 2 commits June 17, 2023 17:49

add adapters in docs

3cb65a6

remove local adapter README, documentations moved to main docs

4b544dc

mergify bot added the Documentation label Jun 17, 2023

pre-commit-ci bot and others added 3 commits June 17, 2023 21:52

[pre-commit.ci] auto fixes from pre-commit.com hooks

c00443d

for more information, see https://pre-commit.ci

example adapter config file

3ac59d0

[pre-commit.ci] auto fixes from pre-commit.com hooks

986c71c

for more information, see https://pre-commit.ci

jwrh and others added 2 commits July 8, 2023 15:15

added(tentative) unit testing for adapters moduels

924cd29

[pre-commit.ci] auto fixes from pre-commit.com hooks

19605d1

for more information, see https://pre-commit.ci

simpleoier reviewed Jul 10, 2023

View reviewed changes

kan-bayashi modified the milestones: v.202307, v.202312 Aug 3, 2023

kan-bayashi modified the milestones: v.202310, v.202312 Oct 25, 2023

Stanwang1210 mentioned this pull request Jan 5, 2024

Integrate adapter for s3prl frontend #5609

Merged

kan-bayashi modified the milestones: v.202312, v.202405 Feb 6, 2024

Fhrozen modified the milestones: v.202409, v.202412 Oct 1, 2024

Fhrozen modified the milestones: v.202412, v.202503 Dec 4, 2024

Fhrozen modified the milestones: v.202503, v.202506 Mar 27, 2025

github-actions bot added the Stale For probot label Jun 27, 2025

github-actions bot closed this Jul 12, 2025

		from espnet2.asr.frontend.adapter_utils import *


		def test_add_adapters_wav2vec2():

Conversation

jwrh commented Mar 19, 2023

Uh oh!

simpleoier left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jwrh Apr 8, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sw005320 commented Jun 12, 2023

Uh oh!

simpleoier left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sw005320 commented Jun 28, 2023

Uh oh!

jwrh commented Jul 8, 2023

Uh oh!

simpleoier left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jwrh Jul 24, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jun 27, 2025

Uh oh!

github-actions bot commented Jul 12, 2025

Uh oh!

Reviewers

Assignees

jwrh Apr 8, 2023 •

edited

Loading

jwrh Jul 24, 2023 •

edited

Loading