Codestin Search App

Stanwang1210 · 2024-01-05T08:09:51Z

What?

The PR is to support add adapter to s3prl frontend SSL models.
I integrate it with original lora implementation.
Currently, it only supports houlsby adapter. However, we can easily integrate other types of adapter very soon.
Two config yaml give examples of the configuration of adapter

TODO: Havn't done with the adapter load/save function. Will handle it soon

Why?

The original lora implementation are not compatible with other kinds of adapter.

For the lora case, I didn't modified any code except for the entry to create_lora_adapter function.
So the main modification is at the houlsby adapter case.
For those target layers idx assigned in the adapter_conf ,
model.frontend.upstream.upstream.model.encoder.layers.idx will become HoulsbyTransformerSentenceEncoderLayer.

Basically, it looks like

Did I make it clear?

sw005320 · 2024-01-08T12:42:20Z

Cool.
Can you add some usages to https://github.com/espnet/espnet/blob/master/egs2/ml_superb/asr1/README.md?

@ftshijt, by following the convention, it would be better to have the result and model link.

Stanwang1210 · 2024-01-17T08:00:52Z

Sorry for the late reply
Now I add model link and result in the README.
Please let me know if I miss anything

@ftshijt

ftshijt

Thanks for the updates! The current implementation seems cool to me, but I would like to hold more discussions on some design points.

ftshijt · 2024-01-25T02:14:53Z

espnet2/train/trainer.py

+            if adapter == "lora" and lora is None:
+                raise RuntimeError("Requiring loralib. Do 'pip install loralib'")
+
+            # TODO: houlsby adapter may need S3PRL?


Is it possible to make it general to other modules as well? If that's difficult, we may keep only to s3prl for now.

In that case, you may consider add a check to ensure that s3prl is installed when using the houlsby adapter.

Is this update compatible with pretrained models which have configs use_lora and save_lora_only?

Sorry for the late reply.
For @ftshijt questions, I think currently it's difficult to integrate adapters and lora together. The reason is adapter implementation needs to modify the forward function of SSL models, like here, while LoRa did not. Therefore, it may requires some efforts to integrate them together.

Sorry for the late reply.
@simpleoier I think it's compatible but need to do some revision.
And I did it by rename use_lora to use_adapter (more general), and the same way with save_lora_only.
The reason is that in most case, we add lora only on the pre-trained model. However, in SSL settings, most of the time we need to initialize a downstream model for each task. That is to say, when applying lora to SSL models as adapters, we need to save not only the lora parameters but also the downstream model and other tunable parameters. Therefore, I choose to save all parameters requires_grad = True.

I made some revisions here.
Please check d7f0b39

ftshijt · 2024-01-25T02:17:43Z

espnet2/train/trainer.py

+                # TODO: This kind of design may not facilitate the integration of other kind of the adapter
+                # Maybe we can only save those with requires_grad = True ?
+                # If we use lora.lora_state_dict, we will not save the downstream model in SSL settings


I do not have a clear question to this one. But I feel only save models with requires_grad = True is a bit risky.

Ping @sw005320 @wanchichen @simpleoier @pyf98 for some discussion here.

I think so. One case is that it can fail if the pretrained SSL models are updated.

For that purpose, I would suggest we simply follow the setting with the current design.

The reason why I choose to save all parameters with requires_grad = True is that in adapter setting, we usually need to save the downstream model or other tunable parameters together (Ex, the weighted_sum). Therefore, we may not able to follow what loralib did, which only save lora_layers.

I understand your concern about the current design. However, for the case SSL models are updated, I think it would be better to save the SSL weights (or I can not see why we need to update it without saving it).

If we have to follow what loralib does by offering an options which saves the adapter parameter only, then I would like to add another option for saving parameters with requires_grad = True. In this case, we can fulfill the requirement but also make sure adapter and lora can work in s3prl settings.

Do you think it's a good idea?

I made some revisions here.
Please check d7f0b39

@ftshijt @simpleoier

mergify · 2024-01-30T07:24:49Z

This pull request is now in conflict :(

ftshijt · 2024-01-30T07:25:30Z

Could you fix the conflicts and update the pre-trained model in the readme session? then we can move forward to merge it as it is a dependency for some other projects~

mergify · 2024-02-06T02:48:30Z

This pull request is now in conflict :(

sw005320 · 2024-02-11T14:24:15Z

Please fix the above conflict.

for more information, see https://pre-commit.ci

Stanwang1210 · 2024-02-21T11:31:20Z

It seems like the CI error happened due to some weird reason. Is this caused by my PR?

sw005320 · 2024-02-21T12:13:48Z

It's not related to your PR.
This happens accidentally.
I reran the CI.

sw005320 · 2024-02-21T12:42:20Z

@ftshijt, do you know why codecov complains about this?
I think @Stanwang1210 correctly prepared the test.

If this is due to some issues in codecov, we can ignore it and merge this PR.

ftshijt · 2024-02-21T15:30:08Z

@ftshijt, do you know why codecov complains about this? I think @Stanwang1210 correctly prepared the test.

If this is due to some issues in codecov, we can ignore it and merge this PR.

Not exactly from the codecov, I think it is mainly due to an in-successfully running of other CI tests (i.e., the time out for vits decoding), which automatically stop some running CIs, resulting in no execuation of the test function from Stan.

sw005320 · 2024-02-22T13:10:30Z

Thanks, @Stanwang1210!

pengchengguo · 2024-09-11T04:01:46Z

It seems like we missed lora.mark_only_lora_as_trainable(model) in espnet2/layers/create_adapter_fn.py and all params will be optimized even with lora finetuning.

sw005320 · 2024-09-11T07:45:42Z

Thanks @pengchengguo!
Can you explain it a bit more?

pengchengguo · 2024-09-11T07:58:04Z

Certainly. Before the integration, the LoRA fine-tuning process was implemented as follows:

## find target param -> add LoRA layer -> make only LoRA params trainable
for key in key_list:
    parent_module, target_name, target_module = get_submodules(model, key)
    if not isinstance(target_module, lora.LoRALayer):
        new_module = create_new_module(target_module, rank, alpha, dropout_rate)
        replace_module(parent_module, target_name, target_module, new_module)
    else:
        continue

lora.mark_only_lora_as_trainable(model)

After the PR, the last line lora.mark_only_lora_as_trainable(model) was missing, which causes all parameters to be updated, even when using LoRA.

I’m not sure if it’s implemented elsewhere and I didn't find it.

sw005320 · 2024-09-11T08:02:43Z

Oh, that would be a big issue.
@Stanwang1210, can you double-check it?

ftshijt · 2024-09-11T08:06:19Z

I remember we used to fix the issue through this PR. #5726 but it seems not available anymore.

Stanwang1210 · 2024-09-11T15:56:24Z

@pengchengguo

Sorry for the late reply!
Based on the discussion in PR #5726, we get to the conclusion that using freeze_param option should be fine.
If you have any questions about freeze_param option, feel free to let us know!

pengchengguo · 2024-09-12T03:05:41Z

I see. Thanks for your responding.

init adapter module

f6f5c6b

mergify bot added the ESPnet2 label Jan 5, 2024

sw005320 added New Features ASR Automatic speech recogntion labels Jan 5, 2024

sw005320 added this to the v.202312 milestone Jan 5, 2024

revise type check

c366908

Stanwang1210 force-pushed the master branch from 461bb33 to c366908 Compare January 6, 2024 10:55

add save adapter only

b111dca

Stanwang1210 force-pushed the master branch from 242dd94 to b111dca Compare January 10, 2024 04:31

add readme && fix some bugs

6e8c4dc

Stanwang1210 force-pushed the master branch from 9b48a6d to 6e8c4dc Compare January 14, 2024 03:35

mergify bot added the README label Jan 14, 2024

Stanwang1210 added 2 commits January 16, 2024 19:36

fix saving bug

aa855df

Add Model link && Add Result

2e58b32

Stanwang1210 force-pushed the master branch from d570ae2 to 2e58b32 Compare January 17, 2024 07:57

ftshijt reviewed Jan 25, 2024

View reviewed changes

mergify bot added the conflicts label Jan 30, 2024

kan-bayashi modified the milestones: v.202312, v.202405 Feb 6, 2024

revise save strategy

d7f0b39

Stanwang1210 force-pushed the master branch from 5173e4b to d7f0b39 Compare February 10, 2024 04:11

add config && fix typo

c13bea2

Stanwang1210 and others added 17 commits February 21, 2024 00:12

add create_adapter test case

723f5dc

Merge branch 'master' of https://github.com/Stanwang1210/espnet

98d0fb1

[pre-commit.ci] auto fixes from pre-commit.com hooks

3f7367a

for more information, see https://pre-commit.ci

add test case

cc8d65f

Merge branch 'master' of https://github.com/Stanwang1210/espnet

9d174e1

[pre-commit.ci] auto fixes from pre-commit.com hooks

65039ab

for more information, see https://pre-commit.ci

add import skip

915b4a3

Merge branch 'master' of https://github.com/Stanwang1210/espnet

925f4df

[pre-commit.ci] auto fixes from pre-commit.com hooks

380012f

for more information, see https://pre-commit.ci

fix typo

2bf76fb

Merge branch 'master' of https://github.com/Stanwang1210/espnet

9dc0a15

[pre-commit.ci] auto fixes from pre-commit.com hooks

abc4c2c

for more information, see https://pre-commit.ci

add test s3prl import

be491c4

fix conflict

c73b887

Merge branch 'master' of https://github.com/Stanwang1210/espnet

5ed233f

[pre-commit.ci] auto fixes from pre-commit.com hooks

405423d

for more information, see https://pre-commit.ci

Merge branch 'master' into master

6f1ae7d

sw005320 merged commit 98b0387 into espnet:master Feb 22, 2024

Conversation

Stanwang1210 commented Jan 5, 2024

What?

Why?

See also

Uh oh!

sw005320 commented Jan 5, 2024

Uh oh!

Stanwang1210 commented Jan 6, 2024

Uh oh!

sw005320 commented Jan 8, 2024

Uh oh!

Stanwang1210 commented Jan 17, 2024

Uh oh!

ftshijt left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Stanwang1210 Feb 7, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Stanwang1210 Feb 7, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mergify bot commented Jan 30, 2024

Uh oh!

ftshijt commented Jan 30, 2024

Uh oh!

mergify bot commented Feb 6, 2024

Uh oh!

sw005320 commented Feb 11, 2024

Uh oh!

Stanwang1210 commented Feb 21, 2024

Uh oh!

sw005320 commented Feb 21, 2024

Uh oh!

sw005320 commented Feb 21, 2024

Uh oh!

ftshijt commented Feb 21, 2024

Uh oh!

sw005320 commented Feb 22, 2024

Uh oh!

pengchengguo commented Sep 11, 2024

Uh oh!

sw005320 commented Sep 11, 2024

Uh oh!

pengchengguo commented Sep 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sw005320 commented Sep 11, 2024

Uh oh!

ftshijt commented Sep 11, 2024

Uh oh!

Stanwang1210 commented Sep 11, 2024

Uh oh!

pengchengguo commented Sep 12, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Stanwang1210 Feb 7, 2024 •

edited

Loading

Stanwang1210 Feb 7, 2024 •

edited

Loading

pengchengguo commented Sep 11, 2024 •

edited

Loading