Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Support LoRA based large model finetuning.#5400

Merged
sw005320 merged 14 commits intoespnet:masterfrom
pengchengguo:whisper
Oct 11, 2023
Merged

Support LoRA based large model finetuning.#5400
sw005320 merged 14 commits intoespnet:masterfrom
pengchengguo:whisper

Conversation

@pengchengguo
Copy link
Collaborator

@pengchengguo pengchengguo commented Aug 9, 2023

What?

Support LoRA-based large model finetuning and also provide results of LoRA-based Whisper finetuning on Aishell corpus.

By finetuning the Whisper large model, we are able to achieve superior results of 2.5/2.7 CERs on dev/test sets, respectively.

See also

  1. LoRA: Low-Rank Adaptation of Large Language Models (https://arxiv.org/pdf/2106.09685.pdf)
  2. https://github.com/microsoft/LoRA.git
  3. https://github.com/huggingface/peft

@codecov
Copy link

codecov bot commented Aug 9, 2023 •
edited
Loading

Codecov Report

Merging #5400 (97f855c) into master (71dc9a3) will decrease coverage by 6.82%.
Report is 187 commits behind head on master.
The diff coverage is 86.61%.

@@            Coverage Diff             @@
##           master    #5400      +/-   ##
==========================================
- Coverage   77.14%   70.33%   -6.82%     
==========================================
  Files         684      707      +23     
  Lines       62713    64932    +2219     
==========================================
- Hits        48383    45673    -2710     
- Misses      14330    19259    +4929
Flag Coverage Δ
test_configuration_espnet2 ∅ <ø> (∅)
test_integration_espnet1 ?
test_integration_espnet2 48.76% <53.27%> (-0.31%) ⬇️
test_python_espnet1 19.19% <1.57%> (-0.76%) ⬇️
test_python_espnet2 51.30% <78.74%> (-1.01%) ⬇️
test_utils 23.10% <66.66%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Coverage Δ
espnet2/bin/whisper_export_vocabulary.py 93.22% <100.00%> (+0.36%) ⬆️
espnet2/text/build_tokenizer.py 78.37% <ø> (ø)
espnet2/text/whisper_token_id_converter.py 89.47% <100.00%> (+3.36%) ⬆️
espnet2/text/whisper_tokenizer.py 89.18% <100.00%> (+0.95%) ⬆️
espnet2/train/preprocessor.py 77.31% <100.00%> (-0.12%) ⬇️
espnet/nets/e2e_mt_common.py 63.93% <66.66%> (-0.48%) ⬇️
espnet2/bin/asr_inference.py 87.50% <83.33%> (+0.51%) ⬆️
espnet2/tasks/st.py 86.66% <88.88%> (-1.66%) ⬇️
espnet2/layers/create_lora_adapter.py 95.74% <95.74%> (ø)
espnet2/tasks/abs_task.py 76.84% <83.33%> (+0.04%) ⬆️
... and 3 more

... and 102 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@sw005320 sw005320 requested a review from pyf98 August 9, 2023 11:14
@sw005320 sw005320 added New Features ASR Automatic speech recogntion labels Aug 9, 2023
@sw005320 sw005320 added this to the v.202312 milestone Aug 9, 2023
@sw005320
Copy link
Contributor

sw005320 commented Aug 9, 2023

Thanks, @pengchengguo!
The results are very impressive.
Can you also add a test so that this implementation becomes more solid?

@pengchengguo
Copy link
Collaborator Author

pengchengguo commented Aug 9, 2023

Sure, the HF model link will also be updated later.

@sw005320
Copy link
Contributor

sw005320 commented Aug 30, 2023

@pyf98, can you review this PR?

@pengchengguo, can you add a test?

@pyf98
Copy link
Collaborator

pyf98 commented Sep 17, 2023

Thanks! It looks good to me!

@pyf98
Copy link
Collaborator

pyf98 commented Sep 17, 2023 •
edited
Loading

Just one question:
Does it support LoRA for all ESPnet models? Or just Whisper? I guess it should support all.

@pengchengguo
Copy link
Collaborator Author

pengchengguo commented Sep 18, 2023

Yes, it supports all.

@pengchengguo pengchengguo changed the title Support LoRA based large model finetuning. [WIP] Support LoRA based large model finetuning. Sep 18, 2023
@pengchengguo
Copy link
Collaborator Author

pengchengguo commented Sep 18, 2023

I am changing this PR to "work in progress" because I also want to add Whisper fine-tuning to ST. The ST part is being refactored now and will be finished soon.

@mergify mergify bot added the ESPnet1 label Sep 21, 2023
@mergify
Copy link
Contributor

mergify bot commented Sep 23, 2023

This pull request is now in conflict :(

@mergify mergify bot added the conflicts label Sep 23, 2023
@mergify mergify bot removed the conflicts label Sep 24, 2023
@sw005320
Copy link
Contributor

sw005320 commented Sep 30, 2023

Can you remove “WIP” from the PR title?

@pengchengguo pengchengguo changed the title [WIP] Support LoRA based large model finetuning. Support LoRA based large model finetuning. Sep 30, 2023
@pengchengguo
Copy link
Collaborator Author

pengchengguo commented Sep 30, 2023

Sure, now I am trying to increase the codecov, it seems the CI test file misses some functions.

@mergify mergify bot added Installation CI Travis, Circle CI, etc labels Sep 30, 2023
@sw005320
Copy link
Contributor

sw005320 commented Oct 10, 2023

We seem to have an issue for CI. Can you check it?

@pengchengguo
Copy link
Collaborator Author

pengchengguo commented Oct 11, 2023

I fixed the CI issues and ST errors as I mentioned before. Now, Whisper fine-tuned ST shows better results compared with E-Branchformer (60.9 vs 53 BLEU scores on Fishercallhome dev sets).
I need time to decode all test sets and upload the pre-trained model, so I want to do it in another PR.
For this PR, I think it's okay to be merged.

@sw005320 sw005320 merged commit 390589a into espnet:master Oct 11, 2023
@sw005320
Copy link
Contributor

sw005320 commented Oct 11, 2023

Thanks, @pengchengguo!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pyf98 pyf98 Awaiting requested review from pyf98

Assignees

No one assigned

Labels

ASR Automatic speech recogntion CI Travis, Circle CI, etc ESPnet1 ESPnet2 Installation New Features README

Projects

None yet

Milestone

v.202310

Development

Successfully merging this pull request may close these issues.

3 participants

@pengchengguo @sw005320 @pyf98