This is the sample training script for the ASR task with espnetez.
All you need is to build a dump file and pass it to the trainer.
You can easily join several datasets by just joining the dump file.

import espnetez as ez

# dataset information
# the format of wav.scp is: <audio_tag><space><file_path>\n
# and the format of text is: <audio_tag><space><text>\n
# Example for wav.scp: audio_1 /database/libri100/train/first.flac
# Example for text: audio_1 HELLO WORLD
data_inputs = {
    "speech": { "file": "wav.scp", "type": "kaldi_ark" },
    "text":{ "file": "text", "type": "text" }
}
train_dump_path = "dump/raw/train_clean_100_sp"
test_dump_path = "dump/raw/test_clean"
output_path = "exp"

# You can use configuration from the ESPnet recipes.
training_config = ez.config.from_yaml("asr", "train_asr_branchformer_e24_amp.yaml")

# and you can update with your config.
preprocessor_config = ez.utils.load_yaml("preprocess.yaml")
training_config.update(preprocessor_config)

# Define trainer
trainer = ez.trainer.Trainer(
	"asr", train_dump_path, test_dump_path, output_path,
	data_inputs, training_config,
	ngpu=1 # you can also update configuration here
)

# If you don't have stats file then you need to run this.
trainer.collect_stats()

# finally run train()
trainer.train()

ToDO

• Add more tasks. (Currently, I added asr/transducer/tts for debugging.)
• Add sentencepiece training
• Add frontend for creating dump files
• Refactor trainer class. Especially the _update_config() function.

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (35b8f01) 76.54% compared to head (c788444) 76.54%.

@@           Coverage Diff           @@
##           master    #5372   +/-   ##
=======================================
  Coverage   76.54%   76.54%           
=======================================
  Files         720      720           
  Lines       66602    66602           
=======================================
  Hits        50978    50978           
  Misses      15624    15624           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@pyf98, can you help this project by reviewing and testing PRs?

@Masao-Someki, can you let me know the progress?
Can you fix the CI and finish refactoring?
I think we can make an LM training step a lower priority.

@sw005320
I'm using this espnetez to train a single ASR model but encountering some issues during training.
For example, the uid variable in the dataloader/dataset becomes float type, and I got assertion error.
I'm not sure why it happens, and debugging if the dump file is properly generated.

I added a demo notebook to train E-branchformer model with Librispeech-100 dataset. (link)
In my environment, the final train() function does not successfully executed on the Jupiter notebook. Maybe we need to run the train() function from command line.

I included a demo notebook on fine-tuning the pre-trained model using LoRA.
ESPnet-Easy simplifies fine-tuning the pretrained model from the Hugging Face hub with a custom dataset.
(Currently, it seems that there is a bug in the training process with the pretrained model.)

Very cool!

@juice500ml, @ftshijt, @simpleoier, can you check this?

Can we provide an example for fine-tuning OWSM (e.g., https://huggingface.co/espnet/owsm_v2_ebranchformer)? It will attract more users.

This PR is awesome!! Can we also consider packaging for pypi, so that people can easily pip install espnetez and directly use this?

Thank you @sw005320, @pyf98, and @juice500ml,

I apologize for the delay in development, but the bug in the fine-tuning process has been successfully fixed.
This PR is now ready for review!

Since I currently have only one GPU on my local machine, I kindly request the reviewer's assistance in checking whether the training process runs successfully with multiple GPUs.

LGTM!

I will fix the CI and add an inference guide to the notebooks to finish this PR!

• Fix CI
• Add inference instruction in notebooks

Thanks a lot, @Masao-Someki!
This is a great first step!