Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Commit 50cd4cf

Browse files
LiuChiachiLiuChiachi
authored
Update dataset API usage in distill lstm (PaddlePaddle#85)
* update dataset usage in distill lstm * update usage of chnsenticorp * use map fn in data augmentation * fix paddlenlp readme typo * convert chnsenticorp to uppercase
1 parent a093555 commit 50cd4cf

8 files changed

Lines changed: 151 additions & 174 deletions

File tree

‎README_en.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,7 @@ from paddlenlp.datasets import load_dataset
4747
train_ds, dev_ds, test_ds = load_dataset("chnsenticorp", splits=["train", "dev", "test"])
4848
```
4949

50-
### Chinese Text Emebdding Loading
50+
### Chinese Text Embedding Loading
5151

5252
```python
5353

@@ -60,7 +60,7 @@ wordemb.cosine_sim("艺术", "火车")
6060
>>> 0.14792643
6161
```
6262

63-
### Rich Chinsese Pre-trained Models
63+
### Rich Chinese Pre-trained Models
6464

6565

6666
```python
@@ -129,7 +129,7 @@ Please refer to our official AI Studio account for more interactive tutorials: [
129129
* [Waybill Information Extraction with BiGRU-CRF Model](https://aistudio.baidu.com/aistudio/projectdetail/1317771) shows how to make use of Bi-GRU plus CRF to finish information extraction task.
130130

131131
* [Waybill Information Extraction with ERNIE](https://aistudio.baidu.com/aistudio/projectdetail/1329361) shows how to use ERNIE, the Chinese pre-trained model improve information extraction performance.
132-
132+
133133
* [Use TCN Model to predict COVID-19 confirmed cases](https://aistudio.baidu.com/aistudio/projectdetail/1290873)
134134

135135

‎examples/model_compression/distill_lstm/README.md‎

Lines changed: 17 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -64,7 +64,7 @@ python -u ./run_glue.py \
6464
--num_train_epochs 3 \
6565
--logging_steps 10 \
6666
--save_steps 10 \
67-
--output_dir ../model_compression/distill_lstm/pretrained_modelss/$TASK_NAME/ \
67+
--output_dir ../model_compression/distill_lstm/pretrained_models/$TASK_NAME/ \
6868
--n_gpu 1 \
6969

7070
```
@@ -81,7 +81,7 @@ python -u ./run_glue.py \
8181

8282
```shell
8383
CUDA_VISIBLE_DEVICES=0 python small.py \
84-
--task_name senta \
84+
--task_name chnsenticorp \
8585
--max_epoch 20 \
8686
--vocab_size 1256608 \
8787
--batch_size 64 \
@@ -90,7 +90,8 @@ CUDA_VISIBLE_DEVICES=0 python small.py \
9090
--lr 3e-4 \
9191
--dropout_prob 0.2 \
9292
--vocab_path senta_word_dict.txt \
93-
--output_dir small_models/senta/
93+
--save_steps 10000 \
94+
--output_dir small_models/chnsenticorp/
9495

9596
```
9697

@@ -103,6 +104,7 @@ CUDA_VISIBLE_DEVICES=0 python small.py \
103104
--lr 1.0 \
104105
--dropout_prob 0.4 \
105106
--output_dir small_models/SST-2 \
107+
--save_steps 10000 \
106108
--embedding_name w2v.google_news.target.word-word.dim300.en
107109

108110
```
@@ -116,6 +118,7 @@ CUDA_VISIBLE_DEVICES=0 python small.py \
116118
--lr 2.0 \
117119
--dropout_prob 0.4 \
118120
--output_dir small_models/QQP \
121+
--save_steps 10000 \
119122
--embedding_name w2v.google_news.target.word-word.dim300.en
120123

121124
```
@@ -125,16 +128,17 @@ CUDA_VISIBLE_DEVICES=0 python small.py \
125128

126129
```shell
127130
CUDA_VISIBLE_DEVICES=0 python bert_distill.py \
128-
--task_name senta \
131+
--task_name chnsenticorp \
129132
--vocab_size 1256608 \
130133
--max_epoch 6 \
131134
--lr 1.0 \
132135
--dropout_prob 0.1 \
133136
--batch_size 64 \
134137
--model_name bert-wwm-ext-chinese \
135-
--teacher_path pretrained_models/senta/best_bert_wwm_ext_model_880/model_state.pdparams \
138+
--teacher_path pretrained_models/chnsenticorp/best_bert_wwm_ext_model_880/model_state.pdparams \
136139
--vocab_path senta_word_dict.txt \
137-
--output_dir distilled_models/senta
140+
--output_dir distilled_models/chnsenticorp \
141+
--save_steps 10000 \
138142

139143
```
140144

@@ -148,9 +152,10 @@ CUDA_VISIBLE_DEVICES=0 python bert_distill.py \
148152
--dropout_prob 0.2 \
149153
--batch_size 128 \
150154
--model_name bert-base-uncased \
151-
--embedding_name w2v.google_news.target.word-word.dim300.en \
152155
--output_dir distilled_models/SST-2 \
153-
--teacher_path pretrained_models/SST-2/best_model_610/model_state.pdparams
156+
--teacher_path pretrained_models/SST-2/best_model_610/model_state.pdparams \
157+
--save_steps 10000 \
158+
--embedding_name w2v.google_news.target.word-word.dim300.en \
154159

155160
```
156161

@@ -163,18 +168,19 @@ CUDA_VISIBLE_DEVICES=0 python bert_distill.py \
163168
--dropout_prob 0.2 \
164169
--batch_size 256 \
165170
--model_name bert-base-uncased \
166-
--embedding_name w2v.google_news.target.word-word.dim300.en \
167171
--n_iter 10 \
168172
--output_dir distilled_models/QQP \
169-
--teacher_path pretrained_models/QQP/best_model_17000/model_state.pdparams
173+
--teacher_path pretrained_models/QQP/best_model_17000/model_state.pdparams \
174+
--save_steps 10000 \
175+
--embedding_name w2v.google_news.target.word-word.dim300.en \
170176

171177
```
172178

173179
各参数的具体说明请参阅 `args.py` ，注意在训练不同任务时，需要调整对应的超参数。
174180

175181

176182
## 蒸馏实验结果
177-
本蒸馏实验基于GLUE的SST-2、QQP、中文情感分类ChnSentiCorp数据集。实验效果均使用每个数据集的验证集（dev）进行评价，评价指标是准确率（acc），其中QQP中包含f1值。利用基于BERT的教师模型去蒸馏基于Bi-LSTM的学生模型，对比Bi-LSTM小模型单独训练，在SST-2、QQP、senta(中文情感分类)任务上分别有3.3%、1.9%、1.4%的提升。
183+
本蒸馏实验基于GLUE的SST-2、QQP、中文情感分类ChnSentiCorp数据集。实验效果均使用每个数据集的验证集（dev）进行评价，评价指标是准确率（acc），其中QQP中包含f1值。利用基于BERT的教师模型去蒸馏基于Bi-LSTM的学生模型，对比Bi-LSTM小模型单独训练，在SST-2、QQP、ChnSentiCorp(中文情感分类)任务上分别有3.3%、1.9%、1.4%的提升。
178184

179185
| Model | SST-2(dev acc) | QQP(dev acc/f1) | ChnSentiCorp(dev acc) | ChnSentiCorp(dev acc) |
180186
| ----------------- | ----------------- | -------------------------- | --------------------- | --------------------- |

‎examples/model_compression/distill_lstm/bert_distill.py‎

Lines changed: 5 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -21,16 +21,15 @@
2121

2222
from paddlenlp.transformers import BertForSequenceClassification
2323
from paddlenlp.metrics import AccuracyAndF1
24-
from paddlenlp.datasets import GlueSST2, GlueQQP, ChnSentiCorp
2524

2625
from args import parse_args
2726
from small import BiLSTM
2827
from data import create_distill_loader
2928

30-
TASK_CLASSES = {
31-
"sst-2": (GlueSST2, Accuracy),
32-
"qqp": (GlueQQP, AccuracyAndF1),
33-
"senta": (ChnSentiCorp, Accuracy),
29+
METRIC_CLASSES = {
30+
"sst-2": Accuracy,
31+
"qqp": AccuracyAndF1,
32+
"chnsenticorp": Accuracy
3433
}
3534

3635

@@ -98,7 +97,7 @@ def do_train(agrs):
9897
mse_loss = nn.MSELoss()
9998
klloss = nn.KLDivLoss()
10099

101-
metric_class = TASK_CLASSES[args.task_name][1]
100+
metric_class = METRIC_CLASSES[args.task_name]
102101
metric = metric_class()
103102

104103
teacher = TeacherModel(

0 commit comments

Comments
 (0)