feat: add OneCycleLR and CyclicLR scheduler #657
Conversation
4b4ce59 to
b5d88b0
Compare
mindcv/scheduler/dynamic_lr.py
Outdated
| return lrs | ||
|
|
||
|
|
||
| def one_cycle_lr(pct_start, anneal_strategy, three_phase, min_lr, max_lr, *, lr, steps_per_epoch, epochs): |
train.py
Outdated
| three_phase=args.three_phase, | ||
| step_size_up=args.step_size_up, | ||
| step_size_down=args.step_size_down, | ||
| decay_mode=args.decay_mode, |
There was a problem hiding this comment.
这堆decay参数过于confusing. 建议优化命名，或者加个前缀区分下。考虑精简下参数个数。如：
decay_strategy -> oc_decay_strategy
three_phase -> is_oc_three_phase
step_size_up -> oc_ascend_steps
step_size_down -> oc_descend_steps
There was a problem hiding this comment.
这个可能先不用暴露出来吧，感觉太过复杂。在create_scheduler这里不传入这些参数
config.py
Outdated
| help='Learning rate (default=0.001)') | ||
| group.add_argument('--min_lr', type=float, default=1e-6, | ||
| help='The minimum value of learning rate if scheduler supports (default=1e-6)') | ||
| group.add_argument('--max_lr', type=float, default=2.0, |
There was a problem hiding this comment.
这个是不是没有必要加，lr就能代表max_lr
There was a problem hiding this comment.
不行，根据论文中的设置，resnet one cycle的学习率变化为0.05（lr）到1.0（max_lr），最后下降至0.00005（min_lr）
There was a problem hiding this comment.
而且这样也无法使用warm_up 来达到初始学习率了
mindcv/scheduler/dynamic_lr.py
Outdated
| return lrs | ||
|
|
||
|
|
||
| def one_cycle_refined_lr(pct_start, anneal_strategy, three_phase, min_lr, max_lr, *, lr, steps_per_epoch, epochs): |
There was a problem hiding this comment.
这个应该没有refined吧，一直都是随step变化的
train.py
Outdated
| three_phase=args.three_phase, | ||
| step_size_up=args.step_size_up, | ||
| step_size_down=args.step_size_down, | ||
| decay_mode=args.decay_mode, |
There was a problem hiding this comment.
这个可能先不用暴露出来吧，感觉太过复杂。在create_scheduler这里不传入这些参数
36354b1 to
ba4c0e1
Compare
| final_div_factor = initial_lr / min_lr | ||
| main_lr_scheduler = one_cycle_lr( | ||
| max_lr=lr, | ||
| pct_start=0.3, |
There was a problem hiding this comment.
这些默认值的，不用写在factory里就行
Thank you for your contribution to the MindCV repo.
Before submitting this PR, please make sure:
Motivation
Add OneCycleLR and CyclicLR scheduler
Test Plan
It's tested in dynamic_lr.
Related Issues and PRs
(Is this PR part of a group of changes? Link the other relevant PRs and Issues here. Use https://help.github.com/en/articles/closing-issues-using-keywords for help on GitHub syntax)