Thanks to visit codestin.com
Credit goes to github.com

Skip to content

fix: fuse optimizer oom and make it stateless#207

Merged
NOBLES5E merged 32 commits into
masterfrom
fuse-opt
Nov 2, 2021
Merged

fix: fuse optimizer oom and make it stateless#207
NOBLES5E merged 32 commits into
masterfrom
fuse-opt

Conversation

@wangraying
Copy link
Copy Markdown
Member

@wangraying wangraying commented Sep 17, 2021 •
edited
Loading

BREAKING CHANGE:

  • contrib: bagua.torch_api.contrib.FusedOptimizer is now removed and replaced by bagua.torch_api.contrib.fuse_optimizer

@wangraying wangraying changed the title fix: fuse optimizer oom and make it stateless [DRFAT] fix: fuse optimizer oom and make it stateless [DRAFT] Sep 17, 2021
@wangraying wangraying requested review from NOBLES5E and removed request for NOBLES5E September 18, 2021 11:27
wangraying
wangraying commented Sep 22, 2021
View reviewed changes
Comment thread bagua/torch_api/contrib/fuse/optimizer.py Outdated
@wangraying wangraying changed the title fix: fuse optimizer oom and make it stateless [DRAFT] fix: fuse optimizer oom and make it stateless Sep 24, 2021
github-actions[bot]
github-actions Bot reviewed Sep 24, 2021
View reviewed changes
Comment thread bagua/torch_api/contrib/fuse/optimizer.py Outdated
@NOBLES5E NOBLES5E mentioned this pull request Oct 28, 2021
Closed
@BaguaSys BaguaSys deleted a comment from todo Bot Nov 1, 2021
@todo
Copy link
Copy Markdown

todo Bot commented Nov 1, 2021

async algorithm conflict with fused optimizer, can only support flattened inplace bucket.

bagua/bagua/torch_api/algorithms/async_model_average.py

Lines 84 to 89 in 2fd9eb2

# TODO: async algorithm conflict with fused optimizer, can only support flattened inplace bucket.
assert do_flatten, "async does not support `do_flatten=False` at present."
if self.step_id < self.warmup_steps:
return super().tensors_to_buckets(tensors, do_flatten)
all_tensors = []

This comment was generated by todo based on a TODO comment in 2fd9eb2 in #207. cc @BaguaSys.

@todo
Copy link
Copy Markdown

todo Bot commented Nov 1, 2021

qadam optimizer maintain `step_id` in its state

bagua/bagua/torch_api/algorithms/q_adam.py

Lines 47 to 50 in 2fd9eb2

# TODO: qadam optimizer maintain `step_id` in its state
self.step_id = 0
self.warmup_steps = warmup_steps

This comment was generated by todo based on a TODO comment in 2fd9eb2 in #207. cc @BaguaSys.

@BaguaSys BaguaSys deleted a comment from todo Bot Nov 1, 2021
@BaguaSys BaguaSys deleted a comment from todo Bot Nov 1, 2021
@BaguaSys BaguaSys deleted a comment from todo Bot Nov 1, 2021
@BaguaSys BaguaSys deleted a comment from todo Bot Nov 1, 2021
@BaguaSys BaguaSys deleted a comment from todo Bot Nov 1, 2021
@BaguaSys BaguaSys deleted a comment from todo Bot Nov 1, 2021
@BaguaSys BaguaSys deleted a comment from todo Bot Nov 1, 2021
@BaguaSys BaguaSys deleted a comment from todo Bot Nov 1, 2021
@NOBLES5E NOBLES5E merged commit 8707188 into master Nov 2, 2021
@NOBLES5E NOBLES5E deleted the fuse-opt branch November 2, 2021 01:18
This was referenced Nov 2, 2021
Open
Closed
Open
Open
@wangraying wangraying restored the fuse-opt branch November 2, 2021 03:39
@wangraying wangraying deleted the fuse-opt branch November 2, 2021 03:41
This was referenced Nov 4, 2021
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@github-actions github-actions[bot] github-actions[bot] left review comments

@NOBLES5E NOBLES5E Awaiting requested review from NOBLES5E NOBLES5E is a code owner

Assignees

No one assigned

Labels

PR: merged

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@wangraying @NOBLES5E