Codestin Search App

NOBLES5E · 2021-07-07T08:16:15Z

No description provided.

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

…s/bagua into feat/async-model-average

wangraying · 2021-08-19T08:51:49Z

update @NOBLES5E

It seems using LL128 protocol will cause the problem to hang. I have reproduced the problem through a sample code backend by torch+cupy. This is consistent with the results on Bagua.

Using the other two protocols LL and Simple does not hang as far as I know.

I have posted an issue on github, which we could track later on.

todo · 2021-08-20T03:48:53Z

remove nccl proto check

bagua/bagua/torch_api/algorithms/async_model_average.py

Lines 14 to 19 in f550bcd

    
           # TODO: remove nccl proto check 
        
           proto_str = os.environ.get("NCCL_PROTO", "") 
        
           if ( 
        
               proto_str == "" 
        
               or ("^" not in proto_str and "LL128" in proto_str) 
        
               or ("^" in proto_str and "LL128" not in proto_str)

This comment was generated by todo based on a `TODO` comment in `f550bcd` in #110. cc @BaguaSys.

wangraying · 2021-08-20T05:52:39Z

@wangraying fix CI so that we can merge this
We also need a tutorial page on this algorithm

This CI seems broken.

wangraying · 2021-08-20T06:01:45Z

And do you have any further comment about the usage, since we add a function barrier to end async threads.

@NOBLES5E

todo · 2021-08-23T03:24:13Z

; remove this after NVIDIA/nccl#549 gets solved

bagua/bagua/torch_api/algorithms/async_model_average.py

Lines 23 to 28 in 1c33cf9

    
                   ) # TODO; remove this after https://github.com/NVIDIA/nccl/issues/549 gets solved 
        
           class AsyncModelAverageAlgorithm(Algorithm): 
        
               def __init__( 
        
                   self, peer_selection_mode: str = "all", sync_interval_ms: int = 500,

This comment was generated by todo based on a `TODO` comment in `1c33cf9` in #110. cc @BaguaSys.

NOBLES5E

I refined the texts and left some comments. @wangraying Please do a final check

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

NOBLES5E added 4 commits July 2, 2021 18:05

chore(release): v0.6.2

27411a9

Merge branch 'master' of github.com:BaguaSys/bagua

1fa5f1f

Merge branch 'master' of github.com:BaguaSys/bagua

a7e334a

update

69e8bf2

pr-triage Bot added the PR: draft label Jul 7, 2021

github-actions Bot reviewed Jul 7, 2021

View reviewed changes

Comment thread bagua/torch_api/algorithms/async_model_average.py

Comment thread bagua/torch_api/algorithms/async_model_average.py

Comment thread bagua/torch_api/algorithms/async_model_average.py Outdated

wangraying added 4 commits July 24, 2021 23:11

Merge branch 'master' into feat/async-model-average

de05ee6

update

3e65319

add async

ee40323

add comment

adcb1e3

github-actions Bot reviewed Aug 6, 2021

View reviewed changes

Comment thread bagua/torch_api/bucket.py Outdated

wangraying and others added 2 commits August 6, 2021 17:03

Update bagua/torch_api/bucket.py

cb6ad87

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

update

5b5f793

wangraying requested a review from a team August 6, 2021 10:13

wangraying marked this pull request as ready for review August 6, 2021 10:13

pr-triage Bot added PR: unreviewed and removed PR: draft labels Aug 6, 2021

update

7196a3d

github-actions Bot reviewed Aug 6, 2021

View reviewed changes

Comment thread bagua/torch_api/algorithms/async_model_average.py

Comment thread bagua/torch_api/algorithms/async_model_average.py

wangraying and others added 2 commits August 6, 2021 18:48

Merge branch 'master' into feat/async-model-average

891fa9a

Apply suggestions from code review

7625f6f

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

github-actions Bot reviewed Aug 6, 2021

View reviewed changes

Comment thread bagua/torch_api/algorithms/async_model_average.py Outdated

wangraying and others added 4 commits August 6, 2021 21:27

Apply suggestions from code review

b4285c8

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

.

6c3274b

fix

9460589

.

e97e17d

github-actions Bot reviewed Aug 10, 2021

View reviewed changes

Comment thread bagua/torch_api/algorithms/__init__.py Outdated

Comment thread tests/torch_api/test_async_model_average.py

wangraying and others added 3 commits August 12, 2021 23:02

Apply suggestions from code review

42a4143

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

little change

edc56ec

Merge branch 'feat/async-model-average' of https://github.com/BaguaSy…

1acd2d5

…s/bagua into feat/async-model-average

add

f550bcd

pr-triage Bot removed the PR: draft label Aug 20, 2021

pr-triage Bot added the PR: unreviewed label Aug 20, 2021

fmt

8dc05a2

github-actions Bot reviewed Aug 20, 2021

View reviewed changes

Comment thread bagua/torch_api/algorithms/__init__.py Outdated

wangraying added 2 commits August 20, 2021 12:02

fmt

28c85c3

.

af18148

Update async_model_average.py

1c33cf9

github-actions Bot reviewed Aug 23, 2021

View reviewed changes

Comment thread bagua/torch_api/algorithms/async_model_average.py Outdated

NOBLES5E commented Aug 23, 2021

View reviewed changes

Comment thread bagua/torch_api/algorithms/async_model_average.py Outdated

Update async_model_average.py

7246854

NOBLES5E commented Aug 23, 2021

View reviewed changes

Comment thread bagua/torch_api/algorithms/async_model_average.py

Update bucket.py

7dd0526

NOBLES5E commented Aug 23, 2021

View reviewed changes

github-actions Bot reviewed Aug 23, 2021

View reviewed changes

Comment thread bagua/torch_api/bucket.py Outdated

NOBLES5E and others added 3 commits August 22, 2021 20:37

Update async_model_average.py

27fa44b

Apply suggestions from code review

e8446d8

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

uu

cf7b15f

NOBLES5E merged commit fcef9ef into master Aug 23, 2021

pr-triage Bot removed the PR: unreviewed label Aug 23, 2021

NOBLES5E deleted the feat/async-model-average branch August 23, 2021 04:53

todo Bot mentioned this pull request Aug 23, 2021

remove nccl proto check #160

Closed

pr-triage Bot added the PR: merged label Aug 23, 2021

todo Bot mentioned this pull request Aug 23, 2021

; remove this after https://github.com/NVIDIA/nccl/issues/549 gets solved #161

Closed

NOBLES5E pushed a commit that referenced this pull request Aug 23, 2021

feat: add async model average algorithm (#110)

96c7bd5

Conversation

NOBLES5E commented Jul 7, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wangraying commented Aug 19, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

todo Bot commented Aug 20, 2021

remove nccl proto check

This comment was generated by todo based on a TODO comment in f550bcd in #110. cc @BaguaSys.

Uh oh!

Uh oh!

wangraying commented Aug 20, 2021

Uh oh!

wangraying commented Aug 20, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

todo Bot commented Aug 23, 2021

; remove this after NVIDIA/nccl#549 gets solved

This comment was generated by todo based on a TODO comment in 1c33cf9 in #110. cc @BaguaSys.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

NOBLES5E left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

wangraying commented Aug 19, 2021 •

edited

Loading

This comment was generated by todo based on a `TODO` comment in `f550bcd` in #110. cc @BaguaSys.

wangraying commented Aug 20, 2021 •

edited

Loading

This comment was generated by todo based on a `TODO` comment in `1c33cf9` in #110. cc @BaguaSys.