Thanks to visit codestin.com
Credit goes to github.com

Skip to content

[c10d] Turn off default non-blocking API mode to work around hang in NCCL 2.26#154055

Closed
kwen2501 wants to merge 1 commit into
gh/kwen2501/155/basefrom
gh/kwen2501/155/head
Closed

[c10d] Turn off default non-blocking API mode to work around hang in NCCL 2.26#154055
kwen2501 wants to merge 1 commit into
gh/kwen2501/155/basefrom
gh/kwen2501/155/head

Conversation

@kwen2501
Copy link
Copy Markdown
Collaborator

@kwen2501 kwen2501 commented May 21, 2025 •
edited
Loading

Stack from ghstack (oldest at bottom):

Work around issues like #153960, #152623

NCCL 2.26 seems to introduce random hang in non-blocking API mode. This PR opts out of non-blocking mode to work around it. Previously torch turned it on by default in eager init (i.e. device_id passed) to avoid init overhead.

cc @H-Huang @awgu @wanchaol @fegin @fduwjj @wz337 @wconstab @d4l3k

@kwen2501
Update
e6b5bbe 
[ghstack-poisoned]
@pytorch-bot
Copy link
Copy Markdown

pytorch-bot Bot commented May 21, 2025 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/154055

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit e6b5bbe with merge base fa85434 (image):
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

kwen2501 added a commit that referenced this pull request May 21, 2025
@kwen2501
[c10d] Turn off default non-blocking API mode to work around hang in …
f7ab6d1 
…NCCL 2.26

ghstack-source-id: e3cd0a6
Pull-Request-resolved: #154055
@pytorch-bot pytorch-bot Bot added oncall: distributed Add this issue/PR to distributed oncall triage queue release notes: distributed (c10d) release notes category labels May 21, 2025
@kwen2501 kwen2501 requested review from d4l3k, eqy, fduwjj and ngimel May 21, 2025 20:44
@kwen2501 kwen2501 added the topic: bug fixes topic category label May 21, 2025
nWEIdia
nWEIdia reviewed May 21, 2025
View reviewed changes
Comment thread torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp
// hang in NCCL 2.26 in non-blocking mode. We can revisit if NCCL fixes the
// bug. See https://github.com/pytorch/pytorch/issues/153960
// else if (getBoundDeviceId()) {
// useNonblocking_ = true;
Copy link
Copy Markdown
Collaborator

@nWEIdia nWEIdia May 21, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any existing unit test that would need to be adjusted when this option is toggled on or off?

Copy link
Copy Markdown
Collaborator Author

@kwen2501 kwen2501 May 21, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The flag is more of an internal choice than contract.
There are several tests that passes device_id, so hopefully they don't break.

Copy link
Copy Markdown
Collaborator

@nWEIdia nWEIdia May 23, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was hoping that this would fix the test_non_blocking_with_eager_init
but with v2.7.1RC, (docker pull http://ghcr.io/pytorch/pytorch-test:2.7.1-cuda12.6-cudnn9-runtime), I am still reproducing the timeout/hang:

root@d70999cd4c34:/my_workspace/wei-pytorch/test/distributed# python test_c10d_nccl.py -v -k test_non_blocking_with_eager_init test_non_blocking_with_eager_init (__main__.ProcessGroupNCCLGroupTest.test_non_blocking_with_eager_init) ...

Copy link
Copy Markdown
Collaborator

@nWEIdia nWEIdia May 23, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am switching to a platform that has better OS (was previously using a ubuntu 20.04 based system and there could be known issues).
But now encountering with v2.7.1RC:
ModuleNotFoundError: No module named 'torch.distributed._spmd'
update: used wrong (runtime) container, should use devel container. Never mind on this command.
cc @atalman

Copy link
Copy Markdown
Collaborator Author

@kwen2501 kwen2501 May 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test didn't hang for me. On H100 machine.

Copy link
Copy Markdown
Collaborator

@nWEIdia nWEIdia May 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I confirm this test does not hang for me as well on H100. I would follow up internally on the potential issues with the ubuntu 20.04 stack.
Below is on H100
`python3 test_c10d_nccl.py -v -k test_non_blocking_with_eager_init
test_non_blocking_with_eager_init (main.ProcessGroupNCCLGroupTest.test_non_blocking_with_eager_init) ... ok

Ran 1 test in 10.152s

OK
`

Though this potentially mean that even if your change lands on main, the upstream CI may still hang, due to the potential OS related issue. I would double check on this front (ubuntu 20.04 or Amazon Linux 2023 + SM75) distributed.
Below is what I get from ubuntu 20.04 based host + ghcr.io/pytorch/pytorch-test:2.7.1-cuda12.6-cudnn9-devel on T4x2

root@d6abe3d5c3dd:/workspace/pytorch/test/distributed# time timeout 30 python3 test_c10d_nccl.py -v -k test_non_blocking_with_eager_init
test_non_blocking_with_eager_init (main.ProcessGroupNCCLGroupTest.test_non_blocking_with_eager_init) ...
real 0m30.040s (i.e. hang)
user 0m4.055s
sys 0m2.664s

atalman
atalman approved these changes May 21, 2025
View reviewed changes
Copy link
Copy Markdown
Collaborator

@atalman atalman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@kwen2501
Copy link
Copy Markdown
Collaborator Author

kwen2501 commented May 21, 2025

@pytorchbot merge -f "Unblocking an urgent issue"

@pytorchmergebot
Copy link
Copy Markdown
Collaborator

pytorchmergebot commented May 21, 2025

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@atalman
Copy link
Copy Markdown
Collaborator

atalman commented May 22, 2025

@pytorchbot cherry-pick --onto release/2.7 -c critical

pytorchbot pushed a commit that referenced this pull request May 22, 2025
@kwen2501 @pytorchbot
[c10d] Turn off default non-blocking API mode to work around hang in …
fecbe87 
…NCCL 2.26 (#154055)

Work around issues like #153960, #152623

NCCL 2.26 seems to introduce random hang in non-blocking API mode. This PR opts out of non-blocking mode to work around it. Previously torch turned it on by default in eager init (i.e. `device_id` passed) to avoid init overhead.

Pull Request resolved: #154055
Approved by: https://github.com/atalman

(cherry picked from commit 87fc5af)
@pytorchbot
Copy link
Copy Markdown
Collaborator

pytorchbot commented May 22, 2025

Cherry picking #154055

The cherry pick PR is at #154085 and it is recommended to link a critical cherry pick PR with an issue. The following tracker issues are updated:

Details for Dev Infra team Raised by workflow job

This was referenced May 22, 2025
Merged
Closed
atalman pushed a commit that referenced this pull request May 22, 2025
@pytorchbot @kwen2501
[c10d] Turn off default non-blocking API mode to work around hang in …
ff4dddf 
…NCCL 2.26 (#154085)

[c10d] Turn off default non-blocking API mode to work around hang in NCCL 2.26 (#154055)

Work around issues like #153960, #152623

NCCL 2.26 seems to introduce random hang in non-blocking API mode. This PR opts out of non-blocking mode to work around it. Previously torch turned it on by default in eager init (i.e. `device_id` passed) to avoid init overhead.

Pull Request resolved: #154055
Approved by: https://github.com/atalman

(cherry picked from commit 87fc5af)

Co-authored-by: Ke Wen <[email protected]>
This was referenced May 22, 2025
Closed
Closed
@github-actions github-actions Bot deleted the gh/kwen2501/155/head branch June 23, 2025 02:20
@kwen2501 kwen2501 mentioned this pull request Dec 3, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@nWEIdia nWEIdia nWEIdia left review comments

@atalman atalman atalman approved these changes

@d4l3k d4l3k Awaiting requested review from d4l3k

@fduwjj fduwjj Awaiting requested review from fduwjj

@eqy eqy Awaiting requested review from eqy

@ngimel ngimel Awaiting requested review from ngimel

Assignees

No one assigned

Labels

Merged oncall: distributed Add this issue/PR to distributed oncall triage queue release notes: distributed (c10d) release notes category topic: bug fixes topic category

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@kwen2501 @pytorchmergebot @atalman @pytorchbot @nWEIdia