Fix vs2022 caused AVX512 illegal instruction issue. #153480

xuhancn · 2025-05-13T18:26:34Z

Add /d2implyavx512upperregs- to disable compiler over-aggressive optimization, which caused involeved AVX512 register on AVX2 machine.

Reference to: #145702 (comment)

Local test passed:

cc @peterjc123 @mszhanyi @skyline75489 @nbcsm @iremyux @Blackhex @jgong5 @mingfeima @XiaobingSuper @sanchitintel @ashokei @jingxu10 @jerryzh168

pytorch-bot · 2025-05-13T18:26:38Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/153480

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 1 Unrelated Failure

As of commit 99978ab with merge base dccd19c ():

NEW FAILURE - The following job has failed:

xpu / linux-jammy-xpu-2025.1-py3.9 / test (default, 6, 6, linux.idc.xpu) (gh)
inductor/test_pattern_matcher.py::TestPatternMatcher::test_bmm_to_mm

UNSTABLE - The following job is marked as unstable, possibly due to flakiness on trunk:

pull / cuda12.4-py3.10-gcc9-sm75 / test (pr_time_benchmarks, 1, 1, linux.g4dn.metal.nvidia.gpu) (gh) (#149370)
MISSING REGRESSION TEST

This comment was automatically generated by Dr. CI and updates every 15 minutes.

Blackhex

LGTM. If it's desired, I'd suggest creating a GitHub issue to solve the root cause - removal of AVX512 intrinsic from code targeting AVX2 - and linking it to this PR.

xuhancn · 2025-05-16T02:55:32Z

LGTM. If it's desired, I'd suggest creating a GitHub issue to solve the root cause - removal of AVX512 intrinsic from code targeting AVX2 - and linking it to this PR.

Upgrade VS2022 to latest can fix this issue, but I don't known how, please check here: #145702 (comment) @Blackhex can you check with Visual studio team to seek a solution?

xuhancn · 2025-05-16T06:42:08Z

@pytorchbot rebase

pytorchmergebot · 2025-05-16T06:43:46Z

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

pytorchmergebot · 2025-05-16T06:43:50Z

Successfully rebased xu_fix_vs2022_illegal_instruction onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout xu_fix_vs2022_illegal_instruction && git pull --rebase)

Blackhex · 2025-05-16T09:25:37Z

Upgrade VS2022 to latest can fix this issue, but I don't known how, please check here: #145702 (comment) @Blackhex can you check with Visual studio team to seek a solution?

IMO, the https://github.com/pytorch/pytorch/blob/56e7d38b9f9ee727caf7b86aa0cca5088c94a489/.ci/pytorch/windows/internal/vc_install_helper.bat script is not used for PR builds that are done with https://github.com/pytorch/pytorch/blob/56e7d38b9f9ee727caf7b86aa0cca5088c94a489/.ci/pytorch/win-test-helpers/build_pytorch.bat script which probably assumes the build tools already installed on the image. If that's true, it should be updated on AMI image or installation added (https://github.com/Blackhex/pytorch-test-infra/blob/main/aws/ami/windows/scripts/Installers/Install-VS.ps1). The installation script should be used for nightly wheels builds. Can you confirm or decline whether the update of build tools work there?

PS: I am discussing with VS team what is the reason that the issue is not happening with newer build tools. I will need a small C++ repro case for that which will take some time to dissect from the codebase, unless someone already did that?

xuhancn · 2025-05-19T12:03:56Z

@pytorchbot rebase

pytorchmergebot · 2025-05-19T12:05:32Z

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

pytorchmergebot · 2025-05-19T12:05:35Z

Successfully rebased xu_fix_vs2022_illegal_instruction onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout xu_fix_vs2022_illegal_instruction && git pull --rebase)

xuhancn · 2025-05-19T20:06:12Z

@pytorchbot rebase

pytorchmergebot · 2025-05-19T20:08:08Z

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

cmake/public/utils.cmake

xuhancn · 2025-05-20T08:21:33Z

Double confirmed the latest changes.

atalman

lgtm

xuhancn · 2025-05-20T18:14:07Z

@pytorchbot merge

pytorchmergebot · 2025-05-20T18:16:39Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2025-05-20T18:54:14Z

Merge failed

Reason: 1 jobs have failed, first few of them are: xpu / linux-jammy-xpu-2025.1-py3.9 / test (default, 6, 6, linux.idc.xpu)

Details for Dev Infra team

Raised by workflow job

xuhancn · 2025-05-20T20:24:48Z

@pytorchbot merge

pytorchmergebot · 2025-05-20T20:27:16Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2025-05-20T20:27:36Z

Merge failed

Reason: 1 jobs have failed, first few of them are: xpu / linux-jammy-xpu-2025.1-py3.9 / test (default, 6, 6, linux.idc.xpu)

Details for Dev Infra team

Raised by workflow job

xuhancn · 2025-05-20T20:29:04Z

@pytorchbot merge -i

pytorchmergebot · 2025-05-20T20:31:09Z

Merge started

Your change will be merged while ignoring the following 2 checks: pull / cuda12.4-py3.10-gcc9-sm75 / test (pr_time_benchmarks, 1, 1, linux.g4dn.metal.nvidia.gpu), xpu / linux-jammy-xpu-2025.1-py3.9 / test (default, 6, 6, linux.idc.xpu)

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

This reverts commit e4f2282. I believe fix PR was landed #153480 that triggered the revert. Hence this is reland. Pull Request resolved: #155478 Approved by: https://github.com/malfet

malfet · 2025-07-29T21:12:04Z

I couldn't find any documentation for d2implyavx512upperregs....

Blackhex · 2025-07-30T07:14:45Z

Yeah, this flag is undocumented. I found out this is a solution to the problem from public (and internal) discussions like https://developercommunity.visualstudio.com/t/Code-gen-bug-uses-ymm16-register-for-AVX/10564317

xuhancn added topic: not user facing topic category ciflow/binaries Trigger all binary build and upload jobs on the PR module: windows Windows support for PyTorch ciflow/trunk Trigger trunk jobs on your pull request intel This tag is for PR from Intel labels May 13, 2025

pytorchbot added the open source label May 13, 2025

xuhancn mentioned this pull request May 15, 2025

PyTorch VS2022 official build Windows binary illegal instruction on AVX2(max ISA level) CPU #145702

Open

xuhancn requested review from Blackhex, atalman and malfet May 15, 2025 17:27

xuhancn changed the title ~~fix vs2022 illegal instruction.~~ Fix vs2022 caused AVX512 illegal instruction issue. May 15, 2025

xuhancn marked this pull request as ready for review May 15, 2025 17:28

xuhancn requested a review from a team as a code owner May 15, 2025 17:28

xuhancn added the module: cpu CPU specific problem (e.g., perf, algorithm) label May 15, 2025

Blackhex approved these changes May 15, 2025

View reviewed changes

pytorchmergebot force-pushed the xu_fix_vs2022_illegal_instruction branch from a6214c5 to 56e7d38 Compare May 16, 2025 06:43

pytorchmergebot force-pushed the xu_fix_vs2022_illegal_instruction branch from 56e7d38 to cd0344d Compare May 19, 2025 12:05

xuhancn added the ciflow/xpu Run XPU CI tasks label May 19, 2025

fix vs2022 illegal instruction.

58089c2

xuhancn force-pushed the xu_fix_vs2022_illegal_instruction branch from ffa3a12 to ad4ada3 Compare May 20, 2025 05:40

cyyever approved these changes May 20, 2025

View reviewed changes

cyyever reviewed May 20, 2025

View reviewed changes

cmake/public/utils.cmake Show resolved Hide resolved

remove debug log.

99978ab

atalman approved these changes May 20, 2025

View reviewed changes

xuhancn mentioned this pull request May 20, 2025

[Windows][CPU] Root cause the VS2022 illegal instruction #153961

Open

pytorchmergebot added the merging label May 20, 2025

pytorchmergebot removed the merging label May 20, 2025

pytorchmergebot added the merging label May 20, 2025

pytorchmergebot removed the merging label May 20, 2025

pytorchmergebot added the merging label May 20, 2025

pytorchmergebot added the Merged label May 20, 2025

pytorchmergebot closed this in 179e7d8 May 20, 2025

xuhancn deleted the xu_fix_vs2022_illegal_instruction branch May 20, 2025 20:37

pytorchmergebot removed the merging label May 20, 2025

atalman mentioned this pull request Jun 9, 2025

Reapply "Cleanup VS 2019 refs in pytorch (#145863)" (#152613) #155478

Closed

Fix vs2022 caused AVX512 illegal instruction issue. #153480

Fix vs2022 caused AVX512 illegal instruction issue. #153480

Uh oh!

Conversation

xuhancn commented May 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented May 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/153480

❌ 1 New Failure, 1 Unrelated Failure

Uh oh!

Blackhex left a comment

Choose a reason for hiding this comment

Uh oh!

xuhancn commented May 16, 2025

Uh oh!

xuhancn commented May 16, 2025

Uh oh!

pytorchmergebot commented May 16, 2025

Uh oh!

pytorchmergebot commented May 16, 2025

Uh oh!

Blackhex commented May 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

xuhancn commented May 19, 2025

Uh oh!

pytorchmergebot commented May 19, 2025

Uh oh!

pytorchmergebot commented May 19, 2025

Uh oh!

xuhancn commented May 19, 2025

Uh oh!

pytorchmergebot commented May 19, 2025

Uh oh!

Uh oh!

xuhancn commented May 20, 2025

Uh oh!

atalman left a comment

Choose a reason for hiding this comment

Uh oh!

xuhancn commented May 20, 2025

Uh oh!

pytorchmergebot commented May 20, 2025

Merge started

Uh oh!

pytorchmergebot commented May 20, 2025

Merge failed

Uh oh!

xuhancn commented May 20, 2025

Uh oh!

pytorchmergebot commented May 20, 2025

Merge started

Uh oh!

pytorchmergebot commented May 20, 2025

Merge failed

Uh oh!

xuhancn commented May 20, 2025

Uh oh!

pytorchmergebot commented May 20, 2025

Merge started

Uh oh!

malfet commented Jul 29, 2025

Uh oh!

Blackhex commented Jul 30, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

xuhancn commented May 13, 2025 •

edited

Loading

pytorch-bot bot commented May 13, 2025 •

edited

Loading

Blackhex commented May 16, 2025 •

edited

Loading