[dynamo][super variable] Fix bug to use correct source #151154

anijain2305 · 2025-04-12T01:49:57Z

Stack from ghstack (oldest at bottom):

-> [dynamo][super variable] Fix bug to use correct source #151154

We should cherry-pick to 2.7 branch if possible, because this breaks torch.compile on some HF models. Look at the issue referenced here.

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @chenyang78 @kadeng @chauhang @amjames

Will add test in a follow up PR, because this PR might need to be cherry-picked to 2.7. So keeping the PR very simple to cherry-pick [ghstack-poisoned]

pytorch-bot · 2025-04-12T01:50:00Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/151154

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit bf0eb70 with merge base 2653498 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

anijain2305 · 2025-04-12T04:47:03Z

Asked for review too early. CI failures say that something is really wrong. Converting to draft.

Will add test in a follow up PR, because this PR might need to be cherry-picked to 2.7. So keeping the PR very simple to cherry-pick Fixes #150994 cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx chenyang78 kadeng chauhang amjames [ghstack-poisoned]

Will add test in a follow up PR, because this PR might need to be cherry-picked to 2.7. So keeping the PR very simple to cherry-pick ghstack-source-id: 7d03594 Pull Request resolved: #151154

Fixes #150994 We should cherry-pick to 2.7 branch if possible, because this breaks torch.compile on some HF models. Look at the issue referenced here. cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx chenyang78 kadeng chauhang amjames [ghstack-poisoned]

anijain2305 · 2025-04-12T17:36:22Z

benchmarks/dynamo/ci_expected_accuracy/aot_eager_huggingface_training.csv



-LayoutLMForSequenceClassification,pass,5
+LayoutLMForSequenceClassification,pass,6


Both GPT2 and Layout are correct increases. There is a legit recompilation

we were not guarding on self.config.problem_type which is None in the first run, and initiailaized there.

Fixes #150994 We should cherry-pick to 2.7 branch if possible, because this breaks torch.compile on some HF models. Look at the issue referenced here. cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx chenyang78 kadeng chauhang amjames [ghstack-poisoned]

Will add test in a follow up PR, because this PR might need to be cherry-picked to 2.7. So keeping the PR very simple to cherry-pick ghstack-source-id: 4a519e6 Pull Request resolved: #151154

anijain2305 · 2025-04-13T04:41:40Z

@pytorchbot merge

pytorchmergebot · 2025-04-13T04:43:19Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

anijain2305 · 2025-04-14T17:48:09Z

@pytorchbot cherry-pick --onto release/2.7 -c critical

pytorchbot · 2025-04-14T17:53:17Z

Cherry picking #151154

Command git -C /home/runner/work/pytorch/pytorch cherry-pick -x 7b1a2373e87f5a2a3582cb4e37e47a7c544edf7a returned non-zero exit code 1

Auto-merging test/dynamo/test_misc.py
Auto-merging torch/_dynamo/guards.py
Auto-merging torch/_dynamo/source.py
Auto-merging torch/_dynamo/variables/misc.py
CONFLICT (content): Merge conflict in torch/_dynamo/variables/misc.py
Auto-merging torch/csrc/dynamo/guards.cpp
error: could not apply 7b1a2373e87... [dynamo][super variable] Fix bug to use correct source (#151154)
hint: After resolving the conflicts, mark them with
hint: "git add/rm <pathspec>", then run
hint: "git cherry-pick --continue".
hint: You can instead skip this commit with "git cherry-pick --skip".
hint: To abort and get back to the state before "git cherry-pick",
hint: run "git cherry-pick --abort".
hint: Disable this message with "git config set advice.mergeConflict false"

Details for Dev Infra team

Raised by workflow job

ydshieh · 2025-04-14T18:07:58Z

Hi @anijain2305 Thank you a lot for the fix. If I want to try the fix, what would the best approach? Like a wheel for torch 2.7 RC with this fix included? Or checkout to branch/tag release/2.7 and build torch from there? I can't see release/2.7 with this fix applied despite I saw you commented cherry-pick --onto release/2.7 -c critical

Fixes pytorch#150994 We should cherry-pick to 2.7 branch if possible, because this breaks torch.compile on some HF models. Look at the issue referenced here. Pull Request resolved: pytorch#151154 Approved by: https://github.com/jansel

gante · 2025-04-24T15:10:11Z

👋 I have the same question as @ydshieh -- what's the best way to try the fix?

We noticed that torch.compile on HF VLMs/LLMs, with torch 2.7, results in large slowdowns, so we're pinning the max version to torch 2.6 for now: huggingface/transformers#37760

atalman · 2025-04-25T19:37:32Z

adding to 2.7.1

anijain2305 · 2025-04-25T21:23:53Z

@gante @ydshieh We apologize for the regression with 2.7. Please allow us till Monday to check if this PR indeed resolves the regression. We can figure out how to proceed then.

ydshieh · 2025-04-29T10:26:20Z

Hi, share an observation I found. Today I tried to update our docker images to use torch 2.7(+cpu), and find we find the same issues happens for 6 tests, see this run

For example: running

python -m pytest -v tests/models/gemma3/test_modeling_gemma3.py::Gemma3Vision2TextModelTest::test_generate_compilation_all_outputs

gives

FAILED tests/models/gemma3/test_modeling_gemma3.py::Gemma3Vision2TextModelTest::test_generate_compilation_all_outputs


 - torch._dynamo.exc.Unsupported: Unexpected type in sourceless builder transformers.models.gemma3.configuration_gemma3.Gemma3TextConfig

from user code:
   File "/usr/local/lib/python3.9/site-packages/transformers/utils/generic.py", line 969, in wrapper
    output = func(self, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/transformers/models/gemma3/modeling_gemma3.py", line 1320, in forward
    causal_mask = self._update_causal_mask(
  File "/usr/local/lib/python3.9/site-packages/transformers/models/gemma3/modeling_gemma3.py", line 1115, in _update_causal_mask
    if self.config.text_config._attn_implementation == "flash_attention_2":
  File "/usr/local/lib/python3.9/site-packages/transformers/configuration_utils.py", line 211, in __getattribute__
    return super().__getattribute__(key)

Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo"
FAILED tests/models/gemma3/test_modeling_gemma3.py::Gemma3Vision2TextModelTest::test_generate_compile_model_forward - torch._dynamo.exc.Unsupported: Unexpected type in sourceless builder transformers.models.gemma3.configuration_gemma3.Gemma3TextConfig

from user code:
   File "/usr/local/lib/python3.9/site-packages/transformers/utils/generic.py", line 969, in wrapper
    output = func(self, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/transformers/models/gemma3/modeling_gemma3.py", line 1320, in forward
    causal_mask = self._update_causal_mask(
  File "/usr/local/lib/python3.9/site-packages/transformers/models/gemma3/modeling_gemma3.py", line 1115, in _update_causal_mask
    if self.config.text_config._attn_implementation == "flash_attention_2":
  File "/usr/local/lib/python3.9/site-packages/transformers/configuration_utils.py", line 211, in __getattribute__
    return super().__getattribute__(key)

Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo"
===== 2 failed, 1267 passed, 406 skipped, 46 warnings in 143.60s (0:02:23) =====

anijain2305 · 2025-05-05T04:16:43Z

@ydshieh @gante We are cherry-picking this PR (along with the other cudagraph bug PR) for 2.7.1 - #152774

I have tested above test passes with the PR built on top of 2.7 (and I verified that if failed on 2.7). Once the PR is merged to a release branch, it would be really helpful to do some extra testing from your side.

[dynamo][super variable] Fix bug to use correct source

2c4e3ad

Will add test in a follow up PR, because this PR might need to be cherry-picked to 2.7. So keeping the PR very simple to cherry-pick [ghstack-poisoned]

pytorch-bot bot added ciflow/inductor module: dynamo labels Apr 12, 2025

anijain2305 requested a review from jansel April 12, 2025 01:50

anijain2305 marked this pull request as draft April 12, 2025 04:46

anijain2305 removed the request for review from jansel April 12, 2025 04:47

anijain2305 added topic: not user facing topic category keep-going Don't stop on first failure, keep running tests until the end labels Apr 12, 2025

anijain2305 commented Apr 12, 2025

View reviewed changes

anijain2305 marked this pull request as ready for review April 12, 2025 17:37

anijain2305 added the ciflow/trunk Trigger trunk jobs on your pull request label Apr 12, 2025

anijain2305 requested review from StrongerXi, jansel, tugsbayasgalan and zou3519 April 12, 2025 17:38

jansel approved these changes Apr 13, 2025

View reviewed changes

pytorchmergebot added the merging label Apr 13, 2025

pytorchmergebot added the Merged label Apr 13, 2025

pytorchmergebot closed this in 7b1a237 Apr 13, 2025

pytorchmergebot removed the merging label Apr 13, 2025

anijain2305 mentioned this pull request Apr 14, 2025

torch.export and torch.compile in torch 2.7 RC fails some cases that work with torch 2.6 #150994

Closed

atalman added this to the 2.7.1 milestone Apr 25, 2025

gante mentioned this pull request Apr 30, 2025

Qwen model export regression huggingface/transformers#37876

Closed

4 tasks

anijain2305 mentioned this pull request May 5, 2025

[v2.7.1] Release Tracker #152627

Closed

ZainRizvi mentioned this pull request May 27, 2025

[Release Impovements] Ensure cherry-pick failures get noticed pytorch/test-infra#6669

Open

atalman mentioned this pull request May 28, 2025

Release 2.7.1 validations checklist and cherry-picks #154512

Closed

49 tasks

github-actions bot deleted the gh/anijain2305/721/head branch June 7, 2025 02:18



		LayoutLMForSequenceClassification,pass,5
		LayoutLMForSequenceClassification,pass,6

[dynamo][super variable] Fix bug to use correct source #151154

[dynamo][super variable] Fix bug to use correct source #151154

Uh oh!

Conversation

anijain2305 commented Apr 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Apr 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/151154

✅ No Failures

Uh oh!

anijain2305 commented Apr 12, 2025

Uh oh!

anijain2305 Apr 12, 2025

Choose a reason for hiding this comment

Uh oh!

anijain2305 commented Apr 13, 2025

Uh oh!

pytorchmergebot commented Apr 13, 2025

Merge started

Uh oh!

anijain2305 commented Apr 14, 2025

Uh oh!

pytorchbot commented Apr 14, 2025

Cherry picking #151154

Uh oh!

ydshieh commented Apr 14, 2025

Uh oh!

gante commented Apr 24, 2025

Uh oh!

atalman commented Apr 25, 2025

Uh oh!

anijain2305 commented Apr 25, 2025

Uh oh!

ydshieh commented Apr 29, 2025

Uh oh!

anijain2305 commented May 5, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

anijain2305 commented Apr 12, 2025 •

edited

Loading

pytorch-bot bot commented Apr 12, 2025 •

edited

Loading